Compare commits

...
Author SHA1 Message Date
Chris LuandClaude Opus 5.5 c7287f8f54 filer: send a read token on the cp.from source chunk GET
With jwt.signing.read set, volume servers refuse unsigned reads, so
cp.from of a chunked file failed with "source returned status 401".
Mint a read token for the chunk on each source attempt, the same way the
filer's own reads do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 04:03:29 -08:00
Chris LuandClaude Opus 5.5 e00698a327 helm: update the volume HTTPS caveats now that masters forward correctly
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 03:56:58 -08:00
Chris LuandClaude Opus 5.5 63a575f28f master: forward to the leader with the masters' own HTTP scheme
A follower took the scheme from [https.client], so with HTTPS clients
toward volume servers and plain-HTTP masters every forwarded request
(/dir/assign, /dir/status, the UI) failed with 502 "first record does not
look like a TLS handshake". Masters share one [https.master] setting, so
use HTTPS exactly when this master serves it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 03:56:47 -08:00
Chris LuandClaude Opus 5.5 ed67f9930c helm CI: check the volume HTTPS rollout annotation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 03:44:01 -08:00
Chris LuandClaude Opus 5.5 dd4d06e40f helm: restart every security.toml reader when volume HTTPS is toggled
security.toml is mounted with subPath and read once at startup, so on an
upgrade that turns volume HTTPS on, the volume servers restarted with TLS
while filer, S3 and the other clients kept speaking plain HTTP. A
checksum/volume-https pod annotation, rendered only while the option is
on, rolls them all.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 03:44:00 -08:00
Chris LuandClaude Opus 5.5 64d5fb5613 filer: copy chunks for cp.from through the shared HTTP client
The copy built a bare client and plain http:// chunk URLs, so with
[https.client] on it could not read from or write to HTTPS volume servers
("source returned status 400"). Use the global client's transport and
redirect policy, and normalize the source and destination URLs to its
scheme.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 03:42:22 -08:00
Chris LuandClaude Opus 5.5 e15aa250e9 helm: document volume server HTTPS
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 03:25:11 -08:00
Chris LuandClaude Opus 5.5 c908497d56 helm CI: check the volume server HTTPS rendering and its guards
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 03:24:09 -08:00
Chris LuandClaude Opus 5.5 9bc1a50b37 helm: reject volume HTTPS for the all-in-one deployment
Its volume server registers by pod IP, so HTTPS clients could not verify
the volume certificate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 03:23:40 -08:00
Chris LuandClaude Opus 5.5 dc2d13a1ee helm: probe volume servers over HTTPS when they serve it
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 03:23:29 -08:00
Chris LuandClaude Opus 5.5 af35fb158c helm: add the volume service names to the volume certificate for HTTPS
Volume servers register as <pod>.<volume service>.<namespace>, and HTTPS
clients verify that name, which none of the certificate's wildcards cover.
With volume HTTPS on, the certificate also lists every volume group's
service.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 03:23:12 -08:00
Chris LuandClaude Opus 5.5 080af98b86 helm: render volume server HTTPS in security.toml
global.seaweedfs.volumeHttps.enabled sets [https.volume] to the volume
certificate and turns on [https.client] with the client certificate and the
chart CA, so the volume data port is no longer plaintext under
enableSecurity. The option fails without enableSecurity.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 03:22:53 -08:00
Chris Lu 4c92a00357 admin/plugin: detect again soon after a capacity give-up (#12825)
A scheduled run whose jobs no executor can take gives them up at once,
leaving them to the next detection. For a job type with a long detection
interval that is a whole interval later: a daily job whose only worker
was briefly full lost a day.

Bring the job type's next detection forward to within 10 minutes when
its jobs are given up for capacity, never pushing back an earlier one.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
2026-11-20 00:31:18 -08:00
Chris LuandClaude Opus 5.5 1d7125abf2 filer: keep AppendToEntry's cluster lock for the whole append (#12819)
* filer: renew AppendToEntry's cluster lock while the append runs

AppendToEntry took a 5 s short-lived lock and never renewed it, so an
append slower than that (slow store, slow volume, GC pause) lost the
lock while still running, and another filer could take it and append
to the same entry concurrently, dropping one append's chunks. Use the
auto-renewing lock the HTTP append path already uses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: refuse an AppendToEntry commit once its cluster lock was lost

A renewal can still fail, or the lock can be lost and granted again
under a new generation, while the append runs; another filer may have
appended in between. Check the lock and its generation right before
the entry is written, fail the append so the caller retries, and drop
the manifest chunks this attempt created.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* cluster: IsLocked reports a lock whose lease ran out as not held

isLocked only flipped when a renewal came back rejected, a second after
the reply, so a holder whose renewal stalled past the lease (a long GC
pause) still read as locked while the server could hand the key to
someone else. Track the lease from each grant's send time and require
it in IsLocked, which the filer append paths check before committing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: hold the slow-append test past the renewing lock's first lease

The renewing lock grants a 7 s lease, so a 6 s hold passed even with
renewal off; wait past LiveLockTTL so only a renewal keeps the lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: say the AppendToEntry lock check is best effort

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 00:01:24 -08:00
Chris LuandClaude Opus 5.5 6018f87c25 test(s3vectors): add an author-filter workload to the perf suite (#12818)
* test(s3vectors): let a perf test restart the engine with extra environment

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test(s3vectors): serve the volume server's metrics in the test cluster

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test(s3vectors): build a named index in the perf suite through one helper

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test(s3vectors): add an author-filter workload to the perf suite

Plants authors at 10%, 1%, 0.01%, fewer than topK and no rows (seed 5),
scattered over the index or written as consecutive keys, and reports the
eligible rows, p50/p95, filtered recall@k against an exact numpy search
over the eligible rows, results per query, hits without a finite
distance, and the engine and volume server read counters, warm or with
the engine's caches at 0 MB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test(s3vectors): reject duplicate hits and clean up a failed filter workload setup

The workload skips the distributed cluster, since only one engine is
restarted, and accepts only the warm and cold modes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 23:48:58 -08:00
Chris LuandClaude Opus 5.5 4bbc03747f s3vectors: match json_extract's escaping of control characters in filter strings (#12817)
* s3vectors: render filter string control characters as serde_json does

json_extract serializes the stored value with serde_json: \b, \f, \n,
\r, \t, \u00XX for the rest of U+0000-U+001F, and U+007F-U+009F raw.
The scalar comparison text used Rust debug escapes (\0, \u{1}, \u{7f}),
so $eq/$in on such strings missed scalar values and $ne/$nin kept them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test(s3vectors): filter on every control character, quote and backslash

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 23:48:27 -08:00
Chris LuandClaude Opus 5.5 e109f91c6f vector: indexed metadata keys, engine side (derive, rewrite, exact planner, filter counter) (#12813)
* vector: derive the indexed metadata key columns

An indexed metadata key is a hidden `_mk_<n>` column of List<Utf8>: the
strings an S3 `$eq` on a string matches, that is the value when it is a
string or the string elements of an array, and nothing for any other type
or a missing key. The field metadata names the key and its state
(`building` or `ready`).

The module derives the column from metadata given as JSON (the write
path) or as stored (jsonb or JSON text, for a backfill), and reads the
indexed keys from a schema. It lives in the worker crate so the engine and
the worker share it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: fill the indexed metadata key columns on PutVectors

PutVectors builds its batch against the table's current schema. Once a
table has indexed metadata key columns, the three-column batch no longer
matches that schema and the write fails. Fill each `_mk_*` column the
schema has from the row's metadata, whatever the key's state; a column
the engine does not know still fails the request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: answer string $eq and $in on a ready indexed key from its column

The gateway renders a string `$eq` or `$in` on a key as an OR of
`json_extract(metadata, P) = '"v"'` (or IN) and
`json_array_contains(metadata, P, '"v"')` over the same values. On the
parsed filter, a top-level conjunct of that shape on a key whose column is
`ready` becomes `array_has` / `array_has_any` on the column, which its
LABEL_LIST index answers. The two halves must name the same key and the
same strings, the scalar half in the text json_extract returns for them;
anything else, including `$ne`, `$nin`, `$exists`, numbers and booleans,
stays on the JSON path, and the other conjuncts still apply.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: score few indexed matches exactly, per scope

With a ready indexed key in the filter, `ann` first counts the rows of its
scope that the key's conjuncts match, through the scalar index. When they
are at most the limit it scores them exactly over that scope, the way
the short-result fallback does; above it, the vector index searches with
the indexed prefilter, and the fallback still applies. The scope is the
one `exact_in_scope` scores, so per-scope exact answers merge into the
global exact top-k.

The limit is `Query::exact_match_limit` (None takes 256; 0 never counts),
set by the engine's --exact-match-limit (WEED_VECTOR_EXACT_MATCH_LIMIT).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: count filtered queries by metadata key

For each index, the engine counts its QueryVectors calls by the metadata
keys of the parsed filter, with their total latency and the operand types
each key was compared with, over the last hour in 5-minute buckets. At most
512 indexes and 32 keys per index are kept (an idle key makes room;
otherwise the index is marked truncated), and the stats are not exported
as metrics labels.

The new FilterStats RPC returns an index's stats; the gateway's discovery
client forwards it like the other calls.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: report the metadata columns capability in gRPC health

An engine whose writes fill the indexed metadata key columns reports the
gRPC health service `weed.vector.capability.metadata_columns` as serving,
so an operator action can refuse to index a key while an older engine
could still write the table.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: expire filter stats' operand kinds and truncation with the window

The operand kinds a key was compared with, and the mark that a key went
uncounted, outlived the queries that set them. Keep the kinds per
5-minute bucket and the truncation as the bucket it last happened in, so
both leave the hour window with the counts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: fill indexed key columns a write left empty racing their add

A PutVectors that opened the table before an indexed key's column was
added, and commits after the add, is rebased by Lance onto the wider
schema: it commits, and the rows it wrote hold NULL in the new column, so
the column's index misses them (reproduced in the test: 100 of 150 rows).

After its commit, PutVectors now looks for indexed-key columns the batch
did not carry, reads those of its keys whose column is still NULL, derives
the values from their stored metadata and patches just those columns with
a partial-schema merge that updates only rows still NULL.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: keep filter stats' truncation mark at the later bucket

record() reads the clock before it takes the lock, so a call that waited
on the lock across a bucket boundary could move the truncation mark (and
the index's last-used bucket) back to its earlier bucket, and the mark
then left the window one bucket early.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* vector: derive a raced write's indexed key columns from current metadata

A PutVectors that opened the table before an indexed key's column was
added is retried by Lance against the wider schema: rows it inserts hold
NULL there, and rows it updates keep the value derived from their old
metadata, which the NULL-only heal never looked at. The heal now derives
the columns for all of the write's keys.

Its merge also ran with Lance's retry, which joins the values derived
before the conflict onto the rows a later write of the same keys left.
It now fails on a conflict and is redone from the latest version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* vector: check the raced write's keys keep the later write's rows

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 23:45:30 -08:00
Chris LuandClaude Opus 5.5 817338c911 volume.merge, volume tail receiver: apply size-0 records for existing keys as deletes (#12816)
* volume: apply a tailed size-0 record as a delete when the key exists

A delete and an empty write both reach the tail receiver as a size-0
.dat record. Writing every one as a live needle turned each delete
caught by volume.move's tail into a 200-empty object. Treat the record
as a delete when the key already has a record on the receiver, and as
an empty write otherwise.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rust volume: apply a tailed size-0 record as a delete when the key exists

Mirrors the Go tail receiver: a size-0 record for a key that already
has a record here is a delete, otherwise an empty write.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: volume.merge applies a size-0 record as a delete when the key exists

The tail stream carries deletes as size-0 records, and volume.merge
wrote each one to the merged copy as a live empty needle, so every
delete in the volume came back as a 200-empty object on all replicas.
A size-0 record for a key already written in the merged stream is now
a delete on the target; for a new key it stays an empty write.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: volume.merge reads a size-0 record as a delete only after its own replica's record

Replicas append the same write at their own times, so a fresh empty write
can reach the merge twice when the copies are further apart than the
dedup window. Tracking keys across all replicas made the second copy a
delete. Track, per source replica, the keys it has returned (duplicates
the merge skips included): a size-0 record is a delete only when its own
replica wrote the key before, which is the only way a replica appends a
delete. A delete for a key the merged copy already deleted sends nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: volume.merge verification counts the deletes the merge applied

Applying deletes lets a correct merge hold fewer live needles than a
replica that missed a delete, and the live-count check then refused it
and kept the divergent replicas. Every key live on a replica is either
live or deleted in the merged copy, so compare that replica's live count
with the merged live count plus the keys the merge deleted. An empty
merged copy is still refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: tail receiver looks a size-0 record's key up in the needle map

The receiver read the needle to decide whether a size-0 record deletes
an existing key. A read reports a needle past its TTL as not found, so
a delete for it was written as a live empty needle, which reads ignore
TTL for. Look the key up in the needle map instead (Volume.LookupNeedle).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rust volume: tail receiver looks a size-0 record's key up in the needle map

Mirror Go: write_tailed_needle read the needle to decide whether a size-0
record deletes an existing key, and a read reports a needle past its TTL
as not found, so the delete became a live empty needle. Look the key up
in the needle map instead (Volume::lookup_needle).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: volume.merge keeps a merged delete when a late empty copy follows

A replica that missed a delete can hold its copy of the empty write
stamped after the other replica's delete. As that replica's first record
for the key it was written, reviving the deleted key. A size-0 record now
never revives a key the merged copy already deleted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: volume.merge refuses more replicas than its key history has bits

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 22:36:44 -08:00
0a1e337302 wdclient: keep a volume's live replicas readable while another volume cannot be found (#12814)
* wdclient: keep a volume's live replicas readable while another volume cannot be found

A filer's "volume not found" answer counted as a filer failure, so three
of them opened the circuit breaker on a healthy filer. With its only filer
in the cool-down, every lookup then failed without asking, as "all 1
filer(s) failed with non-retryable error: %!w(<nil>)". And a refresh whose
lookup failed left the cache without the volume's live replicas, since
InvalidateCache had already dropped them with the dead one.

The breaker now counts only failures to reach the filer, and asks the
filers anyway when all of them are in their cool-down. InvalidateCache keeps
what it removed; a lookup that errors puts it back unless the volume was
learned again, the epoch moved, or a reset would have expired it, and
leaves out locations fenced by a pushed removal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* wdclient: set invalidated locations aside in the map instead of copying them out

The first commit kept what InvalidateCache removed in a side map and put it
back when the lookup failed. The copy left the map's lock, so a removal the
master pushed between the take and the put-back was undone: the removed
server came back into the cache.

The entry now stays in the map with a state. entryStale hides it from readers
and releases its server references, as a delete did. A pushed removal edits
it like any other entry. When the lookup after the invalidation returns an
error, the entry becomes entryRevived and is served again.

A stale or revived entry holds only what was known before the read failed,
so the next thing learned replaces it: a lookup answer drops it before the
answer is cached, including an answer with no locations and an answer to a
lookup that was in flight when another one failed, and a pushed add replaces
it instead of merging.

LookupVolumeIdsWithFallback returns no error when every volume it was asked
for has locations after the revive, so the filer's LookupVolume handler sends
its response instead of an error status.

The not-found breaker test now checks the failure count itself. Without
that, a regression of the classification passed, because a lone filer in its
cool-down is asked anyway.

On the first commit, TestPushedRemovalRacingFailedRefreshes failed in 8 of 8
runs under -race: four with "got [live:8080 old:8080], want [live:8080]",
four with "got [], want [live:8080]".

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* wdclient: retry a filer lookup pass that asked no filer

The pass decides once whether to skip unhealthy filers. A filer whose
breaker trips after that decision, as when a concurrent lookup records
its failure, is still skipped. With every filer skipped the pass ended
with no error and failed the lookup as non-retryable with
"%!w(<nil>)". Such a pass now counts as Unavailable, so the lookup
retries and the next pass asks the filers anyway.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* wdclient: believe a provider answer that reports errors for some volumes

The master provider answers a batch with a map of the volumes it knows
and an error naming the ones it does not. The failed-lookup branch took
that as no answer and revived what InvalidateCache had set aside, so a
volume the master reported as absent came back with its old locations,
and the batch could return them with a nil error. Only a nil map now
means no answer. Otherwise the answer replaces what was set aside, and
the provider's error is still returned.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* wdclient: keep set-aside locations learned after a lookup was asked

A successful lookup dropped every stale or revived entry of the volumes
it asked for, however old the answer was. A lookup asked before the
master confirmed new locations could, on finishing, delete those
locations after a read had set them aside and a failed refresh had
revived them, and its own answer was not cached because the epoch had
moved. Entries now record when their locations were learned, and an
answer drops only what was learned before it was asked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* wdclient: keep an older answer from replacing locations set aside after it was asked

fdcde25 keeps dropSetAside from deleting set-aside locations that were
learned after a lookup was asked. The answer's insertion still replaced
them: fillLocations adds each location with vidMap.addLocation, and
addLocationToMap replaces a stale or revived entry whatever its learnedAt.
The epoch does not stop this when the newer locations came from another
lookup's answer or, on the master client, from a pushed add, since neither
moves it. The master client's provider also caches its answer itself,
before LookupVolumeIdsWithFallback reaches dropSetAside.

vidMap.addAnswer adds an answer's locations under one lock, and adds none
of them while vid has a stale or revived entry learned after the lookup was
asked. fillLocations uses it, and the master client's provider passes the
learned mark it takes next to its epoch. An answer with nothing newer in the
way still replaces what was set aside, with every location it reports.

On fdcde25, TestOlderAnswerDoesNotReplaceNewerRevivedLocations and the
"newer location revived during the lookup" case of
TestMasterAnswerAgainstSetAsideLocations fail with
"got [old:8080], want [new:8080]".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* wdclient: move addAnswer next to dropSetAside

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-19 22:34:35 -08:00
Chris LuandClaude Opus 5.5 374c77bcbe admin/plugin: fail a restored job its worker no longer lists (#12815)
* admin/plugin: fail a restored job its worker no longer lists

An admin restart reloads the jobs that were running from the persisted
state, but the workers drop those jobs when the old admin stream closes.
No pending execution covers a restored job, so only the 24h stale sweep
ended it, and until then it blocked dedupe and scheduling of its job type.

Track the active jobs loaded at startup by worker and fail one after two
itemized heartbeats from its worker leave it out, as is already done for
a dispatch whose worker restarted. A listed job is left to adoption.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* admin/plugin: keep tracking a listed restored job; skip queued ones

- A restored job its worker lists only resets its miss count, so one it
  drops later is still failed.
- Track only jobs that were sent to a worker (request id set, not pending).
- Match restored ids against the trimmed heartbeat ids.
- Fail the restored attempt under the jobs lock that applies the expiry
  (handleJobCompletedIf / expireActiveJobIf), so a newer attempt of the
  same job is neither failed nor canceled.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* admin/plugin: keep a newer attempt's detail from a restored-job expiry

- The conditional completion writes the job detail through
  persistJobDetailSnapshotIf and skips it when the stored detail already
  names another request; the check runs under jobDetailsMu with the write.
- mergeTerminalDetailIntoTracked ignores a terminal detail of another
  request, so a restart does not end the loaded attempt with it.
- Tests wait for the restored-job expiries instead of sleeping.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* admin/plugin: restore a finished retry from its detail

The job detail is saved as each attempt changes and the job list only
periodically, so a terminal detail naming another request is the newer
attempt. Apply it with its request id instead of ignoring it; the
expiry's own detail write already refuses to overwrite another request.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* admin/plugin: restore whichever attempt is newer

The job list and a job's detail can name different attempts either way
round: the list is saved periodically, and a detail write can fail. When
their request ids differ, apply the terminal detail only if it is the
newer attempt (higher attempt number, else later update).

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* admin/plugin: fill a restored job's empty request id from its detail

A job the list saved while still queued has no request id; take the
finished detail's, as the merge did before the newer-attempt check.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* admin/plugin: take the worker and attempt of a newer restored detail

When the restore takes a finished retry's detail over the job list's
attempt, the record kept the older attempt's worker and attempt number
next to the retry's request id and state.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin/plugin: fold the conditional completion into the existing functions

handleJobCompleted, expireActiveJob, trackExecutionCompletion and
persistJobDetailSnapshot take the accept check themselves instead of
gaining *If twins.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin/plugin: ignore a late earlier-attempt completion; replace an older detail

- A completion whose request id differs from the tracked attempt's is a
  late report of an earlier attempt and no longer ends a retry.
- A conditional completion skips the detail write only when the stored
  detail is a newer attempt (detailIsNewerAttempt), so an older finished
  attempt's detail left behind by a failed write is replaced.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* admin/plugin: take only the dispatched attempt's reports

A late progress report of a timed-out attempt replaced the retry's request
id, after which the retry's own completion was rejected as foreign. One
rule now gates both: a record the admin dispatched (attempt > 0) takes
progress and completion only for its own request (reportFromOtherAttempt).

When an expiry replaces an older attempt's detail it also takes the
closing attempt's worker, attempt number and times.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* admin/plugin: restore a newer running attempt's identity from its detail

The job list can name attempt A while the detail shows retry B running
when the admin stopped. The restore now takes B's request id, worker and
attempt number from such a newer running detail, so B's own reports are
accepted. Test helper keeps a given attempt number.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* admin/plugin: keep attemptEnded's doc comment on attemptEnded

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin/plugin: dispatch an attempt-less job as attempt 1

The late-report guard only covers records with a positive attempt, and
ExecuteJob passed a zero attempt through, so a direct caller's dispatch
could be closed by another request's completion. The HTTP API already
raised zero to one; the dispatch path now does the same for every caller.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 22:30:52 -08:00
2382f07788 filer: keep a hard-linked file's chunks when one of its names is replaced (#12812)
* filer: keep a hard-linked file's chunks when one of its names is replaced

Replacing one name of a hard-linked file deleted the file's chunks while
its other names still pointed at them. Reads through those names then
return EIO once the volumes are vacuumed; until then they only work
through readDeleted.

Two everyday ways to get there through a mount:
- conda links an environment's files to its package cache with hard
  links. Python saves a __pycache__/*.pyc by writing a temporary file and
  renaming it over the old name, so the cache's copy loses its data.
- A training script saves model.pt, hard-links it into an epoch directory
  and saves the next epoch by renaming a new file over model.pt. Each
  archived epoch loses its data.

Three write paths deleted the chunks of the entry they replaced without
asking whether that entry was one of several hard links:
- a rename over a file: moveSelfEntry drops the target's link, then
  queues all of the target's chunks for deletion;
- CreateEntry over an existing path (an S3 PUT, for one): the update
  drops the old entry's link, then deleteChunksIfNotNew deletes its chunks,
  and the CreateEntry handler deletes chunks its own list covers;
- UpdateEntry with an entry that no longer carries the old one's link.

A plain delete already keeps the chunks while HardLinkCounter > 1, and a
folder delete asks hardLinkIsOrphaned. The three paths now go through
Filer.KeepHardLinkedChunks, which asks after the write, once the replaced
name's link is gone: the chunks stay unless the file's shared record went
with it, that is, unless the replaced name was the last link. Writing the
file through one of its names (an entry with the same hard link id) still
deletes the chunks the write replaced, since every name reads the new
content.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: keep hard-linked chunks from the shared record, not the replaced row

A lookup that cannot read the hard link record returns the name's own row,
which can lag the file. Read the record when filtering the garbage, and keep
everything when it cannot be read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test the manifest reads behind KeepHardLinkedChunks

The manifest resolution moves into withoutChunksOf, which takes the lookup
function, so tests can serve manifests. Chunks behind nested manifests and
the manifests themselves stay, unrelated garbage is still deleted, and an
unreadable manifest returns an error, on which KeepHardLinkedChunks deletes
nothing. Behavior is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-19 22:14:01 -08:00
Chris LuandClaude Opus 5.5 a91def90c0 volume: count and log needles vacuum drops as unreadable (#12811)
* volume: count needles vacuum drops as unreadable and log each at error level

Vacuum drops a live needle whose record fails to read back intact (CRC
mismatch, malformed fields, short or dangling record). The key leaves the
compacted index with no tombstone, so repair tools later treat it as a
vacuumed delete. Log every dropped key at error level and count the drops
in SeaweedFS_volumeServer_vacuuming_dropped_needles per collection, once
the compacted index is written.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: add a panel for needles dropped by vacuum

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: count needles vacuum drops as unreadable and log each at error level

Mirror the Go volume server: log every dropped key at error level and add
SeaweedFS_volumeServer_vacuuming_dropped_needles per collection once the
compacted index is written.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: count dropped needles only after the compacted dat is flushed

The final .cpd fsync runs in the deferred close and loadTmpNeedleMap can
still fail after the index is saved; a copy failing there is retried and
would count the same needles again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 21:02:31 -08:00
Chris LuandClaude Opus 5.5 2ac8db5cb4 filer: lock the path for plain HTTP overwrites so replaced chunks get deleted (#12801)
* filer: lock the path for plain HTTP overwrites

CreateEntry looks up the entry it replaces and queues that entry's chunks
for deletion. Two plain PUT/POST writes of one path could both read the
same old entry, so the chunks of whichever write lost were never queued.
Take the exclusive path lock around CreateEntry, as the gRPC CreateEntry
already does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test concurrent HTTP overwrites queue every replaced chunk

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 21:01:08 -08:00
Chris LuandClaude Opus 5.5 ac7d949c61 filer: keep a manifest needle until its data chunks are queued for deletion (#12810)
* filer: resolve deleted manifests through the master on a volume cache miss

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: keep a manifest needle until its sub-chunks are queued for deletion

When a manifest could not be read during chunk deletion, the manifest
needle was deleted anyway and the data chunks it listed were orphaned.
Retry the resolve with the deletion retry backoff and delete the
manifest alone only after MaxRetryAttempts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: keep a durable delete that joins a pending manifest retry

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: bound each manifest retry read with a deadline

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: requeue resolved manifest ids on an fsync upgrade instead of rereading

A durable delete that joins a manifest retry in flight used to resolve
the manifest a second time, but the first pass had already queued the
manifest needle, so the reread could fail and the sub-chunks never got
the durable mark. Resolve nested manifests up front and queue the same
ids again with fsync.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: bound each manifest retry pass and stop it on shutdown

A pass takes ready manifests one at a time and stops after
manifestRetryTimeout, leaving the rest queued for the next tick, so a run
of stalled reads cannot keep the shared retry loop from ordinary deletion
retries or from noticing shutdown.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 19:50:40 -08:00
Chris LuandClaude Opus 5.5 8df716b728 volume: make a delete retry after a failed replica fan-out reach the replicas (#12797)
* http: report the status of a failed delete that has no body

A replica that answers a DELETE with a bare 500 left the forwarding
server reporting "[<replica>]: " with an empty reason.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: forward a delete retry that finds nothing here to the replicas

A delete is applied locally before it is sent to the other replicas, so
when the fan-out fails the client is told to retry while this replica
already dropped the needle. The retry then read nothing here and answered
404 without asking the replicas, so the replica that missed the delete
kept the needle for good.

On a replicated volume, a plain delete that finds the needle missing or
deleted now cancels a write still queued for it, as a replica delete
does, and sends the delete on to the other replicas. A failed fan-out is
reported as before, so the client retries until every replica agrees.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): take a replica's 404 to a forwarded delete as done

A replica that no longer holds the needle answers a forwarded delete with
404. The Go server folds that into success, but the Rust server reported
it as a replication failure, so a delete that only had to catch up one
replica failed on the replica that was already in step.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): forward a delete retry that finds nothing here to the replicas

Mirrors the Go server: on a replicated volume, a plain delete that finds
the needle missing or deleted cancels a write still queued for it and
sends the delete on to the other replicas, so a retry after a failed
fan-out reaches the replica that kept the needle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: answer a delete whose needle read fails with 500, not 404

A replica's 404 to a forwarded delete is taken as the needle being gone
there. DeleteHandler answered 404 for any failed pre-read, so a replica
whose disk could not read the needle let the primary acknowledge a delete
that replica never applied. Only a missing or deleted needle (or a volume
this server does not hold) answers 404 now; other read errors are a 500,
so the client retries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): answer a delete whose needle read fails with 500, not 404

Mirrors the Go server: a replica's 404 to a forwarded delete is taken as
the needle being gone there, so only a missing or deleted needle answers
404; other read errors are a 500 and the client retries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): corrupt the needle with portable file writes in the delete read test

FileExt::write_all_at is Unix-only, so the test did not build on Windows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 19:47:02 -08:00
Carlos Leyva 2e4be5e25e mount: report a read-only mount root that does not exist as not found, not as the create error (#12799) 2026-11-19 19:45:41 -08:00
Chris LuandClaude Opus 5.5 f2acfb7be4 volume: serve decoded bytes for a range over a compressed needle (Go and Rust) (#12805)
* volume: serve decoded bytes for a range over a compressed needle

A Range with Accept-Encoding: gzip got Content-Encoding: gzip and a slice
of the compressed bytes, which a decoding client cannot read. The range
is over the stored file's decoded bytes, so pass gzip through only for a
whole read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-volume: serve decoded bytes for a range over a compressed needle

Mirrors the Go volume server: a Range with Accept-Encoding: gzip got
Content-Encoding: gzip and a slice of the compressed bytes. Pass gzip
through only for a whole read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-volume: HEAD of a compressed needle reports the served length

HEAD answered a compressed needle from its meta: the stored gzip length
with no Content-Encoding, whatever the request accepted. Go reads the
payload and reports what the GET would send: the decoded length, or the
gzip length with Content-Encoding: gzip for a whole read that accepts
gzip. Read a compressed needle in full on HEAD too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/volume_server: range and HEAD over a compressed needle

Both volume servers serve a gzip-accepted range decoded, and HEAD reports
the length the matching GET would send.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-volume: decode every gzip member of a compressed needle

Go's gzip reader reads concatenated members; GzDecoder stopped after the
first, so a range over such a needle saw a shorter body and length.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 19:43:02 -08:00
Chris LuandClaude Opus 5.5 b5f23b7dbf vector: answer a short filtered ANN query by an exact scan (#12802)
* vector: answer a short filtered ANN query by an exact scan

A prefiltered query through the IVF index sees only the partitions it
probes. When those hold fewer than k eligible rows the answer is short
(400 matches, topK 500, 63 partitions: about 208 returned), and when at
most k rows match Lance appends the ones its probes missed with an
infinite distance. Either case is now answered again by an exact scan of
the same scope: the listed fragments plus the live coverage of the listed
index segments, or the whole table.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: count an explicit probe count as indexed only with an index

With nprobes set on a table that has no vector index, the scan was marked
indexed, so a filtered query matching fewer than k rows ran the same exact
scan a second time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 19:42:02 -08:00
261c534c19 filer: retry TiKV store writes on a write conflict (#12798)
* filer: retry TiKV store writes on a write conflict

TiKV transactions are optimistic. A commit fails with a write conflict
when another transaction committed one of its keys after it began, even
for a blind Set or Delete. Outside a filer transaction the TiKV store
runs each save and delete in a transaction of its own and returned that
error to the caller, so of two requests writing the same entry at the
same moment one failed, and a mount reported EIO.

A write the store commits itself (InsertEntry, UpdateEntry, DeleteEntry,
one batch of DeleteFolderChildren, KvPut, KvDelete) now runs again in a
new transaction after a write conflict, up to 5 times, waiting 1-2ms,
2-4ms, ... 16-32ms in between. The failed transaction stored nothing,
and the retry lands after the other writer, as if the two had not
overlapped. The retries stop when the context is done.

A write inside a transaction from BeginTransaction is not retried: the
conflict is reported when the caller commits, as before.

deleteBatch now goes through RunInTxn like the other writes.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* filer: stop TiKV write retries once the context is done

When the context was cancelled while the retry wait was running, the
timer case could still win the select and start one more transaction.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: keep a TiKV entry that is no longer expired when listing deletes it

The listing decides an entry has expired from its own read, and the
delete ran later as a blind delete. Once the delete retries after a
write conflict, a live replacement committed in between was removed.
Read the entry again in the deleting transaction and delete it only if
it is still expired.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: MapleWhiteZZZ <18653045+MapleWhiteZZZ@users.noreply.github.com>
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-19 19:17:26 -08:00
github-actions[bot] 5e003bae7d docs: regenerate star history chart 2026-11-20 01:22:33 +00:00
Chris LuandClaude Opus 5.5 4ac3633f1c volume: check the needle id in the streamed-read probe and on blob writes (#12794)
* storage: reject a streamed read whose record id or size differs

The header probe before a streamed read returned not-found only when both
the record's id and its size differed from the index entry, so a record of
another needle with the same size passed and its data was streamed under
the requested key until the final checksum check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: refuse a needle blob whose header names another needle

WriteNeedleBlob compared only the blob header's size with the size it was
asked to index. A blob read from an offset that has since moved to another
record of the same size was appended and indexed under the wrong key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): refuse a needle blob whose header names another needle

Mirror of the Go check: write_needle_blob_and_index compared only the blob
header's size, so a blob of another needle with the same size was appended
and indexed under the requested key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): report not-found when a streamed read's record is another needle

The meta read ahead of a streamed reply took the id from the record header
without comparing it with the requested key. Return not-found when the
index entry resolves to another needle, as the Go header probe does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: check the requested id after a meta-only read

ReadNeedleMeta overwrites n.Id with the record's header id, so a meta-only
read through an index entry that resolves to another needle returned that
needle's metadata, and the streamed-read probe that follows compared the
record with itself. Return not-found when the header names another needle,
as the Rust meta read does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: copy a needle blob to another volume, not under another key

WriteNeedleBlob now refuses a blob whose header names another needle,
so the round trip writes the blob under its own id into a second volume
and checks that storing it under another key is refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: compare the streamed header with a deleted entry's record size

A deleted index entry stores the negated size, so a read-deleted stream
compared the positive header size with a negative one and missed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 15:27:15 -08:00
Chris LuandClaude Opus 5.5 4a5dbe4c6f storage: copy an empty needle written during compaction in makeupDiff (#12796)
* storage: copy an empty needle written during compaction in makeupDiff

A size-0 index row is a live empty needle, not a delete. makeupDiff
grouped it with tombstones and appended a made-up delete marker, so
the compacted .dat lost the record's cookie and held a delete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): copy an empty needle written during compaction in makeup_diff

A size-0 index row is a live empty needle. makeup_diff wrote a made-up
delete marker and a tombstone row for it, so the needle read as not
found after the commit. Copy the record like any other live row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: fail makeupDiff when the replayed needle write fails

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): check the compacted empty needle's id and cookie in the test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 15:26:20 -08:00
Chris LuandClaude Opus 5.5 ba1bfa816c admin: show a pruned job's detail from its run record (#12795)
* admin: show a pruned job's detail from its run record

Tracking keeps the newest 1000 jobs across all job types and deletes
the details of older finished ones, while each job type's run history
keeps its last 10 successful and 10 failed runs. A job listed in the
run history could therefore open as "job detail not found". Fall back
to the run record when the job is no longer tracked.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* admin: show 100% progress for a pruned successful run

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: use the newest run record for a job across both outcomes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: keep the run record a pruned job's detail was built from

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: keep looking for a run record past an unreadable history

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: pick a pruned job's newest run record across job types

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 15:23:46 -08:00
Ping QiuandClaude Opus 5.5 be8ebc2e04 storage: build the temporary needle map in CompactByVolumeData (#12788)
* storage: red test for CommitCompact after CompactByVolumeData with a concurrent write or delete

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: build the temporary needle map in CompactByVolumeData

Fixes #12783

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: require a successful commit in the CompactByVolumeData commit test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 15:06:05 -08:00
Chris LuandClaude Opus 5.5 bfbeed61a7 storage: read only the header and trailer in the append-time search (#12791)
BinarySearchByAppendAtNs, its left/right probes and ResumeByLastEntry read
every probed record through readNeedleAt, which parsed the whole body and
verified the data checksum, although they use only the id, cookie, size and
append time. One record with damaged data on the search path, even a needle
deleted long ago, failed the search and with it the tail sender, a
timestamp-based incremental copy and the volume.move catch-up.

readNeedleAt now reads the header and the trailer and skips the data. The
data checksum is still verified where the data is read, including the tail
receiver for every record it gets. The Rust volume server already parses
only the metadata on this path and does not verify the data checksum.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 15:02:25 -08:00
05427e8108 storage: hold the data-file read lock across the append-time search (#12789)
* storage: red test for the append-time search racing a vacuum commit

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: hold the data-file read lock across the append-time search

Fixes #12785

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: Version() writes the superblock version only when it changes

Version() stored the .vif version into SuperBlock.Version on every call, so
each call was a write that raced with readers of the superblock that do not
take superBlockAccessLock, such as the append-time search and the
compaction setup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: harden the append-time search race test; add a lock-hold budget test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: Linux-only lock budget; stop before reading search failures

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-19 15:01:47 -08:00
Chris LuandClaude Opus 5.5 f7323f2b63 filer: optional HTTP Basic Auth on the filer HTTP ports (#12778)
* security: add a constant-time HTTP Basic Auth check

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: require HTTP Basic Auth on the HTTP ports when configured

With filer.basic_auth.password set in security.toml (or
WEED_FILER_BASIC_AUTH_PASSWORD), a request on the filer HTTP ports, TUS
included, must carry those credentials or a valid filer JWT, and an
access level with no JWT signing key is no longer open. Valid Basic
credentials grant read and write. 401 responses then carry a Basic
challenge so browsers prompt. The credentials reload on SIGHUP with the
signing keys.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* scaffold: document filer.basic_auth in security.toml

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: enable reloaded Basic Auth before swapping the JWT keys

Reload cleared the old signing keys before storing the new Basic
credentials, so a switch from JWT to Basic Auth briefly left both off
and admitted anonymous requests. Store new credentials first and clear
old ones last.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: refuse Basic credentials on cross-origin browser requests

Browsers resend cached Basic credentials on requests other websites
trigger, such as a form POST with mv.from, and the default
cors.allowed_origins "*" lets them through. Accept Basic credentials
only on requests without an Origin, from the filer's own host, or from
an origin explicitly listed in cors.allowed_origins. A JWT is still
accepted from any origin.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: recheck Basic Auth after reading an empty signing key

A request that loaded Basic Auth as off, then read the signing key after
a JWT-to-Basic Reload had cleared it, was let through anonymously. Open
access now also needs Basic Auth to be off after the key read; with
Reload enabling the new credential before disabling the old one, the
two loads cannot both miss a credential that was on throughout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* security: compare Basic Auth credentials as MACs under a per-process key

A plain SHA-256 of the password is what password-hash scanners flag;
keying the digests with a random per-process HMAC key keeps the
length-independent constant-time compare without holding one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* util: read a configuration file without merging into the shared viper

LoadConfiguration merges each read into the shared viper, so a key
removed from the file keeps its old value after a reload.
ReadConfiguration reads the file into a fresh viper with the same search
path and WEED_ environment overrides.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: turn Basic Auth off when security.toml no longer sets it

The shared viper merges each SIGHUP reload into what it held, so
removing [filer.basic_auth] kept the old password valid. Read the
credentials from security.toml alone at startup and on reload; an
unreadable file keeps the current setting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: keep Basic Auth when security.toml cannot be read

A malformed file at startup dropped a WEED_FILER_BASIC_AUTH_PASSWORD
from the environment, and viper reports an unsearchable config
directory as a missing file, which turned Basic Auth off on reload.
ReadConfiguration now always returns the environment overrides and
reports whether the file was read; a reload that could not read it
keeps the current credentials.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: keep reloading Basic Auth from the security.toml first read

A search directory that turned unsearchable made viper fall through to a
lower-priority security.toml, and a reload from a file without
[filer.basic_auth] turned Basic Auth off. ReadConfiguration now takes the
file to read, and the filer reads the file it first found on every reload.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 13:19:46 -08:00
f5e091cf91 wdclient, s3: return an embedded S3 gateway to its local filer after failover (#12772)
* wdclient: sort newly discovered filers and log every current-filer switch

Discovery appended new filers in map iteration order, so after the current
filer left, the fallback at index 0 was a random filer. Append them sorted.
Route every change of the current filer through one helper that logs it at
V(0) with its reason (failover, return_home, discovery), so an operator can
see which filer a gateway uses without new metrics.

* wdclient: prefer a home filer and return to it after failover with backoff

A FilerClient with FilerClientOption.HomeFiler set goes back to that filer
once its circuit breaker allows a re-check, instead of staying on whichever
peer it failed over to. Leaving home again soon after a return doubles the
wait before the next return, up to 5 minutes; a return that holds for 10
minutes resets it. Volume lookups and discovery follow the same preference.

* s3: embedded gateway uses its local filer as home filer

weed filer -s3, weed server -s3 and weed mini now build the gateway's filer
address in the form the filer registers in the master (ip:port.grpcPort),
with the gRPC port default applied, and pass it to the gateway as its home
filer, so the gateway returns to its local filer after any failover.
Standalone weed s3 has no home filer and behaves as before.

Refs #12768, follow-up to #12771

* wdclient: switch the current filer only from the filer a request started on

A volume lookup that started on a peer and finished after the client had
returned home moved it back to the peer and counted that as leaving home,
doubling the return backoff although home never failed. Concurrent
failovers could also each count as a separate departure.

Failovers now name the filer they started on and take effect only while it
is still current, via compare-and-swap, so only the winning switch logs and
feeds the backoff. S3's failover walk uses the same rule.

* wdclient: switch the current filer and record the return backoff in one step

The index moved by compare-and-swap before the switch was recorded, and
both ran under the shared read lock. A request reading the current filer
in that gap saw the peer with the old eligibility time and, with home's
breaker still closed after one failure, switched straight back home.
The eligibility check, the move and the bookkeeping now all run under
homeMu.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* wdclient: discovery falls back past a home that is still backing off

When discovery removed the current filer and home was not yet eligible,
the client took index 0, which is home whenever it is listed first. With
home's breaker still closed after a single failure, requests went back to
the filer they had just left. Take the next filer instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* wdclient: start return timing on every switch back to the home filer

Only a return_home switch stamped lastReturnHome. Discovery or a peer's
failover could also put the client back on home, and a failure soon
after such a return kept the backoff at its base value instead of
doubling it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Konstantin Lebedev <whitefox@mayflower.work>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 13:18:57 -08:00
Chris LuandClaude Opus 5.5 c9f3a02b94 filer: recover a leveldb store whose table a power loss damaged (#12780)
* filer: recover leveldb without rebuilding broken tables

leveldb.Recover rebuilds a table with damaged blocks using the user
comparer and filter on internal keys. The shortened index keys it writes
are not valid internal keys, so a Get on the rebuilt table panics, and
the bloom filter it writes misses every lookup. Recover with a comparer
that keeps full index keys (same name, so the manifest still matches)
and no filter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: recover a leveldb store when a compaction finds a corrupted table

When a compaction reads a damaged table block, as a power loss can
leave one, goleveldb latches the corruption error and returns it for
every later write until the database is reopened. A plain reopen loads
the same table and the next compaction fails again, so the store kept
rejecting inserts even across restarts.

Treat a corruption error from a write like a storage error, and reopen
through leveldb.Recover, which rebuilds the table without the damaged
blocks, then retry the write.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: rebuild damaged leveldb tables before Recover

leveldb.Recover's own table rebuild orders internal keys with the user
comparer, so a damaged table holding several versions of one key fails
with "keys are not in increasing order" and the recovery gives up.
Overriding the comparer cannot fix that, because the same comparer
also serves the recovered database's user keys.

Rebuild every table that has corrupted blocks first, using an internal
key comparer and full index keys, then let Recover rebuild the manifest
from tables it no longer needs to rewrite. Seed the test tables from a
journal flush so they keep several versions of a key, and check keys
the write test does not rewrite.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: leave unreadable leveldb tables for Recover to drop

table.NewReader reports a truncated table or a damaged footer through
the reader, not its error, so rebuildDamagedTables never stopped on one,
but it rewrote it as an empty table first. Skip tables with nothing
readable and let Recover drop them, and test an unreadable table next
to a healthy one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: let StrictRecovery drop damaged leveldb tables

With StrictRecovery set, leveldb.Recover drops a table that has corrupted
blocks. Salvaging the readable blocks first would hide the damage from it,
so skip the rebuild under that option.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: hold the leveldb storage lock while rebuilding tables

The table rebuild ran after the failed open released the storage lock
and before Recover took it, so it could rewrite the tables of a database
another caller had open on the same storage. Take the lock for the
rebuild; Recover takes it again afterwards.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 13:13:30 -08:00
6bdcd17ed4 master: reject an assign on a follower as Unavailable (#12779)
* master: reject an assign on a follower as Unavailable

Assign on a master that is not the raft leader returned
raft.NotLeaderError as a plain error, which reaches gRPC clients as
codes.Unknown. operation.Assign retries only Unavailable,
ResourceExhausted, Canceled and DeadlineExceeded, so every assign made
during a leader election failed at once instead of waiting for the new
leader.

Return it as codes.Unavailable, like the warming-up status in the same
file, so clients retry within their 30 s assign budget. The message is
unchanged, so callers that match "Not current leader" still work.
StreamAssign now answers such a request in-band with the error message
instead of ending the stream, as it already does for warm-up.

TestAssignOnFollowerIsUnavailable covers it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* pb: keep the cached channel when the peer answers Unavailable with details

gRPC's own Unavailable (dial failure, closing transport) never carries
details, so one that does came from the peer over a working channel.
Closing the shared ClientConn for it cancels every other in-flight RPC
on it with "the client connection is closing".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* master: tag the follower's assign rejection with a NOT_LEADER ErrorInfo

The detail marks it as the master's answer, so clients retry it
without tearing down their cached channel to that master.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-19 13:11:40 -08:00
Chris LuandClaude Opus 5.5 729b872f01 test: let the merge-read FUSE test wait out the master's vacuum drain (#12777)
* test: let the merge-read vacuum outlast the master's write drain

Before compacting a volume the master waits up to 30 s for writes it
assigned to that volume to drain. The test cut each /vol/vacuum call at
30 s and gave the vacuum 60 s, so one or two slow drains failed it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: retry the merge-read vacuum until the master runs one

A 409 means another vacuum holds the master's lock; check the chunks
only after a vacuum this test asked for has finished.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 12:44:48 -08:00
Chris LuandClaude Opus 5.5 972f85159e util/http: probe dead replicas in the background and key no-answer dials by machine (#12776)
* util/http: probe an unanswering server in the background, not on a read

Once the retry interval passed, the first read put the dead server first
and waited out the full dial timeout before trying a live replica. Keep it
last and dial it from a background probe with a short timeout instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* util/http: try every server of a machine that does not answer last

Unreachability was kept per host:port, so a powered-off node running one
volume server per disk cost one stalled read per server before each was
known dead. A dial that times out or finds no route now marks the whole
machine; a refused connection still marks only that port and shows the
machine is up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* util/http: probe an unanswering server with an HTTP request

A bare TCP connect cleared servers that accept connections but never send
response headers, and could not dial an address without an explicit port.
Probe with a HEAD request through the global client instead, and have the
tests wait for the probe to settle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* util/http: keep a timed-out port last after a sibling port answers

A dial timeout stored only the machine verdict, so a success on another
port of that machine cleared it and the port that timed out went back to
the front. Record the port as well as the machine.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* util/http: classify refused and unroutable dials with Winsock errnos on Windows

Windows reports these as WSAECONNREFUSED, WSAEHOSTUNREACH and friends,
which errors.Is does not match against the syscall constants. Test the
refusal with a real dial to a closed port.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* util/http: wait for background probes through a WaitGroup in tests

A refused probe can finish before the test reads the claimed record, so
inferring completion from the record's timestamp could wait out the
deadline on a probe that already settled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* util/http: key machine verdicts apart from hosts without a port

A host without an explicit port is also its machine name, so a sibling
port answering cleared the timed-out host along with the machine verdict.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* util/http: group a bracketed port-less IPv6 host with its machine

net.SplitHostPort rejects "[2001:db8::9]", so the machine key kept the
brackets and missed the verdict stored for "[2001:db8::9]:9501".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 12:44:10 -08:00
ed7324611c filer: metrics for metadata subscription delivery (lag, holds, ring fall-offs, claim and watermark ages) (#12773)
* filer: metrics for metadata subscription delivery (lag, holds, ring fall-offs, claim and watermark ages)

* filer: measure subscribe delivery lag after the real send using the mutation stamp, and record a watermark hold once per hold

* filer: skip empty progress markers when sampling subscribe delivery lag

* filer: assert subscribe delivery lag is sampled only after the stream send succeeds

* grafana: panels for the metadata subscription delivery metrics

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: record a watermark hold when the read passes the held entry, not when the next hold starts

* filer: use a monotonic clock for the subscribe hold duration

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 12:43:34 -08:00
Chris LuandClaude Opus 5.5 05a1463855 wdclient: keep the embedded S3 gateway on its local filer across discovery (#12771)
* filer: keep an explicit gRPC port in the embedded gateways' filer address

runFiler handed the embedded S3/WebDAV/IAM/SFTP gateways
NewServerAddress(...).String(), and String() drops the .grpcPort part.
With a non-default -port.grpc the gateways then dialed port+10000 and
did not recognize their own filer in the master's list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* wdclient: match discovered filers by gRPC address

A filer listed as host:port and the same filer discovered as
host:port.grpcPort were compared as raw strings, so a refresh pruned the
configured filer, added every discovered filer as new in map order, and
reset the current filer to whichever came first; a pushed join added it
a second time. Key filers by their gRPC address: a survivor keeps its
health and current position and takes the discovered address form, and
pushed joins and leaves match the same way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 12:42:31 -08:00
c1c95e127f filer: scoped read-only tokens for metadata subscriptions (#12767)
* security: scoped read-only token for filer metadata subscriptions

A token signed with jwt.filer_signing.subscribe.key names the subscribe
methods and the path prefixes its holder may see. mTLS peers and admin
tokens stay unrestricted. Internal subscribers attach an admin token
when both keys are configured.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: enforce subscription token scope on metadata streams

With jwt.filer_signing.subscribe.key set, SubscribeMetadata and
SubscribeLocalMetadata authorize the caller first. A scoped caller gets
a stream that sends only events whose paths are all under its prefixes,
filters batches, and never carries log file refs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: send an admin token on internal metadata subscriptions

Peer filers, metadata followers and the lifecycle reader attach an
admin token when the subscribe key is configured, so they keep the full
stream without mTLS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: add fs.meta.token to mint metadata subscription tokens

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: end scoped metadata subscriptions at token expiry

The scope now carries the token's exp. The scoped stream's context ends
at that time and the call returns Unauthenticated, so an open stream no
longer outlives its token.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.sync: sign metadata subscriptions with the source cluster's key

MetadataFollowOption takes a FilerJwtProvider; filer.sync passes the
one loaded from -a.security or -b.security, so clusters with different
admin keys both accept the subscription.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: report fs.meta.token flag errors

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: give scoped metadata subscribers their own client id

A scoped caller could reuse another subscriber's ClientId with a higher
epoch and replace that registration, disconnecting it. Scoped streams now
register under a server-chosen id that no other subscriber holds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* security: accept only HS256 metadata subscription tokens

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* security: test HS384 metadata subscription token rejection

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: never let a scoped subscriber replace another registration

The server-chosen id can be taken by another subscriber between choosing
and registering it. Registering scoped streams with the lowest epoch makes
addClient refuse them rather than replace the other registration.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: send position markers for events a scoped token withholds

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test a scoped stream withholding a whole event batch

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: filter SendMsg on scoped metadata streams

The scoped wrappers only overrode Send, so a SendMsg on the embedded
stream would reach the client unfiltered. Route it through the same
filter and refuse any other message type. Ported from #12790.

Co-authored-by: mindaeon <30555587+mindaeon@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test a scoped subscription through the real subscribe loop

Run SubscribeLocalMetadata with a scoped token over a flushed backlog,
the in-memory tail and live events: only in-scope events arrive, sibling,
"..", encoded and cross-boundary rename paths are dropped, and no log
file references are sent although the client asks for them. Without the
subscribe key every event still arrives. Ported from #12790.

Co-authored-by: mindaeon <30555587+mindaeon@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test that the HTTP API refuses metadata subscription tokens

GET, HEAD, PUT, POST and DELETE all refuse a subscription token, also
when the subscribe key equals the read or write key. Ported from #12790.

Co-authored-by: mindaeon <30555587+mindaeon@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: mindaeon <30555587+mindaeon@users.noreply.github.com>
2026-11-19 12:41:16 -08:00
Chris LuandClaude Opus 5.5 668ca45029 s3: ListParts returns each part number once, its latest attempt (#12770)
* s3: list each part number once, its latest attempt

Every UploadPart writes its own NNNN_<uuid>.part entry, so a part number
uploaded twice listed twice, superseded ETag included, and a max-parts page
could hold only the stale attempt. ListParts now groups the entries by part
number, reports the attempt sortEntriesByLatestChunk ranks first (the same
order completion uses), and pages over distinct part numbers, reading past a
page boundary that splits one part number's attempts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: test ListParts over repeated part attempts

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: rank an empty part attempt by commit stamp or mtime

An empty UploadPart body stores no chunks, so sortEntriesByLatestChunk gave
it time zero and an empty re-upload always lost to the attempt it replaced.
Attempts now rank by the filer's commit stamp when both carry one, else by
the later of the first chunk's write time and the entry mtime.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: keep only the newest attempt per part while listing parts

listPartAttempts held every attempt of a part number until the page was
built, so a heavily retried part grew one ListParts request's memory past
what max-parts bounds. It now compares each entry against the newest seen
for its part number with compareLatestPartAttempt, the comparator
sortEntriesByLatestChunk sorts with.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: order part attempts by commit stamp, then attempt time

Comparing commit stamps only when both attempts carried one made the order
non-transitive across stamped and unstamped attempts, so listing and
completion could pick different attempts depending on input order. The
commit stamp now leads (unstamped entries predate the field) and the
attempt time breaks ties.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: list part 10000 after the four-digit part numbers

Part names sort as text, so 10000_<uuid>.part lands between 1000.part and
1000_<uuid>.part. A page could end on part 10000 and the next marker then
skipped parts 1000-9999, and a legacy 1000.part could be reported before
the newer attempt of part 1000 was read. List the four-digit part numbers
first and part 10000 in its own prefix listing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: test ListParts paging across part 10000

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 12:36:27 -08:00
Chris LuandClaude Opus 5.5 dcbea0626f test/s3: patch s3-fast-list's data map race in the listing study build (#12766)
test/s3: build s3-fast-list with its data map draining the last page

The data map task reads the channel and then the list-task state. When
the list task sends its last page and completes between the two reads,
the map is dumped without that page and the tool exits 0, so a one-page
prefix comes back empty. Read the state before the channel.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 12:21:32 -08:00
Chris LuandClaude Opus 5.5 de4075660f telemetry: refresh the server module's go.mod so the integration test builds (#12765)
* telemetry: refresh the server module to the root module's dependency versions

The server module replaces seaweedfs with ../.., so root dependency
bumps raised prometheus/common, prometheus/procfs and x/sys past the
versions its go.mod listed, and go build refused to run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: run the telemetry integration test on root go.mod changes

The telemetry server module builds against the root module through a
replace directive, so a root dependency bump can leave its go.mod stale
without touching telemetry/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 00:55:09 -08:00
Chris LuandClaude Opus 5.5 652133a28a telemetry: keep the newest release in the versions chart (#12764)
The versions chart keeps the 8 versions with the most clusters on the last
day and folds the rest into "other". A new release starts small, so 4.54
(282 clusters) was folded in while 4.53 shrank as clusters upgraded,
which showed as a dip across the named versions. The newest plain release
now always keeps its own band.


Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 00:08:19 -08:00
e012fca9a2 mount: keep lookups from stalling behind directory cache rebuilds (#12754)
* mount: release the meta cache lock between batches of a rebuild wipe

deleteFolderChildrenForRebuild held mc's write lock while it listed every
cached child of the directory and deleted them. Every metadata read in the
mount (FindEntry, IsNameFresh, isOversized) takes the read lock, and a
waiting writer blocks new readers, so while a large directory was wiped,
lookups and getattrs in unrelated directories queued behind the scan. The
scan gets slower as leveldb accumulates the deletion markers of earlier
wipes of the same directory, which happens quickly once lookups rebuild
expired directories.

Wipe rebuildWipeBatchSize (1024) children per lock hold and drop the lock
between batches. A child is judged pinned under the same lock that deletes
it, so a local-only create is kept whether it was pinned before the wipe
started or between batches. The listing that refills the directory already
ran outside the lock, so batching adds no interleaving the build did not
allow.

Add LevelDBStore.BatchDeleteEntries, the counterpart of BatchInsertEntries.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* mount: bound how long a lookup waits on the directory rebuild it starts

Since #11268 a lookup whose parent's cache expired rebuilds the parent and
every expired ancestor up to the mount root before it answers, and every
lookup beneath them joins the same builds. With a slow listing (a large
directory, a busy filer, or the meta cache lock held by another rebuild) a
plain stat, an ls -l, or the parent touch after mkdir waited for all of it.

Wait at most lookupRebuildTimeout (2s), then fall back to one LookupEntry
RPC. Since #12206 the shared build does not run on the requester's context,
so it keeps going after the waiter gives up and later lookups find the
directory cached. A timed-out wait records a rebuild attempt, so lookups
during the cooldown do not queue on a build that is already known to be
slow.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* mount: wait for the rebuild's listing in TestLookupDoesNotWaitOutSlowRebuild

The build runs on its own goroutine, so the lookup that stopped waiting
for it can return before the build's ListEntries reaches the filer. The
test read `started` without waiting right after the lookup returned, which
relied on an ordering nothing guarantees, and the ListEntries count taken
afterwards assumed the same. Wait for the listing to arrive, bounded by
the test's deadline.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* mount: keep children inserted while a rebuild wipe runs

A deferred create inserts its placeholder before Create registers the
handle that pins it. The single-lock wipe made such an insert wait until
it finished, so the placeholder survived; a batch that runs in that gap
judged it unpinned and dropped it, and the rebuilt listing then omitted
the file until its flush.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: drop a cached path that an event vacates during a rebuild wipe

A build replays only the events later than its listing's snapshot and
leaves the earlier ones to the listing, which starts after the wipe. With
the wipe in batches a file's whole life fits inside it: created, flushed,
then unlinked or renamed away. Both events are earlier than the snapshot
and are skipped, the listing has no such file, and the placeholder the
wipe kept stays in the published directory until the next rebuild.

While a directory's wipe runs, an event buffered for it now takes the
path it vacates out of the store at once, fenced by versions like any
other event. After the wipe the store holds what the listing put there,
and a removal earlier than the snapshot may still arrive, so from then on
the event is left to the replay as before.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* mount: check which file a removal took away during a rebuild wipe

Two things the removal applied during a rebuild wipe got wrong.

A placeholder is unversioned, so the version gate lets every removal
through. One that arrived late, for an earlier file of the same name,
took out the placeholder of a file created since. The entry the event
took away is now compared with the cached one, by inode or, with no
inode to compare, by creation time. An event that only names the file
cannot tell and is taken at its word.

The entry was deleted without OP=MV, so taking one name of a hard-linked
file out of the cache also took one off the count its other names share.
A replayed event that vacates a path already deletes with OP=MV.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* mount: keep a removal the rebuild wipe declined out of the replay

The wipe declines a removal when what is cached there is newer than the
event or is another file than the one it took away. The event stayed
buffered anyway. An unversioned listing clears the children's version
records and replays every buffered event, so the declined removal then
deleted the entry the wipe kept and the directory was published without
it.

A declined removal now buffers only the name it puts in place, so a
rename still lands on replay.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: check a declined removal again when the build replays it

The previous commit dropped the removal half of an event the wipe
declined, judged by the file cached when the event arrived. Earlier
buffered events can replace that file during the replay: a listing
attempt that failed leaves its events buffered, so the retry's wipe may
see the first attempt's file while the replay puts a later one in place
first. The dropped delete then left that later file published.

The wipe now records which removals it declined, and the replay checks
each against the entry it finds at that point, replaying only the name
the event puts in place when that entry is still another file. A
removal declined by version is replayed as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep a declined removal's invalidation off the placeholder's handle

After the replay the build invalidates every buffered event again,
raised to the snapshot, and it still sent the original of a removal the
replay rejected. A late delete of an earlier file then reached the
handle of the placeholder at that name: the placeholder is unversioned,
so the raised timestamp got past it and the handle was marked deleted.

The replay's decision on a declined removal now replaces the event in
the buffer, and the invalidations use what was replayed. It is made
before the snapshot check, so a removal the listing covered is judged
the same way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep an event's signatures on the halves a rename is split into

A declined removal replays as its rename's create half, and a rename across
a building directory is split in two. Both dropped the event's filer
signatures, so a handle stamped by that filer no longer fenced the replayed
invalidation by timestamp and could take older listed contents over newer
ones it had read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: drain invalidations around the declined-removal build check

Invalidations run on a background worker, so the baseline could miss the
removal's own and the final read could run before the build's arrived.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: MapleWhiteZZZ <18653045+MapleWhiteZZZ@users.noreply.github.com>
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-19 00:04:03 -08:00
Chris LuandClaude Opus 5.5 3159b8408c filer, volume: fsync chunk deletes under an fsync path rule (#12759)
* volume: fsync a batch delete that asks for it

BatchDeleteRequest gets an fsync field, and the Go and Rust volume
servers pass it to the needle delete, which already orders the
tombstone, .dat sync, publish and .idx sync for a durable delete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* operation: pass an fsync flag to batch deletes

DeleteFileIdsWithLookupVolumeId and DeleteFileIdsAtOneVolumeServer take
an fsync argument and send it in the BatchDeleteRequest. Every caller
passes false for now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: fsync chunk deletes under an fsync path rule

DeleteChunks passes the path rule's fsync flag to the deletion queue.
The flag is kept with the pending record until the delete is
confirmed, so retries and re-queues keep it, and the batch and retry
loops send marked ids in a durable BatchDelete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: keep fsync marks in the deletion ledger

Marked ids go into the trailing parts of a manifest ledger, counted by
fsyncParts. Older filers read those parts as plain ids and still delete
them; a reload or a claimed foreign ledger restores the marks. A ledger
without marks keeps the single array form.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: record a queued deletion before the delete loop can see it

queueDeletions published the ids to the hot queue before taking the
ledger lock, so a concurrent flush could delete an id before its fsync
mark landed and then forget the mark with the record.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: fsync a folder delete's chunks by each child's path rule

A recursive delete collected its file children's chunks and resolved
the rule once for the folder, which misses a trailing-slash rule on
the folder itself and file-prefix rules. Each file child's own rule now
decides fsync; the folder rule still decides chunk deletion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: keep a newer fsync request when a plain delete finishes

An id re-queued with fsync while a plain delete of it was in flight
lost its mark when that delete succeeded, so the queued copy also ran
without fsync. A plain attempt now leaves a marked record for its
queued copy; a durable delete of an already-deleted needle still syncs
the tombstone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: delete a folder's plain child chunks without fsync

Children whose own rule turns fsync off were still deleted durably when
the folder's rule asked for fsync. Only the children marked by their
own rule now go out with fsync.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: retry a durable chunk delete the volume server failed

A volume server rolls back a durable delete it cannot sync and answers
with a 5xx carrying the I/O error text, which the classifier took as
permanent, so the record and its fsync mark were dropped while the
needle stayed live. A 5xx answer to a durable delete is now retryable,
within the usual attempt limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 00:02:43 -08:00
Chris LuandClaude Opus 5.5 f88751c3ff s3api: evaluate request condition keys for STS sessions (#12762)
* s3api: build request condition keys in one helper

Split the keys ExtractConditionValuesFromRequest reads from the request
itself into RequestConditionValues, keeping aws:SourceIp and the
bucket-listing keys in the method.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: evaluate request condition keys for STS sessions

The IAM integration context for STS and OIDC sessions carried only the
source IP, legacy user agent/referer keys and claims, so role and session
policy conditions on s3:x-amz-*, aws:SecureTransport, aws:CurrentTime,
s3:RequestObjectTag/* and the other request keys never matched: a Deny
on them never fired, and a negated Deny always did. Add the same request
keys the static identity path evaluates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: report SigV4 presigned requests as REST-QUERY-STRING

s3:authType recognized only SigV2 query authentication, so a policy
condition on REST-QUERY-STRING never matched a SigV4 presigned URL.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 00:01:32 -08:00
Chris LuandClaude Opus 5.5 2089abecf3 ci: cache Snowflake suite Maven deps and fall back to a Central mirror (#12763)
* ci: cache the Snowflake suite's Maven dependencies under their own key

setup-java's maven cache key hashes the repository pom.xml files, which
other Java workflows share. The Snowflake job restored that cache, which
lacks the suite's plugins and dependencies, and never saved its own because
the key already existed. Every run downloaded the whole suite dependency
tree from Maven Central and failed when Central answered 429.

Key the cache on run.sh, which pins the suite revision.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: fall back to Google's Maven Central mirror in the Snowflake job

A cold cache still resolves about 36 MB of plugins and dependencies, and
Maven gives up after three 429 retries within 30 s. Add the mirror as a
settings-profile repository; Maven tries it first and moves on to Central
for any artifact the mirror fails to serve.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 00:00:41 -08:00
Carlos Leyva 6d0a6e0935 filer: mint read_auth for the data chunks behind manifest chunks (#12756)
* filer: mint read_auth for the data chunks behind manifest chunks

* filer: cover every chunk behind manifests in read_auth (data + nested), minted after resolution, and skip when read signing is off

* filer: pass the master client as manifest cache invalidator, and move the read_auth helpers below populateBatchReadAuth for the enterprise merge
2026-11-18 19:37:06 -08:00
Chris LuandClaude Opus 5.5 9b94f00325 test: mount pinned Iceberg jars into the Spark catalog tests (#12757)
* testutil: fetch pinned Maven Central jars with a mirror fallback

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: mount pinned Iceberg jars into the Spark catalog containers

Each test started a new Spark container that resolved the Iceberg runtime
and AWS bundle through Ivy from Maven Central, so a run downloaded them six
times and a single refused request left Spark retrying until the package
timed out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: cache the Iceberg jars for the Spark catalog tests

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* testutil: test the verified jar download

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: bound Maven jar downloads by one deadline shared across retries

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: give each Maven mirror attempt its own timeout within the download budget

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 19:35:54 -08:00
Chris LuandClaude Opus 5.5 d2a0002869 s3api: fix listing regressions from argument validation (empty continuation-token, V1 Prefix encoding) (#12760)
* s3api: accept an empty ListObjectsV2 continuation-token

An empty continuation-token lists from the start and is echoed back
empty, as S3 does; rejecting it broke s3-tests
test_bucket_listv2_continuationtoken_empty.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: leave the ListObjects V1 Prefix unencoded under encoding-type=url

S3 encodes the V1 Delimiter, markers and keys but echoes Prefix as sent,
and botocore, which always asks for url encoding, decodes the V1 Prefix
only for V2. Encoding it broke s3-tests test_bucket_list_prefix_unreadable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: use a codespell-clean key in the versions encoding test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: URL-encode a V1 Prefix that XML cannot carry

Leaving the V1 Prefix raw under encoding-type=url turned a control
character such as \x01 into U+FFFD in the response. Such a prefix is
URL-encoded again; prefixes that are valid XML text, including '\n',
stay as sent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 19:34:57 -08:00
github-actions[bot] c01e87689e docs: regenerate star history chart 2026-11-19 02:58:55 +00:00
Chris LuandClaude Opus 5.5 7b4bdb77bb master: export writable volumes per disk and reset leader-only layout gauges on followers (#12751)
* master: export writable volumes per disk for each volume layout

The layout gauges only report how many volumes of a collection take
writes, not where they sit, so two writable volumes sharing one disk
cannot be seen or alerted on. The leader now exports
writable_volumes_per_disk with the layout labels plus data center, rack,
node and disk id, counting the replicas of the layout's writable volumes
on each disk (state=writable) and those of them that are crowded
(state=crowded). Series for disks that no longer hold a writable volume of
the layout, and for layouts that are gone, are deleted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* master: reset the leader-only layout gauges on followers

The layout writable, crowded, under-replicated and per-disk gauges are only
set by the leader, so a master that lost leadership kept exporting the
values of its last term indefinitely, and dashboards that aggregate over
masters could show numbers days old. A master that is not the leader now
clears them on each periodic pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: chart disks holding several writable volumes of a collection

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: keep disks of different clusters apart in the per-disk panel

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 16:17:04 -08:00
Chris LuandClaude Opus 5.5 1fd5008d63 s3api: listing argument validation and encoding-type=url for versions, prefix and delimiter (#12753)
* s3api: reject an empty ListObjectsV2 continuation-token

AWS answers continuation-token= with 400 InvalidArgument ("The continuation
token provided is incorrect"); SeaweedFS listed from the start of the bucket.
No token this server issues is empty, and raw-key and base64: tokens keep
resuming as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3api: reject an encoding-type other than url on listings

AWS answers any encoding-type other than url with 400 InvalidArgument
("Invalid Encoding Method specified in Request") on ListObjects,
ListObjectsV2, ListObjectVersions, ListMultipartUploads and ListParts;
SeaweedFS ignored the value and listed unencoded keys.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3api: apply encoding-type=url to ListObjectVersions

The handler accepted encoding-type=url but returned raw keys. It now
URL-encodes Key in Version and DeleteMarker entries, CommonPrefixes, Prefix,
Delimiter, KeyMarker and NextKeyMarker with the urlPathEscape form ListObjects
uses, and echoes EncodingType.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3api: URL-encode Prefix and Delimiter in ListObjects responses

Under encoding-type=url, ListObjects and ListObjectsV2 encoded keys,
CommonPrefixes, markers and StartAfter but echoed Prefix and Delimiter raw,
so a client decoding them once corrupted a prefix holding '%' or '+'.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 15:55:41 -08:00
646549eee1 volume growth: spread writable volumes of a layout over servers and disks (#12749)
* volume growth: spread writable volumes of a layout over servers and disks

Volume growth picked every server, and the volume server picked every disk,
independently: the server weighted by free slots, the disk by the lowest
total volume count. A grow request therefore stacks several writable volumes
of one collection on some disks while most disks get none (a Poisson spread:
with 160 replicas on 479 disks about 20 disks get two or three), and their
writes share those disks.

Data centers and racks are still picked weighted by free slots. Inside a
rack the server is now one of those holding the fewest writable volumes of
the layout per disk, weighted by free slots among them; this covers the main
server, same-rack replicas and replicas walked into other racks and data
centers. On the volume server the disk is the one with the fewest writable
volumes of the collection, then the lowest volume count as before.

A grow request registers each volume before placing the next, so a request
smaller than the number of disks lands on distinct disks.

Fixes #12746

* volume growth: scan the layout load once per grow request

Placing each volume rescanned all writable volumes of the layout under its
read lock. RecordAssign takes that lock exclusively on every assign, so with
thousands of writables (0.8 ms for 10000) every volume of a grow request
stalled assigns once more.

GrowByCountAndType now scans once and findAndGrow adds every volume it
creates to the counts. Per volume, placement costs about 130-160 us on 480
servers whatever the number of writables, against 120 us before the spread
change; the single scan is 0.17 ms for 2000 writables and 0.8 ms for 10000.

* volume growth: address review of the spread placement

- Count only disks that report volume slots in a server's load. An
  unavailable disk, or one low on space with no volumes, is still listed in
  DiskTags with no slots; counting it understated the load of a server with
  failed disks and piled volumes onto its remaining ones.
- On the volume server, judge a volume full by its .dat size, which the
  master compares with the limit, not by ContentSize.
- A server that refuses an allocation ranks last for the rest of the grow
  request, and the request retries the volume up to 3 times. The least
  loaded server is picked first deterministically, so without this one server
  that cannot fit a preallocated volume stopped the layout from growing.
- Scan the layout load after warmup, so a grow request issued right after a
  leader change sees the writable volumes the returning servers report.

* volume growth: keep the request-local placement load ahead of the request fields

The JSON request fields stay together at the end of VolumeGrowOption, and
the comment now says the load lives for the whole grow request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): place a new volume on the disk with the fewest writable volumes of its collection

Store::add_volume picked the disk with the most free slots, so the Rust
volume server ignored the collection spread rule the Go server now applies.
find_free_location ranks disks by the writable volumes of the collection
first (not read-only, .dat below the volume size limit), then by free slots.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume growth: leave a refusing server out of every placement tier

A server that refused an allocation only ranked last among the servers of
its rack, so the weighted data center and rack picks could still choose its
rack on every retry while other racks had room. It now offers no free slots
for the rest of the grow request, which takes its slots out of the rack and
data center weights too. If only refusing servers are left, they are tried
again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume growth: require the full same-rack count only in the main rack

The data center check asked every rack for SameRackCount+1 free servers,
but only the main rack holds the same-rack copies; each other rack takes one.
With a refused server left out of its rack, the check rejected a feasible
placement, the retry put the refused server back, and it was picked again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Konstantin Lebedev <whitefox@mayflower.work>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 15:51:13 -08:00
Chris LuandClaude Opus 5.5 9cd183b04f volume: fsync a needle delete that asks for it (#12736)
* volume: fsync a needle delete that asks for it

A delete now takes fsync like a write: the tombstone is synced before the
needle map drops the needle, and the .idx row before the delete is
acknowledged. A failed .dat sync takes the tombstone back off the .dat.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: honor fsync=true on DELETE and forward it to replicas

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): fsync a needle delete that asks for it

Mirrors the Go volume server: DELETE with fsync=true flushes the tombstone
before the index drops the needle and the .idx before the ack, and a failed
.dat flush takes the tombstone back off the .dat.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: let a durable delete through on a full volume

A queued delete hit the batch worker's volume size check, which a
tombstone append is exempt from inline, so fsync=true deletes failed once
the volume reached its hard limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: sync a durable delete of an already-deleted needle

An earlier delete without fsync may have dropped the needle with its
tombstone still unsynced, so the inline path syncs before acking anyway,
as the batch worker already does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: refuse a batch request that races the worker stopping

stopWorker can close the batch channel between startWorker handing it out
and the send, which panicked. The request now falls back to the inline path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): sync a durable delete of an already-deleted needle

Mirrors the Go volume server: an earlier delete without fsync may have
dropped the needle with its tombstone still unsynced.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): run a durable delete on a blocking thread

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: keep syncDelete's signature for existing callers

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: refuse a write a delete cancelled before it appends

A write cancelled while still queued behind the volume locks used to
append, publish and then tombstone itself without a sync, after its
delete may already have been acknowledged as durable. A crash could
keep the row and lose the tombstone. Refuse it before the append.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): refuse a write a delete cancelled before it appends

Mirrors the Go change: a write cancelled while still queued behind the
volume locks no longer appends, publishes and tombstones itself without
a sync after its delete may already be durable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): drop the write guards for a durable delete's flushes

The durable delete held the write lease and the io write guard across
both fsyncs, so reads of the volume waited on them. Append and publish
the tombstone under the guards and flush with only the io read side, as
write_synced_run and the Go path already do. do_delete_request keeps its
signature.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: stop taking writes when a durable delete fails to index

The needle map drops the needle before it appends the .idx row, so after
a failed append a durable retry found nothing to delete and acked, and a
later write could bury the unindexed tombstone so a reload brought the
needle back. Fence the volume the way a failed durable write does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): stop taking writes when a durable delete fails to index

Rust counterpart of the Go change: the needle map drops the needle
before it appends the .idx row, so a failed append left a durable retry
nothing to delete, and a later write could bury the unindexed tombstone.
publish_delete_request now fences the volume for a durable delete, as
publish_write does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: stop taking writes when any delete fails to index

A non-durable delete whose .idx append failed left the volume writable with
the needle already dropped from the map, so a later durable retry found
nothing to delete and acknowledged without the missing row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): stop taking writes when any delete fails to index

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: take the fsync flag on the existing delete functions

DeleteVolumeNeedle, deleteNeedle2 and syncDelete take fsync instead of
gaining *WithFsync variants.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): take the fsync flag on the existing delete functions

delete_volume_needle and delete_needle take fsync instead of gaining
*_with_fsync variants; the durable path and the deletable checks fold
back into delete_needle, and the cancelled-write check mirrors Go's
isCancelledWrite.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 15:18:29 -08:00
Chris LuandClaude Opus 5.5 23eef59417 test/s3: run the s3-listing-study listing tools and strategies in CI (#12750)
* test(s3): s3-listing-study integration suite (WIP)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test(s3): run the s3-listing-study tools against SeaweedFS (WIP)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test(s3): s3-listing-study deleted-prefix probe, fresh data dir per run

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test(s3): listing study ports keep clear of every gRPC companion

free_ports.py compared only base ports, so a base port could land on
another server's base+10000 gRPC listener.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(s3): listing study ports stay below the ephemeral range

Base ports of 20000-55000 put both them and their gRPC companions inside
32768-60999, so an outgoing connection made while weed server starts could
take a port before the server that was given it binds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(s3): listing study checks the order inside each listing page

parse_page sorted Contents and CommonPrefixes together before the probes
compared them, so keys returned out of order within one page passed. Check
each list's byte order first; the versions pager does the same.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(s3): listing study applies --only before running a probe

The filter was applied to the probe results, so a run limited to one tool
still sent every reference listing request and the deleted-prefix check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(s3): listing study requires CommonPrefixes from shallow modes

The CommonPrefixes comparison ran only when a tool returned at least one
prefix, so a delimiter listing of the canary root that came back empty
passed. Compare them for every shallow mode except the two whose
normalizer keeps keys only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 15:17:25 -08:00
Chris LuandClaude Opus 5.5 04c7b12238 s3api: reject unknown notification elements; fail test-with-server on a stuck shutdown (#12752)
* s3api: reject unknown elements in PutBucketNotificationConfiguration

xml.Unmarshal skips children that match no field, so a configuration
whose only child was an unmapped element parsed as empty and returned
success while nothing was stored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: fail test-with-server when the server does not stop

A passing suite returned success even when stop-server gave up with the
server still running.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 15:15:38 -08:00
Chris LuandClaude Opus 5.5 f5418845ce test/s3: run tigrisdata-community s3-api-compat-tests in CI (#12741)
* test/s3: run tigrisdata-community s3-api-compat-tests in CI

Runs the suite, pinned to a commit, against weed mini through a small
provider patch, and fails on any test that is not in
expected_failures.txt or on a listed test that starts passing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3/tigris_compat: prepare the suite again when the patch changes

The checkout was reused whenever it sat at the pinned commit, so an
edited seaweedfs-provider.patch, or a checkout whose patch failed to
apply, kept being built as it was. Record the commit and the patch's
hash once the patch applies, and prepare the checkout again when either
differs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: keep the pid file of a server already running

start-server removed weed-server.pid before it found the port taken, so
a second start lost the pid of the server holding it and stop-server
could no longer stop that one. Remove the file only once the port is
free, and have test-with-server stop only a server it started.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: bound the S3 port probes

A listener that accepts but never answers held curl, and with it the
readiness loop and the cleanup after it, for good. Give each probe two
seconds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: expect a listed test to end with its listed status

A listed test passed the check with any non-pass status, so a setup error
that skipped the call under test (RestoreObject's PutObject, say) went
unnoticed. Entries now expect fail unless they name another status; only
GetBucketAccelerateConfiguration, whose setup PUT is unsupported, expects
error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: refuse to start while the recorded server still runs

A second start-server before the first server answered on the S3 port
passed the port probe and dropped the pid file of the live server, so
stop-server could no longer stop it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: recognize the test server by its command line

start-server and stop-server took any process named weed at the recorded
pid for the test server, so a weed process that reused the pid would be
reported as running and killed by stop-server. Match the command line
start-server launches instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: recognize the test server by its start time

A weed mini started from another checkout has the same command line, so
a pid it reused still matched. Record the process start time next to the
pid at launch and require both before treating the process as the test
server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: PutObjectTagging_VersionId passes now

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: keep the server records until the server exits

stop-server waited a fixed two seconds after SIGTERM and then removed
weed-server.pid and weed-server.started even if weed was still shutting
down, so a later stop-server could not find it and clean could delete
data it still used. Wait up to 30 seconds for the recorded server to exit
and keep the records, failing, while it still runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: start the server on an empty data directory

A data directory left by an earlier run on other ports made the master
freeze volume growth (topology id mismatch), and every PutObject failed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 15:14:52 -08:00
Chris LuandClaude Opus 5.5 53bc62c773 s3api: accept the accelerate and notification configurations the bucket already reports (#12747)
* test/s3: run tigrisdata-community s3-api-compat-tests in CI

Runs the suite, pinned to a commit, against weed mini through a small
provider patch, and fails on any test that is not in
expected_failures.txt or on a listed test that starts passing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3/tigris_compat: prepare the suite again when the patch changes

The checkout was reused whenever it sat at the pinned commit, so an
edited seaweedfs-provider.patch, or a checkout whose patch failed to
apply, kept being built as it was. Record the commit and the patch's
hash once the patch applies, and prepare the checkout again when either
differs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: keep the pid file of a server already running

start-server removed weed-server.pid before it found the port taken, so
a second start lost the pid of the server holding it and stop-server
could no longer stop that one. Remove the file only once the port is
free, and have test-with-server stop only a server it started.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: bound the S3 port probes

A listener that accepts but never answers held curl, and with it the
readiness loop and the cleanup after it, for good. Give each probe two
seconds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: accept PutBucketAccelerateConfiguration with Status=Suspended

GetBucketAccelerateConfiguration always reports Suspended, but the PUT
answered 501 for every request. Suspended is now accepted, Enabled
stays NotImplemented, and the accelerate subresource resolves to
s3:GetAccelerateConfiguration / s3:PutAccelerateConfiguration.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3api: accept an empty PutBucketNotificationConfiguration

GetBucketNotificationConfiguration always reports an empty
configuration, but the PUT answered 501, including for the empty
configuration clients send to clear notifications. An empty one is now
accepted; a topic, queue, Lambda or EventBridge destination stays
NotImplemented.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3api: reject accelerate and notification bodies with the wrong root

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3/tigris_compat: expect a listed test to end with its listed status

A listed test passed the check with any non-pass status, so a setup error
that skipped the call under test (RestoreObject's PutObject, say) went
unnoticed. Entries now expect fail unless they name another status; only
GetBucketAccelerateConfiguration, whose setup PUT is unsupported, expects
error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: refuse to start while the recorded server still runs

A second start-server before the first server answered on the S3 port
passed the port probe and dropped the pid file of the live server, so
stop-server could no longer stop it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: recognize the test server by its command line

start-server and stop-server took any process named weed at the recorded
pid for the test server, so a weed process that reused the pid would be
reported as running and killed by stop-server. Match the command line
start-server launches instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: recognize the test server by its start time

A weed mini started from another checkout has the same command line, so
a pid it reused still matched. Record the process start time next to the
pid at launch and require both before treating the process as the test
server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: PutObjectTagging_VersionId passes now

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: keep the server records until the server exits

stop-server waited a fixed two seconds after SIGTERM and then removed
weed-server.pid and weed-server.started even if weed was still shutting
down, so a later stop-server could not find it and clean could delete
data it still used. Wait up to 30 seconds for the recorded server to exit
and keep the records, failing, while it still runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3/tigris_compat: start the server on an empty data directory

A data directory left by an earlier run on other ports made the master
freeze volume growth (topology id mismatch), and every PutObject failed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 14:55:18 -08:00
Chris LuandClaude Opus 5.5 1f5cfc7552 s3api: return x-amz-version-id from object tagging on versioned buckets (#12745)
* s3api: return x-amz-version-id from object tagging on versioned buckets

PutObjectTagging, GetObjectTagging and DeleteObjectTagging now report
the version they acted on, "null" for an object written before
versioning was enabled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3api: set the tagging x-amz-version-id only on success

The header was set before the tag-conditioned policy recheck, so a
denied request carried the version id on its AccessDenied response.
Also test the "null" version and require distinct version ids.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 14:30:55 -08:00
Chris LuandClaude Opus 5.5 b1c315b837 s3api: report no ownership controls after DeleteBucketOwnershipControls (#12748)
* test/s3: run tigrisdata-community s3-api-compat-tests in CI

Runs the suite, pinned to a commit, against weed mini through a small
provider patch, and fails on any test that is not in
expected_failures.txt or on a listed test that starts passing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3api: report no ownership controls after DeleteBucketOwnershipControls

GetBucketOwnershipControls reports BucketOwnerEnforced for a bucket that
never stored a setting, so it kept reporting it after the setting was
deleted. The delete now records that it happened, and GET answers
OwnershipControlsNotFoundError until ownership is put again. How
objects are written after the delete is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3api: answer DeleteBucketOwnershipControls with 204 No Content

AWS answers a successful DeleteBucketOwnershipControls with 204 and no
body; this answered 200 with an empty OwnershipControls document.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 14:25:15 -08:00
aceb089bc8 lance: per-table cleanup retention and on-demand purge of deleted rows (#12739)
* lance: per-table cleanup retention and on-demand purge of deleted rows

A vector deleted from an index stayed in storage until compaction
rewrote its fragment (only above 10% deletions) and the cleanup job's
168 h retention, set once per worker, aged out every version that
referenced it.

Let a table override retain_hours and min_versions in its Lance config
(seaweedfs.cleanup.retain_hours, seaweedfs.cleanup.min_versions), and
let it request a purge (seaweedfs.purge.requested). The cleanup job
proposes a table with a pending purge regardless of the version floor
and then rewrites every fragment with deletions, retrains the indices
(a remapped IVF_FLAT index still holds the deleted vector), keeps only
the latest version, and records the request as served. A version pinned
by a tag fails the purge and leaves the request pending.

The test scans every file of a local table for a deleted vector's key,
metadata and values, and finds none after the purge.

* lance: refuse to purge a sharded vector index

A retrain builds every index name over the whole table, so the purge
turned shards of one vector index (one name per fragment set) into
names that each covered every fragment, indexing every row once per
shard. Check before compacting and fail with the request left pending,
and retrain only the vector indices.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance: rebuild scalar indices in a purge

Compaction only remaps scalar indices, and a retrain leaves them alone,
so a purge reported success while a bitmap index still listed the
deleted row's value, and under stable row ids a BTree index still held
the deleted key. Build each scalar index again from the remaining rows
with the parameters it was built with.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance: purge through the cleanup job's retention policy

The purge built its own cleanup policy, so a deployment that has the
cleanup job keep more than the floor and the window (versions an
unleased reader may still be on) would have lost them to a purge. Clean
up through retention_policy, the one the job itself uses, with the
latest version kept.

Mark the request served only once the oldest version left is one the
rebuild wrote, and record the rebuild so a purge that has to wait for
its cleanup (a tag, or that policy) retries only the cleanup instead of
compacting and retraining the indices on every pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* vector: lease the version every list page reads

A list page read its version without a lease unless it was a segment
that read data and metadata twice. A purge removes every version but
the latest, and only a lease tag stops it, so a listing that opened the
version before the purge's compaction could lose its files mid-read.
Pin it as get_vectors and query_vectors do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance: give the cleanup job a build's execution time

A purge compacts the table and retrains its indices inside a cleanup
job, but the job declared no runtime defaults, so the admin held it to
its 90 s default and could expire a purge mid-rebuild. Declare the same
budget the indices job uses for a retrain.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 14:24:28 -08:00
c7d8b106d5 mount: follow a peer unlink to the surviving hard link for closed inodes (#12737)
* mount: drop a peer-unlinked name only while its inode keeps another

Move the survivor check from followSurvivingLink into InodeToPath.RemoveAlias
so the check and the removal happen under one table lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: follow a peer unlink to the surviving hard link for closed inodes

A peer deleting one name of a hard link only moved an open handle to the
surviving name. A closed inode kept the deleted name as its primary path, so
the next open by the kernel's nodeid failed with ENOENT.

Co-authored-by: skymazer <54621405+skymazer@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: share the cached-entry freshness check of peer invalidations

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep a name a local rename reclaimed before the peer unlink arrived

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep a name a local rename or link gave the inode after an older peer delete

The cached-entry check only covers a cached parent. InodeToPath now records,
per hard-linked inode, the newest name a local rename or link gave it with
the filer stamp, and the closed-inode peer-delete path keeps that name when
the delete is older. The record lives in a side map dropped on Forget, so
InodeEntry keeps its size class.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: sign rename stream responses with the filer signature

A rename response carries the stamp of the filer that applied it, so the mount
can tell which clock the stamp belongs to, as it does for create acks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: order a peer delete against a local rename or link only on one filer clock

The stamp of a local rename or link comes from the filer that applied it, and
a delete logged by another filer runs on a different clock. Keep the name only
when both carry the same nonzero filer signature; otherwise drop it as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: drop the local-name stamp fence for uncached parents

Ordering a delayed peer delete against a local rename by filer stamp needs a
record per alias, a shared filer clock, and the open-handle path to follow;
that is a larger change than this fix. Back to the cached-entry check, which
covers a cached parent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: skymazer <54621405+skymazer@users.noreply.github.com>
2026-11-18 14:13:40 -08:00
a3b2fd722d mount: report opens for reading with TouchAccessTime (-accessReport) (#12721)
* mount: report opens for reading with TouchAccessTime (-accessReport)

An open answered from the metadata cache never reaches the filer, so a
filer that tracks access cannot see that a cached file is still being
read. With -accessReport, the mount sends one TouchAccessTime per open
for reading, with the open time, whether the open was answered from the
cache or not. Write-only opens are not reported.

Reports go out in the background and never hold up the open: a report
that does not fit in the queue is dropped, and drops and failures are
logged once a minute. A filer that answers Unimplemented pauses
reporting for 10 minutes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: address review of the access report

- Drop reports still queued when reporting pauses; the filer that answered
  Unimplemented would answer them the same way.
- Bound each filer attempt to 2s, so a filer that accepts the connection
  but never answers leaves time for the next one, and keep using the filer
  that answered.
- Fail over only when a filer cannot be reached. Any answer, Unimplemented
  included, ends the walk.
- Report an O_CREAT open of an existing file with read access, which comes
  through Create instead of Open. A new file has nothing to read and is not
  reported.
- Stop the reporter on unmount: stop ends the workers and cancels the
  report in flight.
- Say in the flag help that it has no effect unless the filer implements
  TouchAccessTime.
- Tests no longer wait a fixed time to see that nothing else was sent; they
  look at the queue of a reporter that is not started.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: fail an access report over only when the filer did not answer

A status the filer sent ends the walk even when its message contains a
word the transient classifier matches, so an Unimplemented answer still
pauses reporting and a rejected request is not resent elsewhere.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep a filer selection made while an access report failed over

A report that reached a later filer moves the mount there only if the
selection it started from is still current, so a slower report cannot
undo another call's failover.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: share an access report's time with the filers not yet tried

Each attempt is capped by an equal share of the report time left, so a
row of stalled filers cannot use up the report before a healthy one later
in the list is tried.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: report the path an open for reading came through

A create that opens an existing hard link reports the name it was
given rather than the inode's first known name. Open reads the inode's
path from the inode table rather than the handle's saved path, which
other handle operations may rewrite concurrently.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: check the access report stop and failover tests without races

The post-stop report is checked on a reporter without workers, so no
worker can take an accepted report before the check, and the failover
test asserts that the stalled filer was tried first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* pb: keep the shared channel when the peer answers with its own status

gRPC never raises InvalidArgument, NotFound, AlreadyExists,
FailedPrecondition, OutOfRange or DataLoss itself, so such a status is
the peer's answer. It fell through to the string matching, and a message
naming a path like /transport closed the cached channel and cancelled
every other call on it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: say which filer selection an access report keeps

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: bound the in-flight call in the rejected-report channel test

The call held in flight went through touchAccessTime, whose 2 s attempt
cap could expire on a slow runner and fail the test on a healthy channel.
It now uses callFiler with the test deadline. The wait for it to reach
the filer also ends if it returns early or the deadline passes, and the
blocked handler is released on every exit path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: report each file at most once per -accessReport.interval

Every read open used to send a TouchAccessTime, adding a filer metadata
call per open. A path reported within the interval (default 1h) is now
skipped; the last report times live in a bounded LRU. 0 keeps reporting
every open.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: report an open again once the file was modified since its last report

Within -accessReport.interval a read open was skipped even after a write,
so the filer kept an atime older than the mtime until the interval ran
out. relatime records that first read after a write; do the same by
comparing the entry's mtime with the last report.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep the report time apart from the mtime it saw

Recording max(now, mtime) as the last report let a file dated a day ahead
suppress reports for a day plus the interval. Record when the report was
sent and the mtime the entry had, and treat any change of that mtime as a
modification, so neither clock can stretch or shorten the interval.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: MapleWhiteZZZ <18653045+MapleWhiteZZZ@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-18 14:11:11 -08:00
Chris LuandClaude Opus 5.5 9f413ff2c8 shell: volume.check.disk skips replicas whose live counts match (#12743)
* shell: volume.check.disk skips replicas whose live counts match

FileCount counts every write and DeleteCount every delete or overwrite
until a vacuum drops them, so a replica vacuumed while its peer was not
keeps a different FileCount forever and was re-checked on every run
although both hold the same live needles. Only treat counts as differing
when the live counts (FileCount - DeleteCount) differ too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: volume.check.disk skips equal live counts only across compaction revisions

Equal live counts at the same compaction revision mean a missed write
and a missed delete, not a vacuum-state difference, so keep checking
those pairs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: volume.check.disk real-time recheck ignores heartbeat revisions

VolumeStatus counts are newer than the heartbeat's compaction revisions,
so a vacuum in between could make a pair at the same revision look
vacuum-skewed. Apply the old count rules there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: volume.check.disk skips on live counts only when no repair could follow

Equal live counts across compaction revisions can still hide a missed
write and a missed delete. Keep checking such pairs when -syncDeleted
could propagate the delete, or when -resurrectMissingNeedles could copy
a needle into a never-vacuumed replica.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 14:10:47 -08:00
e037a9c1ce s3vectors: enforce AWS metadata, key and filter limits (#12738)
* s3vectors: enforce AWS metadata, key and filter limits

PutVectors accepted vector metadata of any size and key count, bounded
only by the request body limit, and keys of any length; QueryVectors
accepted filters of any size. Apply the AWS S3 Vectors limits
(https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-vectors-limitations.html),
taking the ENHANCED index values where the modes differ, since this
implementation prefilters and takes no indexMode:

- 100 KB total metadata per vector, which also bounds a single
  non-filterable field;
- 2 KB filterable metadata per vector;
- 50 metadata keys per vector;
- 1024-character vector keys;
- at most 10 nonFilterableMetadataKeys of 1 to 63 characters at
  CreateIndex;
- 100 filter constraints per QueryVectors request.

A violation is a ValidationException before the engine is called.

* s3vectors: accept metadata whose JSON is exactly at the byte limits

metadataEntrySize counts a trailing comma for every member, so starting
from both braces overcounted by one and rejected metadata that encodes to
exactly 2 KB filterable or 100 KB total.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3vectors: rewrap the metadata limits header comment

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 13:49:15 -08:00
Denis 3db54ee7e5 kms/openbao: refresh the AppRole token before it expires (#12734)
* kms/openbao: refresh the AppRole token before it expires

The provider logged in with AppRole once, when it was created, and
never renewed the token. After the token's TTL every SSE-KMS upload,
and every read of an SSE-KMS object whose data key was not cached,
failed with "permission denied" (HTTP 500 to the client) until the S3
server was restarted.

With AppRole authentication the provider now logs in again in the
background once two thirds of the token's TTL have passed, and retries
a failed login every 5 s. Each login is bounded by the client's
timeout, or by 30 s when request_timeout is not positive. Close stops
the refresh. A static token, or an AppRole token without a TTL, is left
as it is.

* kms: keep the OpenBao token refresh across configuration reloads

- AddKMSProvider and InitializeGlobalKMS create a provider instance of
  their own with the new NewProvider instead of the registry's cached
  one: a reload closed the cached instance and then got it back, which
  now stopped its token refresh. The new instance is created before the
  old one is closed, so a configuration that fails leaves the current
  provider working
- openbao: time each refresh from the start of the login that issued
  the token, since its TTL runs from then rather than from the response
- tests: a reload gives a new instance and closes the old one, a failed
  reload keeps the old one; the refresh survives a reload; a slow login
  response

* kms: create a replacement provider outside the manager lock

AddKMSProvider ran the provider factory, which for OpenBao is a login
that may wait for its timeout, while holding the manager lock, so every
lookup, also of unrelated providers, waited for it. The new provider is
now created before the lock is taken; the lock covers only the swap, and
the replaced provider is closed after it. Test: a lookup while another
provider is being created.
2026-11-18 11:07:09 -08:00
Chris LuandClaude Opus 5.5 94de65b3b5 s3api: answer a malformed Cache-Control on PutObject with InvalidRequest (#12733)
PutObject rejected a Cache-Control value it could not parse with
InvalidDigest, whose message is about Content-MD5, even when the request
sent no Content-MD5. CreateMultipartUpload already answers the same parse
error with InvalidRequest; PutObject now does too.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 11:03:57 -08:00
ebc77400ca operation: reassign an upload after a proven-expired volume JWT (#12729)
* operation: reassign an upload after a proven-expired volume JWT

* seaweed-volume: mark an expired write JWT refused before append

Mirror the Go volume server: a multipart POST whose write JWT is signed
with the current key, names this fid and fails only on exp gets
X-Seaweed-Write-Outcome: expired-jwt-before-append on its 401, so the
uploader reassigns instead of failing. Both checks use one guard
snapshot, so a key rotation between them cannot drop the marker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* operation: keep same-FID retry for filer-proxy expiries, snapshot the write signing key, and make the RetryForever budget test fail fast

* operation: re-mint the filer credential before a proxy same-FID expiry retry

* operation: make a proxy marked-expiry terminal in the outer retry too, not only the inner same-FID loop

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 11:02:52 -08:00
Chris LuandClaude Opus 5.5 ecbf7af1ca mount: compare permissions against local ids without mapping them again (#12726)
* mount: compare permissions against local ids without mapping them again

Access, open and create (without -defaultPermissions) ran FilerToLocal on
entries that maybeReadEntry and maybeLoadEntry already return in local ids.
Under a uid/gid map where a local id is also a mapped filer id, the owner
was mapped twice and the check refused the file's own owner.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: cache a deferred create's placeholder in filer ids

The meta cache stores filer uid/gid and maps them to local ids on every
read, but createRegularFile cached the deferred or writeback placeholder
with the caller's local ids. The second mapping in the permission checks
used to cancel that out under a swapping id map; without it the owner of
a placeholder was refused. Cache a mapped clone and keep the handle's
entry in local ids.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: test that a deferred create returns its entry in local ids

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 11:01:30 -08:00
b64895ed0a filer: stop following a peer that stays unreachable (#12725)
* filer: stop following a peer that stays unreachable

* filer: re-track an answering peer only while it is still followed

A signature read that succeeds just before the master removes the peer
called initPeerWatermark after the removal, clearing its mark. The loop
then exits, so the watermark it re-tracked never advances or drops and
pins aggregated readers at the settled horizon.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: forget a peer's unreachable clock when the master removes it

Only a successful connection cleared the clock, so every departed address
that once failed stayed in the map for the filer's lifetime, and a re-added
peer inherited the old outage. The removal now drops it, and the clock is
guarded by peerChansLock so the removed peer's exiting loop cannot start a
new one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: report the unreachable mark only when it is set

Once peerResumeRetention forgets the mark of a peer that stays
unreachable, markPeerWatermarkRemoved no longer marks it (its watermarks
are gone), but notePeerUnreachable still saw no mark and logged
"treating it as removed" on every retry. markPeerWatermarkRemoved now
reports whether it set a new mark.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: fix the unreachable-peer subscriber's start before the baseline write

The subscriber goroutine picked its start time itself, so a baseline write
that finished first fell before the subscription and the test failed after
20 seconds with correct delivery.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 11:00:15 -08:00
72ac9e96b2 mount: answer an interrupted read with EINTR, not EIO (#12723)
* mount: answer an interrupted read with EINTR, not EIO

The kernel's FUSE contract (Documentation/filesystems/fuse/fuse.rst,
"Interrupting filesystem operations") lets a filesystem either ignore an
INTERRUPT or honor it by replying EINTR to the original request. Read did
neither: the FUSE cancel channel cancels the read's context, the chunk
fetch fails with "context canceled", and every error became EIO.

Interrupts come from any signal the reader handles. Go's runtime sends
SIGURG for goroutine preemption, so Go programs reading under load hit
this on healthy data: on a 128-thread 64 KiB random-read test about 250
of 5 million reads failed with EIO, and none with GODEBUG=asyncpreemptoff=1.

Read now returns EINTR when its own context was cancelled, the way
dirReadStatus already does for directory reads and lookups do since
#11717. Go's os.File retries EINTR, so those readers see no error. A
cancelled read also skips the entry refresh retry, and neither read log
line reports an interrupt as a warning or error. Failures without an
interrupt stay EIO.

Tests: TestReadInterruptedReturnsEINTR (fails on the old code with EIO),
TestReadFailureWithoutInterruptReturnsEIO; weed/mount/... passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: require the failed-read test to reach the filer

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: skip the read retry once the request is interrupted

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-18 10:58:05 -08:00
11af0e0c12 mount: move an open file's read view to the chunks a merge wrote (#12722)
* mount: move an open file's read view to the chunks a merge wrote

When a flush finds that a file's chunks take more than twice its size,
maybeMergeChunks reads the file back and writes it again as new chunks.
The commit that publishes them makes the filer delete the chunks they
replaced. The flush put the new chunks into the handle's entry but left
its read view on the old ones, so every descriptor that kept the file
open went on reading the replaced chunks. Those read back until their
volume is vacuumed. After that a read first retries the fetch of the
missing needle for about 13 s; then refreshEntryForRead installs the
filer's entry and the read is retried. The refresh skips a handle with
writes it has not flushed, and such a read returns EIO instead.

Files that are kept open and rewritten in place, such as PostgreSQL
data files and SQLite databases, are the ones whose chunks bloat, so
they are the ones that hit this.

installRewrittenChunks puts the merge's chunks into the entry and the
read view together, under the entry lock. Chunks attached while the
merge ran were not part of it and stay. Flushes without a merge are
unchanged.

TestMergeMovesReadViewOffReplacedChunks covers the handle.
TestReadOpenHandleAfterChunkMerge runs it on a real mount, with and
without an unflushed write; for it, the fuse_integration framework
starts mini on a free master port and exposes the address.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: drain uploads before a merge replaces the chunks

Uploads attach their chunks without the handle lock, first to the entry
and then to the read view, and writes that landed after doFlush drained
the pages may still be uploading when the flush takes its lock. An
upload that attached during the merge could keep a chunk older than
data the merge already holds, or put a chunk the merge replaced back
into the read view. Either outranks the merge's zero-timestamp chunk.

When the chunks are bloated enough to merge, the flush now drains the
pages again while its lock holds writes off, and takes the chunks again.
FlushAll returns only after every upload has attached to both the entry
and the read view, so nothing attaches during the merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: leave a pre-merge upload failure to the next flush

The drain before a merge returned an upload failure through the
metadata flush. doFlush then retried it as a filer error and answered
EIO, where the page flush answers a full disk with ENOSPC. The drain is
there only to wait for the uploads: a failure stays with the pages, and
the next flush reports it as a write error, as it would without a merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep chunks attached while a flush installs its compacted list

A write that lands after doFlush drained the pages can still be uploading
when flushMetadataToFiler takes its chunk snapshot. Its upload attaches the
chunk without the handle lock, and the plain flush then replaced the
entry's chunks with the list compacted from the snapshot, dropping it from
the handle and from every later commit. Keep chunks attached after the
snapshot, as the merge path already does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: fail a merge's flush on a failed drain

A failed upload found by the drain before a merge was only logged, and
the flush went on to publish a size covering bytes that no chunk stores.
It now fails the flush as a pageUploadError before anything is
published; doFlush maps it with writeErrorToFuseStatus, as it does its
own FlushData failure, and does not retry it as a filer error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: clone a flush's request entry under the entry lock

The request snapshot cloned the live entry after GetEntry had released
its read lock. Uploads attach chunks without the handle lock, so an
AppendChunks that landed during the clone raced it on the Chunks field.
Clone inside UpdateEntry, as content flush recovery does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: leave a merge to a later flush while uploads are pending

Before a merge the flush drained the pages while it held the handle and
publication locks. A writeback upload retries a transient storage error
until it succeeds, so that drain could hold both locks for as long as
storage stayed down, stalling reads, writes and stat on the handle.
The flush now checks for pending uploads and, if any, publishes the
compacted chunks and leaves the merge to a later flush. With none
pending the lock still holds writes off, so nothing attaches during the
merge, and a failed upload still fails the flush as a write error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep room for pending uploads when a flush merges onto a moved entry

A flush that leaves the chunk merge to a later flush publishes while
uploads are still pending, and writes that landed after the drain can be
uploading on any flush. Merging a peer's inline content or a shrink into
such a publication lets the upload attach its chunk next to the inline
bytes, where reads never reach it. Tell createContentEntry when uploads
are pending so the merge keeps room for them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: test chunk merges through the filer's chunk proxy, also under load

The tests serve the filer's chunk proxy, and like the filer the fake
deletes the chunks each commit replaces, so a read view still naming
them fails:
- TestFlushMergeThroughChunkProxy/Quiet: the flush merges, and reads
  through the open handle return the file from the merged chunk alone.
- TestFlushMergeThroughChunkProxy/UploadInFlight: an older page is still
  uploading while a newer page of the same bytes has attached. The flush
  leaves the merge, reads return the newer page, and the next flush
  merges.
- TestFlushMergeRacesWithWritesAndReads: four writers, three flushers
  and a reader on one handle, with uploads delayed at random. The
  writers pause now and then, so 14 to 18 flushes a run merge and the
  others leave the merge to a later one. Every read must succeed within
  2 s, and at the end the handle and the filer must both hold exactly
  what was written.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: let a held upload go when UploadInFlight fails

The subtest's proxy holds the older page's upload until the test
releases it, and only the passing path did. A failure in between left
the handler blocked, so the proxy's cleanup waited on it and the failure
surfaced as a suite timeout. Release it from a cleanup as well, once,
and bound the waits for the upload and the drains.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-18 10:51:31 -08:00
de5b3305a7 mount: ask the filer again when a deferred create is published mid-lookup (#12720)
* mount: ask the filer again when a deferred create is published mid-lookup

A program that writes a file under a temporary name and renames it while
the file is still open (safetensors save_file does, through Rust
tempfile's persist) looks the temporary name up before the rename. When
the parent directory's listing is not cached, that lookup goes to the
filer. If a FLUSH publishes the deferred create while the lookup is in
flight (a forked child closing its inherited descriptor sends one), the
filer can answer not-found from a read taken before the create
committed. By the time the answer arrives the flush has cleared the
handle's dirty flag, so lookupEntry finds nothing local to explain the
answer, logs "possible coherence bug" and returns ENOENT, and the rename
fails although the file exists.

Note before sending the request whether the inode had unpublished
metadata (a dirty handle or an async flush in flight). On not-found, if
it had and has since been published, ask the filer once more: the filer
acknowledged the publish before the second request is sent. A second
not-found stands, so a file another mount removed is not brought back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: judge a placeholder's not-found by one look at its base

Review of the previous commit found two problems. The retry check and the
not-found fallback read the handle separately, so a flush that finished
between the two reads still turned the answer into ENOENT. And the
pre-request read of dirtyMetadata, a plain bool that Write and the flush
set under the handle lock, raced with them on every lookup that reached
the filer.

Note instead whether the handle is a local placeholder: a deferred create
has no filer base (baseEntry) until a flush publishes it. A not-found for
a placeholder is judged by one atomic look at that base. With none, the
filer cannot have the file yet and the handle answers. With a base, the
filer acknowledged the publish, so ask once more. dirtyMetadata is not
read for this, and the not-found fallback keeps the existing check, with
the placeholder decision added.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: answer a placeholder from its own handle; a second miss is final

Follow-ups from review of the previous commit:

- Pick the placeholder after the async-flush wait, from the path's
  current mapping.
- A placeholder that still has no base answers from the observed handle
  itself, instead of through the not-found fallback, which re-resolved
  the path and could miss once the handle was released and the cache
  entry evicted.
- A not-found from the second request is final. The flush installs the
  base before it clears dirtyMetadata, and keeps it set when a write
  landed meanwhile, so the fallback could return a file another mount had
  removed.
- Test gates release on cleanup, so a failing test leaves no handler
  blocked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: read a placeholder's entry the way GetAttr does

The previous commit answered a not-found for a placeholder from the
handle's live entry under entryLock. Neither of the entry's writers takes
that lock: Write and the metadata flush change size, times and chunks
under the handle lock, and async upload workers append chunks under the
LockedEntry lock, so the lookup could read an entry while it changed.

Answer with the copy the create cached when it is still this file's
(same inode), as the not-found fallback already prefers. Otherwise copy
the handle's entry under both writers' locks, in GetAttr's order (handle
lock shared, then the LockedEntry lock), and convert the copy. No caller
of lookupEntry holds a handle lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: cache a deferred create's placeholder in filer ids

The meta cache stores filer uid/gid and maps them to local ids on every
read, but createRegularFile cached the placeholder with the caller's local
ids. Under a uid/gid map whose local id is also a mapped filer id, a lookup
or stat answered from the placeholder reported the wrong owner. Cache a
mapped clone and keep the handle's entry in local ids.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: assert the placeholder is cached before checking its owner

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: check the snapshot owner after evicting the cached placeholder

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-18 10:50:28 -08:00
04e135c21e fix(helm): separate volume DNS discovery from ingress traffic (#12719)
* helm: set publishNotReadyAddresses on the volume headless service

* fix(helm): separate volume DNS discovery from ingress traffic

* test(helm): cover volume DNS discovery and ingress routing

* fix(helm): rename the volume routing Service so volumes.client cannot collide

* docs(helm): name the ready-only Service for explicit volume backendRefs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(helm): fail when the volume routing Service name collides after truncation

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(helm): point explicit volume backendRefs at the rendered client-volume name

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 10:49:35 -08:00
Chris LuandClaude Opus 5.5 862913d199 volume, ec_balance: clean up failed EC shard copies; stop re-proposing empty-index volumes (#12718)
* volume: remove files a failed VolumeEcShardsCopy wrote

The shard files were copied before the .ecx, so a source holding a
0-byte index failed the request only after its shards had landed, and
nothing removed them: every retried move stranded another unmounted
shard on a new disk.

Copy the .ecx first, staged beside its target, so a bad index fails
before any shard lands; rename it into place only after the shards and
sidecars have copied, so a failed copy never truncates or replaces an
.ecx the destination already has. On any error, remove exactly the
files this call created; files already on the disk (an existing
holder's shards, index or journal) are left alone. Go and Rust volume
servers behave the same.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* ec_balance: skip EC volumes whose every holder reports an empty .ecx

A shard copy refuses a 0-byte .ecx, so detection proposed the same
failing move of such a volume every cycle. Leave volumes with no
holder reporting a needle out of planning; their shards still count
toward disk occupancy.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* volume: serialize VolumeEcShardsCopy per volume

Overlapping copies of one vid shared the staged .ecx, and a failed copy's
cleanup removed shards that a concurrent copy had already committed and
mounted. Hold a per-vid EC lifecycle lock across the whole copy and its
cleanup, named as in the enterprise build (Store.EcLifecycleLock /
ec_lifecycle_lock).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: overwrite an existing .ecx in place when committing a copied index

A mounted EcVolume reads its index through an open handle, so renaming
the staged copy over it left the reader on the replaced inode: tombstones
the source had folded into its .ecx never reached the mounted holder.
An index already on the disk is now overwritten in place, as before the
staging change; a fresh destination still gets the rename. Go and Rust.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: stage replacements of shards a VolumeEcShardsCopy finds on disk

The shard transfer opened an existing destination shard with O_TRUNC and
removed it on a stream or write error, so a failed retry deleted a shard
this disk already held. A shard already present is now copied to
<shard>.copying and renamed into place only after every other file has
copied; a mounted copy is reopened so it serves the new file. Fresh
shards still write in place and are removed on failure. Go and Rust.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: stage the .vif and .ecsum a VolumeEcShardsCopy brings until commit

The sidecars landed as soon as they transferred, ahead of the deferred
.ecx commit, so a failure after them left a new .vif beside the index
the failed copy kept. They are now staged as .copying and renamed into
place with the shards and index once every transfer has succeeded; a
source without one still stages nothing. Go and Rust.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: run a Rust VolumeEcShardsCopy to completion when its caller goes away

tonic drops a cancelled handler future at its current await, which
released the EC lifecycle lock while file operations the copy had started
(a cleanup remove_file on the blocking pool, a create or rename) could
still land; a retry taking the lock could then lose its files to them.
The copy now runs in a spawned task that owns the lock, as Go's handler
runs to completion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: fail the sidecar copy tests with an unreadable source .ecsum

A directory at the source path reads as "not found" on volume servers
that skip non-regular EC files, which turned the failure into a no-op.
A mode-000 file fails the transfer on every implementation; root and
Windows, which can read it, skip the test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: copy each shard a VolumeEcShardsCopy lists once

A shard listed twice and already held here was staged twice; the first
commit rename consumed the staged file, so the second failed the request
after the shard was replaced, and every retry failed the same way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: wait out an in-flight VolumeEcShardsCopy before mounting its shards

A shard the copy had already landed could be mounted while the copy was
still running; the copy's failure cleanup then removed the file but left
the shard registered and announced. MountEcShards takes the per-volume
lifecycle lock, as the enterprise build already does, and the Rust mount
handler takes the same lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: fail a VolumeEcShardsCopy whose replaced mounted shard cannot be reopened

The copy reported success while the mounted shard kept serving the
replaced file. The error lets the caller retry the copy instead of
treating the shard as delivered.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: check which transfer fails in the EC copy cleanup tests

An earlier failure, before any shard or sidecar landed, would leave the
destination unchanged and pass every assertion without exercising the
cleanup. The sidecar test also skips on Windows, where a mode-000 file
stays readable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: fail the sidecar transfer in the EC copy test by injection

A mode-000 file is still readable to root and on Windows, so the test
skipped there. The source now returns an error for the .ecsum request
only, and the test checks that it served .vif before that failure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: take the EC lifecycle lock for a blanket full teardown

A teardown acknowledged while a detached copy of the same volume was
still waiting on its source let that copy recreate the index and shards
afterwards. Go and Rust now wait for the copy, as the enterprise build
does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: keep the request id on a Rust EC shard copy

The copy now runs in a spawned task, outside the gRPC layer's request id
scope, so its CopyFile calls to the source went out with a new id. Carry
the caller's id into the task.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: unmount a replaced EC shard whose reopen fails

The failed copy left the mount reading the replaced file's inode, so the
destination kept serving the old shard. Unmount it, which also reports
the removal to the master, so a retry copies and mounts it again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: make the Rust EC copy tests fail for the right reason

The sidecar test now serves a directory as the source .ecsum, which root
cannot read either, so it no longer returns early there. The cancel test
holds the store until the handler's join reports the cancellation, so a
copy that finished before the abort can no longer pass it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: stage every shard a VolumeEcShardsCopy transfers

A shard missing here was written in place with a cleanup guard armed
before its transfer started, so a failed copy removed a shard that
ReceiveFile wrote meanwhile. Every shard is now staged, and only one
that did not exist when it is moved into place is removed on failure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: take the EC lifecycle lock for VolumeEcShardsRebuild

A rebuild folds the deletion journal into .ecx. Run during a copy of the
same volume, the copy's index then replaced the folded tombstones. Go
and Rust take the lock the enterprise build already takes here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: check the staged .vif bytes in the EC sidecar failure test

A source without the optional .vif also returns success, so the served
extension list could not prove the transfer. The test now reads the
staged .vif when the .ecsum request fails and compares it with the
source's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: unmount a failed reopen's shard on the copy's disk only

UnmountEcShards walks every disk, so a reopen failure on the destination
also unmounted a healthy copy of the same shard on a sibling disk. The
copy now unloads the shard from its own disk and reports that disk's
removal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: take the EC lifecycle lock for an ordinary EC shard delete

Shards a copy has staged are invisible to the delete's last-shard check,
so deleting the only canonical shard removed the index the copy was
about to commit its shards against. Go and Rust deletes now wait for an
in-flight copy of the same volume.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: leave a mounted shard's copied file in place on copy failure

A shard mounted on the copy's disk whose file was missing got the copied
file reopened into its mount, then that file was unlinked by a later
failure's cleanup while the mount kept serving it. Only an unmounted
shard's new file is removed now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: skip repeated shard ids with a seen set

Scanning the request prefix for each id made a request with many
repeats quadratic while the copy held the volume's lifecycle lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: make the Go EC copy and lock tests observe their effects

The fake source counts requests per extension, so the duplicate-id test
proves .ec05 is fetched once, and returns fixture errors instead of
failing the test from its handler goroutine. The staged .vif reaches
the test through a channel. The mount and teardown lock tests seed real
files and check that nothing happened while the lock was held and that
the operation took effect after it was released.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: poll the handler directly in the Rust EC lifecycle lock tests

A spawned handler that had not run when the 200 ms timer fired passed
the blocking check without ever reaching the lock. The tests now poll
the pinned handler future themselves, which guarantees it is waiting on
the lock, and cover the ordinary shard delete as well.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: observe the lifecycle-lock waiter in EC copy lock tests

The mount, teardown, shard-delete and rebuild tests slept 200 ms while
holding the lock, so a handler scheduled only after the unlock passed
without ever taking it. Wait until the handler's goroutine is blocked on
the mutex instead; each test now fails within its timeout if the handler
skips the lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: check the shard is staged before the failure-path write in a copy test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: recover a missing EC index under the volume's lifecycle lock

Missing-index recovery fetched the index and then ran a store-wide mount
of every orphan shard, without the per-volume lifecycle lock. It could
mount shards an in-flight VolumeEcShardsCopy of another volume had just
renamed in, and that copy's failure cleanup then unlinked a mounted shard.
Each recovered volume now holds its lifecycle lock across a re-check, the
fetch and a mount restricted to that volume.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): recover a missing EC index under the volume's lifecycle lock

Mirror of the Go change: recover_missing_ec_indexes takes the volume's
ec_lifecycle_lock, re-checks that the index is still missing, fetches it,
and mounts only that volume's shards through mount_recovered_ec_shards_only,
so an in-flight VolumeEcShardsCopy of another volume never has its
renamed-in shards mounted from under it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: mount a volume whose index a copy installed during recovery

Recovery re-checks under the lifecycle lock that the index is still
missing. When a copy installed it in the meantime, the re-check skipped
the mount too, and the volume's other shards stayed unmounted because
later scans no longer list it. Skip only the fetch and still mount the
volume's orphan shards under the lock, in Go and Rust.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: fetch a recovered EC index into the disk the locked re-check found

The re-check under the lifecycle lock only kept a yes/no answer, so the
fetch still wrote to the destination from the first scan, which may be a
disk that has since gone away. Fetch into the record the re-check returned.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: mount a copy-installed EC index even when no peer is known

Recovery skipped the locked step when the master was unreachable, the
lookup failed or no peer was listed. A copy that installed the index after
discovery then left the volume's other shards unmounted, and later scans
no longer list it. Always run the locked re-check, and give up for lack of
peers only when the index is still missing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: grow the stack buffer before scanning for a lock waiter

A full buffer can cut off the handler goroutine's stack, so the test would
report a handler that is waiting as one that never waited.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): fetch a recovered EC index into the disk the locked re-check found

Mirror of the Go change: fetch into the record the re-check under the
lifecycle lock returned instead of the destination from the first scan.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): mount a copy-installed EC index even when no peer is known

Mirror of the Go change: a failed lookup or an empty peer list no longer
skips the locked re-check, so a volume whose index a copy installed after
discovery still gets its orphan shards mounted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): run missing-index recovery to completion when the mount RPC is cancelled

Dropping the mount handler mid-recovery released the volume's lifecycle
lock while the .ecj merge it had started on a blocking thread was still
landing, so a teardown could remove the volume before that late write.
Run recovery on its own task, as the copy handler does, so the lock is
held until it finishes; Go's handler already runs it to completion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: skip other volumes before stat in a single-volume EC scan

Recovery re-checks and mounts each volume with its own scans, so a
request with many orphans repeated whole-store discovery once per volume.
The index-owner and orphan-shard scans now take the same volume filter
and drop other volumes' files by name, before the per-file stat, and the
locked re-check scans only its own volume.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): skip other volumes before stat in a single-volume EC scan

Mirror of the Go change: the index-owner and orphan-shard scans take the
volume filter and drop other volumes' files by name before the per-file
stat, so recovery's per-volume re-check and mount no longer repeat
whole-store discovery under the store lock for every orphan.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): announce shards recovered after the mount RPC was cancelled

Recovery now outlives a cancelled mount RPC, but the heartbeat nudge
stayed in the handler, so shards it mounted after the caller left waited
for the next full EC heartbeat. Nudge it from the recovery task, as Go's
loads announce each recovered shard.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: match the collection as well as the volume id in the recovery re-check

Discovery keys orphans by (collection, vid), but the locked re-check
matched the vid only. Two collections with orphans of the same vid could
both resolve to the first record, so the other's index was never fetched.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): match the collection as well as the volume id in the recovery re-check

Mirror of the Go change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 10:48:21 -08:00
Chris LuandClaude Opus 5.5 8c6d09f498 s3api: answer NoSuchBucket from GetBucketRequestPayment for a missing bucket (#12716)
* s3api: answer NoSuchBucket from GetBucketRequestPayment for a missing bucket

The handler wrote the static BucketOwner answer without resolving the
bucket, so a bucket that does not exist looked like one that does.
Check the bucket first, as PutBucketRequestPayment and the other static
configuration getters already do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: test GetBucketRequestPayment on existing and missing buckets

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 22:38:48 -08:00
eb1dca8088 filer: release the stamps of parent directories a rename creates (#12711)
* filer: release the stamps of parent directories a rename creates

* filer: test that a suppressed create keeps the moved entry's stamp in flight

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 22:37:56 -08:00
deacd575ea operation: prepare upload content before assigning the volume (#12714)
* operation: prepare upload content before assigning the volume

* operation: keep the existing upload function names for prepared payloads

UploadData and doUpload prepare once and hand the prepared payload to
retriedUploadData, so the retry loop, doUploadData, uploadChunk and
uploadChunkToHolders take a preparedUpload directly instead of growing
a prepared twin of each and branching on nil.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* operation: send ciphertext whenever the prepared payload carries a cipher key

doUploadData chose the cipher branch from the option, so a payload
prepared under a different option could be sent as plaintext metadata
or lose its key. Decide from the prepared payload itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* operation: build a chunk's preparation and upload options in one place

The chunk loop prepared with a hand-written option while uploadChunk and
uploadChunkToHolders built their own, so a setting that shapes the bytes
could reach one and not the other. chunkUploadOption now feeds all three.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* operation: test that a reassigned chunk reuses its prepared payload

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 22:36:46 -08:00
Chris LuandClaude Opus 5.5 194e6ede9f seaweed-volume: reject FetchAndWriteNeedle reads that miss the requested size (#12717)
The Go volume server fails the RPC when the remote read returns a
different length than req.Size; the Rust server cached whatever came
back as a full-size chunk.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 22:35:38 -08:00
97f49571a1 wdclient: let master-pushed adds replace the gateway's lookup-fed locations (#12705)
* wdclient: let master-pushed adds replace the gateway's lookup-fed locations

After a master leader change the S3 gateway clears its FilerClient cache and
refills it through filer lookups. The filer's own master client may still
answer from the previous leader's generation until the new leader has heard
from the volume servers, so a volume server lost together with the old leader
is cached again. The new leader never knew that server and never pushes its
removal, and the gateway keeps trying it until reads fail on it volume by
volume, one 30 s dial timeout at a time.

The gateway's master client relearns every volume from the new leader's
adds. On each add, cache the volume where the master client now knows it to
be, in place of what a lookup cached, and keep lookups already in flight from
caching their answer over it. A provider answer for a volume is now cached
whole or not at all.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* wdclient: take a lookup's fill epoch before its second cache check

A master add that landed between the second cache check and the read of
the fill epoch was invisible to both: the check had already missed, and
the epoch the lookup then read already included the confirmation, so a
filer answer that still named the lost server was merged back into the
confirmed entry. Taking the epoch first means such an add either shows
up in the check or stops the answer from being cached. The same order
covers a reset or clear in that gap.

TestConfirmDuringLookupIsNotUndone fires a confirmation from a test hook
at that point and fails with the old order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* wdclient: gate the master lookup provider's cache writes on the map epoch

The provider wrote each answer straight into the vid map, so a lookup sent
before a master reset and answered after it landed in the new generation,
which a gateway's FilerClient now reads as the current master's view. The
answer now goes through fillLocations, which drops it once a reset passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* wdclient: keep the vidMapClient doc comment on its type

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-17 19:10:20 -08:00
Konstantin Adzer 91eaabf371 s3: answer ClientDisconnected for a GET the client aborted before the first byte (#12707)
* s3: answer ClientDisconnected for a GET the client aborted before the first byte

Fixes #12706

* s3: tell a gone client on TLS and HTTP/2 too, and not on a connection the server closed
2026-11-17 17:15:00 -08:00
c91da3d1f8 filer: fail health check when absent from lock ring (#12709)
* filer: fail health check when absent from lock ring

If the master broadcasts a lock ring that does not include this filer
(e.g. the late-cleanup race in seaweedfs#11442), every lock acquisition
fails with "no lock server found" and locked writes, including S3 PUTs,
hang indefinitely while /healthz and /readyz still report OK.

Make the filer health handler return 503 once this filer has been
missing from its own lock ring for longer than a 30s grace period, so
Kubernetes liveness probes restart it and it re-registers with the
master. The grace period covers startup before the first ring update
and the master's ring-change batching delay.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: hold the lock-ring health clock until the filer announces

AnnounceOnceCaughtUp can hold the master registration for up to a minute
while HTTP already serves, so the filer cannot be in the ring yet. Report
healthy until the announce gate opens and start the grace period after it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: keep a probe racing a ring rejoin from reading a reset clock

A probe that loses the CompareAndSwap to a concurrent healthy probe could
load the reset zero and report the filer absent since the epoch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: trim the lock-ring health comments

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: restart the lock-ring health grace when a ring update brings the filer back

The clock only reset when a probe saw the filer in the ring, so a rejoin
and a second removal between two probes reused the first absence and
failed the next probe early. Reset it from the ring snapshot callback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test the announce gate leaves the lock-ring health clock unset

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: re-read lock-ring membership after starting the health clock

A probe whose first read raced a rejoin could start the clock after the
snapshot callback reset it, leaving a stale start for the next removal.
The ring is installed before the callback runs, so a second read sees the
rejoin whenever the reset came first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: treat a health clock changed during the probe as a rejoin

A rejoin and a second removal can both land between the probe's two
membership reads. The second read then shows the filer absent again, and
the probe judged the new absence by the expired start it had read before
the rejoin's reset. The snapshot callback resets the clock before the next
ring update is installed, so a clock that no longer matches the probe's
start means a rejoin happened; report healthy and let the next probe use
the fresh clock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test ring updates that land during a lock-ring health probe

Move the absence clock into lockRingAbsence, which takes the membership
read as a function, so the test can install ring updates between the
probe's reads: a rejoin after the first read, and a rejoin plus removal
between the two reads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: give every lock-ring health reset a new clock value

A probe that saw the filer absent started the clock with a CAS from 0,
so a rejoin that reset the clock to 0 again before that CAS was
invisible, and the probe could install a start time from before the
rejoin. Each reset now stores a new non-positive value and the probe
swaps from the value it loaded before the membership read, so any
rejoin during the probe makes the CAS fail. This drops the second
membership read and never waits on the LockRing mutex.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test lock-ring rejoins before and after the probe's membership read

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: keep the lock-ring version lookup out of the unhealthy health path

LockRing.Version takes the ring mutex, which SetSnapshot holds while
the snapshot callback transfers locks, so an expired-absence probe
could wait on those transfers and time out instead of answering 503.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test an expired-absence health probe while the ring mutex is held

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-17 17:13:45 -08:00
c76fb36416 test(fuse): two mounts staging the same files without -dlm (#12704)
* test(fuse): two mounts staging the same files without -dlm

Two jobs that extract the same archive into one shared directory, each
through its own mount and without -dlm, both create many of the same
files: a file that exists is skipped, otherwise it is created, written
and closed, then chmod'ed and given its mtime. Before #12593, #12594 and
#12596 the attribute update and the content flush sent the mount's whole
cached entry. Sent after the other mount had rewritten the file, the
update put back chunks that the rewrite had dropped and the filer had
deleted, so the entry pointed at deleted needles and the file failed to
read once its volume was vacuumed. Nothing told the writers.

TestStageSameFilesWithoutDLM runs that on the fuse_dlm cluster with the
mounts started without -dlm (startTestCluster takes a dlm flag;
startDLMTestCluster is unchanged) and checks every staged file: each
chunk is still on the volume server (HEAD on the fid, without
readDeleted) and the filer serves the bytes written. What the losing
writer is told depends on the version and is only logged.

With a 4.53 weed binary the test fails: 66, 71 and 70 of the 400 files
were broken in three runs. With master it passes in about 48 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(fuse): bound the staging test's filer and volume checks

The checks used the default HTTP client and a background context, so a
server that accepts a request and stops answering held the test until the
package timeout instead of failing the check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(fuse): fail the staging test unless both mounts wrote the same files

A mount whose MkdirAll failed returned early, and a mount that started
late could skip every file, so the test passed with one writer. Start both
loops together, fail on setup errors, and require files both mounts wrote.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(fuse): bound the staging test's checks with one deadline

A per-call timeout still let 400 hung lookups and reads run for hours.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-17 17:08:39 -08:00
0d3f5a220c s3: keep serving for -preStopSeconds after SIGTERM (#12703)
* s3: keep serving for -preStopSeconds after SIGTERM

On SIGTERM the gateway closed its listeners at once, so requests a load
balancer or Kubernetes Service still routed to it failed until the endpoint
was removed. With -preStopSeconds set, the gateway makes /status, /healthz
and /readyz return 503, keeps serving for that many seconds, then runs the
existing graceful shutdown. 0 (the default) keeps the old behavior.

The flag is registered for weed s3 and as -s3.preStopSeconds for weed
filer, weed server and weed mini. A cancelled weed mini cluster context
ends the wait early. The Helm chart passes s3.preStopSeconds and
filer.s3.preStopSeconds through.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: assert the preStopSeconds chart render under set -e, drop timing from the probe test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: wait before weed mini cancels its clients; size grace periods in the chart

weed mini's interrupt hook runs before the S3 one and cancels the S3
gateway's shutdownCtx, so the pre-stop wait never happened there. The wait
is now also registered with beforeMiniClientsShutdown, which runs before
that cancel, and a sync.OnceFunc keeps the interrupt hook from waiting a
second time. Outside weed mini the registration is a no-op.

Helm: the s3 deployment fails to render when terminationGracePeriodSeconds
is below preStopSeconds + 15. The filer statefulset and the all-in-one
deployment, whose grace period is fixed at 60s, add preStopSeconds to it.
allInOne.s3.preStopSeconds is passed through, inheriting s3.preStopSeconds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: keep 64-bit counters aligned on 386; let allInOne.s3.preStopSeconds=0 disable the wait

draining sat ahead of inFlightDataSize and inFlightUploads in S3ApiServer
and shifted them off 8-byte alignment on 32-bit builds, so the atomic
upload-limit counters panicked there. It now follows them.

Helm: allInOne.s3.preStopSeconds inherits s3.preStopSeconds only when null;
default treated an explicit 0 as unset. The s3 values and the grace period
error now say a lifecycle preStop hook's time comes on top.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grace: add OnInterruptFirst for hooks that must run before any shutdown

Interrupt hooks run in reverse registration order, so a hook cannot run
ahead of a service that registers later, such as a volume server that is
still loading its volumes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: run the -preStopSeconds wait before every other interrupt hook

Under weed server the volume server registers its hook once its volumes
are loaded, often after the S3 gateway. Its hook then ran first and shut
the volume server down, so the gateway spent the whole wait serving
requests that could not read or write objects.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-17 17:06:06 -08:00
Chris LuandClaude Opus 5.5 c4e1802e3e s3: route ';' queries as authorized, and report an empty s3:prefix to policies (#12702)
* s3: escape raw semicolons before the router matches a route

registerRouter applied EscapeSemicolonsInQuery with router.Use, but mux
runs router middleware only after it has picked the route. Queries()
matchers split pairs on ';', so GET /bucket?uploads;x=1 reached the
multipart listing handler while the handler and the IAM resolver read
uploads%3Bx=1 and saw no uploads subresource. Wrap the S3 router with the
escape so routing and authorization read the same query.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: report a listing without prefix as s3:prefix "" in policy conditions

The advanced IAM path already sets s3:prefix to "" when a bucket listing
names no prefix, as AWS does, but bucket policies and attached policies
read conditions from ExtractConditionValuesFromRequest, which leaves the
key out. The AWS home-directory policy that grants s3:prefix ["", "home/"]
then denied the root listing, and a Deny keyed on the empty prefix never
applied.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: test the served handler composition for the semicolon escape

Move the escape and the Vectors split into s3Handler so a command test can
send ';' queries through the handler the server actually serves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: keep the isVectorsRequest doc comment on its function

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 17:05:34 -08:00
1d30b0de21 fix(s3api): complete bucket-listing IAM condition handling (#12699)
* fix(s3api): complete bucket-listing IAM condition handling

Populate s3:delimiter and s3:max-keys for bucket. Add regression tests for listing conditions and variants.

Assisted-by: gpt-6.1-sol, deepseek-v4.1-flash

* fix(s3api): ignore max-keys for multipart listing policies

Assisted-by: GPT-6.1 Terra

* s3api: drop max-keys from multipart-upload listing policy conditions

Bucket policies and embedded IAM policies read condition values through
ExtractConditionValuesFromRequest, which still exposed max-keys on ?uploads
listings. Those page by max-uploads, so an unused max-keys parameter could
satisfy an s3:max-keys condition. Match the advanced IAM path and leave the
key absent there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 17:05:00 -08:00
5681be241e topology: never report negative free volume slots (#12698)
* topology: never report negative free volume slots

Free volume slots are computed as max + remote - volume - ecSlots. When phantom or over-registered volumes push the count past the configured max, the result is negative and rolls up into the cluster total, causing volume grow and assign to fail with a negative 'volumes left' count even while healthy servers have capacity. Clamp every free-volume count at zero so an over-committed server contributes no slots instead of subtracting from the cluster.

* topology: drop the issue reference from the free-slot test comment

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* topology: test free slots next to an over-committed server

A server with more volumes than slots must not hide the free slots of
the other servers in its rack, data center or topology.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* topology: sum free slots per data node in racks, data centers and topology

Clamping the rolled-up counters only clamped their sum, so one server with
more volumes than slots still hid the free slots of every other server in
its rack, data center and topology. Interior nodes now add up their
children's free slots, so each data node counts as zero at worst. A data
node in maintenance mode no longer adds slots to its parents either.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* topology: test that disk info keeps an over-committed deficit

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* topology: keep the signed free volume count in disk info

EC encode retries and the balance planners add refunds and planned moves
to the published FreeVolumeCount. Clamping it at zero dropped an
over-committed disk's deficit, so a refund could make a full disk look
free. Volume growth already reads the clamped AvailableSpaceFor.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* topology: test placement next to an over-committed server

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* topology: accept either data center order in the placement spread check

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 17:02:01 -08:00
8ba33fae71 rust volume: bulk-load needle maps on redb 4.6 and compact the old floor (#12695)
* seaweed-volume: expect a fresh redb file to start at 72 KiB

The assertion fails on redb 4.2.0, whose empty database is about 1 MiB.

* seaweed-volume: bulk-load redb rebuilds with the stable cursor

redb 4.6.0 has no experimental_cursor feature. Full rebuilds insert
sorted needle ids through insert_before, and a new .rdb starts at 72 KiB.

* seaweed-volume: test which redb files are still at the old floor

The gate is missing, so the test does not compile yet.

* seaweed-volume: decide when a reused redb is still at the old floor

Compact only a file above 72 KiB and at most 1 MiB plus 8 KiB, and only
when floor_compact has not been recorded.

* seaweed-volume: test one-shot compact of an old-floor redb

Reuse, shrink, and storage-error recovery are specified. The helpers
they call are not implemented, so the tests do not compile yet.

* seaweed-volume: compact a reused redb that is still at the old floor

One compact runs when the file is above 72 KiB and at most 1 MiB plus
8 KiB. A storage error drops the handle and opens the same path again
so the volume does not keep a PreviousIo latch.

* seaweed-volume: publish a redb replay before compacting the old floor

compact() is what would make a Durability::None tail durable, and a
storage error then drops that tail on reopen. An in-band file is
committed first. The floor_compact flag is read only inside the band,
so a meta error cannot rebuild a larger index.

* seaweed-volume: drop the redb experimental cursor CI step

redb 4.6 has no redb-experimental-cursor feature. The needle-map tests
that used it now run in the normal cargo test step.

* seaweed-volume: drop the experimental-api-5 notes on redb reads

redb's experimental-api-5 can no longer be enabled, so these reads always
use the inherent ReadOnlyTable::get.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: drop the always-false flag from the old-floor band check

Both callers pass false and read floor_compact on their own, so the band
check only needs the file length.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 17:00:55 -08:00
d8a47042c5 master: unregister a volume server whose heartbeats stop (#12697)
* master: unregister a volume server whose heartbeats stop

A volume server sends a heartbeat every 5 s. When its host hangs or its
network drops packets, its SendHeartbeat stream stays open, and the
master unregisters it only once gRPC keepalive closes the connection,
60 s plus 20 s after the last frame. Until then the master keeps
assigning writes to volumes with a copy on it, and those writes hang.

SendHeartbeat now ends the stream when no heartbeat arrives for three
pulses (15 s), which unregisters the server through the existing
deferred path and tells clients its locations are gone. The server
reconnects and registers again once it can reach the master.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* master: test that clients learn a silent server's volumes are gone

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* master: fail the silent-stream receive test instead of hanging

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-17 16:59:40 -08:00
Chris LuandClaude Opus 5.5 dbb2fdb93b pb: recover when a local gRPC socket file is deleted (#12694)
* pb: fall back to TCP when the local gRPC socket cannot be dialed

A temp cleaner can delete a local gRPC socket file while its server keeps
running. Every later dial to that server then failed with "no such file or
directory", so the master could no longer reach its own volume server and
volume growth stopped. Dial the server's TCP address when the socket dial
fails, as the S3 gateway already does for the filer's HTTP socket.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* pb: recreate a local gRPC socket whose file was removed

Once its socket file was deleted, a server kept accepting on an
unreachable listener until restart, so same-host RPCs stayed on the TCP
fallback. Check the path every minute and, when the file is gone, close
the old listener and bind the socket again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* pb: retry a failed local socket rebind on the next check

A failed rebind ended the watcher, so the socket stayed missing after the
cause cleared. Bind the replacement before closing the old listener: a
failed bind keeps the old listener serving and is retried on the next
check, and the old Serve still reports server shutdown. The old listener
no longer unlinks the path on close, since the replacement now owns it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* pb: set the local socket unlink policy before serving the listener

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 16:58:19 -08:00
b74971b6d4 server: add -grpc.localSocketDir for the local gRPC sockets (#12693)
* server: add -grpc.localSocketDir for the local gRPC sockets

`weed server` and `weed mini` hard-coded each local gRPC Unix socket
under /tmp. systemd-tmpfiles ages /tmp after 10 days and macOS after 3,
so a long-running server loses its socket, and RegisterLocalGrpcSocket
keeps routing local dials to the missing path: the master can no longer
reach its volume server.

Add -grpc.localSocketDir to both commands, defaulting to /tmp so an
existing setup is unchanged, and build every socket path from it with
filepath.Join. A folder that does not exist or is not a directory is
refused at startup rather than at the first local RPC. A folder whose
socket paths exceed the platform's sockaddr_un limit is refused too,
because every bind then fails with EINVAL while the server keeps running
and each local RPC dials a socket that does not exist.

Refs #12692

* server: refuse a -grpc.localSocketDir the server cannot bind in

An existing folder the server's user cannot write to passed the startup
check, and then every socket bind failed: ListenOnLocalSocket only logs
the error while GrpcDial keeps routing local dials to a socket that was
never created. After the path-length check, create and remove a probe
Unix socket in the folder, and refuse startup with the bind error if
that fails.

* server: give the -grpc.localSocketDir probe a random name

A probe named only by the process id could be precreated by another
user in a shared /tmp, or left by a crashed process whose pid was
reused, and the server cannot remove such a file, so startup failed
although the real socket paths were free. Name the probe with 8 random
bytes, retry a name that is taken, and rely on the listener's close to
unlink it.

* server: keep the -grpc.localSocketDir probe name within the service socket lengths

The probe was 5 bytes longer than the master and volume socket names, so a
folder whose service sockets fit the sockaddr_un limit could still be refused.
A 23-byte probe name is shorter than any service socket name, and a process
with no local sockets skips the check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* server: trim the -grpc.localSocketDir comments and drop issue references

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Brent Pease <179639673+brentDSL@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 16:56:15 -08:00
Chris LuandClaude Opus 5.5 95f46c8fff admin: manage filer webhook targets from the UI with live reload (#12690)
* webhook: share the default target settings

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* webhook: managed webhook targets with a drainable queue

Targets kept as JSON (name, endpoint, token, filters, retry and worker
settings) build their own queue; Close waits for queued and in-flight
deliveries before stopping it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: run webhook targets stored in /etc/seaweedfs when notification.webhook.managed is set

The targets in /etc/seaweedfs/notification_webhooks.json start with the
filer and are rebuilt from the store whenever the file changes on any
filer, alongside notification.toml. The previous queue drains before it
stops. Events for the file itself, which holds bearer tokens, are not
sent to notification queues.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: manage filer webhook targets

A Webhooks page under Management lists, adds, edits and deletes the
targets kept in /etc/seaweedfs/notification_webhooks.json. Bearer
tokens are write-only: the API reports whether one is set, and an empty
token on edit keeps the stored one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: keep the webhook queue when the stored targets did not change

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: keep a replaced webhook queue open until its senders hand over

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: read chunk-backed webhook targets before editing them

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: keep the stored webhook targets out of fs.meta.notify

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* webhook: reject negative managed target settings

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* webhook: no signal handler on queues that reloads replace

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: swap the webhook queue without waiting behind blocked senders

The read lock held across SendMessage made a reload wait for any sender
blocked on a saturated target, and the waiting writer stalled every
other filer write with it, which is when an operator disables that
target. Close now refuses new sends and waits for the senders already
inside SendMessage, and a send that hits the closing queue retries on
its replacement.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* webhook: release blocked test handlers before the server shuts down

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: always run the webhook targets managed in the admin UI

Drop the notification.webhook.managed switch: every filer loads and live
reloads the stored targets. Without stored targets no queue is created.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* scaffold: place the admin-managed webhook note above enabled

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* scaffold: separate the admin-managed webhook note

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* scaffold: note that admin-managed webhooks follow the notification queue

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: retry loading webhook targets after a failed read

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: retire the admin-managed webhook queue on shutdown

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 16:54:54 -08:00
github-actions[bot] 386385101d docs: regenerate star history chart 2026-11-18 00:33:52 +00:00
Chris LuandClaude Opus 5.5 6342876392 ci: run the Python 3.9 jobs on 3.10 (#12691)
* ci: run the Python 3.9 jobs on 3.10

ubuntu-latest now resolves to Ubuntu 26.04, which has no Python 3.9 build
for actions/setup-python, so the Ceph S3 tests and the PyArrow Parquet
3.9 job fail during setup. 3.10 is the oldest version built for 26.04.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: list the Python versions the parquet CI runs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 22:07:34 -08:00
Chris LuandClaude Opus 5.5 40182a68bc admin: keep the dark theme from flashing light on reload (#12689)
* admin: apply the saved theme inline at the top of the page head

The theme script was an external file, so a reload had to fetch or
revalidate it before the root got its theme, and the page could paint
with the light default in the meantime.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: set color-scheme on the root together with the theme

Until the stylesheets load, the browser paints the page canvas from the
root's color-scheme, which only the stylesheets set, so a dark theme
showed a white canvas for the duration of every reload.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 20:31:00 -08:00
github-actions[bot] eed00c3737 docs: regenerate star history chart 2026-11-17 01:00:14 +00:00
Chris LuandClaude Opus 5.5 b449301a4e ec: mount shards through the encode's .vif when a source volume's .vif shares the disk (#12676)
* ec: mount shards against the index .vif when the local one has no EC config

A worker encode can send shards to a disk that still holds the source
volume's .vif while the encode's .vif lands with the index on a sibling
disk. The shard disk mounted through the source's config-free .vif, so
it reported encode time 0 and fell back to the sidecar layout, and reads
carrying the encode time were rejected as a different encode run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): mount shards against the index .vif when the local one has no EC config

Mirrors the Go mount: a data-dir .vif that records no EC config yields
to the idx dir's when that one records it, and load_vif_info_across_dirs
reports the directory the .vif actually came from.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: do not mirror EC sidecars onto a disk whose .vif has no EC config

Startup mirroring copied .ecx/.ecj next to a source volume's leftover
config-free .vif and kept that .vif, so after a restart the shard disk
mounted self-contained with encode time 0. Leave such a disk on the
cross-disk mount, which reads the owner's .vif.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): do not mirror EC sidecars onto a disk whose .vif has no EC config

Mirrors the Go startup mirror: a shard disk holding a source volume's
config-free .vif stays on the cross-disk mount, which reads the owner's
.vif, instead of mounting self-contained with encode time 0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec: take the encode .vif from a sibling data dir when the index dir is shared

With a separate -dir.idx the .ecx is shared by every disk, but the encode's
.vif lands in the data dir of the disk that received it. A disk still holding
a source volume's config-free .vif mounted through it at generation 0.
NewEcVolume now accepts the server's data dirs and takes a sibling's EC .vif
when the index is not local and every sibling EC .vif names the same run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: give EC mounts the data dir of every disk on the server

NewStore records every data dir on each DiskLocation before loading, so the
startup scan, cross-disk reconcile and MountEcShards all pass them to
NewEcVolume.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): take the encode .vif from a sibling data dir when the index dir is shared

Mirrors the Go mount: EcVolume::new_with_vif_dirs takes a sibling's EC .vif
when this disk's own records no EC config, the index is not local, and every
sibling EC .vif names the same run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): give EC mounts the data dir of every disk on the server

main sets Store.store_dirs before the first add_location, so a disk loaded
before its siblings still sees them; each DiskLocation passes them to
EcVolume::new_with_vif_dirs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: size startup EC validation by the sibling encode .vif

With a shared -dir.idx, a disk that still holds the source .dat and its
config-free .vif read the default data-shard count, judged custom-ratio
shards an interrupted encode, and removed them along with the shared .ecx
before the mount could take the sibling's .vif. The startup lookup now
falls back to the same sibling .vif the mount uses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): size startup EC validation by the sibling encode .vif

Mirrors the Go change: validate_ec_volume falls back to the sibling data
dir's encode .vif before judging shards beside a source .dat.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: keep EC shards when sibling .vif files disagree on the encode

SiblingEcConfigVif reports when a sibling .vif cannot be read or the
siblings name different encode runs. The startup data-shard lookup then
returns 0 instead of the default ratio, and validateEcVolume keeps the
files rather than sizing custom-ratio shards by a ratio they do not use.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): keep EC shards when sibling .vif files disagree on the encode

Mirrors the Go change: sibling_ec_config_vif returns Err for an unreadable
sibling .vif or disagreeing runs, ec_data_shards_from_vif returns 0 then,
and validate_ec_volume keeps the files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: release the opened EC shard when its volume cannot be built

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 16:38:17 -08:00
Chris LuandClaude Opus 5.5 9e5357782e volume: keep reads over the download limit local when readMode is local (#12679)
* volume: keep reads over the download limit local when readMode is local

With -readMode=local the server never sends a read for a volume it does
not hold elsewhere, but a read over -concurrentDownloadLimitMB on a
replicated volume was still redirected to another replica. Skip the
replica fallback in local mode, so the read waits for a download slot
and answers 429 when the wait times out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): keep reads over the download limit local when readMode is local

Mirror the Go volume server: with readMode local, a read over the
download limit waits for a slot instead of being redirected to a
replica. proxy_or_redirect_to_target no longer needs the local-mode
redirect switch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: check download capacity under the wait lock

waitForDownloadSlot read the in-flight size outside the cond lock and
the read handler broadcast without it, so a download finishing between
the check and the wait was missed and the read waited out the timeout
for a 429. Check capacity with util.WaitUntil under the lock and
broadcast under it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rs): register for the download notification before checking capacity

The wait loop created its Notified future only after the capacity
check, so a body dropped in between had its notify_waiters() lost and
the read waited out the deadline for a 429. Create the future first,
then check, as Go now checks under the cond lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 16:20:15 -08:00
f4e638fa1d Fix/admin cluster submenu (#12677)
* admin: fix cluster submenu toggle

* admin: update embedded JavaScript

* admin: drop the object store submenu handlers

The sidebar no longer renders an object store submenu, so its expand,
link and toggle handlers match nothing; the toggle handler repeated the
manual collapse that fought Bootstrap on the cluster submenu.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: ivan <i.stepanov@navigine.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 16:13:18 -08:00
Chris LuandClaude Opus 5.5 1e31c6dbf5 test(pjdfstest): check the sudo'd mount with ps, not kill -0 (#12673)
sudo switches its real uid to root while it starts, so kill -0 from the
invoking user fails with EPERM even though the mount process is alive,
and the wait loop reported "weed mount exited early" a few milliseconds
after starting it.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 16:12:30 -08:00
0f0f3811bf filer: serve caught-up aggregated subscribers from memory, not a held disk pass (#12674)
* filer: serve caught-up aggregated subscribers from memory, not a held disk pass

An aggregated SubscribeMetadata stream that does not take metadata chunks
(filer.sync, filer.backup, filer.meta.backup, filer.meta.tail, the
filer.remote.* syncs, the S3 gateway's subscriptions, and mounts that
predate chunk support) re-ran the persisted-log pass whenever its cursor
moved: after every memory hold at the delivery watermark and on every
2 s re-probe. The pass holds at the peers' flush low-watermark. Peers
flush on their own one-minute schedules, so on a cluster that keeps
writing some peer has always flushed past a caught-up cursor while the
slowest has not: the pass held at its first entry, the loop came back
to it, and memory was never read. The stream got each change only once
the slowest peer's flush covered it, in bursts up to a flush interval
late, and every pass listed and re-read the newest log files.

When the aggregated ring was already filling at the cursor and evicted
nothing after it, memory holds everything the persisted logs could add,
since every peer's log reaches the ring through the aggregator. Skip the
pass and read memory, held at the delivery watermark as before. The
skipped pass proves nothing past the cursor, and the next-day scan for
an empty pass does not apply. An empty ring, a ring that starts past the
cursor, and a cursor memory just refused still take the pass. Chunk
subscribers already skip it for a caught-up cursor: their ref listing
stops below the flush hold.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: keep a new peer's pre-start history on the disk pass

The ring only receives a peer's events from the start of its subscription.
A peer followed for the first time after this filer started (a filer added
to the cluster, a restarted pod with a new address, or the late-arriving
peers bootstrap) starts at its notice time, so what it logged before then
is only in the persisted logs. A cursor below that start passed the ring
check and skipped the disk pass, losing those events for subscribers that
do not take metadata chunks.

Record the newest start of a peer subscription that had no resume point and
send cursors below it through the disk pass as before. Peers that rejoin with
a resume point (a restart under the same address) do not move it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-16 16:12:06 -08:00
dependabot[bot] 53b234558e build(deps): bump actions/setup-go from 6 to 7 (#12685)
Bumps [actions/setup-go](https://github.com/actions/setup-go) from 6 to 7.
- [Release notes](https://github.com/actions/setup-go/releases)
- [Commits](https://github.com/actions/setup-go/compare/v6...v7)

---
updated-dependencies:
- dependency-name: actions/setup-go
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-16 15:39:24 -08:00
dependabot[bot] d9c2d99f8f build(deps): bump gocloud.dev/pubsub/natspubsub from 0.46.0 to 0.49.0 (#12680)
Bumps gocloud.dev/pubsub/natspubsub from 0.46.0 to 0.49.0.

---
updated-dependencies:
- dependency-name: gocloud.dev/pubsub/natspubsub
  dependency-version: 0.49.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-16 15:38:44 -08:00
dependabot[bot] e3769592a5 build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob from 1.8.2 to 1.8.3 (#12684)
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob

Bumps [github.com/Azure/azure-sdk-for-go/sdk/storage/azblob](https://github.com/Azure/azure-sdk-for-go) from 1.8.2 to 1.8.3.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azidentity/v1.8.2...sdk/storage/azblob/v1.8.3)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
  dependency-version: 1.8.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-16 14:58:43 -08:00
dependabot[bot] c3f4b3e0eb build(deps): bump go.etcd.io/etcd/client/v3 from 3.7.1 to 3.7.3 (#12682)
Bumps [go.etcd.io/etcd/client/v3](https://github.com/etcd-io/etcd) from 3.7.1 to 3.7.3.
- [Release notes](https://github.com/etcd-io/etcd/releases)
- [Commits](https://github.com/etcd-io/etcd/compare/v3.7.1...v3.7.3)

---
updated-dependencies:
- dependency-name: go.etcd.io/etcd/client/v3
  dependency-version: 3.7.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-16 14:57:32 -08:00
dependabot[bot] 438cc8562f build(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.114.1 to 1.114.10 (#12681)
build(deps): bump github.com/aws/aws-sdk-go-v2/service/s3

Bumps [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) from 1.114.1 to 1.114.10.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.114.1...service/s3/v1.114.10)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/s3
  dependency-version: 1.114.10
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-16 14:56:55 -08:00
dependabot[bot] 51ab451f37 build(deps): bump github.com/rabbitmq/amqp091-go from 1.15.0 to 1.16.0 (#12683)
Bumps [github.com/rabbitmq/amqp091-go](https://github.com/rabbitmq/amqp091-go) from 1.15.0 to 1.16.0.
- [Release notes](https://github.com/rabbitmq/amqp091-go/releases)
- [Changelog](https://github.com/rabbitmq/amqp091-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/rabbitmq/amqp091-go/compare/v1.15.0...v1.16.0)

---
updated-dependencies:
- dependency-name: github.com/rabbitmq/amqp091-go
  dependency-version: 1.16.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-16 14:56:35 -08:00
dependabot[bot] 34927705fa build(deps): bump actions/setup-python from 6 to 7 (#12686)
Bumps [actions/setup-python](https://github.com/actions/setup-python) from 6 to 7.
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](https://github.com/actions/setup-python/compare/v6...v7)

---
updated-dependencies:
- dependency-name: actions/setup-python
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-16 14:56:11 -08:00
Chris LuandClaude Opus 5.5 d5b4e1352e helm: drop the 4.53 vector metrics render guard (#12672)
* helm: pin the 4.53 tag in the vector metrics rejection test

The 4.54 release made 4.54 the chart's default image, and that image
serves weed-vector metrics, so rendering vector.metricsPort with the
default tag no longer fails. Pin image.tag=4.53 for the rejection case
and check that the default image renders the metrics port.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* helm: describe vector metrics support from 4.54 on in the chart README

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* helm: say the vector metrics render guard skips image overrides

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* helm: drop the 4.53 vector metrics render guard

The chart now defaults to 4.54, whose weed-vector serves --metrics-port, so
the check keyed on one old tag only blocked valid installs. The test asserts
the default image renders the metrics flag and HTTP probes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* helm: describe vector metrics without the render guard in the chart README

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 22:49:39 -08:00
github-actions[bot] 5ef99c1d8d 4.54 2026-11-16 05:31:42 +00:00
Chris LuandClaude Opus 5.5 8e15110b4f filer: wake metadata subscribers only for directories they can match (#12669)
* log_buffer: let a subscriber skip wake-ups for unwanted partition keys

Every append woke every registered subscriber. SetSubscriberWakeFilter lets
a reader sleep through appends whose partition key it would filter out; it
still reads them on its next poll. Flushes and keyless appends wake everyone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer_pb: decide subscription matches from an event's directories

MetadataDirectoryMayMatchSubscription rules out events whose directory no
child path of can match the subscription, and ScanMetadataEventDirectories
reads an event's directory and new parent path without allocating.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: wake subscribers of a move's destination directory

Appends are keyed by the source directory, so a filtered subscriber of the
destination would otherwise only see the move on its next poll.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: wake metadata subscribers only for directories they can match

Every metadata event woke and was scanned by every subscriber, so filer CPU
grew with the number of subscribers watching other paths. Readers now
register a directory wake filter, and the per-entry prefilter rules entries
out from their directories before the skeleton scan.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test that moves into a subscribed prefix pass the prefilter

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 20:53:41 -08:00
Chris LuandClaude Opus 5.5 f13b763552 docs: explain the 3-filer recommendation in the production setup notes (#12671)
* docs: explain the 3-filer recommendation in the production setup notes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: note that a filer load balancer must forward the gRPC port

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 20:52:03 -08:00
Chris LuandClaude Opus 5.5 df30c883fe s3api: keep a multipart upload's state on the object's owner filer (#12668)
* s3api: create a multipart upload's directory on the object's owner filer

CreateMultipartUpload wrote .uploads/<id> to the gateway's own filer while
the upload's completion commits on the object's write owner. With
independent filer stores, a part or completion reaching another gateway
before peer metadata replication caught up could not see the upload.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: look up a multipart upload on the object's owner filer

UploadPart, UploadPartCopy, ListParts, CompleteMultipartUpload,
AbortMultipartUpload and the bucket policy SSE lookup read .uploads/<id>
from the gateway's own filer, so a request on another gateway shortly
after CreateMultipartUpload answered NoSuchUpload until replication
caught up. Read it preferring the object's write owner, like object reads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: write multipart parts to the upload's owner filer

UploadPart and UploadPartCopy stored part entries on the gateway's own
filer, so ListParts and CompleteMultipartUpload on another gateway could
miss a part that replication had not delivered yet. Store them next to the
upload directory on the object's write owner.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: list and clean up a multipart upload on its owner filer

ListParts, CompleteMultipartUpload and AbortMultipartUpload listed and
removed part entries on the gateway's own filer. Do it where the upload
and its parts now live, so completion sees every part and cleanup does
not miss entries replication has not delivered yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: test a multipart upload against a lagging gateway filer

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: check the upload's owner before freeing a failed part's chunks

A part write that the owner stored but answered with an error went to
recovery with no object, so the owner was never among the filers checked.
When the owner is outside the gateway's filer list, the others' not-found
answers freed chunks the stored part still references.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: list an upload's parts on the prior owner after a ring change

The upload record lookup already falls back to the key's prior owner while
a ring change is in its cooling-off window, but the part listing read only
the new owner. ListParts then reported no parts for an open upload. Fall
back the same way when the new owner has nothing yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: find the upload record on the prior owner past a bare directory

A part routed to a new owner that has not received the upload yet makes
that filer create .uploads/<id> without the upload's key. The post-write
check then read it as an aborted upload and removed the part. While the
ring change is in its cooling-off window, a record on the prior owner
now wins over such a directory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: merge the prior owner's parts into an upload listing

During a ring change's cooling-off window the new owner may have received
only some of an upload's parts. Listing returned the new owner's parts
alone once it had any, so ListParts dropped the rest and a completion
naming them failed with InvalidPart. Both owners' pages are now merged,
the owner's entry winning a shared name, and cut to the page limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: fail an upload listing the prior owner could not answer

When the new owner has none of an upload's parts yet and the prior
owner's listing fails, the parts may all be on the prior owner. That
listing read as empty, so CompleteMultipartUpload answered NoSuchUpload
for an upload that still exists; it now returns the error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: fail an upload lookup the prior owner could not answer

During a ring change's cooling-off window an upload the new owner has
not received yet, or holds only as a bare directory, read as NoSuchUpload
when the prior owner's lookup failed. That failure is now returned, so
the client sees a retryable error instead of an upload that is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 19:03:41 -08:00
github-actions[bot] 5a1b0abc20 docs: regenerate star history chart 2026-11-16 00:37:53 +00:00
b2dd93eac8 admin: support [maintenance.s3_lifecycle] in admin.toml (#12661)
* admin: support [maintenance.s3_lifecycle] in admin.toml

The admin.toml overlay covers vacuum, volume_balance and erasure_coding but
not s3_lifecycle, so the bucket lifecycle job's settings can only be changed
from the web UI and are lost whenever the admin data directory does not
persist. That is the worst combination for this job type: its
meta_log_retention_days knob decides whether bucket rules ever apply to
objects written before the job was enabled, and silently reverting it to the
default makes the rules no-op without any error.

Wire the four admin-side knobs into the existing overlay:

  meta_log_retention_days    retention window the meta-log replay can reach
  walker_interval_minutes    minimum gap between full bucket walks
  cluster_deletes_per_second cluster-wide delete budget
  cluster_deletes_burst      token-bucket burst for that budget

All four are admin config values, not worker values: the engine partition is
computed on the admin side and the delete budget is divided across
execute-capable workers at dispatch time. enabled/retry_limit/
retry_backoff_seconds come for free from the shared section handling.

Tested with TestApplyPluginConfigFromTomlS3Lifecycle, which also pins the
retention=0 case: 0 means unbounded and must survive as an explicit 0 rather
than being treated as unset.

* admin: fix meta_log_retention_days guidance in admin.toml scaffold

Both reviewers caught a real error. `0` is not "unbounded": runShard does

    retentionWindow := cfg.RetentionWindow
    if retentionWindow <= 0 {
        retentionWindow = maxTTL
    }

so zero falls back to the longest rule TTL, which keeps every replay rule in
replay and promotes nothing to the walk set. Recommending 0 to force full
scans was wrong, and so was the claim that pre-existing objects are only
reachable through the walk set -- the cold-start recovery walk covers them
regardless of the retention value.

Describe zero as "no retention info supplied" and tell operators to set the
window the filer meta-log actually retains. The code change is unaffected:
it only wires the four keys into the existing overlay.

* admin: tie s3_lifecycle walk-set scans to walker_interval_minutes in admin.toml

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: show the s3_lifecycle retry defaults in admin.toml

The s3_lifecycle descriptor sets no retry limit or backoff, so a failed
run is not retried and the scheduler's 5 s backoff applies; the scaffold
listed the vacuum values instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: shorten the s3_lifecycle toml section comment

The retention and walker knobs are read by the worker from the admin
values, not computed on the admin side.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: cover cluster_deletes_burst in the s3_lifecycle toml test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: check the s3_lifecycle retention zero is a stored int64

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 13:46:14 -08:00
Chris LuandClaude Opus 5.5 5f7fefadae s3 lifecycle: write the suspended expiration marker before deleting the null version (#12664)
* s3 test: evaluate mutation conditions and serve DeleteEntry in the fake filer

The fake ObjectTransaction now checks each mutation's own condition before
applying it, as the filer does, and reports the failed mutation. ETag clauses
read the stored ETag attribute first, like the filer. DeleteEntry removes the
entry so unrouted delete paths can run against the fake.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: write the suspended expiration marker before deleting the null version

A suspended-versioning expiration deleted the null version first and then
wrote the delete marker. When the pointer read or the marker write failed in
between, the retry found no null version, resolved as NOT_FOUND or
VERSION_NOT_LATEST, and never wrote the marker, so an older version became
the latest.

The marker now goes first. The latest pointer outranks the null object, so
the key reads as deleted once the marker lands, and a null delete that fails
after it leaves a noncurrent null version rather than exposing an older one.

The null delete is pinned to the entry read before the marker: on the owner
it carries IF_ENTRY_EQUAL on that read, and off the ring the entry is re-read
and compared by identity first. A null object written after the read is a
newer write and is no longer deleted. When the null version is the one that
passed the expiration checks, a missing or changed entry at that read
resolves the request as STALE_IDENTITY before anything is written.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: retry a failed routed null delete instead of deleting on the default filer

A routed null delete that failed fell through to the unrouted path, which
reads and deletes on the default filer. That filer may not have the owner's
newer null object yet, so the fallback could delete it without the owner-side
IF_ENTRY_EQUAL guard. The failure now returns RETRY_LATER; the marker is
already written, so the key keeps reading as deleted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: recheck and delete an unrouted null version under the object write lock

Without a ring owner the identity recheck and the null delete were separate
calls, so a suspended PUT landing between them was deleted. Off the ring
writers take the object write lock, so the recheck and the delete now run
under it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 test: strip quotes from the stored ETag in the fake filer

The filer normalizes both sides of an ETag comparison; the fake now strips
quotes from the stored ETag attribute as well as from the candidate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: retry the null delete when the object lock lookup fails on a routed key

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: check the suspended marker against the validated version

A suspended expiration that validated a named version wrote its marker
without comparing the pointer, so a null PUT landing after the checks was
hidden, and deleted too when it landed before the null read. The marker now
compares the latest the way the enabled path does and resolves the request
as VERSION_NOT_LATEST.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 test: count the marker's object write lock in the unrouted null delete

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: recheck the attempt deadline after reading the null version

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 13:30:00 -08:00
Chris LuandClaude Opus 5.5 c09009a74f s3: accept the governance bypass header case-insensitively (#12667)
* s3: accept the governance bypass header case-insensitively

botocore and the AWS CLI send x-amz-bypass-governance-retention: True,
so the exact "true" match ignored the bypass and governance-locked
deletes and retention changes failed with AccessDenied.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: test governance bypass with SDK-style header values

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 13:18:19 -08:00
Chris LuandClaude Opus 5.5 c0509cf499 s3api: evaluate each policy condition against its own values (#12665)
* s3api: normalize a nil policy value to no values

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: evaluate each policy condition against its own values

The process-wide normalized value cache keyed entries by fmt.Sprint of the
value, so ["a", "b"] and ["a b"] (or [] and [""]) shared one entry and a
condition in one policy could be compared against another policy's values.
Normalizing is cheaper than building that key, so drop the cache.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: test that look-alike condition values stay distinct

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 13:16:27 -08:00
Chris LuandClaude Opus 5.5 63806b7073 s3 lifecycle: bound each LifecycleDelete attempt with its own deadline (#12646)
* s3 lifecycle: bound each LifecycleDelete attempt with its own deadline

Replay and walker deletes used the job context with no request
deadline, so one stalled unary RPC could hold a shard for the whole
job. Each attempt now runs under DeleteTimeout (default 1m), set after
the limiter wait. A timeout while the job is still live is retried like
a transport error and then halts the shard or walk without advancing;
parent cancellation still stops both the attempt and the backoff.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: admin setting for the delete RPC timeout

Adds delete_timeout_seconds to the S3 Lifecycle admin form and passes
it to both the replay drain and the walker. 0 keeps the 60s default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: describe replay and walker timeouts separately in the help text

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: fail the pass when a replay delete exhausts its attempts

A transport error or attempt timeout that outlived the retries halted
the shard with a nil error, so the job reported success. processMatches
now returns the error (the cursor still holds at the event), and
runShard treats a drain error as a clean stop only when its own context
ended, since an attempt timeout also wraps DeadlineExceeded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: test that a replayed LifecycleDelete resolves after a lost reply

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: skip the expiration write once the attempt deadline passed

A worker retries a LifecycleDelete whose attempt timed out, while the
first attempt can still be running on the S3 server. An attempt that
stalled before its marker write would then stack a second delete marker
after the retry wrote one. The gRPC deadline reaches the handler context,
so check it right before the expiration writes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: recheck the attempt deadline after the suspended null delete

The null-version delete can block on the owner filer past the attempt
deadline; the worker may have retried and written a marker by then.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: finish the suspended marker once the null version is deleted

Skipping the marker after a successful null delete left an older version
as the latest, and the retry then resolved as a no-op without hiding it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: let a routed delete marker commit only while its latest guard holds

createDeleteMarkerIf passes an optional condition, evaluated on the
.versions directory entry under the object's lock on the owner filer,
and reports a refused commit as errLatestVersionChanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: write the expiration marker as a compare-and-set on the latest pointer

A timed-out attempt can still be running on the S3 server while the
worker's retry writes the marker. The marker commit is now conditioned
on the .versions pointer as read just before it and skipped when the
latest is already a delete marker, so whichever attempt commits second
writes nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: resolve the marker as stale when another version became latest

The marker's compare-and-set only covered changes after its own pointer
read, so a version promoted after the expiration checks passed could be
hidden by the marker. The enabled path now passes the validated version
and a conflict retry that finds a different named latest resolves the
request as VERSION_NOT_LATEST.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: resolve a null expiration as stale once a version takes the pointer

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: read the marker's latest version the way getLatestObjectVersion does

A pointer names a version only with its file name; an id alone falls back
to the null object, and the null-is-latest signal makes the null version
current. An expiration that validated a named version no longer hides a
null written by a suspended PUT, and a current null under an id-only pointer
now expires instead of resolving as stale.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: hold the object write lock for an unrouted expiration marker

Off the ring createDeleteMarkerIf cannot carry the pointer guard, so a PUT
committing between the pointer read and the marker write was hidden. The
read and the write now run under the object write lock that unrouted
writers take.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: share transient-error retries between walker and replay deletes (#12649)

* s3 lifecycle: retry only transient LifecycleDelete failures

dispatchWithRetry retried every RPC error except parent cancellation, so
an authentication, permission or argument error spent the whole retry
budget before surfacing. Retry only attempt timeouts and the failures
util.IsTransientError recognizes (Unavailable, ResourceExhausted, network
resets); everything else returns on the first attempt. A nil response
is now an error instead of a nil dereference.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: take a delete-rate token for every LifecycleDelete attempt

processMatches waited on the shared limiter once per match, so the
retries behind it ran outside the cluster-wide deletes/sec budget. The
retry loop now waits on the limiter before each attempt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: give walker deletes the replay retry policy

WalkerDispatcher sent each LifecycleDelete once, so one dropped
connection aborted the bucket's daily walk while the same failure on the
replay path was retried. The walker now goes through deleteWithRetry and
shares its transient-error retries, per-attempt limiter wait and nil
response handling.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: retry a DeadlineExceeded the server sends back

A server or proxy can end the RPC with DeadlineExceeded while the
attempt's own deadline is still running. util.IsTransientError rejects
that status, so the delete halted with retries left. It is a timed-out
attempt like the client-side one, so retry it the same way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: say walker deletes get 3 attempts in the timeout help

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 13:09:22 -08:00
Chris LuandClaude Opus 5.5 504de9b562 security: refuse to start when per-target gRPC server allowlists are not applied (#12662)
* security: refuse to start when per-target server allowlists are not applied

A component that dials every server with one option ignored
<target>_allowed_server_* keys, so a config that only set those
accepted any CA-signed server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* scaffold: say which components scope server allowlists per target

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* security: also catch per-target server allowlists set through WEED_ env vars

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 12:41:19 -08:00
Chris LuandClaude Opus 5.5 d80e905fc7 security: optional dialed-host check for gRPC server certificates (#12663)
* security: optionally check the dialed host against gRPC server certificates

verify_server_host under [grpc] or a [grpc.<component>] section switches
the client to certificate-and-host verification.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* scaffold: document verify_server_host

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 12:38:37 -08:00
Chris LuandClaude Opus 5.5 799fd10342 s3 lifecycle: fail over between S3 fronts during a pass (#12647)
* s3 lifecycle: add a LifecycleDelete client that fails over between S3 fronts

When the current front returns Unavailable, the client closes it, marks it
failed for the rest of the pass, picks the next front that passes
checkLifecycleReady and resends the same request there. Callers cut off by
that switch closing their connection resend on the new front. When no front
is left, every later call returns one error naming the fronts tried.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: serve the pass through the failover client

Execute hands the replay drain and the walker one failover client built on
the selected front, reports each switch as job progress, and closes the
current connection when the pass ends.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: keep failing over until a front serves the resent delete

The resend after a switch went straight to the new front, so a second
lost front ended the call even with a healthy third one, and the walker
has no retry of its own. Resend through the failover loop instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: do not recheck fronts skipped when the pass started

The failover client started with no failed fronts, so losing the selected
front checked the fronts selectS3Front had already skipped again, holding
every delete for another readiness timeout per front.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 12:37:07 -08:00
Chris LuandClaude Opus 5.5 a91b333658 s3 lifecycle: persist daily walk checkpoints and resume them safely (#12650)
* s3 lifecycle: walk checkpoints take each entry's resume position

Versions of one object share a Path, so a checkpoint set from Path after
the first version made a resume skip the rest. The ListFunc now gives
each entry the start value that continues after it, and leaves it empty
on all but an object's last version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: resume a bucket listing where the walk stopped

The filer listing skipped entries with key <= start, but its order is
not key order: each directory emits its .versions groups before its
files, and a directory's children come before a sibling like dir.txt.
Positions are now one (pass, name) frame per directory level, so a
resume seeds each listing and skips exactly what was finished. A version
group is one unit, and a skipped group still keeps its bare null
version out of the second pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: content hash of a walk view

Hashes every non-disabled action of a view and the versioning of their
buckets, so a saved walk position can be tied to the walk it came from:
another rule set, another walk mode, or a bucket that turned versioned
gives another hash.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: persist daily walk progress and resume it

A walk that failed or was cut short started over on the next pass,
listing again every bucket it had already finished. WalkBuckets now
keeps the finished buckets and each unfinished bucket's position in a
checkpoint file next to the shard cursors, one per shard set, and the
next walk of the same view and shards resumes from it. The record is
saved every 30s and when the walk stops, and cleared when a walk
completes, so the next walk is full again and sees keys added behind
the position; a clear that fails fails the walk rather than leave a
record that would skip buckets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: a stuck resumed walk starts over

A bucket that fails at the same place on every pass kept the walk from
completing, so the buckets it had finished were never walked again and
their due objects waited forever. A resumed walk that fails again
without finishing anything clears its checkpoint, and the next walk is
full. A canceled walk keeps it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: do not resume a walk checkpoint left for a week

A checkpoint is cleared only by a later walk of the same shard set, so a
set that stopped walking, such as a one-off per-shard walk, left its
record behind, and a walk of that set much later would skip buckets
finished long before. Records now carry their save time and are not
resumed after seven days; a walk that keeps retrying saves on every
attempt and stays fresh.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: a failed walk without progress leaves its checkpoint alone

A pass canceled while it read the checkpoint failed the read, then
saved its own empty progress over the record it never loaded, so the
next pass started every bucket again. A failed walk that finished
nothing has nothing new to record and no longer writes; the stuck
resume reset still clears.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: space out walk checkpoint saves after a failed one

The save time advanced only on success, so once the filer refused a
periodic save every following entry tried again and logged another
warning. The interval now counts from the last attempt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: a walk checkpoint save that errors may still be on disk

A save whose reply was lost left onDisk false, so a walk that then
completed skipped the clear and the next walk resumed from the stale
record instead of walking everything.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: keep a stalled walk checkpoint for a day

A resumed walk that failed again without progress dropped its record
right away, so an outage spanning two attempts restarted the whole
walk. Treat the walk as stuck only once its record has not advanced
for a day.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: a resumed walk counts its earlier buckets as walked

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: replay a resumed recovery walk from the walk that began it

A recovery or cold-start pass rewinds replay to runNow - maxTTL. When the
walk resumed a checkpoint, its finished buckets were listed in an earlier
attempt, so an object there that was not yet due then and whose write is
now older than maxTTL was neither expired by the walk nor replayed.

The checkpoint now keeps the start of the walk that began it, WalkBuckets
returns that time (its own start for a fresh walk), and runShard floors
the recovery rewind and the cold-start replay at it. A record without a
start is not resumed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: a cold start that resumed a walk replays on the next pass

The pass's shared subscription starts at runNow - maxTTL for a shard
without a cursor, before its walk reports an earlier start, so flooring
the cold-start drain at the walk start could not deliver the older
events. Such a shard now saves its cursor at the walk start minus maxTTL
and returns, like the recovery branch, and the next pass replays the
window from that cursor.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: test that a cursor below the floor opens the next subscription

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: test the cold-start replay across two Run passes

A cold start whose walk resumed an earlier attempt sees an older event
only on the next pass. The test runs Run twice against a filer that
serves events past the requested floor, with a later in-window event
that would otherwise move the cursor past the older one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 12:35:49 -08:00
4f70ad015e mount: fail the write when the swap file chunk cannot be created (#12655)
* mount: fail the write when the swap file chunk cannot be created

#7957 added a nil check after NewSwapFileChunk so a failed swap file
creation returns an error instead of panicking. The result was assigned
to pageChunk, a PageChunk interface, before the check, and an interface
holding a nil *SwapFileChunk is not nil, so the check never fired:
SaveDataAt stored the nil chunk and panicked in WriteDataAt.

Check the *SwapFileChunk before converting it. The new test points the
swap directory below a regular file; on master it panics in
SwapFileChunk.WriteDataAt.

Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>

* mount: grow the file only after its write is buffered

Write set the new size, times, pending append offset and uncommitted
bytes before AddPage. When AddPage failed (now that a swap file chunk
that cannot be created returns an error), the write was reported as
failed but the larger size stayed on the handle's entry, so the next
flush published it with a hole where the data should be.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: report a partly buffered write as a short write

PageWriter.AddPage fills one chunk at a time, so when a later chunk
cannot be created the earlier chunks already hold their bytes. Return
how much was buffered, and let Write grow the size and times over that
prefix and report it, instead of failing the whole write while the
prefix still reaches the next flush.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: report the bytes a failed copy_file_range already buffered

A streaming copy whose page fails partway returned an error even though
earlier pages, or the start of the failing one, were already buffered and
would reach the next flush with the destination size left short. Count
the buffered bytes, grow the destination over them, and return them as a
short copy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: free the copy test's destination pages

The buffered memory chunk counts toward the process-wide memory chunk
ceiling, so repeated runs of the test pushed later ones onto the
unusable swap directory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: count only the bytes a swap chunk stored

A swap chunk whose file write fails stores a prefix or nothing, but
SaveDataAt returned no error and AddPage counted the whole span as
buffered. SaveDataAt now reports the short write, and AddPage passes the
stored count up so a write is never acknowledged past it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: fail a partly buffered writeback write

Kernel writeback ignores the count in a WRITE reply, so a short write
there drops the unbuffered rest without an error. A write flagged as
page-cache writeback now keeps its buffered prefix but returns the
error, which the kernel reports on fsync.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 19:34:43 -08:00
github-actions[bot] ddd3b3666f docs: regenerate star history chart 2026-11-15 00:34:44 +00:00
Chris LuandClaude Opus 5.5 cdb01f8150 install.sh: add --enterprise (#12660)
* install.sh: offer the vector engine on linux only

Releases build weed-vector for linux amd64/arm64 only, so --component vector
on darwin or windows downloaded a missing asset.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* install.sh: add --enterprise

Installs from seaweedfs/artifactory: weed-enterprise-<os>_<arch>[_large_disk],
weed-volume-enterprise[_large_disk], weed-worker-enterprise and
weed-vector-enterprise. The latest tag skips vfs-* releases, and the Rust
volume server is linux-only there, so --component all skips it elsewhere.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* Revert "install.sh: offer the vector engine on linux only"

Releases after 4.53 publish weed-vector for darwin and windows too
(rust-vector-platforms.yml).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* install.sh: Enterprise vector engine is linux-only

Enterprise releases publish weed-vector-enterprise for linux amd64/arm64;
open-source releases keep their darwin and windows engines.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test: installer --enterprise asset names, platforms and latest tag

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* install.sh: Enterprise latest is the newest plain version tag

artifactory also releases previews and a dev build, and a prerelease such
as kvcache-filer-native-v0.1.0-preview.1 sits in the release list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* README: show passing --enterprise through bash -s --

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* install.sh: the open-source weed for windows is a .zip too

Releases publish windows_amd64[_large_disk].zip, so asking for a .tar.gz
404s on every Windows host.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* install.sh: install the windows weed as weed.exe

weed-volume and weed-vector keep their .exe on Windows; weed lost it, so
cmd and PowerShell could not run the installed binary.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* install.sh: suggest weed-volume only when it was installed

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 13:50:18 -08:00
b067266e6b shell: fix s3.circuitBreaker help examples and error messages (#12657)
* shell: fix s3.circuitBreaker help examples and error messages

The help examples used -type count, but -type accepts Count and MB (as
its flag description says), so both examples failed with "type not
valid, only 'count' and 'bytes' are allowed", an error that names
neither accepted value. Two other errors pointed at flags that do not
exist: -max and -counts, where the flag is -values.

Use -type Count in the examples, name Count and MB in the type error,
and -values in the value errors. TestCircuitBreakerHelpExamplesRun runs
every help example and fails on master for the two -type count lines.

Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>

* shell: name the bad action and the -actions flag in s3.circuitBreaker

The unknown-action error pointed at a nonexistent -action flag and echoed
the whole -actions list instead of the action that was rejected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 11:17:52 -08:00
b3a58506aa filer: report a SQL listing that fails partway through (#12654)
* filer: report a SQL listing that fails partway through

ListDirectoryPrefixedEntries never checked rows.Err() after its rows.Next()
loop. When the result set failed partway, for example on a lost
connection, rows.Next() returned false and the listing returned the
entries read so far with a nil error, so callers saw a shorter directory
instead of an error. Check rows.Err() once the loop has run to the end,
as the scan in abstract_sql_store_scan.go already does. A listing the
callback stopped on purpose is left alone: rows.Err() would return a
canceled request context's error there.

This covers every store built on AbstractSqlStore (mysql, mysql2,
postgres, postgres2, sqlite).

Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>

* filer: wait for the canceled SQL listing to release its rows in the test

A fixed sleep let the callback return before database/sql closed the
rows, so a listing that always consulted rows.Err could still pass.
Wait, bounded, until the connection is back in the pool instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 11:14:54 -08:00
anantandanant b42993d24a shell: make TestCircuitBreakerShell compare against the expected configs (#12656)
The test unmarshalled the command's output into both actual and expect,
so it compared the output with itself and the expected JSON in each case
was never checked; it also skipped the first case. Compare every case's
output with its expected config as S3CircuitBreakerConfig messages (the
output includes default fields such as "enabled": false that the
expected JSON leaves out), and restore LoadConfig when the test ends.

Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: anant <170962632+antcybersec@users.noreply.github.com>
2026-11-14 11:14:14 -08:00
Chris LuandClaude Opus 5.5 fd4603d657 s3 lifecycle: count filer entries a walk reads but skips as progress (#12658)
* dailyrun: count entries read from the filer, skipped ones included

A versioned bucket's pre-pass and a resumed walk read entries without
emitting them, so a walk can read for hours with no visited entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: count listed entries as progress work

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 11:12:33 -08:00
Chris LuandClaude Opus 5.5 53377bcb5d mount: keep a handle dirty when a write lands during its flush (#12652)
A flush uploads the dirty pages without the handle lock, then commits the
entry under it. A write in between grows the entry's size, but its pages
are not in the committed chunks. Clearing dirtyMetadata then made the next
flush skip the commit, leaving a file with the new size and no chunk for
the write, which reads back as zeros.

Count page writes on the handle and clear dirtyMetadata only when none
landed since the flush started uploading.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 11:11:01 -08:00
1167ca8cf5 vacuum: report a request dropped because another vacuum is running (#12648)
* vacuum: report a request dropped because another vacuum is running

The master runs one vacuum at a time. A request that arrives while
another one is running, the automatic vacuum or an earlier request, is
dropped: Topology.Vacuum logs "Vacuum is already running" and returns.
The VacuumVolume RPC and /vol/vacuum then answered as if the vacuum had
run, so volume.vacuum printed nothing and exited 0 with no volume
compacted. Where the automatic vacuum is busy for hours, an explicit
volume.vacuum -volumeId could do nothing every time, and nothing said so.

Topology.Vacuum now returns ErrVacuumRunning for a dropped request. The
RPC returns it and /vol/vacuum answers 409 with it. volume.vacuum names
the request that failed, because a list of volumes or collections is
sent as several requests and a vacuum that starts in between fails only
the later ones. Requests are still dropped, not queued, and the
automatic vacuum ignores the error as before.

The integration tests that need their /vol/vacuum request to run ask
again while the master answers 409.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* docs: note the 409 answer from /vol/vacuum while another vacuum runs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-14 11:07:44 -08:00
Chris LuandClaude Opus 5.5 512f898600 s3 lifecycle: report work-based job progress during a pass (#12644)
* s3 lifecycle: report work-based job progress during a pass

The daily pass sent progress only while starting, so a healthy scan that
ran for many hours could reach the admin's 24-hour no-progress guard.
Worker heartbeats prove the worker is alive, not that this job moves.

While dailyrun.Run is in progress the handler now sends a progress update
every 5 minutes, and only when the pass's counters (entries visited,
replay events, RPC outcomes including errors) moved since the last
delivered update. An idle tick sends nothing, so a stalled pass still
expires. A failed send is retried on the next tick. The reporter is
stopped and joined before the completion is sent, so no progress can
follow the terminal result.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: test that progress, not heartbeats, keeps an old job active

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: label progress phase and bucket as the last started

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 10:55:08 -08:00
Chris LuandClaude Opus 5.5 858f8cbad8 s3 lifecycle: check internal S3 authentication before starting a pass (#12645)
* s3 lifecycle: add a LifecycleReady RPC that only authorizes the caller

LifecycleReady runs the same checkAdminAuth as LifecycleDelete and reads
no bucket or object metadata, so a lifecycle worker can confirm that an S3
front accepts its internal credentials before it starts a pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: check S3 fronts with LifecycleReady before the pass

Execute used to dial the first registered S3 front and start loading
policies and walking buckets, so a missing or mismatched
jwt.filer_signing.key surfaced only as repeated per-bucket delete
failures. It now checks each front in order with LifecycleReady, using
the credentials LifecycleDelete sends, under a 10s bound that also
follows parent cancellation. The first front that passes serves the
pass; fronts that fail are reported in the job's progress and the log.
When none passes, the job fails before any listing with an error that
names the front and says whether this worker has the key set, without
key or token material. Fronts that predate LifecycleReady are checked
with an empty LifecycleDelete, which they authorize and then reject
before any lookup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 10:53:07 -08:00
Chris LuandClaude Opus 5.5 c819e3fbea s3 lifecycle: per-pass run results and last full-pass status (#12643)
* s3 lifecycle: collect per-pass bucket results and work counters

WalkBuckets kept only the first bucket error and logged the rest. A
RunStats shared by the walker, the walker dispatcher and the replay
drain now records every bucket's walk result (walked, failed, skipped
on cancellation), entry and object visits, replay events, and
LifecycleDelete outcomes under the dispatch_total outcome labels.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: persist the last fully successful pass

Shard cursors and walk timestamps advance on partial passes as well, so
neither says when lifecycle last covered every policy. Keep that in its
own record next to the cursors, plus a worker gauge for it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: report each pass's result in the job record

The job ended with a fixed success summary, or with only the first error,
and malformed lifecycle policies were V(1) logs. The completion now
carries the pass status (complete, partial, failed, canceled), bucket
totals and failures, invalid policies, entry, object and event counts,
LifecycleDelete outcomes, the last phase and bucket, the elapsed time
and the last full pass, on failed passes as well.

Only a complete pass advances the full-pass record; a malformed policy
makes the pass partial.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: keep a failed job's result in its record

Completions with Success=false dropped Result.Summary and
Result.OutputValues, so a handler could not report what a failed run
had done, which buckets failed, or how far it got. Keep them; the error
message still drives the job's message and run record.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: chart the time since the last full lifecycle pass

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: a replay halted on an unresolved delete is a partial pass

BLOCKED, RETRY_LATER and exhausted transport retries stop a shard's
replay without an error, leaving its due events for a later pass. Count
those shards and report the pass as partial, so it no longer advances
the full-pass record.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: a pass with throttled walks is not a full pass

WalkerInterval can skip the walk for walk-only rules (ExpirationDate,
ExpiredDeleteMarker, NewerNoncurrent) with no error. Count those shards
and keep the full-pass record for passes that evaluated every rule; the
job reports full_pass and walks_deferred.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: keep the saved full-pass time when the save fails

Publishing a timestamp that never reached the filer let the gauge move
back on the next job's reload. Keep the previous value and report the
save error in the job result.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: drop a failed attempt's summary when the retry reports none

Retries reuse the job record, and failed completions now keep their
result summary, so a retry that completes without a Result showed the
failed attempt's summary, and a successful one used it as its message.
Clear the summary when a completion carries no Result, as the output
values already are.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 10:50:37 -08:00
Chris LuandClaude Opus 5.5 6ad4b07598 mount: scope and order the final-release writeback fence (#12651)
* mount: fence a dropped handle only when it holds unpublished content

dirtyMetadata is also set by chmod, xattr and fallocate, so a failed
metadata-only close refused every later write of the inode.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: fence the inode before its last handle leaves the handle map

A reopen between the map removal and the fence could publish at the
stale kernel EOF.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: fence a failed metadata flush only when the handle holds content

A refused synchronous flush or an abandoned writeback metadata flush of a
handle with only chmod, xattr or fallocate changes fenced the inode, so
later writes were refused until FORGET.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: count a size change on an open handle as unpublished content

A truncate or extend through an open handle changes the entry's size and
chunks, but it set only dirtyMetadata. A failed close then skipped the
final-release fence, so a reopen before FORGET could publish against the
untruncated entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: mark copied content dirty before each page of a streaming copy

A copy that failed or was cancelled after earlier pages were accepted left
dirtyContent false, so the final-release fence skipped that content.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: mark copied content dirty only after a page is accepted

A copy whose first page is rejected leaves the destination unchanged and
the kernel EOF where it was, so fencing the inode on release only blocked
later writes until FORGET.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: skip the server-side copy while the destination holds unpublished content

A whole-file server copy clears dirtyContent when it commits, so a page a
cancelled streaming copy left behind would lose its release fence. Fall
back to the streaming copy instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: skip the server-side copy while the source holds unpublished content

The filer copies the stored source, so pages still pending on the source
handle would be missing from the destination. The streaming copy reads
them through the dirty pages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: check the publication fence before a server-side whole-file copy

The filer copy publishes the destination without going through the
flush-time guard, so a fenced writeback inode could still take new
content before FORGET. Fall back to the streaming copy, whose flush is
guarded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 10:47:33 -08:00
Ping Qiu 65ecb6aa12 fix(mount): prevent negative activity-score shifts (#12631)
fix(mount): bound activity decay shifts across time regressions
2026-11-14 10:46:57 -08:00
Chris LuandClaude Opus 5.5 0a38738076 filer: keep the listing snapshot below later and unfinished event stamps (#12642)
filer: take the listing snapshot as a claim below later and unfinished stamps

A ListEntries snapshot tells the client that every event at or below it is
already in the listing, so the mount drops those events from a directory
build and fences them with the directory floor. The snapshot was a raw
time.Now(), which an event stamp can tie on a coarse clock: on Windows a
mkdir right after a listing of its parent was stamped with the snapshot's
own timestamp, the mount took it as listed, and the new directory stayed
missing from a parent marked cached. A create stamped before the snapshot
but not yet in the store was dropped the same way.

Take the snapshot through the same in-flight claim the idle heartbeat uses:
it stays below every unfinished stamp, and every later stamp lands above it.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 01:17:51 -08:00
10f8b2d5b1 mount: guard peer-truncate gaps, stale append EOF, and terminal writeback refusal (#12630)
* mount: guard append gaps over concurrent content merges

* mount: fence a writeback inode when its async publication is abandoned

A drain that gives up on a non-precondition error left the token clear, so a
reopened writeback handle could publish past the stored EOF.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: fence a writeback inode when its async data flush fails

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: fence the inode when its last handle drops uncommitted content

A synchronous release whose final flush fails destroys the handle's dirty
pages, while under writeback the kernel EOF already counts them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 01:13:37 -08:00
b50d4a9cab mount: don't spend the close()/fsync commit deadline on uploads (#12641)
* mount: start the close()/fsync commit deadline after the data upload

Flush, Fsync and the truncate path of SetAttr create the 30 s
metadataFlushTimeout context on entry and pass it to doFlush, so the
clock also runs through FlushData, the upload of the dirty pages. That
upload has its own timeouts and retries. When an assign through a stalled
filer or a slow volume server takes most of the 30 s, CreateEntry then
fails at once with "context deadline exceeded" and close() returns EIO
for data that was already stored. The deadline is meant for the commit
alone (#10102: "a healthy CreateEntry completes in well under a second").

doFlush now takes the commit timeout and starts the deadline once the
data is uploaded. close() still cannot hang on a stuck filer: the commit
keeps its 30 s bound. Release and the flush before a rename pass 0, as
before (no overall bound, only the per-call filer timeouts).

Seen in production on a training cluster: a 30 s stall of one replica's
disk, and a filer that stalled for about 30 s, each failed close() with
EIO although the upload had succeeded.

Tests: close(), fsync() and a truncate below dirty data, each after an
assign slower than the deadline (all three fail before this change), and
a filer that never answers CreateEntry still fails close() at the
deadline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: charge the flush commit deadline only for its filer calls

Starting the 30 s deadline after the data upload is not enough.
flushMetadataToFiler does more than call the filer before it commits: it
waits for the handle's locks, reads the file back and uploads it again
when its chunks are bloated (maybeMergeChunks), and uploads a manifest
once enough chunks have built up (MaybeManifestize). None of that takes
the context. When it outlasts the deadline, CreateEntry again fails at
once with "context deadline exceeded", and close() or fsync() returns an
error for data that was stored.

The deadline becomes a budget, commitBudget, charged only while a filer
call of the commit runs: CreateEntry, and the read of a changed hard
link before it. The attempts of one flush share it, and a flush is not
retried once it is spent, so a stuck filer still fails close() after
30 s. Release, the flush before a rename and the async flush pass no
budget, as before.

Seen in production: the merge of a 1.8 MiB sqlite file with 183 chunks
took 32 s because one chunk's read first tried a volume server that was
down, and the commit failed 32 microseconds after the merge finished.

Tests: a flush whose manifest upload is slower than the deadline commits
(it fails without this change), and the filer calls of one flush share
the budget.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* mount: give the stalled-lookup refusal test the new doFlush argument

TestContentFlushRefusalWithStalledLookup came in from master after this
branch changed doFlush's signature, so the merged package did not build.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-13 21:56:44 -08:00
Chris LuandClaude Opus 5.5 e0890391ac mount: keep a refused content create refused when the stored-entry read stalls (#12629)
The read of the stored entry ran in the create's filer callback, so a read
that hit the call deadline turned the refusal into a filer timeout, and
both retry layers resent the refused create. Read it in its own call,
bounded to one filer call.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 20:19:35 -08:00
Chris LuandClaude Opus 5.5 6eb2fb4842 filer: stop full-chunk reads at the chunk size after a mount truncate (#12632)
* util/http: stop a full-chunk read at the chunk's recorded size

weed mount shrinks a chunk's Size in place on ftruncate but leaves its
needle untouched. A view covering the whole shortened chunk is fetched
without a Range header, so the reader returned every stored byte. The
filer GET then wrote past the declared Content-Length and sent an empty
body, and the stream reader served the stale tail.

ReadUrlAsStream and the cipher path now deliver at most size bytes;
size 0 still reads the whole needle for manifest and log chunks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: limit a plain chunk read to the requested size

A full-chunk read sends no Range header, so a chunk shortened by a mount
truncate streamed its whole needle into the S3 response.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* util/http: still report a failure past the chunk size

The volume server aborts a full-needle response when the needle fails
its checksum, and only at the end. Stopping at the chunk size missed
that, so a damaged copy read as healthy. Read the rest of the body and
return its error so the caller tries another copy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 20:17:46 -08:00
Chris LuandClaude Opus 5.5 63c99183d1 s3vectors: name weed-vector processes servers (#12627)
* s3vectors: call the registered weed-vector processes servers

The gateway's discovery named them workers, the name of the admin
plugin workers. Also drops two comments left without a function.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: build the Enterprise engine as weed-vector

In an Enterprise checkout, make ./weed-vector builds the distributed
engine, which also serves VectorService, instead of a separate
weed-vector-worker. An open-source checkout builds weed-vector-server
as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: distributed mode starts two more weed-vector engines

It started weed-vector-worker, which no target builds now, and waited
for a gateway log line that no longer exists. The Enterprise
weed-vector's engines all join the placement controller.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 20:17:15 -08:00
Chris LuandClaude Opus 5.5 a8436a621a admin: vector servers on the dashboard (#12606)
* admin: vector servers on the dashboard

A Vector servers count in the Cluster card and a Vector Servers table
after S3 Servers, from the engines the dashboard already discovers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* admin: drop the dashboard's vector server kind

Every vector server is a weed-vector engine, so the column said the
same thing on every row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 20:16:43 -08:00
Chris LuandClaude Opus 5.5 58d25998d0 admin: Cluster → Vector Servers page (#12603)
Lists the weed-vector engines registered with the master, with version
and health, next to the other server lists.


Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 20:16:00 -08:00
github-actions[bot] d91d58ce47 docs: regenerate star history chart 2026-11-14 02:04:51 +00:00
Chris LuandClaude Opus 5.5 4fc41872e3 mount: keep an open file publishing when a peer unlinks one of its hard link names (#12613)
* mount: record a hard link's looked-up names on its inode

Lookup mapped a second name of a hard-linked file to the existing inode
without adding it to the inode's names, so GetAllPaths missed it and the
final forget left it mapped to a released inode.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep an open file publishing when a peer unlinks one of its names

A delete event from another client marked the inode's open handle deleted
even when other names still held the hard-linked file, so every later
fsync reported success without saving metadata. The delete event's entry
carries the link count the filer decided on, so an alias delete now drops
only that name, keeps the handle on a name this mount knows, and has the
next flush read the survivor's stored entry before publishing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: test peer unlinks of a hard link name held open

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: leave an unlinked file out of the periodic metadata flush

flushFileMetadata did not check the handle's delete mark. A file
unlinked while open before its deferred create reached the filer was
created again by the next periodic flush, which expects no entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: read a pending link change before a periodic metadata flush

Only flushMetadataToFiler applied linkRefresh. After a peer unlinked one
name, the periodic flush kept publishing over the old link count, so the
filer refused every one until a regular flush ran.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: test the periodic flush after a peer unlinks a hard link name

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 16:14:22 -08:00
Chris LuandClaude Opus 5.5 08f9dcb6de admin: vector buckets link maintenance jobs to the Lance lane (#12633)
The Vector Engines and Maintenance card linked to the general plugin
monitoring page; the vector index jobs run in the Lance lane.


Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 16:14:19 -08:00
Chris LuandClaude Opus 5.5 8a756af348 vector: key shared leases by manifest commit time too (#12626)
On Windows Lance commits local tables by rename and leaves the manifest
ETag empty, so a table dropped and written again at its location reused
the old table's lease for the same version number.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 15:52:52 -08:00
Chris LuandClaude Opus 5.5 68b6a2ed08 mount: model the refused async flush as a replaced file in recovery tests (#12628)
A peer's chunk append now merges into the flush, so these tests no longer
saw the refusal they exercise. A replaced file is still refused.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 13:50:51 -08:00
anantandanant fdd3a692d3 shell: show -collection, not -c, in ec.balance and ec.rebuild help (#12617)
Both usage lines showed [-c EACH_COLLECTION|<collection_name>], but the
flag is -collection; there is no -c. ec.rebuild's own Options section
already lists -collection.

Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: anant <170962632+antcybersec@users.noreply.github.com>
2026-11-13 13:03:07 -08:00
Chris LuandClaude Opus 5.5 45cd6033a0 s3: make conditional DeleteObject atomic on versioned buckets (#12624)
* s3: re-check a versioned DeleteObject If-Match under the object lock

A versioned or suspended DeleteObject without a versionId routed to the
owner filer after checking If-Match outside any lock, so concurrent
conditional deletes could all pass the check and each add a delete
marker. Keep conditional deletes on the lock path, which re-checks the
precondition under the object write lock, as conditional versioned PUTs
already do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: guard a conditional version delete on the owner filer

DeleteObject with a versionId checked If-Match before routing the
delete to the owner filer, so concurrent conditional deletes of one
version could all report success. Carry the If-Match clause into the
owner's transaction as a guard on the version (the null entry for
versionId=null), evaluated under the object's path lock before anything
applies, and map that guard failing to 412 instead of the object-lock
403. An If-Match the filer cannot evaluate stays on the lock path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: carry an If-Match ETag list into the routed delete

A DeleteObject whose If-Match listed several ETags fell back to the
gateway's object write lock, while a single-ETag delete of the same key
ran under the owner filer's path lock. The two locks do not exclude each
other, so a list delete and a single-ETag delete could both report
success. Send every candidate to the filer, whose strong comparison
matches checkDeleteIfMatch, so all conditional deletes with an owner are
decided under the owner's lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: fail a conditional null delete on an owner transport error

The owner may have committed the delete before the error, and the gateway
lock path it fell back to does not exclude routed deletes, so a second
If-Match delete could pass there too. Answer 500 as the specific-version
path does; unconditional deletes still fall back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: fail a conditional null delete on any owner error but the nested refusal

Only the owner's non-empty-folder refusal is known to apply nothing. After
any other owner error the delete may have been partly applied, and a lock-path
re-check does not exclude a concurrent routed delete of the same version, so a
conditional null-version delete answers 500 instead of falling back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: strip a nested conditional null delete on the owner

A conditional versionId=null delete that the owner refuses because other
keys are nested under the entry used to fall back to the gateway lock and
re-check If-Match there. That lock does not exclude routed deletes: once
the last nested key is removed, a concurrent routed delete can pass the
same check on the owner, and the fallback then treats the missing entry as
deleted, so both requests answer 204.

Demote the entry on the owner instead, in a transaction guarded by the
If-Match clauses and by the entry it was read as, with the object-lock
guards on the request. A missing or already demoted entry answers 412.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: add an IF_OBJECT_EXISTS write condition

IF_EXISTS also holds for a plain directory, which is what is left of a key
once the object on it is stripped while other keys stay nested under it.
IF_OBJECT_EXISTS holds only for an entry that carries an object.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: require an object for a conditional null delete with If-Match: *

Stripping the null version off an entry other keys are nested under leaves a
plain directory. Once the last nested key is gone, a routed delete with
If-Match: * still found an entry there and removed the empty directory, so a
second conditional delete could also answer 204. The owner now checks that
the entry carries an object.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: settle a wildcard null delete on the entry when the filer predates IF_OBJECT_EXISTS

A filer older than IF_OBJECT_EXISTS fails it closed, so with the gateway
upgraded first every If-Match: * delete of a null version answered 412. When
the narrowed guard fails, the gateway now reads the entry on the owner: absent
or a plain directory is a 412, otherwise the file is deleted, or the directory
stripped, under IF_EXISTS and IF_ENTRY_EQUAL on the entry it read. The demote
keeps the caller's If-Match clauses, since the entry guard already requires an
object.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:47:22 -08:00
d695d101b5 Fix/s3 abandoned chunk download (#12616)
* filer: cancel a chunk download that no reader waits for

* filer: stop waiting for reader cache budget once cancelled

* filer: keep a cancelled download's re-lookups off its context

The refresh and retry lookups took the download's context. FilerClient
coalesces concurrent lookups of a volume, so a cancelled download leading
one failed the other readers waiting on it. Fetches still stop on cancel.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test that cancelling a download leaves its re-lookup running

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: cancel unheld downloads when the reader cache is destroyed

ReaderCache.destroy waited for every downloader, so closing a reader
waited for a stalled prefetch nobody read. Downloads with no reader or
stream on them are now cancelled first; ones still in use keep running.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test that destroying the reader cache cancels an unread prefetch

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: cancel a destroyed download once its last holder leaves

destroy skipped downloads a reader or stream still held, and once they
were out of the map nothing cancelled them, so destroy waited for a fetch
nobody used. removeConsumedLocked now cancels any consumable download,
whether or not it is still in the map.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test a destroyed download cancelled after its last stream leaves

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: release the stream only after destroy dropped the download

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test a cancelled stream leaving a waiting reader's download running

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:45:34 -08:00
Chris LuandClaude Opus 5.5 813ac74d79 s3tables: keep legacy-subject owners and policy principals for federated STS sessions (#12615)
* s3tables: re-anchor legacy owners for federated STS sessions

A web-identity role session was keyed on its bare OIDC subject until
owners became issuer-scoped. Its sessions now carry the scoped aws:userid
but no asserted issuers, so the legacy subject check never applied and the
session lost the table buckets and namespaces it had created. Read the
configured issuer set for such sessions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: keep token claims from asserting trusted issuers on STS sessions

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* iceberg: give the catalog's table handler the local accounts and issuer set

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: test a federated session on a bucket its legacy subject owns

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sts: drop a token-supplied aws:userid when a session has no parent user

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: take a session's aws:userid from its signed parent user

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: test that session claims keep only the server-asserted aws:userid

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: match legacy subject patterns in table bucket policy principals

A policy written before issuer scoping can name role sessions by a pattern
over their bare OIDC subjects. Match such a pattern against the caller's
subject claim once that subject resolves to the session's scoped principal
under the single configured issuer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: test a federated session against a legacy subject pattern principal

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:43:52 -08:00
Minh Vu 4bc7598a43 admin: escape dynamic text in S3 bucket modals (#12625)
admin: escape dynamic text in bucket modals
2026-11-13 12:42:40 -08:00
Chris LuandClaude Opus 5.5 d685709832 s3api: refuse writes to a stored version's physical path (#12622)
* s3api: refuse a PUT to a stored version's physical path

In a Suspended bucket a PUT to <key>.versions/v_<id> replaced the stored
version's body in place. Look the entry up when the key has the version
file shape and deny the write when it is a version file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: refuse copy, multipart, POST and rename writes to a version file

Apply the same version file check to the CopyObject, CreateMultipartUpload,
CompleteMultipartUpload, POST Object and RenameObject destinations.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: read a version file write target on its logical owner

The guard resolved the key as an object, which on a ring reads the
physical path's owner and, in a versioned bucket, may return a nested
version under <key>.versions/v_<id>.versions instead of the stored
version. Look up the physical entry directly, routed by the logical key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: confirm a version file miss on a skipped owner

When the owner was recently unreachable the routed lookup answers from a
replica, whose miss does not show the version is absent. Ask the owner
directly in that case and refuse the write when it cannot answer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: confirm every routed version file miss on the owner

Whether the routed lookup skipped the owner can change between the lookup
and a second unreachable check, letting a replica's miss through. Ask the
owner after any miss instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:39:40 -08:00
d1a97f4c3d fix(mount): retain refused async metadata before releasing handle (#12598)
* fix(mount): retain refused async metadata before releasing handle

* mount: sync the failed-metadata directories with util.FsyncDir

Directory fsync fails on Windows, where the mount also builds, and returns
EINVAL on some filesystems. Either kept the retention loop retrying a record
that could never be saved, so the drain never finished.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep failed-metadata records in a per-user directory

The write cache root defaults to the shared temp directory. The first mount
to save a record created failed-metadata there as 0700, so another user's
mount could never save one and its drain retried forever.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: drop the async conflict probe test

TestContentAsyncConflictKeepsRecoveryRecord drives the same refused drain
with the retry backoff stubbed out; the probe slept through the real backoff
(7 s) and its kept-handle branch could not run. Its peer-entry check moves
into the recovery record test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:34:58 -08:00
5a54009439 filer/etcd: client keepalive and per-request deadlines (#12621)
* filer/etcd: client keepalive and per-request deadlines

An etcd member that goes silent with its connections open (a dead or hung
node, a network partition, a frozen process) keeps the store's requests
balanced onto it waiting forever, because the client has no keepalive and
the calls have no deadline. Writes through the filer stop although the
other members keep quorum.

keepalive_timeout (default 3s, "0s" turns keepalive off) enables gRPC
keepalive with the ping interval fixed at gRPC's 10s minimum. gRPC also
sets TCP_USER_TIMEOUT to it on Linux, so a connection whose sent data goes
unacknowledged that long is closed by the kernel. request_timeout (default
3s) bounds every Get, Put and Delete unless the caller set a deadline;
timeout keeps its meaning as the dial timeout. DeleteFolderChildren gets no
request deadline: its DeleteRange grows with the directory, and keepalive
still ends it if the member goes silent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Update weed/filer/etcd/etcd_store.go

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>

* Update weed/filer/etcd/etcd_store_test.go

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>

* Update weed/command/scaffold/filer.toml

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>

* filer/etcd: no request deadline on entry writes

A Put whose client deadline expires may still have been committed. The
filer's upload handler treats the error as a failed write and deletes the
uploaded chunks, which the saved entry may then point at. InsertEntry and
UpdateEntry therefore keep the caller's context: etcd's own request
timeout ends a Put a live member cannot commit, and keepalive ends one on
a member that goes silent. Reads and deletes keep request_timeout; a
timed-out store delete returns before the filer deletes any chunks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer/etcd: no request deadline on KvPut

The filer wrapper stores a hard-linked file's attributes and chunk list
through KvPut. An HTTP append to a hard-linked file whose KvPut times out
after etcd saved it returns an error, and the upload handler deletes the
appended chunks the saved hard link now points at. KvPut keeps the
caller's context like InsertEntry and UpdateEntry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-13 12:28:48 -08:00
Yip LeoandClaude Opus 5.5 911bfe4014 mount: stop overwriting the mount root's filer entry (#12623)
When a mount is rooted at an existing directory, two code paths replace
that directory's filer entry with one the mount builds itself:

- every mount start re-creates the root with a non-exclusive mkdir,
  which the filer handles as an update of the whole entry;
- creating or removing an entry directly in the root saves the root's
  entry to touch its mtime, and maybeLoadEntry synthesizes that entry
  from the mount options instead of reading the filer.

The filer keeps only crtime and inode on such an update, so the root
loses its mode, owner and extended attributes. When the root is a
bucket it also loses the bucket quota, which silently drops to 0
(unlimited), and the settings S3 keeps on the bucket entry. The explicit
allowEmptyFolders=false that #11370 meant to keep is replaced by true at
every mount start, because the mkdir has wiped it before
ensureBucketAllowEmptyFolders looks.

Create the root only when a lookup says it is missing, and then
exclusively. Leave the root out of the filer-side mtime touch: its
attributes come from the mount options (setRootAttr), so the filer copy
of its mtime was never visible through the mount anyway.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:27:15 -08:00
Chris LuandClaude Opus 5.5 737ef68a11 mount: keep publishing an open file through its surviving hard link (#12609)
* mount: keep flushing an open file whose other hard link survives an unlink

Unlink marks the inode's open handle deleted so no flush recreates the
removed name. When the entry was one of several hard links the inode lives
on under another name, yet the mark stayed and every later fsync and close
returned OK without publishing metadata. Lift the unlink's own mark once
the name is gone and the inode still has a path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: carry a link's stored entry into the file's open handle

Link rewrites the source entry with the new hard link id, count and times,
but an open handle kept its old base and entry. Its next conditional
content flush was refused (EIO), and an unconditional one would have
dropped the hard link id. Move the handle onto the stored entry when only
the link fields and times changed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: carry an alias unlink's stored entry into the open handle

Removing one of several hard links lowers the stored link count and
stamps a new ctime, and the handle may still hold the removed name. Look up
the entry at the name the handle flushes through and adopt it, so the next
content flush matches the stored entry, writes the right count, and
attribute saves address the surviving name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: test an open handle across link and alias unlink

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: read an alias unlink's lost entry again at the next flush

When the read of the surviving entry right after an alias unlink fails,
the handle kept a base with the old link count and ctime, and every later
flush was refused even after the filer recovered. Leave the hard link id
on the handle and have the next flush read the stored entry and adopt it
before publishing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: publish a queued close before unlinking a name another link keeps

With writeback caching, close only queues the flush. Unlink marked the
handle deleted before waiting for that flush, so the flush skipped its
metadata and the surviving link kept the content from before the close.
When another link keeps the file, wait for the queued flush first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: let a queued close reach the filer before linking the file

With writeback caching a new file is created on the filer by its queued
close, and a listed directory serves the source from the cache, so a link
right after close updated a name the filer did not have yet and failed
with EIO.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: skip a link change no newer than the handle's entry

A link change carried into the handle could replace a base already moved
past it by a later acknowledgment. Use the same clock-domain check as
installAckedEntry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep a link change's ctime in the dirty entry

The flush does not restamp times, so a dirty entry carrying an earlier
write's ctime moved the filer's ctime back from the link or unlink.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: flush through the file's current name when the resolved one is gone

doFlush resolves the name before uploading the data. An unlink of that
alias, or a rename, in the meantime sent the publish to a name the file no
longer has.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep the pending link read when its name is already gone

A flush could read the removed alias name, find nothing, and drop the
pending read of the surviving link, leaving the stale base in place.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: test link and alias unlink across queued closes, flush races, versions and ctime

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: pick a flush's name again once its locks are held

An unlink or rename that lands between the name pick and the locks would
otherwise send the flush to the removed name, where the create guard fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: test a flush waiting on its lock through an alias unlink

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: check the flush target after every lock acquisition

A flush whose name kept changing published to the name chosen before its
last lock wait. Check after each acquisition and return a retryable error
once the attempts run out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: test a flush whose name changes during every lock wait

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:25:50 -08:00
bac2196d65 shell: return flag parse errors from every command (#12618)
* shell: return flag parse errors from every command

48 commands returned nil when their flags failed to parse, so a mistyped
flag printed the flag error but the command still counted as a success:
a piped weed shell run exited 0, the master maintenance loop logged no
error, and admin script jobs reported the line as done.

Return the parse error as s3.bucket.create (#12544) and the other shell
commands already do. TestCommandsReturnFlagParseErrors runs every
registered command with an unknown flag and fails for the 48 on master.

Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>

* shell: don't report -h as a failure

With the parse errors returned, -h on those commands now returns
flag.ErrHelp, so the shell printed "error: flag: help requested" and a
piped run exited 1. The flag package has already printed the usage, so
processEachCmd now treats flag.ErrHelp as success for every command.

Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>

* shell: keep the -h test beside the flag parse test

TestProcessEachCmdTreatsHelpAsSuccess covers the same change as
TestCommandsReturnFlagParseErrors, so it lives in the same new file and
shell_liner_test.go stays as it was.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: anant <170962632+antcybersec@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:19:44 -08:00
Chris LuandClaude Opus 5.5 c248ab7865 credential: keep IAM changes made during the legacy-file migration (#12614)
* credential: migrate the legacy IAM file without overwriting newer entries

The migration wrote every identity, service account and group from the
snapshot the load read, over whatever the per-entry files held by then, so
a revoke or delete landing in between was undone for good. It now migrates
only the legacy file's own entries, creates each per-entry file only when
none exists, and waits for the next load when the legacy file changed
after the snapshot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* credential: test changes landing between the IAM load and its migration

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* credential: keep a per-entry file deleted after the IAM load deleted

An entry the load read from its own file is already migrated. When that
file is removed before the migration runs, the legacy copy must not
recreate it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* credential: test a deleted per-entry file the legacy file also holds

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:17:45 -08:00
Chris LuandClaude Opus 5.5 12cb707e58 seaweed-volume: copy master.proto with metrics_port (#12605)
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:10:48 -08:00
Chris LuandClaude Opus 5.5 aa901b7ac7 s3tables, iceberg: create missing directories exclusively so a concurrent create keeps its record (#12600)
* s3tables: create a missing directory exclusively in ensureDirectory

ensureDirectory looked the path up and, when it was missing, created it with a
plain CreateEntry. A table create that lost the lookup race then replaced the
directory another create had made, claimed and confirmed since, erasing that
table's record, so a third create could claim the name and win as well.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* iceberg: create missing metadata parent directories exclusively

saveMetadataBlob created each missing directory on the way to the metadata
directory with a plain CreateEntry after a lookup. A table directory another
create had made and written its record onto since the lookup was replaced by a
bare directory, erasing that record.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* iceberg: create missing stage-create marker directories exclusively

The marker path starts at the table bucket directory, which carries the bucket
record; a plain create after a missed lookup replaces a directory another writer
made in between, with whatever it carries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:10:18 -08:00
Chris LuandClaude Opus 5.5 e0beb59ef0 topology: treat a nil ttl as the empty ttl when looking for duplicate volume ids (#12601)
Volume layouts are keyed by ttl.String(), so a nil ttl and the empty ttl
share one layout. ttlEqual told them apart, so a replica reported with a nil
ttl into a layout created with the empty ttl matched its own layout as a
duplicate, and every copy was quarantined.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:09:07 -08:00
Chris LuandClaude Opus 5.5 fe14880f4d shell: tier download replicates the downloaded volume before deleting the remote object (#12599)
* shell: replicate the downloaded volume before deleting its remote object

volume.tier.download downloaded every replica from the remote tier and let
the last one in the master's location list delete the shared object, so a
replica missing from that list kept reading a deleted object.

Download onto one replica, copy it over every other remote-backed replica,
re-read the topology so replicas that show up meanwhile are replaced too,
and only then delete the last remote-backed replica together with the
object before copying the local volume onto it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: test tier download replicates before deleting the remote object

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: copy the downloaded volume onto each replica's own disk type

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: test tier download keeps each replica's disk type

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: name the hdd disk type when copying a downloaded volume

VolumeCopy treats an empty disk type as the source's, so an HDD replica
copied from an SSD source was placed on SSD or failed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: block deletes on the tier download copy source

A readonly-can-delete source took deletes after VolumeCopy read its
index, so the replaced replica lost them and served the files again.
Hard-mark the source readonly while replicas are replaced, then restore
the writable mark or reload the persisted state.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: keep the remote object while replicas are unlisted

Tier download deleted the shared object once the listed replicas were
local, even when the replica placement expected more copies than the
topology listed. Keep it in that case so a rerun can finish later.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: fail tier download when no replica is listed

Also point the final-replica delete error at volume.fix.replication.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: keep the remote object when a server lists the volume twice

A stale twin on another disk of a replaced server still references the
object but is filtered out with its server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: restore the tier download source when its readonly mark fails

VolumeMarkReadonly blocks local writes and deletes before its second master
notification, so a failed notification returns an error with the source
already frozen. Restore it whenever the status was read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: report a failed tier download source restore

A restore that leaves the source unmounted or frozen was only printed, so the
command still returned success. Join the restore error into the result.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: wrap the tier download restore error

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:07:52 -08:00
Chris LuandClaude Opus 5.5 94c3869057 volume_move: keep the live-move source frozen across volume reloads (#12597)
* volume_move: persist the move's source freeze across reloads

A live move froze the source in memory only, so an unmount/mount or a
volume server restart mid-move reopened it for writes that the move then
deletes with the source. Persist the freeze when the move froze a
writable source, and reopen the target right after the copy since the
copied .vif carries the freeze. An already-readonly source keeps an
in-memory mark, and a failed move still restores the source writable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: live move keeps the source frozen across a reload, on Go and Rust volume servers

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: require 409 from the reloaded frozen source

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume_move: reopen an interrupted move's source only without a target copy

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:05:58 -08:00
Chris LuandClaude Opus 5.5 652648d9c4 mount: merge concurrent content changes from another mount instead of failing or dropping them (#12596)
* mount: merge a refused content flush onto the stored entry

A content flush conditioned on the handle's base failed with EIO whenever a
peer had changed the file since, so two mounts appending to one file could
not both close it cleanly. On a refused condition, reread the entry and
replay this handle's changes since its base onto it: the chunks it added or
dropped, and the attributes, extended keys and inline content it changed.
A local truncation keeps its size and clips the peer's chunks to it;
otherwise the larger size wins. The merged entry is published against the
stored one, up to three times, and installed in the handle so its next
flush keeps the peer's chunks. A replaced or deleted file is still refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: merge a peer's content change into a dirty handle

A subscription event carrying a peer's content change replaced a dirty
handle's entry and discarded its unflushed pages, so the handle's writes
were lost and its close still returned OK. Fold the peer's entry into the
handle with the same merge a refused flush uses, keep the dirty pages, and
take the peer's entry as the new base. A replaced file still invalidates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: stop a flush merge once a filer accepts the write

withFilerClient can call the callback on several filers. A refusal from
one left its stored entry behind, so a later filer's success went on to
another merge round.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: refuse a merge that puts inline content next to chunks

A read inside the inline bytes returns them without looking at chunks,
so one side's inline content merged with the other side's chunks hid
data after a successful flush.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: refuse to merge a whole-file rewrite over a peer's chunk change

maybeMergeChunks re-uploads the handle's whole view as unstamped chunks.
Merged after a peer's change, they tie with a peer's own rewrite or bring
back a range the peer truncated, so that flush fails as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: clip peer chunks at the lowest truncate, not the final size

A handle that truncated and then wrote past the old end ends up larger
than its base, so the merge did not see the truncate and kept the peer's
chunks inside the range the truncate cleared. SetAttr now records the
lowest size truncated to over the current base.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep a peer's truncate when the handle left the size alone

The merge took the larger of the handle's and the stored size, so a peer
truncate came back as a zero-filled tail when the handle only overwrote
bytes inside it. The stored size now stands unless the handle changed
its size or wrote past it, and rebasing keeps only the growth writes made
after the flush was taken.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: retry a read view whose merged manifests did not resolve

A failed SetChunks leaves the chunk group answering every read with the
resolution error, and nothing re-ran it while the entry stayed the same.
The handle now marks such a view (on rebase and on SetEntry) and the
next read or lseek resolves the entry's chunks again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep a dirty handle's writes when a peer change cannot merge

The new merge refusals sent a peer event on the same file down the old
path, which discarded the dirty pages and adopted the peer's entry, so
close returned OK without the writes. The handle now keeps its writes and
base, and the flush meets the conflict and fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep an unpublished truncate across a peer's base change

The truncate mark was tied to the base pointer, so a peer event that moved
a dirty handle's base (a merged content change or a metadata-only change)
dropped it, and a later merge on a regrown file kept peer chunks inside
the truncated range. Those base moves now carry the mark; a flush ack or a
discarded handle state still clears it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: wait for the flush before merging a shrink over buffered writes

A write inside the file leaves its bytes in dirty pages without moving
the size. Merging a peer's truncate then adopted the smaller size, and the
chunk uploaded later sat past it, unreadable through the handle. The event
path and the periodic metadata flush now refuse a merge that shrinks the
handle; the close flush uploads the pages first and merges then.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: tell a shortened base chunk from a whole-file rewrite

A truncate shortens a base chunk in place, so it gets a new reference
key, and an old chunk can carry no timestamp. The rewrite refusal now
applies only to unstamped chunks whose file id is new to the base.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: check the creation time when the base has no inode

An entry without an inode or hard link id matched any replacement at the
same path, so a stale handle could merge into a different file. Without
an inode the merge now needs the same creation time. The filer keeps it
across updates; only a synthesized base has neither and still merges.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep a dirty handle's writes when a peer replaces the file

A peer entry with another inode at the handle's path went down the
fallback that destroys the dirty pages and installs the replacement, so
close returned OK without the writes. The handle now keeps its writes
and base like it does for an unmergeable change, and its flush refuses to
update the replacement.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: build the content flush tests on freebsd

The invalidation tests, which every platform compiles, now use the
content flush helpers, so the freebsd cross-compile vet failed with
undefined: flushConditionEntry. Nothing in the file is freebsd-specific.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep a dirty handle's base when a same-content file replaces it

A replacement with the base's chunks but another inode counted as a
metadata-only change, so the dirty handle adopted it as its base and the
next conditional flush overwrote the new file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: bring a peer's metadata change into a dirty handle

A metadata-only event moved a dirty handle's base but left its live
entry, so the next flush read the peer's chmod as a local edit and
published the old mode back. Replay the handle's own changes onto the
peer's entry instead, as for a content change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: wait for the flush before merging peer inline content over buffered writes

A buffered write is not a chunk yet, so a merge could install a peer's
inline content that later reads return ahead of the write's chunk.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: track a whole-file chunk rewrite on the handle

The merge told a rewrite by its unstamped chunks, which a manifest fold
hides: manifests are unstamped either way. Record the rewrite against the
base it was made from, carried across peer base changes like a pending
truncate, and refuse the merge from that.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: refuse a merge that shortens a manifest at the truncate

Clipping a manifest reference leaves its members whole, so peer bytes past
the handle's truncate could reappear once the file grew again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: flush a dirty handle after a peer replaces its file

The replacement test checked only the retained handle state. It now
flushes against a filer holding the replacement, including one with the
base's content, and checks the flush fails and leaves it unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: test an append fsync after a peer's attribute-only change

A second mount's acknowledged chmod, chown, utimes or setxattr leaves
the content alone, so the writer's conditional content flush must still
publish its append and keep that change. On master the refused flush
surfaces as EIO from Fsync.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:02:42 -08:00
Chris LuandClaude Opus 5.5 a576398662 weed-vector: advertise its metrics port when registering (#12604)
* master.proto: KeepConnectedRequest carries the node's metrics port

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: advertise --metricsPort when registering with the master

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* master.proto: number metrics_port 9

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:36:02 -08:00
Chris LuandClaude Opus 5.5 baaa60b24a admin: an arrow icon for vector buckets (#12602)
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:11:13 -08:00
Chris LuandClaude Opus 5.5 917cf534db s3api: sign every operation subresource in signature V2 (#12575)
* s3api: sign every operation subresource in signature V2

The V2 canonical resource only kept the subresources inherited from the
original allowlist, so tagging, retention, legal-hold, cors, encryption,
object-lock and the other keys the router dispatches on were left out of
the signature. A V2 signature issued for one request therefore also
verified that request with one of those keys appended, running a
different operation under the signer's identity.

Add every operation-selecting subresource to the signed list. Clients
that already sign tagging, cors, object-lock or replication in V2 now
verify as well.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: sign every value of a repeated signature V2 subresource

canonicalizedResourceV2 kept only the last value of a repeated query key,
while handlers read the first. A request signed for versionId=A could be
sent as versionId=B&versionId=A and act on version B. Sign every value in
order, as V2 clients do, so a second value no longer verifies.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 17:22:59 -08:00
github-actions[bot] 5ba77af65f docs: regenerate star history chart 2026-11-13 01:20:00 +00:00
c2b2ed769b fix(mount): preserve appended chunks during stale attribute updates (#12593)
* fix(mount): preserve acknowledged chunks during attribute updates

* mount: keep the WFS field block aligned

Adding the publication-lock fields inside the aligned block made gofmt
re-indent 25 unrelated lines. Put them in their own group after
dirIdleEvict, matching the enterprise layout, so syncs do not conflict there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: finish attribute saves through a FUSE interrupt

saveEntryAttributes derived its RPC context from the syscall's cancel
channel, so an interrupt while the conditional UpdateEntry was at the
owner returned EIO for an update the owner then applied, and skipped the
local install and retention. Keep the metadata deadline but drop the
cancellation, like the other metadata mutations.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: map attribute CAS ids in place instead of cloning the entry

saveEntryAttributes cloned the whole entry three times per attempt only
to rewrite uid/gid for the filer, and each clone is as large as the
chunk list. Switch the ids in place and restore the exact local values
after the call, keeping one clone. The test now checks the reply carries
the local uid.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: compare attribute updates against the filer's own uid/gid

With -map.uid/-map.gid, a filer id that is also a mapped local id reads
back unchanged and maps forward to another filer id. The IF_ENTRY_EQUAL
expected entry then never matched, so chmod, utimes and xattr changes on
such files failed with EAGAIN after three attempts, and an update would
have rewritten the owner. Keep the filer ids read with the entry for the
ids the syscall leaves alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(mount): align attribute retention with read lock order

* mount: take the handle lock before entryLock in the attribute snapshot

attributeEntrySnapshot held entryLock while it queued for the shared
handle lock. A Read holding that lock waits on entryLock, the snapshot
waits behind a queued exclusive (an invalidation or a Write), and the
exclusive waits on the Read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(mount): complete handle lock order and preserve explicit ID intent

* mount: re-read a refused attribute update from the file's owner filer

The filer checks the IF_ENTRY_EQUAL condition on the owner of the path's
write route key. The retry read any filer, so behind one that had not
applied the owner's latest update, every attempt sent the same stale base
and the syscall failed with EAGAIN. Read from the owner once the lock ring
is known, like lookupFilerEntry, but keep the filer's raw ids.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 16:28:47 -08:00
d7ae08ecad fix(mount): refuse stale whole-entry content flushes (#12594)
* fix(mount): condition content flushes on the observed entry

* mount: stop retrying a flush the filer refused on its condition

The filer reports a failed CreateEntry condition as FilerError_PRECONDITION_FAILED
in the response, which surfaces as filer_pb.ErrPreconditionFailed rather than a
FailedPrecondition status. The flush retry loop treated it as transient and
replayed the refused flush three more times over 7 s before returning EIO.

The test filer now answers like the real one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: take the handle base from the entry the filer stored

After an acknowledged flush the handle base was the requested entry, but the
filer stores it with a fresh commit_ts_ns (and fills crtime, inode, atime), so
the next IF_ENTRY_EQUAL flush on the same handle never matched: write, fsync,
write, fsync returned EIO, as did close after a periodic metadata flush and the
first flush after an eager create. Use the stored entry from the
acknowledgement's metadata event.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: condition the periodic metadata flush like the close flush

The periodic metadata flush of an open, dirty handle sent the same whole entry
through an unconditional CreateEntry, so a file held open past the flush
interval still overwrote a peer's acknowledged append, and the close flush then
matched the overwritten state. Both paths now publish through one helper that
carries the handle's write condition.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: republish a refused flush once when the stored entry is unchanged

A handle's base can differ from the stored entry without any peer write: a
rename while the file is open restamps the entry under its new name, and with
-map.uid a filer id that is also a mapped local id does not map back to itself.
Each made every flush of the handle fail with EIO. On a miss, read the stored
entry and, if it still holds the observed state apart from name, commit stamp,
id encoding and derived size, retry once with that entry as the condition.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: take the filer's entry once a write open holds the DLM lock

The lock orders this writer after the previous holder, but the holder's
flush may not have reached the local cache yet, so the handle's base
named the entry from before the wait and the conditional flush refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: share the open-existing path of Create

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: open the file a previous DLM holder created during a deferred create

A deferred create never asks the filer, so a create that waited for the
lock went on to publish over a file the previous holder had just made,
and its flush expecting no entry was refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* master: send the lock ring snapshot to mounts

A mount running -dlm is a lock client like the S3 gateway, and needs the
ring on connect to reach a file's owner filer before the next membership
change is broadcast.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: follow the masters' lock ring under -dlm

The lock client only knew the seed filer, so every lock took the forward
hop and the mount had no way to tell which filer owns a file. Subscribe to
the masters named by the filer, like the S3 gateway, and wait briefly for
the first ring before serving.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: read the DLM-ordered entry from the file's owner filer

The filer checks a flush's condition on the owner of the path's write
route key, and another filer may not have applied the previous holder's
flush yet. Look the entry up on that owner once the ring is known.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 16:15:08 -08:00
Chris LuandClaude Opus 5.5 a2b9b37ee5 volume(rust): untrack queued writes before replying (#12595)
* volume(rust): untrack queued writes before replying

The durable write worker sent each write's result and only then dropped
the batch's pending-write trackers, so a caller could see its write
finish while the write was still tracked as pending. Go untracks in the
caller before writeNeedle2 returns. Drop the trackers before sending the
results so the Rust queue path matches.

test_replicate_delete_cancels_a_queued_write asserted the count was zero
right after the fsync write returned and failed about 1 run in 100 under
load (Rust Unit Tests on master c2625dd).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rust): untrack a write rejected for a remounted volume before replying

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 16:11:10 -08:00
Chris LuandClaude Opus 5.5 a2ffdce2ec volume: skip VolumeDelete when its RPC was cancelled before the delete (#12586)
* volume: skip VolumeDelete when its RPC was cancelled before the delete

A caller that cancels (a lost tiering lease, an expired deadline) still
had its admitted delete run in the Go server, including one that waited
behind the copy guard. The Rust server drops a cancelled handler, so
return the context error before the delete starts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: wait until the queued delete blocks on the guard before cancelling

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 10:19:05 -08:00
Chris LuandClaude Opus 5.5 6388dfca9b s3api: verify signed request bodies against the signed payload hash (#12577)
* s3api: verify signed request bodies against the signed payload hash

A SigV4 signature covers the declared x-amz-content-sha256, but only
getRequestDataReader (PutObject, UploadPart) compared the body with it.
Every other handler parsed the body as received, so a signed request,
DeleteObjects for example, could be replayed with a different body.

After a V4 signature verifies, wrap the body so it is handed out only
once all of it has been read and matched against the hash the signature
covered; a decoder that stops reading early never sees unverified bytes.
getRequestDataReader unwraps it and keeps verifying uploads as they
stream.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: answer XAmzContentSHA256Mismatch for refused DeleteObjects bodies

DeleteObjects answered a body that failed its payload-hash check with
InternalError and CompleteMultipartUpload with MalformedXML. Name the
check that failed instead, as PutObject does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: check Content-MD5 on signed request bodies

Signature V2 signs Content-MD5 rather than a payload hash, and a V4
request sent with UNSIGNED-PAYLOAD may still sign it, but bodies read
outside the upload path never compared it with the body. Check it with
the payload hash once a V2 or V4 signature verifies: BadDigest on a
mismatch, InvalidDigest when it is malformed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: verify presigned uploads against a signed payload hash

A presigned V4 URL can sign X-Amz-Content-Sha256 as a query parameter,
but getRequestDataReader compared upload bodies only with the header of
that name, so the signed hash went unchecked. Verify uploads against the
hash the signature covered.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: check Content-MD5 on bodies buffered for their payload hash

IAM and other non-S3 requests signed without x-amz-content-sha256 have
their body buffered and hashed during signature verification. That
buffer counted as already bound, so their Content-MD5 was never checked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: check Content-MD5 on directory marker bodies

A PUT of a key ending in "/" stores its body on the directory entry
without going through putToFiler, so its Content-MD5 was never compared
with the body. Answer BadDigest on a mismatch, as putToFiler does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: wrap and unwrap signed bodies through helpers

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 10:17:09 -08:00
Chris LuandClaude Opus 5.5 a3b68aa37c shell: confirm trim survivors still serve the volume before deleting (#12590)
* shell: confirm trim survivors still serve the volume before deleting

volume.fix.replication compared the doomed replica's index with every
survivor and then deleted it unconditionally. A survivor that went away
while the indexes were read still let the delete through, leaving fewer
copies than the check had counted. Re-read each survivor's VolumeStatus
after the index checks and skip the trim if any no longer has the volume.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: test a trim whose survivor goes away after its index read

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 09:32:37 -08:00
Chris LuandClaude Opus 5.5 4cba4eb7bf s3: ignore unsigned object headers on POST Object (#12585)
* s3: ignore unsigned object headers on POST Object

The POST policy signs form fields only, but the upload read tagging,
user metadata, storage class, object lock and the stored content
headers from the request headers as well. Drop them before the form
fields are applied.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: test POST Object request headers against signed form fields

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 09:28:33 -08:00
Chris LuandClaude Opus 5.5 249f04e63a s3api: NoSuchKey for object lock calls on missing or deleted keys (#12592)
* s3api: answer PutObjectLegalHold on a missing key with NoSuchKey

In a versioned bucket a key with no latest version resolves to
ErrLatestVersionNotFound, which the legal hold handler did not map, so
the request failed with 500 InternalError. PutObjectRetention already
maps it to NoSuchKey.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: keep failed reads distinct when resolving an object lock target

getObjectEntryForUpdate turned every lookup error into a not-found
sentinel, so a failed filer read made PutObjectRetention and
PutObjectLegalHold answer NoSuchKey for an object that exists. Only a
genuine miss becomes the sentinel now, matching getObjectEntry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: answer object lock calls on a delete marker with NoSuchKey

PutObjectLegalHold and PutObjectRetention on a key whose current
version is a delete marker returned 200 and wrote the lock onto the
marker, and the matching GETs read it back. A key whose current version
is a delete marker does not exist, and delete markers take no lock, so
these calls now answer NoSuchKey like the tagging and ACL handlers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: let a lock recorded on a delete marker not block its removal

Delete markers are not WORM-protected, but a legal hold or retention
written onto one (possible before the previous change) made both the
gateway check and the routed delete's filer-side WORM guard refuse to
remove the marker, so the object could not be restored and lifecycle
could not clean the expired marker up.

Lock enforcement now ignores a lock on a delete marker, and the routed
version deletes in DeleteObject and lifecycle drop the WORM guard when
the version they read is a marker. A version id never changes kind, so
the marker read is the one deleted; the null version keeps its guard
because a PUT can replace it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 09:27:12 -08:00
Chris LuandClaude Opus 5.5 4ee82977af docker: pass entrypoint arguments to weed without splitting them (#12589)
The master, volume, volume-rust, server, mini, filer and s3 cases and the
fallback forwarded the user arguments as an unquoted $@, so the shell split
each one at whitespace and globbed it against /data. An -admin.password with
a space reached weed as two words, and weed ignored every flag after the
stray word. Quote "$@" as the other cases already do; $ARGS stays unquoted
since the script builds it from its own words.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 09:25:25 -08:00
Chris LuandClaude Opus 5.5 bef1b13b3d replication: keep a local sink path the source recreated after a delete (#12581)
* replication: let the local sink check a delete against the live source

LocalSink removed the destination path for any delete it was handed. A
delete that reaches the sink after the source recreated the path (a peer
filer's event delivered late, or a historical event replayed) removed the
newer file, and no later event brings it back.

DeleteEntryVersion receives the deleted entry. Before removing an existing
destination, the local sink looks up the source path and keeps the
destination when the source holds an entry of the same kind committed
after the deleted version (commit_ts_ns, or mtime for unstamped entries).
A live entry that is the deleted version itself or older means the source
store lags the event, so the delete proceeds. A failed lookup returns the
error so the event is retried instead of deleting a file that may be live.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.sync, filer.replicate: hand the deleted version to the sink

Pure deletes, renames without a native move and moves out of the
watched directory now go through sink.DeleteEntryVersion with the old
entry, so a sink that checks the live source (the local sink) can keep a
path the source recreated after the deleted version. Other sinks fall
back to DeleteEntry unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* replication: order the local sink check by the delete's mutation time

A source filer can hold a version written concurrently with the delete:
newer than the deleted entry but older than the delete itself. The
source's own replay applies the delete over it, so keeping the backup
copy because it outranks the deleted entry left a file the source no
longer has.

The sink now gets the delete's mutation time (MutationTsNs, else TsNs)
and keeps the destination only when the live entry was committed after
it, the order Filer.Replay keeps. That also covers a stamped replacement
of an unstamped entry whose mtime did not move. filer.replicate events
carry no time, so they keep the deleted entry's stamp as the bound, and
mtime only when nothing is stamped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* replication: count an unstamped live entry from its mtime

A source filer that predates commit stamps stores entries with no
CommitTsNs, so comparing the live stamp with the delete's time never
kept a replacement there. An unstamped live entry now counts from its
mtime against the delete's time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* replication: keep a stamped live entry when the deleted one is unstamped and the delete has no time

filer.replicate gets no delete time from the notification queue. When the
deleted entry has no commit stamp either, a stamped live entry was written
after it, so its mtime no longer decides.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 09:23:54 -08:00
Chris LuandClaude Opus 5.5 c2625ddf00 seaweed-volume: read security.toml values as TOML strings (#12582)
The line parser only trimmed quotes, so `key = "abc\n"` gave the Rust
volume server the bytes `abc\n` with a literal backslash while the Go
servers sign with a newline, and every JWT failed. Literal strings kept
their quotes and a trailing comment stayed part of the value.

Decode each value with the TOML parser; text it rejects keeps the old
quote trimming.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 09:19:18 -08:00
Chris LuandClaude Opus 5.5 b7d6b48921 operation: probe octet-stream uploads for gzip like untyped ones (#12584)
* operation: probe octet-stream uploads for gzip like untyped ones

The 128-byte gzip probe ran only when the caller passed no MimeType, so an
upload labelled application/octet-stream or binary/octet-stream was never
compressed. S3 PutObject and UploadPart pass the request Content-Type (or the
octet-stream that mimeDetect fills in), so every binary S3 upload skipped
compression that the same bytes get through the filer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* operation: check the probed upload keeps the caller's content type

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 09:16:55 -08:00
Chris LuandClaude Opus 5.5 02b9747bd8 wdclient: keep a shared volume lookup alive when its first caller leaves (#12579)
* wdclient: keep a shared volume lookup alive when its first caller leaves

Concurrent misses for the same volumes share one provider lookup, but it
ran on the first caller's context, so that caller's cancellation failed
every request waiting on it. The lookup now runs on a context no single
caller can cancel, and each caller stops waiting when its own context ends.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* wdclient: wait for the second caller to join before canceling the first

A fixed sleep did not prove the second caller had joined the shared lookup:
a late caller could hit the filled cache and pass without waiting. The test
now waits until the caller reads its own Done channel, which happens only
after DoChan registered it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 09:15:01 -08:00
Chris LuandClaude Opus 5.5 8581151376 filer: gzip metadata log pieces before uploading them (#12580)
* filer: let the fake-master test helper serve any master

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: gzip metadata log pieces before uploading them

Every metadata event carries the old and new entry with their whole chunk
lists, so a file grown by small fsynced appends logs the same chunk records
again and again: 2000 appends of 256 bytes through weed mount wrote 34.8 MB
of log. The uploader only compresses when the first 128 bytes compress, and
a log piece starts with fresh file ids and ETags, so it stored the log as is.

The filer now gzips each piece itself and keeps the compressed bytes when
they save at least 10%. After the same workload the volumes hold 3.05 MB
instead of 35.2 MB, file data included. Readers already honor the chunk's
IsCompressed flag on every path (filer-side decode, log file refs read
straight from volume servers), and the assign request sizes the bytes
actually stored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 09:13:50 -08:00
Chris LuandClaude Opus 5.5 e0f8438284 mount: retry a writeback metadata flush until the filer answers (#12578)
* mount: keep the filer status in the flush create error

flushMetadataToFiler formatted the CreateEntry error with %v, so callers
classifying it saw only the message, which includes the file path. Wrap
it so the gRPC status the filer sent stays reachable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: retry a writeback metadata flush until the filer answers

With -writebackCache, close() returns before the chunk references reach
the filer. The deferred flush gave up after four attempts, about a minute
once each attempt's own filer retries are counted, then released the
handle: a longer filer outage dropped the file even though write() and
close() had succeeded, and its uploaded chunks became orphans.

Keep retrying for as long as the filer is unreachable, as the writeback
chunk uploads already do. A filer that answers with an error still ends
the flush after the bounded attempts, and an unlink of the draining file
still ends it on the next attempt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: retry only an unanswered filer call without a bound

The unbounded phase used util.IsTransientError, which falls back to
matching the error text. A refusal the filer sends in its response
(read-only, name too long, a plain error string) has no gRPC status, and
the flush error carries the file path, so a file named like
"unavailable-transport.log" kept retrying against a filer that answered.

Retry without a bound only when the call failed with a gRPC Unavailable
status: the dial, the connection or the mount's call deadline gave out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 08:57:42 -08:00
Chris LuandClaude Opus 5.5 a6a3bdef8d s3api: deny signed reads of a version file through its physical path (#12574)
* s3api: deny signed reads that name a version file by its physical path

A stored version lives at <key>.versions/v_<id>, and that path is also a
valid object key. Anonymous reads already refuse it; signed GET, HEAD,
copy sources and object subresources did not, so a caller without
s3:GetObjectVersion, or under a Deny on the logical key, could read any
version through it. Apply the same check in the shared entry recheck.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: recheck a version file key before answering a conditional read

A conditional GET or HEAD answered 304 or 412 with the ETag before the
handler's entry recheck, so the physical version path still confirmed a
version's ETag to signed callers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: look up version file keys in the lookup-based policy recheck

Retention, legal hold, tagging writes and deletes only fetched the entry
for the recheck when the bucket had a policy, so without one they still
acted on a version file named by its physical path. Fetch it whenever
the key has the version file shape.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: refuse a directory-marker delete that names a version file

A trailing slash kept "<key>.versions/v_<id>/" out of the version file
shape check, and the lookup recheck misses the entry for that form, so a
DELETE reached deleteDirectoryMarker and removed a chunkless version or
delete marker without s3:DeleteObjectVersion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 08:21:31 -08:00
Chris LuandClaude Opus 5.5 317d907334 filer: route tests forward to a peer port the test holds (#12576)
The ring tests used 127.0.0.1:18889 as an unreachable peer, but its gRPC
port 28889 is a common filer port: on a shared CI host a live filer there
accepted the forwarded exclusive create, and
TestCreateEntryExclusiveLeavesNonOwner failed. The peer now listens on a
port the test owns and drops every connection.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 19:10:58 -08:00
github-actions[bot] d51a79335c docs: regenerate star history chart 2026-11-12 00:47:35 +00:00
Chris LuandClaude Opus 5.5 13334a59b9 vector: reuse the Lance session, version leases and opened tables across reads (#12573)
* vector: share one Lance session across reads

Each request opened its table in a session of its own, so every query
loaded and decoded the index again. Get, list and query now open in one
session the server keeps, sized by -index-cache-mb and
-metadata-cache-mb; writes keep a session of their own.

Lance keys some of its cached entries, such as a version's index list,
by location and version alone, and an index dropped and created again at
its location numbers its versions from 1 again: a shared session answered
a query on the new table with the old table's index. CreateIndex now
marks each table with an id in its schema metadata, and the session
starts over when a location turns up holding another table.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: share a version lease among the reads of that version

Every get, list and query tagged the version it read and deleted the tag
afterwards: two filer writes and a manifest read per request. Reads of
one version now share one tag, which goes 10 s after the last read took
it; a tag is reused for at most 5 minutes, so a busy version never
reaches the reapers' TTLs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: keep opened tables for the next read

A read built the table's store and read its latest manifest each time.
Reads now keep the opened table by location and encryption, up to 1024,
and move it to the latest version, which lists the versions and takes
the manifest from the session. A handle opened in an earlier session is
not reused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: key shared leases by manifest, and take one per version at once

A table dropped and created again within the linger reused, for a
version number the old table had, a lease whose tag the drop had
deleted. Leases are now keyed by the manifest's ETag as well. Reads that
arrive before a version has a lease wait for the first to take it,
rather than each writing a tag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: drop kept tables when the shared session starts over

A kept table holds the session it was opened in, so a replaced session
and its caches lived on in the handles of other tables until they were
evicted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: prune shared lease slots only when they double

Every lease lookup scanned all slots under the map's lock, a cost that
grew with the versions read in the last linger.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: shrink the lease slot map after a prune

After a burst the map kept its peak allocation, which every later prune
scanned.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: keep a lease slot a read has taken through a prune

A read that took a slot and had not yet locked it could see the slot
pruned, and then pin through it while the next read pinned through a
new slot, writing a second tag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 16:26:50 -08:00
Chris LuandClaude Opus 5.5 b924ffabf1 admin: calm console restyle with light and dark themes (#12568)
* admin: restyle the UI as a calm console

Grey canvas with white sheets, a docked sidebar that marks the current
page with an indigo tint, ink primary buttons, tinted status pills,
dense tables with tabular numbers and spectrum capacity meters. The
styles are design tokens layered over Bootstrap's own variables, with
light and dark values, so the page templates stay unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: add a light, dark and automatic theme switch

The choice is kept in localStorage and applied before the page paints,
on the sign-in page too; automatic follows the system setting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: drop hard-coded colors from page templates

The volume id link, utilization track and plugin form borders used
fixed light-theme colors; sparklines now use the data palette.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: theme the plugin job detail panel background

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: keep the selected radio dot visible in dark mode

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: cycle the theme from memory when storage is blocked

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: keep success progress bars green

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: keep keyboard focus visible on the toolbar togglers

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: use contrasting text on danger and success buttons in dark mode

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: lead the dashboard with cluster health and one row of tiles

The header now says in one line whether the cluster is healthy and how
many masters, volume servers and filers report. The two rows of stat
tiles become one: the volume size limit and EC volume count move into
footnotes, the mount client card joins the cluster counts, and the
counts link to their pages instead of sitting in nested cards. The
duplicate version list and the bucket shortcut in the header are gone,
and the section menus become plain View all links.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: export the user columns the table actually has

The users table has username, email and access key columns, but the
export waited for six cells per row and so always produced only the
header line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: drop table titles that repeat the page title

List pages titled their only table with the page title again. The
table now sits right under the summary tiles, and the export actions
that were hidden in the table header menu move to the page header as
an Export button, matching the cluster pages. The policies menu linked
to nothing and is removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: drop clock and duplicate summary tiles

Buckets, users, policies and service accounts spent a summary tile on
the refresh time, which the page footer already shows (policies gains
that footer). The users page also showed its user count twice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: show unset bucket settings as a dash

Rows of No owner, No quota and Not configured made the bucket table
hard to scan. Unset cells now show a quiet dash that keeps the wording
as a tooltip and for screen readers, and the name and creation date no
longer wrap (the full timestamp is in the date's tooltip).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: export buckets through the page's API exporter

admin.js loads after the page, so its copy of exportBucketList replaced
the buckets page's own one. That copy scraped the visible rows, so it
exported only the current page, without CSV quoting, and picked up the
display text of the cells. The page's exporter reads every bucket from
the API with full timestamps and escaped fields.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: escape quotes in the user export

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: download the bucket export as a Blob

The exporter put the CSV into a data URL with encodeURI, which leaves #
alone, so a bucket owner containing # ended the download there. Use the
shared downloadCSV helper instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: count masters as configured, not reporting

The master list comes from raft membership, which keeps a follower
that is down, so it does not show which masters are reporting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: link the Prometheus metrics endpoint from the sidebar

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: drop the sidebar metrics link

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 15:32:48 -08:00
Chris LuandClaude Opus 5.5 1618d613a3 s3: resolve a role session's aws:username from its user, not its session name (#12572)
* s3api: record the caller on AssumeRole sessions

The session name is the caller's choice; the session now carries the
caller itself as the user it was issued to, as the web identity, LDAP
and credential paths already do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: take a role session's aws:username from its user, not its name

An assumed-role ARN ends in the session name the caller picked, so
naming a session after another user resolved ${aws:username} and
${aws:userid} to that user in bucket and attached policies. Resolve
them from the user the session was issued to instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* iam: take a role session's aws:username from its user, not its name

IsActionAllowed set aws:username to the last segment of an assumed-role
ARN, the caller-chosen session name, for role and bucket policies. Use
the user the session was issued to, and drop a forwarded value when the
session names none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: pass a role session caller's user on to the session it assumes

A role session that calls AssumeRole is named by its session ID, so the
new session recorded that ID as its user. An STS session caller now
passes on its own recorded user, or none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: size the session claims map from the request context

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 15:32:13 -08:00
Chris LuandClaude Opus 5.5 3f557b73c8 admin: show the right master ports when -port.grpc is not port+10000 (#12570)
* admin: take master HTTP ports from raft member names

The masters page and dashboard turned each raft member's gRPC address
into an HTTP address by subtracting 10000 from the port, so masters with
a custom -port.grpc showed wrong or negative ports. Raft members are
named after their master address, which carries the HTTP port; use it
and keep the old derivation only for names that are not addresses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: test raft member address mapping

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 15:22:29 -08:00
Chris LuandClaude Opus 5.5 a0f95e16bc s3api: fix TestRoutedMultipartFinalize flake from a reused fake filer port (#12571)
* s3api: drop cached gRPC channels when a fake filer takes a port

A test that dials a dead owner leaves its failed channel in the
process-wide gRPC connection cache. When a later test's fake filer is
given the same port, its first RPC fails at once with the cached
connection refused, which is how TestRoutedMultipartFinalize failed on
the 32-bit job. The fake filer helpers now share fakeFilerAddress, which
invalidates any cached channel for the new address.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: test a fake filer on a dead owner's reused port

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: drop cached channels for dead test filer addresses too

closedFilerAddress can be handed a port whose cached channel belonged to
a fake filer that just stopped and may still look ready; then the dead
owner's transaction is sent on it instead of failing over. Build its
address with fakeFilerAddress as well.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 14:50:16 -08:00
Chris LuandClaude Opus 5.5 6c1eec8703 s3: store the POST Object acl on the uploaded object (#12567)
* s3api: split upload ACL assembly out of preparePutObjectACL

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: store POST Object acl and grant fields on the object

The form acl and x-amz-grant-* fields were authorized but never assembled,
so the object kept no ACL: acl=private on a public-read bucket stayed
anonymously readable and invalid values were accepted. Build them with the
same rules as PutObject, which also covers the Block Public Access check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: test POST Object stores its upload ACL

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: take the POST Object ACL from the signed form only

ACL request headers are not covered by the POST policy, so a holder of a
signed form could add one and have it stored with the upload.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: reject a POST acl field the policy does not cover

The acl form field now sets the stored object ACL, so like the x-amz-*
fields it needs a policy condition, as AWS requires.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: test unsigned ACL headers and uncovered acl fields on POST

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: evaluate POST upload policies on the canonical ACL

Policy conditions on the grant headers saw the raw form value, so an
equivalent spelling of a denied grantee passed. Use the PutObject request
shape that the PUT path evaluates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: check bucket policy for POST ACLs with auth disabled

PutObject applies the bucket policy to an explicit upload ACL when IAM is
off; POST now does the same before storing it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: reject differing acl and x-amz-acl POST fields

Both promote to the canned ACL header, so map order picked the stored ACL.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: test POST grant canonicalization, auth-off ACL policy, acl conflicts

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: check the auth-off POST ACL policy after loading the bucket

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: reject an empty acl or x-amz-acl POST field beside a differing one

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: test the auth-off POST ACL policy on a cold bucket cache and empty acl aliases

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: load the bucket before per-key object authorization

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: test the POST ACL policy on a cold bucket cache with auth on

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: stop per-key object authorization when the bucket fails to load

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: test the cold-bucket POST ACL allow path and a failed bucket load

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 10:36:51 -08:00
Chris LuandClaude Opus 5.5 eb159f3181 lance: deregister and re-register need UpdateTable (#12552)
* lance: require UpdateTable to deregister a table

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance: require UpdateTable to register over an existing table

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance: test that GetTable alone cannot deregister or re-register a table

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance: a register that finds a table the caller cannot read needs UpdateTable to bring it back

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance: decide declare write access by whether the table existed, not by what the caller can read

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance: test register and declare by a caller that cannot read the table

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance: register clears the marker on the create branch only after checking it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance: keep a register whose table was created when the marker check fails

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 10:35:00 -08:00
Chris LuandClaude Opus 5.5 4cb52d7705 filer: reopen a leveldb store wedged by a transient write error (#12565)
* filer/leveldb_handle: reopen a database whose journal latched a write error

goleveldb keeps the first journal write error and returns it for every
later write until the database is reopened, so one transient ENOSPC or
EIO fails all writes until a restart. The handle offers the leveldb.DB
methods the stores use; a write that fails on an OS error probes the
storage and, once it accepts writes and no call holds the database,
closes and reopens it and retries the write once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: open the leveldb stores through leveldb_handle

leveldb, leveldb2 and leveldb3 now recover from a transient storage
write error without a restart.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: leveldb stores return iterator errors from listings and deletes

A listing on a database that cannot be read reported an empty
directory, and DeleteFolderChildren wrote an empty batch, because the
iterator error was never checked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer/leveldb_handle: test a reopen that fails after the probe passes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer/leveldb_handle: probe storage before draining readers

A probe that stalls in Create, Write or Sync held mu exclusively and
blocked every read on the handle. Reopen attempts are now serialized by
their own mutex, and the probe runs before the drain.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer/leveldb_handle: hold new calls back while a reopen drains

TryLock succeeds only at an instant with no reader, so overlapping
listings could keep a reopen from ever draining while each write spun
for the full drain timeout. The drain now holds new calls back for at
most drainTimeout; a call nested inside a running one waits that out
instead of deadlocking. A drain that times out holds off the next
attempt for busyReopenInterval, so new calls are not held back over and
over.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer/leveldb_handle: bound the wait for a running reopen

A write that hit the latched error waited on the reopen mutex for as long
as another caller's storage probe ran, so a stalled probe queued every
writer. Callers now wait at most reopenWait and return their error, and
still retry once a concurrent reopen succeeds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer/leveldb_handle: tests wait for the drain and readers explicitly

The nested-read test polled for a drain that could start and end between
polls and then spun forever; it now syncs on the probe, widens the drain
and fails if the put returns first. The overlapping-readers test waits
until every reader holds an iterator before it writes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 10:23:19 -08:00
Chris LuandClaude Opus 5.5 5150168368 s3: authorize an empty versionId as the current-object action (#12566)
* s3: authorize an empty versionId as the current-object action

GET, HEAD and DELETE treat an empty versionId as the current object, but
the action resolver keyed on the parameter's presence and authorized it
as GetObjectVersion/DeleteObjectVersion, so a Deny on GetObject or
DeleteObject did not apply.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: test an empty versionId against a version-only policy

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: cover a bare versionId on HEAD

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 10:21:28 -08:00
Chris LuandClaude Opus 5.5 0c6c6868de volume: refuse a tier upload whose .dat was written during the copy (#12562)
* volume: refuse a tier upload whose .dat grew during the copy

AttachRemoteDatFile only checked that the data backend was still the one
copied, so a needle write acknowledged after the copy read the .dat was
missing from the remote object, and the upload then switched reads to it
and removed the local .dat. Compare the .dat size with the copy under the
same data-file locks; on a mismatch the handler deletes the copy and keeps
the local file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: refuse a tier upload whose .dat grew during the copy

Mirror the Go AttachRemoteDatFile check. The tier-up attach took the
write lease only inside each step, so a needle write acknowledged after
the upload read the .dat was missing from the object the volume then
switched to, and the local .dat holding it was removed. Hold one write
lease across the instance, revision and size checks, the remote-file
update, the .vif save and the backend swap, as tier-down does; a size
mismatch aborts, deleting the object and keeping the local .dat.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: compare the physical .dat size before a tier attach

DiskFile pads its size past an unaligned tail, which a read-only load
keeps, while the upload copies the physical bytes. Stat the file itself
so such a volume is not refused as written during the copy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 10:19:06 -08:00
dc03c3b1f7 shell: parallel moves and a per-run limit for volumeServer.evacuate (#12561)
* shell: parallel moves and a per-run limit for volumeServer.evacuate

-maxParallelization runs up to N EC shard moves at once (default 1), with the
ec.balance scheduler: never two shards of one volume at a time, free workers
go to the least busy targets. The moves are planned first with the same target
choice as before, then applied through volume_move.Mover.MoveEcShards, which
verifies the target registered a shard before deleting it on the source.

-movesPerExec limits how many volumes and EC shards one run moves (default 0
for unlimited), like -volumesPerExec in volume.balance, so a big server can be
emptied in several runs.

With -skipNonMoveable a shard that fails to move (a read error on a failing
disk, for example) no longer drops the other shards of its volume; they are
moved and the failures are listed at the end. Without it the run stops at the
first failure as before. Losing the admin lock stops the run in either mode.

Refs #12501

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: plan evacuation again after failed moves; fail on a lost lock

A failed shard move kept the target slot it was planned on, so a readable
shard that found no destination in the plan stayed on the server although
the room was never used. Failed moves now give their targets back to the
model and the shards without a destination are planned again, up to
maxEcEvacuationRounds times; the failed shards are not tried again.

With -skipNonMoveable a lost admin lock was printed as a skipped failure and
the run returned success. ec.ErrLockLost is now exported and always fails
the run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: volumeServer.evacuate reports -movesPerExec after an applied EC run

The round loop only checked the limit after a failed move, so an applied run
that stopped at -movesPerExec without failures never said to run it again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: volumeServer.evacuate plans again for as long as failed moves leave room

The ten-round cap left readable shards on the server once more than ten
rounds of unreadable shards had taken the free room first, and a rerun
fails the same shards first again. Every round moves or fails each shard it
plans and a failed shard is not planned again, so the rounds end without it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: volumeServer.evacuate gives a failed move back every slot it took

When the target already held the shard in another disk type, adding it to
the planned disk type still took a slot, but the undo gave back only the
explicit one. Record the slots the plan took and restore exactly those.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec: an evacuation move that landed on its target keeps the target slot

A move can fail after the target registered the shard, when only the source
unmount or delete failed. Evacuate then gave the target's slot back and could
plan another shard onto a full server. The runner now checks the target after
a failed move and leaves a shard the target has out of FailedMoves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: volumeServer.evacuate gives back the slot of a failed move onto a shard the target had

A target that held the shard before the plan reports it after any failed
move, so the runner took such a move for one that landed and evacuate kept
the slot the plan had taken. The runner now lists those moves in
LandedMoves, and evacuate undoes the ones whose target had the shard, then
plans again with the room they gave back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-11 10:18:51 -08:00
Chris LuandClaude Opus 5.5 fcfca3fee3 s3, filer: bound the wait for an upload slot (#12559)
* util: add WaitUntil, a cond wait bounded by a context and a timeout

The caller holds the cond lock. Cancellation and the timeout wake the
waiter through a broadcast taken under the same lock, so no wakeup is lost.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: bound the wait for an upload slot

A PUT that arrived while the concurrent upload limits were exhausted waited
on the condition variable with no bound. The handler reads nothing from the
body while it waits, so a client that gave up and reset its connection went
unnoticed: the socket and goroutine stayed pinned until a slot freed, and a
stalled volume server grew the gateway's fd count by one per aborted PUT.

The wait now ends on InflightUploadDataTimeout with 503 SlowDown, which S3
SDKs retry, and on request cancellation. An admitted upload takes its slot
under the lock, and a finished upload broadcasts, so freed bytes admit every
waiter that fits instead of one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: bound the wait for an upload slot

The filer's PUT/POST upload limits queued requests the same unbounded way.
Reuse the bounded wait: reject with 429 after InflightUploadDataTimeout, as
the volume server does, and stop on request cancellation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* command: add -inflightUploadDataTimeout for the S3 gateway and the filer

weed s3 -inflightUploadDataTimeout, weed filer -inflightUploadDataTimeout and
-s3.inflightUploadDataTimeout, weed server and weed mini
-filer.inflightUploadDataTimeout and -s3.inflightUploadDataTimeout. The
default of 60s matches the volume server's flag of the same name; 0 keeps
the old unbounded wait.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: reject a queued upload without reading its body

WriteErrorResponse drains the request body before answering. Behind the
stats recorder the drain has no read deadline, and reading the body sends an
Expect: 100-continue client the go-ahead to upload what is about to be
discarded. Answer the rejection with an empty-body copy of the request so
the drain is skipped; net/http closes the connection when a large unread
body remains.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3, filer: release an upload slot under the cond lock

A waiter checks the counters under the cond lock and then waits. A release
that decremented and broadcast without the lock could land between the two,
so the waiter slept with a slot free until the next release or its timeout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3, filer: close the HTTP/1 connection when rejecting a queued upload

net/http reads up to 256 KiB of an unread request body before sending a
keep-alive reply, so a client that stalls its body held back the 503 or
429 until the connection idle timeout. Connection: close skips that read.
HTTP/2 is left alone: there it would shut down the whole connection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 10:17:18 -08:00
Chris LuandClaude Opus 5.5 d659228338 shell: s3.clean.uploads frees an open upload under the object write lock (#12558)
* shell: s3.clean.uploads deletes uploads over filer gRPC

Same DeleteEntry the HTTP handler runs, without minting a filer JWT,
and the gRPC path is the one a locked, routed delete needs next.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: s3.clean.uploads frees an open upload under the object write lock

The cleaner deleted a stale upload recursively with no lock, so a
CompleteMultipartUpload could pass its upload-exists check after the
part entries and their chunks were gone, and publish an object over
freed chunks. Do what AbortMultipartUpload does: hold the object's
distributed write lock, and free an open upload in an ObjectTransaction
routed to the object's owner filer, conditioned on the object not
carrying the upload; if it does, delete the metadata only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: s3.clean.uploads keeps the object write lock for the whole cleanup

A large upload's recursive delete can outlast one 15 second lease, and
a completion on the distributed-lock path could then take the lock and
commit over chunks being freed. Renew the lock in the background and
skip the delete once it is lost.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: s3.clean.uploads locks the path a completion commits under

For a trailing-slash key a non-versioned completion writes and locks the
entry inside the directory the key names, not the object path, so the
cleaner's transaction did not serialize with it. Lock and check the
same path the completion uses for the bucket's versioning state; the
route key stays the object's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: abort locks the path a completion commits under

A routed abort locked and checked the object path, but for a
trailing-slash key a non-versioned completion commits under the entry
inside the directory the key names. Use the same path the completion
locks for the bucket's versioning state.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: end s3.clean.uploads bucket failure line with a newline

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* cluster: NewBlockingLongLivedLockWithin bounds the wait on a held lock

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: s3.clean.uploads skips an upload whose object lock stays held

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* cluster: share the contended lock filer setup across lock tests

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: s3.clean.uploads reads the bucket's versioning state per upload

The commit path a routed completion locks depends on versioning for a
trailing-slash key. Reading it once per bucket before the run left a
window as long as the run; read it just before the delete, as the
gateway's abort does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 10:16:20 -08:00
Chris LuandClaude Opus 5.5 b11af30cb1 mount: remove the local control socket when the mount exits (#12560)
* mount: remove the local control socket when the mount exits

weed mount listened on /tmp/seaweedfs-mount-<hash>.sock and removed it only
when a later mount of the same directory started, so every unmount or
SIGTERM left one socket file behind. Remove it on the normal return path and
in the interrupt hook, unless a later mount of the same directory has
already replaced it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: remove the control socket before a failed mount exits

glog.Fatalf and glog.Exitf call os.Exit, which skips the deferred removal,
so a mount that failed after creating its socket (mount point not
available, fusermount failure) still left it behind.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: register the socket cleanup hook before mount setup

A SIGTERM that arrived while the mount was still setting up, for example
waiting on the filer to create the mount root, exited through grace without
running the deferred cleanup. Registering the hook right after the listener
is created makes it run last, after the unmount and filesystem hooks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 10:11:58 -08:00
Chris LuandClaude Opus 5.5 7f18576798 filer/etcd: serve volume.fsck from one etcd revision (#12555)
* filer/etcd: scan entries from one revision

volume.fsck walked etcd one directory at a time, so a file moved from a
directory not yet listed into one already listed was seen under neither
name and its chunks were reported as orphans. The etcd store now
implements filer.EntryScanner: every page is read at the revision of the
first read, and a compacted revision fails the scan instead of falling
back to the walk.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer/etcd: read hard links at the scan revision

Hard-link records resolved during a scan were read at the current
revision, so a link updated meanwhile mixed two instants. Reads made with
the scan's context now use its revision.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: list etcd among the stores volume.fsck reads from a snapshot

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer/etcd: skip hard-link records in a scan

Hard-link records live beside the entries under their random ids, so a
root scan could read one as an entry with a path made of random bytes,
which the traversal stream cannot send. Skip hard-link ids and keys whose
directory is not valid UTF-8.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer/etcd: recognize a hard-link record by the id it stores

A hard-link id can be valid UTF-8, so the key shape alone let some
records through a root scan as phantom files. The record's value is the
shared entry, whose HardLinkId is the record's own key; no entry key
can equal its own hard-link id.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer/etcd: scan entries under an invalid-UTF-8 directory

The filer sanitizes names but stores directories as given, so a path
written over HTTP can leave entries the listing never reaches. Skipping
them let volume.fsck count their chunks as orphans. Send them under the
sanitized directory instead, the substitution the filer uses for names.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer/etcd: fail a scan whose sanitized path names another entry

An entry under an invalid-UTF-8 directory is reported under the sanitized
directory. When that path is also another stored entry at the scan revision,
or another sanitized entry in the same scan, acting on the reported path
(volume.fsck deleting it) would reach the other file, so the scan fails and
names the stored path instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 10:11:21 -08:00
e23a952878 ec: don't index ev.Shards[0] when the last shard was unmounted (#12538)
* ec: don't index ev.Shards[0] when the last shard was unmounted

* ec: skip a runtime emptied before the scrub merge reads its shards

FindAllEcVolumes releases each location lock before MergeEcRuntimes runs,
so an unmount can leave a merged runtime with no shards and the width
scan indexed Shards[-1].

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec: treat a scrub merge with no shard left as a vanished volume

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec: read Shards once in ShardSize so a concurrent unmount cannot empty it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec: read each runtime's Shards once in the scrub merge

The vanished check and the slot construction read Shards separately, so
an unmount between them produced a non-nil view with no shards.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec: anchor the scrub merge on a generation that still holds shards

An unmount can empty the newest runtime while an older one keeps its
shards, which left a merged view anchored on the emptied generation with
no slots and the holder reported as skipped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Mansur Akhmadi <10739460+mansur-akhmadi@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 09:57:39 -08:00
Chris LuandClaude Opus 5.5 641ab6204b shell: fs.mergeVolumes leaves in-progress multipart parts in place (#12556)
* shell: fs.mergeVolumes leaves in-progress multipart parts in place

CompleteMultipartUpload reads the part entries and later commits the
object with their fids, guarded only by the upload directory. A merge
that moved a part chunk in between deleted the source needle the new
object pointed at. Parts stay where they are until the upload completes
or is cleaned, and their chunks still count as referenced sources.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: note in fs.mergeVolumes help that upload parts stay in place

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: match upload parts when the buckets folder is the root

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: count chunks behind a part's manifest as referenced sources

A skipped part whose chunks on a merge source sit behind a manifest was
not counted, so the merge printed no skip line and warned that nothing
references the source.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: keep a part's direct source references when its manifest is unreadable

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: count chunks resolved before an unreadable part manifest

ResolveChunkManifest returns the data chunks of earlier manifests along
with a later manifest's error; record them before returning so the
unreferenced-source warning does not fire for a source they reference.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: test the partial manifest result without a timing delay

The unreadable manifest is now nested under a readable part manifest, so
the first level resolves fully before the failing lookup runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 09:55:03 -08:00
Chris LuandClaude Opus 5.5 e9606c8ecc filer.meta.backup: resume an interrupted metadata copy and add -entriesPerSecond (#12549)
* filer_pb: add a depth-first traversal that can resume after a path

TraverseDfs walks a tree in listing order and reports each finished
path, so a caller can save one path and continue from it later. Pages
are read whole before visiting, and the entries of a page are visited
by up to five workers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.meta.backup: resume an interrupted metadata copy

The walk saves the last finished path every few seconds, and a walk
that continues an unfinished one starts after it. A fresh walk clears
the saved path first, and a finished walk clears it with the walk
start.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.meta.backup: add -entriesPerSecond to throttle the metadata copy

Each entry the full copy reads waits on a rate limiter, and pages are
listed only after the previous page is copied, so the flag also bounds
how fast the copy lists the source filer. 0 keeps it unlimited.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer_pb: report each file of a depth-first walk once all before it are done

A throttled walk only reported progress after a whole listing page, so a
slow copy could go many minutes without a resumable path. Visit entries in
runs that end at a directory and report each file as soon as every entry
before it is done.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 09:08:11 -08:00
Chris LuandClaude Opus 5.5 0ef0260844 volume: fail a vacuum step that finds the volume already compacting (#12553)
* volume: fail a vacuum step that finds a compaction already running

Compact, CompactByIndex and CommitCompact returned success when another
compaction held the volume. A second, uncoordinated vacuum (an expired
admin job still running, a shell volume.vacuum next to the worker) then
went on to commit or clean up the first run's .cpd/.cpx. Return a
"volume N is compacting" error instead, and refuse cleanup while a copy
is in flight, as the Rust volume server already did.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: check the compaction files before a commit closes the volume

CommitCompact closed the needle map and data file and wrote the .cpc
marker before it found .cpd/.cpx missing. A duplicate commit, such as
the second of two overlapping vacuums, then left the volume registered
with no open files: it dropped out of the heartbeat and looked lost to
volume.list and volume.fix.replication until a restart. Check for the
files first so the refused commit leaves the volume as it was.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: run makeupDiff before a commit closes the volume

CommitCompact closed the needle map and data file before makeupDiff, so
a makeupDiff failure (a compaction revision another commit moved, an I/O
error) returned with them closed: the volume stayed registered but
missing from the heartbeat until a restart. makeupDiff reads the live
files through its own handles while the commit holds the write lock, so
close them only once it has succeeded, as the Rust volume server does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: log a volume the heartbeat skips because its files are closed

A volume left without its needle map or data file vanished from the
heartbeat with no log line, so the master saw a replica disappear while
its files were still on disk. Warn on each skip, as the deleted-.dat
case next to it already does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: claim the compaction flag while cleanup removes the temp files

Cleanup only read the flag, so a compaction could start between the
check and the removal and lose its .cpd/.cpx. Claim it for the cleanup
instead, in Go and Rust.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 08:48:58 -08:00
Chris LuandClaude Opus 5.5 a4a68c885a filer.sync: move the watermark over jobs that finished out of order (#12554)
* filer.sync: move the watermark over jobs that finished ahead of an older one

The watermark only took the finishing job's timestamp when it was the
oldest in flight, so jobs that completed before an older one were never
recorded and a quiet stream kept the offset behind finished events.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.sync: keep one finished timestamp per gap behind a running job

A long job kept every later completion in the done heap. Only the newest
one between two running or failed timestamps can become the watermark.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.sync: test done-heap compaction across several running and failed jobs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 08:47:30 -08:00
000baf8dfa s3: return an error for GETs canceled before the first byte (#12502)
* s3: answer GETs canceled before the first byte

A stream that fails before writing anything now always gets an S3 error
response, canceled errors included. The idle timeout and a shared lookup
canceled by another request both cancel the request context while the
client is still connected, and returning silently sent that client a
200 with Content-Length: 0. A client that has gone away never sees the
write.

Refs #12500

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: mark inline GET write errors as written; bound the idle-cancel test

An inline object's write error comes after the status line, so the
caller must not write an S3 error on top of it. The idle-cancel test
caps the client wait and gives the response write its own deadline,
keeping only the read side on the 200 ms idle timeout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: test a GET whose request is canceled before the first byte

Drives GetObjectHandler through a chunk lookup that cancels the request
context, so a handler that returns before answering shows up as the empty
200 again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: check the cancel test reached the chunk lookup

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-11 08:45:20 -08:00
Chris LuandClaude Opus 5.5 ea6e9d670a volume: forward no Content-MD5 for a JPEG the orientation fix rewrote (#12557)
* volume: forward no Content-MD5 for a JPEG the orientation fix rewrote

With -images.fix.orientation the primary stores and forwards the rotated
bytes, but sent replicas the client's digest of the original upload. The
replica checks Content-MD5 against what it receives and rejects the write,
so replicated JPEG uploads with a digest failed. The Rust volume server
already drops the digest once the body is rewritten.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: test the forwarded Content-MD5 through PostHandler and a replica

The test now posts each upload to a primary whose fake master names a
real replica volume server, so it covers the JPEG gate, ReplicatedWrite
and the replica's own Content-MD5 check, and reads the stored bytes back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: fail the replica forwarding test instead of hanging when nothing is forwarded

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: compare the primary and replica bytes in the replica forwarding test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 08:44:06 -08:00
1202c5ba08 filer.copy: compute chunk md5 so files get a real ETag (#12497)
* filer.copy: compute chunk md5 so files get a real ETag

Chunks uploaded by filer.copy had no ETag, so every multi-chunk file got
the md5 of the empty string plus -N. An overwritten file kept its old ETag
and a revalidating client got 304 for changed content.

* filer.copy: store chunks in offset order

Chunks are collected in upload completion order, and the entry ETag
hashes the chunk ETags in stored order. Identical files copied with
concurrent chunk uploads got different ETags, and a file with swapped
equal-sized chunks could keep the old one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.copy: test chunk order and chunk md5 on chunked uploads

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 08:43:23 -08:00
b9866bae4f weed/shell: support -noLock flag in volumeServer.evacuate (#12501) (#12516)
* weed/shell: support -noLock flag in volumeServer.evacuate (#12501)

* shell: return flag parse errors from volumeServer.evacuate

A typo such as -noLock=invalid printed the parse error but the command
still succeeded, so a piped script treated the evacuation as done.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: volumeServer.evacuate -h prints usage without failing

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 21:25:42 -08:00
09a63031c1 helm: set replication and disk on buckets created by the chart (#12543)
* helm: set replication and disk on buckets created by the chart

* helm: test bucket replication and disk without ttl

* helm: reject unquoted bucket replication values

* helm: treat a null bucket replication as unset

A blank replication key in a values file failed the render with the
quoted-string error, while blank ttl and disk are skipped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* helm: require a three-digit bucket replication

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* helm: note that removing bucket settings keeps the stored rule

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* helm: reject bucket replication above 255

fs.configure rejects an xyz value above 255, so the hook Job failed after
the release rendered.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 21:24:55 -08:00
f068fe7f09 fix(shell): fs.log.purge purges every filer, not just one (#12524)
* fix(shell): fs.log.purge purges every filer, not just one

Each filer keeps its own metadata logs in its own store, so purging a
single filer leaves the chunks referenced by the others and the volume
vacuum cannot reclaim anything.

The shell picked that single filer at random (commands.go:208) when no
-filer flag was given, and the admin script worker pinned it to
filers[0] (admin_script_handler.go:574), so the same command purged a
different filer on every run and never the whole cluster.

fs.log.purge now resolves the filer list from the master and purges each
of them, continuing when one is unreachable and reporting the failures
at the end. -filer=<addr>,... restricts the purge to a subset.

Also stops mutating the shared command environment per filer, so a
long-lived shell keeps its original filer.

* fix(shell): bound filer discovery and cover the cluster path

Address review feedback:

- resolveLogPurgeFilers waited on the master with context.Background(): a
  shell whose master connection dropped would hang before purging
  anything, where it used to purge the connected filer. Bound the master
  round trip with a timeout and fall back to the connected filer when the
  master cannot be reached or lists no filer.
- the tests only exercised the single-filer fallback. Rename that test to
  what it actually covers and add one asserting every filer of the list is
  attempted and the shared command environment keeps its original filer.

* shell: reject an fs.log.purge -filer list with no addresses

A comma or whitespace only -filer value fell through to master
discovery and purged every filer instead of the intended subset.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: fall back to the shell's filer without a second master lookup

When the master listed no filer, FilerAddress() ran filer discovery again
and could wait out the shell's ConnectTimeout before failing. Also cover
master discovery with the fake discovery master.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: test the fs.log.purge filer loop against a fake filer

The previous test looped over the addresses itself, so it passed no
matter what Do did.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Martin Morez <martin@morez.org>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 21:24:04 -08:00
Chris LuandClaude Opus 4.8 fb5ffb815f filer: validate bucket names before the SQL DDL hooks (#12547)
* filer: validate bucket names before the SQL DDL hooks

The SQL store builds CREATE TABLE and DROP TABLE statements by formatting
the bucket name into the query. The DML paths already reject names that
fail VerifyS3BucketName, but the bucket-creation and bucket-deletion
hooks ran the DDL without that check, so a name reaching them through the
filer or WebDAV directory path (which skips the check for dot-prefixed
directories) could inject stacked statements. Apply the same validation
the DML paths use.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* filer: check that a valid bucket runs both DDL statements

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-10 21:22:31 -08:00
Chris LuandClaude Opus 5.5 d350a0ad03 weed-db: end startup when password authentication fails (#12550)
* weed-db: end startup when password authentication fails

The failure branches returned the result of writing the error response,
which is nil on success, so the handshake carried on to ReadyForQuery.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-db: test startup with wrong and unknown credentials

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-db: check the auth error code and fail on stray read errors in the startup test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 21:21:05 -08:00
Chris LuandClaude Opus 5.5 f3150c3495 admin: escape file properties before rendering the modal (#12548)
* admin: escape file properties before rendering the modal

The Properties modal concatenated the entry name, path, MIME type and
extended attributes into HTML, so object metadata written over S3 could
inject markup into the admin page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: keep every property key when escaping

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 21:19:53 -08:00
Chris LuandClaude Opus 5.5 8d96985131 security: scoped filer tokens are not admin credentials (#12551)
* security: do not accept a path- or method-scoped filer token as admin

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* security: test that scoped filer tokens fail the admin token check

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 21:18:59 -08:00
2143761319 fix(s3tables): let a table renamed across namespaces keep committing (#12515)
* s3tables: let a table renamed across namespaces keep committing

* s3tables: keep the same-directory exception off a directory another table shares

A record written before metadata locations were confined to their namespace
(6f93f874) and claimed (#12473) can keep its files in another table's
directory, in another namespace. The exception that lets a table renamed
across namespaces keep committing to the directory it left behind accepted
such a record's commits too, where master refused them.

The namespace check now runs first, as on master. Only a location it refuses
is matched against the record's current directory, and the exception holds
only when no other record keeps its files there. Where other records' entries
sit is ignored, so a table created under the renamed table's old name, whose
files go to a directory of its own, does not take the exception away. The
scan runs only on that path, so commits inside the namespace cost what they
did.

Co-Authored-By: Claude <noreply@anthropic.com>

* s3tables: seed the table at the renamed table's old name

A rename may keep the old name taken, so creating a table there can be
refused; seeding the record keeps the test on the update path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-10 21:15:48 -08:00
Tsic 518009dcd4 filer: apply filerGroup prefix to bucket default collection (#12499)
filer: prefix bucket default collection, assign copy collection explicitly

S3 server-side copy paths (CopyObject/UploadPartCopy) assign volumes with an empty collection and relied on the filer's bucket fallback. The fallback used the raw bucket name, while PutObject uses getCollectionName(bucket) = <filerGroup>_<bucket>, so copies landed in a second, unprefixed collection.

Prefix the filer's bucket default with the filer group, and have the S3 copy path pass that collection explicitly, like PutObject. Direct filer writes keep path-rule precedence, so this only aligns S3 copy with S3 put.

Adds regression tests for the fallback, explicit collection precedence, and the S3 copy collection resolver.
2026-11-10 21:12:49 -08:00
Ryan Leung 7ad8419815 shell: preserve completed uploads for bucket-root objects (#12545)
Signed-off-by: rleungx <rleungx@gmail.com>
2026-11-10 19:39:10 -08:00
github-actions[bot] 9799b82edf docs: regenerate star history chart 2026-11-11 01:16:36 +00:00
Mansur AkhmadiandMansur Akhmadi 9886ac0bcf topology: take dn.Lock when a heartbeat links a new disk type (#12535)
Co-authored-by: Mansur Akhmadi <10739460+mansur-akhmadi@users.noreply.github.com>
2026-11-10 12:17:21 -08:00
Mansur AkhmadiandMansur Akhmadi 9f4b652522 master: make admin lock lease and release atomic (#12532)
Co-authored-by: Mansur Akhmadi <10739460+mansur-akhmadi@users.noreply.github.com>
2026-11-10 12:16:23 -08:00
Saarthak Gupta 017e42633f shell: return flag parse errors from s3.bucket.create (#12544) 2026-11-10 12:15:01 -08:00
Mohd Quamar TyagiandChris Lu 5863e32318 weed/s3api: avoid truncated marker rescan in ListObjects (#12540) (#12541)
* weed/s3api: avoid truncated marker rescan in ListObjects (#12540)

Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com>

* fix(s3): validate marker directory candidates

Respect prefix filters and internal-directory exclusions when recovering marker candidates. Restrict lookups to direct children, propagate read failures other than not-found, and deduplicate directory marker echoes.

Add regression coverage and model promoted objects consistently in the XML-unsafe prefix fixture.

---------

Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-11-10 09:58:45 -08:00
Chris Lu d485d2c9cd filer.sync: shorten retries for confirmed missing chunks (#12546)
* filer.sync: shorten retries for confirmed missing chunks

* filer.sync: clarify missing chunk grace period names
2026-11-10 09:56:17 -08:00
Chris Lu 6c473433c5 vector: bound request buffering before authentication and document installation (#12537)
* s3vectors: bound body buffering before authentication

* docs: include vector engine installation and source builds

* s3vectors: prevent MIME sniffing of JSON error responses

* docs: load vector Cargo configuration when building from source

* docs: note the native protoc requirement for Windows vector builds

* docs: make vector source checkout and working directory explicit

* ci: handle deb822 sources during Ubuntu mirror failover

* ci: bound Ubuntu mirror attempts and reject partial index updates
2026-11-10 04:21:56 -08:00
Chris Lu 8236c8d136 vector: integrate engine health, admin management, and local deployment (#12517)
* vector: expose request metrics and registration-aware health endpoints

* vector: report immutable read cache and storage fetch metrics

* docs: describe vector engine health, metrics, and maintenance

* vector: count canceled requests when their handler is dropped

* vector: account for uncached large immutable storage reads

* vector: reliably release the stalled credential reader in tests

* vector: observe large reads as their streamed body is consumed

* vector: bound heartbeats on established master registration streams

* test: exercise hardened vector maintenance on Go and Rust volumes

* vector: register engines in their configured filer group

* vector: make retried index creation preserve compatible datasets

* vector: allow registration to advertise a separate address

* test: release stalled credential readers without blocking FIFO cleanup

* test: allow hardened vector coverage in a named filer group

* vector: discover engines in the same filer group as their gateway

* vector: fall back when the advertised address is empty

* admin: expose vector readiness, maintenance, and management tools (#12520)

* s3vectors: share vector discovery with trusted management clients

* admin: show vector engine readiness and recent maintenance activity

* admin: suggest S3 Vectors IAM actions from the supported operations

* admin: manage vector indexes, policies, tags, encryption, and query data

* admin: validate SSE-S3 support before creating encrypted vector buckets

* admin: discover vector engines in their registration group

* admin: probe vector engines concurrently with independent time budgets

* admin: verify encryption capability on the remote S3 gateway

* admin: defer discovery until after the HTTP server can start

* admin: distinguish unknown vector health services from failed engines

* admin: count workers that can execute vector index maintenance

* admin: allow read-only users to query vectors and inspect tags

* vector: scope gateway and admin discovery to the configured filer group

* admin: refresh policy state after mutations and serialize policy controls

* vector: drain active RPCs before closing replaced discovery connections

* vector: respect request cancellation while waiting for discovery

* vector: prefer another registered engine after a transport failure

* admin: check all masters before reporting no registered vector engines

* iam: honor attached vector policies without broadening legacy grants

* vector: allow short-lived discovery clients to close their connections

* vector: refresh cached master addresses for each discovery lookup

* iam: enforce vector bucket tag conditions in attached policies

* iam: authorize vector bucket creation with the requested tags

* test: exercise vector IAM grants and tag denies on Go and Rust volumes

* admin: reject authenticated vector mutations without an admin role

* test: run vector IAM assertions before the shared engine is stopped

* vector: add native mini startup, shell tools, and a runnable Compose example (#12522)

* s3vectors: share vector discovery with trusted management clients

* admin: show vector engine readiness and recent maintenance activity

* admin: suggest S3 Vectors IAM actions from the supported operations

* admin: manage vector indexes, policies, tags, encryption, and query data

* admin: validate SSE-S3 support before creating encrypted vector buckets

* admin: discover vector engines in their registration group

* admin: probe vector engines concurrently with independent time budgets

* admin: verify encryption capability on the remote S3 gateway

* admin: defer discovery until after the HTTP server can start

* admin: distinguish unknown vector health services from failed engines

* admin: count workers that can execute vector index maintenance

* admin: allow read-only users to query vectors and inspect tags

* vector: scope gateway and admin discovery to the configured filer group

* admin: refresh policy state after mutations and serialize policy controls

* vector: drain active RPCs before closing replaced discovery connections

* vector: respect request cancellation while waiting for discovery

* vector: prefer another registered engine after a transport failure

* admin: check all masters before reporting no registered vector engines

* iam: honor attached vector policies without broadening legacy grants

* vector: allow short-lived discovery clients to close their connections

* vector: refresh cached master addresses for each discovery lookup

* iam: enforce vector bucket tag conditions in attached policies

* iam: authorize vector bucket creation with the requested tags

* test: exercise vector IAM grants and tag denies on Go and Rust volumes

* admin: reject authenticated vector mutations without an admin role

* test: run vector IAM assertions before the shared engine is stopped

* mini: optionally start and supervise the vector engine

* mini: bootstrap vector buckets with existing Lance catalog helpers

* shell: manage vector buckets, indexes, and vector data

* docs: provide a runnable vector and maintenance worker Compose stack

* shell: scope vector discovery to the session and close its connections

* mini: honor the bind address for vector and metrics listeners

* mini: reserve vector ports before assigning gRPC and compatibility listeners

* mini: wait for vector storage setup and master registration

* mini: create requested vector buckets without requiring the Lance HTTP listener

* shell: refresh master addresses during vector discovery

* mini: resolve sibling vector executables with platform extensions

* vector: preserve uncertain creates and fix mini listener allocation (#12525)

* vector: retain catalog records when index creation has an unknown outcome

* mini: keep chosen listener ports reserved throughout allocation

* mini: resolve hostname binds before starting vector listeners

* vector: fix integration test imports and recover interrupted index creation (#12533)

* shell: register vector operations from the command package

* s3vectors: resume interrupted index initialization safely

* mini: exclude disabled WebDAV from listener allocation

* s3vectors: retain uncertain creates across refused retries

* s3vectors: claim pending indexes before retrying initialization

* s3vectors: compare resolved distance metrics during recovery

* s3vectors: preserve metadata keys when retrying initialization

* s3vectors: verify ownership before initializing new catalog records
2026-11-10 04:21:18 -08:00
github-actions[bot] 3f5b07475d docs: regenerate star history chart 2026-11-10 00:45:43 +00:00
Chris Lu 5abb18e678 helm: manage vector engines and provision vector buckets (#12518)
* helm: deploy vector engines with TLS, probes, and monitoring

* helm: provision vector buckets through the S3 Vectors API

* helm: bracket IPv6 pod addresses for vector listeners

* helm: discover vector engines through the all-in-one master

* helm: accept vector component NetworkPolicy overrides

* helm: bind vector metrics to the pod address on both IP families

* helm: provision vector buckets through embedded and all-in-one gateways

* docs: use the full Helm name for external vector certificates

* helm: render vector instance labels as strings

* helm: allow vector bucket hooks to use restricted security contexts

* helm: honor vector security context enable switches

* helm: prevent duplicate vector monitoring labels

* helm: reject metrics flags with the default pre-metrics vector image

* test: skip vector chart compatibility checks when Helm is unavailable
2026-11-09 16:34:57 -08:00
Chris Lu c1afb1b1cb ci: finish native vector checks and fix Windows installer tests (#12526)
* test: pass installer scripts through stdin on Windows

* ci: allow cold Intel macOS vector builds to finish

* ci: retain completed Rust builds when later platform checks fail
2026-11-09 16:33:50 -08:00
Chris Lu 89d3016d5e vector: publish native macOS and Windows engines and installer support (#12523)
* ci: build, test, and publish vector engines for macOS and Windows

* installer: install native vector archives on macOS and Windows

* ci: run desktop vector checks for shared Rust and protobuf changes

* ci: publish vector releases from a separate job with write permissions

* ci: build tagged vector releases without restoring caches

* ci: propagate Windows bash path conversion failures
2026-11-09 16:14:42 -08:00
Chris Lu ac2857911d grafana: add vector API, engine, cache, and maintenance views (#12519)
* grafana: add vector API, engine, cache, and maintenance panels

* helm: include vector panels with Kubernetes metric selectors

* grafana: preserve cluster and namespace identity when matching engine health

* grafana: retain engine identity when scrapes fail

* grafana: show zero cache hits before the first hit

* grafana: keep cluster and namespace identity in vector cache ratios
2026-11-09 16:00:02 -08:00
dependabot[bot] d9d8942caf build(deps): bump docker/login-action from 4.7.0 to 4.8.0 (#12510)
Bumps [docker/login-action](https://github.com/docker/login-action) from 4.7.0 to 4.8.0.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](https://github.com/docker/login-action/compare/v4.7.0...v4.8.0)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-version: 4.8.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-09 11:59:14 -08:00
dependabot[bot] f39b1abd03 build(deps): bump github.com/shirou/gopsutil/v4 from 4.26.7 to 4.26.10 (#12507)
Bumps [github.com/shirou/gopsutil/v4](https://github.com/shirou/gopsutil) from 4.26.7 to 4.26.10.
- [Release notes](https://github.com/shirou/gopsutil/releases)
- [Commits](https://github.com/shirou/gopsutil/compare/v4.26.7...v4.26.10)

---
updated-dependencies:
- dependency-name: github.com/shirou/gopsutil/v4
  dependency-version: 4.26.10
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-09 11:59:07 -08:00
dependabot[bot] 2373207b6e build(deps): bump sigstore/cosign-installer from 4.1.2 to 4.1.3 in /.github/actions/sign-image (#12512)
build(deps): bump sigstore/cosign-installer

Bumps [sigstore/cosign-installer](https://github.com/sigstore/cosign-installer) from 4.1.2 to 4.1.3.
- [Release notes](https://github.com/sigstore/cosign-installer/releases)
- [Commits](https://github.com/sigstore/cosign-installer/compare/v4.1.2...v4.1.3)

---
updated-dependencies:
- dependency-name: sigstore/cosign-installer
  dependency-version: 4.1.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-09 11:58:55 -08:00
dependabot[bot] 82cd43a884 build(deps): bump helm/kind-action from 1.15.1 to 1.15.2 (#12511)
Bumps [helm/kind-action](https://github.com/helm/kind-action) from 1.15.1 to 1.15.2.
- [Release notes](https://github.com/helm/kind-action/releases)
- [Commits](https://github.com/helm/kind-action/compare/v1.15.1...v1.15.2)

---
updated-dependencies:
- dependency-name: helm/kind-action
  dependency-version: 1.15.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-09 11:58:40 -08:00
dependabot[bot] d1d081fcf2 build(deps): bump github/codeql-action from 4.39.1 to 4.39.3 (#12509)
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.39.1 to 4.39.3.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/v4.39.1...v4.39.3)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-version: 4.39.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-09 11:58:24 -08:00
Chris Lu 84cd3c621b s3vectors: integrate bucket metrics, audit events, and request limits (#12514)
* s3vectors: attribute request and traffic metrics to authorized buckets

* s3vectors: emit resource and identity audit events

* s3vectors: apply existing S3 request limits and preserve JSON errors

* s3vectors: select limit resources by the dispatched operation

* s3vectors: count decoded chunked request bodies against byte limits
2026-11-09 11:37:06 -08:00
Chris Lu 620a1c3f39 docs: showcase vector buckets and cluster features (#12513)
* docs: showcase vector buckets and cluster features

* docs: include built-in Iceberg catalog in table bucket overview

* docs: keep feature diagram at cluster level
2026-11-09 08:52:12 -08:00
dependabot[bot] 643784df0d build(deps): bump github.com/aws/aws-sdk-go-v2 from 1.47.5 to 1.47.7 (#12506)
Bumps [github.com/aws/aws-sdk-go-v2](https://github.com/aws/aws-sdk-go-v2) from 1.47.5 to 1.47.7.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.47.5...v1.47.7)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2
  dependency-version: 1.47.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-09 08:29:08 -08:00
dependabot[bot] 9bb33d0e36 build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azcore from 1.23.3 to 1.24.0 (#12505)
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azcore

Bumps [github.com/Azure/azure-sdk-for-go/sdk/azcore](https://github.com/Azure/azure-sdk-for-go) from 1.23.3 to 1.24.0.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.23.3...sdk/azcore/v1.24.0)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/azcore
  dependency-version: 1.24.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-09 08:28:40 -08:00
dependabot[bot] a1ba470b32 build(deps): bump github.com/testcontainers/testcontainers-go from 0.44.0 to 0.45.0 (#12504)
build(deps): bump github.com/testcontainers/testcontainers-go

Bumps [github.com/testcontainers/testcontainers-go](https://github.com/testcontainers/testcontainers-go) from 0.44.0 to 0.45.0.
- [Release notes](https://github.com/testcontainers/testcontainers-go/releases)
- [Commits](https://github.com/testcontainers/testcontainers-go/compare/v0.44.0...v0.45.0)

---
updated-dependencies:
- dependency-name: github.com/testcontainers/testcontainers-go
  dependency-version: 0.45.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-09 08:28:27 -08:00
dependabot[bot] 5b87b7bc95 build(deps): bump github.com/prometheus/procfs from 0.22.0 to 0.23.0 (#12503)
Bumps [github.com/prometheus/procfs](https://github.com/prometheus/procfs) from 0.22.0 to 0.23.0.
- [Release notes](https://github.com/prometheus/procfs/releases)
- [Commits](https://github.com/prometheus/procfs/compare/v0.22.0...v0.23.0)

---
updated-dependencies:
- dependency-name: github.com/prometheus/procfs
  dependency-version: 0.23.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-09 08:28:15 -08:00
github-actions[bot] 7506e5f91c docs: regenerate star history chart 2026-11-09 00:35:41 +00:00
Chris LuandClaude Opus 5.5 5c910ad546 topology: list collections without nesting the collection map read lock (#12495)
* topology: list collections without nesting the collection map read lock

Items() already takes the read lock and copies the values. Holding it
around the call made ListCollections take the read lock twice, so a
DeleteCollection waiting for the write lock in between blocked the
second read lock and wedged the master.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* topology: test ListCollections against a concurrent collection delete

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 15:05:41 -08:00
github-actions[bot] 4ab66fb76d 4.53 2026-11-08 18:10:33 +00:00
Chris LuandClaude Opus 5.5 76fddab5ef master: keep a re-registered filer in the lock ring when its old stream ends (#12494)
* master: move the KeepConnected stream cleanup into removeClusterNode

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* master: drop a filer from the lock ring only with its last registration

A filer's replacement can register under the same address before the
master ends the old stream. The cluster list counts both registrations,
but the old stream's cleanup removed the address from the lock ring
outright, leaving the live filer out of the ring until it re-registered.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* master: order a stream's cluster list and lock ring changes

The cleanup of an ending stream could see its address removed from the
cluster list, then a new registration of that address add it to the list
and the ring, and only then remove it from the ring.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* master: take a leaving filer's reconnect out of the lock ring

A filer that reconnects while leaving registers with LeaveLockRing set.
Its broken stream can still be open on the master, and since that
stream's end no longer drops an address that is still registered, the
reconnect removes the address from the ring itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 09:55:32 -08:00
Chris Lu 35e98177d9 shell: test collection.list keeps live statistics next to offline names (#12492) 2026-11-08 09:49:32 -08:00
Ping Qiuandcodex03 0705e20fe9 shell: include collection names when topology statistics are unavailable (#12490)
fix(shell): include collections without live topology in list

Co-authored-by: codex03 <codex03@local.invalid>
2026-11-08 09:32:47 -08:00
Chris LuandClaude Opus 5.5 c147b775f9 helm: say what the Lance worker needs from security.toml (#12489)
* helm: say what the Lance worker needs from security.toml

Since #12470 the Lance namespace authenticates weed-worker by a token
derived from jwt.filer_signing.key, which the chart renders only with
securityConfig.jwtSigning.filerWrite. Without it, a gateway with
identities lists the worker no tables, and the lance_* jobs silently
find nothing to do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* helm: warn that filerWrite also requires signed filer HTTP writes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 09:19:51 -08:00
Chris LuandClaude Opus 5.5 4e12c13416 Volume encryption follows the filer: weed s3, weed-vector, weed-worker, weed-vector-cli (#12487)
* s3: encrypt volume data when the filer is configured to

weed s3 read the filer's configuration at startup (buckets dir, filer
group, masters, metrics, maxMB) but not its -encryptVolumeData, so a
cluster that turned encryption on at the filer still had S3 uploads
stored plain, while uploads through the filer itself were encrypted.
The gateway now encrypts when either its own -encryptVolumeData or the
filer's is set; weed-vector, weed-worker and weed-vector-cli already
follow the filer's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: weed-vector, weed-worker and the CLI take volume encryption from the filer

They already encrypted chunks for the volume servers when the filer's
configuration said so, or when their own --encryptVolumeData did. With
weed s3 following the filer too, the filer's -encryptVolumeData is the
one switch for every writer, so the three flags are removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* S3 Vectors tests: turn volume encryption on at the filer

VECTOR_ENCRYPT_VOLUMES=1 set -encryptVolumeData on the gateway, the
engine and the worker. It now sets it on the filer alone, so the leg
checks that each of them follows the filer: the gateway's stored chunks
carry a cipher key (with a gateway that ignores the filer, the leg fails
on "the gateway stored a chunk without a cipher key"), and so do the
engine's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker README: volume encryption follows the filer

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3: help text says the filer's -encryptVolumeData turns S3 encryption on

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* S3 Vectors tests: plain legs check the engine stored no cipher key

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 09:17:30 -08:00
Chris LuandClaude Opus 5.5 87627c2d72 S3 Vectors: help text for weed-vector, weed-vector-cli and weed-worker (#12486)
vector: the engine's, CLI's and worker's help say what they do now

weed-vector's --masters and --filer-group said "reads through" and
--security-toml named only gRPC TLS, though the engine keeps tables on the
filers and volume servers and takes https.client, the read JWT key and the
SSE-S3 keys from it too; weed-vector-cli called itself a "query spike";
weed-worker's --filer-signing-key spoke of write credentials the
namespace no longer vends.


Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 05:55:40 -08:00
Chris LuandClaude Opus 5.5 af299655e7 S3 Vectors: weed-vector drops the ignored --namespace and --filer-signing-key (#12485)
vector: weed-vector drops the ignored --namespace and --filer-signing-key

Since #12468 the gateway resolves every table it sends, so the engine has
no catalog to name or authenticate to; the two flags were kept only so an
older command line still started, with a warning. The direct-storage
engine is new, and release 4.52's command lines need changing anyway
(--storage-option is refused), so they are removed.


Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 05:55:30 -08:00
Chris LuandClaude Opus 5.5 cd616a48a8 S3 Vectors: refuse an SSE-S3 index when the engine has no key for it (#12484)
vector: refuse an SSE-S3 index when the engine has no key for it

The engine writes and reads an SSE-S3 bucket's files itself, so without
s3.sse.kek or s3.sse.key it failed inside Lance and the client saw only
InternalFailure. weed mini makes this easy to hit: it generates its key
for its own S3 server only. A table the gateway marks sse_s3 is now
refused up front with FailedPrecondition, which the gateway answers as
ValidationException with the reason.


Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 05:13:19 -08:00
Mansur AkhmadiandMansur Akhmadi 10ea662522 s3: don't report a version delete as done when the filer lookup failed (#12482)
* s3: don't report a version delete as done when the filer lookup failed

deleteSpecificObjectVersion treated any getEntry error as "entry is gone":

- the null-version precheck returned success without deleting anything,
- the recheck after a failed rm returned success while the entry may
  still be there,
- a failed .versions lookup was read as "not the latest version", so the
  latest blob was removed without repointing the pointer first.

Only filer_pb.ErrNotFound now counts as a miss. Other errors fail the
delete, so the S3 client or the lifecycle worker retries it. The lookups
go through retryFilerOp, like the other .versions reads in this file, so
a single transient filer error doesn't turn into a 500.

* s3: include recheck error when a version delete fails

* s3: recheck a failed version delete once instead of retrying

---------

Co-authored-by: Mansur Akhmadi <10739460+mansur-akhmadi@users.noreply.github.com>
2026-11-08 05:06:52 -08:00
f99cc6ca94 volume.fsck: batch and deduplicate filer entry deletions (#12480)
* volume.fsck: batch and deduplicate filer entry deletions

In volume.fsck with -findMissingChunksInFiler and -reallyDeleteFilerEntries, deleting orphaned filer entries was slow and CPU intensive due to tiny worker channel buffers (128), lack of batch processing, redundant HTTP DELETE requests for multi-chunk files, and lock contention on parent directory tracking.

Optimize by:
- Adding startPurgeBatchWorkers with 4096-buffered channels to prevent producer stalls and batch opportunistic deletes up to 64 items.
- Deduplicating missing chunks per file path during volume checks so that multi-chunk files are queued once.
- Processing batches with in-batch path deduplication and aggregating parent directories before updating purgedDirs in a single lock acquisition.
- Buffering temporary fid file writes (64KB) during the traversal to avoid millions of unbuffered syscalls.

* volume.fsck: report a failed write of the filer fid files

The fid files are written through a buffer now, so the last part of
each one reaches the disk only in the deferred Flush. Its error was
dropped: a full temp disk left a short fid file and the collection
still succeeded, and the volume check would then take the chunks
missing from it for orphans.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume.fsck: retry a batched path whose earlier delete failed

A path was marked done before its delete ran, so a second copy of it in
the same batch was skipped after a transient failure and the entry
stayed. Only skip paths already deleted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume.fsck: keep reporting a failed DELETE response read

The non-verbose path now discards the body without checking the copy,
so a connection cut after the headers went unreported.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume.fsck: keep one purge worker starter and one delete function

startPurgeWorkers and the single-path httpDelete were left only as
wrappers the tests called. Give their names to the batch versions and
adapt the tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 05:03:52 -08:00
Copilotandchrislusf 73fa85f54f Retry transient failures in star history generation (#12479)
* Initial plan

* Retry transient star history API failures

Co-authored-by: chrislusf <1543151+chrislusf@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: chrislusf <1543151+chrislusf@users.noreply.github.com>
2026-11-07 18:12:34 -08:00
Chris LuandClaude Opus 5.5 4f24b756ba s3tables, iceberg: give tables created without a location their own directory (#12475)
* s3tables: give a table created without a location its own directory

An Iceberg table created without a location kept its files under its name
path, so a rename left them where the next table of that name would land.
Such a table now gets s3://<bucket>/<namespace>/<name>-<uuid>, stored as the
record's warehouseLocation and returned by GetTable; its first commit writes
there without moving the root. Records without one keep resolving to their
name path, so existing tables need no rewrite. Lance datasets stay at their
name path, where Lance clients and the vector engine look for them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* iceberg: create tables and views in a directory of their own

A table or view created without a location took its name path, or a
unique one only when that path was already taken. It now always gets
<name>-<uuid>, so a later rename never leaves its files under a name
another table can take.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* iceberg: load and commit a table at its warehouse location

A table created through S3 Tables without a location has no metadata until
its first commit; loading or committing it now uses the directory recorded
for it, and only a record without one falls back to its name path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* iceberg: fail a stage-create whose marker is not saved

The marker is how the commit finds the location the stage-create handed
out. Without it the commit fell back to the name path while the client's
files sat under the returned location.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: expect tables created without a location in their own directory

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: rename a table that keeps its files elsewhere before reusing its name

Enterprise keeps the storage marker on a renamed table's files, so the name
of a table whose files sit under its name path stays taken there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: remove the table directory when a create backs out

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* iceberg: answer a create that lost the race with the registered table

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* iceberg: answer a view create that lost the race with the registered view

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* iceberg: fail a staged commit whose name another create registered

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* iceberg: remove the directory of a create whose name another table took

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* iceberg: keep the metadata of a staged commit whose name is taken

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: write the lakekeeper table's data under its warehouse location

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* iceberg: commit a staged table at the location of its own stage

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* iceberg: never continue another stage when a commit's own marker is gone

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* iceberg: test filer honors exclusive creates and token-guarded deletes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 16:37:31 -08:00
Chris LuandClaude Opus 5.5 55a4a5974c ec_balance: make the per-rack cap on global moves a worker setting (#12478)
* ec_balance: make the per-rack cap on global moves a worker setting

The global load-balancing phase planned at most 10 moves per rack per
detection run, hard-coded. On a cluster where one disk per machine holds
several times its siblings' shards, that is about 20 moves an hour against
tens of thousands needed, however many moves the worker can execute.

Add global_max_moves_per_rack to the plugin worker config. The default stays
10, so nothing changes unless it is raised. It is clamped to 1..5000 because
the planner reads 0 as unlimited and would plan a whole drain in one run.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* ec_balance: clamp global_max_moves_per_rack before narrowing it to int

A value past the int32 range made ReadIntConfig return the default, so a
request asking for more than the maximum got 10 moves instead of 5000.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec_balance: test the configured global cap through Detection

The helper test still passed if Detection handed the planner a literal cap.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec_balance: plan no more global moves per rack than the detection keeps

Each global move scans the source node's volumes, and the detection keeps
at most max_results moves. With the cap at 5000, three skewed racks of
20,000 volumes planned 15,000 moves to keep 500, taking about 70 s of the
300 s detection budget; bounded by max_results it takes about 5 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec_balance: test the configured global cap with a result limit

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 16:01:12 -08:00
78a3bc9dc3 filer.remote.sync: -allMounts writes back every mounted directory from one process (#12476)
* filer.remote.sync: -allMounts writes back every mounted directory from one process

A filer with many remote mounts needs one filer.remote.sync process per
mount. -allMounts runs the existing -dir write-back once per directory in
the mount table, in one process: each with its own subscription, client id
(the filer keeps one subscription per client id), offset, processor and
retry loop. A supervisor reads the mount table every -mountsCheckInterval
(5s) and starts a write-back for each mounted directory that has none
running, 10 at a time, 500 ms apart. It never stops one because the live
table changed: a write-back ends itself where the -dir process exits, at the
first mount-table version it receives that ends its mount (#12472), and a
directory still mounted is started again from its saved offset, after a
pause that grows from 1s to 30s while it keeps ending within a minute. A
directory mounted while the process runs, without a saved offset, waits up
to 2 minutes for one (a mount followed by a pull sets it after the pull). A
write-back that ends at an unmount 5 times in a row while the table still
mounts it at the same location is logged as a warning.

To run under a supervisor, a follow:

- returns errMountGone at the end of its mount instead of exiting.
  admitUntilMountEnds takes the end action: the -dir process keeps draining
  and exiting as before; a -allMounts follow admits nothing more (events
  after the end can still arrive in the same batch or before the stream
  closes, including remote.unmount's purge and a remount), ends its stream,
  drains, and saves no offset.
- drains its admitted jobs before it returns, however it ended, so a
  successor never races a job still running for the same path.
- saves its offset only over the value it read at its start or last saved
  (offsetGuard), both at its checkpoints and at its end. Otherwise a
  write-back behind in the log can overwrite the offset remote.unmount
  writes after its purge (or a mount after its mapping write), and its
  successor replays the unmount and never writes the remounted directory
  back. Once it sees the offset moved it saves nothing more and keeps no
  resume position. A value whose write reported an error counts as its own
  when read back.
- reads the stored offset once, before the mount table, and uses it as both
  its start and its guard's value: read after the table, an unmount
  completing in between would start a follow past the unmount's table.
- keeps a resume position for its next attempt only with the mount location
  it was earned under and the offset stored at the time; it is dropped when
  the directory is mounted elsewhere or someone else moved the offset.
- turns a panic in a job into that job's error, so one directory's failure
  does not end every directory's write-back.

Fixes #12471

* filer.remote.sync -allMounts: retire a failed follow whose mount is gone

A follow that failed before its stream reached the unmount (or before it
started) was retried every 30s forever: DetectMountInfo reports the
directory not mounted, which is not errMountGone. No stream runs on the
retry path, so a directory the live table no longer maps is retired there
and started again by reconcile if it is mounted again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync -allMounts: drop the restart pause of an unmounted directory

ended() left notBefore and ends behind for every directory ever mounted.
reconcile now drops them once the directory is neither mounted nor
followed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: log a -dir drain held at the mount end

The -dir exit drained with processor.Drain, so a remote call hung during
an unmount or move held the process silently; the follow's own drain
already warns every 5 minutes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: keep the resume position through a failed mount read

A transient DetectMountInfo error dropped the only position behind the
event a follow failed on. With -timeAgo and no checkpoint saved yet, the
retry then started at a later now minus timeAgo and skipped that event.
The next attempt that reads the table still checks the position against
the mount location and the stored offset.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: judge a kept resume position by the directory's remote location

resumeFor compared the mount's raw location, so a -dir below a mount kept
its position when the same remote was mounted one level up (or down),
although its paths now map to other keys; a replay from that position
skips startAfterMountOffset and writes old events to the new keys. The
position now carries dirRemoteLocation and is checked against it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync -allMounts: warn about a replayed unmount only from an unmoved offset

The warning counted every end at an unmount while the directory stayed
mounted at one location, so write-backs that each saved progress, or
unmount and remount cycles between two reconciles, also tripped it, and
it told the operator to set the offset to now, skipping events. An end
saves no offset, so a run of ends is one replay only while the stored
offset stays put: a moved offset now starts a new run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync -allMounts tests: wait for the supervisor before moving the frozen clock

A reconcile or end recorded after the clock moved would start its wait
or pause at the moved time, which the frozen clock never passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* remote.mount: save the sync offset right after the mount mapping

A new mount had no saved offset, so filer.remote.sync -allMounts waited
the full allMountsOffsetWait before following it and then started from
the directory's creation time anyway. remote.mount and remote.mount.buckets
now save the time taken just before the mapping write, keeping an offset
remote.unmount saved for a remount.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* remote.mount: save the directory's creation time as the sync offset

The mount time skipped local files already in a -nonempty directory, which
a write-back with no offset replays from the directory's creation. Saving
that creation time keeps the start position and only ends the -allMounts
wait.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* remote.mount: warn instead of failing when the sync offset is not saved

The mount mapping is already saved by then, and a mount without an
offset still starts from the same creation time after the -allMounts
wait. Failing stopped remote.mount.buckets before the remaining buckets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 15:43:44 -08:00
Chris LuandClaude Opus 5.5 554593796e ec_balance: drain a server holding far more shards than its siblings (#12477)
* ecbalancer: drain a full server through any target that can take a shard

The global phase paired the fullest server with the single emptiest one and
stopped for the whole rack when that pair had no legal move. Where every
volume already has a shard on each machine, no cross-machine move keeps the
spread, so on hosts running one volume server per disk the first server's
disk kept its pile while its siblings on the same machine stayed nearly
empty: the phase never planned a move.

Try sources fullest first and targets emptiest first and take the first pair
with a legal move, so such a server drains onto its own machine's emptier
servers. A same-machine move goes only to a server holding none of the
volume, so one disk never carries two of its shards. Per-machine counts are
indexed once per rack and pairs without a move are remembered until a move
can change them, so an uncapped drain stays fast.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* ec_balance: keep global moves ahead of within-rack moves under the detection cap

Detection truncated the plan to the per-cycle job limit in planner order,
which puts the global moves last. A long within-rack backlog pushed all of
them past the limit, so even a planned global move never ran. Order dedup
and cross-rack first, then the capped global phase, then within-rack.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* ec_balance: keep each volume's moves in plan order when promoting global moves

The admin scheduler runs only the first proposal per volume each cycle. A
global move promoted ahead of its volume's within-rack moves can take a shard
that one of those moves placed, so it ran while the shard was still on the
original server and failed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ecbalancer: order global targets by fullness after taking the shard

The search stops at the first target that would end up fuller than the
source, which only holds when targets are ordered by that same value. With
unequal capacities an empty one-slot server sorted first and ended the search
before a larger server that could take the shard.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ecbalancer: free the source disk slot on a global move

The global phase cleared the moved shard's bits but left the source disk's
free slots and shard count unchanged, unlike every other phase. Use
releaseShard so a later pick of that server sees the freed disk.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec_balance: test that a capped detection keeps a global move

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ecbalancer: test the same-machine move skips a sibling holding the volume

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 12:55:34 -08:00
Nguyễn Lộc Phúc 195f8156de fix(iceberg): keep a losing table create from deleting the winner's metadata (#12474)
* iceberg: keep a losing table create from deleting the winner's metadata

* iceberg: keep a staged file whose published record was renamed away

A create answers a conflict when its record was already visible and
another request changed it before the create could back it out. The
cleanup then looked only at the original name, so a record renamed in
between read as absent and the file it still points at was deleted.

S3 Tables now reports that case with a conflict of its own, and both
create paths keep the staged file on it instead of probing the name.
A conflict over a still-hidden record, lost to another create, still
removes the file.

* iceberg: discard a losing create's metadata only while it is still its own

The cleanup deleted the staged file by name. The path can be reused
before it runs: the table is dropped and purged, and another create
stages the same v1.metadata.json and registers it, which the losing
request then deleted.

Staged files now carry a per-request token in an internal extended
attribute, and the cleanup deletes with that token as the filer's
ExpectedExtended precondition, checked under the path lock the delete
runs under. A file that took the path since is left in place.

* iceberg: classify a typed S3 Tables error by its code, not its message

isS3TablesAlreadyExists matched "already exists" in the message before
looking at the code, so an InternalError whose text mentioned an
existing entry read as the name being taken: CreateTable answered 409
instead of 500, and registrationRefused discarded a staged file the
catalog may still point at. A typed error now decides by its code; only
an untyped one is matched by message.

* iceberg: pin stage-create on in the stage-create conflict test

Stage-create is on by default, but a shell that sets
ICEBERG_ENABLE_STAGE_CREATE=false turned the expected 409 into a 501.
Setting it in the test keeps the result independent of the environment.

* iceberg: test two creates overlapping at the canonical metadata name

The concurrent-creators test only lines the creators up before they
start, so whether two of them ever collide at v1.metadata.json is up to
the scheduler. The new test fixes the order: the second create runs
while the first has staged v1.metadata.json and not yet registered it.
It checks that every staging attempt is exclusive, that the second falls
back to v1-<uuid>, that both files hold their own metadata while both
are staged, and that the loser removes only its own file.

* s3tables: a create whose unmark write never went out stayed hidden

confirmChild counted any failure to take a create's mark off as possibly
published, the lookup that precedes the write included. That lookup
failing leaves the mark on, so the record stayed hidden; if another
create then adopted it, the back-out reported a published record that
changed, and the Iceberg catalog kept a staged file no record names.

The mark-removal calls now go through a wrapper that notes whether an
UpdateEntry went out without the filer refusing it. Only then may the
record have been seen. A refused compare, a missing entry, or a failure
before any write leaves the create's record hidden.

* s3tables: pass mayBeVisible into backOutChild

Fixes the build after rebasing onto the backOutChild split.
2026-11-07 10:54:04 -08:00
985706cc32 volume: check a needle's CRC the first time a range of it is read (#12467)
* volume: check a needle's CRC the first time a range of it is read

A ranged read never checked the CRC, so a replica with a damaged needle
served the damaged bytes with 206 and the reader never tried another
copy. The first ranged read of a needle now streams the whole needle
through the existing CRC check, with the usual pooled page buffers,
before writing any byte. A mismatch fails the read, so the reader falls
back to another replica. Needles that pass go into a bounded LRU, keyed
by volume, compaction revision, id, append time and checksum, so later
ranges of the same needle stream as before.

Fixes #5959 for ranged reads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: key verified ranges by volume load, share concurrent checks

From review: the cache key read v.SuperBlock.CompactionRevision without
the data file lock, racing a compaction commit that reloads the super
block, and a remounted copy of a volume inherited the old copy's
verified entries. Key entries by a load generation instead, an atomic
counter that changes on every load, including the reload in
CommitCompact, and cache a result only if no reload happened during
the check. Concurrent first ranges of a needle now share one check
through singleflight instead of each reading the whole needle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: keep hot needles verified, leave the io error streak to the range read

From review: Contains does not refresh an entry's place in the LRU, so
even a constantly read needle was evicted after 64k other needles were
verified; look it up with Get instead. The verification pass also
cleared the io error streak before the range read ran, so a range read
failing with EIO after a passing check never built up a streak; the
pass into io.Discard now leaves the streak to the range read, like the
meta read before a streamed GET.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume (Rust): check a needle's CRC the first time a range of it is read

Mirror of the Go change: a range served from the data file now checks the
whole needle's CRC before any byte of it is sent, so a damaged replica
answers 500 and the reader tries another copy. Needles that pass go into a
bounded LRU keyed by the volume's load generation, needle id, append time
and checksum; concurrent first ranges of a needle share one check, and a
check that passes leaves the io error streak to the range read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume (Rust): a range over the whole needle checks its CRC

Go reads a range that covers the whole needle through the checked full-read
path. The Rust volume server read it from the data file unchecked, so
"bytes=0-" served a damaged needle with 206. Check the slice's CRC before
answering, as the full read does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: check the covered blocks of every range against CRCs recorded at the first check

The first ranged read of a needle now records the CRC of each block
(64 KiB to 1 MiB, at most 64 blocks below 64 MiB) while it checks the
whole needle. Later ranges read the block-aligned span around them and
write a block's bytes only once it matches, so damage that appears
after the first check is not served.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume (Rust): check the covered blocks of every range against CRCs recorded at the first check

verify_checksum now records a CRC per block (same sizes as Go's
rangeCheckBlockSize) and returns them from the cache; read_checked_range
reads the block-aligned span of a range and answers 500 if a block no
longer matches, so damage after the first check is not served.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-07 10:53:07 -08:00
Chris LuandClaude Opus 5.5 d0b5529a2d S3 Vectors: weed-vector, weed-worker and the CLI write tables to the filers and volume servers (#12470)
* sw-client: the filer and volume client library

The client the Enterprise repository's VFS and vector engine use, with
the same API: filer gRPC for entries, listings, volume locations,
assignments and remote caching; volume HTTP, plain or TLS, for needle
bytes; chunk resolution for sparse, overlapping and manifest chunks; a
location cache; and the write side. Enterprise-only protocol fields
(license, conditional mutations, hard links, KV listing) are left out.
Compiled against weed/pb/filer.proto like the other worker crates.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-lance-worker: reads straight from the filers and volume servers

direct_read::DirectReads wraps the object store Lance opens so that
entries, listings and bytes come from the filers (gRPC) and the volume
servers (HTTP, signed with the read JWT), whatever the entry holds:
inline content, chunk manifests, sparse or overlapping chunks, the
filer's volume encryption, the gateway's SSE-S3 (sse.rs, with the KEK
from security.toml or the filer) and objects tiered to remote storage.
Writes still go to the inner store. The code and its tests are those of
the Enterprise repository's engine.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: read tables from the filers and volume servers

The engine asks the masters for the filers (and follows the list), takes
jwt.signing.read.key, grpc.client, https.client and the SSE-S3 keys from
security.toml, and opens every table with DirectReads innermost and the
read cache outside it. --masters is required: the gateway finds the
engine through them and the engine finds the filers. Writes still go to
the gateway.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* sse: chunks out of a manifest keep their own SSE-S3 keys

A chunk expanded from a manifest carries its id as fid, so keying the
per-chunk key cache by file_id gave every such chunk the first one's key
and IV. Keyed by the chunk's id however it is carried.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* direct reads: a silent filer is passed over like an unreachable one

Each filer attempt is bounded, so a filer that accepts the connection and
never answers does not hold the read or the startup lookup.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* sw-client: carry the assignment's fsync to the volume upload

The filer's storage rule may want a needle on disk before the volume
answers; the assignment says so and the upload now asks for it, as the
Go uploader does.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: argument tests pass the required masters; connect timeout for filers

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* direct reads: a failed read is tried once more against fresh locations

A fetch that fails forgets the volume locations it was made against; the
request now looks them up again and retries instead of failing on a
volume that moved.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* sw-client: an exclusive create that finds its entry, and needle deletion

create_entry marks ENTRY_ALREADY_EXISTS with EntryExists, so a caller can
tell a lost conditional create from a failure; the transport deletes a
needle whose entry was never created.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* sse: encode chunks as the gateway stores them

EntryEncoder writes SSE-S3 under a fresh object key wrapped by the KEK,
serialized with the IV, key id and commitment the gateway's reader
expects, each chunk under the IV at its offset; and the filer's volume
encryption under a per-chunk key.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* direct storage: writes, deletes and table removal without the gateway

A put uploads its chunks to the volume servers the filer assigns, at
the filer's chunk size, encoded as the bucket and cluster ask, and
creates the entry the gateway would: md5, mime, size, the ETag
attribute, the SSE-S3 attributes. A conditional create is the filer's
exclusive create, so Lance's manifest commit conflicts the same way;
the needles of a create that failed are discarded. A multipart upload
turns parts into chunks and derives its ETag from them. Deletes remove
the entry with its data, and remove_table a table directory. Copy and
rename, which Lance does not use on S3, are not offered.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector.proto: Table says whether its bucket encrypts with SSE-S3

The engine writes the table's files itself, so it needs what the S3 write
path knew: whether the bucket declares SSE-S3. storage_options go: the
engine reaches no S3 store.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors: send the engine the bucket's SSE-S3 setting

The rule the S3 write path applies to a Lance bucket's files, factored out
of DeclaredBucketEncryption, decides it from the vector bucket's record.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: every table operation goes to the filers and volume servers

Opens, index creation and drops all go through the direct store, so the
engine has no S3 client of its own left to protect: --storage-option,
the endpoint pin and the request-option allowlist go, and lance's own S3
store is built unsigned and never dialed. The bucket's SSE-S3 setting
comes with the table; -encryptVolumeData encrypts chunks for the volume
servers as the gateway's flag does.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors tests: the engine takes no credentials

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors e2e: the engine writes what the gateway would have

Every leg checks that the entries the engine wrote carry the gateway's
ETag attribute, the hardened leg that their chunks are encrypted for the
volume servers, and a new test that an SSE-S3 bucket's files are written
under keys the gateway unwraps when it serves them over S3.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* cluster plumbing moves to the worker crate for the engine, worker and CLI

security.toml, the gRPC TLS connector and acceptor, the master dialer,
the filer discovery and the storage connection built from them now live
in weed-lance-worker (cluster, security, grpc_tls), where the worker and
the CLI can use them too. The server crate re-exports security and tls
under their old paths.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker, weed-vector-cli: tables through the filers and volume servers

dataset::Storage replaces the S3 credentials every job and the CLI
carried: a table is opened and written through the cluster's storage,
encrypted as its bucket declares (read from the bucket's record on the
filer), and dropped there. Both take --masters and security.toml like the
engine; --access-key and --secret-key go, and with them the catalog's
x-api-key credential, which the cluster token replaces.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* lance catalog: the cluster token alone authenticates a cluster process

The engine, the maintenance worker and the CLI hold the cluster's filer
signing key and reach a table's files themselves. A request presenting
the cluster token and no other credential is theirs: it is authenticated
by the token, runs through the trusted manager, and is vended no S3
credentials.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* worker chart, README and test design follow the worker's new arguments

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors tests: the worker and the CLI reach the cluster through the masters

The cluster gets a filer signing key, which the catalog authenticates
the worker and the CLI by; both take --masters in place of S3 keys.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* direct storage: encrypted chunks each start their own CTR stream

The gateway encrypts a multipart part from the IV derived for its offset,
from position zero; the encoder encrypted from the exact byte offset of
the object's stream instead, which agreed only at block boundaries. A
part of 3,000 bytes read back wrong; now tested.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* direct storage: keep what a lost reply saved, discard what a failed put left

A create the filer saved but did not answer, or an exclusive create
retried onto this very upload, keeps its chunks and counts as written;
the needles of a put that failed midway are discarded. The report prints
reads as reads.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: the security.toml it is given also holds its signing key

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* worker chart: the Lance worker mounts grpc.client's certificate

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* direct reads: the filer deadline bounds a listing page, not the listing

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: security.toml booleans read as viper reads them

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: a filer the master lists badly does not hide the others

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: register with the masters only once ready to serve

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* sse: every chunk after the first is written under a key of its own

Chunks of one object shared a key with IVs derived from their offsets, so
two chunks meeting inside a cipher block shared keystream. The first
chunk stays under the object's key and IV, which is how the gateway reads
an object of one chunk; every other chunk gets a fresh key and IV, as the
gateway's multipart parts have and as it reads an object of several.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* worker chart: the Lance worker is given the filer group

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors e2e: the gateway serves engine-written SSE-S3 files of one chunk and of several

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* sw-client: the HTTP transport asks the volume to fsync when the assignment does

write_synced fell back to the trait default, which refuses fsync, so every
upload under a fsync storage rule failed before it was sent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sw-client: https.client.ca alone vouches for volume servers

With a CA configured, reqwest still trusted the system roots next to it;
weed processes trust only the configured pool.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sw-client: an empty volume range is read without a request

A zero length underflowed the Range end, and a range past u64::MAX
overflowed it; the first now returns no bytes, the second an error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sw-client: a whole-needle answer to a ranged read is cut to the range

A server or proxy that ignores Range answers 200 with the whole needle;
the read returned its first bytes in place of the ones asked for.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sw-client: a read size past u64::MAX reads to the end of the file

offset + size overflowed before the clamp to the file size.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sw-client: an SSE-encrypted manifest is refused without a decoder

Only filer-encrypted manifests were; an SSE one went to the protobuf
decoder as ciphertext.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sw-client: an empty write leaves the staged file size alone

A zero-byte write past the end grew the size to its offset, as a
zero-byte pwrite does not.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sw-client: the chunk cache keeps no empty chunks

They add no bytes, so nothing evicted them and the map grew with every
distinct empty needle, even at capacity zero.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sw-client: flushing with a zero chunk size is an error, not a panic

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: an empty master list fails at startup

"" split into one empty address that passed the check, and the engine
then waited two minutes for a filer before a generic error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance cluster: an empty master list fails at startup

Cluster::new took "" as one master, so weed-vector, weed-worker and the
CLI waited two minutes for a filer before a generic error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* direct storage: a failed create keeps its chunks when the filer cannot be asked

The lookup after a failed create is what tells a lost reply from a
refusal; when it fails too, the entry may be saved, so its chunks stay.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: a gateway-described table takes the shared unsigned store options

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* vector.proto: give sse_s3 a new field number and reserve storage_options

Field 4 held the storage_options map on master; a gateway from that build
would send a length-delimited value the engine cannot decode as a bool.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector-cli: open s3:// URIs on the cluster's storage

A table named by its s3:// location skipped the store wrapper and opened
Lance's own S3 store, which has no credentials since the fallback flags
were dropped. It now opens like a namespace table and needs --masters;
local paths keep the plain builder.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* vector: an index id is plain path components, and drop trusts describe()

`resolved()` now refuses an id whose part is empty, `.`, `..` or holds a
`/`, since the colocated path would land elsewhere on the filer; and
`drop_index` no longer repeats the colocation check for the location
describe() already enforced for the configured schemes, which made a drop
on a memory:// table fail in the scheme test configuration.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: a file delete takes only its entry; a table path is plain names

The store's delete of one file no longer asks the filer for a recursive
delete, so a path that names a directory does not take its content with
it, and `remove_table` refuses a location whose bucket or path component
is empty, `.`, `..` or holds a separator, which the filer would resolve
outside the table.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: chunks are assigned to the bucket's collection

The filer assigns a chunk's volume from the collection the caller names;
the gateway names the bucket, prefixed by the filer group when the
cluster has one, so a plain assign landed the engine's chunks in the
default collection. The store now asks for `<bucket>` or
`<group>_<bucket>`, with the group taken from the cluster connection.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: a put's content type becomes the entry's mime

Lance sets no content type today, so every entry was written as
application/octet-stream; a put that does carry one now creates the
entry with it, as the gateway does.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: an upload whose create went unanswered is done with

A create the filer took and never answered (a timeout, or a transport
that reports the filer unavailable) may still be applied, so its chunks
are kept, as they already were for a create whose reply was lost. The
upload then records that it is done: a second completion is refused
instead of creating an empty entry, and an abort discards nothing.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: an upload dropped unfinished discards its chunks

A request cancelled mid-write drops its multipart upload with neither
complete nor abort called, leaving its needles on the volume servers.
Dropping an upload that still holds chunks now discards them on the
runtime; a completed upload, or one whose completion is in doubt, holds
none.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3: a cluster process registers and deregisters in the vector namespace

The Lance catalog ran a cluster-token caller through the trusted manager
for every operation but the vector-namespace write check of register and
deregister, which still used the caller-checked manager and answered 403
for an index of a LANCE bucket. One `manager(r)` now picks the manager
for all three.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: the worker's flags take two dashes

The worker parses its flags with clap and, unlike the engine, does not
accept Go-style single-dash long flags, so the harness's hardened worker
leg and the README's `-encryptVolumeData` did not start it. The README's
commands also gain the `--masters` the worker now requires.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: the CLI takes --encryptVolumeData

An index build or optimize from the CLI wrote its chunks unencrypted for
the volume servers unless the filer itself asked for the cipher; the CLI
now takes the flag the engine and the worker have.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: a bucket's encryption is read when a table is opened

The setting was cached for a minute, so a bucket recreated with SSE-S3
within that minute had its first tables written plain, and a record that
could not be decoded was read as "no encryption". A table open now reads
the bucket's record from the filer, once per open, and an unreadable
record is an error.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: a removed table's files leave the object cache

`remove_table` asked the cache to forget everything under the table, but
the cache was built without invalidation closures, so the call was refused
and its error dropped: a data file read before the drop was still served
afterwards. The cache now supports the invalidation, and a refusal is an
error.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: an abort keeps the chunks it could not discard

`discard_all` stopped at the first chunk a volume server would not give
up and dropped the rest from the upload's state, so a retried abort found
nothing to do and the needles stayed. Every chunk is now attempted, the
ones left are kept for the next abort or the drop, and the upload can no
longer be completed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: a chunk write that fails gives its needle back

When the volume server took the chunk but its answer was lost, the needle
stayed under a file id nothing would ever reference. The assignment is
known before the write, so a failed write now discards it, best effort,
before the error is returned.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: the live worker tests require the cluster's masters

With WEED_LANCE_NAMESPACE set and WEED_MASTERS not, the tests seeded their
tables through lance's own S3 store, which reaches nothing, and failed
before a job ran. The missing variable is now named outright.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* direct storage: only the filer's missing-entry error means not found

Matching any "not found" text read failures such as an unresolved remote
storage as a missing entry, so a delete or a listing reported success over
data that was still there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* direct storage test: compare the chunks' SSE-S3 IVs

Each chunk's metadata wraps its key under a fresh nonce, so the metadata
differed even if the key and IV were reused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* direct storage: a filer passed over unanswered keeps the outcome open

When one filer timed out on a create and the next refused it, the refusal
came back without the no-answer mark, so the put discarded chunks the
first filer may still reference.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance test: start every concurrent writer before awaiting them

The lazy map spawned each writer only after the previous one finished, so
the commits never overlapped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* direct storage: count failed uploads and deletes

The storage report counted failed reads and lookups but not a volume
upload, an object delete or a table removal that failed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* S3 Vectors tests: run weed-vector-cli as the cluster's processes run

The CLI ran without the fixture's environment and outside its directory,
so under VECTOR_READ_JWT it could not read the volumes and under
VECTOR_TLS it found no security.toml.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* direct storage: a filer that fell away leaves the outcome open too

An unavailable error (connection reset, deadline, transport failure) can
arrive after the filer took the request, so a refusal from the next filer
no longer settles a create the first one may still apply: the chunks are
kept, as after a timeout.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* direct storage: a chunk write abandoned in flight gives its needle back

The needle is assigned before the write, so a request cancelled while the
write is in flight left it on the volume server with nothing to name it.
`Source::upload` is now `assign` and `write`, and the upload holds the
assignment across the write in a guard: a write that fails discards it
and one abandoned discards it on the runtime, while a write that returns
hands it to the entry.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* lance tests: sign for the namespace as the worker does

The live tests reached the namespace anonymously once the S3 keys went,
and an anonymous caller lists no tables, so detection proposed nothing.
They now take the cluster key from where the worker reads it
(WEED_JWT_FILER_SIGNING_KEY or security.toml) and log at info, so a table
a handler skips says why in the harness output.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3tables lifecycle test: run the Lance worker against the cluster

The worker leg of TestLanceTableLifecycle still handed the Rust tests S3
credentials and no masters. The mini cluster now has a cluster key, and
the tests get WEED_MASTERS and that key; passes with
WEED_LANCE_MAINTENANCE=worker (3 fragments became 1, 4 versions removed).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* direct storage: name no collection outside a filer group, as the gateway

Without a filer group the gateway leaves the collection to the filer, so a
filer.conf rule for the bucket's path picks it; naming the bucket
overrode that rule, and the chunks landed where a bucket delete does not
look.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* direct storage: a read reports the entry's content type

A put now stores its ContentType as the entry's mime, but get_opts
returned no attributes, so the type did not come back on a read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-07 10:51:52 -08:00
Chris LuandClaude Opus 5.5 5cd38527ba filer.remote.sync: skip pre-mount mount-table versions, decide the unmount before admitting later events (#12472)
* filer.remote.sync: judge the mount table before admitting later events

The unmount was decided inside a processor job, so events behind the
mapping write, such as remote.unmount's purge deletes, were admitted and
could delete remote objects before the exit landed. The stream callback
now reads each mount-table version before the next event, admits nothing
after the one that ends the mount, saves no offset, and drains the
admitted jobs before exiting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: ignore mount-table versions from before the mount

Without a saved offset the follow starts at the mounted directory's
creation second, so it replays mount-table versions written before
remote.mount added the directory. Each read as an unmount and the
process exited on every start. A version now ends the write-back only
when the version it replaced mapped the directory to its location.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: judge the mount table by the mount's own key

-dir may lie below a mount, and DetectMountInfo resolves it to the
enclosing mount. Looking the table up by -dir found no mapping in any
version, so a replaced table never counted as mapped and the parent's
unmount was missed, admitting its purge deletes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: a table renamed away ends the mount

A rename of /etc/remote/mount.mapping to another name inside /etc/remote
was ignored, although the filer no longer has a mount table. The table is
now matched by its exact path on both sides of the event, so renaming it
away counts as removing it, and a file of that name elsewhere under
/etc/remote no longer counts as the table.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: a mount on another remote is not the same mount

The mount-table check compared only bucket and path, so remounting the
directory on another remote storage at the same bucket and path kept
writing through the old client, and a replayed unmount from the other
remote stopped the write-back. The remote storage name now counts too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: check the drain at the exit itself

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: resolve the mount covering -dir in each table version

Judging the table by the startup mount key alone missed a mount added at
-dir itself below that key: the key stayed mapped, so the nested mount and
its later removal were both admitted, and the removal's purge deletes ran
against the startup remote. Each version is now resolved the way
DetectMountInfo resolves the live table, and a different covering mount
ends the follow as a move.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: an earlier mount's unmount still ends the follow

Skipping every version whose replaced table did not map the directory to
this mount also skipped the unmount of an earlier mount of the directory,
which a -timeAgo replay reaches, and admitted its purge deletes against
this mount's remote. Only a version in which no mount covers the
directory, replacing one in which none did, is now skipped as history
from before the mount.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: map paths below -dir under its place in the mount

With -dir below a mount, every remote key was taken relative to -dir and
appended to the mount's location, so /mount/sub/x.txt uploaded to x.txt
instead of sub/x.txt and could overwrite another object. The processor
now gets the remote location of -dir itself; the mount's own location
still judges mount-table changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: resume below a mount no earlier than the mount's offset

remote.unmount saves the offset of the directory it unmounts, after its
purge, and leaves the offsets of directories below it alone. A
write-back of -dir=/mount/sub resumed from its own offset after a
remount, replayed the unmount and exited, on every start. Without
-timeAgo it now starts no earlier than the mount's offset. Whatever that
skips was purged, or was synced by a write-back of the whole mount.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 05:06:52 -08:00
Chris LuandClaude Opus 5.5 5b1450b06b s3tables, iceberg: one catalog record per table location (#12473)
* s3tables: refuse a table or view location another record holds

A create, register, update or rename could name a location at, under or
above another catalog record's files, including the files a renamed table
left under its old name. The new record then loaded that table's metadata,
had credentials vended for its files, and could purge them on drop.

Each of those writes now scans the bucket's records and refuses a root
another record claims, and a drop of a record sharing its files is refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* iceberg: check a table's location before writing to it

REST create and stage-create write a metadata file, and stage-create
vends write credentials, before the catalog registers the table, so the
create gate now also validates the location and refuses one another
record holds. RegisterTable runs the same gate before it reads the
metadata file, instead of reading first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* iceberg: keep a dropped location another record holds

DropTable and DropView purge the dropped record's location after the
catalog delete. A view recorded at a table's location took that table's
directory, record included. The purge now leaves a location another
record still holds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: authorize CreateTable before reporting the namespace

A caller without CreateTable got 404 for a missing namespace, 403 for an
existing one and 500 with decoder text for an unreadable one. The grant is
now evaluated against the bucket first, and only a caller that passes
learns the namespace is missing or unreadable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* iceberg maintenance: skip a table whose files another record shares

Orphan removal and snapshot expiry delete what a table's metadata does not
reference. For two records over one location that deletes the other's
files, so detection leaves such tables alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: count another record's catalog entry as claimed

A table keeping its files elsewhere still owns its entry directory, and a
view recorded there would take that entry with it on drop. The ownership
check now refuses a root at, above or under another record's entry too.
Maintenance still compares files only, so a table created over a renamed
table's old name does not stop that table being maintained.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: check a claimed location again once the record is written

Two creates, or a create and an update moving a root, could both read the
same root free and both land. A create now checks again after writing its
still-marked record and backs out if another record claims the root, and an
update that moved its root checks again after the write and puts the previous
metadata back. Of two racing writers at least one sees the other.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* iceberg: enable stage-create in the location ownership test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* iceberg maintenance: check for a shared location again before removing orphans

A job can run long after its table was scheduled, or come from a detector
that never checked, so remove_orphans reads the bucket's records itself and
leaves the files alone when another record shares the table's location.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: check a renamed record's new root again once it is written

A record without a location moves its root with its name, and was checked
only before the rename wrote the destination, so a create landing on that
root in between could share it. The rename now checks again with the
destination written and backs out, like a create.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: put a moved record back when its location cannot be checked again

An update that moved its root kept the move when the check after the write
failed to list the bucket, so a racing writer that relied on that check could
share the root. The move is now undone as it is when the root is taken.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: hold a moved record's old root until the move lands

An update moving a record's root wrote the new metadata first and put the
old metadata back when the check that followed found the new root taken.
The old root was free in between, so a create racing for it could pass its
own check and then share it with the restored record.

The update now stores the metadata it is about to write beside the current
one, so the record holds both roots, checks the new root with that
reservation in place, and only then switches the metadata. A move that
backs out drops the reservation without touching the metadata, and an
update that keeps its root clears a reservation a crashed move left behind.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: test a move racing another update of the record

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: drop a move's reservation when a rename or delete takes the record

A rename or a decoupled delete that took a record while an update held its
move reservation left the reservation on the old name, where the move could
no longer clear it, and later creates of that name were refused as taken.
Both now clear it with the record's other attributes, and a failed rename
does not put it back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: register a table from metadata under its own location

The Spark register test registered a live table's metadata file under a
second name, which now fails because both records would own the source's
files. It checks that refusal instead, and registers a copy uploaded under
the new table's own directory, with its location pointing there, to cover
reading and writing a registered table.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 05:04:09 -08:00
Chris LuandClaude Opus 5.5 b650303aee S3 Vectors: weed-vector reads tables from the filers and volume servers (#12469)
* sw-client: the filer and volume client library

The client the Enterprise repository's VFS and vector engine use, with
the same API: filer gRPC for entries, listings, volume locations,
assignments and remote caching; volume HTTP, plain or TLS, for needle
bytes; chunk resolution for sparse, overlapping and manifest chunks; a
location cache; and the write side. Enterprise-only protocol fields
(license, conditional mutations, hard links, KV listing) are left out.
Compiled against weed/pb/filer.proto like the other worker crates.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-lance-worker: reads straight from the filers and volume servers

direct_read::DirectReads wraps the object store Lance opens so that
entries, listings and bytes come from the filers (gRPC) and the volume
servers (HTTP, signed with the read JWT), whatever the entry holds:
inline content, chunk manifests, sparse or overlapping chunks, the
filer's volume encryption, the gateway's SSE-S3 (sse.rs, with the KEK
from security.toml or the filer) and objects tiered to remote storage.
Writes still go to the inner store. The code and its tests are those of
the Enterprise repository's engine.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: read tables from the filers and volume servers

The engine asks the masters for the filers (and follows the list), takes
jwt.signing.read.key, grpc.client, https.client and the SSE-S3 keys from
security.toml, and opens every table with DirectReads innermost and the
read cache outside it. --masters is required: the gateway finds the
engine through them and the engine finds the filers. Writes still go to
the gateway.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* sse: chunks out of a manifest keep their own SSE-S3 keys

A chunk expanded from a manifest carries its id as fid, so keying the
per-chunk key cache by file_id gave every such chunk the first one's key
and IV. Keyed by the chunk's id however it is carried.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* direct reads: a silent filer is passed over like an unreachable one

Each filer attempt is bounded, so a filer that accepts the connection and
never answers does not hold the read or the startup lookup.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* sw-client: carry the assignment's fsync to the volume upload

The filer's storage rule may want a needle on disk before the volume
answers; the assignment says so and the upload now asks for it, as the
Go uploader does.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: argument tests pass the required masters; connect timeout for filers

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* direct reads: a failed read is tried once more against fresh locations

A fetch that fails forgets the volume locations it was made against; the
request now looks them up again and retries instead of failing on a
volume that moved.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* direct reads: the filer deadline bounds a listing page, not the listing

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: security.toml booleans read as viper reads them

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: a filer the master lists badly does not hide the others

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: register with the masters only once ready to serve

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* sw-client: the HTTP transport asks the volume to fsync when the assignment does

write_synced fell back to the trait default, which refuses fsync, so every
upload under a fsync storage rule failed before it was sent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sw-client: https.client.ca alone vouches for volume servers

With a CA configured, reqwest still trusted the system roots next to it;
weed processes trust only the configured pool.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sw-client: an empty volume range is read without a request

A zero length underflowed the Range end, and a range past u64::MAX
overflowed it; the first now returns no bytes, the second an error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sw-client: a whole-needle answer to a ranged read is cut to the range

A server or proxy that ignores Range answers 200 with the whole needle;
the read returned its first bytes in place of the ones asked for.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sw-client: a read size past u64::MAX reads to the end of the file

offset + size overflowed before the clamp to the file size.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sw-client: an SSE-encrypted manifest is refused without a decoder

Only filer-encrypted manifests were; an SSE one went to the protobuf
decoder as ciphertext.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sw-client: an empty write leaves the staged file size alone

A zero-byte write past the end grew the size to its offset, as a
zero-byte pwrite does not.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sw-client: the chunk cache keeps no empty chunks

They add no bytes, so nothing evicted them and the map grew with every
distinct empty needle, even at capacity zero.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* sw-client: flushing with a zero chunk size is an error, not a panic

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: an empty master list fails at startup

"" split into one empty address that passed the check, and the engine
then waited two minutes for a filer before a generic error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-07 04:21:13 -08:00
Chris LuandClaude Fable 5.1 2fe480a6e2 S3 Vectors: the gateway resolves the table it sends to weed-vector (#12468)
* lance catalog: in-process DescribeTable for trusted callers

DescribeTable and DescribeTableInternal resolve a table's location and
storage options without the HTTP layer; the internal form runs the lookup
through a trusted manager so the caller's catalog grants do not gate a
resolution its own permission already authorized. Same code as the
Enterprise repository.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector.proto: Table carries its location and storage options

The gateway resolves the table it addresses, so the engine can open it
without a describe round trip of its own. Both fields are optional; an
empty location means the engine looks the table up as before.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* lance catalog: test the in-process describe

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors: resolve the table before calling the engine

The gateway describes the table in process, through the trusted catalog
manager, and sends its location and storage options with every engine
call. Storage options are vended for the caller's principal, as the
Enterprise router does for queries. A catalog failure answers with the
code the engine would have given.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: open a table at the location the gateway sent

A request whose table carries a location is opened there, with the
storage options the gateway vended over the engine's fallback; the
catalog is described only for a table that comes without one, as from
an older gateway. dataset::open_at and dataset::drop_at take a
description the caller already holds.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors e2e: run the engine without a catalog

VECTOR_ENGINE_NO_CATALOG=1 gives the engine an unreachable catalog URL;
every CI leg sets it, so the suite proves the gateway resolves each table
it sends.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* lance catalog: an internal describe may write a vector index

With credential vending on, the options vended to the engine followed the
caller's s3tables grants, which never allow writing the vector namespace,
so every write the gateway sent would have run with read-only credentials
and reads would have lost their version lease. The engine is that
namespace's writer and the caller's vector permission authorized the
operation, so an internal describe of a vector table vends write-capable
options; other namespaces keep the UpdateTable check.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3: resolve tables for the engine only while the catalog runs

With -port.lance=0 the gateway has no catalog to resolve through; the
engine then keeps looking tables up itself, as before.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* lance catalog: a deregistered table describes as NoSuchTable

So the gateway answers NotFoundException for it, not InternalFailure.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: trust a sent location only for a colocated vector index

A caller that reaches the engine could otherwise name any storage the
engine's credentials cover. A location is taken from the request only
when the id names a vector index and the location is its colocated path,
which the engine writes already; anything else is described through the
catalog as before.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: scope the read cache by storage endpoint

The cache was keyed by bucket and path alone, so the same path reached
through another endpoint would have shared entries. Each open wraps the
store with the cache scoped to the endpoint its options name.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: keep a configured endpoint over one a request names

A caller that reaches the engine could otherwise send its fallback
credentials to an endpoint of the caller's choosing. When the engine's
--storage-option names an aws_endpoint, a table whose request names
another one is described through the catalog instead.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: use request options only where credentials cannot leave

A request's storage options are taken only when they hold the keys a
gateway's describe produces, so a proxy or an endpoint alias cannot route
the engine's credentials elsewhere, and an endpoint only when it is the
configured one or the engine holds no credentials to send. Anything else
sends the table to the catalog.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors e2e: give the engine its endpoint with its credentials

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: a drop checks the request's options like an open

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: an unpinned endpoint needs the request to sign for itself

Without configured credentials the S3 store still signs with the AWS
default chain, so an endpoint from a request is taken only when the
request skips signing or brings its own credentials.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: a trailing slash does not unpin the endpoint, and say so

A configured aws_endpoint written with a trailing slash matched nothing
the gateway sends, so every table fell back to the catalog in silence.
The comparison ignores the slash, and a fallback for a resolved table is
logged at debug.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: resolved by the gateway, or refused

The engine no longer has a catalog: --namespace and --filer-signing-key
are gone, a request whose table names no location is InvalidArgument, and
one whose location is not the index's own or whose storage options the
engine's credentials must not be used with is PermissionDenied. A drop
removes the colocated path it was given.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3: the catalog resolves vector tables whether or not its port is served

newLanceServer builds the Lance catalog for the S3 Vectors API; the
listener on -port.lance is optional.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors tests: start the engine without a catalog

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3: the aux listener test builds the catalog before serving it

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: an unpinned endpoint takes only credentials the request signs with alone

Lance fills an option a request leaves out from the engine's environment,
under any of object_store's names for it (TOKEN, SESSION_TOKEN, ...), and
asks the AWS default chain when the key pair is incomplete. A request that
named its own endpoint with only an access key id, or a key pair while the
engine's environment held a session token, would have sent the engine's
credentials there. Such a request now needs a full key pair, and its own
session token whenever the environment would supply one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: accept --namespace and --filer-signing-key, ignored

Both flags shipped in 4.52 and are documented, so dropping them made an
existing command line exit at startup with an unexpected-argument error.
They are kept hidden and ignored, with a warning.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3vectors design: the engine is started without a catalog

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: the trust check takes the environment it reads

The options test read the process environment through signs_alone, so a
host exporting AWS_SESSION_TOKEN or TOKEN failed it. The caller now
passes the variable names in; describe passes the engine's own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: log both endpoints when a request's options are refused

The gateway turns the PermissionDenied into a bare AccessDenied and logs
nothing, so a pinned aws_endpoint written in another form than the one the
gateway sends (host case, a default port, an IP for a name) failed every
call with no trace of which value differed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: the Lance catalog sets the credential role whether or not Iceberg runs

-iceberg.credentialRole reached the S3 server only while the Iceberg
listener started, so with -port.iceberg=0 the Lance catalog installed its
vendor but vending stayed off, and the vector engine got no credentials for
the tables the gateway resolves. Both catalogs now set the role through
one helper.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: set the credential role once, before either catalog serves

Both catalogs wrote the role as they were built; the second write landed
while the first was already serving requests that read it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-06 19:39:09 -08:00
Chris LuandClaude Opus 5.5 7beac0e8ea admin: show disk usage on the dashboard instead of the data size (#12466)
* admin: show disk usage on the dashboard instead of the data size

The dashboard's Total Size card summed the volume and EC shard sizes the
volume servers report. That leaves out deleted needles waiting for vacuum,
EC generations kept after a vacuum, and anything else on the disks, so on a
cluster with a lot of EC vacuum activity it showed far less than the disks
held.

The card now shows the bytes in use on the volume servers' disks, the same
statfs numbers the volume server list uses, with the disks' capacity, and
its trend line follows that number. Servers that report no statfs numbers
count their data size, as on the volume server list.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* admin: compute the disk usage percentage in float64

current*100 overflows int64 past about 92 PB of used space, so the
dashboard card showed a negative percentage.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: size the volume server disk bar with calculatePercent64

int(host.DiskUsage) truncates byte counts on 32-bit builds, so the bar
width was wrong for any disk over 2 GB there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: test the cluster disk totals across statfs and fallback servers

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 19:25:10 -08:00
Chris LuandClaude Opus 5.5 fd81a50c57 mount: keep retrying writeback chunk uploads through a volume outage (#12465)
* mount: keep retrying writeback chunk uploads through a volume outage

With -writebackCache, close() returns before the data is uploaded. When
the volume servers stay unreachable past the upload's own retry (~20 s),
the chunk was dropped and the file never appeared, with no error to the
application. Retry transient upload errors with the same bytes until they
succeed, unless the dirty pages were discarded. Permanent errors and the
synchronous mode keep failing as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: test writeback upload retries and the cases that still fail

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* operation: keep the filer's read-only verdict on assign failures

UploadWithRetry flattened AssignVolumeResponse errors into a string, so
callers could not tell a read-only rule from an outage. Wrap the filer's
coded error so errors.Is(err, filer_pb.ErrReadOnly) works; the message is
unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: retry writeback uploads the upload layer reassigns, never read-only

A filer proxy that cannot reach the volume answers 500 with an empty
body, which util.IsTransientError does not match, so that outage still
dropped the chunk. Also retry what operation.ShouldReassignUpload takes.
A read-only rule stays final even when its path or message contains a
transient word such as "unavailable".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: stop a writeback upload once its dirty pages are discarded

Destroy() was only seen after a failed attempt, so a truncate during the
retry wait still started another upload, and a chunk that finished after
its pages were replaced was attached to the new content. Check before
each attempt, and drop the chunk instead of attaching it; volume.fsck
reclaims it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep Destroy from slipping between the discard check and the append

truncateEntry destroys the pages and then installs the new entry, so a
Destroy landing after the check but before AddChunks put the old chunk
into the replacing entry. Hold one lock across the check and the append,
and take it in Destroy while setting the flag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: leave the Windows interrupt drain to the capped filesystem hook

Interrupt hooks run in reverse registration order, so the Windows hook ran
before the filesystem hook and waited on pending writeback flushes without a
bound. With writeback uploads now retrying through an outage, Ctrl-C could
wait until the volume servers came back. Unmount only, as on Linux, and let
the filesystem hook drain for at most 30 s and keep the write cache.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: test a writeback upload whose assign keeps timing out

With every volume server gone the assign RPC times out until the filer
client gives up; that error must be retried like a refused upload.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 19:24:42 -08:00
github-actions[bot] 7d6680c7a4 docs: regenerate star history chart 2026-11-07 00:47:27 +00:00
Chris LuandClaude Opus 5.5 99539484b6 grafana: chart the remote-deletion tombstone gauges (#12444)
* grafana: chart the remote-deletion tombstone gauges

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: describe the fetch hold that outlasts a full tombstone table

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: word the tombstone panel description like the gauge help

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 15:22:38 -08:00
Chris LuandClaude Opus 5.5 1c4fe61fcd master: stop the resumed single-master leader from checking quorum every 1 ms (#12463)
* deps: bump seaweedfs/raft so the leader quorum check follows the election timeout

A single master resuming raft state self-elects on a 1 ms election
timeout and restores the configured one afterwards, but the leader kept
running its quorum check every 1 ms for the rest of the process. The
raft change re-arms that check from the current timeout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* deps: bump seaweedfs/raft to the raft PR head with the extended test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* deps: bump seaweedfs/raft for the non-positive election timeout guard

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* deps: bump seaweedfs/raft so a timeout change re-arms the leader quorum check

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* deps: pin seaweedfs/raft v1.2.4

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 15:22:06 -08:00
Chris LuandClaude Opus 5.5 cdea7e79e0 filer: fold small chunks into manifests 256 at a time (#12462)
* filer: fold small chunks into manifests 256 at a time

Every metadata event carries the file's whole chunk list, so a file grown
by small appends re-sent each chunk record on every append until the
list reached ManifestBatch (10000). Chunks under 1 MiB now fold once 256
of them build up; full-size chunks still fold at ManifestBatch, so large
files keep their flat lists and read without a manifest fetch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: merge appended small chunks at the small-chunk fold count

The append path merges small chunks into full-size ones, but only at
1000 of them; the fold now hides small chunks in a manifest at 256, so
the merge would never see enough of them. Merging at the same count
keeps rewriting them into full chunks first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 13:41:02 -08:00
Chris LuandClaude Opus 5.5 728b476ba2 filer.remote.gateway: keep the remote object while a version under a delete marker is locked (#12459)
* filer.remote.gateway: keep the remote object while a version under a delete marker is locked

Every version of a versioned bucket uploads to the object's key, so on a
remote without versioning that object is the only remote copy of the
latest version. A delete marker deleted it even when a version still held
a legal hold or unexpired retention. Keep it while any version of the key
is locked, the same rule the filer uses before pruning a remote entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.gateway, filer.remote.sync: say that earlier versions need remote versioning

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.gateway: delete a kept delete marker's object once the version lock is released

A delete marker kept for a locked version stays unstamped. When an
update or delete leaves a version that carried a legal hold or retention
without an active lock, look up the key's latest version and, if it is
still that unstamped marker, run the delete again. Only changes to
versions that carried a lock trigger it, so the noncurrent mark and tag
updates of unlocked versions do not send repeat remote deletes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.gateway: test the kept delete marker retry through the bucket mount

The retry reads the key's versions through a filer client parameter, as
the rename helpers do, so a test can check the mapped object key and the
marker stamp, and that lock-releasing updates and deletes reach it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.gateway: check the lock release lookup goes to the gateway's filer

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 13:37:08 -08:00
Chris LuandClaude Opus 5.5 8c22336944 filer.meta.backup: stream from the start of an unfinished walk (#12458)
* filer.meta.backup: stream from the start of an unfinished walk

A walk only inserts entries, so when the first walk is interrupted the
next start walks again and streams from the second walk's start: deletes
and renames between the two walks never reach the backup store. Keep the
first walk's start time in the store until a walk completes and stream
from it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.meta.backup: start fresh once the stream has passed an earlier walk's start

A walk start left in the store by a crash between saving the offset and
clearing it, or by an interrupted -restart followed by a plain restart,
stops mattering once the stream saves an offset past it. Ignore it then
instead of replaying from it on the next walk.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.meta.backup: fail the walk on an unreadable offset

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 13:32:46 -08:00
jsasandjames 5ba97b4432 filer: a full remote-deletion tombstone table keeps the lazy fetch off (#12440)
* filer: a full remote-deletion tombstone table keeps the lazy fetch off

At the cap a new delete under a remote mount was dropped with a log line
and isRemoteDeletionPending answered false for it, so the lazy remote fetch
resurrected deleted paths. While the table is full every path may be a
pending delete: isRemoteDeletionPending answers true until the table has
room again. The size and the full state are gauges
(filer_remote_deletion_tombstones, filer_remote_deletion_tombstones_full).

(cherry picked from commit f3be0ab965cde3bfd3053d3ac720fbbe3f68e98b)

* filer: a full tombstone table drains on reads and lifts after the unrecorded deletes age out

While the table is full, a lookup miss evicts expired tombstones and those
the write-back offset has passed, so the table drains without a mutation.
The full state lifts only once the table has room and the newest
unrecorded delete is past the TTL: an unrecorded delete has no event to
confirm it, so only the TTL makes it safe, as for a pending tombstone.

(cherry picked from commit 19394fe5d67798af7b0d755ac18c83c75e20a12b)

* filer: a full tombstone table drains per mount, only when its offset advances

Offset-based release applies to the tombstones under the mount whose
offset was read; another mount's tombstones answer to their own offset. A
read sweeps the table only once that offset advanced since the last drain
for the mount or the unrecorded deletes aged past the TTL, so a listing of
many missing children does not rescan the table per child.

(cherry picked from commit 9cf00e2e54d70e6ba530e6ec1b1ff4b502873847)

* filer: the tombstones-full gauge help names when it clears

---------

Co-authored-by: james <james@medable.com>
2026-11-06 13:28:47 -08:00
7fc33e7fe3 filer: no lazy remote fetch inside a store transaction (#12439)
* filer: no lazy remote fetch inside a store transaction

A rename onto an existing target deletes the target inside the store
transaction, then CreateEntry's FindEntry misses the row and runs the lazy
remote fetch, which persists the archive copy on a second connection. The
store wrapper drops that call's deadline, so the UPDATE waits without bound
on the row lock of the transaction's own uncommitted DELETE while the
rename goroutine holds the path's exclusive locks: every later lookup,
rename and delete of the path on that filer waits forever.

A lookup whose context carries the store transaction skips the fetch.

(cherry picked from commit b69e9535c2597d0c8e856f031d3778fddfd4159c)

* filer: materialize remote-only rename paths before the transaction

With the lazy fetch off inside a store transaction, a source or target the
filer holds only on the remote would read as absent to AtomicRenameEntry
and StreamRenameEntry: a rename with FailIfTargetExists could overwrite it,
and a remote-only source would skip the deferred copy. PrefetchRemoteEntries
runs the fetch for both paths before BeginTransaction, under the path locks.

(cherry picked from commit e61f9718018669ef59768cd5eda25b6466d69597)

* filer: skip the lazy remote fetch only in CreateEntry's lookup inside a transaction

The guard on every lookup inside a store transaction hid a source or target
the filer holds only on the remote from the rename, and the prefetch before
the transaction did not reach the children of a directory rename. The
deadlock needs a write: CreateEntry's lookup after the transaction deleted
the path. That lookup alone skips the fetch; reads inside the transaction
keep it, and their persist on a second connection holds no lock the
transaction waits for.

(cherry picked from commit f555092c324780a8fe5314b238694551e3ef6be5)

* filer: mark a store transaction independently of the store's own key

ArangoDB and FoundationDB keep their transaction under other context
keys, so the CreateEntry guard keyed on "tx" missed them. The filer now
marks every context returned by BeginTransaction and the guard checks
that marker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit bd0505eb80)

* filer: keep the remote check for an exclusive create in a transaction

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit b6aac15c3b)

* filer: test that a create outside a transaction still consults the remote

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 972daddd84)

---------

Co-authored-by: james <james@medable.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 13:21:37 -08:00
Chris LuandClaude Opus 5.5 86db60b669 s3api: mint a versioned PUT's id at commit, after the lock wait and the upload (#12454)
* s3api: mint a versioned PUT's id after the object write lock

putVersionedObject minted the version id before the upload and the lock
wait. A DELETE that committed while the PUT waited got a newer id, so a
conditional PUT whose If-None-Match passed against the delete marker was
stored behind it: accepted, but GET returned NoSuchKey.

The finalize now re-mints the id once the lock is held and the
precondition passed, so the version orders after everything the check
saw. An ambiguous routed attempt keeps its id, since recovery looks for
that entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: mint a routed versioned PUT's id after the upload

On the routed path the version id was still the one minted before the
upload, and RECOMPUTE_LATEST picks the newest name. A DELETE that
committed during a long upload left its marker latest over the accepted
PUT. Re-mint right before the routed transaction, as the lock path does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 13:20:36 -08:00
1176f975b3 filer: release metadata log chunks a failed append leaves unreferenced (#12455)
* filer: stop unbounded metadata-log growth on append-only files

Two independent leaks make a filer metadata log grow far faster than the
data it describes. On a cluster whose live logs weigh a few hundred
megabytes, they produced tens of gigabytes of metadata per day.

1. Metadata events embed the whole old and new entry, chunk lists
   included. An append-only file — a log or a journal written in small
   pieces — therefore re-serializes every chunk it ever had on every
   append, so the cost is quadratic in the number of chunks. A
   2,700-chunk file emits a couple of megabytes per append. Cap the
   carried chunk list at 256 entries and set
   metadata_event_chunks_truncated so subscribers that need the content
   read the entry from the filer instead of trusting the event.
   Subscribers that only track names, attributes, or chunk deltas keep
   working unchanged.

2. appendToFile uploads the chunk before committing the filer entry.
   When the commit fails — store error, cancelled context, filer
   shutting down — the chunk is stranded: no entry references it, so no
   later delete can mark it, and neither vacuum nor fsck can see it
   because both work from the entry index. The volume server keeps
   counting those bytes as live data forever. Release the chunk when
   the commit fails.

* filer: keep chunk lists in metadata events

Mount meta caches, peer filers replaying into their own store, and the
filer.sync, filer.backup and filer.meta.backup sinks all store the event's
entry as the file. Dropping the chunk list past a count left those copies
of a large file without content.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: release a metadata log chunk the lookup or commit leaves unreferenced

The flush retries a failed append with a new upload, so a store outage
left one stranded chunk per retry. A failed lookup strands the chunk too.
Use DeleteUncommittedChunks, as the other write paths do: the chunk was
never part of the file, so path chunk-deletion rules do not apply.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: keep a metadata log chunk the failed create stored

CreateEntry inserts a new entry before its parent folders, so a parent
failure returns an error with the entry already referencing the chunk.
Release it only when the store shows no entry, or an entry without it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: count a metadata log piece the failed create stored as appended

The flush retry would append the same bytes after it, and a piece cut
inside a record then breaks replay of that segment. The retry takes the
update path, which does not create parent folders either, so returning
the error only duplicated the piece.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Martin Morez <martin@morez.org>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 13:19:09 -08:00
Chris LuandClaude Opus 5.5 2be4498c04 s3 lifecycle: walker deletes carry the listed identity and must target the latest version (#12452)
* s3 lifecycle: send the listed entry's identity on walker deletes

The daily walker dispatched LifecycleDelete with no ExpectedIdentity, so
the server skipped its compare-and-swap and an object rewritten between
listing and dispatch was deleted. The list adapter now records each
listed entry's identity, computed the same way as on the event path, and
the walker sends it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: current-version expiration must target the latest version

The walker addresses the latest version of a versioned key by id. A
version file never changes, so its identity still matches after a newer
PUT, and the delete went ahead: a delete marker over the new version with
versioning enabled, or the new null version deleted while suspended.
LifecycleDelete now resolves the key's latest version for
EXPIRATION_DAYS/EXPIRATION_DATE requests that carry a version id and
answers NOOP_RESOLVED (VERSION_NOT_LATEST) when it is a different one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: check the latest version for unaddressed suspended expirations

With versioning suspended, an expiration without a version id reads the
plain entry, which is the null version but not necessarily the latest:
a newer real version can sit in .versions. The latest-version check now
runs in the expiration branch, by versioning state: for every suspended
request (an unaddressed one addresses the null version) and for enabled
requests that carry a version id.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: test that each version sibling carries its own identity

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 09:39:12 -08:00
Chris LuandClaude Opus 5.5 9af736da62 volume: stop serving EC reads from quarantined shards (#12450)
* storage: skip quarantined EC shards on local reads

A needle read tried the local shard first even when its disk was
quarantined after repeated IO errors, so every read paid the failing
disk's EIO latency before falling back. Treat such a shard as not local
so the read goes to peers or reconstruction, and read it only as a last
resort when no healthy copy answers. Scrub still reads it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: refuse EC shard reads from a quarantined disk

VolumeEcShardRead served peers from a shard whose disk was quarantined,
and it ignored the .ecx lookup error: a failed index read returns a
tombstone size, so the peer was told a live needle was deleted. Refuse
quarantined shards and fail the request on an index read error; a
needle missing from the index still reads, as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: report quarantined EC shards deleted right away

Quarantining an EC volume only dropped it from later full EC heartbeats,
so the master kept routing reads to the faulty disk for up to 17
heartbeat intervals. The heartbeat that quarantines the volume now
carries its shards as deleted, once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: do not announce EC shards mounted onto a quarantined volume

MountEcShards and the shard load notifier reused a quarantined EcVolume
and announced the new shard, pointing the master back at the faulty disk
until the next full EC heartbeat. Skip the announcement, and fail the
mount so a caller moving a shard here keeps its source copy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: count IO errors on local EC shard reads

Go's readLocalEcShardInterval feeds every local shard read into the EC
volume's IO error streak; the Rust needle read and recovery gather did
not, so a failing disk under local reads never moved toward quarantine.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: skip quarantined EC shards on local reads

Mirror the Go read path: a needle read and the recovery gather treat a
shard on a quarantined disk as not local, and read it only as a last
resort once peers and reconstruction fail. Scrub still reads it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: refuse EC shard reads from a quarantined disk

Mirror Go's VolumeEcShardRead: answer Unavailable for a shard whose disk
is quarantined, so the peer tries the next holder or reconstructs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: report quarantined EC shards deleted right away

Mirror Go: the volume heartbeat that quarantines an EC volume carries its
shards as deleted, once, instead of leaving the master to drop them at
the next full EC heartbeat.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: fail EC shard mounts onto a quarantined volume

Mirror Go's MountEcShards: the heartbeat already leaves such a shard
unannounced, so fail the mount and let a caller moving the shard here
keep its source copy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: keep reading EC needles when the shard lookup fails

A failed or short master lookup aborted the read before any shard was
tried, so the quarantined-disk last resort never ran once the master had
dropped those shards, and local shards went unread whenever the master
was unreachable. Read on with the cached locations, as the Rust server
already does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec: log a failed last-resort read of a quarantined shard

When reconstruction and the quarantined read both fail, say why the local
read failed too. Go and Rust.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec: drop a shard whose mount was refused on a quarantined disk

The refused mount left the new shard loaded on the quarantined volume,
where it was neither served nor announced, and the in-memory claim drew
later copies of that shard back to the same disk. Unload it again,
keeping a shard that was already mounted there. Go and Rust.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec: finish a mount batch before reporting a failed shard

A shard that failed to mount, now including one refused on a quarantined
disk, returned before the sidecar reload, and in Rust before the
heartbeat notify, so shards mounted earlier in the same batch kept
checksum protection off and reached the master only at the next full EC
heartbeat. Stop at the failed shard but finish the rest first. Go and
Rust.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec: keep a peer's deleted answer when its stream fails after it

A shard read recorded the holder's deleted answer, and a stream error
after it failed the read: Rust discarded the flag, and Go let the next
source or the reconstruction result overwrite it. With the last-resort
read of a quarantined shard, that could serve a needle a peer had
reported deleted. Once the deleted answer is in, end the read on a later
stream error instead of failing it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: use the platform media error in the quarantine heartbeat test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: rebuild EC intervals from quarantined shards when healthy ones fall short

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: rebuild EC intervals from quarantined shards when healthy ones fall short

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: serve a peer's EC shard read from a healthy copy on another disk

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: serve a peer's EC shard read from a healthy copy on another disk

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: refuse a quarantined EC shard read instead of serving a sibling copy

A sibling disk's copy can have its own deletion journal, and an EC blob
delete is journaled on one local copy only, so the sibling could serve a
deleted needle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: refuse a quarantined EC shard read instead of serving a sibling copy

A sibling disk's copy can have its own deletion journal, and
VolumeEcBlobDelete journals a delete on one local copy only, so the
sibling could serve a deleted needle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: check the disk an EC shard mounted on for quarantine

mount_ec_shard now returns the disk it mounted on, and the mount RPC
checks only that disk, like Go's MountEcShards. Before, a quarantined
sibling holding the same shard failed the mount while the healthy
disk's copy stayed mounted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: test an EC mount beside a quarantined sibling

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: check the quarantined-sibling mount test by its result

A server that remounts a shard on the disk already claiming it refuses
the mount there instead. The test now only requires that the result
matches whether the healthy disk ended up holding the shard.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: check the quarantined-sibling mount test by its result

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: expect the quarantined-sibling mount result from the disk the store picks

The test accepted a refused mount whenever the healthy disk stayed empty,
so a handler that refused before mounting passed too. Ask the store which
disk it picks first, then require success exactly when that is the healthy
disk.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: expect the quarantined-sibling mount result from the disk the store picks

Mirrors the Go test. add_location's cross-disk reconcile mounted shard 1 on
the healthy disk too, so the test now unmounts it there first, leaving the
quarantined sibling as the only claimant, as in Go.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 09:38:15 -08:00
2df7d435f2 filer.remote.sync: bound the stamp and offset writes, startup offset fixes, stamp a copy (#12447)
* filer.remote.sync: bounded filer lookups, one chunk cache per process

The filer lookups on the sync path (currentEntry, isSuperseded,
uploadCurrentEntry, liveRemoteEntry, the mount directory read) ran with no
deadline. A lookup that never answers keeps its job alive, the job keeps
its path in the MetadataProcessor conflict index, AddSyncJob waits on the
condition and the daemon stops consuming events with no log line. Each
lookup now carries a 30 s deadline, so the job fails and the processor's
stop-and-resubscribe path runs.

remoteUploadReader built a chunk cache per upload; the cache's worker
goroutine is never stopped, so each upload leaked one goroutine and its
cache (21k goroutines, ~150 MB/h under lock-file churn, OOM at the memory
limit). One chunk cache per process.

(cherry picked from commit ff77fd222772d4f3b68f34d5ee39a66684f90e78)

* filer.remote.sync: a failed mount lookup at startup never resolves to now

collectLastSyncOffset returned time.Now() when the mounted directory lookup
failed, which skips every event since the saved offset. The saved offset is
used instead; without one the daemon exits so a restart retries.

(cherry picked from commit 8fd3124f5b3d3f8c3a291263a710047b4aff81ae)

* filer.remote.sync: no upload on an unanswered supersession lookup; bounded offset read

retriedWriteFile returns the supersession lookup's error instead of
treating an unanswered lookup as "not superseded": in a replay an older
snapshot could otherwise overwrite a newer remote object. The processor
retries the event. isSuperseded keeps its boolean form for the other call
sites. GetSyncOffset's KvGet carries a 30 s deadline so a startup that falls
back to the saved offset cannot hang on it.

TestGatewayUncachedRewriteIsStillUploaded asserted the upload on a failed
lookup; it now asserts the withheld upload and the lookup error.

(cherry picked from commit 8fc2e718f7cd8322615b4e40007be903ade2dbfa)

* filer.remote.sync test: keep the slices import the other gateway tests use

* filer.remote.sync: an unanswered lookup after a failed attempt ends the retry loop; an unread offset never resolves to now minus timeAgo

After a failed upload attempt the supersession lookup's error ended in
isSuperseded's boolean and the loop could upload the snapshot again
unverified. The lookup error now ends the loop (errLookupUnanswered, not
retried) and the processor retries the event. With -timeAgo set, a failed
read of the saved offset resolved to now minus timeAgo, skipping everything
older; the daemon exits instead so a restart retries.

(cherry picked from commit 55ca2a88e01d7b556d494a404b9cc263c27b2566)

* filer.remote.sync: an unresolved start offset returns to the retry loop instead of exiting

resolveLastSyncOffset returns the lookup or offset-read error; both callers
(directory and buckets) hand it to util.RetryUntil, which retries with
backoff instead of the process exiting.

* filer.remote.sync: keep doc comments on the functions they describe

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: isSuperseded returns the lookup error itself

Its only production callers needed the error, so the boolean wrapper was
left to the tests. The tests now assert the error on a failed lookup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: bound the stamp written after each upload

updateLocalEntry's UpdateEntry was the last filer call on the sync path
without a deadline. A filer that never answers the stamp keeps the job and
its path in the conflict index the same way an unanswered lookup did. The
stamp carries IF_ENTRY_EQUAL, so a retried event after a timeout either
stamps again or skips a stamp that already landed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: an unread offset never resolves to the mount's creation time

With the offset read now bounded, a timed-out read fell through to the
directory's creation time and replayed every event already synced. The
error goes back to the startup retry loop, as with -timeAgo; a missing
offset still reads as zero without an error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: stamp a copy of the event entry

updateLocalEntry set the RemoteEntry and dropped the rename marker on the
caller's entry before the RPC. A transient stamp failure retries the same
event in place, which then read as already synced and skipped both the
upload and the stamp. The stub filer now applies a successful update the
way the filer does, since tests had relied on that shared object.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: a directory not created yet starts now, as before

A gateway started before the first bucket looked up a missing buckets
directory, had no saved offset, and returned an error to the retry loop
on every attempt until a bucket appeared. A missing directory with no
saved offset has nothing to replay, so it starts now as it did before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: take the start time before the mount lookup

A bucket created between a not-found lookup and the end of the offset
read had its creation before a start taken afterwards, so the gateway
never created its remote bucket.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: save the resume position when a subscription ends

A failed event ends the subscription with ErrResubscribe, and the retry
loop runs startup again, recomputing -timeAgo from a later now. The
periodic checkpoint only runs as events arrive, so a failure before the
first one could start the retry after the event it failed on. The resume
position is now saved whenever the subscription returns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* remote_storage: bound the sync offset write like the read

The offset write runs in the subscription's checkpoint callback and, now,
when a subscription ends; a filer that never answers it stalled either.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: say that any saved offset wins over the creation time

The creation time in seconds was compared with the offset in
milliseconds, which made any saved offset win. That is the behaviour to
keep: remote.unmount recreates the directory and then saves the offset,
usually in the same second, so a seconds comparison would start at the
creation second and replay the purge. The comparison now states it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: keep the resume position for the retry when its write fails

The retry in the same process now resumes from the position the ended
subscription returned, as filer.backup does, so a failed offset write
followed by an empty read cannot recompute -timeAgo past a failed event.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync test: give the same-second saved offset a fractional part

With whole seconds, returning the creation second resolved to the same
time and passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: james <james@medable.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 09:36:06 -08:00
206d7b1418 filer.remote.sync: bounded filer lookups, one chunk cache per process (#12441)
* filer.remote.sync: bounded filer lookups, one chunk cache per process

The filer lookups on the sync path (currentEntry, isSuperseded,
uploadCurrentEntry, liveRemoteEntry, the mount directory read) ran with no
deadline. A lookup that never answers keeps its job alive, the job keeps
its path in the MetadataProcessor conflict index, AddSyncJob waits on the
condition and the daemon stops consuming events with no log line. Each
lookup now carries a 30 s deadline, so the job fails and the processor's
stop-and-resubscribe path runs.

remoteUploadReader built a chunk cache per upload; the cache's worker
goroutine is never stopped, so each upload leaked one goroutine and its
cache (21k goroutines, ~150 MB/h under lock-file churn, OOM at the memory
limit). One chunk cache per process.

(cherry picked from commit ff77fd222772d4f3b68f34d5ee39a66684f90e78)

* filer.remote.sync: a failed mount lookup at startup never resolves to now

collectLastSyncOffset returned time.Now() when the mounted directory lookup
failed, which skips every event since the saved offset. The saved offset is
used instead; without one the daemon exits so a restart retries.

(cherry picked from commit 8fd3124f5b3d3f8c3a291263a710047b4aff81ae)

* filer.remote.sync: no upload on an unanswered supersession lookup; bounded offset read

retriedWriteFile returns the supersession lookup's error instead of
treating an unanswered lookup as "not superseded": in a replay an older
snapshot could otherwise overwrite a newer remote object. The processor
retries the event. isSuperseded keeps its boolean form for the other call
sites. GetSyncOffset's KvGet carries a 30 s deadline so a startup that falls
back to the saved offset cannot hang on it.

TestGatewayUncachedRewriteIsStillUploaded asserted the upload on a failed
lookup; it now asserts the withheld upload and the lookup error.

(cherry picked from commit 8fc2e718f7cd8322615b4e40007be903ade2dbfa)

* filer.remote.sync test: keep the slices import the other gateway tests use

* filer.remote.sync: an unanswered lookup after a failed attempt ends the retry loop; an unread offset never resolves to now minus timeAgo

After a failed upload attempt the supersession lookup's error ended in
isSuperseded's boolean and the loop could upload the snapshot again
unverified. The lookup error now ends the loop (errLookupUnanswered, not
retried) and the processor retries the event. With -timeAgo set, a failed
read of the saved offset resolved to now minus timeAgo, skipping everything
older; the daemon exits instead so a restart retries.

(cherry picked from commit 55ca2a88e01d7b556d494a404b9cc263c27b2566)

* filer.remote.sync: an unresolved start offset returns to the retry loop instead of exiting

resolveLastSyncOffset returns the lookup or offset-read error; both callers
(directory and buckets) hand it to util.RetryUntil, which retries with
backoff instead of the process exiting.

* filer.remote.sync: keep doc comments on the functions they describe

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 952d80c4d3)

* filer.remote.sync: isSuperseded returns the lookup error itself

Its only production callers needed the error, so the boolean wrapper was
left to the tests. The tests now assert the error on a failed lookup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 51f603b5ec)

* filer.remote.sync: bound the stamp written after each upload

updateLocalEntry's UpdateEntry was the last filer call on the sync path
without a deadline. A filer that never answers the stamp keeps the job and
its path in the conflict index the same way an unanswered lookup did. The
stamp carries IF_ENTRY_EQUAL, so a retried event after a timeout either
stamps again or skips a stamp that already landed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit c7338fb68f)

* filer.remote.sync: an unread offset never resolves to the mount's creation time

With the offset read now bounded, a timed-out read fell through to the
directory's creation time and replayed every event already synced. The
error goes back to the startup retry loop, as with -timeAgo; a missing
offset still reads as zero without an error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 0a1b6c2944)

* filer.remote.sync: stamp a copy of the event entry

updateLocalEntry set the RemoteEntry and dropped the rename marker on the
caller's entry before the RPC. A transient stamp failure retries the same
event in place, which then read as already synced and skipped both the
upload and the stamp. The stub filer now applies a successful update the
way the filer does, since tests had relied on that shared object.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 1f332e5866)

* filer.remote.sync: a directory not created yet starts now, as before

A gateway started before the first bucket looked up a missing buckets
directory, had no saved offset, and returned an error to the retry loop
on every attempt until a bucket appeared. A missing directory with no
saved offset has nothing to replay, so it starts now as it did before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit c31497a9a3)

* filer.remote.sync: take the start time before the mount lookup

A bucket created between a not-found lookup and the end of the offset
read had its creation before a start taken afterwards, so the gateway
never created its remote bucket.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit f535fca8f6)

* filer.remote.sync: save the resume position when a subscription ends

A failed event ends the subscription with ErrResubscribe, and the retry
loop runs startup again, recomputing -timeAgo from a later now. The
periodic checkpoint only runs as events arrive, so a failure before the
first one could start the retry after the event it failed on. The resume
position is now saved whenever the subscription returns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 7be919ded1)

* remote_storage: bound the sync offset write like the read

The offset write runs in the subscription's checkpoint callback and, now,
when a subscription ends; a filer that never answers it stalled either.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit f888ac6b8f)

* filer.remote.sync: say that any saved offset wins over the creation time

The creation time in seconds was compared with the offset in
milliseconds, which made any saved offset win. That is the behaviour to
keep: remote.unmount recreates the directory and then saves the offset,
usually in the same second, so a seconds comparison would start at the
creation second and replay the purge. The comparison now states it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit a0d60ee00f)

* filer.remote.sync: keep the resume position for the retry when its write fails

The retry in the same process now resumes from the position the ended
subscription returned, as filer.backup does, so a failed offset write
followed by an empty read cannot recompute -timeAgo past a failed event.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 39ef551bcf)

* filer.remote.sync test: give the same-second saved offset a fractional part

With whole seconds, returning the creation second resolved to the same
time and passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 4bb74382a6)

---------

Co-authored-by: james <james@medable.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 09:35:16 -08:00
Chris LuandClaude Opus 5.5 1406335301 volume: a delete cancels writes of its needle still in flight (#12445)
* volume: a delete cancels the writes of its needle still in flight

A write queued behind the volume locks, for example behind another write's
fsync on a busy disk, is not in the needle map yet. A delete of that needle
in the meantime finds nothing, and the write lands afterwards, so the needle
comes back on this copy only while the other replicas have it deleted.

Each volume now tracks its writes from entry until they return. A delete
cancels the ones for its needle, and a cancelled write that landed is
tombstoned before it returns, unless the needle was deleted or rewritten
since.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: a replicate delete that finds nothing cancels the pending write

Rolling back a failed chunk upload sends DELETE ?type=replicate to every
holder, including the slow one whose write timed out on the client. That
holder has not indexed the write yet, so the delete answered 404 and the
write landed live afterwards. Cancel the needle's pending writes, then look
again in case one finished in between. A plain delete that finds nothing
still answers 404 without cancelling: it is not sent on to the replicas.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: a delete cancels the writes of its needle still in flight

Mirrors the Go volume. Volume::track_pending_write registers a write until
its guard drops, delete_needle cancels the needle's pending writes, and
settle_pending_write tombstones a cancelled write that landed unless the
needle was deleted or rewritten since.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: track uploads and cancel them on a replicate delete

An upload is tracked from the handler, and a durable one from submission
to the write queue, where the guard travels with the request: a handler
dropped while the write waits in the queue does not untrack it, and the
worker settles it after the batch. A replicate delete that finds nothing
cancels the needle's pending writes and looks again, like the Go handler.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: take a cancelled write back in the lock span that publishes it

The write tombstoned itself only after it returned, with the volume locks
released. In that gap a compaction could move the record so the offset
check skipped it, a close could drop it, and an identical write sent after
the delete could match the live record and report success before the
tombstone removed it. The check now runs right after the publish, under
the same dataFileSyncLock and dataFileAccessLock hold, in the plain, fsync
and batch write paths.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: a delete cancels only the writes carrying its cookie

A delete with the right needle id but the wrong cookie could cancel an
in-flight write and get it tombstoned. Each pending write now records its
cookie, and a cancel only marks the matching ones.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: take a cancelled write back in the lock span that publishes it

Mirror the Go change: settle_pending_write ran after write_needles_grouped
returned, so a compaction, a close or an identical write later in the same
batch could slip in before the tombstone. drop_cancelled_write now runs
right after the publish, under the same sync and access lock hold, in the
synced run, the unsynced run and write_needle_unsynced_or_refused.
write_pending_needles carries each write's PendingWrite down to that
point.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: a delete cancels only the writes carrying its cookie

Mirror the Go change: a delete with the right needle id and the wrong
cookie no longer cancels an in-flight write. Each tracked write keeps its
cookie next to its cancel flag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: refuse a queued write whose volume was remounted

The write queue tracks a write on the volume found at submit, but the
worker looks the volume up again by id. After a remount, a delete reaches
the new volume and cannot cancel the write registered on the old one, so
the worker now refuses such a write instead of landing it on the new
volume.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: decode the delete's type parameter like Go's FormValue

The delete handler matched the raw query text for type=replicate, so a
percent-encoded name or value was not treated as a replica delete and did
not cancel the write it undoes. Parse the query the way the upload handler
does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: test a replica delete landing in a write's fsync window

Drive DeleteHandler while a durable write is parked in its .dat fsync,
appended but not indexed: the replica delete answers 404 and the write is
taken back when it publishes, while a plain delete leaves it live.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: flush a take-back tombstone before the index row that points at it

A durable write taken back during its fsync appends the tombstone after the
.dat sync, and the .idx sync that follows made its row durable while the
record itself could still be lost, which reloads the volume read only. The
fsync and batch paths now sync the .dat again first when a take-back
appended a tombstone; a failure stops the volume taking writes like a failed
index sync.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: flush a take-back tombstone before the index row that points at it

A durable write taken back in write_synced_run appends its tombstone after
the .dat sync, so the .idx flush that follows could make the row durable
while the tombstone record itself was still in the page cache. The run now
syncs the .dat again first when a take-back appended a tombstone; a failure
stops the volume taking writes like a failed index flush.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: a skip-cookie batch delete cancels the writes of the file id it names

BatchDelete with SkipCookieCheck left the needle's cookie at zero, so the
delete's cancel matched none of the in-flight writes and a retry queued
behind it could land afterwards. The non-EC delete now carries the file
id's cookie, as the Rust volume server already does; the EC path keeps
cookie 0, which DeleteEcShardNeedle reads as "skip the check".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: release a held test write before the store closes

A failed wait ended the test before release ran, so the writer stayed parked
in its fsync and the store cleanup blocked on its lock until the package
timed out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 09:34:20 -08:00
f71c0fe523 fix(s3tables): route REST RenameTable and refuse unrouted S3 Tables requests (#12446)
* s3tables: route REST RenameTable and refuse unrouted S3 Tables requests

PUT /tables/<arn>/<ns>/<name>/rename had no route, so it fell through to the
/{bucket} routers and ran as a PutObject into a bucket named "tables",
answering 2xx while the table kept its name.

- Answer any s3tables-signed request no S3 Tables route takes with 404
  UnknownOperationException, before the health, domain and bucket routers.
- Route the AWS RenameTable REST call (newNamespaceName, newName, both
  optional; versionToken).
- Honor versionToken on RenameTable, checked against the snapshot the claim
  asserts so a commit between the two reads cannot slip a stale token through.

* s3tables: match the credential scope's service, not a region named s3tables

The unrouted-request fallback widens what a false match costs: an S3
client whose region is named s3tables had every request answered as an
unknown S3 Tables operation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: refuse an empty newName or newNamespaceName on REST rename

An empty field was read as omitted, so {"newNamespaceName":"other",
"newName":""} moved the table under its old name instead of failing
validation the way AWS does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: check the rename version token after the snapshot comparison

Once the separately read metadata is known to equal the claimed snapshot,
the token compares against the parsed metadata; no second decode of the
snapshot bytes is needed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables test: cover policy and tag changes in the rename snapshot compare

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables test: give each REST rename subtest its own source table

The token subtests renamed the table the first subtest produced, so run
alone they failed on a missing table instead of testing the token.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 09:32:13 -08:00
Chris LuandClaude Opus 5.5 c84c43f3da master: export Hashicorp raft metrics through the go-metrics global raft uses (#12448)
* master: keep the hostname out of raft metric names

go-metrics prefixes gauge names with the hostname by default, which gives
every master its own metric family, e.g. SeaweedFS_<host>_raft_peers.
Prometheus already tells masters apart by the scrape target.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* master: send Hashicorp raft metrics through the go-metrics global raft uses

hashicorp/raft and raft-boltdb report through github.com/hashicorp/go-metrics,
but the master set up its prometheus sink on the armon/go-metrics global,
so a -raftHashicorp master exported no raft metric families. The sink is now
registered on the hashicorp/go-metrics global, and armon/go-metrics is no
longer a dependency.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: chart raft commit time and leader contact

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: keep the cluster in the raft panels

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: average raft commit time and leader contact over the rate window

The go-metrics summaries keep quantiles for 10s, so a p99 scraped every
30s misses most observations. The _sum/_count rates count every sample.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: leave raft panels empty when nothing was observed

Clamping the count rate plotted 0 ms for windows without commits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* master: let the raft metrics test use its own registry

Registering on stats.Gather made go test -count=2 fail with a duplicate
collector registration.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* master: register raft metrics before opening the raft stores

raft.NewRaft starts its run loop before returning, so registering the
sink afterwards could drop the first state and boltdb samples.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 09:29:42 -08:00
Chris LuandClaude Opus 5.5 7b3983f154 s3api: speed up recursive listing of branched directory trees (#12438)
* s3api: look up a trailing-slash prefix's directory instead of listing its parent

A prefix such as "account-0001/" names one directory, but the walk found it by
listing the parent with the name as a prefix. SQL stores answer that with a LIKE
that bounds the scan from below only, so in a wide bucket root the query reads
every sibling after the name: seconds for a prefix holding a few keys.

Look the directory up by name and serve it as the level's only entry. A miss,
an error or a non-directory falls back to the prefix listing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: track failover receives atomically

The failover wrapper flags a received reply through a plain bool, which a
callback reading from the filer on several goroutines would race on. Use an
atomic.Bool so the callback may share its filer client.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: read directories ahead of a recursive listing walk

A recursive ListObjects walk enters directories one at a time, so a bucket
made of many directories holding one or two keys each costs one filer
round trip per directory, in series: seconds per 1000-key page on a store
whose reads wait on disk.

Read the first window of the directories the walk will enter on a few
workers, nearest first in walk order, and hand each window over when the
walk gets there. Every window the walk reads itself is drained first so its
subdirectories are queued before the walk descends into the first one. The
number of windows held ahead grows while directories end inside their first
window and falls back when one does not, so a page ending inside a large
directory leaves few reads wasted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: drop the prefetcher's record of every directory it was handed

A page can cross any number of empty directories without spending its key
budget, so a set of every announced directory grows with the tree. A
directory is already handled when it is at or behind the walk, held or in
flight, or queued; check those, which the queue and window caps bound.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: end a trailing-slash prefix listing at the file its lookup finds

A file named by the prefix lists nothing under it, the same as the
sibling listing it fell back to, which threw the lookup's answer away
and paid the parent scan anyway.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: read a walk's own listing ahead instead of draining it

Draining the window before the walk took its first entry made a page
wait for the end of the listing, which a SQL store answering a name
prefix reaches only after scanning the rest of the directory; a page
that fills before then used to return at once. A background reader now
hands the window's directories to the prefetcher as they arrive while
the walk takes each entry, holding at most 1 MiB of entries ahead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: bound the bytes a prefetched listing window holds

Entries with long chunk lists or inline content made the up to 64
held windows of 32 entries an unbounded amount of memory per listing.
A window now stops at 128 KiB of entries and the walk lists the rest
of that directory from the window's last entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: test recursive listings whose streams break partway

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 09:28:41 -08:00
Chris LuandClaude Opus 5.5 3e64326f2b s3: require their own actions to change a bucket's CORS or versioning (#12433)
* s3: require their own actions to change a bucket's CORS or versioning

PUT/DELETE ?cors and PUT ?versioning were authorized as Write, so an
identity granted only object writes on a bucket could replace its CORS
rules or suspend versioning. They now need PutBucketCors,
DeleteBucketCors and PutBucketVersioning (or Admin), and inline policies
granting s3:PutBucketCors, s3:DeleteBucketCors or s3:PutBucketVersioning
map to those actions instead of Write.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: test that object Write cannot change bucket CORS or versioning

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: offer the bucket CORS and versioning permissions

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: test each bucket config action with its own identity

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 09:12:00 -08:00
Chris LuandClaude Opus 5.5 3a579ece3b s3: commit the delete marker in the owner's finalize transaction (#12432)
* s3: commit the delete marker in the owner's finalize transaction

createDeleteMarker wrote the marker with mkFile to the gateway's current
filer and then sent RECOMPUTE_LATEST to the object's ring owner. When the
two differ, the owner scans .versions before peer metadata sync delivers
the marker, re-derives the pointer to the previous version, and the
marker stays non-current on every filer.

routedVersionedFinalize now takes an optional version entry and PUTs it
in the same ObjectTransaction ahead of the recompute, the way PutObject
and CompleteMultipartUpload already do. The copy destination keeps
passing nil: its version file is already written on the owner.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: test a versioned delete when the gateway's filer is not the owner

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: assert the delete marker and the pointer flip share one transaction

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 09:10:10 -08:00
a8766f5177 s3api: apply copy bucket-default SSE after the ACL re-auth (#12436)
* s3api: apply copy bucket-default SSE after the ACL re-auth

CopyObject without its own x-amz-server-side-encryption header takes the
destination bucket's default encryption, which CopyObjectHandler writes
into the request headers. It did so before preparePutObjectACL, which
re-authenticates a request that carries an ACL (x-amz-acl or a grant
header) for s3:PutObjectAcl. verifySignedHeadersCoverage then found the
synthesized x-amz-server-side-encryption* headers outside SignedHeaders
and rejected the copy with SignatureDoesNotMatch.

Move the bucket-default injection after the ACL check, so the
re-authentication sees the request as the client signed it. The
destination object still gets the bucket default encryption.

* s3api: assert the copy's KMS key and stored ciphertext

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Stefan Scharlott <stefans@Stefans-MacBook-Pro.local>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 09:08:44 -08:00
d127be610f admin: re-send the master vacuum disable every poll so it survives leader changes (#12430)
* admin: re-send the master vacuum disable every poll so it survives leader changes

The admin disabled the master's scheduled vacuum only when a vacuum plugin
worker connected or disconnected. The master keeps that flag in memory and it
is not replicated through raft, so after a leader change or a leader restart
the new leader ran its own vacuum next to the plugin worker. When both picked
the same volume the master's commit failed ("compact swap aborted: missing
.cpd/.cpx") and it dropped that replica, leaving the volume under-replicated.

Send the desired state on every poll instead, and log on the master only
when the flag actually changes.

* admin: shorten the vacuum sync comments

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Kevin <kevin@leetify.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 09:07:50 -08:00
github-actions[bot] 0dd33f5f0f docs: regenerate star history chart 2026-11-06 00:33:08 +00:00
Chris LuandClaude Opus 5.5 878979af70 filer: hide write controls in the UI on the read-only port (#12429)
* filer: hide write controls in the UI on the read-only port

The read-only port accepts only GET and HEAD, but its directory page
offered New Folder, Upload, drag-and-drop, rename and delete, each of
which failed with 405. Requests that come through the read-only handler
now render the page without those controls or the upload script.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test the directory page controls on each port

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: match the rendered row buttons in the port test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 10:11:52 -08:00
e44c16802d weed/server: verify data shards against bitrot sidecar before ec decode (#12422) (#12427)
* weed/server: verify data shards against bitrot sidecar before ec decode (#12422)

* ec.decode: verify data shards against the mounted volume's bitrot protection

The decode looked up only the generation-0 .ecsum beside the data base and
each location's data directory, so it missed a sidecar kept with the index
and ignored the generation the volume actually serves.

Take the protection the volume's per-disk runtimes resolved instead (On, else
Invalid, else Off), and test a corrupt block past the first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.decode: check for live entries before verifying the shards

A volume whose needles are all deleted gets its shards purged on the
no-live-entries status; a failing checksum must not keep it around.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: verify data shards against the bitrot protection before a decode

Mirrors the Go volume server: the decode takes the protection the volume's
per-disk runtimes resolved and refuses a data shard that fails its .ecsum
checksum, after the no-live-entries check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: a decode refuses a data shard that fails its .ecsum checksum

Runs against both volume server implementations: the refused decode keeps
every shard and the sidecar, and the needle stays readable through EC.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.decode: reload the bitrot sidecar before verifying the shards

A rebuild's backfill writes the sidecar without reloading the mounted
runtimes, which then still report no protection; resolve it again across
the EC metadata directories, as a shard mount does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: reload the bitrot sidecar before verifying the shards

Mirrors the Go volume server: the decode job resolves the sidecar again on
every runtime of the volume before choosing its protection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: a decode finds a sidecar written after the shards were mounted

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: read through the EC shards after the refused decode

The source volume stayed mounted, so the final read never touched the
shards it is meant to check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.decode: refuse when a sidecar reload reports another layout

ReloadBitrotSidecar logged and dropped the layout-mismatch error, so the
runtime stayed Off and the decode skipped the data-shard check. It now
returns that error and the decode refuses on it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: refuse a decode when a sidecar reload reports another layout

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: a decode refuses a late sidecar that records another layout

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.decode: refuse when an unmount left no runtime to resolve the sidecar

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: refuse a decode when an unmount left no runtime to resolve the sidecar

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 10:11:09 -08:00
80f32bc074 weed/command: skip remote bucket delete when bucket is unmounted (#12418) (#12425)
* weed/command: skip remote bucket delete when bucket is unmounted

* remote_storage: count deleting an already-missing bucket as done

filer.remote.gateway can see the same bucket delete more than once, for
example once per filer when several filers peer their metadata. A copy
that still finds the mount mapping calls DeleteBucket again, and S3
(NoSuchBucket), Azure (ContainerNotFound, ContainerBeingDeleted) and GCS
(404) all failed it, which held the sync offset on the delete. A
missing bucket now counts as deleted, the way deleting a missing object
already does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.gateway: keep the mapping table writable after mount.mapping is deleted

Deleting mount.mapping replaced the gateway's mappings with a table whose
map was nil, so the next bucket create panicked writing its mount into
it. Reset to an empty map instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.gateway: count a delete of an absent object as done

filer.remote.sync already treats ErrRemoteObjectNotFound from DeleteFile
as deleted, since GCS reports a missing object that way where S3 and
Azure succeed. The gateway returned it, so a delete event seen twice, as
it is when several filers peer their metadata, held the sync offset on
GCS. Use the same deleteRemoteFile.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.gateway: drop cached remote clients after the new gateway tests

GetRemoteStorage caches clients by configuration, so a repeated run
(-count=2) got the first run's recorder back and saw no calls.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.gateway: keep the default storage across mount.mapping changes

The default storage for new buckets is derived at startup and kept only
in the gateway's copy of the mappings; mount.mapping never stores it.
Replacing or deleting that file dropped it, so a later bucket create
with no -createBucketAt skipped the remote bucket.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 05:06:41 -08:00
Chris LuandClaude Opus 5.5 c971252871 admin: make the cluster lock's fairness window configurable (#12426)
* admin: make the cluster lock's fairness window configurable

Scheduled jobs hold the cluster admin lock for their whole run. Once the
lock had been held for a minute, every new job waited until all running
jobs had released it, so with hour-long EC jobs dispatch ran in waves and
most workers sat idle between them.

-adminLock.fairnessWindow keeps the one-minute default; 0 lets new jobs
join the shared hold, and shell clients get the lock when the scheduler
next drains.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* admin: wait for the blocked acquire before disabling the fairness window

The wake-up test slept 50 ms before SetFairnessWindow(0), so a late
acquire that started after the change joined without exercising the
Broadcast. Wait until it is registered as a waiter instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 05:02:41 -08:00
8c8d0f1cdf weed/shell: resolve collection from topology when ec.decode is given only volumeId (#12424)
* weed/shell: resolve collection from topology when ec.decode is given only volumeId

When ec.decode is invoked with -volumeId for a volume that belongs to a named collection without explicitly specifying -collection, DoEcDecode was called with an empty collection string. Target volume servers then rejected VolumeEcShardsCopy with 'CopyFile not found ec volume id <vid>' because they looked under the default collection.

This commit introduces FindEcVolumeCollection to discover the collection name directly from topology metadata (checking EC shard info and regular volume info), ensuring DoEcDecode receives the accurate collection.

* ec.decode: resolve every batch volume's collection from the topology

The batch passed -collection through unless it looked like a regex, so a
comma list (a,b), a {n} quantifier, ALL or _default_ reached the volume
servers as a collection name and the copy found no shards. The selector
only picks the volumes; each one is decoded in its own collection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.decode: take the collection only from the volume's ec shards

The lookup returned the first record per disk, so a regular volume on an
earlier node answered before the ec shards on a later one. A decode needs
the shards anyway, so their records are the only ones to read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.decode: leave the -volumeId lookup to DoEcDecode

DoEcDecode already resolves an empty collection from the topology.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 05:01:25 -08:00
120d17a092 filer.remote.sync: stop a touched file from being restamped forever (#12419)
* filer.remote.sync: stop a touched file from being restamped forever

A metadata-only update that moves the mtime past RemoteMtime (touch, or
an O_TRUNC open of an empty lock file) goes through UpdateFileMetadata,
which changes nothing on the remote, and the entry is stamped with the
same RemoteEntry and a new LastLocalSyncTsNs. That stamp is an update
event of its own. shouldSendToRemote only looks at LastLocalSyncTsNs
when RemoteMtime and the mtime share a second, so it still reads the
entry as dirty, and the event is stamped again. Nothing is uploaded,
but the daemon keeps rewriting the entry as fast as it reads its own
events back.

Skip the restamp when the backend left the object alone and the event's
entry already carries a stamp taken after its mtime. The touch itself is
still stamped once.

* filer.remote.sync: log the skipped restamp

The other skips in processUpdateEvent log at V(2); this one did not, so
with -v=2 a touched file showed an "update meta" line with nothing after
it.

* filer.remote.sync: skip the restamp of a stamp's own event, whatever the clocks

The skip compared the entry's stamp with its mtime, so an mtime ahead of
the sync's clock (a skewed writer, touch -d) stayed past every stamp and
kept the loop going until the clock caught up. An event that changed only
the RemoteEntry is the sync's own stamp, which needs no other.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: ignore the filer's commit timestamp when spotting a stamp

The filer gives every stored update a new CommitTsNs, so a stamp's event
never compared equal outside the RemoteEntry and a touched file kept being
restamped against a real filer. The test now assigns one per update.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 04:08:28 -08:00
23598afb31 volume: bound each replica forward so a silent peer cannot hold a write (#12417)
* volume: bound replica forwards so a silent peer cannot hold a write

The Rust volume server forwards each replicated write to its peers with the
shared outgoing reqwest client, which had no connect or request timeout.
When a peer is silently partitioned (packets dropped, no reset) the forward
waits until TCP gives up, minutes later, and the write waits with it. A
chaos test (replication 001, one volume server partitioned) saw every write
on a volume with a copy there block.

- Replica forwards go through replica_request, bounded at
  REPLICA_REQUEST_TIMEOUT = 10 s from connect to the response.
- Both outgoing clients start from outgoing_client_builder, which adds a
  5 s connect timeout, so new connections to a peer that never answers
  fail fast. The shared client gets no overall request timeout, since it
  also serves lookups and other transfers.

Tests: a replica that receives the forward and never answers fails the
whole replicated write at the bound, naming that replica; a silent peer
fails the request builder at the bound on a paused clock; a connect that
stalls in the resolver fails at the 5 s connect bound.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: give a replica forward more time for a larger body

The 10 s replica forward bound runs from connect to the end of the
response, so it also covers sending the body. A write near the 256 MB file
size limit needs about 26 MB/s to reach its replica in 10 s, so on a link
that is slower or busy the forward failed and the write returned 500 even
though the peer was answering.

A forward now gets REPLICA_REQUEST_TIMEOUT plus one second per MiB of
body. A bodiless forward keeps the 10 s bound (a 1 MiB write gets 11 s).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: bound each Go replica forward like the Rust one

The Go volume server forwards a replicated write with a 10 s dial bound and
the transport's 30 s response header timeout. A replica that takes the
forward and never answers held the write for 30 s, and a body larger than
the socket buffers to a silently partitioned peer was held until TCP gave
up, since nothing bounds sending the body.

Each forward now runs under replicaRequestBound: 10 s plus one second per
MiB of body, the same bound the Rust volume server puts on its forwards.

Test: a replica that receives the forward and never answers fails the
write at the bound (300 ms in the test); without it the write fails at
30 s on the response header timeout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: keep proxy variables out of the stalled-connect test

reqwest reads HTTP_PROXY and ALL_PROXY by default, so with a proxy set the
test's request went to the proxy instead of the stalled resolver.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: count a replica forward that hit its bound as a timeout

DistributedOperationResult joined the replica errors into a plain string,
and the uploader reports a cancelled request without wrapping the context
error, so a forward cut off at replicaRequestBound was counted under
server_error in the replication failure metric, as was a response header
timeout before it.

The joined error now unwraps to each replica's error, and a forward that
ran out of its bound reports context.DeadlineExceeded.

Test: the silent replica write raises the timeout failure count by one;
with the flattened join it stays put.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: test that a large Go replica forward gets its body's bound

A replica answers after the bodiless bound but within the bound for a
1 MiB body, so the write fails if replicateTo drops the body scaling.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: run the Rust whole-path replica test with a 1 MiB body

The forward then has to wait the bound for its body, so the test fails if
do_replicated_request stops passing the body length to the bound.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: pingqiu <pingqiu@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 04:05:33 -08:00
Chris LuandClaude Opus 5.5 aab947b65e s3api: answer success when a failed completion already committed (#12416)
* s3api: name the check that a completion already committed

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: answer success when a failed completion already committed

A finalize can fail after its commit: the filer stored the object and the
call still reported an error. The client then got a 500 with an unknown
outcome. When the failure may have left this attempt's object behind,
check the key the way a retry would and answer success with the
committed result when it holds this upload's object.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: keep part data when the commit found is another attempt's

The object carrying this upload ID can be a concurrent attempt's commit of
the same upload, and its part selection can include entries this attempt
counts as unused. Free those parts' data only when the committed chunks
are this attempt's own; otherwise drop their metadata only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: test the committed-completion read on versioned buckets

Cover the latest-version read (version id and checksum in the answer),
a suspended bucket whose delete marker is still current, and that the
upload directory is removed without its data.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: test versioned recovery through the completion handler

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 03:38:52 -08:00
Chris LuandClaude Opus 5.5 fe1cf5263f volume: bound each FetchAndWriteNeedle replica write (#12415)
* seaweed-volume: bound each FetchAndWriteNeedle replica write

A replica that accepted the connection and then went silent held the
request until TCP gave up, because the shared outgoing client has no
request timeout. Each replica write now fails after 30 s, from connect
to the response status, matching the Go volume server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: bound each FetchAndWriteNeedle replica write

The default replica path uploads through the guarded peer client, which
bounds only the dial and has no response header timeout, and the filer
calls FetchAndWriteNeedle with no deadline. A replica that accepted the
connection and went silent held the request for minutes. Each replica
write, retries included, now has 30 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: require the silent-peer test to fail at the deadline

A connection error that came back early also passed, since the test
only checked an upper bound. It now also requires the paused clock to
have reached FETCH_AND_WRITE_REPLICA_TIMEOUT.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: grow the FetchAndWriteNeedle replica bound with the chunk

The reqwest timeout includes sending the body, and a remote-cache chunk
is a thousandth of the object once it passes the 1000-chunk cap, so a
fixed 30 s would fail a large chunk on a link under ~35 MB/s. The bound
is now 30 s plus 1 s per MiB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: grow the FetchAndWriteNeedle replica bound with the chunk

The context deadline includes sending the body, so a fixed 30 s would
fail a large remote-cache chunk on a slow link. The bound is now 30 s
plus 1 s per MiB, matching the Rust volume server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 03:37:49 -08:00
Chris LuandClaude Opus 5.5 247e010865 volume: redial a master that just dropped a registered stream after a second (#12420)
* volume: redial a master that just dropped a registered stream after a second

A registered volume server that loses its master with no peer naming a
leader waited a full pulse before redialing. A restarted master is
usually back within a second, so its volumes stayed unregistered, and
uncached lookups kept failing, for most of a pulse.

For one pulse after a stream a leader had answered ends, redial on the
one-second cap. A master still down after that is redialed at the pulse
pace, so redials never come later than before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: redial a master that just dropped a registered stream after a second

Mirror of the Go heartbeat change. The loop keeps when the last stream a
leader had answered ended, which also stands in for the
confirmed-leader-once flag (None until first registration), and redials
on the one-second cap for a pulse after that.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: test that a dropped registered stream records the loss

A fake master confirms the stream and then ends it. The server starts as
registered long ago, so only a recorded loss makes the redial come after
a second instead of a pulse.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: record a lost stream only after its receiver has exited

doHeartbeatWithRetry can return on a send error while the receiver is
still handling the leader's first reply, so the deferred recorder could
read the registered flag too early and keep an old loss time. Cancel the
stream and wait for the receiver before reading it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: require the prompt heartbeat redial to wait out its second

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: fail the down master's dials at once in the redial test

Windows takes about two seconds to refuse a connection to a closed
loopback port, so the stream dial and the leader lookup each held the
prompt case past its two-second bound. A listener that closes every
connection fails both dials immediately on every platform.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 03:27:45 -08:00
Chris LuandClaude Opus 5.5 8cc3a8ae21 ec.rebuild: repair shards that fail a checksum scrub in complete volumes (#12410)
* ec.rebuild: rebuild shards that fail a checksum scrub, also in complete volumes

ec.rebuild skipped every volume with all shards present, so a shard whose
bytes no longer match the .ecsum sidecar stayed in place until some other
shard went missing. With -checksum, ec.rebuild first scrubs each node's
shards in checksum mode and takes the failing copies out of the plan; a
shard left without a good copy counts as missing and is regenerated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.rebuild: remove the copies that failed the checksum once a good copy serves

A shard regenerated on the rebuilder left the failing copy mounted on its
old holder, so the volume kept a corrupt duplicate that a later dedup could
keep instead of the good one. Remove each failing copy once the rebuilt
shard is mounted, and right away when another holder already serves a copy
that passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.rebuild: test the checksum scrub against fake volume servers

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.rebuild: check the lock before removing a corrupt copy

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.rebuild: print the checksum scrub details

A malformed sidecar or a wholesale mismatch comes back only in the scrub
details, with no broken shard, so the volume would look verified.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.rebuild: fail the run when a corrupt duplicate cannot be removed

For a volume whose only problem is a corrupt duplicate, the removal is the
whole repair, so its failure has to reach the caller.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.rebuild: test duplicate removal failures and a lost lock

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.rebuild: fail the run when a node's checksum scrub fails

The node's copies stay in the plan unverified, so a corrupt shard there is
left in place; the rest of the cluster is still repaired.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.rebuild: preview the corrupt-copy removals in dry-run

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec.rebuild: test a failed checksum scrub

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 02:08:22 -08:00
Chris LuandClaude Opus 5.5 fcf62db8d0 seaweed-volume: stage optional EC sidecar copies before replacing (#12403)
* seaweed-volume: stage optional EC sidecar copies before replacing

VolumeEcShardsCopy opened the destination .vif/.ecsum with File::create
and then streamed into it. A source without the file left a 0-byte .vif,
wiped an existing .ecsum (turning bitrot protection off on that holder),
and a failed stream removed the destination's existing copy.

Stream optional copies into <file>.copying and rename into place only
once the source sent bytes, as the Go volume server's writeToFile does
for ignore_source_file_not_found.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* seaweed-volume: keep the drain doc comment on drain_copy_stream_to_file

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: stage the .vif copied during missing-index recovery

fetch_ec_index_from_one_peer drained the optional .vif straight into the
local file, so a peer without one left a 0-byte .vif (losing a local EC
ratio) and a failed stream left a partial one. Stage it and rename only once
the peer sent bytes, as Go's recovery does through writeToFile.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: test optional copies against a stream that fails mid-transfer

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: remove a staged optional copy when the RPC is cancelled

A cancelled CopyFile handler drops its future at the stream await, so the
cleanup after it never ran and the .copying file stayed behind. Go's
writeToFile runs to completion and removes it. A guard now owns the
staging cleanup and is disarmed only once the rename commits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: test optional copies dropped mid-stream

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: cancel the staged-copy test only once the staging file exists

tokio's File::create runs on the blocking pool, so a fixed timeout could
drop the .ecsum copy before .copying existed and pass without exercising
the cleanup guard.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 02:04:49 -08:00
Chris LuandClaude Opus 5.5 85232f7414 seaweed-volume: run the preallocating max-volume test on a fake disk (#12423)
* seaweed-volume: run the preallocating max-volume test on a fake disk

The test sized the volume limit from the real free space but the store
subtracts the 1% floor first, so on a disk more than 95% full the second
slot no longer fit and the count came out 1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: start the preallocating max-volume test above the expected count

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 02:03:07 -08:00
Chris LuandClaude Opus 5.5 b74111ea2f s3vectors test: skip a query lease tag released between list and HEAD (#12409)
A query's lease tag is deleted by a background task after the query
answers, so the encryption helper could list it and then get a 404 on
HEAD. Skip only that case; any other missing object still fails.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 02:01:18 -08:00
Chris LuandClaude Fable 5.1 48df16ea0d weed-worker: keep a vector index covering as its table grows (ANN phase 3) (#12414)
* seaweed-vector: build and extend the vector index

weed_lance_worker::vector_index builds an IVF_FLAT index in the metric the
table was created with, sizes partitions from the row count, and extends an
index with a new segment or a retrain; the core crate re-exports it and adds
the probes a query spends. Tests cover coverage after build, append, retrain
and compaction.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector-cli: index build and index optimize

Logs go to stderr so stdout stays the JSON a caller parses.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: PERF_INDEX builds the index before the timed queries

The cluster fixture hands out its catalog URL, the Makefile builds
weed-vector-cli, and the perf suite builds the index after ingest and
reports the build time and segment count.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: probe a tenth of the index's partitions, more under a filter

A query that names no probe count gets one from the index's partition
count. Lance searches at least that many partitions, and under a prefilter
keeps going until it has k rows; without one it stops there.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: --nprobes overrides the probe policy

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: test that a build takes the table's metric

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: make clean removes weed-vector-cli

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: measure recall with vectors the table does not hold

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: keep the read cache's doc comment on with_read_cache

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: lance_vector_index builds the index of an S3 Vectors table past a row threshold

Detection keeps the tables a vector bucket created, with at least min_rows
rows (50,000 by default) and no index on their vector column. Execution
builds the IVF_FLAT index in the table's metric with sqrt(rows) partitions.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: optimize only the indices on the vector column

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: resolve the query metric with index::distance_metric

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: test the L2 metric under the name CreateIndex stores

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: build the index through the admin server and the Lance worker

VECTOR_WORKER=1 adds weed admin and weed-worker to the cluster. The new
test sets the job's threshold, runs it through the plugin API, checks
coverage and recall, and that a row written after the build is found.
CI gains the worker leg for both volume servers.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: read an index's partitions, and when a table has outgrown them

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: probes from the shared partition reader, partitions in coverage

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: a vector index reindexes at a tenth of its rows and retrains once the table outgrows it

lance_optimize_indices proposes a vector table once its uncovered rows pass
min(the absolute budget, a tenth of the indexed rows), and trains the index
again instead of extending it when sqrt(rows) has reached twice its
partitions. The end-to-end test covers both.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: open a table inside a caller's Lance session

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: keep opened indices between queries in one session

Every open used a fresh Lance session, so the index and metadata caches
died with the request and each query loaded the IVF index again.
--index-cache-mb sizes the shared cache, 512 MiB by default.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: PERF_CLUSTERS draws the perf suite's vectors around shared centres

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: the index test draws embedding-like vectors

Independent Gaussian vectors have no locality for an IVF index to use, so
recall through one only tracks the fraction of partitions probed; points
around shared centres behave like embeddings. The test must also run
before the one that stops the engine.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: the optimize and retrain steps draw embedding-like vectors

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: test that compaction drops the deleted key from tied hits

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: test that an append-only optimize serves the new row

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: record PERF_NOISE in the perf report config

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: a filtered query widens to four times the probes, not without bound

With no upper bound Lance prepares every partition for the late search
even when the minimum probes already hold k rows, and a filtered query
through the index cost as much as the exact scan. A query that names its
own probe count gets exactly that count.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: report recall through an index without asserting it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: count a hit tying the k-th best score toward recall

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: lance_vector_index runs in the Lance lane

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: check a table's index before counting its rows

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: make build and clean cover the worker leg's binaries

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: a retrain rebuilds the index with partitions for the row count

Lance's OptimizeOptions::retrain does not size the partitions: it left a
6,000-row index with one. The retrain goes through the builder with
sqrt(rows) partitions, for the job and for weed-vector-cli index optimize
--retrain alike. The end-to-end test no longer expects a merge to keep the
partition count.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: allow float32 rounding at the recall cutoff

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: count a repeated hit key once toward recall

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: credit cutoff ties only up to the slots the top-k leaves

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: assert every filtered hit passes the filter, indexed or not

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-worker: coverage over an index's names, one name extended, a merge at the delta limit

A sharded build leaves one name per shard, each missing the others'
fragments, so coverage is the union of every index on the column and an
optimize gives new rows to the name with the fewest deltas only. At
MAX_DELTAS the policy passes OptimizeOptions::merge(deltas); Lance's
default options may append instead of merging. A retrain keeps an index's
own metric and rebuilds only IVF_FLAT indices.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: the optimize job measures and extends a vector table's indices together

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: tests for bounded deltas, disjoint shards and a retrain's metric

Nine append-and-optimize cycles give delta counts 2 to 8 and then 1; two
names on disjoint fragments stay disjoint through an optimize; a cosine
index on a table without metric metadata is still cosine after a retrain.
The end-to-end test expects the appended delta after the first optimize.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: forget the shared session when a table was dropped and created again

The session keys index lists and transactions by dataset version, and a
table created again at the same location restarts its versions. An open
that finds an older version than the newest seen at that URI, or the same
version committed later, replaces the session and opens again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: test an open after another engine created the table again

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-worker: retrain only a single IVF_FLAT vector index name

The job compared each shard name's partitions with the whole table's
rows, so a sharded index was always due, and a non-IVF_FLAT index made
optimize bail on every run before the new rows were indexed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-worker: refuse to retrain a vector index sharded over several names

A rebuild per name over the whole table left every shard covering every
fragment, so a query searched each row once per shard.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: test which vector indices a retrain is due for

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: replace the session in the same lock that records a recreated table

A second open of the recreated table could see the new version recorded
and the old session still shared, and keep the dropped table's index list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-worker: skip a vector table whose rows cannot be counted

Detection returned on the first table whose count_rows failed, so one
unreadable table kept every other table from being proposed. It now skips
that table like an open or index failure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: panel for vector indexes the Lance worker built

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: size default probes from the largest index on the column

A sharded build gives the column one index name per shard. The first
name alone could be a small shard, and its probe count would then be
spent on a larger one.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: skip the probe lookup when a query is given no segments

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: open every table in its own Lance session again

One session shared by every open keeps a table's index loaded between
queries, but Lance keys what it caches by location and version, so a
table dropped and created again at the same location could be served
the old table's index list. Telling the two apart from version and
commit time needs a map that only grows, a retry loop and clocks that
agree across engines. A private session per open, as before, has no
such state. Measured at 50k x 384 through an index, eight clients: 164
queries/s with the shared session, 137 without; one client is unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: reject --nprobes 0

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: read the partition count from the opened index

Lance's index statistics took 3.6 ms a query on a 200k-row table with
447 partitions. Opening the index and asking it takes 0.4 ms, and the
scan that follows finds the index already open.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* ci: give the S3 Vectors worker legs the vendored protoc

The worker workspace has no protoc of its own for lance's build
scripts, so the worker legs failed at their first build.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: give a vector index build an hour before the admin times it out

The job set no execution timeout, so the admin's default of 90 seconds
applied, and a build that trains on a large table would be reported as
failed and retried while it was still running.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: lance_vector_index proposes at most the admin's max_results

Every qualifying table went into one message however many there were.
The detector stops at the limit the request carries and sets has_more,
as the Go detectors do.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: a vector index job re-checks the table it was proposed for

A table emptied since detection completes without a build, and one
without a vector column fails instead of reporting it already indexed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: lance_optimize_indices gets a build's execution timeout

A retrain rebuilds the index over the whole table, which the admin's
default of 90 seconds would cut off on a large one.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: a vector index run window as long as one build

An attempt is capped at the end of the job type's run window plus the
drain grace, and the default window is 30 minutes, so a build could be
cut off before its own hour was up. The window is an hour too, as the
Go iceberg handler sets both.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: a lance_optimize_indices run window as long as one retrain

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: vector index proposals declare a build's runtime

A run started by hand while the job type's schedule is off takes no
timeout from the descriptor, only the admin's 90 seconds. The admin
gives a job at least the runtime its proposal declares, so the build
and the optimize proposals of a vector table declare the hour.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: coverage reports no partitions for an index of unknown coverage

Reading the partition count goes through Lance's index statistics,
which fail on an index that records no fragment bitmap, and the whole
coverage report failed with them.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: an index that records no fragments is not read as uncovered

Its rows counted as uncovered, so detection proposed the table and the
execution then failed in Lance's statistics. Coverage fails for such an
index instead, and detection skips the table with a warning.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: coverage reads partition counts from the opened index

Lance's index statistics rewrite the manifest of a table whose
fragments carry no row counts, which a coverage report must not do and
cannot do with read-only credentials. The report asks each opened
segment instead, as the probe lookup does, and the statistics-based
reader has no caller left.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-04 23:30:27 -08:00
Chris LuandClaude Fable 5.1 e465880ee8 weed-worker: build the vector index of S3 Vectors tables past a row threshold (ANN phase 2) (#12413)
* seaweed-vector: build and extend the vector index

weed_lance_worker::vector_index builds an IVF_FLAT index in the metric the
table was created with, sizes partitions from the row count, and extends an
index with a new segment or a retrain; the core crate re-exports it and adds
the probes a query spends. Tests cover coverage after build, append, retrain
and compaction.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector-cli: index build and index optimize

Logs go to stderr so stdout stays the JSON a caller parses.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: PERF_INDEX builds the index before the timed queries

The cluster fixture hands out its catalog URL, the Makefile builds
weed-vector-cli, and the perf suite builds the index after ingest and
reports the build time and segment count.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: probe a tenth of the index's partitions, more under a filter

A query that names no probe count gets one from the index's partition
count. Lance searches at least that many partitions, and under a prefilter
keeps going until it has k rows; without one it stops there.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: --nprobes overrides the probe policy

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: test that a build takes the table's metric

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: make clean removes weed-vector-cli

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: measure recall with vectors the table does not hold

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: keep the read cache's doc comment on with_read_cache

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: lance_vector_index builds the index of an S3 Vectors table past a row threshold

Detection keeps the tables a vector bucket created, with at least min_rows
rows (50,000 by default) and no index on their vector column. Execution
builds the IVF_FLAT index in the table's metric with sqrt(rows) partitions.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: optimize only the indices on the vector column

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: resolve the query metric with index::distance_metric

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: test the L2 metric under the name CreateIndex stores

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: build the index through the admin server and the Lance worker

VECTOR_WORKER=1 adds weed admin and weed-worker to the cluster. The new
test sets the job's threshold, runs it through the plugin API, checks
coverage and recall, and that a row written after the build is found.
CI gains the worker leg for both volume servers.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: open a table inside a caller's Lance session

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: keep opened indices between queries in one session

Every open used a fresh Lance session, so the index and metadata caches
died with the request and each query loaded the IVF index again.
--index-cache-mb sizes the shared cache, 512 MiB by default.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: PERF_CLUSTERS draws the perf suite's vectors around shared centres

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: the index test draws embedding-like vectors

Independent Gaussian vectors have no locality for an IVF index to use, so
recall through one only tracks the fraction of partitions probed; points
around shared centres behave like embeddings. The test must also run
before the one that stops the engine.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: test that compaction drops the deleted key from tied hits

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: test that an append-only optimize serves the new row

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: record PERF_NOISE in the perf report config

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: a filtered query widens to four times the probes, not without bound

With no upper bound Lance prepares every partition for the late search
even when the minimum probes already hold k rows, and a filtered query
through the index cost as much as the exact scan. A query that names its
own probe count gets exactly that count.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: report recall through an index without asserting it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: count a hit tying the k-th best score toward recall

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: lance_vector_index runs in the Lance lane

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: check a table's index before counting its rows

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: make build and clean cover the worker leg's binaries

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: allow float32 rounding at the recall cutoff

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: count a repeated hit key once toward recall

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: credit cutoff ties only up to the slots the top-k leaves

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: assert every filtered hit passes the filter, indexed or not

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: forget the shared session when a table was dropped and created again

The session keys index lists and transactions by dataset version, and a
table created again at the same location restarts its versions. An open
that finds an older version than the newest seen at that URI, or the same
version committed later, replaces the session and opens again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: test an open after another engine created the table again

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: replace the session in the same lock that records a recreated table

A second open of the recreated table could see the new version recorded
and the old session still shared, and keep the dropped table's index list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-worker: skip a vector table whose rows cannot be counted

Detection returned on the first table whose count_rows failed, so one
unreadable table kept every other table from being proposed. It now skips
that table like an open or index failure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: panel for vector indexes the Lance worker built

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: size default probes from the largest index on the column

A sharded build gives the column one index name per shard. The first
name alone could be a small shard, and its probe count would then be
spent on a larger one.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: skip the probe lookup when a query is given no segments

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: open every table in its own Lance session again

One session shared by every open keeps a table's index loaded between
queries, but Lance keys what it caches by location and version, so a
table dropped and created again at the same location could be served
the old table's index list. Telling the two apart from version and
commit time needs a map that only grows, a retry loop and clocks that
agree across engines. A private session per open, as before, has no
such state. Measured at 50k x 384 through an index, eight clients: 164
queries/s with the shared session, 137 without; one client is unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: reject --nprobes 0

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: read the partition count from the opened index

Lance's index statistics took 3.6 ms a query on a 200k-row table with
447 partitions. Opening the index and asking it takes 0.4 ms, and the
scan that follows finds the index already open.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* ci: give the S3 Vectors worker legs the vendored protoc

The worker workspace has no protoc of its own for lance's build
scripts, so the worker legs failed at their first build.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: give a vector index build an hour before the admin times it out

The job set no execution timeout, so the admin's default of 90 seconds
applied, and a build that trains on a large table would be reported as
failed and retried while it was still running.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: lance_vector_index proposes at most the admin's max_results

Every qualifying table went into one message however many there were.
The detector stops at the limit the request carries and sets has_more,
as the Go detectors do.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: a vector index job re-checks the table it was proposed for

A table emptied since detection completes without a build, and one
without a vector column fails instead of reporting it already indexed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-worker: a vector index run window as long as one build

An attempt is capped at the end of the job type's run window plus the
drain grace, and the default window is 30 minutes, so a build could be
cut off before its own hour was up. The window is an hour too, as the
Go iceberg handler sets both.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-04 22:16:31 -08:00
Chris LuandClaude Fable 5.1 b5951c5c7c seaweed-vector: probe policy for indexed queries (ANN phase 1) (#12412)
* seaweed-vector: build and extend the vector index

weed_lance_worker::vector_index builds an IVF_FLAT index in the metric the
table was created with, sizes partitions from the row count, and extends an
index with a new segment or a retrain; the core crate re-exports it and adds
the probes a query spends. Tests cover coverage after build, append, retrain
and compaction.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector-cli: index build and index optimize

Logs go to stderr so stdout stays the JSON a caller parses.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: PERF_INDEX builds the index before the timed queries

The cluster fixture hands out its catalog URL, the Makefile builds
weed-vector-cli, and the perf suite builds the index after ingest and
reports the build time and segment count.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: probe a tenth of the index's partitions, more under a filter

A query that names no probe count gets one from the index's partition
count. Lance searches at least that many partitions, and under a prefilter
keeps going until it has k rows; without one it stops there.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: --nprobes overrides the probe policy

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: test that a build takes the table's metric

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: make clean removes weed-vector-cli

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: measure recall with vectors the table does not hold

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: keep the read cache's doc comment on with_read_cache

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: optimize only the indices on the vector column

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: resolve the query metric with index::distance_metric

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: test the L2 metric under the name CreateIndex stores

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-worker: open a table inside a caller's Lance session

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: keep opened indices between queries in one session

Every open used a fresh Lance session, so the index and metadata caches
died with the request and each query loaded the IVF index again.
--index-cache-mb sizes the shared cache, 512 MiB by default.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: PERF_CLUSTERS draws the perf suite's vectors around shared centres

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: test that compaction drops the deleted key from tied hits

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: test that an append-only optimize serves the new row

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: record PERF_NOISE in the perf report config

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: a filtered query widens to four times the probes, not without bound

With no upper bound Lance prepares every partition for the late search
even when the minimum probes already hold k rows, and a filtered query
through the index cost as much as the exact scan. A query that names its
own probe count gets exactly that count.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: report recall through an index without asserting it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: count a hit tying the k-th best score toward recall

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: allow float32 rounding at the recall cutoff

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: count a repeated hit key once toward recall

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: credit cutoff ties only up to the slots the top-k leaves

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: assert every filtered hit passes the filter, indexed or not

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: forget the shared session when a table was dropped and created again

The session keys index lists and transactions by dataset version, and a
table created again at the same location restarts its versions. An open
that finds an older version than the newest seen at that URI, or the same
version committed later, replaces the session and opens again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: test an open after another engine created the table again

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: replace the session in the same lock that records a recreated table

A second open of the recreated table could see the new version recorded
and the old session still shared, and keep the dropped table's index list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: size default probes from the largest index on the column

A sharded build gives the column one index name per shard. The first
name alone could be a small shard, and its probe count would then be
spent on a larger one.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: skip the probe lookup when a query is given no segments

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: open every table in its own Lance session again

One session shared by every open keeps a table's index loaded between
queries, but Lance keys what it caches by location and version, so a
table dropped and created again at the same location could be served
the old table's index list. Telling the two apart from version and
commit time needs a map that only grows, a retry loop and clocks that
agree across engines. A private session per open, as before, has no
such state. Measured at 50k x 384 through an index, eight clients: 164
queries/s with the shared session, 137 without; one client is unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: reject --nprobes 0

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: read the partition count from the opened index

Lance's index statistics took 3.6 ms a query on a 200k-row table with
447 partitions. Opening the index and asking it takes 0.4 ms, and the
scan that follows finds the index already open.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-04 21:07:13 -08:00
Chris LuandClaude Fable 5.1 f0f0ba85ae seaweed-vector: build a vector index from the CLI and measure it in the perf suite (ANN phase 0) (#12411)
* seaweed-vector: build and extend the vector index

weed_lance_worker::vector_index builds an IVF_FLAT index in the metric the
table was created with, sizes partitions from the row count, and extends an
index with a new segment or a retrain; the core crate re-exports it and adds
the probes a query spends. Tests cover coverage after build, append, retrain
and compaction.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector-cli: index build and index optimize

Logs go to stderr so stdout stays the JSON a caller parses.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: PERF_INDEX builds the index before the timed queries

The cluster fixture hands out its catalog URL, the Makefile builds
weed-vector-cli, and the perf suite builds the index after ingest and
reports the build time and segment count.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: test that a build takes the table's metric

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: make clean removes weed-vector-cli

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: optimize only the indices on the vector column

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: resolve the query metric with index::distance_metric

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: test the L2 metric under the name CreateIndex stores

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: PERF_CLUSTERS draws the perf suite's vectors around shared centres

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: test that compaction drops the deleted key from tied hits

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: test that an append-only optimize serves the new row

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: record PERF_NOISE in the perf report config

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: report recall through an index without asserting it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: count a hit tying the k-th best score toward recall

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: allow float32 rounding at the recall cutoff

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: count a repeated hit key once toward recall

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: credit cutoff ties only up to the slots the top-k leaves

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: assert every filtered hit passes the filter, indexed or not

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-04 19:25:38 -08:00
Chris LuandClaude Opus 5.5 8350f297af seaweed-vector: dev plan for ANN search over S3 Vectors indexes (#12408)
* seaweed-vector: dev plan for ANN search over S3 Vectors indexes

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: dev plan: one probe count per query, a budget that can shrink

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: dev plan: smaller deltas are searched in full only up to the probe count

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: dev plan: drop the private issue reference

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: dev plan: phase 1 probes use the Design count

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: dev plan: status after phases 0 to 3, and what the measurements changed

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: dev plan: name the Enterprise repository, not the private one

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: dev plan: the Enterprise repository by that name

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: dev plan: a filtered query widens to four times the probes, not without bound

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: dev plan: throughput through the index at 50k and 200k

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: dev plan: union coverage over shard names, and an explicit merge at the delta limit

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: dev plan: status table names branches, not pull requests

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: dev plan: maximum_nprobes is always four times the count

Lance reads maximum_nprobes only under a prefilter, and the phase 1
policy sets it the same way for every query.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: dev plan: the retrain follows the partition count, not a recorded row count

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: dev plan: phase 3 is open like the other phases

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: dev plan: the retrain stops past 2,048 partitions

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-vector: dev plan: the merge at the delta limit takes the name's delta count

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-04 19:25:35 -08:00
github-actions[bot] cd10b4a5c1 docs: regenerate star history chart 2026-11-05 02:07:12 +00:00
Chris Lu 683c5ad4d3 admin: regenerate the plugin pages' templ output from weed/admin (#12405)
Both were generated at the repo root, so their embedded source paths read
weed/admin/view/... where every other admin template, generated by
weed/admin's Makefile, reads view/.... No markup changes.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
2026-11-04 11:47:36 -08:00
Chris LuandClaude Opus 5.5 7047d3947a admin/plugin: fail a dispatch its restarted worker no longer lists (#12402)
* admin/plugin: fail a dispatch its restarted worker no longer lists

A job dispatched to a worker that then restarts dies with the old
process. The worker re-registers under the same id on a new stream and
its heartbeats stop listing the job, but nothing ended the pending
execution: the dispatcher kept waiting and held the executor slot until
the attempt's execution timeout.

Each pending execution now records the stream it went out on. An
itemized heartbeat from a newer stream of that worker that leaves the
job out twice in a row fails it through the expire path, so the
dispatch returns, the slot is released and the retry machinery takes
over. Jobs on the heartbeat's own stream, jobs the new stream still
lists, and heartbeats that count slots without itemizing them are left
alone.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* admin/plugin: let a partly itemized listing clear a dispatch's misses

A heartbeat that counted more slots than it itemized returned before the
listed jobs had their misses reset, so an omission, a partial heartbeat
listing the job, and another omission failed a job the worker was still
running.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin/plugin: look up the worker's stream before tracking the attempt

A dispatch to a worker with no stream returned after the job was already
recorded as assigned, and that record kept new proposals with the same
dedupe key out until the stale-job sweep expired it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin/plugin: fail the attempt whose request could not be sent

The dispatch now sends on the stream it looked up before tracking the
attempt, so a worker that reconnects in between makes the send fail.
That returned with the job still tracked as assigned, and with the
worker back nothing closed the record for the 24h stale timeout while
its dedupe key kept new proposals out. The attempt is now completed as
failed, after it is unregistered so the completion does not wait on a
channel nobody reads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin/plugin: leave a send failure to the expiry that already claimed it

The send-error path and an orphan expiry could both complete the same
attempt and record two failed runs for it. The send-error path now
completes the attempt only if no expiry has marked it failed, decided
under the same lock that unregisters it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin/plugin: record a send failure whose entry Shutdown already removed

Shutdown closes shutdownCh before it unregisters pending dispatches, so a
blocked send returns while its map entry is gone. Read the expiry claim
from the dispatch's own state instead of the map lookup, so the attempt
is still failed and does not stay assigned.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin/plugin: match a padded job id against its trimmed listing

Heartbeat work ids are trimmed before the orphan check, so compare the
dispatch's job id trimmed too; otherwise a job id with surrounding
spaces never matches its own listing and the running job is failed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin/plugin: skip orphan misses once a completion was handed over

The buffer length only shows a completion the dispatcher has not taken
yet; after it takes one, the attempt stays registered until the dispatch
returns. Mark the hand-off under the pending lock and check that flag
instead, so heartbeats in that gap cannot fail a finished attempt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 11:45:41 -08:00
Chris LuandClaude Opus 5.5 3d411c6a6c volume: report EC file and delete counts in incremental shard messages (#12404)
* seaweed-volume: report EC file and delete counts in heartbeat deltas

The full EC heartbeat already carries file_count and delete_count, but the
incremental new/deleted shard messages built in
collect_ec_shard_delta_messages left both at zero. A volume the master first
learns of through a delta then reads 0 files until the next full EC
heartbeat. Fill both from file_and_delete_count, the same source the full
heartbeat uses, computed once per volume rather than once per shard.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* volume: report EC file and delete counts in mount notifications

Go parity with the Rust delta change: the NewEcShardsChan messages sent on
MountEcShards and on startup shard load now carry FileCount and DeleteCount
from EcVolume.FileAndDeleteCount, the same values the full EC heartbeat
reports, instead of leaving them at zero.

Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi

* seaweed-volume: leave count-only EC changes out of the shard delta

The delta messages now carry file_count and delete_count, so the whole-message
comparison re-sent every shard of a volume after any delete, and the master
broadcast each one as a new EC vid. Compare the shard fields only; the full EC
heartbeat already carries the counts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* topology: mount messages now carry EC counts, only older servers omit them

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: guard the captured EC mount messages in the disk-type tests

The drain goroutine appended to the slice that waitForShardMsg polled with no
lock, which go test -race reports.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* storage: test EC counts on the startup notification and repeated .ecj ids

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: move file_count too in the count-only delta test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 10:43:04 -08:00
Chris LuandClaude Opus 5.5 3cbace2471 s3: expose ReadAcp/WriteAcp in shell and admin UI, log ACL-upload denials (#12396)
* shell: accept ReadAcp and WriteAcp in per-bucket and service account actions

s3.bucket.access, s3.anonymous.set and s3.serviceaccount.create rejected
ReadAcp and WriteAcp, so the ACL actions the S3 gateway enforces could not
be granted through them. Share one canonical action table across the
three commands and list the ACL actions in their help.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: list ReadAcp and WriteAcp in s3.configure and s3.circuitBreaker help

The -actions usage of both commands stopped at Admin, so the ACL actions
the gateway enforces could only be found in the source. s3.circuitBreaker
now prints the actions it actually accepts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: offer ReadAcp and WriteAcp in the object store user permissions

The create and edit user forms only offered Read, Write, List, Tagging and
Admin among the bucket permissions, so an identity could not be given the
ACL actions from the UI even though uploads with x-amz-acl need WriteAcp.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: log the WriteAcp denial of an upload that sets an ACL

An upload or copy that carries x-amz-acl or an x-amz-grant-* header also
needs WriteAcp. When that check denies, the client only sees a bare
AccessDenied and nothing reached the log at the default verbosity, so
the header was never suspected. Log the identity and the missing action.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: quote the key in the ACL upload denial log

An object key can carry a decoded newline, which let a caller add forged
lines to the default log. Quote the bucket and key, and word the message
so it also fits a policy deny, not only a missing grant.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: say ACL uploads need both Write and WriteAcp

The s3.configure example granted WriteAcp alone, which still fails the
upload's Write check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: parse S3 action lists with one shared helper

s3.bucket.access, s3.anonymous.set and s3.serviceaccount.create each had
their own validation loop, so a test of the table did not cover the
parsing the commands run. They now call canonicalActionList, which skips
empty entries and repeats. s3.anonymous.set validates before contacting
IAM, and s3.bucket.access still rejects a list with no actions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: keep bucket-scoped actions the user form cannot show

When a user has both global and bucket-scoped actions, the edit form
shows only the global ones, and saving replaced the identity's actions
with what the form rebuilt, deleting the scoped grants. Carry those
actions through the save.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: list hidden bucket grants in the user edit form

Restoring the bucket-scoped grants the form cannot show kept them active
even when an admin narrowed the user's permissions, with no way to see
or drop them. Show them as checked boxes under Other Bucket Permissions;
unchecking one removes it on save.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: reject an s3.anonymous.set -access list with no actions

A blank list such as ",," parsed to nothing and cleared the bucket's
anonymous grants. Use "none" for that, as s3.bucket.access does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: skip the ACL upload denial log for anonymous requests

Anonymous uploads with ACL headers can be sent by anyone, so logging
each denial at the default level lets unauthenticated clients flood the
log. Keep the line for named identities.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: count kept bucket grants when checking an edited user for permissions

The empty check ran before the checked Other Bucket Permissions were
merged, so a user left with only those grants could not be saved.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: list bucket grants of Admin users in the user edit form

parseBucketPermissions returned before reading scoped actions when the
user had Admin, so removing Admin and saving dropped grants such as
WriteAcp:bucket.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: keep unpaired bucket grants as separate checkboxes in the user edit form

The form saves every selected permission on every selected bucket, so
grants such as ReadAcp:a with WriteAcp:b came back as all four pairs.
Show the grants through the picker only when they cover that full
product; otherwise list each grant on its own. Opening the form also
resets the bucket scope left over from the previous user.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: keep grants the edit pickers have no option for

A permission x bucket product was shown through the pickers even when a
bucket was not in the bucket list (deleted, prefix or wildcard grants)
or a permission had no picker option (DeleteBucket, PutBucketQuota).
Saving then dropped those grants. Such grants now go to the checkbox
list, global ones included, and are submitted unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: name the s3.bucket.access anonymous example by what it grants

Read,Write,List leaves out ReadAcp and WriteAcp, so it is not full
access.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: keep unlisted global grants of Admin users in the user edit form

parseBucketPermissions returned for Admin before splitting global grants
into picker options and the rest, and only raw option values were
re-selected. Saving an Admin user dropped s3tables:* and grants with no
picker option such as DeleteBucket. Admin users now use the same split.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: keep the /* suffix of bucket grants in the user edit form

parseBucketPermissions dropped a trailing /* before matching bucket
options, so saving rewrote ReadAcp:bucket/* as ReadAcp:bucket. The bare
form also matches bucket-level requests such as GET ?acl, so the save
widened the grant. A /* bucket is not a picker option, so the grant now
stays in the checkbox list as written.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: test that s3.bucket.access saves ReadAcp and WriteAcp grants

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 09:55:54 -08:00
Chris LuandClaude Opus 5.5 c437791991 s3: reject POST Object header fields with control characters (#12400)
A Content-Type form field such as "\n" was copied verbatim into the
multipart part header sent to the volume server, which broke the
framing and surfaced as a Content-MD5 mismatch and a 500. Form fields
promoted to request headers must be valid header values, as on a PUT;
otherwise answer 400 MalformedPOSTRequest.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 09:43:48 -08:00
0742d316aa fix(s3): accept fresh admin tokens when context carries stale authorization (#12390) (#12397)
* fix(s3): accept fresh admin tokens when context carries stale authorization (#12390)

Signed-off-by: Mohd Quamar Tyagi <104281681+Tyagiquamar@users.noreply.github.com>

* security: share the admin Bearer check between S3 and admin-plane peers

Move the authorization scan into security.CheckGrpcAdminToken, built on
IsFilerAdminToken, and call it from CheckGrpcAdminPeer and the S3 IAM cache
check. Both now accept a valid admin token in any authorization value
instead of S3 re-implementing the decode and claim checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: accept a fresh admin token after a stale one on IAM gRPC

The S3 IAM cache check is meant to mirror the filer's, so route the filer's
checkAdminAuth through security.CheckGrpcAdminToken too. IamGrpcStore appends
its token to the caller's context, so a forwarded older token would otherwise
still shadow it here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* security: parse at most four authorization values per admin call

Scanning every value let one request repeat a junk Bearer token (cheap over
HPACK) tens of thousands of times and burn about half a second of JWT parsing
on an unauthenticated port. Real callers send their own token plus at most one
forwarded from their context.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* credential: match the other WithS3InternalAdminAuth replace fix

The plugin worker fix makes the same set-instead-of-append change to
WithS3InternalAdminAuth with different code and a different test at the same
spot, so whichever lands second would conflict. Use the identical code and
test so the two merge cleanly in either order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* security: test an expired admin token ahead of a fresh one

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* security: test that a same-key session token is not an admin token

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test IAM gRPC with only invalid authorization values

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Signed-off-by: Mohd Quamar Tyagi <104281681+Tyagiquamar@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 09:42:25 -08:00
Chris LuandClaude Opus 5.5 51ee165279 ec: rebuild a needle when other shards are corrupt over its range (#12401)
* storage: search corrupt shard sets when an EC needle fails to rebuild

A needle whose reassembled bytes fail to parse is rebuilt with its own
shards excluded. When other shards are corrupt over the same range, that
decode reads them as inputs, and the retry dropped only one input at a
time, so two or more extra corrupt shards left the needle unreadable even
within parity.

After the single pass fails, try every set of up to ParityShards shards
that holds a suspect as the corrupt set, smallest first. Suspect ranges
inside the set are rebuilt from the shards outside it and the rest keep
their bytes, so a clean suspect can still feed the decode. Sets whose
decodes read the same inputs are tried once, and the search is bounded
by set count and by rebuilt bytes past the single-shard sets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: search corrupt shard sets when an EC needle fails to rebuild

Mirror the Go read path: after the single pass over the suspects fails,
gather each interval from every other shard once and try each set of up
to parity_shards shards holding a suspect as the corrupt set, smallest
first, with the same dedup and bounds. The search runs on the blocking
pool, holding the recovery permit, since it can take many decodes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec: charge only multi-shard sets against the rebuild byte budget

The budget check already let single-shard sets through, but each one
was still charged, so a needle spanning many shards could spend the
budget before the first multi-shard set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec: bound the corrupt-set gather to the recovery budget

The search gathered every other shard per interval, so a large needle
spanning several shards held up to Total-1 copies of itself under a
permit capped at the budget. Gather only as many as the budget covers,
but never fewer than the suspects-excluded gather the fallback always
held, so single-suspect recovery keeps every other shard.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec: gather at least one decode's worth for the corrupt-set search

A needle over more shards than parity skips the single pass, and its
suspects-excluded width can fall below DataShards, so the bounded
gather left even one corrupt shard unrecoverable. Floor the width at
DataShards, which the per-suspect rebuild this search replaced read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec: rebuild one suspect at a time when the search gather is narrow

Flooring the search gather at DataShards made a large needle over more
shards than parity hold ten copies of itself, where the per-suspect
rebuild it replaced held one interval's fan-out. Keep that rebuild
after the search, and skip the search when the budget leaves fewer
inputs than one decode. Parity slack still widens the gather to every
shard outside the suspects, as the dropped-input retry did.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ec: keep the dropped-input retry and run the set search within budget

A narrowed search gather skipped only each interval's own shard, so it
could read corrupt suspects in place of healthy shards the old retry
kept, and its byte bound could stop before that retry's hypotheses.
Restore the dropped-input retry as it was, move the search into its
own function after it, and run the search only when every interval's
full gather fits the recovery budget. Past the budget a read takes the
same paths, with the same footprint, as before the search.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 09:41:21 -08:00
Chris LuandClaude Opus 5.5 8f29e38837 grafana: chart EC reconstruction and other unplotted metrics (#12406)
* grafana: chart EC intervals served by reconstruction

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: chart volume server storage IO errors and quarantine

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: chart frozen volume growth and duplicate volume ids

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: chart filer metadata log gaps and replay failures

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: add failed bytes to filer sync throughput

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: chart remote mount cache reads and evictions

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: chart plugin worker detection duration

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: show duplicate volume ids whose first detection was the only one

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* grafana: keep cluster apart in the master growth and duplicate id panels

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 09:37:43 -08:00
d6c65cd924 volume: count EC intervals served by reconstruction (#12392)
* volume: count EC reads that fall back to reconstruction

When a needle read can't read an EC shard interval directly (the shard isn't
mounted here and no holder answers), both volume servers rebuild it from the
other shards and the read still succeeds, with nothing to show for it.

Count those fallbacks in SeaweedFS_volumeServer_ec_degraded_read_total{result}
(success, failure), one per interval, at the one place each server falls back:
Go readOneEcShardInterval, Rust fetch_one_interval. Corruption repair and scrub
aren't counted. A line binding each one to the needle and the volume's encode
generation is logged at V(3) in Go and debug in Rust.

* volume: name the EC counter for what it counts

Reed-Solomon reconstruction can't tell corrupt inputs from good ones, and the
needle is validated later, so "success" only ever meant the reconstruction
produced data. Rename the counter to
SeaweedFS_volumeServer_ec_reconstructed_interval_total and the log line to
"ec reconstructed interval", say so in the help text, and pin the behaviour
with a test: an interval rebuilt from corrupt parity counts success while the
read itself fails.

* storage: restore stderr when the log-line test stops early

A fixture failure in readEcNeedle ends the test through t.Fatalf, which
skipped restoring os.Stderr and closing the pipe, so later tests wrote
into a pipe nobody read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 08:30:41 -08:00
15ff5c113a fix(s3tables): validate policy conditions and fail safe on unusable ones (#12389)
* s3tables: validate policy conditions and fail safe on unusable ones

* s3vectors: reject a condition operator with no keys

The shared evaluator now treats a keyless operator as unusable, so a
stored keyless Allow would never grant while the write succeeded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3vectors: reject an Allow whose Condition is null

A null Condition decodes as no Condition, so the Allow granted
unconditionally. A null Condition on a Deny only widens the veto and
stays accepted, as in the S3 Tables validator.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: treat a NaN numeric condition value as unreadable

strconv.ParseFloat accepts "NaN", but no comparison with it holds, so a
Deny on NumericLessThan never applied and an Allow on NumericNotEquals
matched every request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: reject condition keys the evaluator never supplies

A condition on a key such as aws:SourceIp reads no request value, so a
Deny on it never applied and an Allow never granted. Accept only the
keys getConditionContextValues resolves, as the vector validator does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3vectors: require Statement and Effect spelled as written

The evaluator decodes field names regardless of case, so an "effect" or
"statement" key slipped past the null Condition check and the other
raw-key checks while the stored Allow still granted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: treat an infinite numeric condition value as unreadable

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: a stored Allow with a null Condition never grants

The typed decode reads "Condition": null as no Condition, so an Allow
stored before validation refused it still granted unconditionally, for
table and vector policies alike. Record the literal null while decoding
a Statement and skip such an Allow during evaluation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: validate a null Condition from the decoded Statement

validatePolicyDocument re-parsed the document to find a literal null
Condition; the Statement decode now records it, so the validator reads
the same flag the evaluator does and nullConditionStatements goes away.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: keep Condition objects merged across key spellings

Decoding Condition through a json.RawMessage kept only the last of
"Condition" and "condition", so an empty object under one spelling
erased the other's restriction. Decode the statement as before and
probe for the literal null separately.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: refuse a Condition split across key spellings

Condition objects under differently cased keys decode into one map, and an
operator they share keeps only the last spelling's keys, so a restriction
under the other spelling was dropped. Validation now refuses a statement
whose Condition is under more than one key, and one stored before that
never grants on an Allow and vetoes on a Deny.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: refuse a Condition key repeated under the same spelling

PolicyDocument re-marshaled each statement through interface{}, and
Statement.UnmarshalJSON counted keys from a map, so both kept only the
last of a repeated Condition key. "Condition":null,"Condition":{} or two
Condition objects sharing an operator passed validation with a restriction
dropped. Statements now decode from their original bytes and the keys are
counted while walking the object.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3vectors: refuse a repeated Condition key in a vector bucket policy

The validator read Condition from a map, which keeps only the last of a
repeated key, so it accepted a statement the evaluator never grants on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: keep the re-marshaled statement decode, count Condition keys apart

Decoding statements from their original bytes let document order pick
between spellings of every field, where the re-marshal sorted the keys
first, so a stored "effect":"Deny","Effect":"Allow" became an Allow.
Statements decode as before again; only the Condition keys are counted
from the original bytes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: test that validation refuses a split Condition on a Deny

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3vectors: test that validation refuses a repeated Condition on a Deny

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 08:13:05 -08:00
Chris LuandClaude Opus 5.5 d727c6970f plugin worker: stop forwarding the stream's admin token to job RPCs (#12391)
* plugin worker: keep the stream's admin token out of job contexts

The bearer token minted when the worker opens its control stream was
attached to connCtx, which every detection and execution context derives
from. Job RPCs therefore forwarded that connect-time token, and when a
job added its own fresh token (s3_lifecycle's LifecycleDelete) the S3
server validated the stale one first and rejected the call once it
expired.

Attach the token only to the WorkerStream call, as the Rust worker does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* credential: replace an inherited admin token instead of appending one

WithS3InternalAdminAuth appended its fresh Bearer token, so a context
that already carried an authorization value sent both, and the S3
gateway's checkAdminAuth validates only the first. Set the header on a
copy of the outgoing metadata so the fresh token is the only one sent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* plugin worker: read the captured job metadata after the stream ends

Both result channels could be ready at once, letting select fail a
successful run. The handler reports before JobCompleted reaches admin,
so wait for the stream to finish and then read the captured metadata.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* plugin worker: test that dropping the connection cancels running jobs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 05:26:12 -08:00
Chris LuandClaude Opus 5.5 a0f30afe5f s3: keep service account keys when a pushed identity replaces its parent (#12394)
* s3: keep service account keys when a pushed identity replaces its parent

Service account changes push the parent user, whose proto carries only
the user's own keys. The merge replaced the cached parent with that
proto and dropped every service account key until a restart. Read the
parent's service accounts from the credential store so the push
rebuilds them, deleted and disabled ones included.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: test that a parent identity push keeps its service account keys

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: test a sibling service account survives a deletion push

Also check the restored secret and status, not just the key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 05:26:07 -08:00
Chris LuandClaude Opus 5.5 92bdfeaf29 seaweed-vector: record seaweed-worker-core's toml dependency in Cargo.lock (#12388)
seaweed-worker-core now depends on toml, but the seaweed-vector
workspace lockfile was not refreshed, so cargo build --locked fails
before the S3 Vectors integration tests start.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 20:48:56 -08:00
Chris LuandClaude Opus 5.5 ba3fd257a3 filer: clear a dead test replica's reachability record before closing it (#12387)
* filer: clear a dead test replica's reachability record before closing it

The read path tries hosts that recently failed last. A dead test server
can land on a port an earlier test left marked as failed, so the live
replica answered first, nothing failed and the location refresh under
test never ran. Have the dead replica answer once through util_http so
its record is cleared before it closes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: close the dead test replica when reaching it fails

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: clear the live test replica's reachability record too

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 20:47:43 -08:00
329ccb8bc0 master: tell a warming-up Assign or LookupVolume client when warm-up ends (#12386)
* master: tell a warming-up Assign client when warm-up ends

During warm-up the master refuses Assign with Unavailable and now attaches
the remaining warm-up as a gRPC RetryInfo detail. RetryWithBackoff may
shorten a backoff step to that hint plus 50 ms, never lengthen it, and only
for a valid, positive hint, so a writer retries right after warm-up ends
instead of up to one backoff step later. The warm-up itself is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* master: jitter the retry-hint slack and hint LookupVolume too

Per review: the slack added to a server hint is now random in
[50ms, 300ms], so clients refused at the same moment do not all retry
in one instant. The warm-up refusal of LookupVolume carries the same
RetryInfo hint as Assign; the masterclient's lookup retry already goes
through RetryWithBackoff, so it lands at warm-up end too.

Tests: the slow guard tests are replaced by one sleep-free table test
of the step logic (zero, negative, huge, malformed, valid and wrapped
hints, a hint longer than the step, none). One timing test of the
shortened step stays, plus a jitter-bounds test and a LookupVolume hint
test. The no-hint control test, which was timing-sensitive, is dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* master: name the shared warm-up test and file the lookup test by topic

Rename the no-remaining test to TestWarmingUpErrorWithoutRemainingHasNoHint,
since the helper now serves Assign and LookupVolume, and move the
LookupVolume hint test next to the other LookupVolume warm-up tests.
Tests only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* util: read the retry hint through ServerStatus

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* master: bound the warm-up hint in tests by the remaining warm-up around the RPC

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: leave more scheduler margin in the retry-hint timing tests

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* util, master: trim retry-hint comments

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: widen the retry-hint timing bounds to just under the unhinted wait

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: require a refused warm-up call before the hinted assign retry

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: check the hinted retry wait through a test timer, not the wall clock

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: move the assign warm-up end to 2.6s for more hinted-retry headroom

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: require the hinted retry to wait out its timer

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-03 20:46:58 -08:00
Chris LuandClaude Opus 5.5 cd82e663ad admin: credit an unitemized slot to a pending job only after a later heartbeat (#12385)
* admin: credit an unitemized slot to a pending job only after a later heartbeat

A pending reservation the worker does not list took one of the slots
the worker counts without itemizing, even when that heartbeat came
before the reservation and could not have counted the job. The slot
belonged to unknown work, so the admin undercounted it and could admit
one job too many, past a job type's per-worker limit, until the next
heartbeat.

A reservation now records when it was made, and claims an unitemized
slot only from a heartbeat received at or after that time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: time unitemized-slot credit by the heartbeat, not the last hello

A reconnect's hello moves LastSeenAt but keeps the old heartbeat, so a
pending reservation could claim a slot counted before it existed, and a
completed job's echo could end without a new heartbeat. Record when the
heartbeat itself arrived and compare reservations against that.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: a completed job claims no slot from a heartbeat before its reservation

A job that completes before any heartbeat counts it has no echo, so the
slot the last heartbeat counted is other work. Crediting it to the job's
type let another job type pass its per-worker limit until the next
heartbeat.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 20:45:39 -08:00
github-actions[bot] 995084f71e docs: regenerate star history chart 2026-11-04 00:54:17 +00:00
Chris LuandClaude Opus 5.5 5dfda3645e admin: count executor slots per job type and stop lanes waiting on capacity they cannot free (#12384)
* admin: count executor slots per job type, deduped against the heartbeat

per_worker_execution_concurrency is a per-job-type setting, but the
scheduler compared it against the worker's execution_slots_used, which
counts every job type. One s3_lifecycle job a worker reported running
after a reconnect made admin_script (per-worker 1) undispatchable on a
worker with three free slots. A reservation and the heartbeat entry for
the same job were also charged as two slots, so a per-worker limit of N
let N-1 jobs run.

Reservations now carry the job id and job type. The per-type limit is
checked against that type's work alone, the worker's slot total against
work of every type, and a heartbeat entry covered by a reservation is
counted once. Per-lane reservation pools go away: the per-type count is
what kept lanes from starving each other.

* admin: stop a lane waiting on executor capacity it cannot free

A queued job no executor could take was retried every backoff until
job_type_max_runtime ran out, up to 30 minutes, while the default lane
sat in executing and every other job type in it idled.

The dispatch loop now asks what holds the capacity. Slots held by the
lane's own jobs release when they complete, so it keeps waiting. Slots
held by other lanes or by work this admin never dispatched clear on a
schedule the lane has no view of, so the queued jobs are canceled at
once with a skipped_no_executor activity naming the worker, its slot
count and the job types holding it, and the next detection proposes
them again. A missing worker is waited for only the orphan grace.

* admin: cancel a job on its worker when the admin expires it

Expiring a job, by the stale sweep or by the operator's expire endpoint,
only failed the tracked record. The worker never heard about it and kept
running the job with its slot taken, so a job stuck in "starting" held a
slot for over a day and, with the old slot accounting, blocked a whole
lane.

Expiring a job now also sends the worker a cancel for it, so the slot
comes back on the next heartbeat.

* admin: count a completed job's heartbeat echo as its lane's own slot

The worker sends JobCompleted before the heartbeat that frees the slot, so
right after a dispatch goroutine releases its reservation the job still
shows in the worker's running work. That entry was counted as foreign, and
a lane with per-worker 1 gave up and canceled the rest of its queue after
the first job. A released reservation now stays for completedJobEchoGrace
and claims that heartbeat entry as the lane's own load.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: let the admin's own jobs claim slots a worker does not itemize

The Rust worker reports only a slot count, every 10s, and the Go worker
counts a slot one heartbeat before it lists the job. Those slots were
charged on top of the reservations behind them, and once a job completed
its stale count read as foreign work, so the lane gave up its queue.
Unitemized slots now go first to this admin's unlisted reservations and
recent completions; the rest are of unknown type and count against every
type's per-worker limit, not only the slot total.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: give up pinned jobs whose busy home a free worker cannot relieve

A job homed on a replica-host worker stays on that home while its queue is
shorter than LocalQueuePin, so a free remote worker cannot take it. The
outlook treated that free worker as a reason to wait, and a home held by
foreign work parked the lane for its whole window. A free worker now only
holds the wait for the orphan grace, after which the held homes decide.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: let a redispatched job's reservation replace its released one

A job the worker turns away for capacity is released, kept as an echo, and
queued again under the same id. Once the new attempt ran, both the echo and
the new reservation matched its heartbeat entry and it counted as two
slots. Reserving a job now drops what an earlier reservation of it left.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: credit a count-only slot to a completed job only until the next heartbeat

A released reservation claimed an unitemized slot for its full 30s grace,
so on a count-only heartbeat other work of another type was charged to the
completed job's type. A heartbeat that arrives after the release already
reflects the freed slot; from then on the slot is of unknown type.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: name unitemized slots in the no-capacity reason

holders() listed only the job types the worker itemized, so a worker with
named and count-only work read as "2/2 slots used by <type>".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: keep the blocked clock running until a reservation succeeds

A popped job whose reservation then failed reset the clock before the
failure pushed it back, so pop-then-fail cycles never aged the block.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: resend an expiry cancel the worker did not receive

Expiring a job sends its worker one cancel. When the worker is
disconnected, or its outgoing buffer stays full past the send timeout,
that cancel is dropped. The record is already failed, so later sweeps
and expire calls skip it, and a worker that kept the job running (a
detached job across a reconnect, or a jammed stream) holds the slot
until the job ends on its own.

The admin now holds an undelivered expiry cancel and sends it again on
the worker's next heartbeat that still lists the job. It is forgotten
when that heartbeat no longer lists the job, and skipped when the job is
active again, since a new attempt reuses the job id.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: cancel an expired attempt by its request id

The expiry cancel named the job id, and a retry reuses it. The worker
keys its cancel funcs by both ids, so a cancel arriving after a retry
started on the same worker stopped the retry. The held resend also gave
up on any job that was active again, and the expired attempt's own late
progress marks the job running, so a worker that buffered progress
while disconnected kept the expired job.

The cancel now names the attempt's request id, as the other execution
cancels do, and the held resend no longer depends on the tracked state.
Only a cancel that can name nothing but the job id still skips a job
that is active again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: ignore late reports from an attempt the admin already ended

Expiring a job now cancels it on its worker, and the worker answers
with its own JobCompleted for that attempt. That overwrote the expiry's
failure with the cancel error and logged a second run record under the
same run id. Late progress from the expired attempt likewise reopened
the record as running.

A progress update or completion whose request id names an attempt the
record already closed is now dropped after the dispatcher's waiter is
told. A retry runs under a new request id, so its reports still apply.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 15:10:26 -08:00
Chris LuandClaude Opus 5.5 ca15fdbff6 remote.uncache, remote.meta.sync, remote.cache: keep local writes that land while they run (#12382)
* remote.uncache: skip a file overwritten since the walk read it

The uncache cleared the chunks of the entry its directory walk returned and
wrote it back unconditionally. A file overwritten after the walk read it
was reset to the remote pointer, and its new chunks were dropped, so an
acknowledged write was lost. Write back only while the filer still holds
the entry as read, and report the file as skipped otherwise.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* remote.meta.sync: keep files written locally after the directory listing

meta.sync lists each local directory once and works from that listing.
A changed remote file was written back over the listed entry
unconditionally, and a name missing from the listing was created without
O_EXCL, so a file written locally in between was replaced by the remote
pointer either way. Update only while the filer still holds the listed
entry, create exclusively, and report such files as skipped. The create
now also surfaces errors the filer returns in its response.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* remote.cache: keep files written locally after the lookup

The metadata step looks each file up and then either rewrites it from the
remote unconditionally or creates it without O_EXCL, so a local write
landing between the lookup and the write was replaced by the remote
pointer. Reuse doSaveRemoteEntry, which updates only while the filer still
holds the looked-up entry, create exclusively, and report such files as
skipped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* remote.meta.sync: do not descend into a name a local file took

When the exclusive create of a remote directory finds the name taken, the
sync still descended into it, and creating the remote children under a
local file would turn that file into a directory. Descend only when the
name now holds a directory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* remote.meta.sync: delete only the file version it listed

The type-change and -deleteStale paths removed a remote-backed file
unconditionally, so a local write landing after the directory listing was
deleted with it. Condition the delete on the listed mtime, as fs.verify
and volume.fsck do; the filer then keeps a file written since.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* remote.meta.sync: condition directory and mtime-less deletes too

The listed-mtime cutoff was only sent for files, so a remote-backed
directory replaced locally by a file between the emptiness check and the
delete was removed unconditionally, and so was a file listed without an
mtime. Send the cutoff for directories as well, and for an entry listed
without an mtime keep it once a write has set one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test that DeleteEntry applies the request's mtime cutoff

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* remote.meta.sync: keep the listed nanoseconds as the cutoff of an epoch mtime

An entry listed at the epoch with a sub-second mtime was sent a 1 ns
cutoff, so the filer kept it unchanged although the old code deleted it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* remote.meta.sync: report an entry the filer kept instead of deleting it

The filer answers a delete it skipped under the mtime cutoff with
success, so the sync printed (delete) for an entry a local write kept.
deleteLocalEntry now looks the name up after the delete and prints the
outcome, and a replaced file that was kept is not recreated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 14:53:42 -08:00
Chris LuandClaude Opus 5.5 b0c6297041 s3api: per-bucket Block Public Access (BlockPublicAcls, IgnorePublicAcls) (#12381)
* s3api: store and serve a bucket's PublicAccessBlock configuration

PutPublicAccessBlock, GetPublicAccessBlock and DeletePublicAccessBlock
answered NotImplemented. Store the configuration in the bucket entry's
extended attributes next to the ownership controls, carry it in the
cached BucketConfig, and answer Get on an unset bucket with
NoSuchPublicAccessBlockConfiguration like S3 does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: refuse public ACLs when a bucket blocks them

With BlockPublicAcls set, a PutObject, CopyObject, CreateMultipartUpload,
POST Object, PutObjectAcl or PutBucketAcl whose canned ACL or grants name
the AllUsers or AuthenticatedUsers group is refused with AccessDenied.
Private and account grants keep working, so clients that send
x-amz-acl: private on every upload are unaffected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: ignore public ACL grants for anonymous reads under IgnorePublicAcls

A bucket with IgnorePublicAcls no longer lets an AllUsers grant on the
bucket or on an object answer an anonymous read or listing. Bucket
policies and identity permissions are unaffected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: reject PublicAccessBlock bodies with unknown elements

A misspelled element such as BlockPublicAcl was ignored, so the request
succeeded and stored the flag as false. Answer MalformedXML instead, and
store the configuration without the request's namespace.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: fail closed when a bucket's public access block cannot be read

A stored PublicAccessBlock that does not parse now blocks and ignores
public ACLs instead of acting as if none were set. PutBucketAcl and POST
Object return the bucket lookup error rather than skipping the
BlockPublicAcls check when the bucket config cannot be loaded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: test BlockPublicAcls on copy, multipart initiate and a PUT-then-GET

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: keep stored public access block bytes on unrelated bucket updates

An unreadable stored configuration was replaced with the in-memory
all-true fallback whenever another bucket setting was updated, losing
the original value. Write the key only when the update changes the
public access block.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 14:52:20 -08:00
ca4ce2c581 volume: a read plan waits for a vacuum commit instead of answering 404 (#12375)
* volume: a read plan waits for a vacuum commit instead of answering 404

A vacuum commit closes the needle map, swaps the compacted files in and
reloads the map. It holds the data-file write lease for that and takes no
store lock. needle_read_plan, which the HTTP GET path uses, read the needle
map without the data-file read lease, so a GET planned in that window found
no map and answered 404 for a live needle.

needle_read_plan now takes the read lease first, as read_needle_with_option
does, so the plan waits for the commit and then reads the reloaded map.
Go's readNeedle does the same under dataFileAccessLock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: plan a read with the store guard released

A plan that waits on a vacuum commit's data-file lease under the store
read guard parks store writers (allocate, mount, EC ops) on that commit,
and every store reader queued behind them. Look the volume up under the
guard and plan after releasing it, as Go's readNeedle does, and look the
volume up again if its handle was closed while the plan waited.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: retry a read plan while its handle keeps closing

A plan that waited out an unmount looked the volume up only once more,
so a replacement handle that also closed before its plan took the lease
answered NotFound for a needle a later mount holds. Repeat the lookup
while the planned handle is closed, at most three times, so a closed
handle that stays in the map still ends the loop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: start the swap-window check once the plan holds the volume

The swap-window test gave the plan 300 ms to answer early, so a reader
thread that started late passed the check without waiting on the
commit. Share the writer test's wait for the plan's volume handle and
start both timed checks after it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: run the swap-window read-plan test on a redb map too

A commit reloads the index through load_index_redb for the redb kinds,
a path the in-memory run never took during the swap window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: say a needle read plan is resolved under the data-file lease

The plan no longer resolves under the store guard, and resolving it
takes the data-file read lease.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: read the needle through the plan in the read-plan tests

The tests checked only that the plan resolved and then read the needle
through Store::read_volume_needle, so a plan with a wrong offset or size
after the commit still passed. Read through the plan, as GET does, and
compare the data.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: test the read plan's lookup after an unmount closed the handle

No test reached the closed-handle retry, so dropping it left every test
green. Take the lookup as a closure so a test can hand the plan the
handle an unmount closed and then the remounted volume, and check that
the plan reads the needle from the remount.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-11-03 14:50:30 -08:00
Chris LuandClaude Opus 5.5 9b5f7bf65b weed-worker: find jwt.filer_signing.key where the Go worker does, and let the chart deliver it (#12373)
* seaweed-common: share the security.toml search path with the volume server

* weed-worker: read jwt.filer_signing from security.toml and WEED_JWT_* like the Go worker

* helm: give the worker-lance container security.toml and the extra environment

* weed-worker: treat an empty signing key in the environment as unset

Viper ignores an empty environment value, so the Go worker falls back to
the security.toml key when WEED_JWT_FILER_SIGNING_KEY is blank. The Rust
worker took the blank value and connected with no token. An empty
--filer-signing-key / WEED_FILER_SIGNING_KEY also falls through to the
security.toml lookup now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-worker: read a quoted expires_after_seconds like viper

The Go worker reads jwt.filer_signing.expires_after_seconds with GetInt,
which also accepts a quoted number. The Rust worker dropped a quoted value
and minted tokens with no expiry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-worker: warn when security.toml cannot be read or parsed

Viper logs a warning when an optional config file is broken. The Rust
worker skipped it silently, so a typo looked like an install with no
signing key. Log the path and the parse message, without the quoted line,
which can hold a key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* helm: keep POD_NAME and LANCE_NAMESPACE_URL out of worker-lance's extra environment

A duplicate name renders after the chart's entry and wins, silently
changing the Lance worker's --id or --namespace.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-worker: list /usr/local/etc/seaweedfs in the security.toml search path

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-worker: an explicit --filer-signing-key-expires-after-seconds wins

The flag was silently ignored whenever the key was discovered, from
WEED_JWT_FILER_SIGNING_KEY as well as from security.toml. Use it when
given, else the expiry configured beside the discovered key, else zero.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-worker: test where filer_signing looks for security.toml

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-worker: read a float expires_after_seconds as viper's GetInt does

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-worker: cap the minted exp so a huge expiry cannot overflow

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 14:49:39 -08:00
Chris LuandClaude Fable 5.1 de4a71665e test/s3vectors: wait for the S3 port before handing out the cluster (#12383)
The fixture waited for the Lance catalog port and the engine, but weed s3
opens its HTTP port last, about half a second after the catalog here. The
open cluster's engine now registers within that window, so
test_engine_without_credentials met a connection refused on the go and rust
single legs. Wait for the S3 port like the other listeners.


Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-03 14:08:42 -08:00
Chris Lu ab6aa9b645 admin: let a buffered detection completion win over the stream closing (#12379)
The detection waiter selects on the request's completion channel and on
the stream's done channel. A worker that finishes a scan and then drops
its stream (a restart right after a scan is the usual shape) buffers the
completion into the request before the receive loop sees the EOF, and
the stream's cleanup then closes done. A waiter that reaches its select
after both are ready can take the disconnect branch, and a completed
detection, proposals included, is failed as "detection worker
disconnected" and retried or recorded as an error.

When the stream closes or the liveness tick finds the worker gone, the
waiter now takes a completion that is already queued and returns it as
the success it is; the detector lease still goes, since the worker that
answered is unreachable. A ctx that ended keeps its meaning: the caller
stopped wanting the answer. The wait loop moves into
awaitDetectionOutcome so a test can hand it both channels already
ready; before this change that test fails on about half its runs. A
completion that lands after the waiter gave up stays unmatched, since
the request's pending entry is gone and the next request has a new id.
2026-11-03 12:53:28 -08:00
Chris Lu 836c60b2c9 topology: refresh EC file and delete counts from heartbeats (#12378)
* topology: refresh EC file and delete counts from heartbeats

UpdateEcShards only re-stored a heartbeated EC record when shard
membership changed, so FileCount and DeleteCount stayed at whatever the
first heartbeat after registration carried, while the volume server
reports fresh tombstone counts on every full EC heartbeat. Per-collection
object counts and EC vacuum candidate detection read the stored values.

Compare the counts per (volume, disk) against the record already in
hand and copy them onto the stored record in place, so the refresh
raises no new/deleted shard events and rebuilds no maps. A message
reporting 0/0 did not count (older volume servers, and the incremental
mount message omits the fields), so it keeps the stored values, also
across a membership-triggered rewrite. The incremental path's existing
branch picks counts up the same way.

* topology: swap EC shard records instead of mutating them in place

GetEcShards hands out the stored *EcVolumeInfo pointers and releases
ecShardsLock; CollectionStatistics, CollectionEcVolumeStats, ToDiskInfo
and the master's shard broadcast read fields through them afterwards.
The heartbeat count refresh wrote FileCount/DeleteCount into those
records under the lock, which the readers never take, so it was a data
race and a reader could pair a FileCount from one heartbeat with a
DeleteCount from the next. Disk.AddOrUpdateEcShard had the same write
on the incremental path, and its shard-bits/ratio merge was already
in place.

Stored records are now never written after they are published: a count
refresh, an incremental merge and a shard delete each build a copy
beside the stored record and swap the map entry under ecShardsLock, so
a reader holding the old pointer keeps one heartbeat's snapshot. The
refresh compares against whatever record is current under the lock,
since the incremental path may have replaced it since UpdateEcShards
looked. The 0/0-means-not-reported rule is unchanged.

The count tests now assert that the stored pointer changes while a
previously obtained one still reads its old values, and a -race test
runs full and incremental count refreshes against readers; it failed
on the previous code and passes now.
2026-11-03 12:48:16 -08:00
Chris LuandClaude Opus 5.5 5063aa9be2 weed-vector: check master certificates against the server allowlists, read -config_dir (#12374)
* weed-vector: look for security.toml in -config_dir first

weed processes search the directory -config_dir names before ., so a
cluster keeping security.toml there left the engine in plaintext.
--config-dir (also -config_dir, WEED_VECTOR_CONFIG_DIR) does the same.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: dial the masters the way weed clients do

The master's certificate must chain to grpc.ca and is not checked against
the host dialed, as weed clients verify (advancedtls.CertVerification), so
masters sharing one workload certificate, or dialed by IP, register as
they do from weed. Instead, grpc.client.master_allowed_server_commonNames,
grpc.client.allowed_server_commonNames or grpc.allowed_server_commonNames
(and the wildcard domain likewise) restrict which server identities are
accepted: the common name or a DNS SAN, or an exact URI SAN, as
security.Authenticator.AuthenticateServer does. These used to be refused
at startup; --tls-server-name is gone with the host check.

The TLS runs in a connector that reads the credentials asynchronously at
every dial, so the connect timeout still covers the reads, the TCP connect
and the handshake together. An IPv6 master's brackets are stripped before
the dial.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: the hardened leg allowlists server identities for grpc.client

The gateway and the engine then accept only servers named localhost, which
exercises the engine's allowed_server_commonNames check on the master dial.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: parse the refresh interval with time.ParseDuration's arithmetic

The interval was summed in floating-point seconds, so 0.9ns rounded up to
1ns where Go truncates it to 0 and falls back to the 5h default, 0.3s was
off by a nanosecond, and 3000000h was accepted where Go rejects anything
past the longest time.Duration. Each part is now an integer count of
nanoseconds plus the fraction Go adds, with Go's overflow bounds, and the
value is no longer trimmed, as Go does not trim it either.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: test a master dial whose credential read stalls

The bounded-dial tests passed no credentials or a prebuilt endpoint, so
moving the credential read out of the connect timeout would have failed
none of them. The new test points grpc.ca at a FIFO nobody writes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: dial a closed port in the stalled credential read test

Against a silent listener, a dial that never read the credentials timed
out with the same message, so the test passed without the read. Nothing
listens on the port now, so only the stalled read can time out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: test that Security::load searches the config dir

Only candidates() was tested, so load dropping config_dir on the way
would have passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: expand a leading ~ in -config_dir like weed's ResolvePath

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: hold the port in the stalled credential read test

A released port can be taken before the dial, and a dial that skipped the
read would then time out and pass. A listener that drops every connection
keeps the port and still fails such a dial at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-03 12:41:02 -08:00
Chris Lu 30f0364256 admin: fail an in-flight detection when its worker disconnects (#12376)
* admin: keep a reconnected plugin worker registered when its old stream unwinds

cleanupSession removed whatever session was registered under the worker
ID, so a worker that reconnected while its previous stream was still
unwinding lost the new session and its registry entry to the old
stream's deferred cleanup. Tie cleanup to the stream that is ending and
publish the new stream before the registry upsert so a lingering
predecessor sees itself superseded.

* admin: fail an in-flight detection when its worker disconnects

RunDetectionWithReport waited only on the caller's context and the
DetectionComplete channel, so a detection whose worker stream ended
mid-request could only finish by timeout. On a customer cluster a
planned restart of ec-vacuum-1 during an ec_vacuum detection left the
default lane in 'detecting' with every execution slot idle until the
3600s detection timeout, blocking every job type in that lane.

Bind the request to the stream it was sent on and fail the detection
as soon as that stream closes, whether by disconnect or by a reconnect
that replaced the session. A liveness tick also fails it once the
registry no longer considers the worker available (heartbeat stale),
sending a cancel in that case because the stream may still be open.
The detector lease is dropped so the next pick starts fresh, and a
'disconnected' detector activity is recorded. WorkerDisconnectForTest
now goes through the real session cleanup.

* admin: retry a scheduled detection once when its detector drops

When a detection fails because its worker disconnected, pick a detector
again and retry once within the same iteration while the job type's
runtime budget allows, instead of giving the whole cycle up. Any other
failure, or a failed retry, ends the run as before and the lane moves
on; the next cycle re-detects anyway.

* admin: register a plugin worker's hello and stream as one
 step

WorkerStream published the stream under sessionsMu but ran the registry
upsert after releasing it, so two concurrent streams for one worker ID
could publish their hellos in the opposite order from their sessions:
the registry then described the older hello's capabilities while only
the newer stream was sendable, and detector picks routed on them.

registerSession upserts the hello and installs the stream under one
sessionsMu critical section, and a hello or heartbeat arriving on a
stream that is no longer the registered one is dropped instead of
rewriting the live stream's entry.

* admin: steer a detection retry off the stream that dropped

The retry picked while the lost worker could still be in the registry:
cleanupSession closed done before taking the registry entry down, and a
stream closed by a reconnect keeps its entry on purpose. The lost worker
ranks first on free slots, so the single retry went back to it and
failed on send while a healthy worker sat idle.

The disconnect error now carries the stream it came from and the retry
picks around it; pickDetector only returns workers whose current stream
is open, excluding registry entries whose stream is closed; and
cleanupSession removes the registry entry before closing done, so a
waiter woken by the closure never sees the dead worker as a candidate.
2026-11-03 12:28:39 -08:00
Chris LuandClaude Opus 5.5 eeb8aefe70 weed-vector: gRPC TLS to the masters and for the gateway (#12368)
* weed-vector: read the cluster's security.toml

The engine finds security.toml where weed processes do (., ~/.seaweedfs,
/usr/local/etc/seaweedfs, /etc/seaweedfs) with WEED_* environment
variables over it, and builds gRPC client and server credentials by the
rules of security.LoadClientTLS and LoadServerTLS: a dedicated client
certificate is preferred, both halves of a pair must be set, and grpc.ca
is required once either is.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: gRPC TLS to the masters and from the gateway

The masters are dialed with [grpc.client]'s credentials when the cluster
has them, the master's certificate checked against the host dialed or
--tls-server-name. The engine's own listener takes its identity from
[grpc.vector] and then requires a client certificate chaining to grpc.ca,
which is how the gateway, dialing with [grpc.client], reaches it. The
rustls provider is pinned before the first TLS use, as the other binaries
linking both aws-lc-rs and ring do.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* scaffold: document [grpc.vector]

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: put every gRPC hop under mutual TLS in the hardened leg

VECTOR_TLS=1 now also writes [grpc.master], [grpc.volume], [grpc.filer],
[grpc.s3], [grpc.client] and [grpc.vector], and probes the master's and
the engine's gRPC ports: plaintext HTTP/2 and a client without a
certificate are refused, the client certificate gets a SETTINGS frame.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: give the test volume server 64 volume slots

Every collection grows 7 volumes and test_vector_bucket_encryption opens
two buckets while the previous test's bucket is still being torn down, so
16 slots were marginal: the Go legs failed CreateIndex now and then with
"failed to find writable volumes" and master.log showing "create 7
volume, created 4: Not enough data nodes found!".

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: reload the listener's certificates and enforce allowed_commonNames

The listener now runs the way weed servers run theirs. The certificate, key
and CA files are re-read every WEED_TLS_CERT_REFRESH_INTERVAL (weed's
default of 5h) and on SIGHUP, and new handshakes use the changed files
without a restart; a file that no longer parses is logged and the working
configuration stays. [grpc.vector] allowed_commonNames and
grpc.allowed_wildcard_domain restrict which client certificates are served,
checked against the subject common name as security.Authenticator and the
Rust volume server's CommonNameVerifier do.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* scaffold: allowed_commonNames and the refresh cadence for [grpc.vector]

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: the engine's allowlist keeps an outsider certificate out

The hardened leg sets [grpc.vector] allowed_commonNames = "localhost" and
probes the engine with a second certificate the same CA signed for
CN=outsider: the handshake must be refused, while the gateway's certificate
still gets an HTTP/2 SETTINGS frame.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* weed-vector: take the master's TLS name from the parsed endpoint

A master given without a port (https://master.example) had the whole URL
checked against its certificate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: bound the master TLS handshake

connect_timeout covers only the TCP connect, so a master that accepted the
connection and never answered the handshake held up the other masters.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: look up security.toml keys the way viper does

An empty WEED_* variable no longer hides the file's value, and keys match
regardless of case, as for weed processes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: refuse master allowlists it cannot enforce

weed checks a dialed server against allowed_server_commonNames and
allowed_server_wildcard_domain on top of grpc.ca. The engine's master dial
accepted every certificate the CA signed even when one was set; it now
fails at startup and names the key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: read the whole frame header in the gRPC TLS probe

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: probe the volume, filer and S3 gRPC listeners too

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: read the master dial credentials only when given masters

A standalone engine never dials with grpc.client, so its settings no longer
stop it from starting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: run each listener TLS handshake on its own task

The accept loop awaited every handshake before accepting the next
connection, so a peer that connected and sent nothing stopped every
gateway connection after it. Handshakes now run concurrently, each under a
10s timeout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: read the master dial's TLS files at every dial

The client certificate, key and CA were read once at startup, so after the
files rotated a reconnect still presented the old pair. Each dial now reads
them, as the listener re-reads its own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: read every unit Go durations accept in the refresh interval

WEED_TLS_CERT_REFRESH_INTERVAL values in ns, us or a leading + fell back
to five hours in the engine while weed servers honored them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: fall back on a refresh interval too long to hold

An oversized WEED_TLS_CERT_REFRESH_INTERVAL panicked the engine at
startup; weed servers reject it and keep the default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: refuse allowed_commonNames without the listener's certificate

With [grpc.vector] allowed_commonNames set but no cert and key, the
listener served plaintext and checked no client at all.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: read the master dial's credentials inside its connect timeout

The CA, certificate and key were read synchronously before the timeout
started, so a stalled credential path held up every other master.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: let a set key win over an empty spelling of it

With allowed_server_commonNames = "" beside allowed_server_commonnames
= "master", the lookup stopped at the empty exact match and the
allowlist went unnoticed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-03 11:42:49 -08:00
Chris LuandClaude Opus 5.5 2ec0623073 filer UI, admin: upload a folder (#12371)
* admin: place uploaded files by their path inside an uploaded folder

The multipart file name keeps only the base name, so the upload API takes
an optional relativePaths field listing each file's path under the target
directory. Paths that are empty, absolute or contain . or .. components
are rejected per file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: upload a folder from the file browser

The upload modal gets a folder picker, and a folder dropped on the listing
is walked into, so its files upload with their subfolder paths.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer UI: upload a folder

An Upload Folder button picks a whole folder, and a folder dropped on the
listing is walked into. Each file is posted to its subfolder under the
current directory, which the filer creates on the first upload into it.
Progress rows are keyed by that path, so equal names in different
subfolders no longer share a row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: reject an empty relativePaths field instead of ignoring it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: upload a large selection in batches below the multipart part limit

Go's multipart parser rejects a form of more than 1000 parts, so a folder
of about a thousand files failed as a whole. The upload modal now sends
500 files per request and sums the results; progress covers all batches,
and the button stays disabled until the last one finishes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: escape file names in upload error messages

A failed upload reports the client-supplied file or folder path, and the
toast renders its message as HTML.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: list failed uploads in the modal and reload the listing when it closes

A partial upload logged its failures only to the console before the page
reloaded, and a batch that failed after earlier ones succeeded left the
listing stale. The failures now stay in the open modal, and closing it
reloads the listing once any file was uploaded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: cancel the upload when its modal closes

Closing the modal, including with its Cancel button, left the batch loop
running, so the remaining batches were still written. The batch in flight
is now aborted, which also stops the loop, and the listing reloads since
the aborted request may already have stored files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: count upload progress by files when the selection has no bytes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: keep the newest upload selection when a folder drop finishes late

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: cap the upload preview at a fixed number of rows

A folder of many thousands of files rendered one preview row per file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer UI: upload a few files at a time

Every selected file started its own request at once, so a large folder
opened thousands of requests together.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: regenerate the gzipped admin.js

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: reload the listing after a batch fails without a result

A batch that fails mid-request may still have stored some of its files,
so closing the modal reloads the listing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: stop reading a folder drop once a newer selection replaces it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: skip the read error of a folder drop a newer selection replaced

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer UI: share the four-upload limit across selections

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: regenerate the gzipped admin.js

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: set up the file browser handlers once

admin.js already runs setupFileManagerEventHandlers on DOMContentLoaded,
so the file browser page registered every listener twice and each drop
was read twice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 11:35:32 -08:00
Chris LuandClaude Opus 5.5 d5e359d0e2 rust volume: run each gRPC TLS handshake on its own task (#12372)
* rust volume: run each gRPC TLS handshake on its own task

The incoming stream awaited every handshake before accepting the next
connection, so a peer that connected to a TLS gRPC port and sent nothing
stopped every gRPC connection after it. Handshakes now run concurrently,
each bounded by grpc-go's default 120s connection timeout, as a Go server's
are.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rust volume: retry a failed gRPC accept instead of closing the listener

grpc-go backs off and retries temporary accept errors such as EMFILE.
The TLS incoming stream ended on the first accept error, which stopped
the gRPC port for the rest of the process lifetime.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rust volume: test that the gRPC handshake timeout closes a silent peer

The handshake timeout becomes a parameter so a test can use a short one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 11:23:07 -08:00
Chris LuandClaude Opus 5.5 3b7706bad8 rust volume: exit cleanly on SIGTERM, and make the harness notice when a server does not (#12370)
* rust volume: exit cleanly on SIGTERM and SIGINT

The shutdown path raced the HTTP and gRPC tasks with select!, then
awaited both handles again; tokio panics when a JoinHandle is polled after
it completed, so every signal ended in "JoinHandle polled after
completion" with exit status 101 and the volumes never closed. Waiting
for both tasks with join! is enough: both stop on the shutdown signal, and
a gRPC serve error sends it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/volume_server: fail when a stopped server does not exit cleanly

The harness ignored the exit status after SIGINT, which is how the Rust
volume server's shutdown panic went unseen. Every server it stops must
now exit 0 within 10s. The Go volume servers get -preStopSeconds=0, which
the Rust ones already had, so a stop does not spend the default 10s grace
inside that window.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/volume_server: let assertArgPair find flags after a bare flag

rustVolumeArgs carries --volume.allowUntrustedRemoteEndpoints without a
value, so pairs no longer sit at even indexes and
TestRustVolumeArgsIncludeReadMode reported --readMode missing although it
was there.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* rust volume: stop both servers when either one exits

A gRPC task that panicked left the HTTP server and heartbeat running
until a signal arrived. Whichever server task ends first now sends the
shutdown and the other is awaited once, matching Go where a serve
failure is fatal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rust volume: report an HTTP task panic as a server error

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-03 10:44:01 -08:00
Chris LuandClaude Opus 5.5 cadb614630 filer.sync: stop chunk retries with the same supersession check the skip uses (#12367)
* filer.sync: pass the replayed entry down to the chunk retry gates

The fetch, manifest-resolve, and manifest-upload retry gates only received the
replayed version's mtime, which is not enough to order it against the live
source entry. Pass the entry itself; the gates still compare mtime only, so
behavior is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.sync: stop chunk retries with the same supersession check the skip uses

The fetch, manifest-resolve, and manifest-upload retry gates stopped early only
for a gone or strictly newer source entry. A transient error therefore kept
retrying a version the source had rewritten under an older mtime until the
read recovered, while a live lookup that lags the replay could stop the
retries on its newer mtime alone. The error handler that runs once a gate stops
already decides with sourceMovedOn, which orders by commit timestamp and
compares chunk sets; the gates now use it too, and hasSourceNewerVersion has no
callers left.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.sync: a lagging live lookup cannot prove a chunk dropped

A peer filer's event reaches subscribers before the source store applies it,
so the live lookup can return the version the replay replaces. That entry
never listed the replay's new chunk, so sourceDroppedChunk read a 404 on it as
an obsolete version and cut the missing-chunk wait short. Like sourceVersion,
it now ignores a live entry committed before the replayed version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.sync: test that a replicate failure on a superseded version is skipped

Covers onReplicateChunkError with a transient error for a deleted source entry,
a later commit with an older mtime, and an unchanged source, and points the
unconfirmed-supersession test comment at it instead of the helper test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.sync: test the manifest gate against a missing volume and a lagging lookup

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 10:43:22 -08:00
Chris LuandClaude Opus 5.5 8461ae2fe9 filer.sync: skip a rewritten source version regardless of its mtime order (#12366)
* filer.sync: a chunk the live source entry dropped is gone whatever its mtime says

sourceDroppedChunk refused to judge a 404'd chunk when the live source entry
carried an older mtime than the replayed version, treating that as a rollback.
Timestamp-preserving writers (cp -p, rsync -a, tar extracts) produce exactly
that order on every rewrite, so the vacuumed chunk sat in the retry loop for
the full missingSourceChunkGrace before being written off. Whether the live
entry still lists the chunk is independent of mtime order.

* filer.sync: treat a rewritten source entry as moved on regardless of its mtime

sourceVersion only compared chunk sets when the live entry was at least as
new as the replayed one, so a live entry stamped with an older mtime but
holding different chunks was never recognized as the current version. The
sink then fetched the replayed chunks, hit 404 on each vacuumed one, and the
same mtime guard in onReplicateChunkError kept the event from being skipped.
Whether the source rewrote the file is a chunk-set question, not an mtime
one; the live version is synced in its place as before.

* filer.sync: apply a substituted live snapshot even under a newer destination mtime

After sourceVersion swapped in the live entry, UpdateEntry re-ran
chooseUpdateAction on it and a destination whose mtime was above the live
entry's read as a late update, so the current source content was never
written. The live snapshot is the source's current state, not a replay, so
mtime order says nothing about it; repair from it unless the destination
already holds it.

* filer.sync: a live entry committed before the replay does not supersede it

Dropping the mtime guard let any chunk-set difference read as a rewrite,
including a live lookup that lags the event: a peer filer's event reaches
subscribers before the local store applies it, so the lookup can still
return the version the replay replaces. sourceVersion then synced that
older version over the destination and consumed the event. The meta-log
commit stamps order the two versions where mtime cannot, so a live entry
committed before the replay proves nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.sync: check commit order before the mtime supersession

A lagging live entry can carry a newer mtime than the replay, as when the
replayed version was stamped back to an older time, so sourceSupersedes
took it as newer and the older version was synced in place of the replay.
Rule out a live entry committed before the replay first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 09:36:04 -08:00
Chris LuandClaude Opus 5.5 5c99744e00 rust volume: count sorted-map files and deletes the way Go does (#12365)
* rust volume: count sorted-map files and deletes the way Go does

The Rust sorted needle map still used the counter rule Go dropped when
the index loaders were unified: every .idx row, tombstones included,
counted as a file, and each delete added one file and two deletions.
Go counts put rows as files and a delete as one deletion of the live
row's bytes, so the Go and Rust copies of one volume reported different
FileCount and DeleteCount for the same index.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* rust volume: test canDelete volume counters across restarts

Mirrors the Go store test: a canDelete volume reloaded from .sdx must
count its deletes, and the counters it reports must match what the next
reload computes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 08:13:18 -08:00
5df7faf0e3 fix(storage): update deletion metrics in SortedFileNeedleMap.Delete (#12363) (#12364)
* fix(storage): update deletion metrics in SortedFileNeedleMap.Delete (#12363)

* storage: test canDelete volume counters across restarts

A canDelete volume reloaded from .sdx must count its deletes so
volume.deleteEmpty and vacuum see it drain, and the counters it reports
must match what the next reload computes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 08:05:45 -08:00
Chris LuandClaude Opus 5.5 d13d975b6e shell: keep fs.mergeVolumes and fs.distributeChunks from reverting entries written while they run (#12362)
* shell: keep fs.mergeVolumes from reverting entries written during the merge

The walk reads a page of entries and moves their chunks one at a time, then
wrote each entry back unconditionally. A file overwritten in between got its
old version back, and the filer deleted the new version's chunks as garbage,
so an acknowledged write was lost without any error.

The update now carries IF_ENTRY_EQUAL over the entry as the walk read it. When
the filer refuses because the entry changed or was deleted, the copies made on
the target volumes are deleted, their plan room is released, and the entry is
skipped; the source needles stay where the filer still points.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: keep fs.distributeChunks from reverting a file overwritten while it runs

The command looks the entry up once, moves its chunks, and then wrote that
entry back unconditionally, so an overwrite during the redistribution was
reverted and the filer deleted the new version's chunks as garbage.

The update now carries IF_ENTRY_EQUAL over the looked-up entry. When the filer
refuses because the file changed or was deleted, the moved copies and any new
manifest chunks are deleted and the command reports it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: keep s3.versions.audit -heal from clearing a pointer that just moved

The heal cleared the latest-version pointer on the directory entry the walk
read, unconditionally. A version written between the audit and the heal has
already pointed the directory at a live file, and clearing that pointer hid it.
The heal now carries IF_ENTRY_EQUAL over the audited entry and reports a
refusal as a failed heal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: keep fs.meta.changeVolumeId from reverting entries written meanwhile

The walk rewrote each entry's volume ids and wrote back the copy it had read,
unconditionally, so an overwrite between the read and the write was reverted
and its chunks were deleted as garbage. The update now carries IF_ENTRY_EQUAL
over the entry as read, and a refused entry is skipped and reported.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: keep the target copies another fs.mergeVolumes run committed

moveChunk keeps the key and cookie, so two overlapping merges of the same
source into the same target write the same fid. When one run commits, the
other's update is refused, and deleting its copies deleted the needle the
committed entry now points at.

A refused update now re-reads the entry, resolving its manifests, and deletes
only the copies it no longer references. If the entry cannot be read, the
copies are left for volume.fsck.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: let a later hard-link sibling retry a merge the filer refused

The first sibling of a HardLinkId claims the move for all of them. When its
update is refused, the claim made every later sibling skip, so the shared
chunks stayed on the source volume for the rest of the run. A refused update
now releases the claim, and a sibling listed after the change retries the move
against the entry as it is now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: delete the manifest blobs a failed fold in fs.distributeChunks saved

When re-manifestizing failed partway, the blobs already uploaded were only
counted in a log line and stayed on their volumes with nothing referencing
them. They have freshly assigned fids, so they are now deleted with the
cookie-checked delete the other orphan cleanups use.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: check the heal clears the pointer file name too

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: require the shell lock for fs.mergeVolumes -apply

moveChunk keeps the key and cookie, so two overlapping runs that move the same
chunk into the same volume write the same fid. Re-reading the entry before
deleting a refused run's copies covers the run that already committed, but not
one that commits after that read. With the lock, two applying runs can no
longer overlap, as volume.fsck and volume.merge already require. A dry run
still works without it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: check unreferencedTargets reads the refused entry itself

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: check unreferencedTargets keeps a copy a manifest references

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: check unreferencedTargets keeps the manifest copy and skips an unreadable manifest

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 07:59:18 -08:00
76198d15cf fix(fsck): inspect references to deleted volumes (#12357)
* fix(fsck): inspect references to deleted volumes

* volume.fsck: treat every match-all collection filter as global

ALL_COLLECTIONS and EACH_COLLECTION compile to the same match-all
matcher as an empty filter or "*", but the string check treated them
as a restricted scope: a selected missing volume was rejected and
references to master-absent volumes were reported without being purged.
Use the compiled matcher instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume.fsck: purge an entry whose manifest is on a missing volume

A top-level chunk on a missing volume already proves the entry is
unreadable, whether or not its manifests resolve. Skipping such entries
kept the very case where the manifest needle sat on the retired volume,
and the remaining unresolved manifest then blocked every later orphan
purge. The missing-needle check on live volumes never had this gate, so
both paths now treat these entries the same. The warning now says only
orphan purging is disabled, which is the only thing the counter gates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume.fsck: report a failed Filer delete

A rejected purge DELETE printed nothing unless -v was set, so the run
looked successful after "deleting path" while the entry stayed. The
filer answers 204 for an entry that is already gone, so only real
failures print.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume.fsck: tighten help text, comments and missing-volume tests

The help now wraps like the rest of it and says that a purge removes
the whole entry, including its chunks on other volumes. The live test
header no longer calls the run report-only, and the reverse-scan test
checks the report line for each missing chunk.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume.fsck: drop the missing-volume design notes

The notes said no destructive change was approved while the change adds
a confirmed purge, still described the dropped manifest gate, and
mostly recorded local runs and a reverse index that is not part of
this change. The operator contract lives in help volume.fsck.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume.fsck: allow a report-only missing-volume scan with -collection

The collection check exists to stop a purge that cannot be scoped, but
it also rejected a report-only run that deletes nothing. Apply it only
with -reallyDeleteFilerEntries, and limit the -confirmMissingVolumeDeletion
description to the -volumeId case it guards.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume.fsck: count only a 2xx answer as a purged entry

A redirect the client could not follow left the entry in place but was
recorded as deleted, which also queued its parent for empty-directory
cleanup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume.fsck: say that a filtered purge of a missing -volumeId is refused

The help and the early warning both said a purge with -collection only
reports references to unlisted volumes, but a -volumeId naming one is
rejected before the scan. The help now says so, and the warning prints
after that check, so a refused run no longer claims it will report.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: MZ01-CHOIMINSU <mz01-juwannam@MZ01-CHOIMINSU.local>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 07:57:43 -08:00
Chris LuandClaude Opus 5.5 e8b284fef5 admin: scope plugin job and activity fetches to the selected job type (#12360)
* admin: scope plugin job and activity fetches to the selected job type

The plugin page fetched the newest 500 jobs and 500 activities
cluster-wide and filtered them by job type in the browser. On a busy
cluster that window can hold none of the selected type (433
erasure_coding + 64 ec_balance jobs, 497 ec_balance activities), so
/plugin/lanes/default/execution?job=ec_vacuum showed "No execution
jobs tracked yet" while the server had seven recent ec_vacuum jobs.

When a job-type tab is active, fetch the queue/detection/execution
lists with job_type=<selected> so the server filters before applying
the limit, and refetch when the selected job type changes. The
unscoped lists are still fetched for the status cards and the overview
per-type summary.

* admin: drop a plugin monitor refresh whose job type is no longer selected

A refresh started on one job-type tab could finish after the user moved
to another tab and that tab's refresh had already landed. Its rows, all
of the earlier job type, then replaced the current tab's and the
renderers filtered them down to empty tables until the next refresh.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: show the lane-wide rows when a job-type scoped fetch fails

On a job-type tab a failed scoped jobs or activities request threw before
anything was stored, so the status cards, scheduler state and tables all
stayed stale. Fall back to the unscoped rows, which the renderers already
filter by job type.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: filter plugin jobs and activities by lane before the limit

/api/plugin/jobs and /api/plugin/activities took the newest N rows across
every lane and then dropped the other lanes' rows, so on a busy default
lane a quieter lane's page could come back empty.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: keep a failed plugin monitor fetch's last rows and report it

Falling back to the lane-wide rows when a scoped fetch failed could still
blank the tables, since that window may hold none of the selected type,
and the failure went unreported. A lane-wide or scheduler-state failure
also threw before the scoped rows that did arrive were stored.

Each request now keeps its previous value on failure: the scoped rows
only while the tab is unchanged, otherwise the lane-wide rows. Whatever
arrived is stored and rendered, then the first failure is rethrown so the
caller reports it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: report a monitor refresh failure apart from the action it follows

expireJob and runWorkflow refreshed the monitor inside the action's try
block, so a failed refresh after a successful expire or workflow run was
reported as a failed action, and the expired job's detail was not
reloaded. Catch the refresh separately, as runDetection already does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: show a repeated plugin page refresh failure once

refreshAll runs every 3 seconds and notify opens a modal, so a request
that kept failing reopened the same alert on every tick. Report an error
only when it differs from the last one, and forget it once a refresh
succeeds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 06:32:49 -08:00
Chris LuandClaude Opus 5.5 222cfa8e2e admin: summarize the EC delete_count accounting warning per refresh (#12359)
* admin: summarize the EC delete_count accounting warning per refresh

collectCollectionStats runs on every admin collection, bucket and
topology refresh, and it logged a Warning for each EC volume whose
delete_count summed across shard holders exceeds its file_count. That
sum is an expected accounting artefact (a tombstone can be recorded on
more than one holder), it recurs identically on every pass, and on one
cluster with hundreds of such volumes it grew the admin log to 20 GB,
more than 99% of it this one line.

Log one V(1) summary per pass with the affected volume count and keep
the per-volume detail at V(2). The object count is still skipped for
those volumes.

* admin: test that an EC volume with deletes above file_count is skipped

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 06:31:15 -08:00
Chris LuandClaude Opus 5.5 e5bc658eb7 test/s3vectors: a hardened cluster with TLS volume servers, read JWTs and encrypted chunks (#12356)
* test/s3vectors: a hardened cluster, with TLS volume servers, read JWTs and encrypted chunks

Three switches on the test cluster: VECTOR_TLS=1 puts the volume servers
behind HTTPS requiring client certificates and enables https.client for
every process; VECTOR_READ_JWT=1 gives every process a jwt.signing.read.key,
so volume reads need a token; VECTOR_ENCRYPT_VOLUMES=1 starts the gateway
with -encryptVolumeData. The integration matrix gains a hardened leg per
volume server with all three on.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: the hardened cluster proves its enforcement

Once the gateway is up, the harness checks that the hardening holds, not
only that it is configured: a TLS volume server refuses plain HTTP (Go
with HTTP 400, Rust with a TLS alert) and a connection without the client
certificate, and answers with it; under a read key an unsigned read of a
real chunk, written through the gateway and located through the filer,
gets 401.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: the hardened cluster proves its chunks are encrypted

With VECTOR_ENCRYPT_VOLUMES the probe reads the stored chunk straight
from the volume server, signing the read when a read key is set, and
requires a recorded cipher key and bytes that differ from the plaintext.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: the signed probe read must return the chunk

An empty 200 passed both branches; an unencrypted read now has to return
the bytes written, and an encrypted one has to return something else.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: a read key in the caller's environment does not leak into the cluster

Only VECTOR_READ_JWT turns read JWTs on; an inherited
WEED_JWT_SIGNING_READ_KEY made the volume servers refuse the probe's
unsigned read in an encrypt-only run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: quote certificate paths as TOML strings in security.toml

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: keep non-ASCII certificate paths as UTF-8 in security.toml

TOML has no surrogate-pair escapes, and the Rust volume server reads the
quoted value without decoding escapes at all.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: an encrypted probe chunk must not be a readable gzip stream

The gateway may gzip the body before encrypting it, so a chunk uploaded
compressed but unencrypted would differ from the body and still pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-03 04:56:15 -08:00
Chris LuandClaude Opus 5.5 af45ab8f2a s3api: known-answer fixture generator for SSE-S3 and filer-encrypted chunks (#12355)
* s3api: a known-answer fixture generator for SSE-S3 and filer-encrypted chunks

TestGenerateDirectReadFixture writes, from this package's own encryption,
the pieces a reader outside the gateway must handle: SSE-S3 chunks with
their per-chunk metadata, the object-level key, a single-stream ciphertext,
the KEK derived from s3.sse.key and wrapped under a passphrase in both
on-disk formats, and chunks encrypted by the filer, plain and gzipped. It
is skipped unless DIRECT_READ_FIXTURE names the output file.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3api: the fixture generator restores the key manager and checks the write

The test set the global key manager's KEK and left it; a later test in
the same run then found a signing key where it expects none. The previous
key is restored with t.Cleanup. The fixture file's close error is checked
too, so a delayed write failure fails the test instead of leaving a short
fixture.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3api: the fixture generator unwraps both KEK formats before writing

The legacy v1 wrap is built by hand in the test; reading v1 and v2 back
through unwrapKEK ties both to the format the key manager accepts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3api: pin a legacy v1 wrapped KEK made outside the package

Generating the v1 sample through legacyKEKWrappingSalt let a change to
the salt pass on both sides; a vector wrapped elsewhere under the
historical salt must still unwrap.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-03 04:53:57 -08:00
199b2e2dd9 fix(s3api): reject an invalid storage class before uploading the body (#12354)
* fix(s3api): reject an invalid storage class before uploading the body

* fix(s3api): reject an invalid storage class on CreateMultipartUpload

* fix(s3api): validate storage class and tags before reading the PUT body

* fix(s3api): validate storage class and tags with the other PUT headers

Checking them next to ValidateRequestEncryption keeps the check ahead of
any later hook that may answer the PUT without reaching putToFiler.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(s3api): reuse validatePutHeaders in putToFiler

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(s3api): store storage class and tags on directory-marker PUTs

The marker path validated x-amz-storage-class and x-amz-tagging but its
mkdir callback never wrote them, so a valid value was accepted and then
dropped and GetObjectTagging on the marker came back empty. Store both
the way putToFiler does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(s3api): read directory-marker tags back through GetObjectTagging

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 22:08:21 -08:00
Chris LuandClaude Opus 5.5 c2ca83c0bf filer: say jwt.filer_signing.key gates only the IAM gRPC service (#12353)
* filer: say jwt.filer_signing.key gates only the IAM gRPC service

The key never covered the rest of the filer gRPC port, which can read and
write the IAM store under /etc/iam. Point operators at [grpc.filer] mTLS for a
port reachable beyond a trusted network instead of the key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: note the IAM gRPC OIDC provider and role RPCs are refused without the key

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 18:00:59 -08:00
github-actions[bot] cc71cc9585 docs: regenerate star history chart 2026-11-03 01:08:52 +00:00
Chris LuandClaude Opus 5.5 9b45aae2eb operation: pad the marginal gzip test past Go 1.26's stored block (#12352)
At BestSpeed, Go 1.26's flate writes 4000 random bytes plus a 300-byte
zero run as a stored block, so the gzip output was 28 bytes larger than
the input and the test failed its own precondition on the toolchain
go.mod pins. With 450 zero bytes the saving is 5.7% on Go 1.26 and 8.4%
on Go 1.27, inside the marginal range on both.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 17:01:40 -08:00
dependabot[bot] 794cebe7b0 build(deps): bump cloud.google.com/go/kms from 1.35.0 to 1.36.1 (#12342)
Bumps [cloud.google.com/go/kms](https://github.com/googleapis/google-cloud-go) from 1.35.0 to 1.36.1.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/documentai/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/kms/v1.35.0...dlp/v1.36.1)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/kms
  dependency-version: 1.36.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-02 13:01:04 -08:00
Chris LuandClaude Opus 5.5 8125a1549a s3vectors: support tag conditions in vector bucket policies (#12351)
* s3tables: resolve s3vectors:VectorBucketTag like s3tables:TableBucketTag

Both name a tag on the bucket that holds the target, whichever resource
the request acts on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors: give vector policy conditions the bucket, index and request tags

A vector bucket policy is evaluated with these condition keys:

- aws:ResourceTag/<key>: the tags of the bucket, or of the index when the
  request names one.
- s3vectors:VectorBucketTag/<key>: the bucket's tags, for bucket and
  index requests alike.
- aws:RequestTag/<key> and aws:TagKeys: the tags TagResource and
  CreateIndex set, and the keys UntagResource removes.

An index's tags are read only when the policy has an aws:ResourceTag
condition. The bucket's come from the lookup that already reads its
policy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors: accept tag conditions in vector bucket policies

PutVectorBucketPolicy rejected every Condition, since no condition key
was supplied. A condition is now accepted when its operator is one the
evaluator knows and each key is aws:ResourceTag/<key>,
aws:RequestTag/<key>, aws:TagKeys or s3vectors:VectorBucketTag/<key>.
Any other key would never match, so it is still rejected, as is a
condition field not spelled Condition.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test/s3vectors: cover tag conditions in vector bucket policies

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors: reject condition values the evaluator cannot compare

A null condition value was accepted and then panicked the request that
matched its statement, a Deny reaching even the bucket owner. A value
must now be a string, number or boolean, or a non-empty list of them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 12:43:12 -08:00
dependabot[bot] 97e03f5a9f build(deps): bump actions/setup-go from 5 to 7 (#12347)
Bumps [actions/setup-go](https://github.com/actions/setup-go) from 5 to 7.
- [Release notes](https://github.com/actions/setup-go/releases)
- [Commits](https://github.com/actions/setup-go/compare/v5...v7)

---
updated-dependencies:
- dependency-name: actions/setup-go
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-02 12:35:26 -08:00
dependabot[bot] b91de44615 build(deps): bump github.com/peterh/liner from 1.2.2 to 1.2.3 (#12341)
Bumps [github.com/peterh/liner](https://github.com/peterh/liner) from 1.2.2 to 1.2.3.
- [Commits](https://github.com/peterh/liner/compare/v1.2.2...v1.2.3)

---
updated-dependencies:
- dependency-name: github.com/peterh/liner
  dependency-version: 1.2.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-02 12:01:10 -08:00
dependabot[bot] 20c8876a52 build(deps): bump github.com/aws/aws-sdk-go-v2/service/sts from 1.51.1 to 1.51.6 (#12346)
build(deps): bump github.com/aws/aws-sdk-go-v2/service/sts

Bumps [github.com/aws/aws-sdk-go-v2/service/sts](https://github.com/aws/aws-sdk-go-v2) from 1.51.1 to 1.51.6.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.51.1...service/sts/v1.51.6)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/sts
  dependency-version: 1.51.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-02 12:00:58 -08:00
dependabot[bot] a670553813 build(deps): bump google.golang.org/api from 0.297.0 to 0.310.0 (#12343)
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.297.0 to 0.310.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.297.0...v0.310.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-version: 0.310.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-02 12:00:36 -08:00
dependabot[bot] 9ae31fa425 build(deps): bump actions/setup-python from 5 to 7 (#12345)
Bumps [actions/setup-python](https://github.com/actions/setup-python) from 5 to 7.
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](https://github.com/actions/setup-python/compare/v5...v7)

---
updated-dependencies:
- dependency-name: actions/setup-python
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-02 12:00:20 -08:00
dependabot[bot] e166629f9b build(deps): bump go.etcd.io/etcd/client/pkg/v3 from 3.7.1 to 3.7.3 (#12344)
Bumps [go.etcd.io/etcd/client/pkg/v3](https://github.com/etcd-io/etcd) from 3.7.1 to 3.7.3.
- [Release notes](https://github.com/etcd-io/etcd/releases)
- [Commits](https://github.com/etcd-io/etcd/compare/v3.7.1...v3.7.3)

---
updated-dependencies:
- dependency-name: go.etcd.io/etcd/client/pkg/v3
  dependency-version: 3.7.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-02 11:59:59 -08:00
Chris LuandClaude Opus 5.5 b7b2a5947f admin: configurable file browser upload size limit (#12348)
* admin: carry a file browser upload size limit

The admin server keeps the total size limit of one file browser upload, 500 MB by default as the UI enforced before, and hands it to the file browser page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: reject file browser uploads over the size limit

The 500 MB limit was only checked in the browser, so any client could post larger uploads and fill the temp dir with spooled parts. Cap the request body at the limit plus room for the multipart framing, and answer 413 when the files add up to more than the limit.

Parse the form before reading the path field. FormValue used to parse it first with the default 32 MB memory, which made the later 1 GB ParseMultipartForm a no-op; keep 32 MB explicitly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: show and check the configured upload limit in the file browser

The upload dialog reads the limit from the page instead of a hard-coded 500 MB, lists it under the file picker, and skips the check when uploads are unlimited.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: add -uploadMaxSizeMB to set the file browser upload limit

weed admin -uploadMaxSizeMB and weed mini -admin.uploadMaxSizeMB set the total size limit of one file browser upload; 0 removes it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: note why the upload framing allowance is enough

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: test an upload whose files only exceed the limit together

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: remove spooled upload parts when the upload handler returns

Parts over 32 MB spool to temp files. The router passes the handler a copy of the request, so net/http never sees the parsed form and never removes those files. Every large upload, and every one rejected for size after parsing, left them on disk.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 11:32:03 -08:00
Chris LuandClaude Opus 5.5 a074538c8f admin: manage S3 vector buckets (#12334)
* s3vectors: run an operation for trusted local tooling

Like s3tables.Manager.Execute, Server.Execute runs one S3 Vectors operation
through its handler as a given identity, without a signed request. The admin
console can then manage vector buckets through the same validation,
authorization and catalog steps as the API.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* admin: list, inspect, create and delete vector buckets over the API

/api/s3vectors runs the S3 Vectors operations through Server.Execute as the
admin account: list vector buckets with their encryption, read one bucket
with its policy and each index's data type, dimension, distance metric and
non-filterable keys, create and delete a bucket, and delete an index. Going
through the operations keeps the API's rules, such as ICEBERG buckets not
being vector buckets and the implicit index namespace not blocking a delete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors: export the rule for which table buckets are vector buckets

IsVectorBucketFormat names the check ListVectorBuckets and isVectorBucket
already make, so the admin console can tell the same buckets apart.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* admin: vector bucket pages

Object Store > Vector Buckets lists the vector buckets with their ARN,
encryption and creation time, creates and deletes them, and opens one bucket
with its indexes (data type, dimension, distance metric, non-filterable keys)
and its policy, where an index can be deleted. A LANCE row on the Table
Buckets page links to the same bucket's vector page, since both APIs serve it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* admin: stop the vector bucket and index lists at the list limit

A page cut short by filtered ICEBERG buckets let the next page carry
the list past s3TablesAdminListLimit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: say when a vector bucket or index list is cut at the limit

The lists stop at s3TablesAdminListLimit while more remain, so mark
them truncated and point to an S3 Vectors client for the rest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: indent the vector bucket policy

A policy stored as one line ran off the right edge of the details page.
Policies that are not valid JSON are shown as stored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: show a deeply nested vector bucket policy as stored

Indenting multiplies the size of a deeply nested document, so a policy
past 32 levels of nesting is shown without indenting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 11:25:24 -08:00
Chris LuandClaude Opus 5.5 0176231ea4 volume: replicas keep the primary's compression and metadata (#12350)
* volume(rust): replicate the stored needle instead of the request body

The primary forwarded the original request body and headers, so a
replica re-derived compression from the raw bytes. Send what Go's
ReplicatedWrite sends: a one-part multipart upload of the stored bytes,
gzip-marked when compressed, with the name, mime, pairs and Content-MD5,
and carry ts, ttl, cm and fsync in the query so form-field values reach
the replica too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rust): keep the primary's compression on replica writes

A type=replicate upload now stores the bytes as received, like Go's
ParseUpload, so a replica no longer compresses data the primary stored
uncompressed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rust): replicas keep the primary's name and mime

Forward the primary's filename and mime even past the 255-byte needle
fields, so a replica applies the same limit and sets the same flags,
and skip the filename MIME guess on type=replicate, so an untyped
upload stays untyped on every copy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rust): keep the multipart part Content-Type as sent

Parsing the part type as a mime value lowercased it and dropped values
it could not parse, so a replica stored a different mime than the
primary. Read the header verbatim, as Go's part.Header.Get does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: replicas keep the primary's mime as forwarded

A Go replica guessed the mime from the file extension, so an upload the
primary stored as application/octet-stream, or with no mime at all, got
a different type on the replica. On type=replicate keep the part's
Content-Type as sent, and have the replication sender leave an empty
mime empty instead of filling it in from the extension. This matches
what the Rust replica does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rust): pick the JPEG orientation fix by file name, as Go does

Go checks the mime and the upload's file name, Rust checked the mime and
the URL path, so the same upload could be rotated on one server and not
on the other. With the name and mime forwarded as stored, a replica now
reaches the primary's decision, and a rotated image has no EXIF tag left
to act on, so the fix stays a no-op there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 11:24:51 -08:00
Chris LuandClaude Opus 5.5 ec968aa016 s3vectors: support TagResource, UntagResource and ListTagsForResource (#12335)
* s3vectors: tag vector buckets and indexes

TagResource, UntagResource and ListTagsForResource answer on
/tags/{resourceArn} (POST, DELETE, GET) and by name. Each authorizes its
own s3vectors action on the bucket or index ARN, then reads or writes the
catalog record's tags. A resource holds at most 50 tags.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3vectors: apply the tags passed to CreateVectorBucket and CreateIndex

The tags field was accepted and dropped. It is now stored on the new
bucket or index, and creating with tags needs s3vectors:TagResource in
addition to the create action.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test/s3vectors: cover resource tagging end to end

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: keep tag operations to a bound table bucket

TagResource, UntagResource and ListTagsForResource run through Execute
with WithTableBucketCreatedAt now answer not found when the bucket name
holds another bucket, as the bound creates and deletes already do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3vectors: bind tagging to the authorized bucket and to vector indexes

The catalog tag calls carry the creation time of the bucket authorizeVector
approved, so a bucket created again under the name is not touched. An index
ARN must name a Lance table: another table in the vector namespace of a
format-less bucket is refused, as DeleteIndex does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3vectors: match a TagResource grant scoped to the bucket being created

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: merge tag writes into the tags the record holds at the write

TagResource and UntagResource merged into the tags they read before the
compare-and-swap and wrote that map on every attempt, so a tag write that
landed in between was undone. The merge now runs inside the mutation.
WithMaxTags lets an in-process caller cap the merged set there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3vectors: check the 50-tag limit inside the catalog's tag write

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: check a tag operation's bound bucket after its reads

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3vectors: validate each tag key UntagResource removes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3vectors: validate untag keys one at a time

UntagResource sized a map by the number of submitted tag keys before
validating them, so a large body of repeated keys allocated far more than
the request itself, before authorization ran.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3vectors: authorize UntagResource before validating its keys

Validating a near-32 MiB body of tag keys costs about 13x its JSON
decode, so an unauthorized caller could make the gateway spend that
work on a request it then refuses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 10:33:26 -08:00
Chris LuandClaude Opus 5.5 690aabda0e filer: reuse IsEqualOrUnder for the rename subdirectory check (#12338)
* filer: test the rename subdirectory check through CanRename

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: reuse IsEqualOrUnder for the rename subdirectory check

A trailing slash on the old name made the source path end in a slash,
so a child of the source was no longer seen as under it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 10:28:48 -08:00
d650abe0e7 compression: compress JSON uploads, and match file types by extension (#12339)
* compression: compress JSON uploads, and match file types by extension

IsCompressableFileType did not recognise JSON media types, so S3 uploads of application/json (which carry no file name) were never compressed. Its callers also passed filepath.Base of the file name where it expects the extension, so the .json/.js/.html/... extension rules never matched a real file name. Both callers now pass filepath.Ext; the function lower-cases the extension and media type, ignores media-type parameters ("application/json; charset=utf-8"), and treats */json and *+json types as compressible.

* compression: treat application/json-seq as JSON

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rust): compress JSON uploads and match media types without parameters

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* operation: keep uploader gzip only when it saves at least 10%

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Pat White <5015863+patwhite@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 10:15:01 -08:00
Jiri PucandClaude Opus 5.5 1637941368 filer: allow renaming into a sibling that shares the name prefix (#12337)
CanRename rejected a move whenever the new parent directory's path started
with the source path as a plain string, so moving /dir/a into /dir/a.d
(or anything under /dir/ab) failed with "can not move directory to a
subdirectory of itself". Check the path boundary instead: reject only
when the new parent is the source itself or lies under it.


Claude-Session: https://claude.ai/code/session_01VncHKTehVroSAzRMUHykEX

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 10:09:22 -08:00
Chris LuandClaude Opus 5.5 7298b4ed37 fix(s3tables,s3vectors): bind DeleteNamespace and the vector bucket cleanup to the authorized records (#12331)
* s3tables: bind DeleteNamespace to the authorized namespace

The delete claimed whatever finished namespace held the name at claim
time, so one deleted and created again by another owner after the
authorization read was removed under the first one's authorization.
The claim now requires the creation time that was authorized.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3vectors: bind DeleteVectorBucket's namespace cleanup to the authorized bucket

After the bound DeleteTableBucket reported the bucket not empty, the
index namespace cleanup ran unbound, so a bucket deleted and created
again in between lost its empty index namespace under the first
bucket's authorization. DeleteNamespace now honors the bucket binding,
checking the bucket after reading the namespace, and the cleanup
carries it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 09:55:09 -08:00
github-actions[bot] 984eac6a79 4.52 2026-11-02 11:24:06 +00:00
Chris LuandClaude Opus 5.5 aa2d94cc50 ci: cancel a PR's runs when it is merged or closed (#12330)
A merged or closed PR kept every pull_request run going: concurrency groups
cancel a run only when a newer push arrives. After #12329 merged, 16 of its
runs were still queued or running. On pull_request_target closed, cancel the
queued and running pull_request runs on the PR's head branch whose commit
belongs to the PR.


Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 03:21:26 -08:00
Chris LuandClaude Opus 5.5 66a6598fdd fix(s3tables,s3vectors): bind table and vector bucket writes to the bucket they were authorized against (#12329)
* s3tables: bind table bucket policy writes to the authorized bucket

PutTableBucketPolicy compared only the policy bytes, and
DeleteTableBucketPolicy removed the policy unconditionally. A bucket deleted
and created again under the same name between authorization and the write
took a policy change nobody authorized for it. Both writes now require the
bucket's CreatedAt and the policy they were authorized against, answer 409
otherwise and 404 for a bucket that is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: bind bucket tag writes to the authorized bucket

TagResource and UntagResource checked the creation time only for a table
ARN. On a bucket ARN the merged tags landed on a bucket deleted and created
again since authorization. extractResourceOwnerAndBucket now returns the
resource's CreatedAt for both kinds, and the write requires it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: bind bucket maintenance writes to the authorized bucket

PutTableBucketMaintenanceConfiguration wrote onto whatever bucket held the
name at write time. authorizeMaintenanceBucket now returns the bucket's
CreatedAt like authorizeMaintenanceTable, and putMaintenanceConfiguration
takes the check and creation time from both callers instead of an optional
closure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3vectors: bind vector bucket policy writes to the authorized bucket

Put/DeleteVectorBucketPolicy authorized inside the compare-and-swap with
the owner from an earlier bucket read, so a bucket deleted and created
again by another account in between had its policy rewritten by the old
owner. The mutation now requires the entry's createdAt to match the
record it authorized against.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3vectors: bind DeleteVectorBucket to the authorized bucket

DeleteVectorBucket authorized one bucket record, then ran the trusted
catalog DeleteTableBucket on whatever bucket held the name by then, so an
empty bucket deleted and created again by another account in between was
deleted by the old owner. The catalog delete now accepts the authorized
bucket's createdAt through an in-process context value and treats any
other bucket as gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3vectors: bind CreateIndex and DeleteIndex to the authorized bucket

Both authorized the bucket, then ran trusted CreateNamespace, CreateTable
or DeleteTable on whatever bucket held the name by then, so an index could
be created in, or deleted from, a bucket another account created again in
between. Those catalog operations now honor the bucket binding
DeleteTableBucket already takes, and the vector handlers pass the
authorized bucket's createdAt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 03:00:30 -08:00
4850def831 fix(s3tables): bind table and view writes to the record they were authorized against (#12328)
* s3tables: bind table and view writes to the record they were authorized against

* s3tables: bind UpdateNamespace to the namespace it was authorized against

UpdateNamespace writes back the whole record it read, owner and creation
time included. A namespace deleted and created again before the write took
the first one's owner and CreatedAt, which also undoes the CreatedAt binding
creates now rely on. Refuse the write with 409 when the namespace at the
path has another CreatedAt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 02:50:45 -08:00
Chris LuandClaude Opus 5.5 ce292a4e5e s3: require Admin to put or delete a bucket's lifecycle configuration (#12327)
PUT and DELETE ?lifecycle were authorized as Write, so an identity granted
only object writes on a bucket could add expiration rules or remove the ones
the bucket administrator set. They now need Admin on the bucket, like the
encryption, public access block and ownership controls writes. Identities
authorized by IAM policies are unaffected: they are still checked against
s3:PutLifecycleConfiguration.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 01:26:16 -08:00
Chris LuandClaude Opus 5.5 d431c1b35a s3api: let the vectors route, not the credential scope, raise the body limit (#12326)
SigV4 verification picked the 32 MiB S3 Vectors body limit from the
credential scope, which is not verified until the body is hashed. A request
claiming the s3vectors scope on any other route could make the gateway buffer
32 MiB before authentication instead of 10 MiB. The vectors handler now sets
the limit on the request context, and every other route keeps 10 MiB.


Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 01:15:42 -08:00
Chris LuandClaude Opus 5.5 02d22800aa s3vectors: accept bodies up to the 32 MiB limit, report larger ones as a size error, log engine failures (#12320)
* s3api: hash an s3vectors body up to the Vectors API limit

SigV4 verification of a non-S3 request hashed at most 10 MiB of the body
and silently dropped the rest, so any S3 Vectors request over 10 MiB failed
its signature check with 403, although the API accepts 32 MiB. The limit
now follows the signing service, and a body over it is reported as
ErrEntityTooLarge instead of being truncated; the embedded IAM answers that
with InvalidInput, as it already does for a body over its own limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors: answer an oversized request with a ValidationException

A body over 32 MiB was refused as "403 AccessDenied: authentication
failed". A declared Content-Length over the limit is now refused before
authentication, and a streamed body over it, whether the signature check
or the handler reads it, gets the same "request body exceeds 32 MiB"
ValidationException. The limit is the one SigV4 verification uses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors: log engine and catalog failures at the default level

An engine error reached the client as a bare "internal error", and the
cause was logged only with -v=1, so a misconfigured engine left nothing in
a default log. Engine and catalog failures are now logged as errors, and a
reported unavailability as a warning, with the operation, bucket and
index. The response still leaves the engine's message out, since it can
name internal hosts and paths, and points at the gateway log instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3api: hash the signed body through readRequestBody

streamHashRequestBody repeated readRequestBody's read-past-the-limit
check; it now reads the body with readRequestBody and hashes the bytes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 00:18:44 -08:00
Chris LuandClaude Opus 5.5 2eb219df61 s3vectors: encrypt the objects of a vector bucket declared AES256 (#12324)
* s3api: encrypt the objects of a vector bucket declared AES256

CreateVectorBucket recorded encryptionConfiguration sseType AES256 and
returned it, but nothing on the write path read it: the index's data
files, manifests and transactions were stored without SSE.

A LANCE table bucket's record now supplies the bucket's default
encryption whenever the bucket has none of its own, so PutObject,
CreateMultipartUpload and CopyObject apply SSE-S3 to every object the
engine writes. The S3 bucket metadata itself is unchanged.

ICEBERG table buckets keep recording the declaration only: their
catalog reads the manifests clients upload straight from the filer,
which would see ciphertext.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3tables: refuse an AES256 vector bucket on a gateway without an SSE-S3 key

A vector bucket's AES256 is now applied to every index write, and SSE-S3
fails each write when the gateway has no KEK. Creating the bucket there
succeeded and the first CreateIndex came back as InternalFailure.

CreateTableBucket now refuses a LANCE bucket declaring AES256 when the
gateway reports no SSE-S3 key, naming s3.sse.kek and s3.sse.key;
CreateVectorBucket returns it as ValidationException. ICEBERG buckets,
whose declaration is only recorded, and tooling that sets no checker
are unaffected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3api: refuse DeleteBucketEncryption on a bucket's declared encryption

GetBucketEncryption reports a vector bucket's declared AES256, but
DeleteBucketEncryption answered NoSuchBucketEncryptionConfiguration and
the declaration kept applying. The declaration is fixed when the bucket
is created, so the delete now returns MethodNotAllowed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test(s3vectors): keep an inherited WEED_S3_SSE_KEK out of the test cluster

The gateway refuses to start SSE-S3 with both s3.sse.kek and s3.sse.key
set, so a developer shell exporting WEED_S3_SSE_KEK made the AES256
bucket test fail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 23:57:40 -08:00
Chris LuandClaude Opus 5.5 d72f761d83 seaweed-vector: work on its defaults, and without identities or credentials (#12323)
* seaweed-vector: default --namespace to the gateway's Lance port

The old default, http://localhost:8888/lance, is a path nothing serves:
the Lance Namespace listens on its own port, 9101 by default on weed s3,
weed server, weed filer and weed mini. With it, the filer answered the
engine's catalog calls with a 500 and every CreateIndex failed with
InternalFailure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: default --listen to a port no SeaweedFS server takes

127.0.0.1:8080 is the volume server's default port, so weed-vector next
to weed server exited with "Address already in use (os error 98)". 8090
is not among the defaults of master, volume, filer, S3 and its Iceberg
and Lance catalogs, IAM, WebDAV, SFTP, the message queue broker and
agent, the admin server or weed mini, nor any of their gRPC ports
(HTTP + 10000).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* lance: take any key while the S3 server runs without identities

Without identities the S3 port accepts any signature and treats the
caller as the admin. The Lance catalog still checked an x-api-key or
Bearer token against identities that did not exist, so weed-vector
started with --storage-option credentials on such a server got 401 on
every catalog call. The catalog now skips both checks when S3 auth is
off and leaves the caller to the S3 authenticator, which admits it as
the admin, as it already did for a request with no key. With
identities configured nothing changes: an unknown key is still 401.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* lance: tell clients to skip signing while S3 auth is off

A client with no credentials of its own could not reach a table's data
on a server without identities, though the S3 port takes unsigned
requests there: lance fell back to the AWS default credential chain and
failed with CredentialsNotLoaded. weed-vector without --storage-option
failed every CreateIndex that way. The catalog now adds
aws_skip_signature to the storage options it hands out while S3 auth is
off, and leaves it out once identities exist.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-vector: bind the listener before registering with the master

The engine spawned its master registration and then bound --listen, so
an engine whose port was taken could reach the master before it exited:
on weed server with 18080 taken the master logged "vector node
127.0.0.1:18080 added" and removed it in the same millisecond, and the
gateway noticed it. Bind first, and name the address in the error
instead of "transport error".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* seaweed-worker: keep an unsigned store off the AWS default chain

A store the catalog says takes unsigned requests needs no credentials,
but lance builds the AWS default credential chain for any store without
them, and the chain probes instance metadata on every dataset open.
With no credentials of its own, weed-vector spent a second on each
query (1030-1066 ms against 32-46 ms with --storage-option ones). The
merge of the operator's fallback and the vended options, which the
engine and the Lance worker each did in their own copy, moves into
table_options, and gives such a store empty credentials.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test(s3vectors): serve an index with no identities and no engine credentials

A second cluster runs weed s3 without -config and weed-vector without
--storage-option, with no AWS credentials in either environment: weed s3
would turn them into an admin identity, and the engine would find them.
On master its CreateIndex fails with InternalFailure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 23:52:30 -08:00
Chris LuandClaude Opus 5.5 ff7045a4e5 s3vectors: report the Euclidean distance, not its square (#12318)
A euclidean index answered QueryVectors with Lance's L2 distance, which is
the squared Euclidean distance: [3, 4] was 25 from the origin, not 5. AWS
defines the metric as the straight-line distance between the vectors. The
gateway now returns its square root, so the single engine and the enterprise
coordinator, which both hand back the raw Lance value, report the same
distance. Ordering is unchanged.


Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 23:50:46 -08:00
Chris LuandClaude Opus 5.5 d6c0d728e4 s3vectors: resolve the caller like table buckets and honor bucket-scoped identity actions (#12319)
* s3vectors: resolve the caller the way table buckets do

The vector checks used the identity name as principal, while the catalog
records a bucket's owner through getAccountID: the account id, or the
issuer-scoped user id of a federated identity. An identity with an account
or a federated identity did not own the vector bucket it created, a Deny
naming its account did not bind it, and an Admin identity in the admin
account did not get the admin bypass it has on table buckets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors: honor identity actions scoped to a vector bucket

An identity action such as s3vectors:QueryVectors:team-a granted nothing,
because the vector policy context never named the bucket. Table buckets
match <action>:<bucket> through the same context field, so pass the bucket
the check is about. Creating a bucket stays unscoped, as for table buckets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 23:50:19 -08:00
Chris LuandClaude Opus 5.5 54777d7fb9 vector: build weed-vector without a system protoc (#12317)
* vector: build without a system protoc

lance's build scripts run $PROTOC and fail without one. Point it at the
protoc that protoc-bin-vendored ships, at the version in Cargo.lock, so
building inside seaweed-vector/ needs only the Rust toolchain. An explicit
PROTOC still wins, and a registry without the crate falls back to PATH.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test(s3vectors): build weed-vector inside its workspace

Cargo reads seaweed-vector/.cargo/config.toml only when it runs there, so
build from that directory to pick up the vendored protoc.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* ci(vector): drop the PROTOC step the workspace config replaces

Builds run inside seaweed-vector/, where the cargo config already points
lance at the vendored protoc.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test(s3vectors): resolve CARGO_TARGET_DIR before changing directory

A relative CARGO_TARGET_DIR given to make would point cargo, which now runs
in seaweed-vector/, and the cp after it at different directories.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: pick the vendored protoc on every Linux host it ships for

protoc-bin-vendored also carries ppc64le, s390x and x86_32 binaries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 18:56:09 -08:00
Chris LuandClaude Opus 5.5 80e0bf97dd shell: dial volume servers on their registered gRPC port in volume.scrub, ec.scrub, volume.server.state (#12315)
* shell: dial scrubbed volume servers on their registered gRPC port

volume.scrub, ec.scrub and volume.server.state built each data node's
address from its HTTP address alone, so a volume server whose gRPC port
is not the HTTP port + 10000 was dialed on the wrong port.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: resolve -node volume servers to their registered gRPC port

An explicit -node/-nodes list was dialed on HTTP port + 10000 too.
Match each entry against the topology and take the data node's gRPC
port; an address the master does not know is used as given.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: keep a gRPC port given in -node

A node given as host:port.grpcPort was rewritten to the registered gRPC
port; only a bare host:port takes it from the topology.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: dial -node volume servers when the topology is unavailable

An explicit node list did not need the master before; keep using it as
given if the volume list cannot be fetched. Selecting every node still
requires the topology.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: bound the topology lookup for an explicit -node list

With the master down, volumeServer.state -nodes waited on the master
forever instead of reaching the given volume servers. Give the lookup a
30s deadline and fall back to the addresses as given.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: resolve volume server addresses from a topology-only listing

Only the data nodes are needed to find each server's gRPC port, so ask
the master for the topology without its volumes and ec shards. A large
cluster no longer risks running past the -node lookup deadline and
falling back to the HTTP port + 10000.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* shell: skip the topology lookup when every -node names its gRPC port

A host:port.grpcPort entry is dialed as given, so a list made only of
those does not need the master and no longer waits on it during an
outage.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 18:50:48 -08:00
Chris LuandClaude Opus 5.5 b9bc42c217 s3vectors: vend write credentials to index data only to the vector engine and Lance worker (#12316)
* security: share the filer admin token check

CheckGrpcAdminPeer verifies an admin Bearer token inline. Move the check
into IsFilerAdminToken so an HTTP endpoint can accept the same token.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance: recognize the cluster's own components by a signed token

A request whose X-Seaweedfs-Cluster-Token header carries an admin token
signed with jwt.filer_signing.key comes from a component that holds the
cluster key: the vector engine or the Lance maintenance worker. Mark such
a request as trusted. It still authenticates as an identity, which stays
the principal for vended credentials. A dedicated header leaves the
Authorization header to Lance OAuth clients. A gateway with no key
trusts no one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance: vend write credentials to a vector bucket's indexes only to trusted callers

An index in a vector bucket's vector namespace is also a Lance table, so
a caller granted s3tables:UpdateTable could describe it, or declare it
again, and get credentials that write the index data without passing the
vector bucket policy. In that namespace both now vend read-only
credentials unless the caller is trusted, and a declare from anyone else
no longer writes the dataset markers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance worker: prove the cluster key to the Lance namespace

The gateway now vends write credentials to a vector bucket's indexes only
to callers that send a token signed with jwt.filer_signing.key. The
worker already takes that key for the admin stream; its namespace client
now mints a token from it for every catalog request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* weed-vector: describe the index the gateway created and prove the cluster key

CreateIndex declared a table the gateway had just created, which only
worked because declare is idempotent. Describe it instead. The Lance
namespace vends write credentials to a vector bucket's indexes only to
callers that send a token signed with jwt.filer_signing.key, so take the
key with --filer-signing-key, the same flag the Lance worker has.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance worker: keep the cluster key out of Debug output

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance: answer a declare of an existing table with its own location

CreateTable succeeds for a name that already exists without looking at
the location requested, and declare kept using that location for the
dataset markers and the vended credentials. Declaring an existing table
with another table's location, such as a vector index, got credentials
to that table's data. Use the location the catalog holds, which
CreateTable now also reports for a new table.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance: sign cluster tokens with a key derived for the catalog

A cluster token was a filer admin token signed with jwt.filer_signing.key
and, by default, never expired. Sent to the Lance namespace on every
request, often over plain HTTP, a captured one would also authorize the
filer's IAM service. Sign it with a key derived from the cluster key for
this one use, which the admin services do not accept, and mint it with a
five minute lifetime. weed-vector drops its token lifetime flag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance: follow a reloaded jwt.filer_signing.key

The Lance namespace copied the cluster key at startup. A SIGHUP reload
refreshes the gateway's filer guard but not that copy, so after a key
rotation the restarted vector engine and worker were no longer trusted.
Read the key from the guard on every check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* iceberg: keep a vector bucket's indexes read-only

An index in a vector bucket is also a catalog table the Iceberg REST
catalog loads, and LoadTable vended credentials that could write it to
any caller granted s3tables:UpdateTable. A commit staged its metadata
file in the index directory on the same grant. Apply the vector
namespace rule to both, after the caller's own UpdateTable check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* iceberg: confine a created table's location to its namespace before writing

Stage-create skips the registration that confines a location to the
table's namespace, yet vends write credentials for that location. A
caller allowed to create a table in any namespace of a bucket could name
another namespace's prefix, a vector index included, and get write
credentials to it. The non-staged create also wrote its first metadata
file there before the registration refused the location.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance: a declare that finds an existing table answers like describe

Creating a name that already exists is idempotent, so a caller allowed
only to create tables could declare someone else's table and get write
credentials to it, and clear its deregistered marker. Describe would have
vended that caller read-only credentials. A declare now looks the name up
first; when it only finds the table, writing needs UpdateTable, as it does
for describe.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 18:49:01 -08:00
Chris LuandClaude Opus 5.5 06b9d01423 s3vectors: match a filter value against the elements of an array (#12321)
* vector: compare json_array_contains elements as typed JSON values

Lance's json_array_contains compares each element's text with the value, or
with the value wrapped in quotes, so 1 matches "1", true matches "true", and
1.0 misses a stored 1. The filter swap now replaces it like json_extract and
json_exists: the value is parsed as JSON once per batch and compared with
jsonb's typed equality, and only literal arguments are swapped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors: match a filter value against the elements of an array

A filter never looked inside an array: {"tags": "space"} and
{"tags": {"$in": ["space"]}} both missed "tags": ["space", "classic"].
AWS documents that $eq, and so a bare value, matches an array metadata value
when any element equals the input. $eq, a bare value and each $in member now
also match through json_array_contains; $ne and $nin remain their negations.
Range operators, $exists and nested objects are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 18:44:49 -08:00
Chris LuandClaude Opus 5.5 533f5060b9 filer: scope a prefix JWT to the directory a zip extract writes under (#12305)
* filer: scope a prefix JWT to the directory a zip extract writes under

Without a trailing slash, an extract request writes members beside the
named path, in its parent directory, while the JWT check only saw the
request path. A token scoped to /tenant1 could therefore extract into /
by posting to /tenant1?extract=zip. Add the extract root to the paths a
prefix-restricted token must cover.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: do not keep an extracted archive under the name ".."

A multipart file name of ".." passes path.Base unchanged, so
extract.keep stored the archive as an entry literally named ".." in the
target directory, a name that local sinks resolve to the parent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* leveldb3: keep a non-local bucket name out of the per-bucket folder

findDB splits the bucket out of a /buckets/ path and uses it as a folder
name under the store directory. The name cannot hold a slash and dot
names already go to the default DB, so this only rejects an empty or
non-local name, but it also lets static analysis see the folder cannot
escape the store directory.

* filer: clean the zip extract root before scoping and writing

A percent-encoded dot segment survives the path cleaning in front of the
mux, so a request path ending in "/" reached extractZipRoot with ".."
in it. The JWT scope check cleans the path it compares, but the members
were written under the raw root, so the two could disagree.

* leveldb3: apply the bucket folder check to bucket create and delete events

findDB already keeps dot and non-local bucket names out of the per-bucket
folders, but OnBucketCreation and OnBucketDeletion used the name as is. A
directory event for /buckets/.. opened a DB in the parent of the store
directory, and its delete removed that parent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* leveldb3: keep a bucket delete event off the default DB

A bucket named _main maps to the default DB in findDB, so deleting
/buckets/_main closed the default DB and removed its folder, taking every
non-bucket entry with it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* leveldb3: keep bucket names with a path separator out of the store dir

A bucket event named x/../_main passed the local-name check, and its
delete removed the default DB folder once store.dir/x existed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* leveldb3: only reject the platform separator in a bucket name

A backslash is an ordinary file name character on Unix, and a bucket
folder named that way already exists for anyone who created one. Rejecting
it on every platform would route those entries to the default DB after an
upgrade. The slash can never appear in a bucket name here, and on Windows
the backslash is the separator, so rejecting os.PathSeparator alongside
the slash keeps the folder flat without changing Unix routing.

* leveldb3: check for an escaped bucket db before its delete event

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 18:36:31 -08:00
Chris LuandClaude Opus 5.5 d84ae4f2ab volume: don't let an unreachable tier backend hold up volume server startup (#12312)
* s3 tier backend: bound connect and response-header waits

The tier client used the SDK default transport: a 30s dial timeout and no
limit on waiting for response headers. A blackholed endpoint cost about
two minutes per request with the SDK's retries, and an endpoint that
accepts connections but never answers held the request forever.

Dial with a 10s timeout and give up on a response whose headers do not
arrive within a minute.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: stop the disk scan from waiting on a stalled tier backend

A volume server registers with the master only after every volume on its
disks has loaded, and loading a tiered volume reads its superblock from the
remote object. With the S3 endpoint unreachable, each read waited out the
client's timeouts and retries on one of the loader workers, so start-up grew
with the number of tiered volumes and the local volumes were not served
until it finished.

A tiered volume already loads without its superblock when the read fails.
Give that read 10s, and once it stalls, let the backend's other volumes skip
it for a minute instead of each waiting out the timeout. The volume reads
from the backend as before once the endpoint answers again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: bound connect and read waits of the S3 tier client

Mirror the Go tier client: a 10s connect timeout instead of the SDK's
3.1s default, and a minute to receive a response, so an endpoint that
accepts connections but never answers cannot hold a read forever.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: stop volume loading from waiting on a stalled tier backend

Mirror the Go loader: the superblock read of a tiered volume gets 10s, and
once it stalls the backend's other volumes skip it for a minute. The
volume loads with its .vif version as before when the read fails, and
reads from the backend once the endpoint answers again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: keep the .vif replication when a remote superblock read fails

Go applies the .vif replication to a tiered volume whether or not its
superblock read succeeded, so a volume loaded while its backend is down
still reports its replica placement. The Rust loader applied it only on
success and reported the default placement instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: skip the .idx head repair while the tier backend is unreachable

The repair reads needle headers from the remote .dat, so after the
superblock read failed it would wait on the same backend again and hold
up the load. It runs on the next load that reaches the backend.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: skip the .idx head repair while the tier backend is unreachable

Mirrors the Go load: after the remote superblock read failed, the
repair's remote header reads would wait on the same backend again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 tier backend: give up a ranged read whose body stops arriving

The response-header timeout ends once headers arrive, so an endpoint
that answers and then sends no body held ReadAt, and a superblock
read's worker, forever. Cancel the request once the body makes no
progress for s3ReadTimeout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* seaweed-volume: test that a tier read whose body stalls gives up

The SDK's stalled-stream protection already fails it, matching the Go
read's no-progress cancel.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rust): keep the S3 read timeout off upload_part

The SDK's read timeout runs from the start of the request, so a 64 MiB
part on a slow link would time out while its body is still being sent.
Go's ResponseHeaderTimeout only starts after the body is written.
upload_part now keeps the connect timeout and drops the read timeout;
stalled-stream protection still fails an upload that stops moving.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rust): bound the upload_part response wait from when the part is sent

Without a read timeout, an upload_part whose body was accepted but never
answered waited forever. The part body now goes to the connection in 64 KiB
chunks, and once the last chunk is taken the response must arrive within
S3_READ_TIMEOUT, like Go's ResponseHeaderTimeout. A retry rebuilds the body
and restarts that clock. The body keeps its in-memory contents, so signing
and the checksum header are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume(rust): keep a retried part's response clock from an earlier attempt

The send state was a single flag, and a retry body could be rebuilt and
fully sent before the waiter saw the flag drop, so the waiter kept the
earlier attempt's deadline. Each body now carries its attempt number, only
the latest attempt marks the part sent, and the deadline holds only while
no later attempt has started.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 18:35:37 -08:00
Chris LuandClaude Opus 5.5 965c6c06fb Ship weed-vector in release assets and Docker images (#12322)
* ci(release): publish weed-vector binaries

Tagged releases now carry weed-vector_linux_{amd64,arm64}.tar.gz, static
musl builds made the same way as weed-worker's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* docker: carry weed-vector in the image

The image takes a pre-built weed-vector the way it takes weed-worker, or an
empty placeholder where none was built. `vector` runs it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* ci(docker): build weed-vector for the images

The container workflows cross-compile weed-vector for amd64 and arm64
beside weed-worker and place it in the Docker context.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* install.sh: add the vector component

`--component vector` installs weed-vector from a release, and `all` adds it
when the release carries one, as it does for the worker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* ci(docker): skip weed-vector for source refs that predate it

The manual latest-image rebuild accepts any source_ref; one without the
engine's server crate now builds its image without weed-vector instead of
failing the Rust job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* docker: have the vector engine listen beyond loopback

weed-vector listens on 127.0.0.1 by default, which a gateway in another
container cannot reach. `vector` now adds --listen 0.0.0.0:8090 unless a
listen address is given by flag or WEED_VECTOR_LISTEN. The engine registers
its container address for a wildcard listen.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 18:34:04 -08:00
Chris LuandClaude Opus 5.5 2ea63bad48 filer.backup: keep local sink writes inside the sink directory (#12310)
* localsink: skip keys that resolve outside the sink directory

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: reject entry paths with dot segments

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: test renames onto dot-segment targets

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: keep the dot-segment rename test inside one bucket scope

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* localsink: resolve keys through an os.Root on the sink directory

The lexical check alone lets a symlinked directory that already exists
under the sink directory carry writes and deletes outside it. Create,
rename and delete now go through an os.Root opened on the sink
directory, so a symlink can only resolve inside it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* localsink: follow absolute symlinks that resolve inside the sink directory

os.Root rejects every absolute symlink target, so a backup through an
absolute link that points back into the sink failed, and a delete through
it left the file behind. Resolve the parent of the key first and use the
resolved path when it stays under the sink directory. Operations still go
through os.Root, so a link swapped after the check cannot escape.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 18:33:07 -08:00
github-actions[bot] 88b76b5890 docs: regenerate star history chart 2026-11-02 00:59:31 +00:00
Chris LuandClaude Opus 5.5 5cf053c32e volume: read a manifest chunk from another replica when one fails (#12313)
* operation: read a manifest chunk from the next replica when one fails

ChunkedFileReader read each chunk from one random location and returned
the first error, so a chunk whose cached replica went down failed the
whole file although another replica had it. Try the chunk's locations in
turn (local-data replicas first) and resume each attempt at the byte the
previous one reached. A failed write to the caller still ends the read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* operation: drop a chunk volume's cached locations after a failed read

A chunk read that failed on a replica left that replica in the
10-minute location cache, so later reads kept trying it first. Drop the
volume's cached locations after a failure, and when every cached
replica has failed, ask the master once more and try the locations it
now lists.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* operation: don't continue a partly sent chunk on another replica

A volume server streams a large needle before it has verified the
checksum and aborts the response on a mismatch, so bytes that already
reached the reader may be corrupt. Continuing such a chunk on another
replica would hand back a mix of bad and good bytes as a complete read.
Move to the next replica only while nothing of the chunk has been
written; a mid-body failure still fails the read, and the dropped cache
entry lets the next read pick a live replica.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* operation: treat a short chunk body as a failed replica read

A replica that ends a chunk's body cleanly before the chunk's size made
the read look successful, so the next replica was never tried and the
following chunks landed at the wrong offsets. Read exactly the chunk's
remaining length and report a short body as an error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 16:31:22 -08:00
ab204a03e6 fix(s3): verify SigV4 signature with decoded path behind X-Forwarded-Prefix (#12299)
* fix(s3): verify SigV4 signature with decoded path behind X-Forwarded-Prefix

When SeaweedFS is hosted behind a reverse proxy that strips a URL prefix (X-Forwarded-Prefix) and forwards the percent-encoded request path, signature verification fails for object keys containing non-ASCII characters. The client signs the canonical URI as the single percent-encoded path including the prefix, but the server verifies against the already-encoded forwarded path joined to the prefix, producing a mismatch (SignatureDoesNotMatch).

Decode the forwarded path and rejoin the prefix before verifying, mirroring the existing un-prefixed decoded retry. ASCII keys are unaffected because the decoded path equals the original, so the retry is skipped.

Signed-off-by: Mohd Quamar Tyagi <104281681+Tyagiquamar@users.noreply.github.com>

* s3: build the SigV4 candidate paths once and try them per host

The forwarded-prefix and plain branches each repeated the decode-and-verify
step for every host candidate. List the escaped and decoded paths once, the
prefixed forms first, and verify them in the same order as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Signed-off-by: Mohd Quamar Tyagi <104281681+Tyagiquamar@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 16:27:45 -08:00
d6ec6bc4e7 fix(s3tables): let DeleteTableBucket remove empty leftover directories (#12307)
* s3tables: let DeleteTableBucket remove empty leftover directories

Page through the bucket instead of reading one page, and check each bare
directory a deleted namespace left behind for files: empty ones go with the
bucket, one holding files refuses with 409 naming it.

The delete claims the bucket before scanning, and CreateNamespace refuses a
claimed bucket and confirms it again after its write, so a namespace created
meanwhile is seen by the scan or turned away instead of deleted with the bucket.
The bucket is removed only while it still carries that claim, otherwise 409.
A delete that refuses lifts the claim, retrying a failed release, since a claim
left behind refuses every create in the bucket.
The new namespace stays marked unfinished until that check passes, so no table
or view is created in one that gets backed out; a create takes over a namespace
still carrying the mark. Both checks accept only the bucket the create was
authorized against, by its creation time: one recreated under the same name
answers 404 and gets nothing written into it.

The removal is not recursive. Each leftover is claimed first, only while it is
still the directory the scan saw, and goes only while it holds no file or
catalog record; the bucket goes only while nothing is left in it. A file
uploaded or a namespace taken back after the scan is refused with 409 instead
of deleted.

* s3tables: keep directory-backed S3 objects in DeleteTableBucket

* s3tables: keep a file that replaced a leftover directory in DeleteTableBucket

* s3tables: claim only the authorized bucket in DeleteTableBucket

* s3tables: assert the prefix-object mark when claiming a leftover directory

* s3tables: hide unfinished namespaces from readers and DeleteNamespace

* s3tables: stop CreateNamespace on any failed bucket check

* s3tables: retry lifting claims on leftover directories

* s3vectors: declare the index namespace again after losing it to a concurrent declare

CreateNamespace now writes a namespace marked unfinished and confirms it
after checking the bucket. A second CreateIndex that declares the namespace
in that window takes the marked record over, and the first one's
confirmation answers 409, which failed its CreateIndex. Declare it again,
a bounded number of times, when the create answers a conflict.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: name an unfinished namespace that keeps a bucket from being deleted

A CreateNamespace that dies between its write and its confirmation leaves
a namespace marked unfinished. Listings and GetNamespace hide it and
DeleteNamespace answers that it does not exist, while DeleteTableBucket
counts it, so the bucket could not be deleted and nothing said why. Name
it in the BucketNotEmpty answer; creating it again takes it over.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: read a refused empty-directory delete with filer.DeleteEntryError

Use the filer's own reading of a non-recursive delete's answer instead of
a copy of its message prefix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3vectors: release the held create when the concurrent-declare test fails

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3vectors: buffer the held create's result so it can exit after a failure

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: retry lifting a refused namespace delete's claim

A claim left on a namespace refuses every create in it, the same harm the
bucket's claim does, so its release retries too. With no one-shot caller
left, the retrying release takes the releaseDeleteClaim name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: keep the data under a hidden directory on bucket delete

The scan filed every hidden entry by name for a recursive delete, so .hidden/data went with the bucket. A hidden directory waits with the leftovers instead: the walk refuses one that holds a file or an object, and removeEmptyTree takes only one that is still bare and empty. A hidden file is removed by name as before.

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-11-01 13:43:15 -08:00
Chris LuandClaude Opus 5.5 55f1e1efc5 vector: cache reads of write-once Lance files across requests (#12303)
* lance worker: let dataset::open take an object store wrapper

open_with is open with the dataset's object store wrapped, so a caller
such as weed-vector can put a read cache in front of the table's files.
open keeps its signature and opens unwrapped, as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: cache reads of write-once Lance files across requests

Every VectorService call opens the table afresh, and Lance keeps no
data pages between opens, so each QueryVectors re-read the whole
vector column through the S3 gateway: 83 MB and about 228 range GETs
per query at 50k x 384. The gateway and volume server spent their CPU
streaming it and throughput stayed at 6-8 queries/s whatever the
client concurrency.

weed-vector now wraps each table's object store with a bounded
in-memory cache (--read-cache-mb, 512 MiB by default, 0 disables). It
only serves data files, deletion files and index files: Lance writes
each once under a fresh random name, so a cached read cannot go stale,
even when an index is dropped and recreated at the same location.
Manifests, tags and every other path still go to the store, and each
call still describes the table, loads the latest version and takes
the query lease, so reads see every write committed before them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: cache an owned copy of each fetched read

A range from get_ranges can be a slice of a larger response buffer,
such as a coalesced fetch, and caching the slice kept the whole buffer
alive while the weigher counted only the slice. The cache now copies
each fetched range and whole-object read into its own buffer, so
--read-cache-mb bounds what it actually holds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 12:59:47 -08:00
Mohd Quamar Tyagi 5f4d05b103 admin: measure volume server usage like its capacity (#12306)
The volume server views summed the logical size of the volumes into
DiskUsage while DiskCapacity came from the filesystem statfs size. On a
compressing filesystem the logical sum exceeds the physical capacity, so
the server details popup derived a negative available space and rendered
it as "NaN undefined".

Pair each disk's capacity with the usage measured the same way, using the
statfs numbers the volume server already heartbeats, as CollectTierStats
does. Volume servers without statfs numbers keep the previous slot-based
capacity and logical bytes, and EC shard bytes are only counted there
instead of on top of the filesystem usage.
2026-11-01 12:59:21 -08:00
Chris LuandClaude Opus 5.5 bcb3940b37 test(s3vectors): add an opt-in end-to-end perf suite for vector buckets (#12297)
* test(s3vectors): add an opt-in end-to-end perf suite

perf_test.py runs on the same cluster fixture as the integration suite
(Go or Rust volume, single or distributed queries) and times the data
plane through the S3 Vectors API: PutVectors ingest and upsert, QueryVectors
latency with and without metadata and a 10% filter, query throughput
across client threads, GetVectors, segmented ListVectors, and
DeleteVectors. Recall is checked against an exact client-side search and
listings must be complete; timings only report. Sizes come from PERF_*
variables, and results go to a JSON report and a printed summary.

It runs through run_perf.sh or `make perf`. run_test.sh does not collect
it, so CI is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test(s3vectors): check every listed key in the perf suite, not just the count

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test(s3vectors): cap perf samples at PERF_VECTORS so small runs work

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test(s3vectors): warm perf worker clients before starting throughput timers

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* test(s3vectors): report the key count of perf GetVectors and DeleteVectors calls

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 12:42:51 -08:00
Chris LuandClaude Opus 5.5 5a60ce04e3 vector: stop re-parsing a filter's JSONPath for every scanned row (#12302)
* vector: parse a filter's JSONPath once per batch instead of per row

Lance's json_extract and json_exists parse their JSONPath argument again
for every row they see. Every translated S3 Vectors filter goes through
them, so a flat scan paid about 1 µs per row before the predicate
compared anything, and a 10%-selective {"category": {"$eq": "c3"}} ran
about 50% slower than the same query unfiltered.

The path is always a literal, so the scanner now parses the filter as
before and swaps in copies of the two functions that parse it once per
batch. They call the same jsonb selector and serializer, so results and
errors do not change. Calls whose path is not a literal keep Lance's
functions. The filter SQL is unchanged, so the gateway, existing tables
and the distributed workers (which run query::ann) need no change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors: read the field once per extractor in $in and $nin

$in and $nin rendered one IN per member, each with its own json_extract,
so an N-member list parsed the row's metadata N times. Members that share
an extractor (strings and booleans, or numbers) now go into one IN list,
so a list reads the field at most twice. Matching is unchanged: each
group still only sees its own JSON type and still coalesces NULL before
$nin negates it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors: rename the $in literal map so codespell passes

Codespell reads "lits" as a misspelling of "list".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 12:06:20 -08:00
Chris LuandClaude Opus 5.5 a1a208506e volume: serve a ranged GET without reading the whole needle (#12301)
* volume: serve a ranged GET without reading the whole needle

A ranged GET read every needle up to 1MB in full, checksum and all,
then copied out the requested bytes. Read the metadata and stream only
the range instead, as the Go server already does above 1MB and the Rust
server does at any size. A small range of a 790KB needle goes from
~340us to ~160us.

The metadata read stays non-terminal: the range read that follows
reports the io outcome, so a failing disk still builds its error streak.
HEAD, compressed needles and chunk manifests keep their current reads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* volume: read the whole needle for a range that may need an image transform

Without a URL extension, the stored name can still make a ranged GET
with width, height or crop parameters an image transform, which needs
the payload. Keep the full read there instead of streaming the range.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* volume: stream a range from the record its metadata came from

The streamed read looked the needle up again, so a replacement written
after the metadata read sent new bytes under the old length, checksum
and headers. Read the record the metadata read resolved, as the Rust
server does, and after a compaction refuse a record whose size changed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* rust volume: read the whole needle for a range that may need an image transform

A ranged GET with no URL extension but width, height or crop parameters
took the metadata-first path, so a stored image name never got its
transform. Read a small needle whole there, as the Go server does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* volume: keep later parts of a multi-range reply on a moved record

When a compaction moved the record before one part of a multi-range
reply, that part re-resolved it, but the next part went back to the
offset the metadata read saw. Remember the re-resolved offset too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* volume: check a compaction-moved record is the one the metadata came from

After a compaction, a range re-resolved the needle and accepted any record
of the same size, so a replacement written in between could send its
bytes under the old headers. Compare the moved record's checksum and
append time with the metadata already read, and fail otherwise.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* volume: accept a re-encoded checksum and count read errors on a moved record

Compaction rewrites a pre-3.09 checksum in the current encoding, so the
moved-record check now accepts either. A failed read of the moved
record's metadata now feeds the io-error streak and surfaces as that
error instead of not-found. The pinned record is a fresh value rather
than a write through the index's pointer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* volume: keep a moved record's wrapped offset when a range follows it

With 4-byte offsets, a record past the 32GB wrap is read at offset plus
MaxPossibleVolumeSize. The compaction refresh dropped that, so it checked
and streamed the wrong location. Retry the wrapped offset in the moved
record check, as the metadata read does, and apply it to the stream.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 12:00:22 -08:00
Chris LuandClaude Opus 5.5 4be191173a vector: keep ListVectors pages and segments flat as an index grows; read only keys in GetVectors (#12300)
* vector: scan only the fragments a ListVectors page covers

A page pushed its offset into the scan, but the scan still loaded every
fragment first, reading each one's deletion file: about 105 S3 GETs per
page at 50k rows against 25 at 10k. The page's fragments now come from the
live row counts in the manifest, so a page reads only its own. Tokens and
row order are unchanged; a manifest without the counts keeps the old scan.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors: list a ListVectors segment in the engine

The gateway paged through the whole index for every segment and kept one
row in segmentCount, so a segment cost segmentCount engine pages and 16
segments listed slower than one. The engine now takes segment_count and
segment_index: it reads keys across the stride, keeps the segment's rows
and takes their data and metadata alone, one call per page.

A segment is still the rows whose position modulo segmentCount is
segmentIndex, and its token still the position to resume from. An engine
that predates the fields does not set segmented, and the gateway then
pages through it as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: read only the key column across the table in GetVectors

The key lookup scanned every column. On S3, lance reads a column during
the scan when a row of it is under 1000 bytes, so a 128-dimension vector
was read for every row of the table to return 100 of them, and data and
metadata were read even when the request did not ask for them. The scan
now projects the requested columns and materializes everything but the
key late, for the matching rows only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* s3vectors: start the legacy segment fallback from the page already read

With an engine that ignores segment_count, the gateway dropped the page
it had just read and fetched it again. The fallback now consumes that
page first, so a one-row segment page still costs one engine call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: check metadata on segmented ListVectors pages

The segment test listed data only. Its rows now carry metadata on two in
three keys, and every segmented row, read through take_rows, must equal
the same row in a plain listing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: lease the version while a segment takes its data

A segmented page with data or metadata reads keys first and then takes
the other columns from the same version, so maintenance could reclaim
its files in between. That path now holds a lease like GetVectors. A
keys-only page is one scan and stays unleased, as plain pages are.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

* vector: leave GetVectors materialization to lance

Forcing every column late made a 100-key GetVectors with data slower at
10k x 128 in the perf cluster: 54.7 ms against 46.3 ms. The take issued
a read per row, where lance's heuristic reads a narrow vector column in
one pass per fragment. The projection keeps the gain on its own: with
neither data nor metadata requested, the call went from 38.6 ms to
25.3 ms and from 150 to 89 S3 GETs. At 50k x 384 the vector is already
late under the heuristic, and a full lookup is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 11:57:44 -08:00
Chris LuandClaude Opus 5.5 a7a6482752 vector: follow a leader hint at once and skip one that failed this pass (#12295)
* vector: follow a leader hint at once and skip one that failed this pass

A follower naming a dead leader stalled master failover. Each pass broke
off at the first hint, tried that leader first on the next pass, then
got the same hint from the same follower, so the later configured
masters were never tried. A pass now works like wdclient.tryAllMasters:
it follows a hint at once, does not follow a hint to a master that
already failed in that pass, and moves on to the rest. The hops are
bounded by the master count, as the volume heartbeat bounds its hints,
so two masters naming each other cannot spin.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* vector: test a hint naming a failed master in its gRPC form

A pass keys failed masters by host:http, so a configured m1:9333 that
failed must also stop a hint of m1:9333.19333. Cover that equivalence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 03:11:03 -08:00
Chris LuandClaude Opus 5.5 6cdddcd76d vector: confine DropIndex to the table's own path, bound master dials, check the gRPC port (#12294)
* vector: refuse to drop a location that is not the table's own

DropIndex removed everything under whatever location the catalog
described. That location is caller-supplied through the Lance
namespace and only confined to the bucket, so a table registered at
s3://bkt/vector, or at another index's directory, wiped every index in
the bucket or that index. drop now accepts only the colocated path the
gateway's tableLocation derives from the table id, after trimming a
trailing slash, and the RPC answers FailedPrecondition otherwise. This
mirrors the ancestor and live-entry guards on the catalog's DeleteTable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* vector: bound the master dial and try a leader hint once

A remembered leader hint was never cleared and the dial had no connect
timeout, so a black-holed former leader added the kernel's SYN timeout
(about two minutes) to every reconnect pass before the configured
masters were tried, and registration lapsed. The dial now gives up
after 5s, the same bound wdclient puts on a master call, and a hint is
tried once ahead of the configured masters, as wdclient.tryAllMasters
keeps nextHintedLeader for one cycle only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* vector: refuse a master port whose gRPC port would pass 65535

grpc_addr derived the gRPC port as http + 10000 in u16, which panics in
debug builds and in release silently dials a wrapped port for HTTP
ports above 55535. Go's pb.ServerToGrpcAddress adds in int, so the port
never wraps and the dial fails instead. The addition is now checked,
and weed-vector refuses to start with an error asking for the explicit
host:http.grpc form.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* vector: refuse a numeric master port past 65535

A master given as m1:65536 or m1:9333.65536 failed both u16 parses and
fell through to the raw address, so startup accepted it and every dial
failed. Ports now parse as u64, as Go's ServerToGrpcAddress does with
ParseUint, and one outside u16 is refused at startup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* vector: bound the master's first keep-connected answer

The connect timeout only covers TCP. A master that accepts the stream
and never answers held stream_to forever: grpc-go sends the response
headers with the first message, so the call never returned and later
seeds were never tried. A healthy master always answers at once (a
follower with its leader hint), so the stream open and the first
response now share the same 5s bound; after that the long-lived stream
stays untimed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 18:42:18 -07:00
github-actions[bot] 455400ed4a docs: regenerate star history chart 2026-11-01 00:37:26 +00:00
Chris LuandClaude Opus 5.5 ab355b43b9 s3tables: keep the vector-namespace fence on a declare of an existing index (#12293)
s3tables: keep the fence on a declare of an existing index

Deciding the existing-index exception before the handler ran left a
window: an index deleted between that lookup and the handler's own check
would be recreated, unfenced, by a caller without s3vectors:CreateIndex.

Drop the up-front exception, so every CreateTable runs fenced and a
fresh create is refused at its first write. The engine's declare still
passes because the idempotent path writes nothing; for a build whose
idempotent path re-applies state already in place, the fence answers an
UpdateEntry that leaves the entry exactly as stored without writing.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 16:07:31 -07:00
Chris LuandClaude Opus 5.5 e2826c48bd s3tables: make the vector namespace of a vector bucket read-only (#12292)
* s3tables: make the vector namespace of a vector bucket read-only

S3 Vectors authorizes index operations against the vector bucket policy,
but an index is also a catalog table in the bucket's "vector" namespace,
and the S3 Tables API only checks s3tables grants. A caller the vector
policy denies DeleteIndex could still delete the index, and its data,
with DeleteTable.

Untrusted writes that name the vector namespace of a LANCE bucket now run
behind a fence that passes filer reads and refuses filer writes. The
operation's own checks still answer everything short of a write, so a
caller without the grant gets its usual denial and learns nothing about
the bucket, and a CreateTable that finds the index already there, which
is what the vector engine's declare does, still succeeds. A refused write
answers 403 AccessDenied. Trusted callers (the gateway, the shell) are
unaffected, and so is a namespace named vector in an Iceberg bucket.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* lance: refuse register and deregister in a vector bucket's vector namespace

Every catalog write the Lance namespace makes goes through the S3 Tables
handler and its vector namespace fence, but register over an existing
table at the same location and deregister only add or remove a marker
file in the dataset directory. Check the vector namespace before those
writes, after the table read that already gates them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: refuse deferred creates in a vector bucket's vector namespace

Manager.AuthorizeCreateTable gates the Iceberg create paths that write
metadata into the table bucket before the catalog registers the table.
Those writes never pass the handler's fence, so the gate applies the same
read-only rule once the caller's own create permission is confirmed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: let a declare of an existing index past the vector namespace fence

The vector engine declares every index the gateway has just created, and
that CreateTable only reaches its idempotent path. Upstream that path
writes nothing, so the fence never trips, but a build whose idempotent
path repairs table state would refuse the engine. Decide the exception
up front instead: a CreateTable naming a finished index in the vector
namespace runs unfenced, while a new name or an unfinished create stays
fenced.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 15:36:48 -07:00
Chris LuandClaude Opus 5.5 e118ba7127 vector: keep S3 Vectors requests off the master-discovery lock (#12291)
* vector: throttle worker discovery from lookup end

cli() set nextResolve before resolve() ran, so a lookup that takes as
long as the throttle interval (an unreachable first master costs the
full 5s dial timeout) left the next lookup already due when it
returned. Discovery then ran back to back and, since it holds d.mu,
every vector request queued behind it.

resolve() now sets nextResolve when the lookup lands, using the short
recheck window only while no client is cached.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* vector: resolve workers outside the discovery lock

resolve() held d.mu across the master lookup, so while it waited out an
unreachable master every vector request blocked on cli(), even with a
healthy worker already cached.

The lookup now runs in a goroutine without d.mu, and d.resolving keeps
it to one at a time. Callers with a cached client use it while the
lookup is in flight; callers without one wait for that lookup and share
its result. Throttling, the sticky worker and reset(c) are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* vector: keep the cached worker connection when a redial fails

resolve() closed d.conn before dialing the replacement worker. A
synchronous dial error then left d.client bound to a closed connection.
Requests got codes.Canceled, which never triggers reset, and later
refreshes saw the unchanged d.worker and never redialed. The old
connection is now closed only after the new dial succeeds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 15:35:52 -07:00
Chris LuandClaude Opus 5.5 9a09506be6 s3vectors: bound ListVectors parameters and lowercase distanceMetric (#12290)
* s3vectors: bound ListVectors maxResults and segment parameters

segmentCount had no upper bound, so segmentCount=1000000,
segmentIndex=999999 made the segment emulation scan the whole index to
return nothing. Apply the AWS ListVectors windows before authorization
and any engine call: maxResults 1..1000, segmentCount 1..16,
segmentIndex in [0, segmentCount), and segmentCount and segmentIndex
given together. The fields become pointers so an explicit 0 is rejected
while an omitted one still takes the default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3vectors: store and return distanceMetric in lowercase

CreateIndex stored distanceMetric exactly as sent, so an index created
with "COSINE" came back as "COSINE" from GetIndex and QueryVectors.
AWS returns the lowercase enum values. Lowercase the metric when the
index is created, and again when the record is read so indexes already
stored in mixed case render the same.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 15:08:46 -07:00
Chris LuandClaude Opus 5.5 21fc46692c s3vectors: authorize CreateVectorBucket as a vector action, close DeleteIndex race, fold policy service case (#12287)
* s3vectors: authorize CreateVectorBucket as an s3vectors action

CreateVectorBucket went through the catalog's CreateTableBucket check, so
an identity granted s3vectors:* was denied the create while s3tables:*
was allowed it. Check s3vectors:CreateVectorBucket in the gateway, the
same way the catalog checks a bucket create, and create the record on
the internal manager under the caller's name so the caller stays the
owner.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3vectors: delete only the index record DeleteIndex checked

DeleteIndex read the record, checked it is a Lance index, then deleted
by name. A table that replaced the index in between was deleted too.
Send the checked record's version token, so a replaced record fails the
delete with a 409 ConflictException.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3tables: reject s3vectors rules in any letter case

validatePolicyDocument kept s3vectors actions and resources out of the
shared policy slot with case-sensitive checks, so "S3Vectors:GetIndex"
was stored as a rule that never takes effect. IAM service prefixes are
case-insensitive; fold case in both checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 15:07:23 -07:00
Anton Rozanov e7f5c6873a ec.balance: spread concurrent shard moves over target nodes (#12283)
* ec.balance: spread concurrent shard moves over target nodes

The planner gives every freshly encoded volume the same target order, so
running each volume's moves in plan order sends all concurrent copies to the
same node while the other nodes idle. Pick, among the volumes not already
moving, the pending move whose target receives the fewest moves right now.
One volume still never has two moves at once, and moves of one shard keep
their plan order.

* ec.balance: let a move into a node wait for the move of its volume leaving it

The planner counts the slot an outgoing shard frees as room for a later
incoming shard of the same volume, and the receiver refuses a shard it has no
slot for. Make the spread tests deterministic: hold the first moves until all
have started, and check that errors of several volumes are all returned.

* ec.balance: pin that the spread moves run concurrently and in plan order without parallelism
2026-10-31 15:05:32 -07:00
Chris LuandClaude Opus 5.5 3a8d756bdd s3: drop retired volume servers from the gateway location cache (#12281)
* wdclient: publish applied volume-location deletes to registered listeners

The master already tells each connected client which volumes a server
stopped hosting; only the MasterClient's own vidMap applied it. Expose
the same filtered deletions (moves between a server's disks excluded)
through OnVolumeLocationDelete so clients keeping a second location
cache can drop the retired server too.

* wdclient: drop retired volume servers from the FilerClient location cache

The S3 gateway's FilerClient keeps its own vidMap, filled by filer
lookups, and only forgot a retired volume server after a read against
it failed (a ~15s header-timeout stall when the host still resolves).
When built with a MasterClient, subscribe to its applied deletions so
the cache forgets the server as soon as the master reports it gone.

* wdclient: drop retired EC servers from the regular location map too

Filer lookups cache EC locations alongside regular ones in vid2Locations,
so an EC removal has to clear the retired server there as well as in the
EC map for the delete to reach what reads actually use.

* wdclient: tombstone pushed deletions against in-flight provider fills

A filer lookup already running when a delete lands can return the retired
location afterwards and re-cache it. Keep a short tombstone on each
deleted (vid, url) pair so fills neither cache nor return it while the
lookup source is still settling.

* wdclient: clear the FilerClient location cache on master reconnect

Removals the master pushed while the gateway link was down never arrive
as deletes, so on reconnect the lookup-fed cache is dropped outright
rather than kept generation-wise like the stream-fed one, and relearned
through filer lookups as before.

* wdclient: invoke push hooks outside their registration locks

A callback that registers or replaces a hook would wait on the read lock
the invocation already holds and stall the master stream. Copy the hook
under the lock and call it after the lock is released.

* wdclient: report pushed adds so confirmed re-adds clear the suppression

A server coming back at the same URL inside the tombstone window stayed
hidden from provider fills. updateVidMap now carries the additions it
applied through the same hook (renamed OnVolumeLocationChange), and the
registered cache untombstones them without eagerly caching them.

* wdclient: amortize tombstone expiry over a delete batch

A server leaving takes every volume it hosted with it, and scanning all
recentDeletes per removed vid squared the batch. Prune lazily on reads
and at most once per half TTL on writes.

* wdclient: let every location cache subscribe to pushed volume changes

The setters were single-slot, so a second FilerClient built on the same
MasterClient would have silently replaced the first one's subscription.
Registration now returns an unhook func, delivered to all listeners, and
FilerClient.Close releases its own.

* wdclient: hold the deletion fence until the master re-confirms a location

* wdclient: check the deletion fence and the cache add under one lock

* wdclient: drop provider answers that crossed a cache reset

* wdclient: bound the deletion fence by evicting its oldest mark

* wdclient: refresh a repeated removal instead of evicting another mark

* wdclient: evict the oldest deletion-fence mark in constant time

* wdclient: fence a server only once it serves the volume in no form

ec.encode registers the shards before the regular copy is deleted from the
same server, and ec.decode mounts the regular copy before the shards come
off it. Both arrive add-first in separate heartbeats, so the later removal
tombstoned a (vid, url) pair the server still serves and dropped it from
the FilerClient map; the filer's answer naming that server was then fenced
out and the volume read as not found until the gateway reconnected.

Tombstone and report a removal only when neither map still names the url
for that volume.

* wdclient: serve a provider answer that crossed a cache reset

The answer is as fresh as the provider's view; only the cache it would
land in has been re-told. Dropping it failed every read whose lookup was
in flight during a master reconnect with 'volume not found'. Return it
and leave the new view to be filled by the next lookup.

* wdclient: test the moved volume whose old server keeps proxying

* wdclient: stop caching inside the master lookup provider

The provider wrote every answered location through addLocation, which
clears the deletion fence: a lookup still naming a server a pushed delete
had just retired untombstoned and re-cached it. Fallback lookups already
gate caching through fillLocation, and authoritative lookups bypass the
cache by contract, so the provider no longer writes the map at all.

* wdclient: count only master-pushed locations when suppressing a removal

hasLocation could not tell a confirmed regular copy from a lookup fill:
the master answers an EC volume with its shard servers, so the regular map
holds shard locations that arrived by lookup, and entries an earlier
generation learned linger past a reconnect. An EC removal whose url still
showed either was suppressed, leaving the retired server served.

The vid map now records the generation each regular location was pushed
at, drops an unconfirmed regular entry alongside the shard that fed it,
and answers hasServingLocation from pushed marks and current-generation
entries only.

* wdclient: test removals against lookup-cached and stale locations

* wdclient: fence removals only in the gateway cache

The fence and the pushed-location marks ran inside every MasterClient,
including the filer's, whose map is also filled by lookups and cleared
by InvalidateCache. A removal reported while the server still served the
volume (e.g. a decoded copy whose mark a failed read had wiped) left the
pair fenced until the master pushed it again, with reads failing as
volume not found meanwhile.

Only the S3 gateway's FilerClient now tombstones, from the removals its
push-only MasterClient reports. That client decides a removal from the
current generation of both its maps after the whole message is applied,
so pushedLocations goes away, and the master lookup provider caches as
before. The hooks follow the existing single-slot SetOn...Fn pattern and
are wired next to SetOnPeerUpdateFn.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* wdclient: let a removal fence expire after a minute

The master drops a broadcast when a client's channel is full, so the
re-add that would lift a fence can be lost while the gateway stays
connected. A mark held until that re-add then hid a live server, and a
volume whose only copy it held read as not found until the next master
reconnect. Bounding marks by a minute still outlasts a lookup in flight
when the removal landed; expiring early only falls back to learning by a
failed read, as before this change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 15:05:00 -07:00
Chris LuandClaude Opus 5.5 fdbaba52f9 s3 sink: forward user metadata to the destination object (#12286)
* s3 sink: forward user metadata to the destination object

filer.backup and filer.sync to an S3 sink copied the object body and tags
but dropped the X-Amz-Meta- attributes the S3 gateway stores with each
object, so the destination object came back without its user metadata.
Send them as the upload's metadata, keyed by the lowercased suffix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 sink: send mtime as metadata without writing it into the entry

The sink stored X-Amz-Meta-Mtime into the replicated entry's Extended map,
modifying the caller's entry, and since tags were limited to X-Amz-Tagging-
keys it was never sent anywhere. Add it to the upload metadata instead,
keeping an mtime the source object already carries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 sink: RFC 2047 encode metadata values that are not printable ASCII

The S3 gateway keeps user metadata values as raw bytes. Forwarded as is, a
UTF-8 value fails SigV4 on AWS and a control character makes the HTTP client
reject the request, and filer.backup retries such an object forever. Encode
those values as RFC 2047 encoded words, which S3 decodes on write.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 sink: upload without user metadata past the S3 2 KB limit

The S3 gateway does not cap user metadata, but S3 rejects more than 2 KB of
it, and filer.backup would retry that object forever. Upload such an object
without its metadata and log a warning, as the sink did before metadata was
forwarded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 sink: skip metadata keys that are not valid header names

Filer gRPC clients can store any Extended key. An X-Amz-Meta- key with a
space or control character makes the HTTP client reject the whole upload,
which filer.backup retries forever. Leave such keys out with a warning.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 sink: pick the same value for metadata keys that differ only in case

An entry holding both X-Amz-Meta-Project and the legacy x-amz-meta-project
maps both to project, and map order decided which value was uploaded. Keep
the lowest key, so the canonical form wins over the legacy lowercase one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 sink: use a test value codespell accepts

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 14:15:13 -07:00
Chris Lu 1c84f98cdf master: deleting a read-only volume no longer drifts the active count (#12285)
* master: undo the stored record when a delete report arrives

Deleting a volume decremented the active count from the delete report's
ReadOnly flag, which the volume server never sets, so a volume the master
had already stopped counting as writable was uncounted a second time.
Every volume.move does this to its source, and the node, rack, DC and
topology counts drifted negative; the full-list path already used the
stored record. Undo exactly what was counted for the record held.

* volume: say whether a deleted volume was read-only

Explicit deletes, unmounts, collection deletes and the departures named
in a delta heartbeat all left ReadOnly unset, so an older master undid
the writable count for a volume it was no longer counting. Carry the
flags, as the Rust server already does.
2026-10-31 13:28:32 -07:00
Chris Lu 42b6bc3d26 release: ship the Linux Rust binaries as static musl builds (#12284)
* ci: share the musl Rust target setup as a composite action

Three container workflows carried the same toolchain, musl-tools and
aws-lc-sys link shim block verbatim.

* release: ship weed-worker as a static musl binary

The gnu build inherits the runner's glibc, 2.39 since the move to
ubuntu-24.04, so the released worker failed to start on Ubuntu 22.04
with 'GLIBC_2.38 not found'. The container images already build it
against musl with the same recipe.

* release: ship the Linux weed-volume as a static musl binary

Same glibc trap as the worker. The arm64 job also installed libssl,
which nothing in the volume links.

* ci: resolve the manual latest build's actions from master

Its checkout is an arbitrary source ref that may predate a local
action; the workflow itself always runs from master.
2026-10-31 12:28:07 -07:00
Chris Lu 8d3cb2ed2d filer: resume a rejoining peer's meta subscription at its last position (#12279)
* filer: resume a rejoining peer's meta subscription at its last position

OnPeerUpdate starts every peer add at the update's arrival time. For
filers sharing one store there is no persisted offset to correct it, so a
peer that drops out and rejoins (a graceful restart) resubscribes from
"now" and the events it logged while its stream was down never reach the
aggregated buffer - filer.backup and other meta subscribers silently
skip them.

Track each peer's delivered-through position (the same timestamp domain
as the stream's events and idle-heartbeat claims) and rewind the start
of a rejoining subscription to it. First-seen peers keep the given
start, and non-shared peers still resume from their persisted offset.

* filer: keep the re-add resume point behind a replay-floored event

The session's replay floor already pins the persisted offset and the
reconnect resume point so an event that outlived its retry budget is
redelivered on resubscribe; cap peerResumeTsNs to the same position on
session exit so the new re-add resume cannot skip past it either.

* filer: stop test aggregators' log-buffer workers

* filer: cover peer rejoin resubscribing at the resume position

* filer: drop the unreachable replay-floor cap on the resume point

Only a peer with its own store can floor an event, and that peer's
subscribe start always comes from its persisted offset, never from the
re-add resume point. The cap also ran in the old session's defer, after
the re-add had already read the value.

* filer: say why a re-add may resume from an idle heartbeat

The heartbeat comments promised a reconnect never resumes from one,
while the re-add resume point does. Spell out the difference and the
fence that makes the local-stream claim safe to resume from.

* filer: forget a removed peer's resume point after an hour

The map held an entry for every address ever seen; with pod IPs that
is one per restart. The removal mark now outlives the watermark grace
and takes the resume point with it once the retention passes.

* filer: advance the resume point only for tracked peers

Same rule as the watermarks: a straggler from a dropped session must
not recreate the entry.

* filer: set the test aggregator's subscribe loop instead of a nil fallback
2026-10-31 11:49:50 -07:00
Nicholas Oh f5989a21fb s3: honor assignment fsync in streaming copies (#12277) 2026-10-31 05:28:54 -07:00
a94edebf00 fix(s3tables): keep the data under a deleted namespace (#12276)
* fix(s3tables): keep the data under a deleted namespace

A rename leaves the source table directory holding data the renamed table
still points at, and DeleteNamespace removed it recursively. The delete now
keeps the directory, minus its catalog record, when files remain under it;
CreateNamespace can take it back.

* fix(s3tables): keep hidden-directory data, report name takeover as 409

- DeleteNamespace also looks for files under hidden directories, since a
  table location may sit under one.
- CreateNamespace answers 409 when a file takes the name during the
  take-back of a kept directory.

* s3tables: a vanished leftover holds nothing to keep

A bare directory removed between the namespace listing and the walk
made dirHoldsFiles answer its ListEntries NotFound, and the delete
reported that miss as a 404 or 500 on a live namespace. Skip the gone
directory instead: whatever it held is already gone, so it cannot keep
the namespace.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3tables: a delete's gone entry is a conflict, not a success

The filer answers a precondition delete on an entry that is already gone with FailedPrecondition carrying the not-found text, which DoRemoveRequest reads as a completed delete -- DeleteNamespace answered 200 for a delete that never ran. deleteDirectoryIfMatches now calls DeleteEntry directly, keeping the FailedPrecondition-to-conflict mapping and refusing a response-level error. keepAsLeftover likewise reports a directory removed since the scan as ErrConcurrentUpdate, so the handler answers 409 rather than 500.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-31 05:27:58 -07:00
Jay HemnaniandChris Lu ecd716e0e2 s3: allow POST Object uploads when auth is off (#12273)
* s3: allow POST Object uploads when auth is off

With no identities configured every other handler skips authentication,
but PostPolicyBucketHandler still verified the form signature, so a
browser form upload failed with InvalidAccessKeyId (or a malformed
credential error without signature fields). Skip the signature check
when auth is off; the policy conditions are still enforced.

* s3: update AuthPostPolicy comment for the auth-off path

---------

Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-31 05:19:17 -07:00
Chris LuandDevin 05584c541c s3: take list markers as raw keys under encoding-type=url (#12275)
* s3: take ListObjects marker/start-after as raw keys

Under encoding-type=url AWS url-encodes the response fields but treats
request markers as raw keys: SDKs decode Marker/NextMarker/StartAfter and
send the decoded key back. Unescaping the marker server-side read a literal
"%XX" key as the key it encodes — StartAfter=a%2Bb resumed after "a+b",
skipping it, and a paginator whose marker decoded to an earlier key got the
same NextMarker twice and failed.

Take the request value as-is; only the server's own base64: continuation
token still decodes. The marker-echo dedup now compares contents in the
response wire encoding and NextMarker in its raw form, since a raw "%XX"
marker equals the escaped emission of a different key.

* s3: take ListMultipartUploads key-marker as a raw key

Same raw-marker rule as ListObjects: under encoding-type=url the emitted
key markers are url-escaped keys, and a client decodes them before sending
the marker back, so the request value is already the raw key. Keeping the
token decode only for the non-url path, where the response could not carry
the raw key and falls back to the base64: token form.

* s3: cover ListMultipartUploads url key-marker at the handler

The selectMultipartUploads tests bypass the request marker decoding, so a
regression that unescapes an encoding-type=url key-marker would pass them.
Drive a listing through ListMultipartUploadsHandler with a literal
"a%2Bb" marker and assert it resumes after that exact key.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-31 05:15:47 -07:00
Jay Hemnani 26783fbcb3 s3: keep an unset CORS MaxAgeSeconds unset (#12272)
A rule without MaxAgeSeconds was stored as 0, so GetBucketCors returned
MaxAgeSeconds 0 and preflights sent Access-Control-Max-Age: 0. Store -1
for unset, which corsRuleFromProto already reads back as nil.
2026-10-31 05:11:42 -07:00
hsdfat 9df2125bb0 fix(s3): reject non-integer max-parts, part-number-marker and max-uploads (#12270)
getObjectResources and getBucketMultipartResources dropped the strconv
error, so a value that does not parse was read as 0. ListParts with
max-parts=abc then reached the filer with limit 0, which the filer reads
as its default listing limit, and returned every part with
<MaxParts>0</MaxParts>; part-number-marker=abc restarted the listing
from part 1; and ListMultipartUploads with max-uploads=abc answered an
empty, untruncated list even with uploads in progress.

Return the existing InvalidArgument errors (ErrInvalidMaxParts,
ErrInvalidPartNumberMarker, ErrInvalidMaxUploads) instead, the way
max-keys is validated. max-parts and max-uploads are parsed as 32-bit
integers to match the documented 0..2147483647 range.
2026-10-31 05:10:50 -07:00
6cf6ce81d7 fix(volume): reject a negative suffix length in Range headers (#12269)
* fix(volume): reject a negative suffix length in Range headers

parseRange accepted "bytes=--5" as a suffix range of -5 bytes and
returned {start: size+5, length: -5}. The volume server then answered
206 with an empty body and Content-Range "bytes 15-9/10", and the
filer's inline-content reader sliced entry.Content[15:10] and panicked.

Restore the check Go's net/http parseRange has: a suffix-length must be
a non-negative integer (RFC 7233 2.1), so a leading '-' or a negative
value is an invalid range. Apply the same check to the Rust volume
server's parse_range_header.

* test: negative-suffix manifest read is now rejected with 416

The parser change makes a negative suffix an invalid-range error, so
the duplicate-range case in this test can no longer reach the "empty
200" path. Assert the new rejection and that nothing is read.

* volume: treat a zero-length suffix range as unsatisfiable

"bytes=-0" resolved to {start: size, length: 0}, and the range handler
then answered 206 with "Content-Range: bytes 10-9/10", a value clients
must not recombine (and a first-byte-pos RFC 9110 cannot express).
Classify the zero-length suffix with the existing no-overlap case —
its resolved first-byte-pos sits at the content size — so the request
gets a 416 "bytes */size" unless another range in the set overlaps.
Applies to the Go and Rust volume servers alike.

---------

Co-authored-by: chrislusf <chrislusf@gmail.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
2026-10-31 05:09:14 -07:00
Chris LuandDevin 97e1758bcc s3: auto-create bucket on POST Object uploads to a missing bucket (#12266)
* s3: auto-create bucket on POST Object uploads to a missing bucket

PutObject and CreateMultipartUpload lazily create the bucket, but a
signed POST Object upload answered NoSuchBucket. Mirror the PUT path:
when the versioning-state lookup reports the bucket missing, run the
shared handleAutoCreateBucket gate (bucket-name check, AutoCreateBucket
option, admin identity) and then re-read the versioning state. The POST
handler already records the verified signer identity in the request
context, so the admin check and bucket-owner attribution work unchanged.

* test: assert the POST auto-create upload lands, not just non-404

The missing-bucket POST test passed whenever the handler created the
bucket, even if the upload then failed. Drive the full path through the
existing putToFiler fakes — assigned volume accepts the chunk, object
entry and payload land — and assert the 201 success response.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: record verified post-policy signer's account id

The POST policy path authenticates in the handler, after the internal
account header was already scrubbed, so auto-created buckets and the
uploaded objects stored no canonical owner.

* test: assert POST auto-create records bucket owner, keep writes applied in denied case

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-31 05:08:01 -07:00
Chris LuandDevin b70c6dc6ed s3: let bucket-scoped Admin grants auto-create matching buckets (#12268)
* s3: let bucket-scoped Admin grants auto-create matching buckets

autoCreateBucket required a global Admin identity, so an identity
scoped to a prefix like "Admin:outline*" could explicitly CreateBucket
but was denied when an upload auto-created the same bucket. Gate on
CanDo(ACTION_ADMIN, bucket) — the same check explicit CreateBucket
enforces — so a prefix-scoped key lazily creates only matching buckets.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: run auto-create bucket admin through the policy-aware path

A scoped Admin identity with an attached IAM policy that denied
s3:CreateBucket could still auto-create a matching bucket, because
userCanAdminBucket only consulted legacy Actions. Evaluate the same
VerifyActionPermission explicit CreateBucket gets, over a synthesized
PUT /bucket request so the attached-policy check resolves to
s3:CreateBucket on the bucket ARN regardless of the upload's shape.

* s3: record verified identity as owner of auto-created buckets

An admin authenticated outside the middleware (IAM integration, post
policy signature) left no identity in the request context, so an
auto-created bucket stored no owner and never reached the owner index.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-31 05:04:56 -07:00
bed0d7401d volume: after losing the leader, dial the leader the other masters name (#12259)
* volume: after losing the leader, dial the leader the other masters name

When the heartbeat stream to the leader breaks and that master stays
unreachable (partitioned or powered off, so dials hang rather than being
refused), the volume server could take close to a minute to register with
the new leader, and the new leader answered assigns with "no volume server
capacity registered yet" in the meantime.

Two things in the retry path added up:

- refreshLeaderFromPeers asked the seed masters in order with up to 5s
  each, inside a budget of one pulse (5s by default). With the lost
  master listed first, the whole budget went to it and the live peers
  that already knew the new leader were never asked.
- Even when the refresh found the new leader, the loop moved on to the
  next seed master. When the lost leader had been reached through a
  redirect, the next seed can be that same master, so the loop dialed it
  again and waited out the 20s gRPC connect timeout.

Ask the lost master last, and when the refresh names a leader other than
the lost master, dial it next.

Assisted-by: Claude <noreply@anthropic.com>

* rust volume: dial the leader the other masters name after losing one

Mirror the Go heartbeat change: refresh_leader_from_peers asks the lost
master last and returns the leader it found, and the loop dials that
leader next instead of a seed that can be the unreachable master itself.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: let a named leader consume the next seed slot

Restarting the seed scan on every redirect let an earlier seed that keeps
reporting the dead leader starve every seed after it: the loop bounced
between the redirect target and the next seed forever. Go consumes the
next range value for the redirect instead, so a pass always reaches every
seed. Drop the breaks so new_leader overrides the next slot the same way.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: try a named leader on top of the seed scan

A reported leader used to replace the next seed slot: with seeds A,B
and A persistently naming a dead leader X, X consumed B's slot on
every pass so B was never dialed. The hint is now a one-shot attempt
made before the seed it would have displaced. Its own result stays
pending when a seed attempt names nothing, so a live follower's
redirect chain is followed while every seed is still dialed on every
pass. Mirrors between the Go and Rust volume servers.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: follow a fresh leader hint before the pending seed

A named leader whose stream breaks can leave the peer refresh naming its
replacement; the loop then dialed the pending seed first and paid a dead
master's connect timeout ahead of the fresh name. Fresh hints are now
followed right away, bounded by the seed count and stopping once a name
repeats, so a peer that keeps naming the same dead leader still cannot
starve the seed scan. Go and Rust mirrored.

* volume: dial each named leader once per pass (Go + Rust)

A name that stays pending after the hint chain — the cap was hit, or the
refresh named one already dialed — was retried before every later seed in
the pass: each seed reset the hint budget, so a dead leader still cost a
sleep and a connect timeout per seed. Track the names dialed in a pass,
seeds included, and skip a hint that repeats one.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: let a seed's named leader replace a pending hint (Go + Rust)

The seed dial runs after the hint chain, so the leader it names is the
freshest answer the pass has — but it was dropped whenever a stale hint
was still pending, and the next seed then chased the older name first.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: keep an untried leader hint over the seed's stale answer (Go + Rust)

A seed that keeps naming a stale leader spends the per-pass hint budget
on it; its own answer then overwrote the live name the hint chain had
just reached, so that leader was never dialed. A pending hint now
survives the seed dial until it has been tried.

---------

Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-31 04:10:36 -07:00
Chris LuandDevin e98adfd3bb security: fail closed on gRPC TLS errors, verify dialed server identity (#12263)
* security: fail closed on misconfigured gRPC TLS

LoadServerTLS/LoadClientTLS returned plaintext credentials on every
error path once TLS was configured: a typo in tls.min_version or
tls.cipher_suites, cert and key in different directories, or cert/key
set without grpc.ca all left the component serving or dialing
plaintext. A partially configured TLS setup now stops the process at
startup instead; leaving cert/key unset still means no TLS, as before.

The loaders also stat cert/key/ca files up front since the pemfile
providers read lazily and would otherwise only fail at handshake time.

Mirrored in the Rust volume server: partial grpc.volume cert/key, a
missing grpc.ca, or a rustls build failure now abort startup rather
than warn-and-continue on plaintext.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* security: optional server-identity allowlist for gRPC clients

The mTLS client accepts any certificate the configured CA signs, so a
per-workload CA lets one component stand in for any other: a filer
identity can answer as the master. The server side already screens
callers via allowed_commonNames and allowed_wildcard_domain; nothing
screened the server.

LoadClientTLS now wires AdditionalPeerVerification when the caller
configures a server allowlist: grpc.<component>.allowed_server_commonNames
(falling back to a shared grpc.allowed_server_commonNames) matched
against the peer cert's subject CN, DNS SANs, or URI SANs such as a
SPIFFE id, plus a grpc.allowed_server_wildcard_domain suffix match.
Leaving them unset keeps the previous behavior.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* security: URI SANs are exact-match only in the server identity allowlist

A URI identity whose path happens to end in the wildcard domain would
otherwise pass the DNS suffix check.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* security: tighten gRPC TLS fail-closed and server-identity paths

- URI SANs match the server-identity allowlist exactly; the wildcard
  domain suffix check applies only to CN and DNS SANs.
- A partially set client_cert/client_key pair is an error when the
  cert/key fallback is empty, instead of dialing plaintext.
- Cert/key/ca files are loaded and parsed at startup (LoadX509KeyPair
  and CA PEM validation) since pemfile providers read lazily.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: error on a partial grpc client_cert/client_key pair

Mirror the Go loadClientTLS path: when client_cert or client_key is
set without its mate and the cert/key fallback is empty, fail instead
of dialing plaintext.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* security: wildcard domain match requires a valid hostname

A CA-signed identity like ".example.com" satisfied the bare suffix check
for wildcard ".example.com" despite not being a hostname. Require a real
subdomain: non-empty label before the suffix, a '.' boundary when the
configured wildcard has no leading dot, and DNS-name syntax. A leading
"*." label still matches so wildcard certs keep working. Applied to both
the Go server/client authenticators and the Rust volume's client-cert
verifier.

* security: ignore a trailing root dot in wildcard domain matching

* tls: reject a partial dedicated client cert/key pair instead of falling back

When exactly one of component.client_cert/client_key is set, warn-and-
fallback silently dialed with the serving identity. Fail closed like the
other TLS misconfigurations in this change; mirrored in the Rust loader.
Regression tests cover a partial pair with a valid fallback.

* rust volume: harden partial client cert pair test with real pem files

* security: verify dialed server identity per target kind on the filer

The filer reused one grpc.filer-verifying dial option for master, volume,
filer and s3 targets, so a broad allowlist accepted the wrong component at
the wrong port and a narrow one rejected legitimate peers. Outbound calls
now pick grpc.<target>.allowed_server_commonNames (falling back to the
client component's and the global list) while still presenting the
grpc.filer client identity.

* security: scope per-target server allowlists under the client component

The filer read grpc.volume.allowed_server_commonNames when dialing a
volume, but that key already denotes the volume's own outbound allowlist
(e.g. the master identity). A deployment restricting volume->master
calls would make the filer reject real volume servers. Per-target policy
now lives under the client component: grpc.filer.volume_*
falls back to grpc.filer.* then grpc.*.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-30 20:04:45 -07:00
Nguyễn Lộc Phúcanduser.email 7d8b72c2b4 fix(s3tables): answer 404 for a table name that was renamed away (#12258)
* s3tables: answer 404 for a table name that was renamed away

A rename leaves the source directory in place to hold the data under
it, with its catalog attributes cleared. UpdateTable, DeleteTable, the
table policy calls and the table maintenance calls read the metadata
attribute of that directory and reported its absence as an internal
error, so they answered 500, and GetTablePolicy answered NoSuchPolicy.

An absent metadata attribute now reads as a missing entry. tableMetadata
reports it as both filer_pb.ErrNotFound and the attribute error, so
callers that test for either keep working. The policy and maintenance
handlers read the attribute through the new readTableMetadataAttr, which
reports filer_pb.ErrNotFound, so each call answers NoSuchTable.

* s3tables: a view record without its metadata reads absent too

* s3tables: refuse a catalog write that lands on a metadata-less record

---------

Co-authored-by: user.email <chrislusf@users.noreply.github.com>
2026-10-30 19:53:57 -07:00
468aabc962 helm: add commonLabels to the metadata of every object (#12257)
* helm: add commonLabels to the metadata of every object

The chart lets users label pods (podLabels, <component>.podLabels) and
annotate workloads, but offers no way to label the objects themselves.
Backup, ownership and cost policies usually select StatefulSets,
Deployments, Services, PVCs or Secrets by label, and umbrella charts
that set such labels on all their workloads have to patch this chart
to do it.

commonLabels is added to the metadata labels of every object the chart
renders, through one seaweedfs.commonLabels helper. Values may hold
templates, and the whole value may be a template string rendering to a
map, so a parent chart can pass its own labels down. The chart's
app.kubernetes.io/{name,instance,component,managed-by} and helm.sh/chart
keys are dropped, and keys an object already sets (ingress.labels,
httpRoute.labels, monitoring.additionalLabels, grafana_dashboard) win,
so selectors never change and no key renders twice.

Pod templates are left alone (podLabels covers them, and a label change
would roll the pods), as are volumeClaimTemplates, which cannot change
on an existing StatefulSet. checksum/master-config now hashes the master
ConfigMap as rendered without commonLabels, so labels do not roll the
masters either; its value is unchanged.

With commonLabels empty, the default, every template renders exactly as
before. Adds a helm_ci check that labels a release with most components
on and asserts every object carries the labels and nothing else changes.

* helm: do not reject a commonLabels key literally named Error

fromYaml reports parse failures as an {"Error": msg} map, so checking
for that key also matches a user-supplied label of the same name. Only
treat Error as a parse failure when the raw value was not already a
map, so {Error: tracked} renders while non-map input still fails.

* helm: exclude the filer monitoring label from commonLabels

filer.metricsPort adds a hardcoded monitoring: "true" label, which
collided with a monitoring key in commonLabels and rendered a duplicate
map key. Exclude it while metricsPort is set so the object's label wins,
and keep the user's label when it is not.

* helm: accept a templated commonLabels string with an Error key

fromYaml reports a parse failure as a one-key map {Error: "error ..."}, so a
templated string such as 'Error: x' was taken for a parse error. Only fail
when the value is not a map and fromYaml returned its error shape.

* helm ci: keep ConfigMap data in the commonLabels render comparison

Generated Secrets still need normalizing, but the security ConfigMap was
only non-deterministic because of its default jwt signing key. Disable
that key for this render and compare ConfigMap payloads like everything
else so a commonLabels regression in data is caught.

* helm ci: render the security ConfigMap in the commonLabels comparison

Disabling jwtSigning.volumeWrite alone makes securityConfigEnabled false,
so the ConfigMap was absent from the render instead of compared. Enable
mTLS without the generated key so the object is rendered deterministically
and both its labels and data are covered.

* helm ci: strip only generated-credential secret payloads in the commonLabels comparison

* helm ci: assert the effective non-reserved commonLabels

* helm: render integral numeric commonLabels without exponent

---------

Co-authored-by: jeffWild <20067853+jeffWild@users.noreply.github.com>
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-30 19:53:43 -07:00
Chris LuandDevin cd9f9ed7e8 deps: bump seaweedfs/raft for PreVote (leader survives partition rejoins) (#12264)
* deps: bump seaweedfs/raft for PreVote

The default raft gains the pre-vote phase so a master returning from a
partition can no longer depose a healthy leader with an inflated term.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* deps: bump seaweedfs/raft for pre-vote review follow-ups

Picks up the post-merge fixes on top of f227dce: per-round vote workers with per-peer dedupe, pre-vote grants counted at a newer term so mixed-version clusters cannot livelock, and state/conf field locking on the startup paths.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* deps: pin seaweedfs/raft to released v1.2.3

PreVote has merged and shipped as raft v1.2.3, so switch off the
PR-branch pseudo-version.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-30 19:46:52 -07:00
github-actions[bot] 82cb79ceb2 docs: regenerate star history chart 2026-10-31 00:32:30 +00:00
b7bfbaf9eb helm: add namespaceOverride to deploy into another namespace (#12256)
* helm: add namespaceOverride to deploy into another namespace

When the chart is a subchart of an umbrella chart, every object lands in
the umbrella's release namespace, with no way to give SeaweedFS its own.

namespaceOverride, empty by default, now sets the namespace of every
object through a new seaweedfs.namespace helper, which falls back to the
release namespace. All uses of .Release.Namespace go through it, so the
in-cluster addresses, certificate SANs, ClusterRoleBinding subjects,
HTTPRoute backendRefs and lookups follow the override too. The
self-signed Issuer, the bucket hook Job and the volume resize hook
objects, which had no metadata.namespace, now set it.

With namespaceOverride unset the default render is unchanged. Adds a
helm_ci check that renders with an override and asserts nothing is left
in the release namespace.

* ci: cover the bucket hook Job in the all-in-one namespace test

The all-in-one render set s3.createBuckets, which bucketHookEnabled
ignores in allInOne mode, so the Job never rendered and the namespace
check could not see it. Use allInOne.s3.createBuckets and assert the
Job is present.

* helm: fail on an invalid namespaceOverride instead of reshaping it

Truncating and trimming could silently map an override onto a different
existing namespace and deploy into it. Validate against the DNS-1123
namespace rules and fail the render instead.

* helm: bound the master data volume name at 63 characters

data-<namespace> can exceed the Kubernetes name limit when the release
or override namespace is near 63 characters; truncate the composed name
and trim a trailing hyphen like the other name helpers.

* helm: quote the rendered namespace in YAML fields

* helm: warn that changing namespaceOverride strands claims and data

* helm: say namespaceOverride only moves namespaced objects

* helm: retrigger the PR head sync (no changes)

* helm: warn that secrets and hook resources do not follow namespaceOverride

* helm: warn that hostPath data does not follow namespaceOverride

---------

Co-authored-by: jeffWild <20067853+jeffWild@users.noreply.github.com>
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
2026-10-30 11:24:36 -07:00
Chris LuandDevin 816f9356b4 volume.copy: reject only on file-count drift, warn on deleted-count (#12253)
* volume.copy: reject only on file-count drift, warn on deleted-count

Go warned and kept the copy on any count mismatch. The file count is
exact on both sides of a stable source, so a mismatch means a genuinely
bad copy and should reject; only the deleted count is a rebuild-time
estimate that legitimately drifts. Match the Rust check_copy_counts,
which warns solely on the counter that can drift.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume.copy: stage the copy and validate before dropping the old replica

The destination replica was deleted before the copy even started, so a
copy that failed its checks — now including the file-count validation —
left the server with no replica at all. The copy now writes under a
copying/ staging dir the volume loader never scans, the file-size and
count checks run on the staged .idx (LoadCompactNeedleMap gives the same
counters the mount would), and only then is the existing replica deleted
and the staged files renamed into place. A rejected or failed copy
leaves the previous replica untouched, and the .note marker is no longer
needed since partial staged files can never be loaded.

* volume.copy: remove only the files a failed copy moved into place

* volume.copy: keep the .note in-progress marker through the renames

A crash between the staged-file renames left a loadable volume without
its .vif (writable where the source was read-only) or a partial set the
load scan could not find. Write the .note before the renames and land
the .vif first so any partial state is discovered and swept, then drop
the note before the mount.

* volume.copy: serialize copies of the same volume

Two copies of the same volume write the same staged copying/ file names
and can pass the existing-volume check before the first one mounts.
Fail a concurrent second copy as retryable, matching the Rust volume
server's volume_copies_in_flight.

* volume.copy: restore the replaced replica when the swap fails

Once the staged copy validates, the outgoing replica's files are
hard-linked into copying/ before DeleteVolume unlinks them. If a later
rename or the mount then fails, the preserved files are renamed back
and remounted, so a failed swap no longer leaves the node with no
replica. The links are removed on success.

* volume.copy: clear stale needle-map files around the swap

A leftover .ldb/.sdx/.rdb at the destination base is trusted over the
fresh .idx the mount loads, and a mount that fails leaves the backends
it built. Clear them before the staged files land and again when a
failed copy cleans up, gated on no live replica owning them. Also keep
<base>.vif on cleanup while an .ecx index shares the basename.

* volume.copy: skip the swap when the caller has gone

A caller that left during the copy cannot use the result; bail before
the swap deletes the replica it would replace.

* volume.copy: discard a mounted copy when the result send fails

A mounted copy was already announced to the master, so a later failure
deletes it through the store to withdraw the announcement before the
files go away, rather than unlinking files out from under a registered
replica.

* volume.copy: keep a vif-less copy discoverable on a split index dir

With -dir.idx the staged .idx lands outside the data dir, so a copy
whose source has no .vif leaves nothing the volume scan can find and
the mount fails after the replica it replaced was already deleted. An
empty .vif marks the basename; the loader treats it as absent and
writes a real one on mount.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume.copy: preserve the replaced replica on its own disk

* volume.copy: restore the replaced replica's .vif past a kept incoming one

* test: cover the rollback link/restore helpers

* volume.copy: preserve the mounted replica's own files for swap rollback

The swap backup linked files at the copy's target paths, which are wrong
when the replica sits on another disk or keeps its .idx beside the .dat;
a failed swap then restored nothing. Link the mounted volume's real paths
and require every present file to link before the delete, since a partial
backup cannot roll back. Re-check the store at swap time so a replica
that mounted mid-copy is replaced rather than overwritten, and drop a
.kept incoming .vif that occupies the path the replica's own .vif
restores to.

* volume.copy: clear a stale .idx a mount would prefer over the copy

A load switches dirIdx to the data dir when an .idx sits beside the .dat,
so a leftover <data>.idx shadows the fresh index the copy lands in the
idx dir, and a surviving <idx>.idx fills in when the source had no index
at all. Clearing only the derived backends still let a stale needle map
reach the mount.

* volume.copy: leave the shared copying/ dir for racing copies

Copies of different volumes serialize per volume id but share the disk's
copying/ dir; a finishing copy could rmdir it between another copy's
MkdirAll and its first staged write, failing that copy with ENOENT.

* volume.copy: fail replica preservation on stat errors other than not-found

* volume.copy: refuse a mount while that volume is being copied

A copy claims the volume id, then waits on the source with the
replacement staged. A mount in that window loads the replica whose files
the swap then clears; refuse it with Unavailable, retryable once the
copy settles. Mirrors the same refusal on the Rust volume server.

* volume.copy: refuse configure while that volume is being copied

Configure unmounts and remounts; during a copy that already claimed the
volume id the remount loads files the swap then unlinks. Refuse with
Unavailable, retryable once the copy settles. Mirrors the Rust volume
server's same refusal.

* volume.copy: leave a still-standing replica's index on a rejected copy

* volume.copy: preserve an unmounted replica's files for swap rollback

* volume.copy: claim the in-flight id in mount and configure, not just check it

* volume.copy: keep an EC-shared .vif when displacing an unmounted replica

* volume.copy: restore replaced files over a surviving partial

restoreReplacedVolumeFiles deleted a preserved link whenever the base
name was still taken, on the assumption a landed file there belonged to
the copy. When a moved-file removal fails, that left a surviving partial
paired with the restored set's other files — mixed generations in one
replica.

A leftover at the basename now comes off first so the backup goes back,
and the link is kept whenever it cannot go back.

* volume.copy: link a repeated replaced path once so rollback cannot delete it

* volume.copy: refuse allocation and staged adoption under the copy claim

AllocateVolume and adoptStagedVolume still registered a volume while a
copy of it held the destination: NewVolume loads the index before
registration, and the copy then truncates or replaces files the live
replica already opened, leaving it mounted on a stale index.

Both paths now claim the volume id like VolumeMount and VolumeConfigure,
so registration either wins the claim before the copy starts or sees it
and refuses with Unavailable.

* volume.copy: serialize reload loads with the copy claim

A SIGHUP reload ran loadExistingVolume outside the copy's claim: its
.note branch could unlink a copy's files mid-write, and its
NewVolume+SetVolume could register a replica between the copy's recheck
and its swap, leaving a live volume on a stale index.

loadExistingVolume now consults a per-location hook the server installs:
a vid whose claim is held is skipped for the next pass; otherwise the
load takes the claim and its release drops it after registration, so
whichever side claims the id first owns the files for the operation.

* volume.copy: let a claim holder's own mount past the load guard

VolumeMount, VolumeConfigure, VolumeCopy, and adoptStagedVolume all hold
the per-volume claim while calling store.MountVolume, which reaches
loadExistingVolume where the load guard saw the caller's own claim and
skipped the disk: every mount failed with 'volume not found on disk'.

Plumb claimHeld through MountVolume -> LoadVolume -> loadExistingVolume
via the new MountVolumeClaimed entry point so an owner's own load
proceeds while loads without the claim are still skipped.

* volume.copy: keep the strict count check when only deletes drift

Deletes append tombstone rows without moving the file count, so a copy whose source only deleted mid-flight still has an exactly comparable file count. Gate the strict check on file-count stability alone; the deleted-count drift is already a warning inside checkCopyCounts.

* volume.copy: abort the swap when stale index files cannot be removed

A surviving .ldb/.sdx/.idx is trusted over the copied .idx at mount, so
a failed removal must stop the copy before the staged files land, not be
swallowed. Mirrors the Rust side's remove_partial_files abort.

* volume.copy: clear staged leftovers before starting a copy

The staging names are fixed per volume, so a crashed copy can leave
files under copying/; a retry whose new source lacks an optional file
(.vif, or .dat on a remote volume) would rename the previous run's
bytes into place. Clear them up front and refuse the copy when they
cannot be removed.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-30 11:22:40 -07:00
80aad76986 rust volume: remove the redb index when a failed VolumeCopy cleans up (#12250)
* rust volume: remove the redb index when a failed VolumeCopy cleans up

A copy that mounts and then fails validation is unmounted, not deleted. With
a redb needle map the mount has already built <idx base>.rdb, and
cleanup_failed_copy removed .dat/.idx/.vif/.note but not the .rdb, so the
index database outlived the copy. A later copy of the same volume to that
server reuses it (RedbNeedleMap::try_reuse_rdb keeps an .rdb whose recorded
idx size is not larger than the new .idx and replays only the tail), which is
wrong once the source has been compacted in between.

Move the partial-file removal into CopyDestination::remove_partial_files and
include the .rdb, next to the .idx where Volume::file_name puts it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq

* rust volume: clear a stale index before a copy writes, and report cleanup failures

A crashed copy, or one whose partial-file removal failed, leaves a stale
.rdb/.sdx next to the .idx; the next copy's mount would reuse it via
try_reuse_rdb's tail replay. remove_partial_files now also covers .sdx
and returns the first non-NotFound error: callers that can still abort
(the pre-copy clear) do so, while the post-failure cleanup warns so a
surviving index is visible instead of silently poisoning the next copy.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: restore the pre-copy leftover clear

A leftover index check-in accidentally gated remove_partial_files behind
a constant false, so a stale .rdb/.sdx at the destination survived the
copy that was meant to clear it.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(volume): expect 409 when a read-only upload is refused

x-seaweed-write-outcome: refused-before-append. The HTTP integration
test still expected 500, so Rust unit tests failed on Linux and Windows.
Deletes stay 500.

* rust volume: keep a shared .vif, and clear a stale index beside the .dat

remove_partial_files deleted <data>.vif even when a sealed .ecx still
shares it, undoing the keep in Volume::destroy. It also cleared the
index only under the index base, so a leftover .idx beside the .dat
still made load trust that directory's .rdb.

Skip .vif while a non-empty .ecx is present, and clear both bases when
the index directory differs from the data directory.

* rust volume: do not delete .dat files in the index directory

The index directory can be another location's data directory. Clearing
.dat and .note there removes that replica. Those files come off the
data base only; .idx, .rdb, and .sdx are still cleared on both.

* rust volume: serialize copies of a volume on one server

A second VolumeCopy of the same volume could pass the existing-volume
snapshot before the first copy mounts, then clear or overwrite the
files the first copy still owns. Hold a per-volume claim for the whole
copy, like ec_decodes_in_flight does for decodes, and fail a duplicate
fast so the caller can retry. Mirrors the Go volumeCopiesInFlight
guard.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: keep files when the mounted-copy delete is refused

When a copy mounted and then failed, deleting through the store can be
refused (e.g. the volume is compacting), leaving it registered. The
cleanup then unlinked its files anyway, stranding a live replica. Keep
the files while the volume is still in the store. Mirrors the Go copy
cleanup.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: init volume_copies_in_flight in the http test state

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: refuse a mount while that volume is being copied

volume_copy claims the vid, snapshots the store, then waits on the
source. A VolumeMount in that window can load the replica, and the
copy's pre-write cleanup then unlinks its files. Check the claim under
the store write lock and return Unavailable.

* rust volume: refuse configure while that volume is being copied

volume_configure unmounts and remounts. A copy that already snapshotted
the volume as absent can then unlink the files that remount loaded.
Check the copy claim under the store write lock and return Unavailable.

* rust volume: preserve files a copy displaces so a failed copy restores them

* rust volume: keep an interrupted copy's .note with the files it marks

displace_existing_files skipped the .note, so a crashed copy's leftovers
were saved without the marker the load scan rejects them by, and
remove_partial_files then deleted it. A retry that failed restored the
partial set marker-free: after a restart the server could load an
unfinished copy as a normal replica and serve wrong data.

Displace the .note like every other file at the basename so the restore
puts back the exact state the copy found.

* rust volume: restore displaced files over a surviving partial

restore_displaced_files dropped the backup whenever the basename was
still occupied. When the partial cleanup could not remove every file the
copy landed, an existing path just meant the removal failed — deleting
the saved file destroyed the old replica and let the surviving partial
pair with restored siblings of another generation.

Clear the leftover first so the displaced file goes back, and keep the
backup when it cannot: the operator can recover from copying/ rather
than lose the replica outright.

* rust volume: back up an EC-shared .vif the copy truncates in place

A .vif a sealed .ecx shares was left out of the displaced set so the EC
volume keeps it — but the incoming copy opens the same path with
truncate, so a failed copy left the shared .vif overwritten by the
rejected copy's settings with no way back. Go keeps the original via a
hard link because its staged file swaps in by rename; truncating the
shared inode makes a link useless here.

Copy the content into copying/replaced_ like the other backups instead:
the file stays in place for the EC volume during the copy, and the
restore puts the original settings back on failure.

* rust volume: serialize reload loads with the copy claim

A SIGHUP reload scans every volume basename and sweeps .note-marked files
as interrupted copies — including one a live VolumeCopy is still writing.
Mirror Go's loadGuard: the copy-claim set is shared with each disk
location, the scan skips a claimed vid outright and claims any other
through its load, and reconcile_compact_states takes the same skip before
rolling a leftover commit over the copy's files.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: hold the load claim through open_volumes

The scan released each claim when its iteration ended, before the queued
volume's index read ran in open_volumes. A VolumeCopy taking the vid in
that gap could displace or remove files out from under Volume::new — the
register the guard exists to prevent. Queued claims now live until the
open pass returns; a second collection of an already-claimed vid merges
into the entry instead of re-claiming.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: refuse allocation while that volume is being copied

AllocateVolume registered a volume while a copy of it held the
destination: create_volume loads the index before registration, and the
copy then clears or rewrites files the live replica already opened,
leaving it mounted on a stale index — the window VolumeMount and
VolumeConfigure already refuse.

Claim the volume id like the Go AllocateVolume does, so registration
either wins the claim before the copy starts or sees it and refuses
with Unavailable.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: refuse staged adoption while that volume is being copied

The from_staged branch of volume_ec_shards_to_volume renames .copying
files into place and mounts, all while a copy of the same vid could be
rewriting the destination — the same registration window VolumeMount
refuses. Hold the copy claim for the whole adoption, as Go's
adoptStagedVolume does.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: drop a partial .note when its write aborts the copy

A write_note failure after displace_existing_files restores the saved replica but leaves the empty or partial marker it just made; the next load scan then deletes the restored replica as an unfinished copy. Remove the marker before restoring, as the Go side's error defer does.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: chrislusf <chris.lu@gmail.com>
2026-10-30 11:21:38 -07:00
43a312e309 rust volume: keep a copied replica on deleted-count drift, as Go does (#12251)
* rust volume: keep a copied replica on deleted-count drift, as Go does

check_copy_counts rejected a VolumeCopy whenever the target's deleted count
differed from the source's. On a redb needle map that count is an estimate:
metrics_from_idx rebuilds it at load through a bloom filter (0.1% false
positives per entry, each counted as an overwrite), and the source holds the
estimate from its own last load plus exact runtime updates. A source written
since its load and a fresh load of the same files disagree by a few, so good
copies were refused with "target deleted count [N+1] is not same as origin
deleted count [N]". An index of unique keys 1..=128 already reports one
deletion.

Go fixed the same rejection in #11701 (issue #11678) by warning instead of
failing. Do the same for the deleted count. The file count stays strict: it
is exact on both sides (every non-tombstone .idx row is one file), and the
.dat/.idx size checks still gate integrity.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq

* test(volume): expect 409 when a read-only upload is refused

x-seaweed-write-outcome: refused-before-append. The HTTP integration
test still expected 500, so Rust unit tests failed on Linux and Windows.
Deletes stay 500.

* rust volume: keep the strict count check when only deletes drift

Deletes append tombstone rows without moving the file count, so a copy whose source only deleted mid-flight still has an exactly comparable file count. Gate the strict check on file-count stability alone; the deleted-count drift is already a warning inside check_copy_counts.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
2026-10-30 11:20:33 -07:00
Chris LuandDevin cc04c805a4 volume.copy: drop the needle-map index a failed copy leaves behind (#12252)
* volume.copy: drop the needle-map index a failed copy leaves behind

A VolumeCopy that mounts and then fails validation is only unloaded, not
deleted, so a leveldb (.ldb) or sorted-file (.sdx) index built by the
mount outlives the copy. The next copy of the same volume to this server
then trusts the stale index (generateLevelDbFile replays only past its
watermark), which is wrong once the source has compacted in between.
Mirrors the Rust server's remove_partial_files.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume.copy: clear a stale index before a copy writes

A crashed copy, or one whose cleanup failed, leaves a stale .ldb/.sdx/.rdb
next to the .idx that this copy's mount would trust (generateLevelDbFile
replays only past the stored watermark, and a same-size .idx keeps every
un-replayed offset). Abort on a removal error rather than risk a mount
reusing a poisoned index. Mirrors the Rust server's pre-write
remove_partial_files call.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume.copy: keep a mounted replica when the final response send fails

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume.copy: discard a mounted replica the caller abandoned

A send failure after MountVolume means the caller is gone, but the
replica was already announced to the master. Deleting its files under
the live mount breaks the loaded index; leaving it mounted keeps an
orphan the master routes to. Mirror the Rust copy cleanup: track the
mount, and on error delete through the store (unmount + withdraw the
announcement + remove files, keeping remote data), and check the stream
context before mounting.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume.copy: serialize copies of a volume on one server

A second VolumeCopy of the same volume could pass the existing-volume
check before the first copy mounts, then clear or overwrite the files
the first copy still owns. Mirror ecDecodesInFlight: the claim covers
the whole copy, and a duplicate fails fast so the caller can retry.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume.copy: remove only files the copy owns, and every stale index

Three cleanup gaps around which files a copy may remove:

- The failure cleanup ran on any error, so an abort before the first
  write (a failed stale-index clear or a failed .note write) could
  delete a previous replica's files the copy had not touched. File
  removal is now gated on the copy having started writing.
- A mount that finds an .idx beside the .dat builds the needle-map
  index under the data base, so leftovers there poison the next copy
  the same way the idx-dir ones do. Clear and clean up .idx/.ldb/
  .sdx/.rdb under both bases when the dirs differ.
- A regular volume and an EC volume for the same id share <base>.vif.
  Keep it while the EC index is on the disk, as the destroy path does.
- When the discard of a mounted copy is refused (e.g. it is compacting)
  the volume stays registered; unlinking its files would strand a live
  replica.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume.copy: do not unlink a previous replica on an early abort

An abort before the first write now leaves the destination files alone,
but three removal paths still leaked past that gate: a partially-written
.note, and the post-copy defer's .idx/.vif at the data base. Remove the
.note unconditionally (it only ever marks an in-flight copy) and gate the
data-file removals in the outer defer on copyOwnsFiles too.

* volume.copy: refuse a mount or configure while a copy is in flight

A copy claims the volume id, then waits on the source before it clears or
overwrites the destination files. A mount or configure landing in that
window loads the replica whose index files the copy is about to remove.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume.copy: abort when a replica registers during the source RPCs

The existing-volume snapshot runs before the copy calls the source, so a
mount that lands in between escapes it. Recheck registration just before
the destructive clear, which can share the index directory with the
mounted replica's bases.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume.copy: keep files whenever a replica of the volume stays registered

The cleanup already kept files after a refused delete; a replica a racing
mount registered during the copy needs the same protection, since its
index directory can coincide with the bases being cleaned.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume.copy: skip the index cleanup when the copy never picked bases

An early failure leaves the base names empty, so the removals would
otherwise run against ".ldb"/".sdx"/".rdb" in the process directory.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume.copy: serialize mount and configure registration under a per-volume guard

The in-flight flag check in VolumeMount and VolumeConfigure was not
atomic with volume registration, and the copy's registration recheck was
not atomic with its stale-index clearing. A mount that passed the check
just before the copy claimed the volume could register after the recheck
and then lose its index files to the clearing.

Replace the bare flag with a per-volume guard: the copy claims it for
the copy's life, mounts and configures hold it across check plus
registration, and the copy holds it across the recheck plus clearing.
Whichever side takes the guard first wins, so a mount either registers
before the recheck (and aborts the copy) or sees the claim and refuses.
Guards persist in the map so a mutex handed to a waiter is never
replaced mid-wait.

* volume.copy: refuse allocation and staged adoption under the copy guard

AllocateVolume and adoptStagedVolume still registered a volume while a
copy of it held the destination: NewVolume loads the index before
registration, and the copy then truncates or replaces files the live
replica already opened, leaving it mounted on a stale index.

Both paths now take the per-volume guard through their registration, so
they either register before the copy's recheck (and make it abort) or
see the claim and refuse with Unavailable.

* volume.copy: serialize reload loads with the per-volume guard

A SIGHUP reload ran loadExistingVolume outside the copy's guard: its
.note branch could unlink a copy's files mid-write, and its
NewVolume+SetVolume could register a replica between the copy's recheck
and its file clearing, leaving a live volume on a stale index.

loadExistingVolume now consults a per-location hook the server installs:
a vid with a copy in flight (or whose guard the copy currently holds) is
skipped for the next pass; otherwise the returned release keeps the
guard held through registration, so the copy's recheck either sees the
volume and aborts or clears first.

* volume.copy: skip compaction-state reconcile while a copy owns the volume

* volume.copy: let a guarded operation's own mount past the load guard

VolumeMount, VolumeConfigure, adoptStagedVolume hold the volume guard
locked, and VolumeCopy holds copying=true, while calling store.MountVolume.
The mount reaches loadExistingVolume where the new load guard saw the
caller's own hold (TryLock fails, or copying set) and skipped the disk:
every guarded mount failed 'volume not found on disk' — a completed copy
deleted its copied files, configure left the volume unmounted, staged
adoption failed after renaming.

Plumb ownLoad through MountVolume -> LoadVolume -> loadExistingVolume via
the new MountVolumeClaimed entry point so the owner's own load proceeds
while unrelated loads are still skipped.

* volume.copy: fix own-load test call, assert unguarded mounts still skip

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume.copy: refuse unmount and delete while a copy owns the volume

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-30 09:20:55 -07:00
Chris LuandDevin ad6257ebbc filer.backup: add -s3.flattenVersions to mirror the current S3 view of versioned buckets (#12255)
* filer.backup: add -s3FlattenVersions to mirror the current S3 view

Versioned buckets are stored as <key>.versions/v_<versionId> files plus a
latest-version pointer on the .versions directory, so a backup mirror held
internal version files and could never be restored through the S3 API.

With -s3FlattenVersions the backup maps events on .versions directories
onto the flattened object key: a new latest version uploads to <key>, a
latest delete marker or a cleared pointer deletes <key> when
-doDeleteFiles is set, and the suspended-versioning null object resolves
to the regular path. Version files themselves are never mirrored. The
initial snapshot seeds only the version the pointer names.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer.backup: serialize flattened writes per object key

Same-path non-barrier dir updates run concurrently in the metadata
processor, so two pointer updates for one object -- or a pointer update
and a regular-path write to it -- could overlap and let the older upload
finish last. Lock on the flattened key: versions events lock the object
key, every other event its own path, via a striped mutex.

* filer.backup: apply exclusions to the flattened object key

File-name and path excludes ran against the .versions entry, so an
excluded object was still copied flat and its deletion still removed the
destination copy. Match the rules against the flattened key on both the
apply and the versions-dir-removal paths.

* filer.backup: re-resolve the pointer on version-file events

A tag or attribute update on the current version touches the v_* file,
not the .versions pointer, so dropping every version-file event left the
mirror with stale tags until the next pointer flip. Re-resolve the
pointer on each version-file event; noncurrent versions still stay out.

* filer.backup: tolerate vanished versions dirs in flattened snapshot

The flattened seed path returned source errors before the existing
missing-source policy ran, so one version deleted mid-walk could force a
full re-walk. Apply the same 404/superseded skip as the generic seed
branch, and cover the review fixes with tests.

* filer.backup: lock both flattened keys an event touches

A rename or delete can retire one flattened key while writing another,
so serialize on the old- and new-side keys together.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer.backup: refresh the flattened key only for the still-current child

A v_* event on a non-pointed version cannot move the flattened key, so
skip the re-upload; tag and attribute updates on the current version
still flow through.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* replication: let resolved-latest writes bypass the source staleness check

A filer sink skips a CreateEntry whose source path no longer holds the
entry, but a flattened write lands on a key whose source path is not the
one it was read from, so the check dropped every flattened upload.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer.backup: reclaim destination chunks on flattened deletes

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer.backup: route regular-path events for versioned keys through the pointer

A null-object write or delete at the regular path can coexist with a
.versions pointer that names a live version; the pointer decides the
flattened key, so deleting the null object must not remove it.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer.backup: make the checkpoint identity cover the flattening mode

Resuming a checkpoint taken in the other mode would skip the flattened
re-seed and leave mirrored version files behind; a mode change now
starts a fresh snapshot.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test: cover flattened regular-path routing and per-mode checkpoints

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer.backup: resolve both sides of a flattened rename

* test: cover flattened renames with one versioned side

* replication: let resolved-latest writes overwrite a newer destination

The stale-destination skip trusted a newer destination mtime, but a
resolved-latest write can legitimately install an older version over the
copy of a since-deleted latest. Skip only on provably identical content
(matching chunk provenance, or equal ETag with full byte coverage).

* filer.backup: rescan .versions when the latest null object is gone

Deleting ?versionId=null removes the regular-path object without touching
the null-is-latest pointer, so the flattened key kept its stale copy.
Fall back to the newest remaining version (or a flat delete), the same
rescan the S3 read path applies.

* replication: compare inline and remote content in the destination-holds checks

A chunk-less entry keeps its bytes in Content and a remote-only entry in
the object RemoteEntry points at, but the identical-content shortcuts
only compared chunk lists and chunk-derived ETags. Two different inline
files looked identical, so a flattened re-copy or a same-mtime rewrite
kept the previous version's bytes. Compare Content bytes and remote
object identity wherever the checks claim the destination already holds
the reference's content.

* filer.backup: paginate the null-latest .versions rescan

A limit=0 listing is capped at DirListingLimit server-side, and
old-format version names sort oldest-first, so a history longer than one
page left the flattened copy on an older version after the null object
was removed. Page through the directory like the S3 read path does.

* replication: require matching writable metadata for resolved-latest skips

A skip keeps the destination's stored metadata, so identical bytes were not
enough: stale attributes or extended entries (e.g. a superseded version id)
survived under equal content. Compare the fields a rewrite controls —
attributes minus filer-assigned inode/crtime, with the store's atime/ctime
zero-defaults applied — plus the extended map, in both the provenance and
ETag fast paths.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* replication: require the extended key to exist when comparing metadata

* replication: ignore a stored rule TTL when the resolved write carries none

* filer.backup: rename -s3FlattenVersions to -s3.flattenVersions

Match the -s3.* flag naming convention used by the other s3 options.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-30 09:09:05 -07:00
Chris LuandDevin 1a62b1d09f filer: buffer recently fetched chunks so parallel readers do not refetch (#12254)
* s3sink: upload through a concurrent-safe ReaderAt so parts do not thrash

s3manager splits a seekable body into parallel section readers, and every
part's ReadAt landed on the single chunk buffer in ChunkStreamReader: parts
at different offsets kept evicting each other's chunk and refetching it
whole, which produced massive read amplification against the volume
servers and collapsed throughput.

Build the upload body from filer.ChunkReadAt instead, the same
concurrent-safe reader the mount and query paths use: each part's range is
served from a shared reader cache with parallel chunk fetches and
range-reads for clipped views, so every byte is fetched once per pass.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: buffer recently fetched chunks for concurrent readers

* Revert "s3sink: upload through a concurrent-safe ReaderAt so parts do not thrash"

This reverts commit be1ec2d7c7.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-30 08:58:48 -07:00
Chris LuandDevin 8b0bd48755 seaweed-volume: pin the read-only write status to 409 in the test (#12244)
* seaweed-volume: pin the read-only write status to 409 in the test

PR #12236 moved read-only upload refusals from 500 to CONFLICT with the
refused-before-append proof; the test still asserted Go's 500 for the
write. Deletes on a read-only volume keep surfacing 500.

* seaweed-volume: correct the read-only status comment in the test

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-30 08:55:53 -07:00
Chris Lu a1f2886647 vector: accept Go-style single-dash long flags (#12247) 2026-10-30 03:44:25 -07:00
Chris Lu 452306776f Delete test/s3vectors/README.md 2026-10-30 03:24:39 -07:00
Nguyễn Lộc Phúcandchrislusf eba2cb926d fix(s3tables): refuse a create that a namespace delete can lose (#12239)
* s3tables: refuse a create that a namespace delete can lose

DeleteNamespace checks the namespace for children and then deletes its
directory recursively in a second filer call. The filer locks each path
on its own, so a table created under the namespace between the two is
not seen by the check and is removed by the delete, while both requests
answer 200. The mirror case loses the table too: a create that passed
its namespace check before the delete finished re-created the namespace
directory through the filer's parent creation and recorded a table in a
namespace that no longer exists.

Each side now writes before it reads. DeleteNamespace claims the
namespace record with a fresh token before it scans, deletes only while
the record still holds that claim, and lifts its own claim when it
stops. CreateTable, RegisterTable and CreateView re-read the namespace
after writing their record and back out when it is claimed or gone:
they remove what they wrote and answer 409 or 404 instead of reporting
an entry that does not survive, or 500 when the record could not be
removed and may remain. An entry the scan missed was written after the
claim, so its create sees the claim; a create and a delete can both
fail, but no longer both succeed. A create is also refused up front
while a claim is held, but only after its authorization check, so a
caller that may not create there does not learn of the delete.

A create writes its record marked "create" and lifts the mark once the
namespace is confirmed open. Until then GetTable, GetView, UpdateTable
and a second create treat the record as absent, so no request is told
about a table or view that the first create may still back out.

* s3tables: keep a create's unfinished record out of reads and listings

* s3tables: refuse catalog writes on a record an operation claimed

* s3tables: answer 404 when a create's write path loses the namespace

* s3tables: confirm the rename destination's namespace like a create

* s3tables: cover unfinished-record reads, claimed-record writes, rename claim

* s3tables: lift a leftover claim when a no-token delete clears the record

* s3tables: answer 409 when a claimed record turns away a write

UpdateView, TagResource, and UntagResource surfaced ErrConcurrentUpdate
as a 500, so an expected conflict looked like a server fault to the
caller. Map it to Conflict the way UpdateTable and the policy handlers
already do.

* s3tables: name the lost source when a rename finds it deleted

* s3tables: report an entry deleted before its update as not found

The filer answers an UpdateEntry on an entry removed since the lookup
with codes.NotFound. mutateEntryExtended returned that error unchanged
instead of filer_pb.ErrNotFound, so a rename whose source was deleted
mid-claim answered 500 instead of NoSuchTable or NoSuchView.

MemFiler now answers a missing entry the same way, so the rename tests
for tables and views take the real filer's path.

* s3tables: unpublish the destination when a rename's source clear fails

* s3tables: refuse to delete a record another operation claimed

* s3tables: release catalog claims on an uncanceled cleanup context

* s3tables: answer 404 when a racing delete already removed the table

* s3tables: keep the rename destination marked until the source clears

* s3tables: guard the rename rollback against a recreated source and an unfinished destination

* s3tables: drop the unfinished destination only once the source is restored

* s3tables: re-resolve the purge target on the record a delete claims

* s3tables: cover a delete that meets a moved metadata location

* s3tables: answer 409 when a create's record changed under it

* s3tables: adopt a record still marked by its unfinished create

* s3tables: skip unfinished creates in the namespace empty check

* s3tables: delete only the record the caller was authorized on

DeleteView and DeleteTable check permissions on the record they read
first, then read the name again to claim it. A record that took the name
in between, such as a table or another owner's view created after a
concurrent delete, was claimed and removed on the first record's
authorization, and DeleteView never compared the caller's version token
with it.

The second read now repeats the first one's checks: a record of the
other kind or an unfinished create answers 404, and a token that no
longer matches answers 409. A delete without a token also compares the
record's version token with the authorized one and answers 409 when they
differ.

* s3tables: re-check the parent after a create lifts its mark

A create that passed checkParentOpen and paused before lifting its mark
could still confirm while a namespace delete held the claim: the delete
scans the still-marked record, skips it, and removes it under a create
that already answered success. confirmParent now re-reads the same
parent record after the mark clears and backs the confirmed record out
when the namespace moved on.

* s3tables: mark the rename destination with a claim of its own

claimCatalogEntry adopted every create-marked record, including a
rename's destination: a create landing after the source cleared could
replace the destination before the rename published it, and with the
source name reused the restore could leave the table on neither name.
The destination now marks with the rename's own claim, which readers
still treat as unpublished but a create must refuse, and the commit
re-checks the destination namespace after the record goes live. The
source keeps its rename claim until the commit lands so a namespace
delete scanning the cleared record still refuses.

* s3tables: compare the parent's createdAt, not its editable metadata

A namespace update rewrites ExtendedKeyMetadata in place, so a create or
rename whose second parent read landed across an UpdateNamespace saw a
changed blob and reported the still-live namespace as gone: creates backed
their record out and answered 404, renames undid their destination.
createdAt survives the edit but changes on a delete+recreate, so it still
identifies the record.

* s3tables: retry lifting the rename's claim off the cleared source

The release's only failure mode is a transient filer error — nothing else
can write the cleared record — yet a single loss left the claim behind:
the old name stayed refused to creates and the namespace counted the
record as a child. Retry the release before warning.

* s3tables: page the namespace scan before deleting it

A page of skippable entries — bare directories or unfinished creates —
ended the scan with hasChildren false, so a table on a later page was
removed with the namespace. Keep listing with StartFromFileName until a
child shows or the listing ends.

---------

Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
2026-10-30 03:18:52 -07:00
Chris Lu e80430db8e test: add s3 integration tests for SQLite Cloud Backed Storage (#12243)
* test: add s3 integration tests for SQLite Cloud Backed Storage (#12235)

SQLite CBS stores a database as named objects in a bucket and relies on
conditional PUT — put-if-absent for new blocks and ETag compare-and-swap
on the manifest — so concurrent clients cannot corrupt the database.

The suite runs a weed mini cluster (conditions are evaluated under the
object's write lock on the owner filer via the routed-write path) and
covers:

- the CBS storage primitives: create/put/fetch/list/delete/destroy
- If-None-Match: * put-if-absent, success and 412 rejection
- If-Match ETag compare-and-swap, including stale-ETag rejection and
  the re-read/retry loop CBS clients run on 412, on plain and
  versioned buckets
- racing put-if-absent and manifest CAS writers: exactly one commit
  wins, every update survives
- conditional CompleteMultipartUpload, sequential and raced
- an end-to-end bcv-style upload, download-and-verify, and checkpoint

A matching job is added to the S3 mutation regression workflow.

* test/s3: drop issue references from the sqlite cbs suite comments

* test/s3: abort every losing upload after the conditional multipart race

* test/s3: assert the winning writer's body after the conditional write races

* test/s3: spell out the get-returned etag in the cbs suite

* test/s3: assert the completed multipart ETag the cbs suite feeds back

The conditional-completion checks pass completeOut.ETag into If-Match;
if the response carried no ETag the check would silently run
unconditional.

* test/s3: drop the unused generation parameter from the cbs block body
2026-10-30 03:17:16 -07:00
Chris Lu 12dac68d66 seaweed-volume: replace deprecated AtomicU64::fetch_update with try_update 2026-10-30 01:37:42 -07:00
Chris Lu 56c081f90e test/s3vectors: data-plane lifecycle integration test (#12242)
* test/s3vectors: data-plane lifecycle integration test

TestVectorDataPlaneLifecycle spawns a real weed-vector against the lance
listener, mirrors the pytest cluster's flags, waits for the gateway to
discover the registered engine, and walks the full lifecycle: create
bucket and index, put/get/list vectors, a filtered query that excludes the
nearer non-matching vector, delete vectors, index, and bucket. The test
skips when no engine binary resolves (VECTOR_BINARY, ./weed-vector, PATH).

* test/s3vectors: fail fast when the engine dies before listening

* test/s3vectors: harden engine launch and tighten lifecycle assertions

* test/s3vectors: wait on the engine's exit once through a broadcast channel
2026-10-30 01:34:02 -07:00
Chris Lu ee7e4a64d2 s3vectors: record request metrics on the dispatch path (#12241)
* s3vectors: record request metrics on the dispatch path

The S3 API tracks every operation through S3RequestHistogram,
S3RequestCounter, and S3InFlightRequestsGauge; the vectors dispatch rode
the same listener with no instrumentation. The op name labels each
series under s3vectors.<Operation>.

* s3vectors: count auth rejects and keep bucketless metrics alive
2026-10-30 01:27:23 -07:00
Akshat Vikram Singhandchrislusf e9f0d3ad2c helm: set resources on the worker-lance container (#12240)
* helm: set resources on the worker-lance container

Signed-off-by: AkshatIIITD <akshatvikramsingh@gmail.com>

* helm: Updated worker-deployment according to bot input

Signed-off-by: AkshatIIITD <akshatvikramsingh@gmail.com>

* helm: drop trailing whitespace in values.yaml

---------

Signed-off-by: AkshatIIITD <akshatvikramsingh@gmail.com>
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
2026-10-30 01:15:36 -07:00
Chris LuandDevin 72ad975803 s3vectors: bound GetVectors and DeleteVectors key lists (#12238)
* s3vectors: bound GetVectors and DeleteVectors key lists

Both forwarded req.Keys to the engine unbounded; AWS caps these at 500,
the same window a PutVectors batch shares.

* s3vectors: cap GetVectors at the AWS 100-key limit

GetVectors allows at most 100 keys per request while DeleteVectors
allows 500, so give vectorKeysBounds the per-call maximum instead of
sharing the put-batch window.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3vectors: assert out-of-window key lists never reach the engine

Record get/delete calls on the fake engine and cover both handlers
rejecting empty and oversized lists with 400 ValidationException.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-30 00:50:55 -07:00
Chris Lu a2496ad7ae ci(vectors): split e2e into per-volume legs, build only needed binaries 2026-10-29 23:39:54 -07:00
Chris Lu 94e9b8a49a fix(volume): return conflict and explicit proof for read-only upload refusals (#12236)
* fix(volume): report read-only HTTP refusals in Rust

* fix(volume): mirror HTTP write refusal proof in Go

* test(volume): cover HTTP read-only refusal parity

* test(volume): cover local refusal after remote append

* fix(volume): honor decoded replica controls before retry proof

* fix(operation): reassign proven HTTP conflict refusals

* fix(operation): stop same-target retries on HTTP conflict

* fix(volume): replace replica marker when forwarding requests

* fix(java): gate conflict reassignment on refusal proof

* fix(volume): preserve authentication query encoding in replication

* fix(operation): stop unproven conflicts in forever retries
2026-10-29 23:33:56 -07:00
665659097d helm: mount existingClaim data and logs volumes for filer and master (#12234)
filer.data, filer.logs and master.logs accept type existingClaim and render
the pod volume for it, but the volumeMount (and, for logs, -logdir) only
covered hostPath, persistentVolumeClaim and emptyDir. With
filer.data.type=existingClaim the claim was attached to the pod but never
mounted, so the filer kept its metadata store on the container's own
filesystem and lost it on every restart. admin and worker already include
existingClaim in the same conditions.

Adds a helm_ci check that renders both StatefulSets with existingClaim and
asserts the mounts and -logdir.

Co-authored-by: Pat White <5015863+patwhite@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 23:30:42 -07:00
9ee8604ff3 webdav: answer 204 to a PUT that replaces a file (#12233)
* webdav: answer 204 to a PUT that replaces a file

* webdav: decide the 204 from what OpenFile actually replaced

A Stat before the write left a window where a delete or create in
between made the status wrong, and it cost a second filer lookup. The
OpenFile create path already knows whether it removed an existing
entry, so it now marks the request context and the writer maps 201 to
204 only on a real replace.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test/webdav: cover OpenFile's own put-outcome flag for both create paths

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* webdav: take the put outcome from the create's OldEntry

A stat-then-create still races: a second writer landing between the
check and the write makes the reported status describe a state the
write never saw. The filer decides create-vs-replace under its per-path
lock and reports the previous entry in the response's metadata event,
so flag the PUT replace from OldEntry instead of a pre-write stat. The
in-place update keeps the replaced entry's crtime and inode and reuses
the filer's own chunk cleanup, so the separate removeAll goes away, and
an exclusive create now defers to the filer's atomic O_EXCL instead of
a racy local check.

* webdav: send no body with the 204 a replacing PUT answers

---------

Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-29 23:14:07 -07:00
Chris LuandDevin cba409cae7 backup: carry index-only tombstones through incremental copies (#12231)
* volume server: add idx_content to VolumeIncrementalCopyResponse

A remote-tier delete is an index-only tombstone row with no .dat record,
so it cannot reach an incremental copy through the .dat stream. Carrying
the .idx tail verbatim lets the backup reproduce those rows at their
exact positions.

* backup: carry index-only tombstones through incremental copies

A remote-tier delete leaves an offset-0 tombstone in .idx with no .dat
record. The copy only streamed .dat bytes and regenerated .idx from
them, so the tombstone never reached the backup: the deleted needle
stayed live and every later incremental diverged and rebuilt again.

backupCursor and ResumeByLastEntry now accept an index-only tombstone
as a boundary, matched by key and size since there is no record to read
a cookie from. VolumeIncrementalCopy appends the .idx tail after the
.dat tail, and IncrementalBackup writes it at the index boundary,
truncates, and loads the new rows — so tombstones land at the exact
positions the replica has them.

* seaweed-volume: mirror index-only tombstone handling in incremental copy

resume_by_last_entry accepts an offset-0 tombstone boundary matched on
key and size, skips its missing cookie check, and resumes .dat streaming
at the next entry that has a record — index-only rows ride the new
idx_content stream instead of reporting divergence. The server captures
the .idx bound before the .dat one (a row is published only after its
bytes) and streams the tail after the data.

* backup: keep incremental copy bounds and streams consistent

Review fixes:

* Bound capture moved after the resume reads and reads .dat/.idx sizes
  together (Go FileStat, Rust file_stat under one lock), so every row
  resume saw is inside the index bound and each row's record is always
  inside the data bound.
* A compaction commit changes CompactionRevision: the server now rejects
  the copy when the revision moved between resume, file opens, or the
  end of the stream, so a backup never mixes revisions. The .idx is
  opened up front to pin the inode.
* An index tail that ends before the captured bound is an error, not a
  silent success.
* IncrementalBackup rolls the .idx tail and .dat bytes back to the
  pre-copy boundary when the stream fails, so a partial row cannot wedge
  backupCursor and unapplied rows cannot pass for current ones.
* The index tail is replayed through a small indexTailReceiver interface,
  covering the leveldb needle maps too, not just the in-memory map.

* backup: bound the resume scan to the captured index size

The resume and the stream bounds are now captured in one order:
revision, then the .dat/.idx pair, then the scan. ResumeByLastEntry and
resume_by_last_entry take the index bound and scan no further, so a row
appended while the copy is being set up can never enter the index
stream without its .dat bytes.

The index-tail replay stays on the in-memory map only: the leveldb
map's DoOffsetLoading writes to the .cpldb compaction database, not the
live one, so it cannot serve a live tail replay; other maps keep
regenerating the index from .dat as before.

* seaweed-volume: guard incremental copy total against a located offset past .dat

A corrupt index can place the resume offset beyond the captured .dat
size; has_data already refuses the reader, but total still evaluates
dat_size - s and underflows.

* backup: bound the incremental .dat tail by index-covered records

A record staged but not yet indexed can sit inside the captured .dat
bound while its row stays outside the index bound — the fsync gap
between staging and publishing is not covered by one stat. Streaming
those bytes leaves them unindexed on the backup, and the next run
appends the record again. Bound the .dat tail by the last record the
streamed rows cover on both producers, and cover the bound walk with
tests.

* backup: cap the .dat stream at the captured bound and roll back replay failures

sendFileContent could read past stopOffset on its last chunk, so a
needle appended after FileStat arrived without its index row and the
next copy appended at the wrong position. Each read is now limited to
the remaining captured bytes.

A failed truncateIndex or DoOffsetLoading after a successful stream
left the index tail on disk unapplied, which a later backupCursor could
mistake for applied rows; both now roll the tail back to the pre-copy
boundary like the stream-error path.

* backup: bound the incremental .dat tail by the maximum covered offset

A key-sorted .idx ends on the highest-key row, which need not be the
record physically last in .dat, so the last-row bound could cut off
live records. Take the end of the highest-offset row inside the bound
instead, and cover the case with a sorted-idx test in Go and Rust.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* backup: restore .dat and the needle map when index replay fails

A failed DoOffsetLoading left the received .dat bytes in place and could
leave partial mappings for rows the index rollback just removed, so the
next copy appended the same records again or read stale entries. Truncate
.dat back to the copy boundary and restore the touched mappings and
metrics from a pre-replay snapshot.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* backup: cover no index rows with the superblock bound

When no row inside the index bound names a record — an empty index or a
bound of index-only tombstones — the covered end is the superblock, not
the captured .dat size, so staged bytes are never streamed without a
covering row.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* seaweed-volume: apply the covered bound even when the index streams no rows

The bound was gated on idx_end > idx_start, so an empty captured index
skipped it entirely and a staged .dat tail still shipped. Gate on the
stream-index predicate instead, matching Go's streamIndex check.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* backup: propagate index coverage read errors

* backup: rebuild the needle map from the truncated index on replay failure

* seaweed-volume: propagate index coverage read errors

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-29 22:37:31 -07:00
0a4668fcfa webdav: overwrite on MOVE when the Overwrite header is missing (#12232)
* webdav: overwrite on MOVE when the Overwrite header is missing

* webdav: defer a MOVE destination's delete into the rename

x/net removes an existing destination before Rename, so a refused or
failed move — a CanRename rejection, a vanished parent, a transient
filer error — loses the target entirely. For an overwriting MOVE onto
an existing destination, run the request against a filesystem that
skips the destination RemoveAll and lets the filer's rename replace the
target inside one transaction instead; a file vs. collection clash
still takes the delete + rename path. Rename itself now tolerates an
existing target, which the atomic filer rename already handles.

* webdav: only default Overwrite when the header has no values

An Overwrite header sent with an empty first value makes Header.Get
return "", which the defaulting treated as absent and rewrote to T —
overriding a client's explicit F in a later value. Check the value list
instead so a present-but-empty header is left alone.

* webdav: move a collection onto a non-empty one via delete + rename

deferredDeleteFS skips the destination RemoveAll so the filer rename can
replace the target in one transaction, but the filer only replaces an
empty collection: moveSelfEntry deletes the target non-recursively and
refuses ErrNonEmptyFolder. Detect a non-empty collection destination and
take the depth-infinity delete + rename path moveFiles would have run.

* webdav: park a clashing MOVE target aside until the rename lands

A refused rename (cross-collection move, bucket target) after deleting
the destination lost its contents; the target now renames to a hidden
sibling first and is restored when the winning rename fails.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* webdav: restore a parked MOVE target under a detached context

A client disconnect cancels the request context mid-move, which also
canceled the restore and stranded the destination under its hidden
sibling name; the restore and parked-entry removal now run under
context.WithoutCancel with a bounded deadline.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* webdav: bound the parked MOVE target's restore after the move

The detached restore deadline started before the source rename, so a
long move could exhaust it; each detached context now starts where its
work begins. A failed parked-entry removal retries briefly and its
warning names the parked sibling for recovery.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* webdav: restore a parked MOVE target without replacing a new claimant

A client can create the destination name while the old target is parked, so the restore rename must fail instead of replacing that new entry.

* webdav: propagate RemoveAll's context to the filer delete

The parked-target cleanup runs under a bounded detached context, but the delete RPC discarded it; passing ctx through makes the deadline effective.

---------

Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-29 22:24:43 -07:00
Chris Lu 74f727d987 test: move the s3vectors pytest suite under test/s3vectors (#12230)
* test: move the s3vectors pytest suite under test/s3vectors

The vectors API tests lived in test/vectors next to the new s3vectors
integration home; the rust-vector-tests workflow's paths and working
directory move with it.

* gitignore: cover the binaries the s3vectors Makefile builds
2026-10-29 19:34:48 -07:00
Chris LuandDevin 665315d75e test/s3vectors: integration test design and catalog-surface suite (#12229)
* test/s3vectors: integration test design and catalog-surface suite

The design doc lays out a two-tier suite: catalog/policy ops that need no
vector engine run against a real weed server over HTTP, while the data
plane skips itself when no engine is reachable. The first test brings up
the server and exercises the bucket/policy lifecycle, encryption-config
round-trip, and pre-engine validation boundaries.

* test/s3vectors: sign API calls so requests reach the vectors router

The gateway selects the vectors router off the SigV4 credential scope —
unsigned requests with only X-Amz-Target fall through to the plain S3
router, so every call in the suite failed before dispatch. Sign each
request with the s3vectors service name and point the spawned server at
a written identity file so the signature is actually verified.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test/s3vectors: bound the spawned server's ports, listeners, and clients

weed server derives each gRPC listener as HTTP+10000, so ephemeral
allocation could hand back a port whose pair is invalid or taken; the
-s3 defaults also bind the Iceberg and Lance listeners. Allocate the
HTTP/gRPC pairs together, pin -ip/-ip.bind, disable the extra listeners,
and give the readiness probe and API client deadlines.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test/s3vectors: verify policy round-trip and gate the engine skip on 503

The stored policy check accepted any JSON body, and a delete that left
the document behind could not be caught — compare the returned document
verbatim and assert a follow-up Get returns 404. The engine probe never
created its bucket, so a healthy engine still produced a skip; create it
first, skip only on a confirmed 503, and fail on any other error, then
exercise the catalog-side index ops when an engine answers.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-29 19:29:33 -07:00
Chris Lu ef597cdad3 test: use an s3vectors action in the vector policy round-trip (#12226) 2026-10-29 19:13:17 -07:00
Ping QiuandClaude Opus 5.5 2a1afadcd0 s3api: return NoSuchUpload when aborting an upload that was already completed (#12227)
s3: AbortMultipartUpload answers NoSuchUpload for an upload that no longer exists

Complete and abort both remove .uploads/<id>. abortMultipartUpload answered
204 whenever that record was missing, so aborting an upload that had already
completed reported success while the object stayed in place, and a client
could read it as the object being discarded. AWS returns 404 NoSuchUpload
("The specified multipart upload does not exist") for that case, and
ListParts here already does.

Answer ErrNoSuchUpload when the record is not found. checkUploadStillOpen
reuses abort for cleanup after the upload is gone; NoSuchUpload is not a
failed cleanup there.

The leftover-directory path from #11385 is unchanged: a .uploads/<id> that
outlived its completed object is still removed metadata-only and answered.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 19:02:23 -07:00
Denis Kieftanduser.email a2f595b207 backup: don't Close a volume after Destroy (#12225)
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
2026-10-29 19:00:15 -07:00
github-actions[bot] cd3e2c8b2c docs: regenerate star history chart 2026-10-30 01:57:47 +00:00
Chris Lu 349576b63e backup: resume incremental copies by index position instead of appendAtNs (#12228)
* volume server: add last-entry cursor fields to VolumeIncrementalCopy

* backup: resume incremental copies by index position instead of appendAtNs

appendAtNs is stamped by whichever replica received the write, so the
same needle carries a different timestamp on every replica. A backup
that resumes from a different replica than last time can append the
last needle again or skip needles written inside the clock gap.

Send the local copy's index entry count and last entry identity (key,
size, cookie) with VolumeIncrementalCopy. The replica verifies the same
record sits at that position and streams from the next index entry, so
the resume no longer depends on any replica's clock. When the boundary
record does not match, or the local copy is ahead, the replica returns
FailedPrecondition and the backup rebuilds the local copy from scratch.

Old servers ignore the new fields and keep the timestamp behavior.

* rust volume server: mirror positional resume in VolumeIncrementalCopy

* volume server: add entries_hash prefix checksum to VolumeIncrementalCopy

* backup: verify the whole index prefix, not just the boundary entry

* rust volume server: verify the index prefix hash in resume_by_last_entry

* backup: keep the source's compaction revision on rebuilt copies

* backup: diverge on a zero-offset boundary entry

A remote-tier tombstone lives in the .idx with no .dat record, so
readNeedleAt(0) parses the superblock and its error bypassed the
ErrIncrementalCopyDiverged mapping; the backup got a generic failure
instead of FailedPrecondition and never rebuilt.

* rust volume server: mirror zero-offset boundary divergence

* rust volume server: gate the empty-dat early return on entry_count

With a positional resume the Go handler still reaches ResumeByLastEntry,
so an empty replica reports divergence; the Rust early return answered a
clean empty stream and weed backup kept a stale copy.

* backup: hash the index entry's size field, not trailing offset bytes

In a 5BytesOffset build the offset occupies bytes 8-12 and the size
starts at byte 13, so the hash folded in one byte of the offset.
Matching records at different offsets then hashed differently and a
valid backup was reported diverged and rebuilt.

* backup: read the index in whole-entry chunks for the prefix hash

When entries are 17 bytes a 1 MiB read buffer does not end on an entry
boundary: the partial tail entry was skipped by the loop but still
counted as consumed, so every later chunk was hashed at a misaligned
position.

* rust volume server: hash the index entry's size field after the offset

Mirrors the Go hashIndexEntries change: with the 5bytes feature the
offset occupies bytes 8-12, so starting the size slice at byte 12
folded one offset byte into the hash and matching records at different
offsets compared diverged.

* rust volume server: read the index in whole-entry chunks for the prefix hash

Mirrors the Go change. With 17-byte entries a 1 MiB buffer does not
end on an entry boundary, so slicing the partial tail entry panicked
once a backup held 61681 entries.

* rust volume server: default the new incremental copy request fields in tests
2026-10-29 18:56:25 -07:00
Chris LuandDevin d0441bbdbb s3vectors: validate index ARNs and expand live integration coverage (#12222)
* test: align vector bucket policy coverage with vector actions

* s3vectors: validate ARN service and resource shape before resolving indexes

* test: exercise signed vector bucket policies and catalog lifecycle

* ci: require the expanded vector integration matrix and all gateway tests

* test: reject index ARNs in bucket-level API requests

* s3vectors: scope ARN resolution to the gateway namespace

* s3vectors: reject name fields that conflict with resolved ARNs

bucketARNs and resolveIndexARNs already reject an indexArn that names a
different bucket than vectorBucketArn, but a vectorBucketName or
indexName disagreeing with the ARNs was silently dropped. Validate the
plain names against the resolved ARN values the same way.

* test: cover conflicting vector names and ARNs in live requests

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-29 17:57:11 -07:00
Chris Lu cc5912b8b8 s3vectors: e2e conformance for vector bucket policy evaluation (#12221)
* s3vectors: reject empty boolean-operator lists and sub-objects in filters

An empty $and/$or list or a nested empty object rendered '()'' — invalid
SQL surfaced as an engine error instead of a clean ValidationException.

* s3vectors: e2e conformance for vector bucket policy evaluation

The handler-level test covers statements; this runs the whole stack a
client sees — HTTP dispatch, authentication, policy evaluation, catalog,
engine — and walks the full lifecycle: a non-owner is denied without a
grant, an Allow on the bucket ARN grants every index under it, a Deny
vetoes an Allow on the same action, and deleting the policy restores
owner-only access.

* s3vectors: prove deny precedence and grant removal in policy e2e

* s3vectors: reject field operator objects with no operators

* test: use an s3vectors action in the vector policy round-trip
2026-10-29 16:19:22 -07:00
Chris Lu 9d5929a73f s3vectors: reject empty boolean-operator lists and sub-objects in filters (#12220)
* s3vectors: reject empty boolean-operator lists and sub-objects in filters

An empty $and/$or list or a nested empty object rendered '()'' — invalid
SQL surfaced as an engine error instead of a clean ValidationException.

* s3vectors: reject field operator objects with no operators
2026-10-29 16:16:34 -07:00
Ping QiuandClaude Opus 5.5 3f954e34dc s3api: label bucket-level requests correctly in the access log (#12224)
* s3api: label bucket-level requests correctly in the access log

getResourceType treated a request as bucket-level only when the object key
was "/". NormalizeObjectKey strips the leading slash, so bucket routes arrive
with an empty key and every bucket operation was logged as an object
operation: ListObjectsV2 as REST.GET.OBJECT, DeleteObjects as
REST.POST.OBJECT, and the lifecycle, ACL and policy calls likewise.

Treat an empty object key as bucket-level too. The new test covers five
bucket-level and two object-level request shapes; it fails without the fix.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* s3api: label only POST ?delete as a batch delete in the access log

With bucket-level requests now reaching the bucket branch, any method on
/bucket?delete was labelled BATCH.DELETE.OBJECT. Only the POST route is
DeleteObjects; label other methods as the plain bucket operation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-29 16:10:36 -07:00
Chris Lu fd493f2547 mount: refuse to attach opens to a draining file handle (#12219)
* mount: refuse to attach opens to a draining file handle

With the writeback cache, releasing the last open of a file leaves the
shared FileHandle in the map at counter zero while an async flush
drains it; the worker removes the map entries once the flush commits.
AcquireHandle waits for a pending flush and then attaches, but the two
steps are not atomic: a release can slip in between, drop the counter
to zero, and queue the drain while the opener goes on to mint a new
file-handle id on the draining handle. When the drain completes,
RemoveFileHandle deletes every fh id for the inode -- including the one
just minted. The kernel still holds that id, so its next FUSE_READ
gets ENOENT from GetHandle; the failed read leaves the folio not
uptodate and the kernel surfaces EIO to userspace. This was the flaky
TestWritebackCacheConcurrentMixedOps failure.

AcquireFileHandle now refuses to attach when the shared handle has
reached a zero counter and reports it as draining. AcquireHandle and
the Create path retry the whole wait-resolve-attach sequence; the next
pass blocks on the pending flush (flag registration and the counter
drop share pendingAsyncFlushMu) and builds a fresh handle from the
post-flush entry. The DLM-failure release in AcquireHandle also goes
through ReleaseHandle so it cannot strand a counter-zero handle
outside the drain bookkeeping.

Removes the refreshEntryForRead retry from #12206, which worked around
the symptom by refetching the entry after the fh id was already gone.

Test: TestAcquireFileHandleRefusesDrainingHandle covers the refusal
and fresh-handle path; TestWritebackCacheConcurrentMixedOps and an
amplified concurrent-overwrite repro run clean.

* mount: keep refresh-on-error retry for stale-chunk reads
2026-10-29 16:06:57 -07:00
Chris Lu ac7b4f4cf6 s3vectors: bound ListVectorBuckets and ListIndexes maxResults (#12218)
* s3vectors: bound ListVectorBuckets and ListIndexes maxResults

Both pass maxResults to the catalog unbounded; AWS caps these list pages
at 100, the same way ListVectors is bounded and topK is bounded.

* s3vectors: raise ListVectorBuckets/ListIndexes maxResults cap to 500

AWS documents a valid range of 1-500 for both list APIs; the shared
limit of 100 rejected legitimate page sizes.

* s3vectors: reject explicit maxResults of zero on list calls

AWS requires maxResults >= 1, but a plain int field cannot tell an
omitted field from an explicit zero. Making MaxResults a pointer keeps
the catalog default for omitted fields while a supplied zero fails
validation.

* s3vectors: test max ListIndexes page against an existing bucket
2026-10-29 15:02:30 -07:00
Chris LuandDevin f122f313e1 s3tables: keep the admin bypass off an identified caller's empty action list (#12217)
Generated with [Devin](https://devin.ai)

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-29 13:41:26 -07:00
Chris Lu 229ad84e2d s3tables: reject s3vectors resources in every ARN partition (#12215)
* s3tables: reject s3vectors resources in bucket policies

An arn:aws:s3vectors: Resource is as dead as an s3vectors action: the
evaluator compares against the generated s3tables ARN, so the rule the
writer believes enforced never matches.

* s3tables: reject s3vectors resources in every ARN partition

* s3tables: test wildcard-partition and mixed-list s3vectors resources
2026-10-29 13:38:53 -07:00
Chris LuandDevin 8a2fec8610 filer.sync: skip superseded replays before fetching any chunks (#12212)
* filer.sync: skip superseded replays before fetching any chunks

On a directory under heavy overwrite, most entries the sync replays are
already superseded on the source. The sink only learned that after chunk
reads failed — or worse, after fetching and uploading bytes a later event
replaces — so the backlog drained slowly and the log filled with 404s.

CreateEntry and UpdateEntry now check sourceMovedOn before touching a
chunk: one entry lookup skips a stale replay entirely. A mid-replication
supersession is still caught by the existing retry gates.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* replication: bound the source liveness probes with a timeout

The supersession checks added a source entry lookup to the replication
path, and every lookup ran on context.Background() with no deadline. A
source that accepts the call but whose metadata store stalls would hold
the sync forever — before a single chunk fetch, even when every chunk is
still readable. Bound the probe at 30s; an unanswered lookup proves
nothing, so replication proceeds and the existing retry gates decide.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* replication: sync the live version a superseded replay points at

Skipping a stale replay assumed a later event always re-carries the
current content. It does not when the superseding write was itself
replicated in from another cluster: those events are dropped before they
reach a filer sink, so the destination could be left missing the file.
When the source reports a live entry newer than the replay, commit that
version through the normal path instead — equivalent to applying the
later event, with no dependence on its delivery. A gone source entry
still skips, since the destination already matches the source.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* replication: reconcile moved entry content after a native rename

AtomicRenameEntry relocates whatever version the destination held. If the
preceding update replay was skipped because the source had already moved
to the new name, the moved entry keeps the old bytes and — with no
further writes — stays stale forever. MoveEntry now runs the same
reconciliation as an in-place update against the rename's snapshot, so
the moved entry picks up pending content changes.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* replication: repair from the live snapshot a superseded replay resolves to

An incremental diff against the replayed oldEntry can resolve manifests the
overwrite already vacuumed, which fails every retry of the event.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-29 13:37:27 -07:00
Chris LuandDevin 98fe70a247 s3vectors: keep bucket policies in vector form and evaluate them in the gateway (#12214)
* s3vectors: translate vector bucket policies to the catalog form

The document is stored verbatim on the shared table bucket and evaluated
against s3tables actions and ARNs, so a policy written for the vector API
was a dead rule. Actions now map to the catalog operation the request
authorizes against, and an arn:aws:s3vectors: resource becomes the
bucket/.../table/vector/... ARN the index's catalog entry answers to.

* s3tables: expose PrincipalFor and PolicyContext for façade evaluation

A façade authorizing against its own action namespace needs the same
identity resolution and evaluation context the catalog uses internally.

* s3vectors: keep bucket policies in vector form and evaluate them in the gateway

The earlier approach translated the document to catalog actions and ARNs,
which lost the caller's original policy for GetVectorBucketPolicy and could
not cover data-plane calls that never touch the catalog. The document now
stays in s3vectors form under a dedicated bucket attribute and every
handler evaluates it through authorizeVector against the action and ARN
the request names; a dedicated key also keeps a vector grant from ever
authorizing a catalog call.

* s3vectors: close policy-mutation races and keep vector grants visible

- Authorize Put/DeleteVectorBucketPolicy inside the policy slot's
  compare-and-swap, so a grant revoked between reads cannot authorize
  the pending write.
- Assert the policy attributes stay absent in the update precondition,
  so a concurrent policy or tag write trips the retry instead of being
  silently reverted.
- Migrate a raw vector document out of the shared policy slot on
  replace or delete, keeping its catalog rules.
- Pass the unresolved identity to PolicyContext so open-by-default
  deployments still work.
- Filter ListVectorBuckets on the vector policy itself, so buckets a
  caller was granted under s3vectors actions stay listed.

* s3vectors: reuse SnapshotExtended for the vector-policy CAS precondition

* s3vectors: skip buckets that vanish mid-list instead of failing the page

* s3vectors: reject non-vector bucket formats inside authorizeVector

* s3vectors: check bucket format after authorization in authorizeVector

An unauthorized caller probing an ICEBERG bucket got NoSuchBucket,\nleaking that the bucket exists. The format check now runs after the\npermission check so denied callers always see AccessDenied.

* s3vectors: cover legacy shared-slot policy removal on delete

* s3tables: resolve the forwarded identity name like getAccountID in PrincipalFor

A synthesized manager request resolves its caller through the identity-
name branch of getAccountID, which normalizes the name (ARNs collapse to
their trailing segment, whitespace trims) before falling back to the
handler's account ID. PrincipalFor returned the raw name, so ARN-shaped
identities never matched their own grants.

* s3tables: assert the vector policy key in update preconditions

s3vectors-policy is stored on table bucket entries but was missing from
s3tablesExtendedKeys, so a catalog update that read the entry before the
first vector policy was saved could overwrite it without tripping the
compare-and-swap. Asserting the key makes such a race fail instead of
silently erasing the policy.

* s3vectors: reject rules the evaluator cannot honor in vector bucket policies

The shared Statement shape has no negated fields, the vector evaluation
supplies no condition keys, and a document that fails the typed decode
is discarded wholesale at evaluation — including denies the writer
believed active. Reject NotPrincipal, Condition, foreign-service actions
and resource ARNs, and require the typed PolicyDocument decode. Also
cover that an admin-named identity still reaches its own buckets.

* s3vectors: read vector policy fields the way the evaluator decodes them

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3tables: match ARN policy principals through caller normalization

Caller principals are normalized ids, so a statement naming a full IAM
ARN could never match: an ARN-scoped deny was silently dead and an
ARN-scoped allow unreachable. Compare arn: specs against the caller
under the same normalizePrincipalID the ownership path uses, while an
ARN caller keeps verbatim wildcard matching.

* s3tables: grant the admin bypass on the caller's actions, not the name

A principal that only normalizes to the admin id skipped every policy
deny. When the request carries an identity, the bypass now requires the
identity's own admin action — the same gate getAccountID applies — while
name-only transports with no action list keep it. PolicyContext takes
the request so it can carry IdentityActions, and the vector context
helpers route through the shared identity keys so the resolved name and
actions agree.

* s3tables: match ARN policy principals only for deny statements

Normalization drops the ARN's account and path, so an allow matched on the
trailing name could grant a same-named caller the statement never verified.
Denies keep the normalized match so the rule cannot be ducked through the
same collapse.

* s3tables: scope service wildcards to their own service

s3tables:* counted as blanket admin inside the shared permission check, so
an identity holding only catalog grants administered s3vectors operations
too. hasAdminActionFor now requires the wildcard to name the operation's
service; the global admin markers and s3:* stay service-agnostic.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-29 13:35:34 -07:00
Chris Lu 4ef123d764 s3vectors: fix shared API behavior and test Go/Rust volume parity (#12216)
* s3vectors: reuse shared authenticated identity context

* s3vectors: encode empty results as arrays

* test: cover S3 Vectors with Go and Rust volume servers

* ci: require both volume implementations in vector integration tests

* s3vectors: keep identityOf as a shim over the shared identity context

Concurrent branches add identityOf call sites; deleting the helper would
break the build when they merge in either order. Route it through
GetIdentityNameFromContext so every caller resolves the same identity.

* test: wait for the vector engine to register before running

The gateway resolves weed-vector lazily through the master cluster list,
so a listening port does not mean the engine is reachable yet; the first
vector write or query could race KeepConnected and fail. Restore the
cluster.ps registration wait the previous harness had, and ignore the
pytest cache directory the suite creates.

* test: wait for master registration across OSS and enterprise

Use the exact engine registration event after AddClusterNode completes. The current enterprise cluster.ps does not list vector nodes, so waiting on its output prevents the shared fixture from starting there.

* s3vectors: keep contextWithIdentity as a shim alongside identityOf

In-flight callers still write the caller name through it; route the
write through SetIdentityNameInContext so both helpers agree on the
shared key whichever lands first.
2026-10-29 13:35:05 -07:00
Chris Lu 88565df885 s3tables: reject s3vectors resources in bucket policies (#12213)
An arn:aws:s3vectors: Resource is as dead as an s3vectors action: the
evaluator compares against the generated s3tables ARN, so the rule the
writer believes enforced never matches.
2026-10-29 11:59:07 -07:00
Chris Lu 15066c8a9f vector: implement drop_index and have DeleteIndex drop the data (#12204)
* vector: implement drop_index in the engine

The RPC was a stub; it now resolves the table's vended location and
removes every object under it through the same credential path open uses.
A table the catalog can no longer describe is already dropped, so the call
stays idempotent.

* s3vectors: drop index data after the catalog delete

DeleteTable carries the delete authorization, so it runs first; the engine
DropIndex then removes the data under the index's location. NotFound and
Unimplemented are tolerated so an index whose data went away with the
catalog entry, or an older engine, still answers 200.

* s3vectors: let the catalog delete carry the index data removal

A vector index is colocated: the catalog entry sits at the dataset root,
so DeleteTable's recursive delete already removes entry and data. The
engine DropIndex call afterwards could only ever describe a missing or
recreated record — a no-op in the common case, and a delete of a
replacement index's files when the name was reused in between.
2026-10-29 11:57:23 -07:00
Chris Lu 2dbf9c9108 s3: evaluate all conditional write headers under the object lock (#12211)
* filer_pb,filer: per-mutation preconditions on ObjectTransaction

A transaction's request-level condition evaluates one entry before any
mutation runs, so a commit guarding two paths — the object it publishes
and the upload directory it assembles — could not be expressed. Give each
ObjectMutation an optional condition (evaluated against condition_key,
else the mutation's target, under the same lock just before the mutation
applies) and report which mutation failed through failed_mutation on the
response, so callers can tell a mutation guard's rejection apart from the
request-level precondition's.

* s3: evaluate multipart completion conditions under the object lock

CompleteMultipartUpload checked conditional headers gateway-side before
committing, and the routed commit transaction guarded only the upload
directory — nothing made If-None-Match: * atomic with the publish, so
concurrent conditional completions of the same absent key could all
report success.

The request's reducible write condition (If-None-Match/If-Match, star or
single strong ETag) now rides the commit as the transaction's request
condition on the owner filer: on the object path for unversioned buckets,
falling back to the .versions directory for suspended and enabled
versioning. The upload-exists guard moves onto the PUT mutation, keeping
its check-before-publish ordering. Conditions the reducer cannot express
stay on the distributed write lock, where check and commit already
serialize.

* filer,s3: resolve the latest version first for versioned conditional writes

* filer: keep directory entries holding content out of the absent-slot rule

* s3: keep a gateway-side upload-exists check for filers without mutation conditions

* filer: honor null-version-is-latest resolving prefer-alt conditions

* drop scratch multipart-race repro programs

* s3: note the write lock does not cover routed writes on non-reducible conditions

* filer: mirror the gateway's remote-etag precedence in condition evaluation

storedEntryETag only knew the extended attribute and the chunk fallback,
so a synced remote entry compared chunk-derived ETags while the gateway
compares the remote ETag. Follow the same precedence: extended attribute,
remote ETag while the entry still holds the remote bytes, Md5, then the
chunk hash — and no ETag for an entry holding none.

* s3: reduce all conditional write headers into owner-lock clauses

Time-based headers, combined conditions, ETag lists, and weak ETags stayed
on the distributed write lock because buildWriteCondition could not express
them, leaving their gateway check racing owner-routed writes. Reduce every
parseable header form to WriteCondition clauses — multiple clauses compose
AND, ETag candidates carry the raw list for the filer's canonicalization,
and RFC 7232 precedence keeps If-Unmodified-Since hidden by If-Match and
If-Modified-Since hidden by If-None-Match — so the owner filer evaluates
them under the same object-path lock as the commit. Only unparseable
headers still fall back, where the lock path reports the same parse error.
2026-10-29 11:49:48 -07:00
Chris Lu 2737315e9d s3: evaluate multipart completion conditions under the object lock (#12209)
* filer_pb,filer: per-mutation preconditions on ObjectTransaction

A transaction's request-level condition evaluates one entry before any
mutation runs, so a commit guarding two paths — the object it publishes
and the upload directory it assembles — could not be expressed. Give each
ObjectMutation an optional condition (evaluated against condition_key,
else the mutation's target, under the same lock just before the mutation
applies) and report which mutation failed through failed_mutation on the
response, so callers can tell a mutation guard's rejection apart from the
request-level precondition's.

* s3: evaluate multipart completion conditions under the object lock

CompleteMultipartUpload checked conditional headers gateway-side before
committing, and the routed commit transaction guarded only the upload
directory — nothing made If-None-Match: * atomic with the publish, so
concurrent conditional completions of the same absent key could all
report success.

The request's reducible write condition (If-None-Match/If-Match, star or
single strong ETag) now rides the commit as the transaction's request
condition on the owner filer: on the object path for unversioned buckets,
falling back to the .versions directory for suspended and enabled
versioning. The upload-exists guard moves onto the PUT mutation, keeping
its check-before-publish ordering. Conditions the reducer cannot express
stay on the distributed write lock, where check and commit already
serialize.

* filer,s3: resolve the latest version first for versioned conditional writes

* filer: keep directory entries holding content out of the absent-slot rule

* s3: keep a gateway-side upload-exists check for filers without mutation conditions

* filer: honor null-version-is-latest resolving prefer-alt conditions

* drop scratch multipart-race repro programs

* s3: note the write lock does not cover routed writes on non-reducible conditions
2026-10-29 11:46:25 -07:00
3955e7218f mount: keep the shared directory build out of the requester's context (#12206)
* mount: keep the shared directory build out of the requester's context

The shared directory build behind EnsureVisited is a singleflight call: the
first requester for a path owns the build, and every later requester for the
same path waits on that same build. The build ran on the owner's request
context, so a FUSE request that returned -- or that the kernel interrupted --
canceled the build for everybody waiting on it.

A waiter whose own context was still alive then received somebody else's
cancellation. weed/mount/weedfs_dir_read.go classified a canceled read as
EINTR only when the local context was done, so such a waiter fell through to
fuse.EIO: the kernel handed EIO to readdirent, kubelet's applyFSGroup failed,
and the volume never mounted.

Build the shared directory on a context that keeps the request's values but
not its cancellation, and report a canceled read as EINTR even when the local
context is still alive. Waiters still abandon on their own context, so a
canceled request stops waiting immediately.

This follows up #11717, which made the wait abandonable and mapped the local
cancellation to EINTR, but left the shared build itself on the owner's
context. The surrounding build lifecycle calls already use
context.Background() for the same reason.

Test: TestEnsureVisitedSharedBuildSurvivesOwnerRequestCancel reproduces the
cancellation with a fake filer and fails without this change.

* mount: classify gRPC-canceled directory listings as EINTR

* mount: log only real directory read failures, not interruptions

* mount: keep shared section refresh out of the requester's context

* mount: let section-refresh waiters leave on their own context

The shared refresh flight now outlives the requester that started it,
but callers still blocked in singleflight.Do until it finished — an
interrupted directory read could not stop waiting, and the caller's
own deadline never applied to the wait. DoChan + select on the
caller's context restores early abandon while the detached refresh
still completes for the remaining waiters.

* ci: re-run FUSE integration after a flaky writeback-cache failure

* mount: refresh a stale entry once before reporting a read error

A read resolves chunk views from the handle's entry. A concurrent
overwrite commits a new entry whose replaced file ids the filer queues
for deletion; the open handle only learns the new chunk list when the
metadata invalidation lands. A fetch that lands between the volume
delete and the invalidation names data that is already gone and the
read reports EIO. On a chunk read error, wait out any in-flight async
flush, re-read the entry from the filer, install it when it outranks
the handle, and retry once. An unchanged version means the failure is
genuine.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* mount: fence the read-error entry refresh by file identity

Two guards on refreshEntryForRead's install path: the unlocked
dirtyMetadata read raced with Write under the handle lock — drop it,
installAckedEntry already checks under that lock, and it now fences
isDeleted handles the same way. And a saved path can name a different
file after unlink/reuse, so refuse the refresh when the fetched inode
disagrees with the one the handle holds.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-29 11:32:57 -07:00
Chris Lu c7d77eb9a8 s3vectors: accept and return encryptionConfiguration on vector buckets (#12203)
* s3tables: carry encryptionConfiguration in table bucket metadata

* s3vectors: accept and return encryptionConfiguration on vector buckets

The configuration is validated (sseType AES256 or aws:kms) and stored on
the bucket record, so the API round-trips what the caller declares.

* s3vectors: reject caller-chosen encryption keys on vector buckets

* s3tables: reject caller-chosen encryption keys on table buckets

* s3tables: reject unknown sseType in bucket encryptionConfiguration

The bucket check rejected aws:kms and kmsKeyArn but stored any other
sseType — a made-up or mistyped value would round-trip through
GetTableBucket as an encryption claim the storage does not keep. The
validator now admits only the AES256 declaration, shared so the vector
layer can drop its copy.

* s3vectors: delegate encryption validation to the shared catalog check

* s3tables: accept the sseAlgorithm name in bucket encryptionConfiguration

S3 Tables declares the algorithm as sseAlgorithm while S3 Vectors uses
sseType; the shared check now accepts either spelling, still admits only
AES256, and still refuses any caller-chosen key.

* s3vectors: call the shared encryption validator directly

* s3vectors: require each API's own encryption field name

The shared validator accepted sseAlgorithm on the vector path, so a
CreateVectorBucket carrying the s3tables field stored a configuration
GetVectorBucket could never echo back in the vectors shape (and vice
versa). ValidateEncryptionConfiguration now takes the wire name its
caller expects; the vectors create path normalizes to sseAlgorithm for
the catalog and GetVectorBucket renames it back to sseType.
2026-10-29 11:29:00 -07:00
Chris Lu fc9be77dbb s3vectors: bound QueryVectors topK at the AWS limit (#12202)
* s3vectors: bound QueryVectors topK at the AWS limit of 30

topK passed through unbounded; AWS caps a query at 30 results, so a
larger request is a ValidationException before it reaches the engine.

* s3vectors: raise QueryVectors topK bound to the AWS limit of 10,000

* s3vectors: validate QueryVectors topK before the catalog lookup

* s3vectors: cover the QueryVectors topK bounds
2026-10-29 11:22:48 -07:00
eecc461b39 s3vectors: put, get and delete vector bucket policies (#12201)
* util: do not bind outbound dials to a loopback -ip.bind

`weed server -ip.bind=127.0.0.1` keeps every listener local, but the
same address became the source of outbound connections, so the filer
dialled its remote mount from 127.0.0.1 and failed with "connect:
invalid argument". A loopback source cannot reach off the host; leave
the source to the OS, as for a wildcard bind.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3vectors: put, get and delete vector bucket policies

The AWS S3 Vectors policy operations had no handlers; the policy document
is stored on the shared table bucket, so the handlers delegate to the
catalog's PutTableBucketPolicy, GetTableBucketPolicy and
DeleteTableBucketPolicy and keep its guard.

* s3vectors: treat ICEBERG buckets as absent in policy handlers

GetVectorBucketPolicy/PutVectorBucketPolicy/DeleteVectorBucketPolicy
delegated straight to the table bucket, so a bucket created for ICEBERG
tables — invisible to every other vectors operation — could still have
its policy read and written through the vectors API. Gate all three on
lanceBucket, the same format check DeleteVectorBucket and CreateIndex
already apply.

* test/vectors: cover vector bucket policy put/get/delete

Round-trips a policy document through the new ops and asserts a get on
the deleted policy surfaces NotFoundException, so the boto3 conformance
run exercises the added handlers end to end.

* s3vectors: validate the bucket ARN and policy document

Bucket-level operations now reject an index ARN and an ARN minted for
another service, and PutVectorBucketPolicy refuses a document carrying
s3vectors actions — the evaluator only honors s3tables actions, so such a
rule would be a dead grant.

* s3vectors: validate the bucket ARN on the remaining bucket-level operations

* s3vectors: cover bucket ARN and policy document validation

TestBucketARNs pins the accepted forms and the index/foreign-service
rejections; TestValidatePolicyDocument covers a valid document, a
malformed one, and the dead s3vectors action namespace.

* s3vectors test: use an enforceable action in the bucket policy test

The catalog refuses s3vectors actions in a stored policy, so the put call
rejected the document before the get and delete assertions ran.

---------

Co-authored-by: Ólafur Bjarki Bogason <olafur@genkiinstruments.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 11:15:59 -07:00
Chris LuandDevin 8e44178dc6 s3vectors: cap PutVectors at 500 items and check data against the index dimension (#12200)
* s3vectors: cap PutVectors at 500 items and check data against the index dimension

A batch could carry any number of vectors and a vector of the wrong
dimensionality reached the engine: AWS caps a PutVectors call at 500 items
and requires each vector to match the index's declared dimension.

* s3vectors: cover the PutVectors batch and dimension checks

* s3vectors: reject query vectors that do not match the index dimension

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3vectors: cover the PutVectors batch size bounds

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3vectors: cover the catalog-backed PutVectors handler path

Run PutVectors through the real handlers over the in-memory catalog: a
matching vector reaches the engine, a mismatching one is rejected with
ValidationException before PutVectors is called, and a full
vectorsMaxPutBatch batch is accepted.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-29 11:14:30 -07:00
Javier de la Torreanduser.email 4be28fef2d iceberg maintenance: cache a remote-only file before reading it (#12191)
* iceberg maintenance: an unreadable parquet file fails the job, not the process

parquet.NewReader panics on any file it cannot open (wrong magic, a
truncated footer, zero bytes), and parquet-go's row decoder panics on a
corrupt page under a valid footer. Maintenance runs in the worker's
goroutines with no recover, so one bad delete or data file in any table
ended the whole process.

- Every parquet open in compaction goes through newParquetReader, which
  opens with parquet.OpenFile and returns the error.
- A panic while decoding a delete file or a data file's rows is recovered
  into an error naming the file (recoverRead/guardRead).
- As a backstop, detection recovers per table and Execute per operation:
  the table is skipped or the operation fails, and the worker keeps
  running.

A failed delete file still fails the table's compaction (no commit), as a
read error did before; a failed data file still skips only its bin.

* iceberg maintenance: cache a remote-only file before reading it

On a bucket mounted with remote.mount, a file that is not cached locally
has no chunks and no inline content, only its RemoteEntry. Maintenance
read it as an empty file: a parquet reader failed on zero bytes (before
the previous change, it panicked and ended the process) and a manifest
failed to parse, so a table whose files had been uncached could not be
maintained.

lookupFileByIcebergPath, the lookup every read goes through, now calls
CacheRemoteObjectToLocalCluster for a remote-only entry, as the S3
gateway does on a read, and reads the entry it returns; the call is tried
twice, and an entry still remote-only after caching is an error. The
lookups that only resolve a path (orphan removal) do not cache.
readEntryContent reports a file with no content and no chunks as empty
instead of returning zero bytes.

The fake volume server in the tests now serves chunk reads, so the
remote-only tests read the cached file through the chunk reader as a
real one is read.

* iceberg maintenance: bound the metadata location read before caching a remote file

* iceberg maintenance: bound the metadata read on the entry the cache returns

* iceberg maintenance: enforce the metadata size bound inside the remote cache call

The worker checks the resolved entry before asking the filer to cache a
remote-only object, but the filer resolves the entry again at cache time
and downloads whatever its remote size then says. A replacement landing in
between could push an oversized object through the download even though
the post-cache check would refuse to read it.

CacheRemoteObjectToLocalClusterRequest gains an optional max_size; the
filer rejects an over-limit remote object before the download starts.
loadMetadata passes metadataFileMaxBytes; data-file reads leave it unset.

* iceberg maintenance: key the remote cache singleflight by the size limit

Callers sharing one flight share its outcome, including a bounded
caller's over-limit rejection: an unrestricted reader joining the same
path would fail even though its own bound allows the download. The
limit joins the key so only callers with the same bound share a flight.

* iceberg: gofmt compact_order_test.go

* filer: bound remote cache callers before the shared download

Each caller's max_size is now enforced on the entry the filer resolves
before that caller joins the singleflight, which is keyed by path again.
A bounded caller's rejection is never shared with other callers, and
callers with different bounds share one download instead of pulling the
same remote object twice.

* filer: bind the remote cache size bound to the downloaded entry

The pre-flight check and the shared download resolve the entry
separately, so a remote metadata refresh landing in between lets a
larger object through: the download ran to the refreshed size and only
the reader's post-check refused it. The bound is now re-checked inside
the flight on the entry the download streams.

The rejection is typed so a co-tenant sharing the flight does not
inherit it: callers whose own bound permits the reported size retry on
a fresh flight instead of failing for another caller's limit.

---------

Co-authored-by: user.email <chrislusf@users.noreply.github.com>
2026-10-29 11:13:23 -07:00
9df7d62b4b mount, s3: clarify allowOthers requirement and bucket name error (#12197)
* mount, s3: clarify allowOthers requirement and bucket name error

`weed mount -allowOthers` defaults to true. On Linux, a non-root
user's mount then needs `user_allow_other` in /etc/fuse.conf, but the
help text only said "allows other users to access the file system".
Say so in the help, and point at -allowOthers=false.

Also fix the grammar of the bucket name length error ("must between
[3, 63]" -> "must be between 3 and 63").

Help and error text only; no behaviour change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: normalize the remaining VerifyS3BucketName error messages

The length message was clarified; the rest of the function still had
inconsistent phrasing ("name must start", "cannot be ip addresses",
"prefix xn-- ... in bucket prefix"). Align them on "bucket name ..."
wording matching the AWS bucket naming rules.

---------

Co-authored-by: pingqiu <pingqiu999@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
2026-10-29 01:36:16 -07:00
multivac61andClaude Opus 5.5 d21a1eda54 util: do not bind outbound dials to a loopback -ip.bind (#12198)
`weed server -ip.bind=127.0.0.1` keeps every listener local, but the
same address became the source of outbound connections, so the filer
dialled its remote mount from 127.0.0.1 and failed with "connect:
invalid argument". A loopback source cannot reach off the host; leave
the source to the OS, as for a wildcard bind.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 01:35:00 -07:00
Chris Lu 43626431c3 vector: recheck master discovery quickly while no worker is cached (#12199)
The vectors gateway backs off discovery for 5s between resolutions, but
the backoff also suppressed rediscovery after an empty lookup. A worker
that landed its KeepConnected registration a moment after the first
query stayed invisible for the whole window — longer than a client's
retry budget — so the S3 Vectors e2e flaked: CreateIndex raced the
registration, got InternalFailure, and every boto retry fell inside the
same 5s fast-fail window.

With no cached backend, recheck every 500ms instead; the 5s interval now
only bounds membership refreshes against a live client, and an absent
worker still cannot flood the master with lookups. The empty-membership
error also becomes codes.Unavailable so the API answers 503 rather than
InternalFailure.

cluster.ps lists registered vector nodes like the other node types, and
the e2e harness polls it for the engine's registration instead of only
checking the gRPC port — a listening engine is not discoverable until
the registration lands.
2026-10-29 01:15:29 -07:00
Chris LuandDevin 2c510cc30e ec.rebuild: remount shards the rebuild replaced, keep regenerated copies (#12193)
* volume: remount ec shards the rebuild replaced under a mounted handle

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ec.rebuild: keep copied shards the rebuild regenerated in place

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: reopen regenerated ec shards in place instead of remounting

UnmountEcShards and MountEcShards send deltas through separate buffered
channels, so a remount can reach the master as add-then-delete and drop
the repaired shard from lookups until the next full EC heartbeat.
ReopenEcShard swaps the mounted shard's descriptor under the volume lock
instead: registration, disk type, and heartbeat state never flip, and
readers holding the old shard degrade to a clean error the parity path
already recovers from.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: reopen rebuilt ec shards in place, mirroring Go

volume_ec_shards_rebuild recreates a shard file under a mounted handle
whose descriptor still serves the detached inode (a mounted shard whose
file was deleted under it). DiskLocation::reopen_ec_shard swaps the
mounted descriptor for a fresh open, matching Go's ReopenEcShard — the
registration and heartbeat view never flip.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: fail ec rebuild when a regenerated shard cannot be reopened

* rust volume: fail ec shards rebuild when a rebuilt shard cannot be reopened

* volume: repoint mounted ec shards after rebuild across disks

A mounted shard whose file was deleted under the mount is "missing" to
the rebuild: the regenerated file lands on the rebuild location while
the handle keeps serving the detached inode on another disk. Reopening
the mounted disk then failed the whole RPC, so the caller never saw the
rebuilt ids and its deferred cleanup deleted the regenerated copies.

Repoint every mounted handle of the rebuilt volume instead of only this
run's outputs: reopen in place when the rebuilt file is on the mounted
disk, otherwise drop the stale handle and remount the shard from
whichever local disk holds the file via a delta-free load split out of
MountEcShards — a same-server move, so the master's registration never
flips. A shard that can be neither reopened nor remounted is left
unmounted rather than serving replaced bytes, and the repoint step no
longer fails the RPC, so RebuiltShardIds always reaches the caller.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: repoint mounted ec shards after rebuild across disks

Mirror the Go repoint loop: iterate every mounted shard of the rebuilt
volume, reopen in place when the rebuilt file is on the mounted disk,
and otherwise drop the stale handle and remount the shard from
whichever local disk holds it. The repoint no longer fails the RPC, so
rebuilt_shard_ids still reaches the caller and its copy cleanup keeps
the regenerated shards.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: still rebuild ecx and repoint mounts when no shard is missing

* master: keep a node in an ec shard's lookup while a disk still holds it

A rebuild that relocates a mounted shard between a node's disks reports
the move as a delete on the source disk plus an add on the destination.
Registering first is a no-op because shard lookups key on the node, then
the unregister dropped the node outright — and steady-state heartbeats
never restore it, so readers and later rebuilds lose the repaired copy.
Skip the delete while another disk on the node still holds the shard.

UnregisterDataNode now clears the node's own ec records before dropping
its registrations, so the retained-shard check cannot skip the teardown,
and each unregister gets a ShardsInfo copy since DeleteEcShard drains
the shared one.

* rust volume: fold the journal on a mounted index instead of rewriting it

The rebuild handler regenerated .ecx through File::create even while the
volume was mounted, truncating the index under readers that search the
open descriptor with its old size — a live lookup could hit EOF mid-scan.
Go folds the deletion journal in place on a mounted volume; do the same
through rebuild_ecx_from_journal and reserve the full regen for the
unmounted path, where nothing holds the file open to interrupt.

* volume, rust volume: report rebuilt shards on late failures, fix rust corrupt discovery

Two review findings the repoint rework left open:

- A RebuildEcxFile failure still returns a plain error after shards were
  regenerated, so the caller discards the response holding the rebuilt
  ids and the deferred cleanup deletes copies the rebuild regenerated
  in place. The Go handler now packs the ids into the gRPC status
  details and the shell pulls them back out on error; the Rust handler
  carries them in grpc-status-details-bin the same way.
- The Rust rebuild still counted a sidecar-corrupt shard as present, so
  it fed bad bytes into reconstruction and never regenerated the shard.
  Discovery now loads the generation-0 .ecsum the way Go does, verifies
  every present shard, and reclassifies mismatches (and zero-size
  residue) as rebuild targets written to a .rebuilding temp file,
  verified, and atomically renamed over the original only after every
  output passes the sidecar check. Honors unsafe_ignore_sidecar, added
  to VolumeEcShardsRebuildRequest to match the Go proto.

* volume: repoint sibling journals sharing a folded .ecj after rebuild

RebuildEcxFile unlinks the deletion journal and the mounted path only
reopens its own handle — a sibling mount journaling into the same file
keeps appending to the detached inode, and those deletes never reach a
path-based reader again. Reopen every mounted volume sharing the index
base before leaving the rebuild.

* rust volume: repoint sibling journals sharing a folded .ecj after rebuild

rebuild_ecx_from_journal replaces the journal and reopens only its own
handle; mounts on sibling disks sharing that .ecj keep the detached
handle, so their next appends are lost at restart. Adopt the folded
journal on every mounted runtime sharing the path before releasing the
store lock, clearing their in-memory set — every journaled id is now an
.ecx tombstone.

* ec: swap the deletion journal handle only after the new file opens

A failed reopen left ecjFile nil, so every later delete returned
"ec volume closed". Open first, then close and swap — mirroring the
Rust adopt_folded_journal ordering.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ec.rebuild: fail the rebuild when a sibling journal cannot be repointed

A warning left the sibling mount journaling to a detached inode while
the RPC reported success. Propagate the error like the Rust handler's
adopt_folded_journal map_err.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ec: keep deletes loud when the journal reopen cannot open the replacement

Reverts the swap-on-success ordering: a detached handle kept on failure
lets deletes append and sync to the removed inode, then vanish at
restart. Closing first also frees the descriptor before the open.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: drop the folded journal handle before reopening it

adopt_folded_journal and the post-fold recreate now clear ecj_file
before opening the replacement, so a failed reopen leaves deletes
erroring instead of appending to the detached inode. Mirrors Go's
ReopenDeletionJournal.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ec: drop the cached journal size when its reopen fails

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-29 00:58:30 -07:00
5c92a54a12 remote: reuse one guarded remote storage client per configuration (#12189)
* remote: reuse one guarded remote storage client per configuration

BuildGuardedRemoteStorageClient (the filer's lazy remote stat and list,
remote deletes and renames, the volume server's cold reads, the S3
gateway's remote reads) built a new client, with its own http.Transport,
on every call. Each transport kept its connection idle for
IdleConnTimeout (60s) and was never reused or closed, so a steady stream
of remote lookups held one open connection per request. An endpoint
that caps connections per source then refuses every new connection.

- remote_storage: GetRemoteStorage's per-name cache becomes a bounded
  LRU (WEED_REMOTE_CLIENT_CACHE_MAX, default 64) keyed by kind, name, a
  hash of the whole configuration and a credentials version, built
  outside the lock once per key (singleflight) from a copy of the
  configuration. A client leaving the cache has its idle connections
  closed. GetRemoteStorageWith exposes the cache to other builders.
- BuildGuardedRemoteStorageClient caches the guarded client per
  configuration. The endpoint is checked against the deny-list when a
  client is built; every new connection still dials through the guarded
  dialer. A gcs credentials file, and for an S3-compatible remote using
  the default credential chain the shared credentials file and AWS_*
  variables, are part of the key by content.
- The shared transport keeps 32 idle connections per host, caps
  connections per host (WEED_REMOTE_MAX_CONNS_PER_HOST, default 64) and
  has dial, TLS handshake and response header deadlines, so a hung
  remote frees its slots.
- The filer drops the clients of removed remotes (and all but the newest
  client per name) on every /etc/remote reload, also when /etc/remote
  does not exist; reloads are serialized.

* remote: drop clients whose build spans a reload or whose remote is gone

* server: hash the shared AWS config file into the guarded credential version

* server: bound a stalled body on the shared remote transport

* remote: waive path-injection on the configured credentials file

* remote: keep caching remotes the local filer does not name

* remote: drop the inline codeql waiver the analyzer ignores

* remote: version the plain gcs client's credentials file

* remote: hash the environment's inline gcs credentials

GOOGLE_APPLICATION_CREDENTIALS can hold the JSON itself, which the gcs
client accepts; it is not in the configuration hash, so a changed value
kept the old client.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* remote: version the well-known gcs credentials file

With neither the configuration nor GOOGLE_APPLICATION_CREDENTIALS set,
the SDK reads application_default_credentials.json; rotating it left the
cache key unchanged and the old client behind.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* remote: share the gcs credentials version with the guarded client

The guarded gcs client resolves the same environment and well-known
credentials as the plain one; keying it only on a configured path left
those rotations invisible to the cache.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* remote: version s3 keys resolved from the environment

A remote whose access and secret keys both arrive through its type's
environment fallback (aliyun, baidu, contabo, filebase, storj, tencent)
never entered the default-chain branch of guardedCredentialsVersion, so
an in-process environment change kept serving the stale cached client.
The version now hashes the resolved keys for every S3-compatible remote.

* volume: reuse one remote storage client per configuration (rust)

fetch_and_write_needle built a fresh aws_sdk_s3::Client per request,
so every cold read kept its connector's idle connections without ever
reusing them — the same leak the Go side fixes. Cache clients keyed by
the configuration's protobuf encoding in an LRU of
WEED_REMOTE_CLIENT_CACHE_MAX entries (default 64), mirroring
weed/remote_storage/client_cache.go. An evicted entry drops the client's
last Arc and with it the connector's idle connections. The per-request
endpoint validation is unchanged: Rust has no dial-time guard yet, so
the up-front DNS check stays on every call.

---------

Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-29 00:57:37 -07:00
Chris LuandDevin 9e41500a3a mount: rebuild dropped directory listings and stale sections on lookup (#12195)
* mount: rebuild dropped directory listings on lookup

Invalidation, idle eviction, and a kernel Forget all clear a directory's
cached-listing flag, but only an expired listing was rebuilt on lookup — a
dropped one read through per entry until a ReadDir happened to run. Track
directories that once held a listing in wasCachedDirs, which survives those
drops, and let ShouldRebuildExpiredDir rebuild them on the next lookup.
Never-cached and read-through directories keep reading through.

* mount: re-validate a stale directory section on lookup

A burst of remote changes invalidates only the directory section it lands
in, but nothing re-validates that section until a ReadDir runs — so a
lookup-heavy workload reads through per entry indefinitely. EnsureNameFresh
re-lists just the section covering the looked-up name, after which lookups
in it are served from the cache again.

* mount: wait for pending flush before a stale-section refresh on lookup

A section refresh lists the range from the filer; without the wait a
pending async flush of the looked-up entry could land after the listing
and leave the refreshed section holding pre-flush metadata — the same
hazard the rebuild path already waits out.

* mount: bound the dropped-listing rebuild history

wasCachedDirs grew one full path per directory ever cached, removed only
by the read-through mark, so deleted and forgotten directories
accumulated forever. Remove the record when its name is deleted, move it
on rename so rebuild eligibility travels with the directory, and cap the
history with FIFO eviction — past the bound an evicted directory reads
through per entry, the same end state as a never-cached one.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* mount: drop oversized stale sections and back off failed refreshes

A section that outgrew one refresh failed EnsureNameFresh with
ErrRefreshRangeTooLarge on every lookup, repeating the multi-page
listing before each per-entry read, and other refresh failures retried
with no cooldown. Drop the cached listing on ErrRefreshRangeTooLarge so
a later lookup rebuilds it re-tiled, as ReadDir does, and stamp the
rebuild attempt on other failures so retries back off under the same
cooldown the rebuild gate uses.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* mount: version hard-link sibling syncs at the committed log position

* mount: tag dropped-listing queue slots so stale ones cannot evict

RemovePath, MovePath, and MarkDirectoryReadThrough delete the rebuild
record but leave the FIFO slot behind, so a path removed and re-cached
holds two queue entries. When the stale slot reaches the front it evicts
the newer record, and the directory reads through per lookup until the
next listing. Tag each record with an insertion generation and only
evict on a match, keeping the newer record for its own slot's turn.

* mount: refresh hard-link siblings from the filer, not the cached copy

Stamping a cached sibling with the mutation's version claimed that
position for content that may predate it — an in-flight event with
newer chunks or attributes then hits the version gate and the sibling
serves stale data until a refresh. Fetch each cached sibling's entry
from the filer after the commit and record the higher of the fetched
and committed positions, falling back to the in-place counter patch
only when the fetch itself fails.

* mount: back off failed section refreshes per section, not per directory

A failed refresh of one stale section marked a directory-wide attempt,
so every other stale section in that directory waited out the cooldown
with per-entry filer reads. Key the attempt timestamp by the section's
lower bound so unrelated ranges still refresh on their own schedule.

* mount: fence hard-linked entries by the group's newest write position

Every path of a hard-link group reads one shared content blob, but the
version gate only tracks per-path records — a sibling absent from
inodeToPath got no version from the sibling sync, so a stale section
refresh or event could roll the shared blob back for all of them.
Record a monotonic version floor per hard-link id wherever versioned
content lands, and check it in the section-refresh and event gates.

* mount: keep hard-link names landing when the group floor fences content

The per-group version floor rejected a delayed event's newEntry while
its oldPath still applied — a rename could drop the source without
adding the destination, and a section refresh could skip then mark a
live link fresh. Rebase the group's current cached content onto the
new path so the name lands without rolling the shared blob back, only
skip refresh entries already cached, and gate the sibling refresh
write under the same lock so a racing event or delete wins.

* mount: rebase hard-link entries from the shared blob when no sibling is cached

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* mount: park group content on uncached hard-link names a stale refresh installs

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* mount: stale the destination section when a fenced hard-link name cannot land

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* mount: bound the sibling hard-link refresh under one request deadline

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* mount: replace a reused-name occupant the hard-link floor would skip

* mount: claim no version for a sibling counter patched without a fetch

* mount: vacate a reused-name occupant without decrementing its group

A sibling write can fold the occupant's filer-side unlink into the shared
blob counter while the stale entry is still cached, so replacing it through
the plain insert may decrement a group that no longer counts the name and
drop a blob live siblings still read. Vacate the path like a rename first;
the counter then only ever drifts high, which leaks a blob rather than
losing shared content.

* mount: keep a fetch-failed sibling's version when patching its counter

The fallback marked the entry unversioned, which fences nothing — a
replayed delete older than the cached content could remove a live link.
Claim the version the entry already holds instead: newer in-flight events
still land and correct it, older replays stay fenced.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-29 00:55:47 -07:00
Chris Lu 8f6db360da admin: warn on dashboard when cluster nodes run mixed versions (#12196)
* admin: warn on dashboard when cluster nodes run mixed versions

A partially-finished upgrade leaves servers on different releases, and
nothing surfaces that drift today. The dashboard now compares the
binary version reported by every node:

- volume servers are asked directly via VolumeServerStatus (the
  heartbeat topology carries no version), fanned out with bounded
  concurrency and a 60s cache since versions only change on restart
- filers, S3 servers, and message brokers already report their version
  over KeepConnected; ListClusterNodes returns it, the dashboard just
  never read the field
- masters are excluded: raft membership does not carry versions

When every node matches, the Cluster panel shows the shared version.
When they differ, a warning banner calls out the mixed cluster and the
panel lists each node type with its count grouped by version, e.g.
"31x 4.39, 24x 4.44". Version columns on the volume server, filer, and
S3 tables (and the dedicated volume servers page) identify which nodes
are stale. Versions are compared without the volume-size-limit prefix
("8000GB 4.44 abc123" -> "4.44 abc123") so capacity config does not
masquerade as version drift; nodes reporting nothing count as
"unknown".

* admin: run the volume version queries concurrently, not under the map lock

The per-server VolumeServerStatus call ran while holding the versions
mutex, so the semaphore-bounded fan-out was fully serialized: every
server's RPC waited for the previous one, up to the 3s timeout each.

* admin: export the volume server table's real columns

The new Version cell shifted every later column, so the CSV read the
version as Rack and dropped Max Volumes, EC Shards and Usage a slot
early. Map the cells the table actually renders and add Version to the
export.

* admin: warn on mixed versions only when nodes report versions

An empty or master-only cluster produced a non-uniform summary with an
empty node list, which rendered the "Mixed cluster versions detected"
banner and warning icon over nothing. Mixed() requires reported nodes,
and the version card shows a dash when none answered.
2026-10-29 00:13:51 -07:00
Javier de la Torreanduser.email e9c280e095 iceberg maintenance: bound decimal, time, uuid, fixed and rescaled timestamp columns on compacted files (#12190)
* iceberg maintenance: bound decimal, time, uuid, fixed and rescaled timestamp columns on compacted files

The column statistics written for a compacted data file give lower and upper
bounds only for booleans, integers, floats, dates, strings, binaries and
timestamps already stored in the Iceberg type's unit. Every other column is
left unbounded, so a reader can never skip a compacted file on it: a
DECIMAL(18,2) column written by DuckDB (stored as INT64) loses its bounds
after compaction, as does a millisecond timestamp, a TIME or a UUID.

boundLiteral now also converts:
- DECIMAL stored as INT32 or INT64 at the column's scale. A decimal stored
  as a byte array still gets no bound: Parquet compares those statistics as
  bytes, which does not order negative two's-complement values;
- TIME, TIMESTAMP, TIMESTAMPTZ and TIMESTAMP_NS from any Parquet time unit,
  rescaled to the Iceberg unit; a value rescaled to a coarser unit is
  rounded down for a lower bound and up for an upper bound so it stays a
  bound, and a value that would overflow gets none;
- UUID and FIXED[n] from a FIXED_LEN_BYTE_ARRAY of the right length;
- a DOUBLE column read from a FLOAT file.

arrow-go becomes a direct dependency (decimal128, already in the module
graph through iceberg-go).

* iceberg maintenance: bound TIME(MILLIS) columns stored as INT32

* iceberg maintenance: bound timestamptz_ns columns too

---------

Co-authored-by: user.email <chrislusf@users.noreply.github.com>
2026-10-29 00:12:46 -07:00
Javier de la Torreanduser.email edf2b0f5ee iceberg maintenance: an unreadable parquet file fails the job, not the process (#12188)
* iceberg maintenance: an unreadable parquet file fails the job, not the process

parquet.NewReader panics on any file it cannot open (wrong magic, a
truncated footer, zero bytes), and parquet-go's row decoder panics on a
corrupt page under a valid footer. Maintenance runs in the worker's
goroutines with no recover, so one bad delete or data file in any table
ended the whole process.

- Every parquet open in compaction goes through newParquetReader, which
  opens with parquet.OpenFile and returns the error.
- A panic while decoding a delete file or a data file's rows is recovered
  into an error naming the file (recoverRead/guardRead).
- As a backstop, detection recovers per table and Execute per operation:
  the table is skipped or the operation fails, and the worker keeps
  running.

A failed delete file still fails the table's compaction (no commit), as a
read error did before; a failed data file still skips only its bin.

* iceberg maintenance: test a corrupt page inside a merge fails it naming the file

* iceberg maintenance: assert a skipped bin's files stay in the snapshot

---------

Co-authored-by: user.email <chrislusf@users.noreply.github.com>
2026-10-29 00:10:47 -07:00
Javier de la Torreanduser.email 55f2234026 iceberg maintenance: apply equality deletes only to older data files in their partition (#12187)
* iceberg maintenance: apply equality deletes only to older data files in their partition

Compaction applied every equality-delete key to every data file of every
bin. The Iceberg spec ("Scan Planning") applies an equality delete file only
to data files with a strictly lower data sequence number, and only to data
files in the delete file's own partition unless its spec is unpartitioned.

An upsert writer (Flink, the Iceberg Kafka Connect sink, CDC tools) commits
an update as the row's new version plus an equality delete of its key in the
same commit, so the compaction deleted both the old and the new version of
every updated row.

- equalityDeleteGroup is keyed by equality field ids and scope (unpartitioned,
  or spec id plus partition) and keeps, per key, the highest data sequence
  number of the delete files carrying it. A delete file without a data
  sequence number is an error rather than a guess.
- Partition values are compared typed through the spec's partition type, so
  one partition decoded in different Go shapes (a uuid as uuid.UUID or
  [16]byte, a long as int32 or int64, a null present or missing) is one
  partition. A spec whose fields are all void is unpartitioned. A delete whose
  scope cannot be decided (unknown spec, no partition values under a
  partitioned spec, a value that does not convert) stops the compaction; a bin
  whose own partition cannot be typed while a delete of its spec exists is
  not compacted.
- Each bin is merged with only the groups that apply to its partition, and a
  row is deleted only by a key whose sequence number is higher than its data
  file's.
- A where filter no longer drops unpartitioned delete files (the clause's
  field is not in their spec), which let deleted rows return in a filtered
  compaction.

Tests: the table fixture now commits its delete files after the data (data
sequence number 1, deletes 2), as writers do; hand-built entries in the unit
tests get sequence numbers. New eqdelete_sequence_test.go covers rows written
with and after their delete, partition scope, unpartitioned and all-void
specs, typed partition matching, missing sequence numbers and partition
values, where filters, and group-key collisions.

* iceberg maintenance: split the undecidable-scope branch in deleteInPredicateScope

---------

Co-authored-by: user.email <chrislusf@users.noreply.github.com>
2026-10-29 00:09:09 -07:00
Chris Lu ec43f4ca00 s3: accept bare composite checksum on CompleteMultipartUpload (#12194)
* s3: accept bare composite checksum on CompleteMultipartUpload

Clients such as minio-go send the composite object checksum without the
"-<parts>" suffix. Compare against the base64 part of the computed
composite value as well as the full form.

* s3: test object checksum matching with and without part suffix
2026-10-28 23:01:48 -07:00
Chris LuandDevin 4ed9595a58 admin: explain detection cooldowns and skipped scans (#12185)
* plugin: describe detection cooldowns and report skipped scans

* admin: expose cooldown eligibility and preserve skipped detection outcomes

* admin: show cooldown eligibility alongside maintenance lock waits

* plugin: retain cooldown activity for older admin servers

* admin: keep manual detection skips out of scheduler run state

* plugin: refresh descriptors on reconnect without downgrading saved schemas

* plugin: allow valid schema refreshes to repair corrupt descriptors

* admin: store eligible_after activity detail as RFC3339

* worker: report admin_script interval skips as skipped detection

* worker: report admin_script interval skips after the last completed job

* admin: show the selected job's cooldown while scheduling is disabled

* admin: report an unsigned interval skip instead of a completed scan

* worker: populate new DetectionComplete fields in Rust senders

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-28 22:55:36 -07:00
Chris LuandDevin cb993ab83b filer.redis3: accept username and keyPrefix options (#12192)
* filer.redis3: prefix every key via getKey, honoring keyPrefix

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer.redis3: read username and keyPrefix options into client config

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer.redis3: reject a username configured without a password

go-redis only sends AUTH when a password is set, so a configured username
with an empty password silently connects as the default user instead of
the intended ACL user. Fail loudly at initialize instead.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer.redis3: cover keyPrefix scoping in the store tests

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer.redis3: derive the test prefix and kv key from the per-run directory

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-28 22:54:07 -07:00
Chris Lu e7c1713457 admin: show maintenance lock blockers in scheduler UI (#12184)
* admin: expose maintenance lock waiters and current owner

* admin: display maintenance lock waits across scheduler pages

* admin: preserve the active lock message while acquisitions wait

* admin: synchronize lock fairness assertions with waiter registration

* admin: display lock wait start times alongside elapsed duration
2026-10-28 21:28:15 -07:00
6cd14f06c9 filer.remote.sync: upload a holed entry zero-filled instead of skipping it (#12178)
* filer.remote.sync: skip an event the filer has written past, not only rewritten

isSuperseded only looked for chunks the event named that the filer no
longer holds. A file copied in through a mount logs a create while the
copy is still running: the mount flushes metadata with chunk uploads in
flight, so that chunk list can have holes. ChunkStreamReader stops at
the first hole, the multipart upload sends a short part ("ContentLength=
8388608 with Body length 4194304"), and since every chunk the event
names is still live the event was never superseded. The daemon retried
it forever and the offset never moved past it.

Treat chunks the filer has added since the event the same way. The
event that added them follows in the log and uploads the finished file.

* filer.remote.sync: skip a holed snapshot even while the filer still holds it

A mid-write snapshot the writer never finished is neither deleted nor
written past: the filer holds it unchanged, so the superseded check never
fires and every retry reads short at the same gap, pinning the sync offset
forever. The upload can never succeed for that entry — skip it like a
superseded one. A finishing write logs its own event and uploads the full
content; a dead write has nothing readable to lose.

The check also covers a chunkless entry that is not a dropped local copy
(e.g. a truncated remote-only entry): its reader yields zero bytes against
a nonzero size, which can silently clobber the remote object with an empty
body where the backend does not verify the length.

* filer.remote.sync: upload a holed entry zero-filled instead of skipping it

Skipping the event left a completed sparse file absent or stale on the
remote, and the coverage check on the raw chunk list missed both gaps a
manifest's bounding extent hides and a FileSize smaller than the chunk
extent. ChunkReadAt already resolves manifests the way the upload reader
does and zero-fills every uncovered range up to the effective size, so
route the upload through it: a sparse file, a mid-write snapshot, and a
chunkless entry all send the bytes a read of the entry returns, and a
hole can neither wedge the event nor publish a truncated object.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer.remote.sync: skip a superseded entry on reader-construction error too

---------

Co-authored-by: Alex Russell <alex@sengine.cloud>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-28 19:39:31 -07:00
github-actions[bot] 07636335cf docs: regenerate star history chart 2026-10-29 01:08:49 +00:00
Jay Hemnanianduser.email 6225e0a705 s3: return x-amz-copy-source-version-id from CopyObject and UploadPartCopy (#12181)
* s3: return x-amz-copy-source-version-id from CopyObject and UploadPartCopy

* s3: cover x-amz-copy-source-version-id on latest copies and part copies

The explicit-version case was already covered; add a .versions pointer
fixture so the unversioned request resolves the latest version and
asserts the header on CopyObject, plus the same assertion on
UploadPartCopy.

* s3: carry the resolved id on a version file that lacks it

A .versions child written before the version-id attribute resolves to an
empty source version even though the pointer or the rescan already knows
it. Stamp the id on the returned entry so copy and get readers report it;
the version copy tests now drop the attribute to cover that shape.

* s3: report the null version in x-amz-copy-source-version-id

A regular-path source in a versioning-configured bucket is the null
version; echo it like every other version id, matching the GET path and
AWS. Unversioned sources still report no header.

* s3: report the version a self-copy retry actually read

---------

Co-authored-by: user.email <chrislusf@users.noreply.github.com>
2026-10-28 15:49:37 -07:00
48782e2b30 s3: allow PutBucketAcl when auth is off (#12180)
* s3: allow PutBucketAcl when auth is off

* s3: resolve the PutBucketAcl owner against the recorded owner

The unauthenticated path now resolves the account as anonymous, but XML
ACLs were still checked against it while GetBucketAcl reported an empty
owner for ownerless buckets, so a read-modify-write ACL update failed
with AccessDenied. GetBucketAcl and the XML check now both use the
recorded owner when the bucket has one and the resolved account
otherwise. The canned-ACL grantee moves to the same owner, and a request
carrying no account no longer erases a stored owner.

* s3: cover PutBucketAcl without auth for XML bodies and stored owners

The XML case posts the anonymous owner GetBucketAcl reports for an
ownerless bucket; the canned case asserts a recorded owner survives an
unauthenticated update and receives the full-control grant. Ambient AWS
credentials are cleared so the memory store never turns auth on.

* s3: keep the recorded owner through a PutBucketAcl update

The grantee and the stored owner were resolved differently: grants
named the recorded owner while config.Owner still took the caller's
account, so an admin updating another account's bucket rewrote the
owner and broke an XML retry naming the original one. An ownerless
bucket adopts the ACL's owner; an owned bucket keeps it.

* s3: fall back to the admin owner for an ownerless bucket in ACL handlers

An unauthenticated PutBucketAcl resolved the owner as anonymous and stored
it on the bucket, but the registry reports an ownerless bucket as
admin-owned: BucketOwnerEnforced uploads then stamped anonymous instead of
admin, and ownership checks after enabling auth compared against the wrong
owner. Both ACL handlers now resolve an anonymous fallback to the admin
account, so GetBucketAcl reports the same effective owner the registry
does and the stored owner matches it.

* s3: resolve the admin owner for bucket grants when the caller is anonymous or untrusted

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: skip recording anonymous identity or account as bucket owner

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: report stored bucket ACL grants verbatim

Explicitly saved anonymous full-control grants must survive a
GET-to-PUT round trip unchanged, so the read path no longer rewrites
them to the reported owner.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: expect the stored anonymous grant verbatim in GetBucketAcl

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-28 14:45:23 -07:00
Alex K fffdb7cc75 filer.remote.sync: skip an event the filer has written past, not only rewritten (#12176)
isSuperseded only looked for chunks the event named that the filer no
longer holds. A file copied in through a mount logs a create while the
copy is still running: the mount flushes metadata with chunk uploads in
flight, so that chunk list can have holes. ChunkStreamReader stops at
the first hole, the multipart upload sends a short part ("ContentLength=
8388608 with Body length 4194304"), and since every chunk the event
names is still live the event was never superseded. The daemon retried
it forever and the offset never moved past it.

Treat chunks the filer has added since the event the same way. The
event that added them follows in the log and uploads the finished file.
2026-10-28 14:44:36 -07:00
Chris LuandDevin 74f196ceab s3: route regular copy source reads to the write owner (#12173)
* s3: read regular copy sources from their write owner

* filer: report mutation sentinel codes in ObjectTransaction responses

A failed mutation only surfaced resp.Error text, so routed writers could
not distinguish an existing-directory conflict from a generic failure and
could not apply the prefix-object retry the plain CreateEntry path has.

* s3api: retry a routed mkfile as a prefix object over a directory

mkFile retries ErrExistingIsDirectory by marking the entry a prefix
object; the routed path returned the conflict instead, so a routed write
to a key that is also a prefix failed where the lock path succeeds. The
filer now reports the code on the transaction response, making the retry
recognizable.

* s3api: commit copy destinations on the key's write owner

An owner-routed source read still left the destination on the gateway's
default filer, so a second copy resolved the first copy's destination on
the owner, found nothing, and returned NoSuchKey. Write unversioned and
suspended destinations through routedMkFile, and write the version file
on the owner the finalize recomputes on.

* s3api: cache remote objects on the filer that served the entry

An owner-routed lookup can return a remote-only entry that has not
replicated to the gateway's default filer; the cache RPC then cannot
find it and the copy or read degrades to a 503. Ask the entry's write
owner to cache it first, falling back to the default filer.

* s3: read versioned object state from the key's write owner

CopyObject destinations now commit version files and the .versions
pointer on the owner's object-scope ring key, so versioned reads through
the default filer could miss them until replication. Generalize the
routed read to dir/name lookups and paginated lists scoped to the
object's owner, and point the versioned read paths at it:
getSpecificObjectVersion, doGetLatestObjectVersion, the .versions
quick-checks in Get/Head/attributes, the self-heal rescans, tagging,
delete conditionals, directory-marker history checks, rename
destination, multipart completion checks, and the version-id format
probe. Reads fall back to the default filer when no ring owner is
resolvable, preserving single-filer deployments.

* test(s3api): fake RECOMPUTE_LATEST/DELETE and cover versioned copy reads

The ambiguous-PUT fake filer now creates parent directories on PUT
(mirroring InsertEntry), applies DELETE mutations, and re-derives the
.versions pointer for RECOMPUTE_LATEST the way applyRecomputeLatest
does, so owner-vs-replica tests can exercise versioned writes.

TestVersionedCopyDestinationReadsOnOwner copies into a versioned bucket
whose writes commit on the owner while the gateway's default filer is a
replica with no replication, then asserts reads by versionId,
latest-version resolution, and a copy chained off the new version all
resolve on the owner.

* test(s3api): gofmt the versioned-copy coverage

* s3: only swap in a cache fill of the resolved remote generation

A cache response can come back from a filer holding an older generation
of the key. Replacing the authorized source or swapping its chunks under
a size and range computed for the resolved entry serves the wrong object
or truncates the response. Match RemoteEntry identity and size before
using a cached entry, in both the copy and streaming paths.

* filer: keep directory attributes when an object lands as prefix object

The object-onto-directory retry replaced the existing directory's mode,
owner, and extended attributes with the object's. Preserve the directory
identity; merge directory-held extended keys only when the old entry
never held an object, so a prior generation's object metadata is not
carried forward.

* s3: keep marker lock checks on the filer that replaces the marker

The object-lock precondition read the key's write owner while mkdir
still replaces the marker on the default filer, so a locked marker not
yet replicated to the owner could be overwritten. Read the entry where
the replacement lands, in both PutObjectHandler and zip extraction.

Also hash a trailing-slash marker key to the same route key as the
object it names, so marker and object share one owner and one write
lock.

* s3: route tag writes to the owner tag reads resolve

GetObjectTagging reads the key's write owner, but PutObjectTagging and
DeleteObjectTagging still wrote through the default filer, so a
successful change returned the owner's stale tags until replication.
setTags, rmTags, and the versioned tag updates now write on the owner
first, falling back to the default filer when no owner resolves.

* s3: keep rename lookups on the filer that performs the move

AtomicRenameEntry and the retry token it carries both execute on the
default filer, but the source and destination reads went to the write
owner — a completed rename's token was invisible to its own retry, and
a source that only exists on the default filer looked absent. Read
both ends on the default filer.

* s3: send the .versions pointer repair with the object's route key

The heal reads the .versions entry on the object's owner, but the
conditional UpdateEntry is routed by the .versions path's own ring key,
which can hash to a different filer — leaving the pointer the routed
reads consult unchanged. Ship the CAS repair as a PATCH_EXTENDED
ObjectTransaction on the object's route key instead.

* filer: preserve only directory identity on a prefix-object write

Merging the old directory's extended attributes could carry a marker's
encryption, version, or etag fields onto the new plaintext object, since
a directory's own keys cannot be told apart from the object it held.
Keep only mode, uid, gid, user name and group names.

* s3: also require matching size when a cache fill replaces the source

The remote-generation check covers identity; size equality is the cheap
guard that keeps the streamed or copied range consistent with what was
resolved.

* s3: lock the object path for a routed .versions pointer repair

conditionalUpdateEntryRouted held the lock on the .versions path while
version writes and deletes serialize on the object path, so a repair
could pass IF_ENTRY_EQUAL and then apply over a pointer a writer had
advanced in between. Lock the object path like the writers and keep the
condition on the .versions entry via condition_key.

* s3: do not replay an owner-side tag mutation on another filer

withWriteOwnerFilerClient re-ran fn on the default filer after any owner
error, so an authoritative not-found could update a stale replica entry
and still report success, and a post-apply transport error could apply
the mutation twice. The failover walk now ends on any error fn returns;
only a channel that never came up moves to another filer.

* filer: keep a directory's xattrs when a prefix object replaces it

The prefix-object branch preserves the directory's identity but dropped
its Extended map wholesale; FUSE xattrs live there under the xattr-
prefix and belong to the directory, not to the object the marker held.

* s3: delete null and specific versions on the filer that serves them

On the default filer an absent entry deletes as success, so a version
that only reached the write owner survived a reported success while
owner-routed reads kept serving it. The suspended-versioning and
lifecycle deletes now go through routedDeleteNullVersion and
routedDeleteSpecificVersion when an owner resolves; an unsettled
attempt keeps the lock path.

* s3: route a lifecycle null-version delete to the object entry

The "null" version is the regular object entry, not a .versions file,
so it goes through routedDeleteNullVersion; only a real version id takes
the .versions-file delete.

* s3: repoint .versions only while it still names the deleted version

routedDeleteSpecificVersion recomputed the latest pointer on every
version delete, so removing a noncurrent version re-derived the pointer
to the newest remaining file — resurrecting an older version over a
cleared pointer (a suspended bucket's null write keeps it clear while
the null object stays current) or over a pointer that moved on. The
pointer is now read on the owner and repointed under an IF_ENTRY_EQUAL
guard only while it still names the deleted version; a contested pointer
is left for the self-heal path rather than overwritten by a stale scan.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: let lifecycle expiry keep its metadata-only deletes on routed paths

The routed lifecycle deletes hardcoded IsDeleteData, so expiring an entry
with a volume TTL queued per-chunk deletions instead of leaving the data
for the volume to reclaim — extra work on the owner for deletes
recordMetadataOnlyIf still counts as metadata-only. Pass the caller's
cleanup choice through routedDeleteNullVersion and
routedDeleteSpecificVersion; the interactive DELETE path keeps deleting
data.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: let RECOMPUTE_LATEST skip a pointer that moved on

A version delete repoints .versions only while the pointer still names the
deleted version, but the gateway cannot carry the pointer's current state
into the transaction's condition — the condition slot is already spent on
the version's WORM guards. only_if_current_name moves the check under the
transaction lock: the filer compares the pointer's name_to_key value before
scanning and skips the recompute when it moved (or was cleared by a
suspended bucket's null write), so a noncurrent delete cannot resurrect an
older version.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: repoint inside the version's locked delete

repointLatestBeforeVersionDelete ran in its own transaction ahead of the
delete, so a version delete denied by retention or legal hold still moved
the latest pointer — GET and HEAD went on serving the older version it
rolled to. The repoint and the delete now ride one transaction behind the
version's WORM guards, atomically gated by the new only_if_current_name
recompute guard instead of a stale-read IF_ENTRY_EQUAL.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-28 14:35:39 -07:00
Chris Lu 8daa1b6ddd vector: Lease::forget hands the tag name off without releasing (#12174)
* vector: Lease::forget hands the tag name off without releasing

A distributed planner pins the planned version and forwards the tag
name to the executor — the guard must not release on drop.

* vector: forget untracks the lease so an abandoned handoff expires

The tag left LIVE_TAGS but the drop-time release was disabled — an
abandoned handoff survived forever locally. Untrack it so the lease
TTL still reaps it.
2026-10-28 09:22:22 -07:00
Chris Lu 511727cf3a vector: DeleteIndex format guard, sticky discovery, advertise normalize (#12171)
* vector: DeleteIndex format guard, sticky discovery, advertise normalize

- handleDeleteIndex verifies the table carries the Lance format before
  DeleteTable — an Iceberg table under the same namespace is not an
  index and must not be removed by it.
- discovery keeps the current worker while it is still registered —
  membership-order churn can no longer cancel in-flight calls, and the
  resolver closes each master conn inside a 5s deadline.
- advertised_addr strips scheme and .grpcPort so the UDP probe reaches
  every supported master form.

* vector: empty-follower retry, format check without read grant

- discoverVectorWorkers keeps looking past an empty master reply —
  followers list nothing when workers registered with the leader.
- the DeleteIndex format check uses the internal catalog read, so
  DeleteTable-only callers don't gain a GetTable requirement, and the
  format must be Lance exactly — format-less tables no longer pass.
2026-10-28 09:22:05 -07:00
Chris Lu ae95b2c777 ec: rebuild a needle's suspect shard ranges when it fails to parse (#12170)
* ec: let shard-interval recovery exclude a set of suspect shards

gatherEcShardIntervals and recoverOneRemoteEcShardInterval skipped only
the shard being rebuilt. A needle-level rebuild wants every suspect
shard kept out of the gather so a bad copy cannot poison the
reconstruction it is being compared against.

* ec: rebuild a needle's suspect shard ranges when it fails to parse

A bit-rotted data shard serves a clean-looking but wrong interval, so the
missing/unreadable-shard fallback never engages and the needle CRC check
in ReadEcShardNeedle returns the corruption to the caller. Each interval's
source shard is a suspect: rebuild the suspect ranges from the surviving
shards — all of them at once while parity covers the suspect set, or one
at a time when the needle spans more shards than that — and let a
successful needle parse arbitrate which copy was corrupt. Shards whose
range changed are dropped from the location cache so later reads go
straight to reconstruction.

* ec: mirror the corrupt-shard needle rebuild in the Rust volume server

The snapshot now keeps every interval's shard placement so a parse
failure can name its suspects. rebuild_ec_shard_needle rebuilds the
suspect ranges through recover_one_remote_ec_shard_interval with the
suspect set excluded from the gather, and the winning candidate's
differing shards are dropped via forget_shard_location.

* ec: recover suspect ranges under the requested needle id

A failed ReadBytes re-reads the id out of the possibly corrupt header, so the rebuild passed whatever it decoded to peer recovery requests; carry the requested id through instead, and reject a candidate whose parsed id does not match it.

* ec: retry a failed needle rebuild with each gathered input dropped

The gather behind a suspect-range rebuild can itself serve a second corrupt shard: Reed-Solomon then reconstructs wrong bytes and the needle parse rejects the only candidate. Gather every surviving shard once, then re-decode with each of the first decode's inputs dropped in turn; the parse arbitrates the honest subset.

* ec: mirror the dropped-input rebuild and id check in the Rust volume server

Same two guards as the Go side: a candidate that parses under a different needle id is rejected, and a failed candidate whose gather fed it a corrupt input is retried by dropping one gathered shard at a time. The gather is shared through a want-bounded gather_ec_shard_intervals so the retry pays no extra fan-out.

* ec: answer deleted when a rebuild's holder reports the needle deleted

The rebuild discarded the deleted flag its interval recovery collected, so a needle deleted on its other holders but live in the local index came back as a corruption error. A deleted report is authoritative, so it now short-circuits the hypothesis search and answers ErrorDeleted.

* ec: hydrate rebuild candidates into a scratch needle

A failed ReadBytes leaves partially parsed fields on the needle, and optional sections do not reset when absent, so a corrupt first parse could leak stale name/mime/ttl into a successful candidate. Parse each candidate into a fresh needle and adopt it only on a matching id.

* ec: corrupt parity+1 shards in the exceeds-parity read test

The loop skipped the suspect's own shard when it fell inside the parity range, leaving only ParityShards corrupt shards — a case recovery can legitimately fix. Count the additional corruptions so the test always exercises the boundary it names.

* ec: mirror the deleted-report short-circuit in the Rust volume server

A holder's deleted answer during a rebuild propagates out as EcMiss::Deleted instead of surfacing as InvalidData, matching the Go path's authoritative-delete rule.
2026-10-28 08:45:57 -07:00
Chris LuandDevin eec2452e30 master: replicate the recovery-wait growth unfreeze through raft (#12165)
* master: replicate the recovery-wait growth unfreeze through raft

state_loss_recovery_wait lifted the volume-growth freeze only in
memory, while the freeze itself was replicated. A restart replayed the
frozen flag and growth stayed refused until the wait expired again, on
every restart, unless an operator ran volume.growth.resume.

The lift now applies GrowthFrozen=false like ResumeVolumeGrowth, from a
goroutine since IsGrowthFrozen is a read path that cannot block on
raft. The apply runs under reportedTopologyIdsLock so a freeze landing
after the lift keeps its command last in the log; the stale unfreeze
skips itself when the flag is frozen again.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* master: cover raft replication of the recovery-wait unfreeze

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* master: retry and re-arm a growth unfreeze raft failed to record

applyGrowthFreeze swallowed the raft error, so a rejected lift during a
leader change left the flag clear while the log still held the freeze;
nothing retried and a restart re-froze growth anyway. The apply now
returns the error, replicateGrowthUnfreeze retries it, and on
persistent failure re-arms the flag so the next IsGrowthFrozen
attempts the lift again instead of diverging from the log.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* master: release the topology-id lock before the freeze raft write

CheckReportedTopologyId and SetTopologyInitPending held
reportedTopologyIdsLock across the blocking raft apply, so a slow
apply stalled every later report — including the heartbeat that
should trigger the next freeze. The lock now covers only the
mismatch decision; the replicate step runs after it is released.

* master: bind the recovery unfreeze to the freeze instant it lifts

An unfreeze submitted while a newer freeze sat committed but not yet
applied — e.g. after this node won an election — could land behind
that freeze in the log and erase it. The command now carries the
freeze instant cluster-wide (GrowthFrozenAt) and the lift is bound
to the instant its submitter observed (GrowthUnfreezeFor): the state
machine no-ops a lift while a strictly newer freeze exists, which
also lets the unfreeze run without holding the topology-id lock at
all. Concurrent freeze writes keep their own mutex so log order
cannot regress the instant.

* master: a superseded unfreeze retry leaves the resumed flag alone

replicateGrowthUnfreeze broke out of its retry loop when a newer\nfreeze superseded the lift, but the break fell through to the\nre-arm: an operator resume in between was silently undone. Return\ninstead — the newer freeze generation owns the flag.

* master: replayed freeze entries without an instant stay ageless

Freeze commands written before the instant field existed carry zero.\nSubstituting the replay time made each restart stamp the freeze as\nnew, so a committed bound unfreeze always looked older and was\nrejected — the freeze came back on every restart. Keep the zero\ninstant: the recovery window reads as elapsed and bound lifts apply.

* master: pack the growth freeze flag and generation into one atomic

Separate flag and timestamp atomics left three holes: a resume's\nload-and-clear could lift a generation stamped in between, a bound\nlift compared stamps numerically although masters' clocks need not\nbe monotonic, and an ageless generation had no identity a bound lift\ncould match without colliding with the unconditional encoding.\n\nOne value now carries the state — 0 unfrozen, -1 an ageless freeze,\nelse the freeze instant — so every transition is a single atomic\nswap or compare-and-swap bound to an exact generation.

* master: a repeat mismatch report keeps the armed freeze generation

Re-stamping while already frozen never reaches the raft log, so the local generation diverges from the replicated one and every bound unfreeze becomes a no-op on replay — the freeze returns after restart or failover.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* master: snapshots read the freeze flag and generation in one load

Save, Snapshot, and PersistMaxVolumeId read IsGrowthFrozen and the generation separately, so a resume landing between them persisted frozen=true with stamp 0 — restoring an ageless freeze a committed bound lift cannot clear.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-28 08:44:45 -07:00
Nguyễn Lộc Phúcandchrislusf b62bbb99fc fix(iceberg): serve a registered table's metadata on load and commit (#12159)
* iceberg: serve a registered table's metadata on load and commit

RegisterTable saved only the metadata location, and loads and commits
never read the file there. They served a placeholder with no schema, no
snapshots and a new random table UUID on every load, so a registered
table read as empty, an INSERT failed against the empty schema, and
every other commit failed its assert-table-uuid requirement. Metadata
uploaded through the S3 gateway, as a migrated table's is, is stored in
chunks that registration did not read, so it registered an empty table.

Read the file following its chunks, reject one that is not Iceberg
metadata with a 400, and keep its content and table UUID in the record
the way CreateTable does.

Records registered before this change hold only the location. Loads and
commits now read the file there, and the first commit stores it in the
record under the file's table UUID.

* iceberg: transactions read a location-only registration's metadata file

* iceberg: cover transactions on a location-only registration

* iceberg: refresh the default name mapping on transaction commits

* iceberg: cover name-mapping refresh in transactions

* iceberg: roll a transaction back to a registration's metadata file

A transaction that failed after flipping a table registered before the
record kept its metadata could not undo the flip. The rollback restored
the record's prior metadata, which for such a table is only a location,
and UpdateTable keeps the full metadata a flip stored when the restore
sends none. Loads went on serving the change the client was told had
failed, and a client retrying the transaction could apply it twice.

Restore the metadata loads served from the file instead, under the
file's table UUID.

* iceberg: register oversized metadata by its location

Registration sends the metadata file's content to the S3 Tables
manager, which refuses a request over its 10MB body limit, so a table
whose metadata is larger could no longer be registered at all.

Register such a file by its location and table UUID alone; loads read
it from that location, as they do for a record that never kept its
metadata. Export the limit as s3tables.MaxRequestBodySize so the
catalog can check against it. A commit to such a table still sends the
whole metadata to UpdateTable and stays over the limit.

* iceberg: commit oversized metadata by its location

* iceberg: let maintenance follow a location-only table record

* admin: read a table's metadata file when the record keeps none

* iceberg: bound the metadata file a location-only record reads

* iceberg: bound only chunked metadata reads in the worker

Inline content is what the catalog itself wrote and the REST load path
accepts it at any size, so rejecting it in loadMetadata stranded tables
over 64 MiB from every maintenance scan. Cap only the chunked read, where
the recorded location can name any object in the bucket.

---------

Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
2026-10-28 08:15:42 -07:00
Jay Hemnanianduser.email a816c6202c sftp: fail rm and rmdir of a missing path, and mkdir of an existing one (#12168)
* sftp: fail rm and rmdir of a missing path, and mkdir of an existing one

* sftp: create directories exclusively

---------

Co-authored-by: user.email <chrislusf@users.noreply.github.com>
2026-10-28 08:11:21 -07:00
Chris LuandDevin 3fd8d0cf09 master: let location-constrained assigns wait for pending growth (#12167)
* topology: add AvailableSpaceWithinLocation for location-scoped free slots

AvailableSpaceFor answers for the whole topology; a location-constrained
assign needs to know whether the requested data center, rack, or data node
still has room before deciding growth can satisfy it.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* master: let location-constrained assigns wait for pending growth

An assign carrying a data center or rack constraint failed on the first
unsatisfied pick, even when the same request had just queued volume growth
for that location — so the first write into a fresh collection errored out
instead of picking up the volume growth landed a moment later.

Keep the fail-fast path only when growth cannot help: growth disabled, no
growth pending, or no free slots left inside the constraint. Otherwise the
request takes the same wait/shed path as an unconstrained assign.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* topology: sum leaf free slots in AvailableSpaceWithinLocation

Rolled-up rack and data center counters keep a maintenance-mode server's
capacity, and one overcommitted node can cancel a sibling's slots. Sum the
matching data nodes directly so the location check agrees with what growth
can actually use.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* master: let constrained growth initiators reach the shed path

When a constrained assign's own growth concluded unfulfilled, the fast-fail
check saw only that no grow was pending and returned the plain pick error —
a code the assign client does not retry, so the first write died instead of
re-triggering growth. Exclude the request's own initiated grow from the
no-pending-growth condition so it reaches the existing re-pick and
ResourceExhausted shed like an unconstrained assign.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* master: judge targeted grow requests by the location they name

A grow request carrying a data center, rack, or data node constraint was
discarded whenever the layout's writables elsewhere were healthy, so a
location with free slots but no writable could never receive the volume its
assign was waiting for. Add ShouldGrowVolumesWithinLocation and use it for
constrained requests; the layout-wide check still governs untargeted ones.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* topology: only positive leaf slots count in AvailableSpaceWithinLocation

An over-committed node reports negative free slots; summed plainly it\ncancelled a sibling's room and a constrained assign could fail fast\nwhile the location still had space to grow into.

* master: fail constrained assigns fast when growth cannot place

A constrained assign with free slots in its location still waited out\ngrowth that could never satisfy the replica placement — a single-rack\ndata center asked for 010 sheds the same doomed grow on every client\nretry. Run the grow-time node search as a dry run and return its\nplacement error instead of a retryable shed.

* master: judge constrained growth by layout, not free slots

A slot-level feasibility probe races with the grow it judges: a\npending grow consuming the last slot in one rack reports the\nplacement impossible while it mints the request's own volume, and\nthe hard error skips the re-pick that would find it. GrowthBlocker\nnow counts racks and data nodes inside the named location — shape a\ngrow cannot consume — so only genuinely unplaceable layouts fail\nfast.

* topology: GrowthBlocker mirrors the grow selector's feasibility

The location filter bound every replica tier, so a 010 request naming one rack needed two racks inside it while the selector constrains only the first placement and counts the rest by room. And counting nodes without free slots let a full location queue a grow it could never serve, so the assign shed ResourceExhausted at timeout instead of failing fast. GrowthBlocker now applies the option's location to the primary placement chain only and leaf-sums free slots the way the selector does, so it reports a blocker exactly when the grow can never place.

* master: repick once before surfacing a growth blocker

An in-flight grow can spend the last free slot minting the request's own volume: the blocker's slot counts predate that registration by a heartbeat, so returning the placement error on the first failed pick could reject a write whose volume lands a moment later. A pending grow now earns one repick grace before the blocker error is returned.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-28 08:10:45 -07:00
Chris LuandDevin 18691e7b5b server: keep the first write on a fresh weed server from stalling (#12149)
* master: end the post-mint allocation hold on the embedded reporter's clean inventory

DeferAllocation armed a warmup-length hold whenever the leader minted a
fresh topology id, and on weed server the hold ran the whole window: the
embedded volume server's join kept refreshing the quiet evidence clock, so
the two-pulse early release still cost the first write 10-15s on every
fresh start. weed mini already releases on its single pinned reporter's
first inventory, but pinning refuses outside volume servers, which weed
server must keep admitting.

Add a non-exclusive variant: the topology names one expected reporter
whose first clean full inventory lifts the hold, while other reporters
stay admitted and still feed prior-state evidence and the topology-id
freeze. A single-master weed server points it at its embedded volume
server — the one reporter guaranteed to arrive quickly; a veteran's
heartbeat carries prior state and keeps the window through the regular
checks, so a wiped -dir with surviving external volume servers still
trips the freeze on first report. Restarted clusters are untouched: a
recovered topology id means no mint and no hold.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: announce on the startup peer scan for embedded single-master setups

A fresh filer whose startup peer scan came back empty re-polled for the
full confirm window before registering on the master, so the lock ring
had no member and routed traffic such as S3 PUTs spun on "no lock server
found" for ~10s on every fresh weed server / weed mini start.

For setups that start with exactly one filer the scan is already final:
a single-master weed server and weed mini announce as soon as it
completes. A late external filer still joins and replays through the
normal path — the confirm window only ever caught peers arriving inside
it, and the residual risk mirrors the accepted volume-side trade-off: an
embedded filer announcing a few seconds earlier can own the lock ring
while a surviving external filer's backlog is still on its way.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: retry a failed heartbeat promptly until first registration

A heartbeat stream refused before the leader is known — the common race
when a volume server starts while an election is still running — always
fell back to sleeping out the full pulse, so a fresh volume's first
registration could wait ~5s behind a leader elected a second later.

The peer refresh already identifies the leader during the wait, so cap
the retry pause at a second once a leader is named; while no leader is
known the same short retry applies until the first stream is confirmed,
since an unregistered server has nothing else to do. Once the server was
registered the pulse backoff is unchanged, so a lost leader still paces
reconnects the way it did before.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: retry a failed heartbeat promptly until first registration (rust)

Mirror the Go heartbeat retry: cap the post-refresh wait at a second when
a leader is named, and keep the short retry until the first stream is
confirmed so a fresh volume registers promptly when it races an election.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* master: leave admitted reporters a confirm pulse when the embedded report ends the hold

The single-master weed server releases its post-mint allocation hold on
the embedded volume server's clean report, but outside volume servers
stay admitted and one may still carry prior cluster state — an inventory
from the embedded server alone cannot prove none is due. Clearing the
hold outright let a rebuilt master re-issue a volume id a reconnecting
veteran still owned.

Collapse the released hold to one heartbeat interval instead: late
reporters land their evidence or a topology-id freeze inside the pulse,
and a re-mint re-arms that pulse rather than the full warmup. The pinned
single-reporter release (weed mini) still drops the hold outright, since
no other reporter can be admitted.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: keep the empty-membership confirm on embedded single-master weed server

A single-master weed server still admits external filers, so an empty
startup peer scan is not final: after a master restart the existing
filers simply have not reconnected yet. Announcing on the scan let the
embedded filer take lock-ring ownership and serve writes that a peer's
replayed backlog could then overwrite.

Only weed mini keeps the skip, where the embedded filer is the only
member its dev cluster can have. The confirm window only delays the
master listing for a fresh store; the filer serves while it runs.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: tolerate an expired heartbeat retry deadline (rust)

refresh_leader_from_peers can consume the whole pulse budget, leaving
retry_at in the past; subtracting a later Instant panics and kills the
heartbeat task instead of retrying. Match the Go side, where
time.Until of a past deadline just sleeps zero.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: retry the seed list without a second pulse sleep (rust)

The error path already slept its retry delay; an extra full pulse after
losing the round delayed the next attempt past what Go's heartbeat()
does, which paces retries only inside the per-master attempt.

* master: keep a configured allocation hold shorter than the confirm pulse

Once the embedded report lands, the collapsed hold replaced the
configured deferral with a full pulse even when the operator set a
shorter one, including zero (disabled). Cap at the pulse instead.

* filer: end the empty-membership confirm early on a fresh-minted embedded master

On a fresh single-master weed server the announce gate still held the
filer for the full confirm window, so the master's lock ring stayed
empty and the first write retried 'no lock server found' for ~7s on
top of the collapsed allocation pulse. A fresh TopologyId mint proves
the master's persisted state was empty, making the empty peer scan
final; skip the rest of the confirm only then, and keep it whenever
the master recovers a prior id.

* filer: keep a short reconnect window on a fresh-minted embedded master

A fresh TopologyId mint proves the master's persisted state was empty,
not that no external filer survived its loss — one can still be
mid-reconnect on its keepalive cycle when the confirm starts. Ending
the empty-scan confirm on the first poll let the fresh filer register
and serve routed reads before that peer was found and backfilled, so
existing files could read as missing. The fresh-mint path now shortens
the confirm to a few keepalive cycles instead of skipping it: a
reconnecting filer still lands in time, while a genuinely fresh cluster
announces after ~3s rather than ~10s.

* volume: skip the heartbeat loop when no seed masters are configured

An empty seed list made the outer loop spin with no await between
passes and no path to observe the stop flag. A masterless volume server
has nothing to register with, so the heartbeat exits after a warning.

* volume: skip the heartbeat loop when no master addresses are given (rust)

Mirrors the Go heartbeat: an empty seed list spun the outer loop with
no await and no shutdown checks. Exit after a warning so library
callers get the same behavior as the binary path that validates the
list up front.

* filer: re-check the fresh-mint hint on each confirm pass

The topology id is minted after raft election, which can land after
the empty-scan confirm has already started — a one-time check at entry
missed the mint and held the full window. Evaluate the hint on every
poll and shorten the confirm from the first observed mint, still
capped by the normal confirm deadline.

* master: mark a fresh topology mint only when the volume bound is zero

Recovery restores the max volume id unconditionally but the topology id
only when present, so an upgraded or partially rebuilt master can mint a
new id while still holding prior volume state. Marking fresh there would
let the embedded filer shortcut its peer confirm on a cluster that is
not empty; only mark the mint fresh when no bound survived.

* filer: drop the fresh-mint shortcut once a peer has been observed

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-28 08:08:26 -07:00
Jay Hemnanianduser.email 9059cc2a07 s3: allow PutObjectAcl when auth is off (#12155)
* s3: allow PutObjectAcl when auth is off

* s3: ownerless objects claim the admin account when auth is off

An object written through the filer carries no owner metadata, so
objectOwner fell back to an empty account id and ExtractAcl rejected the
canned ACL's canonical grantee. Extend TestPutObjectAclWithoutAuth to
cover the ownerless case.

* s3: GetObjectAcl reports the admin owner for ownerless objects when auth is off

The same empty-owner fallback produced an ACL response with a blank
Owner ID and a blank canonical grantee.

* s3: ignore the client-supplied account id when auth is off

Auth does not strip AmzAccountId on the iam-disabled path, so the
ownerless fallback could persist a forged owner. requestAccountId
substitutes the admin account, matching the idiom already used in the
upload ACL path.

---------

Co-authored-by: user.email <chrislusf@users.noreply.github.com>
2026-10-28 08:07:36 -07:00
Chris Lu b3d0caabe2 vector: only follow a leader hint that points elsewhere (#12164)
* vector: only follow a leader hint that points elsewhere

The master sends volume_location on every KeepConnected response —
the leader always names itself, so the worker exited the stream on
the first response and flapped every 3s, leaving ListClusterNodes to
see the node intermittently. Only redirect when the hint points at a
different master.

* vector: leader hints carry host:http.grpc — normalize before comparing

A self-hint arrives as host:9333.19333 while --masters gives host:9333,
so the raw comparison still redirected and grpc_addr could not parse
the dotted port. grpc_addr now prefers the explicit .grpcPort, and the
hint comparison normalizes both sides to host:httpPort first.
2026-10-28 08:07:23 -07:00
Chris Lu c2863d9a4f vector: owner-prefixed lease tags stop cross-server reaping (#12158)
* vector: owner-prefixed lease tags; foreign leases get a long backstop

reap_expired exempted only process-local LIVE_TAGS — server B could
reap server A's live query lease at the 1h TTL. Tags now carry a
per-process instance id: own leases use the short TTL, foreign leases
only the 24h backstop, so a live query elsewhere keeps its pin while
dead-process tags still collect eventually.

* vector: pick the lease-tag format by segment count, not numeric-ness

An owner that is all decimal digits parsed as the legacy timestamp, so
its real timestamp was ignored and the lease never expired. The third
segment is what distinguishes {instance}-{ts}-{uuid} from {ts}-{uuid}.
2026-10-28 01:49:49 -07:00
Chris Lu 7a25983266 vector: cover index names in named_segment_fragments (#12161)
vector: cover the index name in named_segment_fragments

The planner groups tasks by index name — assert the helper actually
returns the logical name so a regression dropping it fails.
2026-10-28 01:48:24 -07:00
Chris Lu d5433ae745 vector: cover the identity object in s3vectors auth (#12162)
vector: cover the identity object in the s3vectors auth context

The auth middleware must carry the resolved identity object — only
the name survives otherwise and authorization loses actions, account,
and claims.
2026-10-28 01:47:12 -07:00
Jay Hemnanianduser.email 9b3f09d0a5 sftp: refuse mkdir and uploads under a missing folder (#12156)
* sftp: refuse mkdir and uploads under a missing folder

* sftp: re-check the parent directory when a buffered upload commits

The open-time check cannot catch a parent removed while the upload
buffered; the filer would recreate it when the entry lands.

* sftp: cover mkdir and upload commit against a missing parent

Extend the test filer with CreateEntry so makeDir can be exercised end
to end.

* sftp: refuse a rename into a missing folder

The filer's rename path creates the target entry through CreateEntry,
which recreates absent parents the same way uploads did.

---------

Co-authored-by: user.email <chrislusf@users.noreply.github.com>
2026-10-28 01:45:08 -07:00
Chris Lu afb8472c5e s3vectors: cover non-filterable key round-trip and nested filter rejection (#12160)
* s3vectors: cover non-filterable key round-trip and nested filter rejection

The catalog writes the key list as a JSON string while older records
may still carry the array form; indexFromRecord must decode both. The
rejection path extracts field names through nested $and/$or filters, so
a blocked key deep in the tree still fails the query.

* s3vectors: factor the non-filterable rejection into a testable gate

Pull the blocked-key check out of handleQueryVectors into
nonFilterableField so the rejection itself is covered without standing
up the catalog stack; the handler now 400s on whatever the gate names.

* s3vectors: report the non-filterable match separately from the key

An empty metadata key in nonFilterableMetadataKeys collided with the
"no match" sentinel, so a filter on that key skipped rejection. Return
a match flag and cover the empty-key case in the test.
2026-10-28 01:41:37 -07:00
Chris Lu 7d823265e5 wdclient: regression test for the elapsed caller-deadline lookup path (#12154)
* wdclient: cover a lapsed caller deadline that Err() has not observed yet

* wdclient: reuse isCanceledErr for the elapsed-deadline check

* wdclient: reinterpret only deadline errors as caller-gave-up

isCanceledErr also matches gRPC Canceled, which a filer can return for
its own reasons; treating it as an elapsed caller deadline hid the
failure from the filer's health record. The fallback now matches only
context and gRPC DeadlineExceeded, so filer-side cancellations keep
counting as filer failures.
2026-10-28 01:08:04 -07:00
dependabot[bot]andchrislusf 2b7c65048d build(deps): bump com.google.guava:guava from 32.0.0-jre to 33.7.1-jre in /other/java/client (#12147)
* build(deps): bump com.google.guava:guava in /other/java/client

Bumps [com.google.guava:guava](https://github.com/google/guava) from 32.0.0-jre to 33.7.1-jre.
- [Release notes](https://github.com/google/guava/releases)
- [Commits](https://github.com/google/guava/commits)

---
updated-dependencies:
- dependency-name: com.google.guava:guava
  dependency-version: 33.7.1-jre
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* other/java/client: bump guava to 33.7.2-jre

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
2026-10-28 01:04:45 -07:00
Chris Lu 237db29dca vector: round-trip and enforce non-filterable metadata keys (#12157)
vector: non-filterable metadata round-trips and filters reject it

- GetIndex omitted nonFilterableMetadataKeys — the catalog stores the
  list as a JSON string but the reader only accepted an array; both
  forms now decode.
- QueryVectors now rejects filters referencing non-filterable keys
  instead of silently scanning them.
2026-10-28 01:02:41 -07:00
Chris Lu 190ab32cfb vector: weed-vector registers to master; s3 discovers it there (#12146)
* vector: weed-vector registers with the weed master

weed-vector-server --masters holds a KeepConnected stream to the weed
master, the same mechanism filer and s3 use via wdclient — the server
shows up in ListClusterNodes(client_type=vector). Resends registration
and rediscovers on failure.

* vector: s3 discovers weed-vector through the master

With -vectors.grpc unset the vectors gateway resolves one registered
weed-vector via ListClusterNodes on the masters it already derives
from filer config — discovery replaces the static dial for the
single-worker OSS path. The conformance harness registers the engine
to mini's master and drops -vectors.grpc.

* vector: default-fill registration fields

..Default::default() keeps the source compiling against proto variants
that carry extra fields, so the same file works in either tree.

* vector: serve the Vectors API on the main s3 port

The vectors router sits on the main -port listener behind a dispatch
that checks the SigV4 credential scope for /s3vectors/ — no separate
port needed. -port.vectors stays for explicit dedicated-port
deployments; the conformance harness now hits vectors through :8334.

* vector: drop -port.vectors entirely

It was new — no compat to preserve. The main -port listener serves the
API whenever a weed-vector backend is reachable.

* vector: drop -vectors.grpc; discovery is the only locator

weed-vector always registers to master, the gateway always resolves it
through ListClusterNodes — a static address just forks membership.

* vector: binary is weed-vector

Matches the enterprise binary; package and lib names are unchanged.

* vector: review fixes — cluster type, grpc port, lazy resolve, leader hint, advertise

- cluster: register 'vector' as a node type so ListClusterNodes returns
  weed-vector members
- s3: dial masters via ToGrpcAddress (HTTP port + 10000)
- s3: backend resolves lazily through vectorDiscovery — workers joining
  after gateway startup work on first query, and address changes re-dial
- master_client: convert master HTTP addr to grpc +10000, honor http(s)
  scheme, follow the leader hint in KeepConnectedResponse, and
  --advertise covers wildcard/unreachable listen addresses

* vector: no --advertise flag; derive the outbound address

A wildcard listen resolves to the source IP the master sees via a
route lookup, so registration always advertises something reachable
without another flag.

* vector: review fixes — plaintext dispatch, conn leaks, discovery backoff

The two plaintext http servers now serve the vectors dispatch too —
signed s3vectors calls on the main port reach the right router.
discoverVectorWorker closes each master conn and bounds lookups to 5s;
a dead or absent worker no longer re-hits master on every RPC — the
resolver backs off for 5s between rediscovery attempts.

* vector: 1 GiB grpc cap, dead-worker invalidation, CLI rename

Valid PutVectors batches exceed the tonic 4 MiB default — match the
gateway's 1 GiB cap. A replaced worker no longer sticks: any RPC error
resets the cached address and the next call rediscovers through the
master. The core CLI bin renames to weed-vector-cli so the two crates
do not collide on the same output path.

* vector: retry-only-transport reset, periodic re-resolve, cli rename

- reset() takes the failed client and runs only on transport errors —
  an InvalidArgument on a healthy conn no longer interrupts unrelated
  requests, and a concurrent re-resolve wins the race.
- cli() re-resolves once the interval passes even with a live client,
  so a worker that moves addresses is picked up.
- verify.rs spike docs use weed-vector-cli after the bin rename.

* vector: only Unavailable counts as a transport failure

A request deadline or an application error can fire on a healthy
connection — resetting the shared conn on those churned reconnects.
2026-10-28 00:58:31 -07:00
Chris Lu 81879fba7b vector: preserve typed error chains in lease pin/release (#12153)
vector: keep typed error chains in lease pin/release

Error::msg discarded the source chain — denied() could not downcast to
object_store::PermissionDenied, so try_pin surfaced the error instead
of degrading to an unleased query and read-only credentials broke
GetVectors/QueryVectors. Plain ? preserves the chain.
2026-10-28 00:38:32 -07:00
Chris Lu a2863bf83b wdclient: count an elapsed caller deadline as caller-gave-up in LookupVolumeIds (#12151)
* wdclient: treat an elapsed caller deadline as caller-gave-up on lookup errors

gRPC can surface its own DeadlineExceeded a tick before ctx.Err() becomes
visible, so a lookup that died on the caller's deadline was sporadically
recorded as a filer failure (seen in the linux/386 job).

* wdclient: reinterpret only deadline errors against an elapsed caller deadline
2026-10-28 00:35:52 -07:00
Chris Lu 67ce9a5f6d vector: expose index names in segment planning (#12150)
vector: named_segment_fragments carries the logical index name

Sharded builds create one named index per shard and a
with_index_segments scan must not mix names — a fanout planner needs
the name to group tasks, so expose it alongside uuid and coverage.
2026-10-28 00:34:45 -07:00
Chris Lu 10ba1b782f vector: carry the full identity through s3vectors auth (#12152)
vector: keep the full identity in the request context

Auth discarded the resolved identity object — the s3tables manager
request then re-resolved by name and denied authenticated users their
actions, policies, account, and claims. Store the object via
SetIdentityInContext so downstream authorization sees it.
2026-10-28 00:33:52 -07:00
dependabot[bot] b6851e561f build(deps): bump github.com/go-jose/go-jose/v4 from 4.1.5 to 4.1.6 (#12148)
Bumps [github.com/go-jose/go-jose/v4](https://github.com/go-jose/go-jose) from 4.1.5 to 4.1.6.
- [Release notes](https://github.com/go-jose/go-jose/releases)
- [Commits](https://github.com/go-jose/go-jose/compare/v4.1.5...v4.1.6)

---
updated-dependencies:
- dependency-name: github.com/go-jose/go-jose/v4
  dependency-version: 4.1.6
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-28 00:32:11 -07:00
dependabot[bot]anduser.email 3b65169d6c build(deps): bump com.google.guava:guava from 32.1.3-jre to 33.7.1-jre in /test/java/spark (#12141)
* build(deps): bump com.google.guava:guava in /test/java/spark

Bumps [com.google.guava:guava](https://github.com/google/guava) from 32.1.3-jre to 33.7.1-jre.
- [Release notes](https://github.com/google/guava/releases)
- [Commits](https://github.com/google/guava/commits)

---
updated-dependencies:
- dependency-name: com.google.guava:guava
  dependency-version: 33.7.1-jre
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* test/java/spark: bump guava to 33.7.2-jre

33.7.1 is covered by a published advisory for uncontrolled memory
allocation during CompactHashMap/CompactHashSet deserialization;
33.7.2 is the patched release.

* test/random_access: bump guava to 33.7.2-jre

Same deserialization advisory; the pin predates the fix.

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
2026-10-27 23:43:49 -07:00
dependabot[bot] bb5375d2b9 build(deps): bump rustls from 0.23.45 to 0.23.46 in /seaweed-volume (#12144)
Bumps [rustls](https://github.com/rustls/rustls) from 0.23.45 to 0.23.46.
- [Release notes](https://github.com/rustls/rustls/releases)
- [Changelog](https://github.com/rustls/rustls/blob/main/CHANGELOG.md)
- [Commits](https://github.com/rustls/rustls/compare/v/0.23.45...v/0.23.46)

---
updated-dependencies:
- dependency-name: rustls
  dependency-version: 0.23.46
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-27 23:43:16 -07:00
Ege Zambelliandchrislusf 89cc910a86 helm: render the filer's mysql credentials once when the user sets them (#12139)
* helm: render the filer's mysql credentials once when the user sets them

With the MySQL filer store, the filer reads WEED_MYSQL_USERNAME and
WEED_MYSQL_PASSWORD from the chart's db secret. Setting either one in
extraEnvironmentVars or secretExtraEnvironmentVars, as helm_ci.yml does for
the password, added a second entry with the same name: Helm 4 rejects the
install (env: duplicate entries for key), and Helm 3 deploys with a warning
that one definition may be dropped on apply.

The db secret's entry is now left out for a key the user sets, so each key
is rendered once and the user's value wins, the rule #8488 applied to a key
set in both global and filer extraEnvironmentVars. Without an override the
filer StatefulSet is unchanged.

* helm: render the selected filer mysql credential ahead of the merged vars

A credential the user sets now renders in the early position the db
secret's entries used to occupy, and is excluded from the merged and
secret loops. Kubernetes resolves a $(VAR) reference only against names
declared earlier, so a later entry such as WEED_MYSQL_DSN can again
reference the active credential. Secret-backed values win over plain
ones; the db secret's entry remains the fallback.

* helm: let a secret-backed filer env key replace the plain one

secretExtraEnvironmentVars now wins over extraEnvironmentVars for the
same name, rendering once at the merged key's sorted position instead
of twice. The values.yaml WEED_MYSQL_* defaults no longer duplicate
when overridden through the secret map - e.g. enabling the store via a
secret WEED_MYSQL_ENABLED also rendered the "false" default before,
which Helm 4's server-side apply rejects as a duplicate.

* helm: keep a credential that references another env var in sorted position

A user-supplied credential whose value itself holds a $(...) reference
stays in the merged vars' sorted order so its own references still
resolve; only reference-free values, valueFrom entries, and the
db-secret defaults move to the early slot.

* helm: render vars a filer mysql credential references ahead of it

Hoisting the selected WEED_MYSQL_USERNAME/PASSWORD entries above the
merged extra vars let WEED_MYSQL_DSN-style $(...) references resolve,
but it broke the opposite direction: a user-set credential such as
WEED_MYSQL_USERNAME: "$(DB_USER)" was emitted before DB_USER, so
Kubernetes left the reference literal. Pull every merged or secret var
a credential value transitively references out of the sorted set and
emit them first, dependencies before dependents, so both directions
resolve.

* helm: emit filer mysql credentials after the chart's own env vars

The dependency-ordered credential block rendered before
SEAWEEDFS_FULLNAME and the s3 credentials, so a user-supplied
WEED_MYSQL_USERNAME could not reference them via $(...). Move the
block below those fixed entries so references in either direction
resolve.

* helm: bound filer mysql credential ordering passes by var count

Chains deeper than the fixed eight passes forced a credential out ahead
of the vars it references. Cap discovery at the number of merged/secret
vars and emit passes at the ordered list length, so any resolvable chain
finishes before the last-pass fallback emits cyclic leftovers.

---------

Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
2026-10-27 23:32:16 -07:00
Ege ZambelliandChris Lu 3a0af308ce helm: keep the filer db credentials across uninstall and install (#12138)
* helm: keep the filer db credentials across uninstall and install

The db secret is kept on uninstall (`helm.sh/resource-policy: keep`), but it
is a pre-install hook, and Helm deletes an existing hook resource before
creating it again. The next install therefore wrote a new random password
and reset the user to YourSWUser, so the filer could no longer log in to a
database set up with the old ones, including credentials edited into the
kept Secret.

The template now reads the existing Secret with lookup and reuses its user
and password; a first install still generates them as before. A kind step
in helm_ci.yml installs, edits the Secret, uninstalls, installs again and
checks both values.

* helm: README, the db secret is kept and reused across a reinstall

* helm: make the db secret hook-delete-policy explicit

before-hook-creation is Helm's default for hook resources, but spell it
out so the reuse contract is obvious: lookup reads the kept Secret at
render time, the old copy is deleted before the hook recreates it, and
the stored credentials carry over.

---------

Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-27 17:48:22 -07:00
dependabot[bot] 1e9cc44697 build(deps): bump rustls from 0.23.45 to 0.23.46 in /seaweed-worker (#12142)
Bumps [rustls](https://github.com/rustls/rustls) from 0.23.45 to 0.23.46.
- [Release notes](https://github.com/rustls/rustls/releases)
- [Changelog](https://github.com/rustls/rustls/blob/main/CHANGELOG.md)
- [Commits](https://github.com/rustls/rustls/compare/v/0.23.45...v/0.23.46)

---
updated-dependencies:
- dependency-name: rustls
  dependency-version: 0.23.46
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-27 17:47:27 -07:00
dependabot[bot] f16f234c53 build(deps): bump org.xerial.snappy:snappy-java from 1.1.10.4 to 1.1.10.9 in /test/java/spark (#12143)
build(deps): bump org.xerial.snappy:snappy-java in /test/java/spark

Bumps [org.xerial.snappy:snappy-java](https://github.com/xerial/snappy-java) from 1.1.10.4 to 1.1.10.9.
- [Release notes](https://github.com/xerial/snappy-java/releases)
- [Commits](https://github.com/xerial/snappy-java/compare/v1.1.10.4...v1.1.10.9)

---
updated-dependencies:
- dependency-name: org.xerial.snappy:snappy-java
  dependency-version: 1.1.10.9
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-27 17:46:11 -07:00
dependabot[bot] 7b0e57eada build(deps): bump github.com/go-jose/go-jose/v4 from 4.1.5 to 4.1.6 in /test/kafka (#12140)
build(deps): bump github.com/go-jose/go-jose/v4 in /test/kafka

Bumps [github.com/go-jose/go-jose/v4](https://github.com/go-jose/go-jose) from 4.1.5 to 4.1.6.
- [Release notes](https://github.com/go-jose/go-jose/releases)
- [Commits](https://github.com/go-jose/go-jose/compare/v4.1.5...v4.1.6)

---
updated-dependencies:
- dependency-name: github.com/go-jose/go-jose/v4
  dependency-version: 4.1.6
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-27 17:45:25 -07:00
github-actions[bot] a1f6621553 docs: regenerate star history chart 2026-10-28 00:38:09 +00:00
Ege Zambelli ef16b4ed02 helm: render numbers and booleans in extraEnvironmentVars as values (#12137)
Only a string rendered as `value`; anything else went to `valueFrom`. An
unquoted number or boolean in a values file or `--set`, such as
`WEED_MYSQL_PORT: 3306` or `WEED_MYSQL_ENABLED=true`, rendered as
`valueFrom: 3306` and the API server rejected the workload.

A map still renders as `valueFrom`, a null still renders an empty variable,
and strings are unchanged. Numbers go through toJson because values files
decode them as float64, and quote would print 1000000 as 1e+06.
2026-10-27 17:34:46 -07:00
Chris Lu 5f26ef6ad1 filer,webdav,iam,sftp: add -idleTimeout flag for connection idle timeout (#12136)
* filer: make the HTTP connection idle timeout configurable

The filer applied a hardcoded 10s activity timeout to its HTTP
listener. A buffering proxy in front of the filer can stall upstream
reads longer than that when the end client drains slowly, so large
downloads through it were truncated (#8062).

Expose the value as -idleTimeout (default 10, unchanged), matching the
existing -idleTimeout flags on the volume and s3 servers, and wire
filer.idleTimeout through 'weed server' and 'weed mini' too. A nil
option keeps the 10s default so embedders constructing FilerOptions
directly keep working.

* filer: apply the idle timeout to the readonly port too

The -port.readonly listener ran without any connection timeout while
the main API port timed out. Give it the same -idleTimeout so both
ports behave identically.

* webdav: make the HTTP connection idle timeout configurable

The WebDAV server used the same hardcoded 10s activity timeout.
Expose it as -idleTimeout (default 10, unchanged), including
-webdav.idleTimeout when embedded via 'weed filer -webdav',
'weed server -webdav', and 'weed mini'.

* iam: make the HTTP connection idle timeout configurable

The IAM API server used the same hardcoded 10s activity timeout.
Expose it as -idleTimeout (default 10, unchanged), including
-iam.idleTimeout when embedded via 'weed filer -iam' and
'weed server -iam'.

* sftp: make the connection idle timeout configurable

The SFTP server used the same hardcoded 10s activity timeout.
Expose it as -idleTimeout (default 10, unchanged), including
-sftp.idleTimeout when embedded via 'weed filer -sftp' and
'weed server -sftp'.

* command: clamp idle timeout seconds before converting to duration

A seconds value past time.Duration's range would overflow on
multiplication and silently disable the timeout. Bound it in one shared
helper, which also deduplicates the nil-default handling at the four
call sites.

* command: return zero for non-positive idle timeout seconds

A large negative seconds value could wrap on multiplication and come
out positive. Guard before converting so non-positive input always
disables the timeout.

* command: apply idle timeout to local unix socket listeners

The unix-socket listeners in filer, s3, and sftp served accepted
connections with no activity deadline while the TCP listeners already
wrapped conns in util.Listener. Wrap the socket listeners the same way
so a positive idleTimeout also bounds idle socket connections.

* command: clamp s3 idle timeout seconds before multiplying

s3 still multiplied the flag inline, so an out-of-range value could
overflow time.Duration. Route it through idleTimeoutSeconds like the
other commands; non-positive keeps meaning disabled.

* volume: clamp idle connection timeout seconds before converting

* command: cover idleTimeoutSeconds conversion semantics

* command: gate idleTimeoutSeconds clamp check on 64-bit int
2026-10-27 17:31:23 -07:00
Chris Lu 6536fac9bd s3: answer NoSuchBucket for encryption and policy writes to a missing bucket (#12133)
* s3: answer NoSuchBucket for encryption and policy writes to a missing bucket

PutBucketEncryption, DeleteBucketEncryption, and PutBucketPolicy surfaced
the filer's bucket-miss as InternalError; GetBucketEncryption masked it as
ServerSideEncryptionConfigurationNotFoundError. Map the miss to
NoSuchBucket instead, matching AWS. Internal readers still treat a missing
bucket as "no encryption configured" so uploads keep their behavior.

* s3: wrap ErrBucketNotFound only for verified misses in bucket policy ops

The filer lookup helpers in get/set/deleteBucketPolicy marked every
LookupDirectoryEntry failure as bucket-not-found, so a transient filer
outage answered NoSuchBucket for a live bucket. Route them through
filer_pb.LookupEntry, which normalizes only true misses to ErrNotFound.

* s3: answer NoSuchBucket when the policy delete's own write loses the bucket

* s3: classify a filer NotFound on the policy UpdateEntry as ErrBucketNotFound

* s3: require the bucket entry to exist on config patch transactions

* s3: answer NoSuchBucket when tagging and CORS writes lose the bucket mid-request

* s3: keep the existence check on no-op bucket config patches

An unchanged config write produced an empty patch that early-returned
before the transaction, so a bucket deleted between the read and the
write still reported success. Ship the condition-only transaction so
IF_EXISTS runs on its own.

* s3: guard the iam step and cover failure-vs-miss classification

Nil-guard s3a.iam in the bucket policy handlers so the handlers do not
panic when no IAM server is wired (unit tests, embedded servers), treat
a "precondition failed" response string the same as the error code when
mapping a failed patch to ErrNotFound, and add handler-level tests that
pin the miss/failure distinction: missing buckets and vanished mid-write
buckets answer 404 NoSuchBucket while transient filer failures stay 500
InternalError.
2026-10-27 13:07:05 -07:00
Chris Lu 4c9e62177d s3: keep multipart part encryption on the initiate-time decision (#12134)
* s3: keep multipart part encryption on the initiate-time decision

putToFiler applied the *current* bucket default encryption when writing a
staged part, so a PutBucketEncryption landing between two UploadPart calls
produced an object whose parts disagree: some chunks ciphertext, some
plaintext. Completion then marks the object by its majority and GetObject
fails decrypting the rest. Parts staged under .uploads now skip the
bucket-default lookup; the decision snapshotted into the upload entry at
CreateMultipartUpload is replayed through request headers instead, which
matches AWS (default encryption applies at initiation).

* s3: mark multipart part writes via request context, not the staged path

Inferring "is a part" from the .uploads file path let an ordinary PutObject
named .uploads/x.part skip bucket default encryption entirely. The part
decision now travels on the request context from the two callers that
stage parts (UploadPart, UploadPartCopy); a client-chosen key cannot reach
it.
2026-10-27 09:13:04 -07:00
Chris LuandDevin 660a826eab remote_storage/s3: send Content-MD5 on RemoveDirectory DeleteObjects (#12129)
S3DisableContentMD5Validation skips the request header that DeleteObjects
requires, so AWS rejects every directory removal and the filer remote sync
offset stalls. Issue the delete through DeleteObjectsRequest and re-enable
the flag on that request's config copy so the SDK's own checksum handler
adds the header without changing any other operation.

Generated with [Devin](https://devin.ai)

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-27 09:11:26 -07:00
Chris LuandDevin 7a47c63d54 volume: replicate FetchAndWriteNeedle writes over the configured TLS scheme (#12128)
* volume: replicate FetchAndWriteNeedle writes with the configured TLS scheme

The guarded upload path hardcoded an http:// URL and used a bare
*http.Client, so a peer fronted by https rejected the replica write and
read-through of a remote-evicted object failed whenever the volume had
replicas. Normalize the upload URL through the shared global client and
let the guarded transport borrow its https.client TLS configuration, the
same settings the regular replication path resolves.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: keep the global HTTPS downgrade guard on replica uploads

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: cover TLS replicate uploads and the redirect downgrade guard

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: pin the downgrade guard against a dialable plaintext target

The previous redirect pointed at 169.254.169.254, which the guarded dialer
already refuses, so the test passed even with CheckRedirect removed.
Redirect to a live plaintext server on a dialable peer address and assert
it never receives the resent body.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume-rs: keep the HTTPS downgrade redirect guard on outbound writes

Mirrors the Go peer upload client: the shared outgoing HTTP client now stops redirects that would downgrade an https request to plaintext, so replicate uploads to TLS peers cannot leak data or auth headers via a 307 to http.

* volume-rs: fail rejected redirects instead of returning the 30x response

Go's rejectHttpsDowngrade uses ErrUseLastResponse, which surfaces to the
caller as a non-2xx status it rejects; attempt.stop() returned the redirect
response as Ok, so a refused downgrade could masquerade as a completed write
to any caller that does not check the status.

* volume-rs: fail replica writes on a non-2xx status

fetch_and_write_needle discarded the replica response without checking its
status, so a peer rejection — or a redirect the client refused to follow —
was reported back as a successful copy while no replica was written. Go's
doUploadData fails on any non-success status.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-27 09:09:04 -07:00
Hector Peeters c46a42694b Fix s3 cancelled read relookup (#12130)
* filer: keep cached volume locations when the reader cancels

* wdclient: do not count a caller's cancelled lookup as a filer failure
2026-10-27 08:21:32 -07:00
Chris LuandDevin af77406cf9 vector: index-segment restriction + partition helper (P3 seam) (#12124)
* vector: allow restricting ANN search to index segments

Query gains an optional index_segments list; when set and the index
path is used, the scanner restricts the vector index search to those
segments via with_index_segments. This is the seam the Enterprise
planner uses to fan one query out across per-segment workers; OSS
keeps passing None.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: enumerate and partition index segments for fanout

segments::index_segments lists the physical segment UUIDs covering the
vector field at the pinned version; partition() assigns them
round-robin to workers. The parity test proves two workers scanning
disjoint shares merge back to the full-dataset top-K — the P3 exit
gate, in-process.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: let a query carry fragment ids for the unindexed sweep

Lance only applies the index-segment restriction to indexed rows —
unindexed fragments outside the selected segments are skipped. For a
partitioned query to stay correct, the planner needs to hand the
unindexed sweep to a worker too, so Query gains a fragments list that
the scanner resolves to manifest fragments via with_fragments. A task
carrying both searches its segments via the index and its fragments
with flat KNN in the same scan.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: empty scope returns no hits; cover the sweep in tests

Some([]) fragments or index_segments fell through to an unrestricted
scan — a worker assigned nothing would have searched everything.
empty_scope short-circuits those queries, and the segment restriction
only calls with_index_segments on a non-empty list.

segments::segment_fragments also returns each segment's live fragment
bitmap, and the parity test now leaves a fragment unindexed so the
worker carrying it exercises the flat sweep — the contract the
Enterprise planner relies on.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: scope the scan to listed ∪ segment-covered fragments

Lance intersects with_fragments with the selected index segments, so a
worker given its segment share plus the unindexed sweep lost the share
entirely — only the listed fragments were scanned. Expand the fragment
list with the live coverage of the selected segments. The list is now
also applied when it resolves empty: stale ids or an empty assignment
must scan nothing, not fall through to an unrestricted scan. The merge
test now leaves a fragment unindexed so the combined path is exercised.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: partition assigns nothing when there are no workers

n.max(1) returned every segment in one phantom partition for a
nonexistent worker, silently dropping coverage.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-27 02:22:22 -07:00
a5036f568f webdav: refuse MKCOL and PUT under a missing folder with 409 (#12122)
* webdav: refuse MKCOL and PUT under a missing folder with 409

RFC 4918 9.3 and 9.7.1 say a MKCOL or PUT whose parent collection does
not exist must fail with 409 Conflict. The filer creates missing parents,
so both succeeded and made the intermediate folders. Check the parent in
Mkdir and in OpenFile with O_CREATE and return os.ErrNotExist, which
x/net/webdav turns into 409.

* webdav: check the PUT parent before an overwrite deletes the target

OpenFile deleted an existing destination before the new parent lookup,
so a PUT whose parent check failed left the old file gone.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* webdav: report a missing rename source as not-exist

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* webdav: refuse MOVE to a missing destination folder

AtomicRenameEntry grows any missing parents itself, so a MOVE to
/missing/file created /missing; check the destination parent first.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* webdav: answer 409 for COPY and MOVE under a missing collection

The webdav handler reports a failed directory Mkdir or Rename as 403,
so the destination parent is checked at the request layer instead.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* webdav: defer foreign-host MOVE and COPY destinations to the 502 handler

* webdav: only check the destination parent for absolute paths

* webdav: cover foreign-host and relative MOVE destinations in the guard test

---------

Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-27 01:34:11 -07:00
Jay Hemnani 5c14e566bc s3: allow bucket ownership controls when auth is off (#12121)
checkAccessByOwnership compared the caller's account with the bucket
owner even with no identities configured, so Get/Put/Delete
BucketOwnershipControls answered AccessDenied to every request. With
auth off there is no owner to compare against, as the other auth-off
checks already assume.
2026-10-27 00:36:53 -07:00
Chris Lu 8557771fc2 build(deps): bump google.golang.org/grpc past the #9330 deadlock fix (#12123)
The pinned dev snapshot (v1.85.0-dev.0.20260915183914) predates
grpc/grpc-go#9563, which fixes the control-buffer throttle deadlock
(grpc/grpc-go#9330) that wedges weed mini under concurrent S3 writes
over unix sockets.

Verified with the issue's standalone repro (one ClientConn over a unix
socket, concurrent unary RPCs): the old snapshot deadlocks at c=512
after 743 RPCs; the new snapshot completes 3.06M RPCs/20s at c=512 and
1.97M/15s at c=2048 with no stall -- above the v1.82.0 pre-regression
baseline of ~112k ops/s.
2026-10-26 23:43:24 -07:00
80f3cc0613 topology: keep oversized volumes out of the writable list across re-registration (#12120)
* topology: keep the oversized verdict after the location list is gone

The per-replica oversized marks live in the volume's location list, which
is deleted when its last location leaves and recreated flagless on the
next registration. Between that recreation and the next heartbeat that
re-measures the volume, ensureCorrectWritables saw no mark and
re-admitted an oversized volume to the writable list; two other paths
(quarantine release, mark-writable) never consulted the marks at all.

VolumeLayout now remembers the oversized verdict per vid, written only
by heartbeat sizes and forgotten only when a report measures the volume
under the limit. The writable admission gate itself refuses a marked
volume, and the vacuum commit path drops a verdict its live size check
has just disproved.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* topology: mark oversized at registration when the report carries a size

RegisterVolumeLayout ran EnsureCorrectWritables while the recreated
location list still held no per-replica marks. When the registering
report knows the volume's size, refresh the oversized state before the
writable check instead of waiting for the next heartbeat pass.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* topology: test that oversized volumes stay out of writables across re-registration

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* topology: remember the oversized verdict per replica

A shared per-volume verdict was cleared by any single replica's
under-limit report, so a still-oversized replica whose location entry
had died could re-register from a size-less report and put the volume
back on the writable list. Keep one verdict per reporting replica: an
under-limit report clears only its own, and a replica that never
returns cannot pin its volume out of writables forever because the
verdict ages out. The vacuum commit now feeds the node's fresh answer
through the same per-replica bookkeeping instead of erasing the whole
verdict.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* topology: count a volume at the size limit as full on vacuum commit

The layout's oversized test is >= the limit, but the vacuum commit and
status checks used >, so a replica sitting exactly at the limit passed
as not-full and the commit path could re-admit it.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* topology: cover per-replica oversized verdicts in tests

One replica shrinking must not clear a still-oversized replica's
remembered verdict; cover that plus a size-less re-registration before
the second replica is re-measured. The vacuum-commit case now shrinks
the stored record first, since the commit re-derives the mark from the
node's own answer.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* topology: judge the vacuum commit by the size it just measured

SetVolumeAvailable rebuilt the oversized verdict from the cached
VolumeInfo, which still carries the pre-compaction size until the next
heartbeat. A commit that shrank a volume under the limit re-marked the
replica oversized and never restored writes. The commit path already
asks every replica for its fresh size; pass it through per replica and
record it on the mark, the verdict, and the size tracking seed.

* topology: hold an expired oversized verdict while its replica is back

The TTL forgives a replica that never returns, but a remount announces
the volume without a size and registration skips the oversized check.
Deleting the verdict on expiry then re-admitted a still-oversized
volume until the next size-bearing heartbeat. Keep counting an expired
verdict while its replica sits in the live location list, and keep
sweeping verdicts whose replicas stayed gone.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: chrislusf <chrislusf@gmail.com>
2026-10-26 23:22:15 -07:00
Chris LuandDevin d182be6a9e s3api: cap SOSAPI capacity by the reported physical disk bytes (#12118)
calculateClusterCapacity multiplies advertised volume slots by the
volume size limit. Slot counts are an admission bound rather than a
capacity measurement — an explicit -max can be set higher than the
volumes a disk actually fills — so a small disk can report more total
and available capacity than it physically has. Bound both numbers by
the per-disk filesystem counters the volume server already reports in
its heartbeat, falling back to the slot estimate only when they are
absent.

Generated with [Devin](https://devin.ai)

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 23:18:43 -07:00
Chris LuandDevin 0ea818a44f mount: drain OFD locks when a file description is released (#12117)
* mount: drain OFD locks when a file description is released

The kernel scopes OFD fcntl locks to the open file description but
reports no lock_owner for them at RELEASE — it only sends one for flock
holders — so an OFD write lock taken on a file handle outlived the
holder and blocked every later open of that file (e.g. etcd failing to
reopen its WAL after a restart). The mount now records the owner of each
granted non-flock lock under its file handle and releases those owners
on RELEASE, which also covers any POSIX locks left over when FLUSH could
not run.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* mount: give every open its own fh value over the shared handle

AcquireFileHandle reuses one FileHandle per inode, so every open reported
the same fh to the kernel. A RELEASE for one description could then not
be told apart from a sibling still holding the inode open — draining
lock owners recorded under the shared fh dropped locks belonging to
descriptions that stay open.

Each open now registers its own fh value in fh2inode, all resolving to
the shared FileHandle, and RELEASE removes exactly the id that is
closing. This also stops a duplicated RELEASE on one id from
double-decrementing the open count.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* mount: forget flushed lock owners so a later release cannot re-drop locks

POSIX lock owners are process-scoped: Flush releases every lock the
process holds on the inode, but the owner stayed recorded under each
open's fh for the Release-time drain. If the process took a fresh lock
through another open before an old description finished closing, the
stale record made that RELEASE drop the new lock too.

Flush now drops the released owner from every recorded open of the
inode, so the Release drain only ever fires for owners whose locks were
never flushed.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* mount: keep close-time lock owner records when the release fails

A routed POSIX owner release that errors keeps the flush hint for a
retry, but Flush also dropped the owner from every open's release
records unconditionally — so the final Release skipped the retry and
the owner filer's lock was stranded until lease reaping. Only forget
the owner once its locks were actually released.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 22:02:13 -07:00
b747e0d3b1 cluster: bound each DistributedLock attempt with a timeout (#12113)
* cluster: bound each DistributedLock attempt with a timeout

An unreachable lock owner blocked the attempt on the connection timeout.
Bound it so the write fails fast; a timed-out attempt keeps no renew token.

* cluster: keep renewToken across failed lock attempts

A request may have renewed the lease server-side before its response was
lost, so the last token can still be valid on the next attempt. Clearing
it on any transport failure strands a live lease until its TTL expires.
A definitive rejection still clears the token through the response's
empty RenewToken, and falling back to the seed filer no longer abandons
the lease either.

* cluster: bound each lock attempt by the remaining route window

The 15s unavailability window started only after the first failure, so
three 10s attempts plus sleeps outlasted it: the third failure selected
the seed fallback at the same moment the loop gave up, and a reachable
seed route was never tried.

Start the clock before the first attempt and cap each attempt at what
the window has left. When the window is spent on a non-seed target,
fall back to the seed filer — it can forward to the real owner — and
give that route its own window, so the fallback is always exercised
before acquisition fails.

* cluster: bound DistributedUnlock by the lock RPC timeout

The unlock used context.Background(), so Stop against an unreachable
lock server could wait on the TCP connect timeout far past the renewal
goroutine's bounded shutdown wait. A timed-out unlock is harmless: the
lease expires via its TTL regardless.

* dlm: replay the current token on a lost renewal's retry

A renewal rotates the token server-side before the response is sent. When
that response is lost the client only knows the superseded token: retries
mismatched, the token was cleared, and the holder conflicted with its own
lease until it expired. The lock now remembers the superseded token, so a
retry from the same owner replays the current token, and a superseded
token can still release its own lock.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* cluster: give the seed lock fallback one fresh window per acquisition

The seed can redirect back to a dead primary via LockHostMovedTo, so
refilling the retry window on every return to the seed made the
primary->seed->primary cycle unbounded. The first seed fallback keeps
its fresh window; later returns to the seed keep the running deadline.

* cluster: advance the lock seq when a lost-renewal replay extends the lease

A retry of a renewal whose response was lost updates ExpiredAtNs in place
but returned the existing Seq, so the replicated extension was rejected
by InsertBackupLock as a stale mutation. A promoted backup would then
grant the lock when the old expiry passed even though the holder still
believes the lease is renewed. Bump Seq on the replay so the backup
accepts the new expiry.

* cluster: carry the superseded lock token through replication and transfer

A renewal rotates the lock token and keeps the superseded one in
PrevToken so a holder whose renewal response was lost can still recover.
That recovery broke as soon as the lock moved: neither the replicated
backup copy nor a transferred lock carried PrevToken, so a new primary
rejected retries of the old token until the lease expired.

Add prev_renew_token to the Lock and ReplicateLockRequest messages and
thread it through ReplicateLock, TransferLocks, InsertBackupLock and
InsertLock.

* cluster: count clean lock redirects against the retry window

A bare LockHostMovedTo response (no error, no token) returns nil from
attemptToLockWithin, and retryUntilLocked reset the retry window on every
nil result. Two ring-disagreeing filers forwarding to each other therefore
bounced the client in an unbounded loop; the spent-window exit could also
return a nil error with no token, handing callers an unacquired lock.

* cluster: do not mark a lock held or hot-loop on tokenless attempts

A bare LockHostMovedTo redirect returns nil with no renewToken; the
attempt previously flipped isLocked to 1 anyway and retried immediately.
Gate the flag on a granted token and pace the retry like other outcomes.

* cluster: keep renewToken across bare lock redirects

A LockHostMovedTo response carries no token and no error; overwriting
renewToken with it dropped a held token, so the next renewal re-acquired
fresh and could contend against its own still-live lease until TTL.

* cluster: poll for replicated backup state in lock tests

Replication runs in a goroutine, so a fixed sleep followed by an
immediate backup read is racy. Wait for the expected backup state
with require.Eventually instead.

* cluster: mark a lock held only on a grant verdict

A bare LockHostMovedTo redirect with a saved token left isLocked set
without the redirected filer granting anything, and retryUntilLocked
could exit on a stale token with no attempt made. doLock now reports
whether the response was a grant, and both the retry loop and isLocked
follow that verdict.

---------

Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 20:53:46 -07:00
Chris LuandDevin ee315527cb vector: address remaining s3vectors review findings (#12115)
* vector: address remaining s3vectors review findings

Catalog and permission model:
- Vector buckets are declared Lance table buckets end to end: create sets
  format=lance, get/list/delete reject or skip ICEBERG buckets, and the
  format lookup runs on an internal trusted catalog so callers need no
  GetTableBucket grant.
- The implicit "vector" index namespace is created through the same
  internal manager, so a CreateTable grant alone authorizes CreateIndex;
  typed S3TablesError codes replace string matching on err.Error().
- CreateIndex pre-checks existence after the authorize, tags the record
  with a create nonce, and rolls back on engine failure under
  context.WithoutCancel with a versionToken CAS, so a retry never deletes
  a table it did not create.
- DeleteIndex drops the catalog record directly; DropIndex is
  unimplemented on the shared VectorService.
- DeleteVectorBucket removes the empty internal namespace on
  BucketNotEmpty and retries once.
- PutVectors/DeleteVectors require the update grant, and engine gRPC
  errors map to typed API errors instead of generic 500s.

API correctness:
- CreateVectorBucket/CreateIndex return their resource ARNs.
- ARNs reject empty bucket segments; vectorBucketArn must be an ARN.
- ListIndexes returns summaries built only from catalog fields.
- ListVectors honors maxResults/nextToken inside a segment via
  "seg:<pageStart>:<nextIndex>" resume tokens and validates
  segmentCount/segmentIndex.
- QueryVectors requires positive topK; PutVectors rejects empty keys.
- CreateVectorBucket rejects a supplied encryptionConfiguration rather
  than silently dropping it.
- Metadata filters emit Lance's real JSONB UDFs: json_get_string/
  json_get_float/json_get_bool picked by operand type, json_exists with a
  bracketed JSONPath, and per-member accessors for $in/$nin.

Engine:
- weed-vector-server stores the metadata column as Lance JSONB
  (lance.json extension) so the json_* UDFs apply; reads accept legacy
  Utf8 JSON columns.

Vectors port TLS:
- The TLS config is built before binding listeners, loads -cacert.file
  into ClientCAs when client-cert verification is on, fails startup on an
  empty/invalid CA file, and closes the gRPC conn on every failure path.

* s3vectors: authorize index create before writing the implicit namespace

A denied caller could still leave a persistent namespace behind because
ensureIndexNamespace wrote through the trusted catalog before the
caller's create grant ran. AuthorizeCreateTable now tolerates the
not-yet-declared implicit namespace (bucket policy alone decides) and
runs before any write; real table creates keep requiring the namespace.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3vectors: keep engine error details out of the API response

Unavailable now maps to 503 ServiceUnavailableException and unclassified
codes return a generic message; the real gRPC error stays in the log.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3vectors: exact-type metadata filters + NULL-safe $nin

The jsonb extractors convert numeric-looking strings through
json_get_float and 'true'/'false' through json_get_bool, so a stored '3'
could satisfy a filter for the number 3. Numeric and boolean comparisons
now also require the stored value not be a string, and each $in/$nin
group coalesces NULL so NOT cannot invert NULL into NULL.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: validate the vectors TLS key pair before binding

The reloading certificate provider reads the files asynchronously, so a
missing or mismatched pair would only surface as failed handshakes on an
announced HTTPS port. Load it once up front.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3tables: make a token-checked table delete atomic

The version token was read in one filer call and enforced in another, so
a delete+recreate slipping between them let the delete erase the
replacement. When a token is supplied the name path now carries the
observed mtime as an if_not_modified_after precondition (evaluated under
the path lock), and a decoupled delete strips the catalog marker under a
compare of the captured attributes before touching the data path.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3vectors: compare filter operands against the stored JSON type

jsonb's strict converters make json_get_string return text for numbers
and booleans too, so a get_string IS NULL guard is never true for scalar
metadata — numeric and boolean comparisons silently matched nothing.
Numbers and booleans now compare through TRY_CAST of the serialized
json_extract text, which yields NULL on any other stored type; string
comparisons stay on json_get_string behind a CASE guard so the extractor
only runs on rows storing a JSON string.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3tables: allocate implicit namespace metadata before use

A first table create in an undeclared namespace took the
allowMissingNamespace branch with a nil record and panicked on the
owner assignment.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3tables: read the delete target's metadata and snapshot in one lookup

Two separate lookups let a delete+recreate in between validate the
caller token against the old record while the conditional delete aimed
at the replacement.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3tables: assert the captured record before a conditional table delete

An xattr update keeps Mtime, so the IfNotModifiedAfterNs guard alone
could not see a metadata write that had already rotated the version
token, and a recreate preserving the mtime could slip the mtime check.
The delete now first proves the entry still carries the captured
attributes under the filer's path lock, then keeps the mtime
precondition.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3vectors: match only JSON boolean literals in metadata filters

* filer,s3tables: delete versioned tables under one attribute precondition

* s3vectors: compare serialized JSON for string filter operands

json_get_string converts numeric-looking strings and booleans, so it
cannot serve exact-typed string comparisons even behind a guard. String
operands now compare the serialized json_extract text directly, which is
exact-typed and never errors; a Rust test pins the semantics against a
mixed-type dataset.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3vectors: compare boolean filter operands as serialized text

Arrow's boolean cast accepts '1' and '0', so a TRY_CAST column let a
stored number satisfy a boolean filter. Booleans now compare the
serialized json_extract text like strings do — a JSON boolean only ever
serializes as 'true' or 'false'.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3tables: claim a catalog name's markers in one compare-checked write

A create wrote its markers attribute-by-attribute after checking the
name was free. A delete's restore landing in the check-to-write gap put
the old markers back first, and the create then overwrote only the ones
it sets — leaving the old tags, policy, and maintenance settings on the
new table. CreateTable, RegisterTable, and CreateView now write all
their markers in a single update that refuses to run while any catalog
marker is present, so the second writer loses with a conflict instead
of inheriting them.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3tables: test the attribute-preconditioned table delete and name claim

Adds a BeforeDelete hook plus ExpectedExtended evaluation to the
in-memory filer so tests can land a competing write in a handler's
check-to-delete window, and covers two losing races: a token delete
whose record changed since the read, and a create colliding with a
delete's marker restore.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3tables: reserve a catalog record under one compare-checked claim

claimCatalogRecord writes a deleting marker onto the record while it
still holds exactly the caller's captured attributes, so a delete or
rename can hold the record through a multi-step operation and a losing
writer backs out before any data or destination entry is written. The
claim key joins the markers a name-claim asserts absent, so the name
cannot be reused while the reservation stands.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3tables: claim the record before purging or publishing past it

A token delete of a decoupled table purged the data first and checked
the marker afterwards, so a rename that finished in between left the
moved record pointing at purged data while the delete reported a
conflict. The delete now claims the record under the captured
attributes before any data is removed and releases the claim if the
purge fails, keeping the record intact for a retry.

The rename claims the source before publishing the destination, so a
losing rename backs out while no destination entry exists instead of
leaving one behind pointing at data a delete went on to remove.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3tables: test the record claim across delete and rename races

A rename completing in the token delete's check-to-claim window turns
the delete into a conflict with the data intact, and a delete's claim
on the source turns the rename into a conflict before the destination
entry is published.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 20:49:44 -07:00
Chris LuandDevin 8379ee9251 vector: snapshot lease + maintenance safety tests (P2) (#12107)
* vector: version-lease helpers + maintenance safety tests

Pinned queries must survive Lance maintenance. maintenance::pin tags the
current dataset version so cleanup refuses to reclaim it; release drops
the tag. Tests prove the P2 gate: a pinned query returns identical hits
across compact_files, a tagged version survives cleanup_old_versions
(error_if_tagged protects it, the non-strict run skips it), and a
missing data file fails the query instead of returning partial hits.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: lease the query's dataset version in the VectorService

query_vectors pins the resolved version with a query tag for the
duration of the RPC and releases it after, so concurrent maintenance
cleanup cannot reclaim the files out from under an in-flight read.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: release the version lease on every query exit path

The lease tag was only released on query_vectors' success path — get_vectors
never released and query_vectors leaked its tag on early returns. pin now
returns a guard whose drop deletes the tag, so every path frees the version.

* vector: unique lease tag names + reap abandoned leases

Nanos-since-epoch tag names can collide across concurrent readers and
processes, so each lease now carries a unique suffix. pin() also drops
query tags older than an hour, since a killed or cancelled request
leaves its lease behind and would otherwise block cleanup forever.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: keep live lease tags + throttle abandoned-lease scans

An expired tag is only reaped once no live lease in this process owns
it, so a query running past LEASE_TTL keeps its pin. Tag scans run at
most once per 60s per dataset instead of on every request.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: answer read-only queries without a version lease

A namespace may hand the server credentials that can read a table but
not update it, so the tag write in pin() fails and the whole query
errors out. try_pin() degrades a denied tag write to an unleased query
instead of failing the request.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: fail a lease whose pinned version is already gone

Cleanup lists tags once at the start, so a tag created mid-cleanup does
not protect the version it names. After the tag lands, pin now verifies
the version still exists and drops the lease when it does not, failing
the read at pin time instead of mid-query.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 20:11:58 -07:00
Chris Lu bfb6e40fb4 test: assert the resumed segment token in ListVectors paging (#12108)
test: assert the resumed segment token

The second listSegmentVectors result was never checked (staticcheck
SA4006); assert it resumes scanning at row 19 after matches 10..18.
2026-10-26 18:08:48 -07:00
Chris LuandDevin 67240f538d filer.remote.gateway: stop zero-byte uploads of remote-only entries (#12110)
* filer.remote.gateway: skip uploading a dropped local copy on update

The bucketed update path uploaded the chunkless entry whenever the update
was not metadata-only. remote.uncache, eviction, and remote.meta.sync drop
a synced entry's chunks and clear its sync stamp, which reads as dirty
when the remote mtime shares the entry's second; the upload then replaced
the object that is the entry's only content with zero bytes. Skip the
upload the same way the mounted-directory sync does.

* filer.remote.gateway: copy a remote-only entry's object on rename

A rename deleted the source object and uploaded the chunkless entry,
replacing the entry's only content with zero bytes — the same loss
remote.uncache caused on plain updates. Complete the rename on the
remote the way the mounted-directory sync does: copy the source object
over (or take the destination when it already describes the entry),
restamp so reads stop resolving through the pending-source marker, and
remove the source only while it is still the object the stamp describes.
A remote-only entry renamed out of the gateway's coverage keeps its
object, and the plain rename's old-key delete only removes the object
the event's stamp still describes.

* filer.remote.gateway: scope the sync-stamp skip to same-path updates

shouldSendToRemote reads the entry's sync stamp, which still covers a
rename's unchanged mtime, so a synced or pending-stamped rename event was
skipped before its copy or upload could run. Only updates in place may
skip on the stamp.

* filer.remote.gateway: keep the source a pending rename moved on from

A rename event whose pending destination has since been renamed again
still names the source a later event reads through its own marker;
deleting the old key here loses the only copy. Mirror the remote.sync
check: leave the source when the event's entry carries the pending-source
marker and no entry holds the destination any more.

* filer.remote.gateway: release a rename's pending source once complete

A rename whose old entry read through a pending-source marker leaves that
earlier source behind: the destination is released by its own key, never
the marker's. Mirror releaseRenamedSource — once the renamed entry no
longer reads through a marker, delete the pending source guarded by the
old entry's stamp, keeping it while a remote-only entry holds its path.

* filer.remote.gateway: finalize a complete rename before failing its source

A rename source outside the gateway's coverage errored before the
destination was checked, and the recorded failure held the event offset.
Stat the destination first: when it already holds the content the entry
describes, refresh the stamp and leave the unmanaged source untouched;
fail only when nothing holds the content.

* filer.remote.sync: document the source-of-truth protocol both processors share

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 18:03:12 -07:00
github-actions[bot] 0f1e73cc3a docs: regenerate star history chart 2026-10-27 00:56:18 +00:00
Chris LuandDevin e2d899ac25 test/vectors: fix the S3 Vectors e2e harness (#12112)
* test/vectors: give the vector server static storage credentials

The e2e harness ran weed-vector-server without credentials, so lance's
AWS object store fell back to IMDS and CreateIndex died on the runner's
411 from the token endpoint. Point it at the test credentials the s3
gateway now reads from s3.json.

* vector: take the list item field from the schema when batching vectors

Lance stores the fixed-size list item as nullable even when the write
schema declared it non-null, so PutVectors built batches that no longer
matched: expected FixedSizeList(4 x Float32) but found FixedSizeList(4 x
non-null Float32).

* vector: store index metadata as lance jsonb so filters resolve

The metadata field was plain Utf8, but the filter functions lance's
planner registers (json_get_string, json_get_float, json_exists) take a
LargeBinary jsonb column, so QueryVectors with a metadata filter failed
to plan. Declare the field with the arrow.json extension — lance encodes
it to jsonb on write and back on read — and encode the batch's strings
when the stored column is LargeBinary. Also rename the filter's
json_get_str call to json_get_string, the name lance registers.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 17:52:05 -07:00
Chris LuandDevin a9d6bfc51f test: boto3 s3vectors conformance harness (P1) (#12105)
* vector: boto3 s3vectors conformance harness

test/vectors spins a mini cluster, an s3 gateway with -port.vectors and
-vectors.grpc, and a weed-vector-server, then exercises the AWS client
end to end: bucket and index lifecycle, put/get/list/delete vectors,
and a filtered query asserting distances and metadata.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: run the s3vectors conformance subset

An e2e job in rust-vector-tests: builds weed and weed-vector-server,
then runs test/vectors against a mini cluster plus the vectors gateway.
The workflow now also watches vector.proto, its stubs, and test/vectors.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: fix e2e protoc and codespell findings

The e2e job needed the same vendored-protoc step as the unit-test job —
lance's build scripts locate protoc themselves. Codespell flags the AWS
`$nin` operator literal and the `lits` variable: exempt `nin` in
.codespellrc like the existing word exemptions, rename the variable,
and fold the negation into the IN/NOT IN symbol directly.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: classify catalog errors by code, not message text

* s3: drop the implicit vector namespace on vector-bucket delete

* test: point the vector engine at the gateway lance port

* test: bind the vectors gateway listeners to loopback

* test: make the filter and distance assertions load-bearing

* test: rebuild the harness binaries on every run

* ci: fix the vectors end-to-end job

* ci: run the vector tests on gateway and command changes

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 15:53:50 -07:00
Chris Lu c55b7c5a29 s3: fix CreateIndex metadata panic and ListVectors segmented paging (#12106)
* s3: nil-check metadataConfiguration before reading its keys

* s3: honor maxResults/nextToken in segmented ListVectors

* s3: exclude non-Lance buckets from ListVectorBuckets

* s3: test segmented ListVectors paging token math

* ci: silence codespell on the "$nin" operator and rename lits
2026-10-26 15:40:25 -07:00
Chris LuandDevin dcc94f818d vector: S3 Vectors API gateway (P1) (#12103)
* vector: S3 Vectors API gateway over VectorService

weed/s3api/s3vectors serves the AWS S3 Vectors JSON API: bucket CRUD
maps to S3 Tables table buckets, index CRUD to Lance table records plus
the VectorService CreateIndex/DropIndex RPCs, and the data plane
(Put/Get/List/Delete/QueryVectors) calls the shared service. The AWS
filter document translates to a Lance SQL predicate over the metadata
JSON column; unsupported operators fail loudly rather than overmatch.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: -port.vectors and -vectors.grpc on weed s3

Mirror the Lance server startup: the S3 Vectors API binds its own port
and dials the weed-vector-server gRPC address; both disabled by default.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: resolve index ARNs, authorize index ops, fix catalog fields

- vector buckets declare FormatLance so a new bucket accepts index
  tables instead of defaulting to ICEBERG.
- ListIndexes decodes the `tables` response field; `tableSummaries`
  hid every index.
- resolveIndexARNs honors indexArn alone; handlers no longer treat
  indexName as the only carrier.
- data-plane RPCs fetch the index record first, so callers authorize
  against the index resource and a missing index 404s before touching
  the engine; QueryVectors reports the index's configured metric.
- IndexArn added to the index-level request types.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: serve the vectors port over TLS when configured

When -key.file/-cert.file are set, the vectors listener mirrors the
main API and serves TLS rather than a second plaintext endpoint.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: harden the s3vectors gateway handlers

- $ne no longer shadows the operator: exclusion filters emit <>
  instead of equality.
- a null element in a PutVectors list fails the request instead of
  panicking.
- requests are capped at 32 MiB; distances return via *float32 so an
  exact match still reports 0 when returnDistance is set.
- CreateIndex declares the `vector` namespace first and validates
  dimension (1..=4096) before the catalog write; a failed
  VectorService.CreateIndex rolls the table record back.
- non-filterable metadata keys persist as JSON rather than a
  comma-joined string, and index records carry their createdAt.
- ListVectors honors segmentIndex/segmentCount by slicing the full
  listing in the gateway.
- catalog errors map to AccessDeniedException/ConflictException instead
  of a blanket ValidationException, and malformed ARNs 400 instead of
  slicing.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: /{Operation} routes and vectors-port TLS, conn cleanup

Path-dispatched clients (POST /QueryVectors) reach the same handlers as
X-Amz-Target calls. A failed listener bind now closes the vector gRPC
connection instead of leaving it open with the endpoint disabled.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 14:57:44 -07:00
Chris Lu 74abcb630e ci: retry the timing-sensitive s3 lifecycle expiration tests once (#12073)
* ci: retry the timing-sensitive s3 lifecycle expiration tests once

test_lifecycle_expiration and test_lifecyclev2_expiration assert exact
object counts after fixed sleeps. The lifecycle worker expires objects
via meta-log replay, and under the suite's write flood a shard's replay
can trail an object's PUT event past the test's window (observed on
master and PR runs, SQL-store job only). Split them out of the main
list and allow one retry; each attempt is independent so real
regressions still fail.

* ci: allow a second retry for s3 lifecycle expiration tests
2026-10-26 13:39:40 -07:00
Jay Hemnanianduser.email 1e7afbacf4 s3: list every bucket when auth is off (#12077)
* s3: list every bucket when auth is off

With no identities configured, every request may create, read and
write any bucket, but ListBuckets treated the caller as an
unauthenticated user and returned an empty list, so `aws s3 ls` showed
nothing. List all buckets in that mode, with prefix, max-buckets and
continuation tokens as before. Listing with auth on is unchanged.

* s3: cover the auth-on anonymous ListBuckets case

The no-auth listing change also moves the nil-identity gate; pin the
security-relevant direction that a request without an identity still
sees no buckets once auth enforcement is on.

* s3: verify ListBuckets continuation token reaches the next page

---------

Co-authored-by: user.email <chrislusf@users.noreply.github.com>
2026-10-26 13:38:36 -07:00
Chris LuandDevin 87f15e4d6c vector: weed-vector-server, the VectorService gRPC (P1) (#12102)
* vector: VectorService protobuf contract

The shared RPC boundary between the Go S3 gateway and the Rust vector
engine from the vector design doc: CreateIndex/DropIndex, Put/Get/
List/Delete/Query vectors. A Table reference carries the Lance table id
and an optional pinned version, matching the snapshot discipline queries
are required to keep.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: generated Go stubs for vector.proto

protoc output via weed/pb/Makefile.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: carry return_data through QueryVectors

S3 Vectors QueryVectors honours returnData, but the contract had no
way to ask for or carry vector data in a query response, so the
gateway would need a second GetVectors round-trip per query. Add
return_data to the request and data to QueryResult now, while the
contract is still new.

* vector: pin metadata numbers to double precision in the contract

Metadata arrives as JSON, where numbers carry double precision; say
so on the field so neither side promises integer fidelity past 2^53.

* vector: declare_table on the lance namespace client

CreateIndex needs a way to ask the catalog for a new table's location;
expose POST /v1/table/{id}/declare.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: cargo fmt coverage.rs

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: weed-vector-server, the VectorService gRPC

A tonic server over weed-vector-core: CreateIndex declares the table
through the namespace and writes an empty {key, vector, metadata}
dataset; PutVectors upserts by key (delete + append until merge_insert
is exercised); Get/List/Delete/Query route through the engine's
resolved table, honoring pinned versions and the prefilter default.
Metadata travels as a Utf8 JSON column, proto Struct on the wire.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: stamp the metric in the schema, tighten put validation

CreateIndex now bounds dimension at the S3 Vectors 4096 maximum and
records the distance metric in the vector field's Arrow metadata, so a
later reader recovers the index's configured metric from the table
itself. batch_of checks every vector's data length individually before
flattening — equal totals could still let a short vector borrow floats
from its neighbor — and writes against the dataset's own schema.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: upsert vectors in a single merge commit

PutVectors ran delete-then-append: a failed append left the old keys
deleted, and two concurrent puts could both append a replacement. Use
lance's merge_insert on the key column — one commit, no delete window,
conflicts fail the request instead of duplicating keys.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: read the index's metric; fetch metadata in one take

QueryVectors hardcoded L2 — it now reads the metric the index was
created with from the vector field's metadata. return_metadata fetched
each hit's row one at a time; collect the row ids into a single
take_rows call and join metadata back on the hit key.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: honor return_data in query results

QueryResult grew a data field but query_vectors never populated it.
Fetch the vector column in the same batched take_rows as metadata and
return each hit's stored embedding.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: reject a malformed query as invalid_argument

A zero top_k silently became one result and a dim-mismatched query
vector surfaced as an internal scan error; reject both before querying.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: reject a malformed list next_token

A token that failed to parse silently restarted the listing from
offset zero; surface it as invalid_argument instead.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: answer an empty put without a commit and cover ragged input

An empty vectors list would still run a merge commit; return the
current version instead. Pin the ragged-vector check with a test.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: keep non_filterable_metadata_keys on the table schema

create_index dropped the declared filter policy. Persist it as JSON on
the metadata field so the index carries which keys a filter may touch;
enforcing the list belongs to the gateway that translates the request
filter into Lance SQL.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: reject duplicate keys in one put

A repeated key for a new row committed both copies, and for an existing
key the merge failure surfaced as internal. Reject duplicates up front
as invalid_argument.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: report open failures as not_found or unavailable

A catalog 404 or a missing dataset/version is not a client mistake, and a
catalog outage is retryable: mapping every open error to invalid_argument
made both indistinguishable from a bad request.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: page list_vectors when max_results is unset

A proto3 client that omits max_results sends 0, which used to limit the
scan to zero rows and return no token.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: drop stray empty service.rs.tmp

* vector: rename datas map to data_map

codespell flags datas as a misspelling of data; use a distinct map name.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 13:32:34 -07:00
Chris LuandDevin d771a91571 vector: VectorService gRPC contract (P1) (#12101)
* vector: VectorService protobuf contract

The shared RPC boundary between the Go S3 gateway and the Rust vector
engine from the vector design doc: CreateIndex/DropIndex, Put/Get/
List/Delete/Query vectors. A Table reference carries the Lance table id
and an optional pinned version, matching the snapshot discipline queries
are required to keep.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: generated Go stubs for vector.proto

protoc output via weed/pb/Makefile.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: carry return_data through QueryVectors

S3 Vectors QueryVectors honours returnData, but the contract had no
way to ask for or carry vector data in a query response, so the
gateway would need a second GetVectors round-trip per query. Add
return_data to the request and data to QueryResult now, while the
contract is still new.

* vector: pin metadata numbers to double precision in the contract

Metadata arrives as JSON, where numbers carry double precision; say
so on the field so neither side promises integer fidelity past 2^53.

* vector: clarify the query filter is raw Lance SQL over one metadata column

The doc said 'metadata columns' but the table stores all metadata in a
single JSON column named metadata; the gateway translates S3 Vectors
filter expressions onto it.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 13:26:11 -07:00
Chris Lu 6b379badbb filer: shut down test filers' meta-log buffers, fixing the 32-bit OOM (#12104)
* filer: shut down test filers' meta-log buffers, fixing the 32-bit OOM

Each log_buffer.NewLogBuffer preallocates five 8 MiB windows and spawns
loopFlush/loopInterval goroutines that keep them reachable. The
weed/filer tests built ~70 filers and aggregators that almost never
called ShutdownLogBuffer, pinning ~2.8 GB plus live churn, and the
linux/386 job ran out of its ~4 GiB address space inside
newSealedBuffers.

- MetaAggregator.Shutdown now retires MetaLogBuffer too: the "aggr"
  buffer's loops previously outlived the aggregator.
- newTestFiler and newTestFilerWithStubStore register t.Cleanup
  shutdowns; the one-off filer/aggregator literals defer theirs.

* filer: wait for the aggregated meta-log buffer to stop in Shutdown

ShutdownLogBuffer only signals the stop; loopFlush and its pooled windows
exit asynchronously. Match Filer.Shutdown's local-buffer sequence so the
aggregator returns with the buffer's memory already released.

* filer: cancel test filers' flush context before buffer shutdown

The stub-store helpers build real NewFiler instances, so the queued
shutdown flush retries appendToFile every ~737ms under flushCtx, which
nothing ever cancelled — the buffer and its pool stayed alive past each
test. Cancelling first lets the flush drop the window and WaitForShutdown
confirms loopFlush exited.

* filer: wait for test log buffers to drain after shutdown

The stub-flush buffers cannot hang, but without WaitForShutdown their
loopFlush goroutines release the pooled windows asynchronously — the
memory this PR means to free is still counted as live while the next
test allocates.
2026-10-26 13:24:48 -07:00
Chris Lu aff18b2336 vector: restore spike CLI flags lost in the #12098 rebase (#12099)
* vector: --vector takes hyphenated floats again

* vector: --vector-column/--key-column on the get subcommand

* vector: --vector-column/--key-column on describe and coverage

* vector: surface describe resolve errors when column flags are set

* vector: cover the restored CLI flags with parse tests
2026-10-26 12:24:01 -07:00
Chris LuandDevin 72ac424198 filer: deliver meta-log events under their stream position, not their stamp (#12084)
* filer: deliver meta-log events under their stream position, not their stamp

A mutation's event timestamp is stamped before the filer store write so
the entry and its event share a commit time, while the log-buffer append
happens after the write. A write delayed behind later-stamped mutations
keeps its early stamp in the payload while AddDataToBuffer bumps
LogEntry.TsNs to keep the stream monotone - so subscribers could
receive events whose payload TsNs arrives out of order.

Consumers treat the delivered TsNs as a stream position (cursors,
watermarks, replay windows). The s3 lifecycle daily-run drains events in
order and pins its per-shard cursor behind the first not-yet-due match;
an event delivered with a stale stamp after the cursor already passed it
is filtered out forever and its pending match silently never fires -
the flaky test_lifecyclev2_expiration signature of exactly one object of
a same-rule pair surviving, with errors=0 on every pass.

Normalize event.TsNs to logEntry.TsNs at delivery (and for the
skeleton-filtered heartbeat) so delivered timestamps are monotone by
construction. The commit-time stamp remains available on the entries
themselves via CommitTsNs.

* pb,filer: stamp metadata events with a mutation timestamp

TsNs on the wire now carries the delivery stream position, so the
mutation's commit stamp needs its own field. Add mutation_ts_ns to
SubscribeMetadataResponse and set it when the event is created.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: carry the mutation stamp through event delivery

Delivery rewrites TsNs to the stream position; the payload stamp survives
in MutationTsNs, backfilled from TsNs for payloads written before the
field existed. The notification callback forwards it to the wire message.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* pb: normalize chunk-replay timestamps like eachLogEntryFn

ReadLogFileRefs bypasses the server decode path, so a delayed append
still arrived under its stale payload stamp and could hide behind a
subscriber cursor. Deliver the stream position and preserve the stamp in
MutationTsNs.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: order replay and tombstones by the mutation stamp

Replay treated the delivered TsNs as the commit time, so a peer write
delayed in its own log could overwrite a newer local entry. Compare and
stamp with MutationTsNs, falling back to TsNs for older peers.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* remote.sync: carry the mutation stamp through reshaped renames

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: test replay ordering on the mutation stamp

* filer: tombstone remote deletes by delivery position, order by commit stamp

* filer: keep the peer commit stamp when restamping aggregated events

* pb: keep the log cursor on the entry position past callback restamps

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 12:18:40 -07:00
Chris Lu 68565fa883 admin: bump a-h/templ to v0.3.1100 to match @latest codegen (#12100)
CI installs the templ CLI with `@latest`, which now generates code
calling templruntime.NewAttributeExpression — a symbol added after the
v0.3.1070 runtime pinned in go.mod, breaking `make install` builds on
clean runners. Bump the runtime to v0.3.1100 and regenerate.
2026-10-26 12:03:41 -07:00
Chris LuandDevin 3abf8b2572 filer,pb,mount: keep per-chunk cipher keys out of logs (#12085)
* filer: format ChunkView without its cipher key

readChunkSliceAt logs a failed chunk fetch with %+v, which printed the
per-chunk AES-256-GCM key into error-level logs on every failed read.
Give ChunkView a String() that omits CipherKey so %v and %+v stay safe.

* pb: scrub cipher keys when an entry must be logged whole

ScrubCipherKeys clones an Entry with chunk CipherKey fields cleared, so
a warning that needs the full entry still cannot leak the keys.

* log entry identifiers instead of whole metadata messages

Several log lines formatted whole protobuf messages — SubscribeMetadataResponse,
EventNotification, Entry-carrying requests — whose chunks include cipher_key
under -encryptVolumeData, so the keys escaped into logs (some at default
verbosity). Log the directory, entry name and event timestamp instead; where a
chunk list was printed, print file ids via FileIdsString.

* log queue: unwrap SubscribeMetadataResponse and never print raw protos

The fallback printed any message with %+v, so a SubscribeMetadataResponse
carrying entries would still put cipher keys in the log. Unwrap its event
notifications (batched events included) and reduce unknown messages to
their type name.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* remote sync logs: name the event's real path

Rename targets logged under resp.Directory while the entry lives under
NewParentPath; route them through MetadataEventTargetDirectory. Sync
failure logs printed only the directory, so same-dir failures shared an
identifier — log the source and destination paths instead.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* log queue: log the entry's destination path on moves

A cross-directory rename printed "file" -> "file" under the source
path, hiding where the entry landed. Use NewParentPath so the log
shows the destination directory next to the new name.

* admin: install the go.mod-pinned templ, not @latest

install-templ fetched templ@latest, which now emits
templruntime.NewAttributeExpression - a symbol absent from the
runtime pinned in go.mod - so every fresh 'make' regenerated
uncompilable _templ.go files. Install the go.mod version and
reinstall when the binary on PATH reports a different one.

* admin: drop stray Makefile change belonging to the templ pin PR

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* notification: join the move destination path in log queue output

A NewParentPath of "/" printed //name for moves into the root;
path.Join keeps a single leading slash.

* log queue: skip a nil event notification instead of logging an empty move

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* pb: tolerate nil chunk entries while scrubbing cipher keys for logs

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* subscription and backup failure logs name the event's file

The process/marshal failure logs printed only the event directory and
timestamp, so a failed sync left operators guessing which file failed.
Use MetadataEventSourceFullPath/MetadataEventTargetFullPath so each
failure names the source and target paths without printing the entries.

* replication: identify entry-less events without printing the proto

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: scrub cipher keys from the bucket-metadata debug dump

buildBucketMetadata marshals the whole filer entry to JSON for a V(3)
log line; a chunked entry would emit each chunk's cipher_key base64.
Marshal the scrubbed copy instead.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 11:52:56 -07:00
Chris Lu c27d7fe5a4 s3: keep transient object-entry errors out of ErrObjectNotFound (#12083)
* s3: keep transient object-entry errors out of ErrObjectNotFound

getObjectEntry wrapped every fetch error as ErrObjectNotFound, so a
transient filer failure (busy filer, ctx deadline, transport blip) was
indistinguishable from a real miss. LifecycleDelete resolves
ErrObjectNotFound permanently as NOOP_RESOLVED, so one flaky lookup at
dispatch time silently skipped the object's expiration - errors=0, no
retry, no log. This matches the flaky s3tests
test_lifecyclev2_expiration signature: exactly one object of a
same-rule pair surviving past its window.

Only collapse genuine misses (filer not-found, version/latest-version
sentinels); everything else propagates so the worker retries it as
RETRY_LATER instead of resolving it.

Also seed bucketConfigCache in the retention test server and teach the
fake filer to inject per-path lookup errors.

* s3: keep invalid version ids on the not-found path in getObjectEntry

A malformed versionId used to collapse into ErrObjectNotFound like every
other fetch error, so the handlers answered NoSuchKey. With transient
errors now propagated, an invalid id would surface as InternalError;
fold it back into the miss classification.

* filer_pb: keep Unavailable out of LookupEntry's not-found normalization

* s3: reject Unavailable in isObjectEntryMiss before message matching

* s3: keep transient fallback errors out of latest-version resolution

* filer_pb: limit LookupEntry's not-found text match to miss statuses

* s3: limit isFilerNotFound text matching to miss statuses
2026-10-26 11:52:15 -07:00
Chris LuandDevin 8498ad716b vector: spike CLI, reference-comparison suite, CI (P0 phase 3) (#12098)
* vector: project _distance and _rowid explicitly

Lance warns that a knn projection without _distance will stop
auto-including it; spell both scoring columns out.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: reference-comparison tests

The spike's exit gate: assert the engine's answers equal a bare
scan().nearest() on the same pinned version — over a fresh index, over
unindexed fragments, over a two-segment index after optimize_indices,
and after a delete. A version-pinned handle querying through
maintenance sees only its own version.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: weed-vector spike CLI

describe / coverage / query / get against a namespace-registered table
or a plain dataset URI, so the spike runs against a real SeaweedFS Lance
table bucket and against a local fixture alike. Flag and env names
mirror weed-worker.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: rust-vector-tests workflow

Build and test seaweed-vector in release on pushes and PRs touching it
or its path-dependency trees, mirroring rust-worker-tests.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: derive Serialize for the spike CLI's JSON output

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: --postfilter on the spike CLI

Track the Query field rename; the flag stays an explicit opt-out of
prefiltering for comparison runs.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: assert the pinned query's answers

The maintenance test passed even with zero pinned hits: an empty result
satisfies all() trivially. Assert the hit count and compare the pinned
answers against the reference on the pinned handle.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: vector tests also watch plugin.proto

weed-vector-core path-depends on weed-lance-worker, which compiles
weed/pb/plugin.proto through seaweed-worker-core; a proto-only change
would skip these checks and could break this workspace.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: describe prints layout and coverage

The describe doc comment promised the vector layout and index coverage;
print both, and propagate count_rows errors instead of printing 0.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 11:48:46 -07:00
eda4b9efdd vector: ANN coverage introspection + prefilter (P0 phase 2) (#12097)
* vector: share the test dataset builder

The coverage tests need the same tempdir Lance datasets the query tests
build; move them into a cfg(test) helper module.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: prefilter the metadata filter

prefilter=false post-filters the top-K rows, which can return fewer than
k results and never sees unindexed rows beyond the nearest cut. Expose
the choice so the S3 Vectors layer can honour AWS filter semantics.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: index coverage introspection

coverage() unions each index's fragment bitmaps against the manifest's
fragment list to report which fragments a vector index covers, which it
has not reached, and how many rows the flat pass still has to scan.
Tests pin the behaviour the spike is for: rows appended after an index
build stay in the top-K, and a version-pinned dataset never sees the
next commit.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: default filtered queries to prefilter

postfilter=false is now the normal path: the metadata filter cuts the
eligible rows before ranking, so a query returns k hits whenever k
eligible rows exist. Lance's post-filter remains reachable through
Query::postfilter for comparison runs.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: intersect index bitmaps with the live manifest

A bitmap records fragments the index covered when built; a fragment the
manifest dropped afterwards stayed in the reported coverage.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: keep bitmap-less indices in coverage

An index whose metadata has no fragment bitmap was dropped from the
report entirely, indistinguishable from no index at all. Count its
segments and mark its coverage unknown instead.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: count live rows in unindexed coverage

physical_rows counts deleted rows and can be absent in the manifest;
sum FileFragment::count_rows for the unindexed fragments instead.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: error on a null key in a selected key column

* vector: compare filtered ANN and flat hits as sets

* vector: coverage reports the checked-out version

* vector: project _distance and _rowid explicitly

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: chrislusf <chris@seaweedfs.com>
2026-10-26 11:45:07 -07:00
Chris LuandDevin 7388d0d4fc vector: seaweed-vector workspace + flat/indexed top-K engine (P0 phase 1) (#12096)
* vector: scaffold the seaweed-vector workspace

Standalone cargo workspace for the vector query engine, mirroring the
seaweed-worker layout: own lockfile, same lint policy, path dependencies
across the two trees since there is no common root manifest.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: resolve a table's vector layout

VectorTable::resolve finds the vector column (explicit name, else the
column named vector, else the sole fixed-size float list), the optional
key column, and the metadata columns a query can project.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: flat and indexed top-k, key lookup

query::flat runs an exact scan (use_index(false)); query::ann lets Lance
split the query between index segments and a flat pass over fragments no
index covers yet, so rows written after the last index build stay
eligible. Both honour a metadata filter and return the raw _distance in
the query's metric.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: reject a non-utf8 or non-identifier key column

An explicit --key-column naming a non-UTF8 field made hits() silently
report row ids as keys, and get() interpolated the name into a SQL
filter verbatim. Require Utf8 at resolve and in hits(), and require a
plain identifier before building the IN predicate.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: fall back to row id on a null key

keys.value() on a null slot fabricates an empty-string key; a row with
no key reports its row id instead, matching the no-key-column case.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: quote the key column in the IN filter

A Utf8 key column named e.g. document-key made the predicate parse as
subtraction, and an identifier interpolated unquoted is injectable.
Backtick-quote it and double its inner backticks; double quotes are
string literals in the filter dialect, not identifiers.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: project only the columns a hit needs

The knn projection pulled every metadata column — including other
vector columns — although Hit returns none of them. Filters still
evaluate off the scan path, so restrict the output to the key and the
scoring columns.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 10:13:05 -07:00
Chris LuandDevin 6f8f8e7079 admin: install the go.mod-pinned templ, not @latest (#12090)
* admin: install the go.mod-pinned templ, not @latest

install-templ fetched templ@latest, which now emits
templruntime.NewAttributeExpression - a symbol absent from the
runtime pinned in go.mod - so every fresh 'make' regenerated
uncompilable _templ.go files. Install the go.mod version and
reinstall when the binary on PATH reports a different one.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* admin: invoke templ from Go's install dir, not PATH

A templ binary earlier on PATH than GOBIN still shadowed the pinned
compiler after install-templ ran, so generate kept emitting code for
the wrong runtime. Resolve GOBIN (or GOPATH/bin) and call the pinned
binary by absolute path for install checks, generate, watch, and fmt.

* admin: regenerate templ output with the pinned v0.3.1070

The checked-in _templ.go files were stamped v0.3.1020, older than the
go.mod pin. Regenerate so the committed code matches the compiler the
runtime links against.

* admin: resolve templ under the first GOPATH entry

go install drops binaries in the first GOPATH element's bin, so a
multi-element GOPATH left TEMPL pointing at a path that does not exist.
Split on the OS list separator so Windows GOPATHs resolve too.

* admin: install pinned templ before fmt and pattern-rule generates

fmt and a direct %_templ.go rebuild invoked TEMPL without ensuring the
pinned binary exists; an order-only prerequisite keeps the check cheap
without forcing regeneration.

* admin: keep spaced GOPATH/templ paths intact in the Makefile

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 09:30:33 -07:00
6ebc055a94 master, volume, filer: start on a read-only mounted directory and serve existing data read-only (#12081)
* master, volume, filer: start on a read-only mounted directory and serve existing data read-only

A read-only mount was treated as writable (permission bits only), so every
service failed on its first write: the master panicked writing raft conf.tmp,
the filer exited opening the leveldb LOCK, and the volume server skipped every
volume while still offering the directory as free capacity.

- util.IsReadOnlyMount: access(W_OK) == EROFS.
- volume: on a read-only -dir take the existing O_RDONLY/noWriteOrDelete load,
  index .idx in memory when no .sdx exists, offer no free slots (same branches
  as isDiskSpaceLow), skip the vid stamp there.
- master: on a read-only -mdir run raft from a private copy and refuse new
  volume ids.
- filer: leveldb2/leveldb3 set their existing ReadOnly option.

* volume: skip read-only mounts when picking EC shard copy targets

A shard copy landing on a read-only mount fails at write time; offer
only writable locations like the other placement paths do.

* volume: mirror the read-only mount handling in the Rust volume server

The .dat open only fell back to read-only on EACCES, so a real read-only
mount (EROFS) still failed the load. The index side already degrades
once no_write_or_delete is set. Add is_read_only_mount via access(W_OK)
== EROFS, mark DiskLocation.read_only_mount, and mirror the Go checks:
no free slots, heartbeat max = used, no vid-stamp writes, no EC shard
copy targets, and a read-only index dir forces the volume read-only.

* volume: write vid stamps while only the index dir is read-only

* filer: skip the write-bit check when a leveldb store opens read-only

* filer: serve reads of buckets with no leveldb3 dir on a read-only store

* volume: derive a stable dir uuid when a read-only mount cannot store one

* master: keep the read-only raft copy across restarts so votes survive

* volume: index a missing .idx in memory on a read-only mount

* volume: release the .idx handle after an in-memory read-only load

* volume: mount EC shards read-only when their index dir is on a read-only mount

* volume: mark the index dir read-only before the tail repair opens .idx

* volume: write vid stamps while only the index dir is read-only (Rust)

* volume: derive a stable dir uuid and report no free slots on a read-only mount (Rust)

* volume: index a missing .idx in memory on a read-only mount (Rust)

* volume: refuse an EC copy whose shard owner is on a read-only disk

* master: key the read-only raft copy by the absolute -mdir

Two masters sharing the same relative -mdir text from different working
directories would have collided on one temp copy and shared raft state.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: mix device and host into the read-only dir uuid

Deriving the uuid from the path alone let independent servers at the
same mount point report one uuid, which RegisterUuids rejects. Hash in
the device id and hostname so restarts stay stable but disks differ.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: vic6996-web <320442181+vic6996-web@users.noreply.github.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 09:28:49 -07:00
dependabot[bot] 0214f9b34e build(deps): bump github.com/pkg/sftp from 1.13.11 to 1.13.12 (#12086)
Bumps [github.com/pkg/sftp](https://github.com/pkg/sftp) from 1.13.11 to 1.13.12.
- [Release notes](https://github.com/pkg/sftp/releases)
- [Commits](https://github.com/pkg/sftp/compare/v1.13.11...v1.13.12)

---
updated-dependencies:
- dependency-name: github.com/pkg/sftp
  dependency-version: 1.13.12
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-26 09:24:50 -07:00
dependabot[bot] b7d3c8a5a6 build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azcore from 1.23.2 to 1.23.3 (#12087)
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azcore

Bumps [github.com/Azure/azure-sdk-for-go/sdk/azcore](https://github.com/Azure/azure-sdk-for-go) from 1.23.2 to 1.23.3.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.23.2...sdk/azcore/v1.23.3)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/azcore
  dependency-version: 1.23.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-26 09:24:36 -07:00
dependabot[bot] 74161edb7a build(deps): bump github.com/getsentry/sentry-go from 0.49.0 to 0.50.0 (#12089)
Bumps [github.com/getsentry/sentry-go](https://github.com/getsentry/sentry-go) from 0.49.0 to 0.50.0.
- [Release notes](https://github.com/getsentry/sentry-go/releases)
- [Changelog](https://github.com/getsentry/sentry-go/blob/master/CHANGELOG.md)
- [Commits](https://github.com/getsentry/sentry-go/compare/v0.49.0...v0.50.0)

---
updated-dependencies:
- dependency-name: github.com/getsentry/sentry-go
  dependency-version: 0.50.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-26 09:24:25 -07:00
dependabot[bot] 07cfe2d5e2 build(deps): bump github.com/redis/go-redis/v9 from 9.22.0 to 9.23.0 (#12088)
Bumps [github.com/redis/go-redis/v9](https://github.com/redis/go-redis) from 9.22.0 to 9.23.0.
- [Release notes](https://github.com/redis/go-redis/releases)
- [Changelog](https://github.com/redis/go-redis/blob/master/RELEASE-NOTES.md)
- [Commits](https://github.com/redis/go-redis/compare/v9.22.0...v9.23.0)

---
updated-dependencies:
- dependency-name: github.com/redis/go-redis/v9
  dependency-version: 9.23.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-26 09:24:09 -07:00
dependabot[bot] 692423d641 build(deps): bump github.com/ThreeDotsLabs/watermill from 1.5.2 to 1.5.3 (#12091)
Bumps [github.com/ThreeDotsLabs/watermill](https://github.com/ThreeDotsLabs/watermill) from 1.5.2 to 1.5.3.
- [Release notes](https://github.com/ThreeDotsLabs/watermill/releases)
- [Changelog](https://github.com/ThreeDotsLabs/watermill/blob/master/RELEASE-PROCEDURE.md)
- [Commits](https://github.com/ThreeDotsLabs/watermill/compare/v1.5.2...v1.5.3)

---
updated-dependencies:
- dependency-name: github.com/ThreeDotsLabs/watermill
  dependency-version: 1.5.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-26 09:23:51 -07:00
dependabot[bot] 2ffe61189f build(deps): bump actions/github-script from 7 to 9 (#12095)
Bumps [actions/github-script](https://github.com/actions/github-script) from 7 to 9.
- [Release notes](https://github.com/actions/github-script/releases)
- [Commits](https://github.com/actions/github-script/compare/v7...v9)

---
updated-dependencies:
- dependency-name: actions/github-script
  dependency-version: '9'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-26 09:23:32 -07:00
dependabot[bot] 41c33bdb21 build(deps): bump github/codeql-action from 4.39.0 to 4.39.1 (#12092)
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.39.0 to 4.39.1.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/v4.39.0...v4.39.1)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-version: 4.39.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-26 09:20:41 -07:00
d010664634 build(deps): bump docker/login-action from 4.6.0 to 4.7.0 (#12094)
* build(deps): bump docker/login-action from 4.6.0 to 4.7.0

Bumps [docker/login-action](https://github.com/docker/login-action) from 4.6.0 to 4.7.0.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](https://github.com/docker/login-action/compare/v4.6.0...v4.7.0)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-version: 4.7.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* ci: correct stale version comments on SHA-pinned docker actions

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 09:20:15 -07:00
dependabot[bot] 66e8211821 build(deps): bump actions/download-artifact from 7 to 8 (#12093)
Bumps [actions/download-artifact](https://github.com/actions/download-artifact) from 7 to 8.
- [Release notes](https://github.com/actions/download-artifact/releases)
- [Commits](https://github.com/actions/download-artifact/compare/v7...v8)

---
updated-dependencies:
- dependency-name: actions/download-artifact
  dependency-version: '8'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-26 09:19:02 -07:00
a235473105 [Volume] Preserve request semantics in replica proxy reads (#12075)
* fix 12074

* volume: shuffle a copy of cached replica locations for proxy reads

LookupVolumeId serves cached location slices shared between lookups; shuffling in place races with concurrent proxied reads of the same volume.

* seaweed-volume: forward inbound method and surface lookup failure in proxy reads

Mirror the Go handler: proxy_request used to always send GET, turning a client HEAD into a replica download, and a failed master lookup returned 404 like missing data. Forward the inbound method and answer 500 for lookup failures in proxy mode.

* volume: keep master lookup error detail out of the proxy error response

* volume: bound the proxied-request wait in the method-preservation test

* seaweed-volume: answer 404 for missing volumes in proxy-mode lookups

Mirror Go's LookupVolumeId: a per-volume error in the master's lookup
reply (e.g. a missing volume) means "no locations" and answers 404
downstream, not a lookup failure that proxy mode answers 500.

* volume: guard nil lookup result in proxy path

A master response that omits the requested vid leaves
LookupVolumeId returning a nil *LookupResult with a nil error;
dereferencing it panics the read handler.

* seaweed-volume: answered lookup errors are missing data, not failures

A master reply carrying a per-volume error with HTTP 404 (volume not
found) now yields empty locations like Go's LookupResult, so proxy mode
answers 404. Other failure statuses such as 503 during master warmup
remain lookup failures that proxy mode answers 500.

* seaweed-volume: drop hop-by-hop headers from replica proxy reads

Go's inbound request drops Host, Content-Length and Transfer-Encoding
into dedicated fields, so its forwarded header set never carries them.
hyper keeps them; forwarding a client's Content-Length with no body
stalls the replica's read, and forwarding its Host misnames the target.

* volume: bound the proxy master lookup to the request context

* volume: drop hop-by-hop and Connection-nominated headers from proxy reads

* seaweed-volume: drop Connection-nominated headers from proxy reads

* volume: skip the proxy lookup error when the inbound request is gone

* volume: drop proxy-specific headers from replica proxy reads

* seaweed-volume: drop proxy-specific headers from replica proxy reads

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: drop hop-by-hop headers from replica proxy responses

* seaweed-volume: drop hop-by-hop headers from replica proxy responses

* seaweed-volume: a failed lookup status without an error body is still a failure

* volume: do not re-proxy an already-proxied read for a missing volume

A proxied request that reaches a server which still lacks the volume was
looked up and forwarded again, so a stale location answer could bounce
the read between two servers until the client gave up. The Rust volume
server's proxy_missing_volume already returns 404 for proxied=true;
mirror it here.

* seaweed-volume: replace stale proxied params in the forwarded query

Go builds the proxied URL with query.Set(reqIsProxied), which replaces
every proxied param the client sent. Appending after a client-supplied
proxied=false left the replica's first-value lookup reading the stale
marker, defeating the loop guard.

---------

Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 08:13:34 -07:00
Chris LuandDevin dfd9ecab05 mount: resume metadata-log replay where a dropped stream stopped (#12079)
* pb: checkpoint metadata-log replay progress per settled entry

A chunk-mode subscriber's resume cursor only advanced after the whole
LogFileRefs batch drained, so a stream dropped mid-replay resubscribed
from the original stale position and re-read the same log files while
live traffic kept appending - under a flapping filer the replay window
never shrank, and each reconnect re-listed and re-fetched the same
backlog.

ReadLogFileRefs now reports every entry the merge has settled -
delivered or filter-skipped - and FollowMetadata checkpoints StartTsNs
on it, so a reconnect resumes where the previous stream stopped and the
backlog drains across restarts. The peer aggregator wires the same
progress into its resubscribe/offset position.

* pb: emit a replay checkpoint only after the merge passes its timestamp

Checkpointing an entry's timestamp the moment it settles could strand a
same-timestamp entry still queued on another filer's stream: resuming is
exclusive, so a reconnect would skip the unsettled half of the tie. The
merge now emits a timestamp only once a later entry proves every tied
entry was delivered (and once at clean EOF).

* pb+filer: never regress the replay cursor below the settled checkpoint

The drain's final assignment took the last settled entry's own timestamp,
which can lag the log position progressFn just checkpointed - a skewed or
legacy entry rewinds option.StartTsNs (and the peer's lastTsNs), so the next
drain or a resubscribe re-reads the settled prefix. Guard both assignments
so the cursor only moves forward.

* pb: cover the resubscribe path of replay checkpoints

A drain that aborts mid-replay leaves StartTsNs at the last checkpoint
the merge passed, and the next subscribe resumes there instead of
re-reading the whole ref set. The test drives the real
makeSubscribeMetadataFunc path: refs, a mid-batch apply failure, then a
resubscribe whose SinceNs must carry the checkpoint.

* pb: emit the tail replay checkpoint only for a settled timestamp

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 08:06:15 -07:00
0d721fccee fix(volume): keep preallocated growth above -volume.minFreeSpace (#12067)
* fix(volume): keep preallocated growth above -volume.minFreeSpace

When free space fell below -volume.minFreeSpace, every volume went
read-only and the master kept growing new volumes. With preallocation on,
each new volume reserved its full size, driving the disk further below the
floor (15 Oct incident: 66 empty volumes reserved about 1980 GiB).

Two causes, both in weed/storage/store.go:

- MaybeAdjustVolumeMax sized a disk's volume slots from raw free space, so
  space inside the floor was counted as available. It now counts only free
  space above the floor.
- addVolume trusted the once-a-minute low-disk flag. A preallocated volume
  is now refused when the disk is low now, or when its reservation would
  not fit above the floor. Volumes without preallocation, including merges
  that copy existing data, are not affected.

addVolume runs under a store-level lock, so two concurrent creates cannot
both pass the room check against the same free space.

* volume: mirror headroom guard in the rust store

find_free_location only trusted the once-a-probe is_disk_space_low flag,
so a burst of preallocated creates could still run the disk under
-volume.minFreeSpace between probes, and maybe_adjust_volume_max sized
slots from raw free space. Adds MinFreeSpace::is_low/available_space to
match Go and applies both checks the same way. store.write() already
serializes add_volume, so no extra lock is needed.

* volume: read rust headroom tests from a fake disk

The floor arithmetic used a live free-space reading, so unrelated disk
activity could flip a test between pass and fail. Store.disk_stats_of is
now the same seam Go uses (diskStatusOf): the growth-loop and floor tests
run against a controlled disk, mirroring the Go suite one for one.

* volume: hold back create metadata from the headroom check

An exact-fit preallocation left free space exactly at the floor, so the
.vif and .idx writes that follow a create could dip under it and get the
location flagged read-only at the next probe. Reserve room for .vif on
the data filesystem, and for .idx growth only when dir_idx shares it.

* volume: mirror the metadata allowance in the rust store

Same rule as the Go side: charge .vif against the checked filesystem and
.idx only when dir_idx shares it, and grow the exact-fit test's fake
disk by the per-create cost.

* volume: skip the reservation check where create cannot preallocate

* volume: mirror the platform preallocation gate in the rust store

* volume.copy: count the preallocation reservation in the destination space check

The copy path picked a location with room for the copied .dat, then reserved
the master's full volume size limit on it. A volume smaller than the limit
could land on a disk whose free space above -volume.minFreeSpace fit the
copy but not the reservation, driving the disk under the floor the same way
the growth loop did.

Fetch the preallocation setting before choosing the location and require
room for the reservation itself, only where CreateVolumeFile actually
reserves blocks.

* seaweed-volume: store min-free-space-adjusted available space like Go

Go's DiskLocation.AvailableSpace holds free space above the
-volume.minFreeSpace floor; the Rust probe cached raw free instead, so the
VolumeCopy location check could accept a disk already inside the floor.
Store the same adjusted figure and cover it with a test.

* seaweed-volume: reserve the copied .dat like Go's CreateVolumeFile

VolumeCopy called set_len for the reservation, which only grows a sparse
file — no blocks are held, so a disk with room for the copied bytes but
not the full reservation only fails mid-copy. preallocate_file fallocates
KEEP_SIZE on Linux like Go, reserving the blocks up front; off Linux it
is a no-op, matching PREALLOCATE_SUPPORTED.

* seaweed-volume: charge a copy's destination pick for the .dat reservation

Go's VolumeCopy now sizes its free-location check against the master's
volume size limit when the platform preallocates, not just the copied
bytes. The Rust pick only measured the copied bytes, so a destination
could accept a copy whose reservation then dips under the floor. The
config fetch moves ahead of the pick (same order as Go) and feeds both
the space check and the reservation itself.

* volume.copy: charge the picked location's free space under volumeCreateLock

The copy path picked a destination off the once-a-minute AvailableSpace
probe and released the decision long before the .dat reservation landed.
Two copies could pass the same cached headroom and together drive the disk
under -volume.minFreeSpace.

ReserveFreeLocation serializes the pick under volumeCreateLock, verifies
the disk still has live room for the reservation, and charges it against
the location's cached free space until the next probe refreshes it.

* seaweed-volume: mirror the copy pick's live-check and headroom charge

Go's VolumeCopy now picks through Store.ReserveFreeLocation: live disk
stats under the store write lock, then the reservation charged against the
location's cached free space. The Rust plan_copy_destination gains the
same store.reserve_free_location so two copies cannot spend the same
headroom before the first .dat reservation lands.

* volume.copy: saturate the free-space charge against a live-space drop

The two statfs reads bracketing a pick can race a large external write; a
plain subtraction would underflow and leave the location looking roomy
until the next probe.

* volume.copy: release the headroom reservation when a copy does not land

ReserveFreeLocation charges the picked location's cached free space, but
a failed copy never returned it, so the headroom stayed booked until the
next disk-space probe and blocked retries. ReleaseFreeLocation refunds
the charge; the copy's cleanup defer now registers with the reservation
so every failure path (existing-replica delete, .note write, stream send)
releases it, including the post-transfer mount/validate failures covered
by the outer defer.

* seaweed-volume: mirror the copy headroom release on failed copies

release_free_location refunds reserve_free_location's charge. The copy
returns it when the plan's existing-replica delete fails, when the .note
write fails, and in cleanup_failed_copy, so a failed copy stops holding
the headroom until the next disk-space probe.

* volume.copy: keep pending copy headroom out of probe readings

ReleaseFreeLocation refunded the reservation into AvailableSpace even when
a CheckDiskSpace probe had already rebased the cache on live stats — the
probe's reading still shows the in-flight copy's bytes as free, so the
refund double-counted headroom and two copies could spend the same space,
driving the disk under -volume.minFreeSpace.

Track the outstanding charge in DiskLocation.ReservedSpace instead: probes
subtract it so they cannot re-spend it, a failed copy removes only its own
charge, and a landed copy commits it — the charge and the new usage cancel
out. hasRoomForNewVolume respects pending headroom too, so a preallocating
create cannot overrun an in-flight copy.

* seaweed-volume: mirror pending copy headroom across disk probes

reserved_space tracks the in-flight copy's charge separately from the
cached free space: check_disk_space subtracts it, release removes only the
failed copy's charge, a landed copy commits it, and has_room_for_new_volume
respects it — matching the Go side.

* volume.copy: settle a copy's headroom as its bytes land on disk

The pending charge held the copy's full reservation until commit, but
written and fallocated bytes already show up in statfs, so the live-space
check counted them twice. A slow copy could then block other copies and
volume growth despite free space above -volume.minFreeSpace.

Settle the charge instead: commit the bytes CreateVolumeFile actually
reserved (a failed fallocate leaves nothing committed), then settle each
file's write progress as it lands. The failure paths now release only the
unsettled remainder; allocatedSize is exported for the .dat's st_blocks.

* seaweed-volume: mirror copy headroom settling as bytes land

reserved_space keeps only the promised bytes statfs cannot see yet:
preallocate_dat settles whatever blocks the filesystem actually reserved
(0 when fallocate cannot), each copied file's progress settles the rest as
it lands, and a failed copy releases only the unsettled remainder —
matching the Go side.

* volume.copy: release a failed copy's headroom charge only once

Both defers fired on the same in-closure error: the inner released the
unsettled charge and the outer released it again, driving ReservedSpace
negative (an atomic.Uint64 underflow) and inflating AvailableSpace —
the location would then fail every later headroom check, or worse,
pass them with fabricated space. Zero the charge once it is released;
the Rust mirror already releases via swap(0) for the same reason.

* volume.copy: write into the preallocated .dat instead of staging over it

The .dat stream staged into .copying and renamed over the file
CreateVolumeFile had just fallocated, so the copy's transient footprint
was preallocate+dat — more than the reservation the pick just checked —
and the reservation itself was discarded at rename. Writing in place
reuses the fallocated extent, keeps the peak at the reserved maximum,
and leaves the copied volume preallocated like a created one.

* seaweed-volume: mirror the in-place write into the preallocated .dat

The .dat copy opened the just-fallocated file with truncate, freeing its
blocks at open and spending the headroom again as the stream re-grew it.
Append mode preserves them, matching the Go side.

---------

Co-authored-by: chrislusf <chrislusf@gmail.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
2026-10-26 08:03:25 -07:00
b5af82409f fix: volume.fsck -volumeId purges live chunks referenced from outside the volume's bucket (#12076)
* fix: volume.fsck -volumeId purges live chunks referenced from outside the volume's bucket

-volumeId narrowed the filer read to <bucketsPath>/<collection> when
the selected volumes shared one bucketed collection (#9347). A filer
started with -collection, or an fs.configure rule, stores files from
any path in that collection, so the narrowed read misses their
references and reports live chunks as orphans; with
-reallyDeleteFromVolume they are deleted. #11794 already made
-collection read from the root for this reason. Do the same for
-volumeId: the flag keeps selecting which volumes are checked and
purged, and the filer is always read from "/".

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

* test: volume.fsck keeps chunks referenced outside the volume's bucket

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
2026-10-26 07:58:43 -07:00
Nguyễn Lộc Phúcanduser.email 937633a77d iceberg: fix CREATE TABLE rejecting a partition on the first column (#12072)
* iceberg: fix CREATE TABLE rejecting a partition on the first column

Apache Spark number the columns of a new
table from 0 and leave assigning fresh IDs to the catalog, so a table
partitioned or sorted by its first column arrives with source-id 0. The
create-table request decoded its partition spec and write order into
iceberg-go's bound PartitionSpec and SortOrder, which require positive
source IDs, so every such CREATE TABLE, and CTAS through stage-create,
was refused with "Malformed request: Invalid request body".

Decode both as iceberg.UnboundPartitionSpec and table.UnboundSortOrder,
as iceberg-go documents for catalog create-table requests;
newTableMetadata already resolves the placeholders through the request
schema by name and assigns fresh IDs. A spec or order that does not
resolve against the schema is now answered with 400 instead of 500, and
the decode error message carries the underlying reason.

Tests check that Spark's request bodies resolve to the same metadata as
1-based placeholders, that the handler accepts them, that unresolvable
source IDs get 400, and add a Spark integration test that partitions by
the first column.

* iceberg: gofmt the create-table source-id tests

---------

Co-authored-by: user.email <chrislusf@users.noreply.github.com>
2026-10-26 07:54:10 -07:00
a6be3c89c2 filer: support nanosecond timestamps for conditional delete (#11339) (#12071)
* filer: support nanosecond timestamps for conditional delete (#11339)

* shell: fs.verify prune carries the entry mtime in nanoseconds

IfNotModifiedAfter now resolves to sub-second precision on the filer,
so pass the looked-up mtime_ns along with mtime; the re-lookup check
catches same-second rewrites that share the second, and the test mock
mirrors the filer's seconds/nanoseconds cutoff.

* shell: volume.fsck guards empty-directory deletes with mtime nanoseconds

The looked-up mtime_ns joins the seconds in IfNotModifiedAfter so a
directory rewritten within the same second is not dropped; the stub
filer mirrors the filer's scale heuristic and the race fixtures use a
realistic mtime so the nanosecond branch engages.

* filer: cover the seconds guard and equality boundary

Test second-resolution cutoffs on both sides of the entry's second and
the exact-equality nanosecond guard that still deletes.

* filer: compare the nanosecond guard as a time, not UnixNano

UnixNano overflows for mtimes outside 1678-2262, letting a far-future
entry slip past the guard; time.After compares exactly at every epoch.
The seconds branch keeps Unix() so a same-second cutoff still deletes.

* filer_pb: add explicit if_not_modified_after_ns to DeleteEntryRequest

* filer: prefer explicit nanoseconds guard, keep if_not_modified_after as seconds

* shell: send seconds and nanoseconds delete cutoffs, skip unexpressible mtimes

* filer: merge conditional-delete guards into one nanosecond cutoff

DeleteEntryMetaAndData carried both the seconds and nanoseconds guards
even though the precedence between them is fixed, so every caller passed
a dead parameter. The gRPC handler now folds if_not_modified_after into
nanoseconds at the boundary, where (sec+1)*1e9-1 keeps the whole-second
cutoff exact, and the filer takes a single ifNotModifiedAfterNs.

* filer: hold the conditional-delete cutoff as a time, not int64 nanoseconds

A seconds-only guard at or past year 2262 saturated to MaxInt64
nanoseconds (~2262), so a delete whose cutoff matched a far-future mtime
was skipped while the request still reported success, and any entry
between 2262 and the real cutoff survived pruning. The merged cutoff is
now a time.Time, where a whole-second guard maps to the last nanosecond
of its second for any int64 second.

---------

Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-26 07:51:33 -07:00
b85281f0da s3: group ListMultipartUploads by delimiter into CommonPrefixes (#12069)
* s3: group ListMultipartUploads by delimiter into CommonPrefixes

The delimiter was echoed but ignored, so every upload was listed. Keys
that continue past the delimiter after the prefix are now listed once as
a common prefix, counted toward max-uploads like an upload.

* s3: bound tracked multipart common prefixes to the page

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: omit upload id marker when a page ends on a common prefix

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 06:10:09 -07:00
b7b571d469 webdav: copy a file onto a collection URL without losing the collection (#12068)
* webdav: copy a file onto a collection URL without losing the collection

An overwriting COPY of a file to dir/ removed dir, then OpenFile refused
the name ending in / and the request failed with 403, leaving neither.
Drop the trailing slash from the destination when the source is a file.

* webdav: cover file-over-collection COPY through the real handler

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* webdav: keep a root destination out of the slash trim

A file COPY or MOVE onto / would rewrite the Destination to an empty
path, turning the existing refusal into x/net's bad-gateway error; leave
the root untouched so RemoveAll refuses it as before.

---------

Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 06:08:24 -07:00
github-actions[bot] 96f310dec8 docs: regenerate star history chart 2026-10-26 08:02:43 +00:00
Chris Lu ffa87c25fd ci: fix aarch64/amd64 musl rust builds on ubuntu-24.04 (#12070)
The 24.04 bump broke build-rust-binaries: aws-lc-sys C objects compile against noble's glibc 2.39 headers, which redirect sscanf/strtol to the C23 __isoc23_* variants musl does not provide, failing the link.

Same fix as the enterprise build on seaweedfs/artifactory:
- arm64: keep aarch64-linux-gnu-gcc, link a tiny forwarding shim that
  resolves __isoc23_sscanf/__isoc23_strtol to musl's vsscanf/strtol.
- amd64: point CC at musl-gcc so C deps compile against musl headers.
2026-10-25 21:09:37 -07:00
github-actions[bot] 153ab668c9 4.51 2026-10-25 19:39:07 +00:00
Chris Lu 6dd6ad512c iam/oidc: drop single-use web-identity tokens and the jti replay cache (#12065)
* iam/oidc: drop single-use web-identity tokens and the jti replay cache

Reverts the single-use semantics added in #11936 (and the #12017
refactor): AssumeRoleWithWebIdentity consumed each web-identity
token's jti into a bounded replay cache, so a second exchange of the
same token was rejected until exp + skew.

AWS STS accepts a web-identity token repeatedly until it expires,
and Kubernetes workload identity depends on that: the kubelet keeps
a projected ServiceAccount token until ~80% of its lifetime, so pod
restarts, co-located containers sharing the token file, repeated CLI
invocations, and pre-rotation SDK refreshes all re-present the same
jti and were refused on 4.50.

The replay machinery also carried a real maintenance cost — a
bounded concurrent cache, fingerprint-keyed entries for jti-less
tokens, adopt/merge handoffs so provider rebuilds could not reopen
consumed tokens, and provider-reuse bookkeeping in the refresh path
that existed only to preserve that state. Removing it restores AWS
semantics and deletes the whole apparatus.

Refs #12063. Alternative to the per-provider opt-out in #12064.

* iam: reuse unchanged OIDC providers on refresh

refreshOIDCProvidersFrom rebuilt every managed provider on any store
change, so an unrelated write discarded each provider's JWKS cache and
an unreachable issuer could then fail exchanges the cached keys would
have allowed. Keep a per-record provider cache keyed on the record
content fingerprint and only rebuild when the record changes.

* iam/oidc: require exp in ValidateToken

Dropping the single-use check also removed the implicit bound on
expiry-less tokens: a signed web-identity token without exp could
mint STS sessions indefinitely. Require the exp claim, matching AWS
AssumeRoleWithWebIdentity which rejects tokens without one.

* iam/oidc: bound idle connections on the provider transport

A provider replaced on refresh drops its http.Transport without a
close; give it an IdleConnTimeout so pooled connections and their
goroutines cannot linger.

* s3api: cover web-identity token reuse over HTTP

Sign one token and exchange that exact string twice, directly covering
the Kubernetes projected-token reuse this change restores.

* test(iam): refresh reuses an unchanged record's provider

Covers the provider-reuse cache: an unrelated record's refresh keeps a
provider's JWKS state, a changed record rebuilds it.
2026-10-25 12:04:59 -07:00
515380c349 s3: answer GET and HEAD of a delete marker as S3 does (#12051)
* s3: answer GET and HEAD of a delete marker as S3 does

A delete marker as the current version now returns 404 NoSuchKey with
x-amz-delete-marker: true and the marker's x-amz-version-id. Asking for the
marker by versionId returns 405 MethodNotAllowed with the same headers and
Last-Modified, instead of 404 NoSuchKey.

* s3: check the tag-conditioned read grant before answering a delete marker

The marker branch returned before the entry-based policy recheck the rest
of the read path performs, so an s3:ExistingObjectTag-conditioned Deny did
not apply to the delete-marker response and its version headers.

* s3: send Allow: DELETE with the delete-marker 405

RFC 9110 15.5.6 requires a 405 answer to list the methods the resource
supports; S3 sends allow: DELETE for a delete marker named by versionId.

* s3: keep delete-marker headers on conditional-read misses

If-Match or If-Unmodified-Since against a current delete marker short-
circuited to a bare NoSuchKey. Carry the resolved marker through the
conditional result so the 404 keeps x-amz-delete-marker and
x-amz-version-id, matching the unconditional read.

* s3: exercise delete-marker responses through GET and HEAD handlers

* s3: read marker version id from the entry before the caller's guess

A delete marker written outside the normal PUT path can miss the
version-id attribute while the v_<id> file name still names it; the
new x-amz-version-id header would then report "null". Reuse
versionIdFromEntry so both the handler and conditional-miss paths
emit the real id.

* s3: answer 405 for versionId=null on a suspended-versioning marker

The null marker lives in .versions while getSpecificObjectVersion only
checks the plain path, so GET/HEAD ?versionId=null answered
NoSuchVersion. On that miss, look up the null marker and answer it
like any other marker named by versionId.

* s3: propagate null-marker lookup failures instead of answering 404

A transient filer error reading .versions/v_null was swallowed as
"no marker", so GET/HEAD answered NoSuchVersion for a live read
failure. NotFound and absent entries still mean no marker; anything
else is InternalError now.

---------

Co-authored-by: Chris Lu <chris.lu@gmail.com>
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
2026-10-25 12:02:19 -07:00
Chris LuandDevin 9620e7ebfe filer: auto-enable per-path commit records when a peer has its own store (#12062)
* filer: enable per-path commit records only when a peer has its own store

Since seaweedfs#11698 every metadata change writes a per-path commit
record into the filer store (an MpTs key, plus an MpDl key on deletes),
and the records stay after the path is deleted. They order the events a
filer replays from peer filers that keep their own store, since replay
only runs when peerSignature != f.Signature.

Filers that all share one store (one Redis, MySQL, TiKV, ...) never
replay each other's events, so the records are never read, yet each
change still costs one or two extra store writes and the key count grows
with every path ever changed — unbounded growth for in-memory stores.

This turns the records off by default and latches them on the first time
a peer reports a different store signature — no option needed. The latch
is persisted in the store (MpRep) so it survives restarts and converges
filers sharing a store even before they discover the peer themselves.
While off, a filer keeps an MpOff marker so enabling later bumps the
records' generation (MpGen, folded into each record key) past the
unrecorded window: a record older than a change made while off cannot
let an older replayed peer event win. Ordering for unrecorded changes
falls back to the entry's mtime, the same as entries written before the
records existed.

Supersedes the manual option approach in seaweedfs#12060.

* filer: keep delete tombstones while records off, harden enable path

Review on the first commit:

- A delete committed while the records were off left no entry and no
  usable tombstone: after enabling, an older replayed create could
  resurrect the path. Deletes and rename sources are now always recorded
  under the generation-0 MpDl key — they are never fenced by a
  generation bump, and they cannot go stale because every delete is
  recorded. Creates/updates still write nothing while off.

- bumpPathCommitGen could regress MpGen under concurrent bumpers on a
  shared store. It now loops: each pass writes only a generation higher
  than any it has seen, so concurrent bumps converge upward.

- doSubscribeToOneFiler proceeded into replay when
  enablePathCommitRecords failed, running the whole session without
  ordering. It now aborts the subscribe so the reconnect retries with
  the records on.

* filer: bind pending commit-record repairs to their generation

A failed MpTs write kept in metaTsPending could land after a shared-store
generation bump, writing a pre-bump timestamp as a current-generation
record and overriding newer unrecorded state. Each pending change record
now carries the generation it failed under: repairs write it only while
that generation is current, and ordering reads ignore it otherwise.
Pending delete records need no binding — MpDl keys are never fenced.

* filer: never move the local records' generation backward

The converge loop in bumpPathCommitGen seeded from zero and both it and
syncPathCommitGen stored unconditionally: a stale writer could leave the
stored generation below what this filer already adopted, and the next
sync or bump then walked the local value back — orphaning records it
wrote under the higher generation. The bump now seeds from the local
generation and every store only moves it forward.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: name a records generation after the off marker it consumes

Two filers consuming the same off marker each counted one past the
stored generation, so the second could store N+2 after the first
stored N+1 and write records under a key the other could not read
until the next check. The bump now takes the marker's own timestamp —
filers racing on one marker land on one value — a marker older than
the stored generation is cleaned as a leftover, and a stored value
that already passed the bump is adopted instead of counted past. A
newly enabling filer whose marker was already consumed names its own
off window with the current time, so its unrecorded period is still
fenced.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: tell a fresh off marker from one the generation already consumed

A marker whose timestamp reads at or below the stored generation was
dropped as leftover, but a new off period can also read that old — the
off filer's clock lagging, or a host clock moving back — and dropping
it kept records from before that period live, so stale replay could
win.

The stored generation now names the marker it was bumped for: a marker
the store still shows is consumed unless it is that one, however old
it reads, and only the consumed marker itself is cleared without a
bump. The generation trails in the value, so a whole-slice read still
yields it on stores written before the pair was stored. Enabling
fences the filer's own off window with the current time instead of a
marker a sharer may already have consumed.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: cover racing off-marker consumers settling on one generation

* filer: adopt the stored generation, not the consumed marker it names

A filer syncing with no off marker present read the value's first field —
the consumed marker's timestamp — so a restart after a lagging-clock off
period under-counted the generation and kept records the bump dropped.

* filer: re-check the off marker before consuming it

A sync whose marker read ran behind a later off period's consume cycle
still bumped for its stale read: the stored generation already covered a
different marker, so the bump overwrote the consumed name, counted the
generation past records a newer consumption meant to keep, and cleared
whatever marker was live by then. Each pass now re-reads the marker and
only bumps or clears while it is still the one the pass saw.

* filer: keep a failed generation read from resetting the records' generation

The final generation read ignored errors and then stored whatever gen
held — possibly zero — so a transient store failure could hide records
the current generation wrote. Unexpected read errors now return before
the local generation changes.

Also cover a sync whose marker read runs behind another consumer's
delete: it must adopt the stored generation, not bump for the stale read.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: fail a bump that cannot verify the stored generation

A generation read that failed mid-bump was treated as absent: the
write could land below a generation a racing bumper had stored, or
re-store a marker that bumper had already consumed. Read failures
now abort the pass like every other store error on the path.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: cover generation reads that fail instead of returning a value

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: carry the commit timestamp on the entry itself

The per-path MpTs record only exists because a commit timestamp had
nowhere else to live. Stamping commit_ts_ns on the entry before its
store write - and reusing that stamp as the emitted event's TsNs -
makes the entry its own commit record: replay and bootstrap traversal
order on it with zero extra writes for creates and updates.

Deletes still need a durable key since a removed entry cannot carry a
timestamp, so MpDl tombstones stay (one write per delete, growth
bounded by delete history). Entries predating the field fall back to
mtime, matching what peers and old records already assume.

This replaces the MpTs/MpOff/MpGen/MpRep machinery entirely: shared-
store filers write nothing extra, and dedicated-store replication
keeps last-writer-wins ordering plus tombstone coverage for deletes
and ancestor drops.

* filer: keep a directory update replay from tombstoning its subtree

A same-path update wrote a delete record before the delete-and-reinsert,
so a directory metadata change read as a subtree delete: a later peer
event for a child older than the update was then skipped even though
nobody deleted the children, leaving files missing on this filer. Only
file updates take the tombstone now — the re-inserted directory entry's
own commit stamp orders the path itself.

* filer: order unstamped entries by their recorded commit before mtime

Entries written before stamping existed have no CommitTsNs, so the
ordering check fell back to the client-settable mtime and dropped the
MpTs record those versions wrote: an entry committed at event time 300
with mtime 100 lost to a delayed peer event at 200 after upgrade. Read
the legacy record for unstamped entries in localCommitTsNs, CommitTsNs,
and the traverse stamp (which also stopped falling back to mtime,
regressing entry_ts_ns to 0).

* filer: release queued rename event stamps when the rename fails

Each moved entry carried an in-flight stamp its queued event would
adopt at emit time. When the move succeeded but a later child move or
the commit failed, the handler rolled back without emitting the queued
events or releasing their stamps, pinning the delivered and flushed
watermarks below the aborted rename forever. Release every queued
stamp on both failure exits in both rename handlers.

* filer: apply a directory update replay with an in-place upsert

Deleting the stored entry before inserting its replacement left a
window where the directory had no entry and no ordering record — a
tombstone cannot stand in for it without reading as a subtree delete
that suppresses the children's own records. A stale peer event accepted
in that gap could replay an older update or delete and sweep the
directory's children. Upsert the replacement instead: the entry keeps
its recorded commit order across the whole update, and a directory has
no chunks or hard links for a delete to release.

* filer: update an existing directory entry on replay instead of upserting

InsertEntry is not an upsert on SQL stores configured without upsert
support — PostgreSQL with enableUpsert=false issues a plain INSERT and
its duplicate-key error does not match the MySQL-only fallback, so a
replay updating a present directory failed on every retry. Pick
UpdateEntry when the entry exists and InsertEntry when it is absent;
UpdateEntry touches no row that is not there.

* filer: detect a shared-store delete racing a replayed directory update

UpdateEntry reports success even when the row was deleted between the
find and the update — the UPDATE changes zero rows and the replayed
change is dropped while the subscription offset advances. Re-read the
entry after the update and retry the write, re-checking the recorded
commit order each pass so a newer peer change wins instead of being
resurrected over.

* filer: declare commit_ts_ns in field-number order

Field 6 sits between extended and hard_link_id; the declaration now
reads in the same order. Wire format unchanged. Also syncs the Java
client's checked-in copy of filer.proto, which the pb Makefile copies
on gen.

* filer: option to skip delete tombstones entirely

filer.options.deleteTombstones=false stops writing MpDl records on
deletes - a single filer or shared-store cluster never replays peer
events, so the tombstones are dead weight. The tradeoff is explicit:
if a dedicated-store peer does replicate, a stale event can resurrect
a deleted path.

* filer: drop delete tombstones entirely

Deletes no longer write MpDl records, and the pending-repair machinery
they needed is gone with them: per-mutation metadata overhead is now
zero for every store topology. Ordering information lives only on the
entry itself via commit_ts_ns.

The accepted tradeoff: a dedicated-store peer replaying an event older
than a delete can resurrect the path, and a kept directory no longer
fences stale child creates. Recorded tombstones and MpTs records left
by older builds are still read, so upgrades keep the protections they
already have.

* filer: stop reading legacy per-path records too

Drops the MpTs/MpDl read path along with the writes: meta_path_ts.go
keeps only the entry-field readers (CommitTsNs, EntryCommitTsNs,
localCommitTsNs). Entries written before stamping existed fall back to
mtime, and tombstones or commit records older builds left behind no
longer fence replayed events.

* filer: trim commit timestamp comments

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-25 11:37:06 -07:00
Chris Lu 53834117a6 mini: lift the post-mint allocation hold on first inventory report (#12061)
* mini: lift the post-mint allocation hold on first inventory report

DeferAllocation armed a fixed warmup window whenever the leader minted a
fresh topology id, so the first write on a brand-new weed mini blocked
for the whole window even though no prior cluster state could exist.
The window still has to cover a rebuilt master's veteran servers, which
report on their first post-reconnect heartbeat.

Mini has exactly one possible reporter — the embedded volume server,
whose first heartbeat carries the full inventory — so hold allocation
only until that report is applied. The bound raise and topology-id
freeze checks run while the heartbeat is processed, so a wiped master
state still raises the bound or trips the freeze before the hold lifts.

Standalone weed master and weed server keep the fixed window: volume
servers outside the process can still be on their way.

* master: apply queued topology-id reports before opening the allocation gate

SetTopologyInitPending(false) cleared the pending flag before replaying
topology ids stashed while the local id was still being minted. A
NextVolumeId landing in the drain window saw no gate and no freeze yet.
The post-mint allocation hold used to cover the gap by delaying the
winner past the freeze; once the hold can release on first inventory,
the gate must stay closed until the queued checks have run.

* mini: gate the first-report release on single-master mode

releaseAllocOnFirstReport assumes the embedded volume server is the only
possible reporter of prior cluster state. A mini started with
-master.peers joins other masters, whose own volume servers can still be
on their way when the first report lands, so keep the regular
quiet-pulse hold there.

* master: check reported topology id on empty full reports

A HasNoVolumes heartbeat carries the same LastSeenTopologyId a populated
list does, but the mismatch freeze only ran for non-empty reports. An
EC-only veteran could therefore release the post-mint allocation hold
through NoteInventoryApplied while its stale topology id never reached
CheckReportedTopologyId.

* master: drain the topology-id stash inside the init-gate lock

CheckReportedTopologyId only held reportedTopologyIdsLock while stashing,
so a report in flight during SetTopologyInitPending(false) could land its
freeze after topologyInitPending cleared, letting a NextVolumeId slip
through the gap. The whole check now runs under the lock, and the gate is
released inside the same critical section as the drain.

* mini, master: pin and enforce the single volume-server reporter

isSingleMasterMode only checks master peers; SendHeartbeat still accepted
any volume server, so a late foreign report could carry prior cluster
state after the first-report release dropped the hold. Mini now pins the
embedded volume server's address and the master refuses registration from
any other reporter while ReleaseAllocOnFirstReport is set.

* master: admit the volume-server reporter before applying heartbeat state

* test: cover SendHeartbeat admission ordering

A refused reporter must not freeze growth or raise the id bound, and the
pinned reporter's stale topology id still must.

* mini: re-derive the single-reporter release on every run

The package-level master options survive in-process reruns, so a
multi-master mini following a single-master one must not inherit the
first-report release or a stale reporter pin.

* master: admit addressless first heartbeats under the reporter pin

An unregistered heartbeat with no Ip skipped the single-reporter
admission check but still reached the seen-bound raise and the
topology-id freeze before the delta-heartbeat skip. Admission now runs
for every unregistered heartbeat; an empty address can never equal the
pin.
2026-10-25 09:25:39 -07:00
Chris LuandDevin 88b91cccba command: no extra localhost listener when -ip.bind is specified (#12055)
* util: add listenOnLocalhost parameter to NewIpAndLocalListeners

Nil keeps the current behavior of also binding localhost when the host
is a specific address; all callers pass nil for now.

* command: add -ip.bind.localhost flag to opt out of the extra localhost listener

Binding a command to a specific -ip.bind address also opened a second
listener on localhost for every port, with no way to turn it off. Each
command that calls NewIpAndLocalListeners (master, master.follower,
filer, s3, iam, volume, sftp, mq.broker, mq.agent) now accepts
-ip.bind.localhost=false to bind only the requested address. Embedded
services get prefixed flags (s3.ip.bind.localhost, sftp.ip.bind.localhost,
iam.ip.bind.localhost), and the command-level flag also disables them so
a single switch turns off every extra listener the command opens.

* mini: only require all-interface port availability while the loopback listener is on

With -ip.bind.localhost=false each service binds just -ip.bind, so a port
held on another address no longer collides; the all-interface precheck
would reject or relocate usable ports. WebDAV, the Admin UI, and the
Admin worker gRPC (which binds -admin.worker.ip, not -ip.bind) never open
a loopback companion listener, so their checks now use the address they
actually bind.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* mq.broker: bind the gRPC listener to -ip.bind instead of all interfaces

The listener used an empty host so -ip was ignored for binding and the
localhost companion was never created, leaving -ip.bind.localhost with
nothing to control. Give the broker the same -ip/-ip.bind pair the other
servers use; embedded mode binds to -ip.bind like its peers.

* server: add -sftp.ip.bind.localhost and -iam.ip.bind.localhost switches

weed filer already exposes the prefixed opt-outs for its embedded sftp
and iam; give weed server the same pair so each extra listener can be
disabled individually while -ip.bind.localhost=false still turns them
all off.

* mq.broker: keep the default all-interface bind unless localhost is off

An unset -ip.bind used to mean the standalone broker listened on every
interface; defaulting it to -ip silently cut off multi-homed clients. Only
fall back to -ip when -ip.bind.localhost=false, where a specific bind is
needed for the opt-out to take effect.

* mini: validate the Admin HTTP port on the address it will bind

The port check always used -ip.bind while startMiniAdminWithWorker falls
back to loopback when Admin has no credential, so a loopback-held port
passed validation and killed the admin at startup.

* mq.broker: keep the all-interface default bind when -ip.bind is unset

-ip only becomes the fallback once -ip.bind.localhost=false needs a
specific address for the listener.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* mini: validate the Admin HTTP port on its effective bind IP

When the Admin listener falls back to loopback (no credential or mTLS
configured), the pre-flight port check now probes that same address
instead of -ip.bind.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* util: keep NewIpAndLocalListeners source-compatible for 3-arg callers

* command: drop the -ip.bind.localhost flag; an explicit -ip.bind drops the extra listener

Instead of an opt-out flag, the loopback companion listener now simply
follows whether the bind address was chosen: it is created only while
-ip.bind is unset and the bind address falls back to -ip. Passing
-ip.bind=<addr> binds each port to exactly that address.

Each service options struct carries a listenOnLocalhost *bool that
embedders (weed server, weed mini, embedded s3/sftp/iam in weed filer)
set from the parent's -ip.bind state, captured before the bind flag is
repointed or filled in by the -ip fallback; standalone commands derive
it from their own flag before the fallback runs.

iam and mq.agent gain a real -ip.bind flag so they have a way to pin a
single address. mq.broker keeps its new -ip.bind flag but drops the
-ip.bind.localhost fallback hack: empty still listens on all interfaces.

* server: default embedded mq.agent brokers to the broker's bind address

With an explicit -ip.bind the embedded broker no longer listens on
localhost, so an unset -mq.agent.brokers defaulting to localhost:17777
could not reach it. Default to <serverBindIp>:<mq.broker.port> (the
address the embedded broker binds), falling back to localhost when the
bind address is a wildcard. An explicit -mq.agent.brokers still wins.

Also fix the NewIpAndLocalListeners skip message to say localhost.

* server: bind embedded mq.agent to serverBindIp

The embedded agent's ipBind was never set, so startQueueAgent fell back
to the advertised -ip address and an explicit -ip.bind did not pin the
agent listener.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-25 06:52:16 -07:00
fefac028ec feat: send EC shards in parallel per destination disk (#12050)
* feat: send EC shards in parallel per destination disk

One stream per destination disk. max_shard_transfers unset means no global cap, 0 sends shards sequentially, and N is a global cap that still keeps one stream per disk. The value is persisted through TaskPolicy. The maintenance form is a checkbox plus Max transfers because unset and 0 differ. The key is intentionally omitted from GetConfigSpec.

* ec shard distribution: treat negative max_shard_transfers as sequential

The config layer (normalizedMaxShardTransfers) clamps negative values to 0,
which selects sequential distribution. shardTransferCap diverged: a negative
cap that reached ShardDistributionOptions directly would have selected
uncapped parallel mode, the most aggressive setting. Align the options layer
with the same normalization so invalid input degrades to the conservative
mode.

* ec: serialize shard transfers to a server when a target leaves disk choice to the receiver

disk_id 0 is auto-placement on the receiving volume server, so an
auto-placed stream can land on a disk another queued stream is pinned
to. Grouping by (destNode, disk_id) alone let both run at once; collapse
all of that server's transfers under the destNode key instead.

* worker: clamp max_shard_transfers to the int32 wire range

ReadIntConfig returned the 0 fallback for values above MaxInt32, turning
a large positive cap into sequential mode. Read int64 and clamp into
range instead.

* worker: drop unused shard-transfer form helper

The maintenance form renders and collects max_shard_transfers in
plugin.templ JavaScript; the Go copy had no callers.

* worker: bound max_shard_transfers conversion under an in-range guard

---------

Co-authored-by: Chepurnov Simon <195125934+CHEPURNOV-S-S@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-10-25 06:37:26 -07:00
Jay HemnaniandChris Lu 0c9872f842 filer, volume: ignore Range when If-Range no longer matches (#12052)
* filer, volume: ignore Range when If-Range no longer matches

A GET with Range and a stale If-Range validator got 206 with the requested
bytes of the current file. RFC 9110 13.1.5: send the whole content instead,
so a resumed download restarts rather than mixing two versions.

* volume(rs): drop Range when If-Range no longer matches

Mirror the Go fix for the Rust volume server: evaluate If-Range against the
response ETag (strong comparison) or Last-Modified once the needle's
validators are known, then drop the range on a stale validator so the whole
current representation is sent. A dropped range on the RangeFromSource
strategy falls back to a full stream of the data-file region; buffered and
chunk-manifest paths already take Option<&str>.

* volume(rs): fail If-Range when the header value is not readable

* volume(rs): accept all HTTP-date formats in If-Range like Go

* server: cover alternate HTTP-date formats in If-Range test

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-10-25 06:36:40 -07:00
Chris LuandDevin da1d7444b8 master: shorten the first-write hold after a fresh topology id (#12059)
* master: release the post-mint allocation hold after a quiet pulse

DeferAllocation armed a fixed warmup window whenever the leader minted a
fresh topology id, so the first write on a brand-new cluster blocked for
the whole window even though no prior state could exist. The window
still has to cover a rebuilt master's veteran servers, which report on
their first post-reconnect heartbeat.

Track the last heartbeat that could carry prior cluster state — a server
joining, held volumes or shards, a nonzero id bound, or a foreign
topology id — and lift the hold one pulse after it goes quiet, capped by
the original window.

* master: make the post-mint allocation hold configurable

master.volume_growth.allocation_deferral overrides the warmup window a
leader holds volume id allocation for after minting a fresh topology
id. Unset keeps the existing window; "0s" disables the hold for setups
that know the cluster is fresh (test harnesses, single-node labs); any
other duration replaces the cap while the quiet-pulse early release
still applies.

* master: re-check the allocation deadline while it is held

NextVolumeId slept for the whole computed delay, so a heartbeat landing
mid-sleep that pushed the quiet deadline out was ignored — the pending
allocation could still run ahead of a straggler's report. Wait in short
slices, re-evaluating the deadline and the growth freeze each round.

* master: count only ids above the issued bound as prior-state evidence

Heartbeat echoes of volumes and shards this master allocated during the
hold refreshed the quiet window, so each growth allocation re-extended
the deferral until the hard cap. Evidence now requires an id beyond the
bound this leader has issued, a higher reported bound, or a foreign
topology id; a joining server (dn == nil) still counts once.

* master: reject an unparseable allocation_deferral instead of disabling the hold

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* master: do not count a skipped heartbeat as a joining server

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test: assert the allocation hold extends past the original quiet deadline

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* master: release the post-mint hold after two quiet pulses

One pulse proved short for a veteran that reconnects on a staggered
restart: with only five seconds of quiet, allocation could resume and
reuse ids before a slower server landed its first heartbeat. Two pulses
still halves the fresh-cluster first-write stall versus the full warmup
cap, and known-fresh setups keep the "0s" override.

* master: pick once more when assign times out as growth concludes

The pick loop's final attempt can predate the growth it triggered
concluding: growth registers its volumes in the layout before clearing
the pending flag, so a timeout landing on that boundary used to shed a
hard 406 over volumes that had just arrived. Pick once more after the
loop in both the http and grpc assign paths before deciding how to
shed.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-25 06:17:42 -07:00
Chris LuandDevin d15396fc4d s3tables: scope the anonymous identity's grant on the table catalogs (#12056)
* s3tables: scope the resolved anonymous identity instead of defaulting to allow

Catalog authorization fell back to the open default whenever the request
resolved to the configured anonymous identity, ignoring its grant. The
fallback now applies only when no principal was resolved (zero-config),
so anonymous requests are checked against the identity's configured
actions like any other principal. The S3 Tables API and the Iceberg and
Lance catalogs share this handler, so all three are covered.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test: authenticate the catalog clients in the s3tables suites

Anonymous calls into the Lance and Iceberg catalogs are now scoped by
the anonymous identity's grant, so the suites that used to rely on the
anonymous pass authenticate like their real deployments do: x-api-key
for the Lance probes and REST drivers, client-credentials OAuth for
pyiceberg.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* worker(lance): authenticate namespace calls with the storage credential

Anonymous catalog calls are now scoped by the anonymous grant, so a
worker that holds an S3 credential sends it as x-api-key on the
namespace requests instead of relying on the anonymous pass.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test: authenticate the pushdown catalog connection in catalog_lancedb

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* lance: follow catalog redirects only within the same host

reqwest forwards custom headers on a follow, so a redirect to another
host would carry the x-api-key credential to a server it was never
meant for. Keep same-host hops for proxies that normalize the path or
upgrade the scheme.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* lance: warn when the catalog api key travels in cleartext

The x-api-key pair is a real storage credential: over plain http to a
non-loopback host the network can read it. Warn at client build so the
operator sees it; https and loopback stay quiet, and the key still
sends since an http catalog is a valid deployment.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* lance: require the same origin on catalog redirects

A same-host hop can still downgrade https to http and put the api key on
the wire; matching scheme, host, and port keeps the credential on the
listener it was issued for.

* lance: bound catalog redirect hops so loops fail fast

The custom redirect policy drops reqwest's default 10-hop limit, so a
catalog that redirects back to itself would hold each call open until
the request timeout. Keep the bound alongside the origin check.

* lance: gofmt the redirect helpers

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-25 06:13:22 -07:00
Chris LuandDevin 39bdbc18db admin: keep read-only sessions out of the filer /etc tree (#12057)
* admin: keep read-only sessions out of the filer /etc tree

The file browser read routes are open to any authenticated session, so a
read-only login could download or view arbitrary filer paths. The filer
stores the credential and cluster configuration under /etc (identity JSON
with plaintext S3 secret keys, SSE-S3 keys, remote storage config), which
let a view-only session recover live secrets that the user API redacts.

Gate the read paths on the session role: read-only sessions get 403 for
/api/files/download, /view, /properties, /metadata on /etc paths, the
browser page redirects away, and recursive metadata exports skip the /etc
subtree for read-only sessions.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* admin: only reject real .. path components in file browser paths

path.Clean already resolves interior parent references, so any '..'
surviving in the cleaned path can only be a leading escape from a
relative input or part of a valid name. Matching on the substring
rejected legitimate entries like /buckets/reports/report..csv.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* admin: cover the read-only /etc export exclusion with a test

Pull the excluded-prefix selection into exportExcludedPrefixes so the
role-dependent /etc skip is exercised directly — the cleanReadPath
tests alone would still pass if the exclusion were dropped.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-25 06:06:21 -07:00
Chris Lu 403d51631b s3api: stop echoing stored tags and internal attributes on GET/HEAD (#12053)
* s3api: stop echoing stored tags and internal attributes on GET/HEAD

setResponseHeaders copied every entry.Extended key into the response,
only filtering xattr- and x-seaweedfs-. Stored object tags
(X-Amz-Tagging-*) and the gateway's own attributes (Seaweed-X-Amz-Acl,
-Owner, -Version-Id, -Object-Lock-Mode, ...) therefore leaked to any
caller holding s3:GetObject, and to anonymous readers on public-read
objects, without s3:GetObjectTagging or READ_ACP.

Tag count, user metadata, storage class and the real S3 headers
(x-amz-version-id, x-amz-object-lock-*) are unaffected.

* s3api: match stored tag and internal attribute keys case-insensitively
2026-10-25 06:05:06 -07:00
Chris Lu 52d91008ac ci: bump hosted runners to ubuntu-24.04
GitHub is retiring ubuntu-22.04; move all 94 runs-on pins across 40 workflows to ubuntu-24.04. rust_binaries_release: the arm64 cross step was jammy-era — noble keeps apt sources in deb822 ubuntu.sources (the sources.list edit was a silent no-op that left amd64 repos open to arm64 and 404'd) and ports suites must say noble. Pin deb822 stanzas to amd64 and add noble ports entries.
2026-10-24 23:07:28 -07:00
github-actions[bot] ec8f979350 4.50 2026-10-25 03:09:42 +00:00
Chris LuandDevin 1a0485a43d test: bound trino container memory + dump logs when it exits before ready (#12044)
* test: dump trino container state and logs when it exits before ready

When the Trino container dies during startup, waitForTrino only
reported the docker exec error ("container is not running"), which
gives no hint whether the JVM was OOM-killed, the image broke, or the
config is bad. Capture docker inspect exit state and the container
logs in the failure message so runner-side flakes are diagnosable
from the test output alone.

* test: cap trino container memory at 4g in catalog tests

Trino's jvm.config sizes the heap at 80% of host RAM. On a loaded
self-hosted runner the JVM attempts to claim most of system memory
at boot and the container is killed ~15s after docker run — the
flake seen in CI today. Bounding the container to 4g gives the JVM
a fixed ~3.2g ceiling (observed usage is ~750mb), immune to host
memory pressure.

* test: keep docker errors in trino container diagnostics

Report docker inspect/logs failures instead of dropping them, and
include the last docker exec output in the container-dead fatal so a
missing container still explains what happened.

* test: detect trino container exit in readiness fallback

When the polling loop times out while the container still appears
alive, the trino --version fallback may itself fail because the
container has since exited. Re-check its output for the
exited-container signature and route through the diagnostic dump.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test: dump trino container diagnostics on readiness timeout too

The alive-but-never-ready timeout path reported only the last docker
exec error, losing the container state and logs that the dead-container
path already captures.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 20:00:06 -07:00
Chris LuandDevin 8c1a0a0cc2 filer: fix prefixed listing when the resume point sorts below the prefix (#12049)
* test: prefixed directory listing with start below the prefix

A listing constrained to prefix "item-b" and resumed from "item" must
return item-b.versions. The seek-based stores return an empty page
because the iterator lands before the prefix range and breaks on the
first non-matching name.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: clamp prefixed-listing seek to the prefix range

Seek-based stores (leveldb, leveldb2, leveldb3, rocksdb, tikv, etcd)
started the scan at startFileName and stopped at the first key outside
the prefix. A startFileName below the prefix landed the iterator on a
non-matching sibling first, so the listing returned an empty page even
though matching entries followed. etcd had the mirror problem: a range
beginning below the prefix end leaked non-matching entries. Begin the
seek at max(startFileName, prefix), which is what foundationdb and the
sql stores already do.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer(hbase): honor startFileName lower bound in prefixed listings

The scan starts at the prefix but only skipped the exact startFileName,
so entries sorting before it were returned again on resume.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: keep paging a prefixed listing while matches can still follow

The fallback for stores without native prefix support stopped paging as
soon as the last scanned name reached the prefix, dropping prefix
matches that continue on the next page.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: return the last delivered name from the prefixed-listing fallback

The fallback for stores without native prefix support pages raw entries
and filters client-side. Keep the scan cursor and the returned cursor
separate: resuming must continue after the last delivered match, not the
last scanned name, or matches still inside the current raw page are
skipped on the next page. Also keep paging while the scan cursor is
inside or below the prefix range instead of stopping at the first name
that reaches the prefix.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer(hbase): start prefixed-listing scans at the resume key

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 19:59:19 -07:00
github-actions[bot] 0c32eab48e docs: regenerate star history chart 2026-10-25 00:48:29 +00:00
Chris Lu 85470416cb volume: overlap the local needle write with the replica fan-out (#10074)
* volume: overlap the local needle write with the replica fan-out

ReplicatedWrite wrote the needle to local disk and then fanned it out to
the replicas serially, so a replicated write cost local + replicas. The
needle is already fully buffered, so the two can run at once. Snapshot
the fields the fan-out reads first (the local write stamps the volume
TTL and updates needle flags, so the replica goroutine must not touch
the shared needle), then run the local write and the replica fan-out
concurrently and ack only after both. Where local disk and replica
network are separate resources the write latency drops to max(local,
replicas) instead of their sum; co-located on one disk the two writes
contend, so this helps distributed placements, not single-host tests.

* volume: return the local isUnchanged from the write goroutine

* volume: trim overlap-write comments

* volume-rs: overlap the replica fan-out with the local needle write

* volume: test the overlapped local write and replica fan-out

* volume: decode replica pairs only when replicas exist

A single-copy or incoming replicate write never uses the decoded pair map, so a malformed Pairs blob only burned an error counter and a log line. Gate the decode on replicaCount.

* volume: release the stalled replica on test failure paths

An early t.Fatal left the httptest handler blocked on releaseReplica, so the deferred Close hung instead of reporting the failure. Release idempotently ahead of the server close.

* test: mismatched If-None-Match serves the object over If-Modified-Since

The precedence case asserted the old If-Modified-Since-first ordering and
expected 304. The volume servers now evaluate If-None-Match first, so a
non-matching tag must return 200 with the body even when If-Modified-Since
would allow 304 (RFC 9110 13.1.3).
2026-10-24 15:16:16 -07:00
Chris LuandDevin e442eda88e s3: authorize POST policy uploads through the shared object check (#12047)
* s3: authorize POST policy uploads through the shared object check

doesPolicySignatureMatch gated a verified POST Object signature on the
identity's legacy Write action, which cannot represent attached IAM
policies, STS session policies, or key-scoped grants. Users authorized
only by an attached policy (e.g. s3:PutObject on bucket/*) were denied
with AccessDenied even though the equivalent PUT succeeds.

The handler already runs AuthorizeObjectWrite against the real object
key after the form's x-amz-* fields are promoted to headers, so the
coarse check only shadowed the full evaluation. Signature verification
now just returns the identity; authorization stays in one place.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test: POST Object authorizes attached-policy identities, still denies others

Cover the signature check and AuthorizeObjectWrite for an identity whose
only write grant comes from an attached IAM policy, plus the deny leg
for an identity whose attached policy omits s3:PutObject.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 15:09:34 -07:00
Chris Lu 216c1032e6 test: mismatched If-None-Match serves the object over If-Modified-Since (#12045)
The precedence case asserted the old If-Modified-Since-first ordering and
expected 304. The volume servers now evaluate If-None-Match first, so a
non-matching tag must return 200 with the body even when If-Modified-Since
would allow 304 (RFC 9110 13.1.3).
2026-10-24 15:06:19 -07:00
03f773f087 redis2: pipeline entry reads when listing a directory (#12042)
ListDirectoryEntries fetched every member with its own GET, so a page of
1024 entries cost 1024 round trips. Fetch the values in pipelined batches
of 128 as the loop reaches them and walk the replies in order. Each reply
keeps its own error, so missing and expired members are cleaned up and
skipped exactly as before, and an unreadable member still ends the
listing at that name. Batching keeps a caller that lists with a huge
limit and stops after a few entries from reading the whole directory.

Co-authored-by: Patrick Schmidt <1310072+Woellchen@users.noreply.github.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
2026-10-24 15:04:37 -07:00
Junker der Provinz f9dac64374 helm: remove filer.redirectOnRead, which weed filer does not accept (#12040)
weed filer lost -redirectOnRead in 9b3109a5d (1.59), which also removed
the chart value, but 2bed15f1d added it back. Setting it makes the filer
exit with "flag provided but not defined".
2026-10-24 10:45:57 -07:00
7b43dec665 shell: run volume.configure.replication in parallel per volume server (#12041)
* shell: run volume.configure.replication in parallel per volume server

Volume servers were configured one after another, and each server's
result overwrote the previous error, so a failure on one server was
lost when a later server succeeded. Run one job per volume server,
bounded by -maxParallelization like the other shell commands, and
return the errors from all servers.

VolumeConfigure unmounts and remounts the volume, so replicas of the
same volume on different servers take turns instead of all going
offline at once.

Co-authored-by: NyaMisty <5344431+NyaMisty@users.noreply.github.com>

* shell: test that volume.configure.replication overlaps servers up to the limit

* shell: assert every server ran in the parallel configure-replication test

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: NyaMisty <5344431+NyaMisty@users.noreply.github.com>
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 10:29:38 -07:00
Chris LuandDevin 95d8a709e8 ci: run s3-go-tests on remote_storage changes (#12043)
The S3 Remote Cache tests exercise the remote-storage S3 client, but the
path filter did not include weed/remote_storage, so a tagging change
there can regress the suite without running it on the PR.

Generated with [Devin](https://devin.ai)

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 09:42:17 -07:00
Chris LuandDevin e432b75822 remote_storage/s3: send only real object tags to the remote (#12039)
* remote_storage/s3: send only real object tags to the remote

WriteFile and toTagging copied every entry.Extended attribute into the
remote object's tag set, including Seaweed-internal keys such as
Seaweed-X-Amz-Acl whose values are not legal S3 tag values. Remotes
that validate x-amz-tagging now reject these uploads with InvalidTag.

Only X-Amz-Tagging-* attributes are S3 tags: filter to that prefix,
strip it, and encode through s3TaggingHeader so keys and values are
properly escaped.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* remote_storage/s3: skip remote tagging call when the tag set is unchanged

UpdateFileMetadata fired PutObjectTagging on any Extended diff, but most
attribute changes are not tag changes. Compare the effective tag sets and
return early when they match.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* remote_storage/s3: assert WriteFile errors in tag-header tests

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 09:41:01 -07:00
81c90cb446 filer.backup: retry a failed checkpoint read instead of replaying from the epoch (#12036)
* filer.backup: retry a failed checkpoint read instead of replaying from the epoch

Fixes #12035

* test: reset sink.local.directory in backup offset test cleanup

The new test sets sink.local.directory on the shared viper but only
reset the enabled flag, leaking a key that points at a deleted
t.TempDir() to later tests in the package.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test: cover the retried attempt re-reading the checkpoint

The fake filer now fails only its first KvGet, and the test runs a
second doFilerBackup attempt asserting the checkpoint is re-read and
the attempt proceeds past the offset stage. LookupDirectoryEntry is
stubbed so the SSE-S3 KEK probe returns a miss instead of retrying
10x2s.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 08:22:29 -07:00
Chris LuandDevin 4c2c0ff21c s3: fix wildcard matcher skipping '*' targets in policy patterns (#12034)
* s3: check pattern '*' before literal equality in matchWildcardString

When the target string carried a '*' at the offset of the pattern's '*',
the literal-equality case consumed it as a one-character match and no
backtrack point was recorded, so the comparison returned false. A Deny
Resource carrying a policy variable, or an Allow carrying NotResource,
could then be sidestepped by an object key containing '*' at the aligned
offset.

* test: cover literal asterisks in wildcard match targets

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 08:20:22 -07:00
Chris Lu 1d2d9dfed2 test(fuse): raise mini/filer and mount readiness waits to 90s (#12037)
Same startup-timing flake class as the Makefile waits: volume/filer
readiness at 30s is marginal on loaded runners — observed failing on a
github-hosted runner too (TestPOSIXCompliance +
TestExternalPOSIXSuites). The process-death early return still reports
real startup failures quickly.
2026-10-24 08:13:36 -07:00
c4d743a69f s3: make ListObjectsV2 continuation tokens round-trip XML-unsafe keys (#11996)
* s3: keep ListObjectsV2 continuation tokens round-tripping XML-unsafe keys

A page ending on a key XML 1.0 cannot carry (for example one containing
\x01) produced a NextContinuationToken from that raw key. The XML
encoder replaced the control characters with U+FFFD, so a client resuming
at the token restarted after the mangled name and skipped every key
sorting between it and the real one. encoding-type=url encoded the keys
but not the token, so it hit the same skip.

Encode the token as base64url behind a prefix when the key is not valid
XML text; keys that survive XML unchanged keep the bare key as the token,
so their tokens are byte-identical to before. The handler decodes the
prefix back to the raw key before listing.

Fixes #11990

* s3: keep undecodable base64: list tokens resuming at the raw key

* s3: round-trip ListObjectsV1 markers over XML-unsafe keys

* s3: round-trip ListObjectVersions key markers over XML-unsafe keys

* s3: round-trip ListMultipartUploads key markers over XML-unsafe keys

* s3: decode only tokens whose key could not have stayed bare

* s3: document and pin the encoded-wins rule for ambiguous list tokens

A token like base64:AQ reads as both a legacy bare key and the encoded form of an XML-unsafe key. The encoded reading already won in code; state why and cover it so the choice is not reverted as a false simplification.

* s3: keep request-marker echoes in the response's wire encoding

The decoded marker/key-marker is a raw key; echoing it writes XML-unsafe
bytes straight into the response, where the encoder turns them into
U+FFFD. Re-encode each echo through encodeListContinuationToken so the
client reads back the token it sent, and apply urlPathEscape for V1 when
encoding-type=url is active, matching the listed keys.

* s3: wire-encode the V1 NextMarker echo like the request marker

* s3: pin resuming an XML-unsafe key past a sibling directory

* s3: wire-encode ListMultipartUploads markers and keys under encoding-type=url

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: unescape url-encoded marker echoes on list resume

Under encoding-type=url the V1 and ListMultipartUploads responses emit
marker echoes through urlPathEscape, and clients pass them back verbatim.
The incoming marker was used as the resume position without undoing that
escaping, so resuming after a url-unsafe-but-XML-safe key like "a b"
(marker "a%20b") restarted the listing at the literal escaped text.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: assert the full decoded V1 sequence in the marker round-trip test

The test only checked the page count, so a regression that skipped one
key while repeating another still passed. Requesting encoding-type=url
keeps the XML-unsafe keys legible for the decoded comparison and
exercises resuming with the escaped marker echo.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: only unescape markers in the canonical urlPathEscape form

A url-encoded request marker like a%41 cannot be encoder output, since
urlPathEscape never emits %41 for 'A'. Unescaping it anyway resumed at
aA and skipped every key sorting between a%41 and aA, including a%42.
Require the unescape to re-escape to the received marker so only
verbatim echoes decode; anything else stays the raw key the client sent.

* s3: pin a raw-key marker that looks like an escape sequence

* s3: cover the marker-echo decode guard with unit cases

* s3: wire-encode the ListObjectsV2 StartAfter echo under encoding-type=url

AWS encodes Delimiter, Prefix, Key, and StartAfter in V2 responses when
url encoding is requested, but StartAfter was echoed raw. The echo now
takes urlPathEscape and the incoming start-after param runs through
decodeMarkerEcho, so a client resuming from the emitted form lands on
the raw key.

* Merge master and reconcile ListMultipartUploads marker encoding

Resolve the overlap with the merged encoding-type=url pass: keys and
markers now go through the single post-pass so nothing is url-escaped
twice. Markers are still token-encoded when the response is not
url-encoded, and the key-marker decode checks the query-escaped wire
form the endpoint actually emits.

* s3: echo the decoded start-after under encoding-type=url

The escaped echo of the raw input is not a fixpoint: a client resubmitting
the echoed StartAfter decoded it to the literal escape text and resumed at
the wrong position.

* s3: keep url-mode multipart key-marker decodes off the token path

Under encoding-type=url the emitted KeyMarker/NextKeyMarker are the
url-escaped raw key, never a base64-prefixed token, so the request decode
must unescape only. Running the token decode afterward misread a literal
"base64:..." key as an encoded token and resumed at the decoded bytes,
skipping the keys in between.

---------

Co-authored-by: Chris Lu <chris.lu@gmail.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 07:18:10 -07:00
fcd28031e7 mq/kafka: wait for a filer to register instead of panicking on an empty list (#12019)
* mq/kafka: wait for a filer to register instead of panicking on an empty list

NewSeaweedMQBrokerHandler queried the masters for filers once, ~2 s after
start, and handed an empty list to NewFilerClientAccessor, which panics. A
fresh filer registers only after its empty-membership confirm window
(10 s), so a gateway started with a fresh cluster crashed (Kafka Quick
Test, kafka-gateway exit 2). Retry while the list is empty for up to 60 s,
then return an error.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* mq/kafka: wait out a transient-empty broker listing too

A fresh SMQ broker registers with the masters on the same announce
delay as a filer, so the one-shot broker lookup could fail startup the
same way. Generalize waitForFilers into waitForClusterNodes and bound
both discoveries by the same window; the empty-broker check becomes
unreachable and drops away.

* mq/kafka: wait out the broker health check like node discovery

A broker announces to the masters before it has acquired its balancer
lock, so the gateway's three-attempt probe could still fail fast on a
fresh cluster. Retry FindBrokerLeader for the same window the caller
waits out empty listings.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 07:09:43 -07:00
Chris LuandDevin 34c7bd03ad ci: clean up stale FUSE mounts before running tests (#12020)
* ci(s3tables): tolerate stale FUSE mounts in disk diagnostics

* ci(s3tables): bound df diagnostics with timeout, explain once

A stale FUSE mount can also hang statfs, not just fail it, so wrap df in
timeout 15; one comment at the first site covers all sixteen.

* ci: tolerate stale FUSE mounts in df diagnostics elsewhere

The same self-hosted runners serve the other workflow files; a bare
df -h there can stall or fail a job the same way.

* ci: unmount disconnected FUSE test mounts before jobs start

* ci: skip FUSE cleanup when the variant checkout is skipped

* ci: reclaim the crashed run's test dir once its stale FUSE mount is detached

* ci: stop tracking python bytecode for the FUSE cleanup action

The helper's __pycache__ slipped into the tree; ignore it so local
test runs cannot re-add it.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: run the FUSE cleanup action's unit tests on action changes

test_cleanup.py was never wired into CI, so a regression in cleanup.py
would ship silently. A path-filtered workflow runs it on every change
under .github/actions/.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: match every FUSE test mount layout in the cleanup action

The candidates list only matched seaweedfs_fuse_test_*/mount, so stale
mounts left by fuse_failover and fuse_p2p runs (mountN leaves under
seaweedfs_fuse_* dirs) and the posix-compliance workflow's fixed
/tmp/seaweedfs/mount were never reclaimed and could still wedge df.

* ci: run the FUSE mount cleanup in the fuse_failover and fuse_p2p workflows

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 07:07:55 -07:00
18f9d9a0b6 feat: Free Volume Deadband (M) for EC shard placement (#12029)
* feat: Free Volume Deadband (M) for EC Place nodes and racks

|ΔfreeSlots| ≤ M is a tie; remainder is liveShards. M=0 stays strict
freeSlots. M is free volume slots (× dataShards). pickNodeInRackEligible
and pickRackEligible share the rule. Worker plugin config
free_slots_deadband, default 0. Not on the write path.

* fix: anchor EC freeSlots deadband on class maxFree

Pairwise |Δfree|≤M vs the current winner is not transitive: 120→110→100
at M=10 slots picked 100. After evenness, rank by maxFree, keep
|free-maxFree|≤M, then min liveShards. Shared by node and rack. M=0 is
still only maxFree then live.

* worker: add free_slots_deadband to ErasureCodingTaskConfig

* erasure_coding: persist free_slots_deadband through TaskPolicy

ApplyMaintenanceConfigFromToml accepted the key but the admin save path
round-trips config through ToTaskPolicy, so the value was dropped on
reload; serialize it like the other EC settings.

* erasure_coding: bound free_slots_deadband at its declared maximum

The TOML loader and the plugin worker config only clamped negatives, so
a value above the spec's 100000 could reach the int32 TaskPolicy field
and wrap; cap at 100000 on both entry paths and declare MaxValue on the
plugin field.

* erasure_coding: document free_slots_deadband in scaffold admin.toml

---------

Co-authored-by: Chepurnov Simon <195125934+CHEPURNOV-S-S@users.noreply.github.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
2026-10-24 07:06:46 -07:00
52e05ccda4 s3: accept a self-copy that changes the storage class (#12031)
* s3: accept a self-copy that changes the storage class or encryption

AWS changes an object's storage class or encryption by copying the
object onto itself with the new x-amz-storage-class or SSE headers, and
only rejects a self-copy that changes none of metadata, storage class,
website redirect location or encryption. CopyObject rejected every
self-copy without REPLACE metadata or tagging directives as
InvalidRequest, so these requests failed.

* s3: reject an invalid x-amz-storage-class on CopyObject

A self-copy with a bad storage class used to fail as InvalidRequest;
once such copies are admitted the value must be validated like PutObject
does, or it would be stored as opaque metadata.

* test: cover the invalid storage class rejection on self-copy

* s3: only a storage class change admits a self-copy

Admitting encryption changes on a self-copy is unsafe: the metadata-only
path compares only KMS key IDs, so a new encryption context or bucket-key
setting is advertised over the old data key, and SSE-KMS key rotation
onto the same entry keeps the old data key.

Count only a requested storage class that differs from the object's
(STANDARD when none is stored) as a change, so a self-copy to the
current class or with only encryption headers is rejected as before.
Such a self-copy updates the object in place and has to keep its
encryption; one that would re-encrypt it or uses SSE-KMS is rejected
too.

---------

Co-authored-by: Wleeaf <179125857+wleeaf@users.noreply.github.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
2026-10-24 07:06:20 -07:00
Chris LuandDevin 514c0c1e99 topology: stop a stale duplicate-vid report from re-quarantining the survivor (#12032)
* topology: drop a duplicate-vid report whose own copy is already gone

flagDuplicateVolumeId runs on a goroutine after registration, so by the
time it executes the reporter's copy may have been unregistered (a
finished move drops the losing copy). The surviving copies are then
legitimate, yet the stale report still quarantined them and stored a
phantom (node, vid) key for the departed reporter. Skip the marks when
the reporter no longer serves the volume under its own layout key.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(topology): cover a stale duplicate report against a lone survivor

The existing tests only flag the conflict before the losing copy is
unregistered. Run the delayed duplicate check after the reporter's copy
is already gone and assert nothing is quarantined and the survivor stays
writable, so removing the stale-report guard fails the test outright
instead of needing a lucky schedule.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(topology): cover the enforceQuarantine-driven release retry

A failed writable push re-arms the record, and the next volume report
drains that retry through enforceQuarantine itself - a different entry
point than the pulse-driven pass the other tests exercise.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 07:05:46 -07:00
8841eb6c70 volume: check If-None-Match before If-Modified-Since and match ETag lists (#12027)
* filer: match If-Match and If-None-Match against "*" and ETag lists

checkPreconditions compared the whole header as one ETag, so "*", a
list of ETags and a weak W/ tag in If-None-Match never matched
(RFC 9110 13.1.1, 13.1.2).

* volume: check If-None-Match before If-Modified-Since and match ETag lists

A volume GET answered 304 from If-Modified-Since even when If-None-Match
named a different ETag, and compared If-None-Match as one exact string,
so "*", lists and weak tags never matched (RFC 9110 13.1.2, 13.2.2).

* filer: split etag lists only outside quoted tags

A comma inside a quoted opaque tag is a legal etagc, not a list
separator; strings.Split cut such tags into fragments that
CanonicalizeETag could turn into a false match.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: match repeated If-Match and If-None-Match header lines

Header.Get reads only the first field line, so a match on a later
repeated header was missed; join all field values like any other
comma-separated list.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: match repeated If-None-Match header lines

Header.Get reads only the first field line; join all field values so a
match on a later repeated header is seen. Also covers a comma inside a
quoted opaque tag.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: send the etag on a 304 like the filer does

RFC 9110 15.4.5 lists ETag among the fields a 304 must carry when a 200
would send it; SetEtag now runs before the notModified check.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: mirror the conditional-GET fixes in the rust server

not_modified_response now checks If-None-Match before If-Modified-Since,
matches "*", weak tags, quoted commas and repeated field lines through
the same list rules, and sends the etag on a 304.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: treat an empty If-None-Match like the Go server does

* filer: ignore empty elements in etag precondition lists

* volume: ignore empty elements in etag lists like the filer

* filer: let a present-but-empty precondition line suppress the date check

* volume: gate If-Modified-Since on a present If-None-Match line

* volume: gate If-Modified-Since on header presence like the Go server

---------

Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: chrislusf <chris.lu@gmail.com>
2026-10-24 07:05:29 -07:00
Chris LuandDevin 59cccc788c master: read the writable-volumes cap from master.toml (#12009)
* master: read the writable-volumes cap from master.toml

-maxWritableVolumes becomes the master.max_writable_volumes config key,
so the new master limit is set the same way as other master options
instead of a flag wired into every command that embeds a master.

* master: name the writable cap master.max_writable_volumes_per_layout

The cap applies per volume layout (collection + replica placement + ttl +
disk type), so the key names its scope instead of reading as a global cap.

* master: keep an embedder's MaxWritableVolumes when the key is unset

NewMasterServer overwrote option.MaxWritableVolumes with the viper
value, which reads as 0 whenever master.toml omits the key, silently
dropping a cap a Go caller supplied through MasterOption. Seed the
default from the option instead so the field survives an absent key;
an explicit config entry still wins.

* master: drop stray nested-checkout gitlinks

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 07:04:45 -07:00
9e7fd2eb90 filer: match If-Match and If-None-Match against "*" and ETag lists (#12026)
* filer: match If-Match and If-None-Match against "*" and ETag lists

checkPreconditions compared the whole header as one ETag, so "*", a
list of ETags and a weak W/ tag in If-None-Match never matched
(RFC 9110 13.1.1, 13.1.2).

* filer: split etag lists only outside quoted tags

A comma inside a quoted opaque tag is a legal etagc, not a list
separator; strings.Split cut such tags into fragments that
CanonicalizeETag could turn into a false match.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: match repeated If-Match and If-None-Match header lines

Header.Get reads only the first field line, so a match on a later
repeated header was missed; join all field values like any other
comma-separated list.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: ignore empty elements in etag precondition lists

* filer: let a present-but-empty precondition line suppress the date check

---------

Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: chrislusf <chris.lu@gmail.com>
2026-10-24 07:04:41 -07:00
Chris LuandDevin d6a68468ad volume: keep a below-minimum fsync batch open for stragglers (#12016)
* storage: hold a below-minimum fsync batch open for stragglers

When the durable-write queue drained, the worker flushed whatever it
held, so low-concurrency fsync writes collapsed to one sync per
request -- a seek plus a rotation on rotational media no matter the
payload. Once the queue empties with the batch still below the
configured minimum bytes and requests, the batch now stays open up to
SyncBatchOption.Wait, armed once per batch so the added latency stays
bounded. DiskLocation carries the option and defaults it; Store exposes
a setter the server calls once at startup.

* volume server: expose the fsync batch window as -fsync* flags

-fsyncMinBatchBytes, -fsyncMinBatchRequests and -fsyncBatchWait carry
the operator's batching floor into every disk's SyncBatchOption.
-fsyncBatchWait=0 restores the previous flush-on-drain behavior.

* server/mini: register the fsync batch window flags

weed server and weed mini build their own VolumeServerOptions and left
the three new fsync flag pointers nil, so startVolumeServer panicked
dereferencing them at startup.

* storage: bound the fsync batch window by Wait while the queue refills

Once the straggler timer was armed, a steady trickle of queued writes
kept taking the len(requests)>0 continue path, so the first write could
wait past Wait until the size or count cap flushed the batch. Track the
armed deadline and flush once it passes even with requests buffered.

* storage: sync the straggler test on the worker taking the first write

The fixed sleep let both writes queue together (passing under the old
flush-on-drain behavior) or the straggler arrive after the window closed
(flaky failure). Poll the drained queue instead so the second write can
only land inside the armed window or a later batch.

* volume server: default the fsync batch window when its options are unset

weed server and weed mini build VolumeServerOptions without the fsync
flag pointers, and outside embedders can too. Resolve the batch window
from the defaults so startVolumeServer never dereferences nil options.

* storage: drive the batch-window test through an injectable timer

The window is armed by the worker now, so the straggler provably joins
an open batch and expiry is fired by hand instead of racing a sleep.

* seaweed-volume: accept -fsyncMinBatch/-fsyncBatchWait flags

Same flags and defaults as weed volume so the Rust server's durable
write batching can be tuned identically.

* seaweed-volume: hold a below-minimum fsync batch open for stragglers

Mirrors the Go worker's batchWait: once the queue drains under the
minimum batch, the window stays open for stragglers and an expired
window still bounds the batch while the queue refills. Paused-time
coverage replaces any sleep-based assertion.

* seaweed-volume: sync the straggler test on the armed batch window

The test fired both writes together, so the straggler could already be
queued when the worker drained — a regression that drops mid-window
writes from the open batch would still pass. Wait until the worker
arms the window before sending the last write (Go's test polls the
drained channel for the same guarantee).

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* seaweed-volume: reject fsyncMinBatchBytes that does not fit usize

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 07:04:05 -07:00
b0e6f7a096 s3: allow up to 50 tags in a bucket tag set (#12030)
* s3: allow up to 50 tags in a bucket tag set

PutBucketTagging validated the tag set with the object tag rules, which
allow 10 tags. AWS allows a bucket up to 50 tags, so tag sets of 11 to 50
tags were rejected with InvalidTag. Validate bucket tags against 50 and
keep the limit of 10 for objects and S3 Tables.

A tag set that repeats a key is rejected with InvalidTag, as AWS does,
instead of keeping one of the values, so 51 tags with a repeated key no
longer pass as 50.

* s3: reject duplicate tag keys in PutObjectTagging

The tag set collapses to a map, so a repeated key was silently kept once.\nMatch the bucket tagging check and answer InvalidTag instead.

* s3: reject empty tag keys

AWS requires a tag key to be at least one character; the shared\nvalidator only checked the 128-char upper bound.

* s3: reject duplicate tag keys in the tagging header parser

parseTagsHeader (the x-amz-tagging path used by CopyObject with
TaggingDirective=REPLACE) silently kept the last value when a key
repeated, while the PutObject and multipart header paths and the new
XML checks return InvalidTag for the same input. A repeated key after
URL decoding now fails the same way.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: validate x-amz-tagging tags on PutObject before storing the entry

The header path only rejected duplicate keys, so an upload could store
more than ten tags or keys and values that fail S3 tag rules. Run the
parsed set through ValidateTags like the XML tagging handlers do.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: validate x-amz-tagging tags on multipart initiate

ParseS3Metadata stored header tags with no tag-set validation, the same
gap putToFiler had. Multipart uploads now reject oversize or malformed
tag sets with InvalidTag.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: count tag length in characters, not bytes

* s3: validate object tags before uploading chunks

* s3: reuse parseTagsHeader and drop raw header from warnings

---------

Co-authored-by: Wleeaf <179125857+wleeaf@users.noreply.github.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 07:04:01 -07:00
de02158212 filer: let NewChunkStreamReaderFromFiler fall back to the master on a cache miss (#11964)
refreshUrls invalidates every cached location of a volume after a failed
read and re-looks it up through the reader's lookup function. With the
cache-only masterClient.LookupFileId, one failed read to one replica left
the volume missing from the filer's cache until restart, so metadata-log
replay skipped it. Use GetLookupFileIdFunction like the other read paths.

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-10-24 07:03:47 -07:00
Chris LuandDevin 8844d9823f iam/oidc: wrap the jti replay cache in its own type (#12017)
* iam/oidc: wrap the jti replay cache in its own type

The replay-protection state lived as ten fields and nine methods spread
across OIDCProvider. jtiReplayCache now owns the store, count, bound,
sweep scheduling, and cleanup worker; the provider keeps a single jti
field and delegates configure, consume, stop, adopt, and merge.

* iam/oidc: lazily allocate the jti replay cache

A zero-value OIDCProvider left jti nil, so Initialize, single-use
validation, Shutdown, and the replay-state handoffs panicked on a nil
cache. Allocate the cache through a sync.Once-guarded getter instead.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 07:03:42 -07:00
Chris LuandDevin 3e5aa8c3b3 topology: mark replica quarantined before recording the pair (#11983)
* topology: serialize quarantine records with their layout marks

A quarantine publishes a (node, vid) record and marks the replica read-only at the layout level; release and heartbeat flag updates must observe that pair as one state. Store-then-mark let a concurrent release delete the record before the mark landed, stranding the surviving replica read-only; mark-then-store let a heartbeat clear the mark before the record was visible, returning a conflicting replica to the writable list.

quarantineLock now covers every record<->mark pair: flagDuplicateVolumeId's quarantine, releaseResolvedQuarantine's undo, the isQuarantined-gated marks on registration, and the quarantine-folded read-only flag writes on heartbeat paths. Remote server pushes stay outside the lock; a push that lands after the record was released is undone, and a failed release push re-adds the record for a later retry.

The release test now waits for in-flight async flags to settle instead of asserting on a single read.

* topology: fold quarantine into RegisterVolume's flag rebuild

RegisterVolume rewrites every replica's location read-only flag from what
its node last reported, and reported flags deliberately exclude quarantine.
Any registration of a shared vid -- a reconnected replica, a new copy, the
self-heal and delta-heartbeat re-register paths -- therefore cleared the
quarantine mark on every replica in the location, while the gate afterwards
only re-marked the registering node. Two call sites had no gate at all.

RegisterVolume now takes an isQuarantined predicate and evaluates it under
accessLock for every replica it touches, so the mark is never dropped in
the first place: a release unmarks after the rebuild, and a quarantine
mark re-applies on the next rebuild, with no window between.

* topology: keep quarantine when a vacuum commit re-adds a replica

SetVolumeAvailable re-adds the replica into the location list; a re-added
slot starts with a cleared read-only flag, so a quarantined replica came
back flag-free and the next ensureCorrectWritables could return a
conflicting volume to the writable list. Pass the quarantine state in and
set the flag back under the same accessLock, with the caller holding
quarantineLock so release cannot interleave between the check and the mark.

* topology: merge quarantine records instead of overwriting them

enforceQuarantine wrote back the state copy it loaded before its remote
push, so a concurrent writer's enforced=true was lost: the server stayed
read-only while the record said unenforced, and a later release skipped
the writable push it owed. The write-back now reloads the record under
quarantineLock and merges -- enforced wins from either side, reported is
taken from the newer pre-enforcement observation only while no push has
landed yet.

All three compensating writable pushes had the same hole in reverse: a
failed un-quarantine left the server read-only with no record to retry
from, and release's restore blindly overwrote any record a fresh flag
pass had created in between. They now go through restoreQuarantineRecord,
which adds the record only when absent and otherwise sets enforced on
the current one, preserving its reported flag.

* topology: hold quarantineLock over registration's quarantine check-and-mark

RegisterVolume reads the quarantine record and marks the replica
read-only under accessLock, but the pair ran without quarantineLock.
A release could push writable, watch the record get re-read, and then
delete it: the replica stayed read-only with no record left to clear,
and unchanged writable heartbeats skip SetReplicaReadOnlyFlag so the
mark never repaired. Holding quarantineLock around the check-and-mark
in all three production callers serializes it against the release
clear-and-delete.

* topology: serialize remote quarantine pushes behind a record recheck

releaseResolvedQuarantine deletes the record under quarantineLock and
sends the writable request after unlock. A conflict flagged while that
request is in flight creates a fresh record and marks the replica
read-only again, and the older writable request can still land last,
leaving the server writable under an active quarantine. The reverse
flap strands a stale read-only mark the same way.

Route every QuarantineVolumeFn call through pushQuarantine, which
holds quarantinePushLock across the remote call and re-checks the
record under quarantineLock right before sending: a read-only push
requires the pair to exist, a writable push requires it to be gone.
A skipped push means the decision already moved on, so the final
server state always follows the newest record.

* topology: serialize quarantine pushes per replica, not globally

A single quarantinePushLock held across the remote mark call let one slow
volume server (up to the 10s client timeout) delay pushes for unrelated
replicas. Keying the lock by (node, vid) keeps pushes to one replica
ordered in record-decision order while pushes to others run in parallel.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* topology: restore the layout read-only mark with a released record

A failed writable push re-armed the quarantine record but left the
replica marked writable in the layout, so PickForWrite could route to a
replica the server still holds read-only.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* topology: wait out async duplicate checks in the release-retry test

TestFailedReleasePushRestoresQuarantineMark asserts layout marks around a
release pass, but the two RegisterVolumeLayout calls each start a
background flagDuplicateVolumeId that can still be mid-flight. A late
mark could land after the final release and fail the writable assertion
even though the code behaved correctly. Track the async passes in a
WaitGroup and drain them before the explicit flag call so the rest of
the test is deterministic.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* topology: keep quarantine when a replica's layout key is unreadable

* topology: keep a failed writable push's restore ordered under the push lock

* topology: retry the quarantine release on every duplicate check

* ci: isolate Java and S3 test servers on persistent runners

* ci: drop the duplicated -ip flags in the lifecycle test server

The bind flags were already on the command; the isolation pass added a
second pair.

* topology: retry a failed quarantine writable push on the next report

* topology: retry pending quarantine releases off the leader pulse

A failed writable push re-arms the quarantine record with a retry that only
enforceQuarantine fires, and that runs only for volumes a heartbeat carries.
Once the server has reported the replica read-only it drops out of the delta
reports, and restoreQuarantineRecord keeps the stored flag matching the
server's so the digests agree and no full list is requested - the pending
release then never retries and the idle volume stays read-only forever. Run
the same release pass from the leader's pulse loop so a pending retry no
longer waits on a report.

* topology: exercise the pulse-driven quarantine release retry

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 07:00:45 -07:00
5700aa5e91 s3: evaluate bucket CORS for path-style OPTIONS on the bucket root (#12028)
* s3: evaluate bucket CORS for path-style OPTIONS on the bucket root

A path-style preflight for /bucket (no trailing slash) missed the bucket
OPTIONS route, whose template is /{bucket}/, and was answered by the global
OPTIONS handler from the global AllowedOrigins instead of the bucket CORS
configuration. Register the bucket OPTIONS route without a path matcher, and
keep OPTIONS for the health probe paths on the service-level handler ahead of
each path-style router.

Fixes #12025

* test: drop the issue reference from the bucket-root preflight test comment

---------

Co-authored-by: Wleeaf <179125857+wleeaf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 06:58:59 -07:00
Chris LuandDevin 9281ed8007 master: never auto-admit raft peers on a single-node configuration (#12021)
* master: never auto-admit raft peers on a single-node configuration

A master started without cluster peers (peers=none, or no peers so the
list holds only itself) declares a one-node cluster, yet any foreign
master that registered over KeepConnected was admitted as a second
voter. The resulting two-voter cluster cannot reach quorum when the
foreign peer is unreachable, wedging a previously healthy single
master. Gate raft admission on having been configured with cluster
peers; masters started with a peer list keep the announce-to-join
flow used to grow the cluster.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* master: source SingleMaster from -peers=none, not the peer count

A master started with an empty peer list ends up with a one-peer map
containing itself, and NewMasterServer inferred declared single-master
mode from that count. The master still mints a raft cluster through the
bootstrap path, but AdmitRaftPeer then ignores every later joiner, so a
deployment that grows from one master to several can never form a
quorum. isSingleMasterMode already spells out the declared case
(-peers=none); populate MasterOption.SingleMaster from it instead.

* master: cover clustered raft admission with a real in-memory leader

The previous clustered test had no raft leader and asserted nothing, so
an AdmitRaftPeer that always returned early still passed. Bootstrap a
single-node hashicorp raft over the in-memory transport, wait for
leadership, and assert the leader reports and admits a missing peer.

* master: return the raft admission result and assert committed join

AdmitRaftPeer ran AddVoter on a detached goroutine, so the only observable
state was an in-flight marker deleted on success and failure alike, and the
clustered test could never tell whether the join worked. Admission now
hands back a result channel; the test runs a connected second in-memory
node and asserts the peer sees itself a committed voter.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 06:58:16 -07:00
Chris LuandDevin fd3b437e04 s3api: write the null version's object-lock metadata to the plain entry (#12033)
* s3api: write the null version's lock metadata to the plain entry

PutObjectRetention and PutObjectLegalHold with versionId=null loaded the
pre-versioning entry but wrote the update under <key>.versions/v_null, so
the acknowledged lock never bound the object it was read from. Resolve the
entry and its write path in one helper that applies the same null-version
rule as getSpecificObjectVersion.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3api: cover the null-version lock write path

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3api: persist object-lock metadata with an in-place entry update

mkFile rebuilds the target entry from chunks alone, so a lock write on a
plain (null-version) entry discarded attributes, inline content, and the
remote reference. UpdateEntry keeps the whole fetched entry, matching how
the tagging and ACL writers persist metadata.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 06:57:02 -07:00
Chris LuandDevin 689ea6ea3e release: digest-bound provenance and SBOM attestations for container images (#12024)
* release: emit digest-bound provenance and SBOM attestations

Per-platform builds now record SLSA provenance at mode=max and a
BuildKit SBOM, so the builder identity and resolved dependencies are
captured instead of the empty predicate the 4.48 build record carried.
After the multi-arch index is assembled and signed, the merge job
publishes GitHub artifact attestations bound to the merged digest and a
syft SPDX SBOM of the image, giving digest-pinning consumers a
verifiable record on the exact index they pull.

* release: keep unfixed findings in the trivy SARIF upload

ignore-unfixed dropped every finding without an available patch, so the
uploaded report understated the image's exposure. The scan stays
report-only; only the completeness of what is uploaded changes.

* release: generate and attest SBOMs per platform digest

An index-wide syft scan only describes its default platform, so the SBOM
attested to the merged multi-arch digest misrepresented every other
manifest. Scan and attest each platform digest instead, which also drops
the sbom-action upload defaults that raced release-asset permissions and
artifact names across the merge matrix.

* release: resolve per-platform index to its image manifest before syft/cosign

push-by-digest records point at OCI indexes (image + attestation
manifests), so scanning or attesting the index digest picks the
runner's platform and fails outright for the other arches under
set -e. Resolve the non-attestation manifest inside each index and
bind the SBOM attestation to the image digest a same-platform pull
of the merged index actually returns.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 06:47:24 -07:00
Chris LuandDevin be081b49b7 notification: named webhook targets for per-path event routing (#11925)
* notification: honor the config prefix in the multi-queue guard

validateOneEnabledQueue checked "<name>.enabled" without the
notification prefix, so the fatal-on-two-queues guard never saw the
real keys and could never fire.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* notification: support named webhook targets under [notification.webhook.<name>]

The webhook queue only accepted one endpoint, so events could not be
routed to different services by path prefix. Each enabled
[notification.webhook.<name>] subsection now spins up its own target —
own filter, http client, retry middleware and worker budget — published
on a per-target topic so every matching endpoint gets each event exactly
once. The flat [notification.webhook] section keeps working as the
unnamed target, and named subsections alone now activate the queue.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* notification: require enabled for the flat webhook target

buildTargets included the flat section whenever an endpoint was set, so a
disabled [notification.webhook] block — which the scaffold ships with a
placeholder endpoint — would start receiving events as soon as any named
target activated the queue.

* notification: resolve named target enabled flags through config

namedTargetPrefixes read the raw map value, which never sees environment
variable overrides; GetBool resolves WEED_* env bindings, so a named
target disabled in the file can now be switched on by
WEED_NOTIFICATION_WEBHOOK_<NAME>_ENABLED.

* notification: cancel the webhook queue only after all workers stop

Each target's router goroutine deferred the shared cancel, so the first
worker to exit tore down the queue context for every other target and
the dead letter logger.

* notification: isolate webhook targets behind per-target queues

SendMessage published to each target's topic inline, so a subscriber
channel that filled up blocked every later target and the filer's
notification path with it. Each target now drains its own inbox through
a forwarder goroutine; SendMessage fails fast on a full inbox, matching
the "throw message" contract the filer already applies to errors.

* notification: name the webhook target in failure and dead letter logs

Every target dead-letters to the same topic, so without the name there
is no way to tell which endpoint failed or needs a replay.

* notification: assert delivery in the named targets test

The test only exercised SendMessage; it now records requests on both
endpoints and checks matching, non-matching, and unrouted keys reach
the right servers exactly once.

* notification: wait for the webhook router before accepting events

* notification: apply backpressure instead of dropping a full webhook inbox

* notification: bound queued and in-flight webhook deliveries per target

* notification: release the pending token when shutdown races the enqueue

A SendMessage that wins the pending select as the queue is canceling
could block forever on a full, undrained inbox.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* notification: stop waiting for a router that failed to start

A router that exits before reaching Running() would leave initialize
blocked on a context nothing had canceled yet.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* notification: route webhook events past saturated targets

A full pending reservation made SendMessage wait on the slow target
before it ever reached the healthy ones after it, coupling every
endpoint's latency to the most congested one. Targets that can take a
reservation now get the event immediately; saturated targets defer to a
second pass that still blocks for a slot, so backpressure is kept and no
event is dropped.

* notification: retry webhook deliveries before dead-lettering

PoisonQueue must wrap the retry middleware: added after it, the poison
middleware intercepts the first handler error, republishes to the dead
letter topic, and returns nil — the retry chain never sees a failure, so
max_retries/backoff settings were dead code and every delivery failed
straight to dead letter on the first attempt.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 02:12:17 -07:00
fb3fb9dcb8 s3api: support s3:RequestObjectTagKeys in bucket policies (#11993)
* s3api: support s3:RequestObjectTagKeys in bucket policies

Condition operators accept the ForAllValues: and ForAnyValue: qualifiers,
and s3:RequestObjectTagKeys holds the keys of the tags taken from
X-Amz-Tagging wherever s3:RequestObjectTag/<key> is set.

CopyObject no longer stores an empty-key tag for an empty segment in
X-Amz-Tagging, so it keeps the key set a policy has checked.

Fixes #7679

* s3: parse tagging header like the tag handlers, not url.ParseQuery

url.ParseQuery stops at 10,000 parameters, so an X-Amz-Tagging header
padded with empty & segments made requestObjectTags return nil: the copy
handler still stored the denied tag while every s3:RequestObjectTag*
condition saw nothing. Scan non-empty segments the way parseTagsHeader
does so the policy sees the same tags the object gets.

* test: gofmt s3tables test files after #11997

The format check fails on these files on master as merged, so every
PR's S3 Tables Format Check goes red until they are re-formatted.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 02:04:39 -07:00
Chris LuandDevin 7a9a9f10af ci: invoke pip via python -m in s3tests clone steps (#12010)
* ci: invoke pip via python -m in s3tests clone steps

The self-hosted runner reaches the Clone s3-tests step with the
setup-python toolcache interpreter on PATH but no pip shim, so the
bare 'pip install' exits 127. python -m pip resolves through the
interpreter that is already on PATH, matching the helm_ci workflow.

* ci: invoke pip via python3 -m in s3tests clone steps

The adjacent fix_s3_tests_bucket_conflicts step already runs python3, so
python3 is the interpreter guaranteed on the self-hosted PATH; a missing
python alias would fail the same way the missing pip shim did.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 23:05:56 -07:00
ecd778a4a0 iceberg maintenance: a merged file with null row ids names no first_row_id (#11973)
* iceberg maintenance: a merged file with null row ids names no first_row_id

Compaction stamps a merged v3 data file with the _row_id of its first row as
first_row_id (mergedFileFirstRowID). When the bin mixed rows that carry a
materialized _row_id with rows that carry none (a data file with no
first_row_id, as a v2 table's files are after an upgrade to v3, gives null
ids), the null rows then inherit first_row_id + their position in the file:
ids that other rows already hold, so two rows answer to the same row id.

A merged file now names a first_row_id only when its footer shows no null
_row_id in any row group. Otherwise the commit assigns the file a fresh range
from the table's next-row-id: the null rows get new, unused ids and the
materialized ones keep theirs (a materialized value overrides inheritance).

Test: TestMergedFileFirstRowIDOnlyWithoutNullIDs ("materialized first, a null
later" fails without the fix).

* iceberg maintenance: name no first_row_id when the merged file has a null _row_id

mergedFileFirstRowID stamped the merged file with its first row's id
without checking later rows. A null _row_id in a merged file inherits
first_row_id+position, an id another row already holds, so the commit
produced two rows answering to the same row id.

Check the footer's null counts for the _row_id column across row groups
first: any null means the file is committed without a first_row_id, so its
nulls take fresh ids from the table's next-row-id while materialized
values still override inheritance.

* shell: retry filer discovery until master knows a filer

A shell started against a freshly booted cluster (weed mini, the
s3tables/lance test harnesses) can list cluster nodes before the filer's
first heartbeat lands on the master. The empty list left FilerAddress
unset and the next command died with "dial tcp: missing address". Poll
ListClusterNodes until a filer appears or the connect timeout expires.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* shell: discover the filer on first use instead of at startup

The startup retry loop made every weed shell invocation wait out the
connect timeout on clusters that have no filer, even for commands that
only talk to the master. Discovery now runs once, on the first command
that actually needs a filer, keeping the retry-until-registered fix for
the fresh-cluster race without stalling master-only commands.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* shell: stop filer discovery retries on non-transient errors

A master that keeps answering ListClusterNodes with a permanent error
(permission denied, unimplemented) cannot gain a filer by retrying, so
discovery returns instead of polling until the deadline.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-23 22:53:43 -07:00
Chris LuandDevin 6a34d82f05 shell: retry filer discovery until master knows a filer (#11989)
* shell: discover the filer on first use instead of at startup

The startup retry loop made every weed shell invocation wait out the
connect timeout on clusters that have no filer, even for commands that
only talk to the master. Discovery now runs once, on the first command
that actually needs a filer, keeping the retry-until-registered fix for
the fresh-cluster race without stalling master-only commands.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* shell: stop filer discovery retries on non-transient errors

A master that keeps answering ListClusterNodes with a permanent error
(permission denied, unimplemented) cannot gain a filer by retrying, so
discovery returns instead of polling until the deadline.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* shell: explain an empty filer list when discovery times out

* shell: retry filer discovery while the address is still empty

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 22:52:51 -07:00
dependabot[bot]anduser.email b86b2f3cee build(deps): bump github.com/go-chi/chi/v5 from 5.3.1 to 5.4.0 (#12004)
Bumps [github.com/go-chi/chi/v5](https://github.com/go-chi/chi) from 5.3.1 to 5.4.0.
- [Release notes](https://github.com/go-chi/chi/releases)
- [Changelog](https://github.com/go-chi/chi/blob/master/CHANGELOG.md)
- [Commits](https://github.com/go-chi/chi/compare/v5.3.1...v5.4.0)

---
updated-dependencies:
- dependency-name: github.com/go-chi/chi/v5
  dependency-version: 5.4.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
2026-10-23 22:47:23 -07:00
57a122fbcf filer.remote.sync: dispatch an update event by the side of the mount it touches instead of slicing the mount prefix off a path outside it (#11958)
* filer.remote.sync: dispatch an update event by the side of the mount it touches instead of slicing the mount prefix off a path outside it

The daemon subscribes to its mount prefix and to /etc/remote. An update event
whose path lies outside the mount still reaches processUpdateEvent: an update
of the /etc/remote directory entry itself (which fs.meta.load of a mapping dump
re-creates) through the /etc/remote prefix, or a rename across the mount
boundary through the path on the inside. processUpdateEvent then maps both
paths to remote keys by slicing the mount prefix off, which panics with
"slice bounds out of range [20:11]" on a path shorter than the mount and names
a key in the wrong tree on a longer one. The event sits in the replay window,
so every restart dies on it and the persisted offset never moves.

Before dispatch, an update event is now mapped onto the side the mount covers:
a rename into the mount is the creation of the new entry, a rename out of it
the deletion of the old entry's object (kept when the entry is remote-only and
the object holds its only copy), an event with neither path inside is not the
mount's and is dropped, and an update with both paths inside is unchanged.

* filer.remote.sync: extract processCreateEvent

Moves the create branch of eachEntryFunc into a function, matching
processDeleteEvent and processUpdateEvent, so the create path can be
exercised directly in tests. No behavior change.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer.remote.sync: upload a create reshaped from a rename into the mount

A file synced under the mount keeps its RemoteEntry across renames. The
rename out deletes the remote object, and the rename back in reshapes to a
create whose inherited stamp still matches the entry, so shouldSendToRemote
skipped the upload and the new key stayed empty. The reshaped create now
sends unconditionally: the stamp describes the old key, not this one.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer.remote.sync: a create reshaped from a rename into the mount writes an empty entry as a zero-byte object

* filer.remote.sync: a create reshaped from a rename sends a stamped entry truncated since its sync

A stamped entry still reports remote-only once truncated to zero (the
stamp's RemoteSize stays positive), so the remote-only exemption skipped
it and the new key got no zero-byte object. Gate the exemption on
isDroppedLocalCopy: only an entry whose size still matches its stamp has
its content already at the new key via the rename's remote copy.

* filer.remote.sync: recover a superseded create reshaped from a rename-in

A create reshaped from a rename into the mount whose write is superseded
was skipped like a plain create, but no later event covers this
destination: remote.uncache accepts the rename's inherited RemoteEntry
stamp and drops the local chunks, and the update it logs skips the
upload through isDroppedLocalCopy, while the source mount's sync still
deletes the source object. The destination object is then missing and
the content exists nowhere.

recoverSupersededCreate finishes the event from the entry the filer
holds now: a rewritten entry is left to its own event's upload or
uploaded here on the stamp edge, and a remote-only entry is completed by
copying the object its stamp describes -- resolved through a
pending-rename marker under this mount, or through the filer's mount
mappings to the source mount's remote location.

---------

Co-authored-by: james <james@medable.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: chrislusf <chris.lu@gmail.com>
2026-10-23 22:43:39 -07:00
Chris LuandDevin 5e01caa522 master: cap parallel vacuums per rack via master.vacuum.max_parallel_per_rack (#11935)
* master: cap parallel vacuums per rack via -maxParallelVacuumPerRack

Hosts that run one volume server per disk under a single rack label could
not bound host-level vacuum I/O: maxParallelVacuumPerServer limits each
process, so every disk on the machine compacted at once. A second quota
map keyed on rack id debits alongside the per-server one, so at most N
volume copies compact in parallel within a rack. Defaults to 0 (no
limit); the flag is wired on master, server, mini and master.follower.

* master: resolve a data node's rack key once per vacuum sweep

A volume server that unregisters mid-sweep loses its parent node, so the
quota credit path calling dn.GetRack() panicked and crashed the master.
Rack names are also local to their data center, so same-named racks in
different centers shared one quota bucket. Each node's rack key is now
captured when the limiters are built and carries the data center id, so
checks, debits and credits never touch the live parent pointer.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* master: reserve vacuum quota per needed copy count, not per copy

Each copy of a volume checked the shared server/rack quota independently,
so several same-rack replicas all passed against one remaining slot and
the debits drove the bucket negative. Needs are now counted per server
and per rack first; a volume whose own copies exceed a cap reserves the
whole bucket once it is free instead of pending forever.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* master.follower: register -maxParallelVacuumPerRack as a flag

It was only a fixed default, so passing the flag on the command line
failed flag parsing.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* master: key the rack vacuum quota by rack node identity

Joining data center and rack names with "/" still let two different racks
share one bucket when the names themselves contained slashes. Key the
limiter by the rack node instead, which is unique per rack regardless of
naming.

* master: keep the tiered-replica vacuum test through the rebase

* master: document 0 as no limit for -maxParallelVacuumPerServer

The clamped per-server need makes a non-positive cap mean "no limit",
matching the new rack flag; say so in the flag help.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* master: key the rack-key cache by data node identity

Node ids are only unique within a rack, so two servers that share an id
in different racks would have reused the first server's cached rack and
escaped the per-rack vacuum cap.

* master: read the per-rack vacuum cap from master.toml

-maxParallelVacuumPerRack becomes the master.max_parallel_vacuum_per_rack
config key, so the new master limit is set the same way as other master
options instead of a flag wired into every command that embeds a master.

* master: let master.max_parallel_vacuum_per_server override the flag

The released -maxParallelVacuumPerServer flag keeps working, but a value
in master.toml now wins when set, matching how master.mdir already
overrides -mdir.

* master: group the vacuum caps under master.vacuum in master.toml

Rename master.max_parallel_vacuum_per_rack to master.vacuum.max_parallel_per_rack
and master.max_parallel_vacuum_per_server to master.vacuum.max_parallel_per_server,
so the keys sit in one subsystem table like master.volume_growth and
master.maintenance.

* master: deprecate -maxParallelVacuumPerServer for master.toml

The flag still works - and is overridden by
master.vacuum.max_parallel_per_server when set - but the help text now
points operators at the config key, matching the codebase style of
help-text-only deprecation.

* topology: read node parent atomically

UnlinkChildNode clears a data node's parent under the rack's lock while
readers like the vacuum sweep's rack capture reach it with no shared
lock, so a torn interface read could crash the master. Store the parent
in an atomic.Pointer and nil-guard the detached walk in
DataNode.GetTopology.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 22:41:36 -07:00
Chris Lu a1b3131bf6 test: handle unsupported catalog clients in S3 Tables CI (#12023)
test: distinguish unsupported catalog clients from S3 Tables failures
2026-10-23 22:40:54 -07:00
Chris Lu aba0cb7b2a ci: isolate S3 lifecycle mini from LAN clusters (#12022) 2026-10-23 22:31:15 -07:00
Chris Lu 81e0859c12 test(s3 remote cache): wait for the master to list the filer before shell commands (#12018)
A fresh single-filer mini registers its filer only after the ~10s
empty-membership confirm window, and weed shell locates the filer via
the master's listing, so remote.configure sent right after the S3
health check races it. Ports the wait from the enterprise test to the
OSS copy.
2026-10-23 20:54:52 -07:00
Chris LuandDevin 0d00e92178 iceberg maintenance: match percent-decoded keys when checking file references (#11978)
* iceberg maintenance: match percent-decoded keys when checking file references

Writers may percent-escape partition directory names in the file paths they
record in manifests (e.g. "part%3D1"), while the S3 layer decodes the key
before storing the object. remove_orphans compared the escaped reference to
the decoded filer path, counted the live file as an orphan, and deleted it
once it passed the age window — readers then got NoSuchKey on the next scan.

Match both spellings of every referenced path in remove_orphans, apply the
same coverage to the kept-file set in expire_snapshots, and delete expired
files at their decoded key so the object the S3 layer stored is the one
removed.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* iceberg maintenance: retry filer lookups at the percent-decoded key

A manifest reference carrying an escaped segment resolves to a filer name
that does not exist, because the S3 layer decoded the key on write. When a
lookup misses and the reference contains escapes, retry once at the decoded
spelling so maintenance can still read the file.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* iceberg maintenance: resolve referenced files verbatim-first, decoded fallback

filer_pb.LookupEntry reports misses as ErrNotFound rather than a gRPC
NotFound, so the escaped-reference retry never fired against a real filer.
Both maintenance reads and expired-snapshot deletes now resolve through one
verbatim-first/decoded-fallback lookup, and expiry refuses to delete a
resolved path outside the table's data directory.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* iceberg maintenance: test expiry of a file stored at a literal percent key

* iceberg maintenance: resolve referenced files in one filer lookup

lookupFileByIcebergPath re-fetched the entry resolveFilerPath had just
looked up, doubling the RPC per referenced file during compaction and
expiration. The resolver now returns the entry alongside the path so a
hit costs one lookup and a decoded fallback two.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* iceberg maintenance: keep every decoded spelling of a referenced path

The kept-reference sets recorded a path plus a single decode of it, but
lookupResolvedEntry decodes recursively until the spelling is stable. A
doubly-escaped reference could resolve to a spelling absent from the set
and be deleted while still referenced. addFilerPathSpellings now walks
the same decode chain the resolver does.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* iceberg maintenance: keep every decoded spelling of a referenced path

* iceberg maintenance: test expiry keeps a doubly-escaped kept reference

* iceberg maintenance: clean decoded spellings in the referenced-path set

* iceberg maintenance: test expiry keeps a kept reference resolved via decoded parent segment

* iceberg maintenance: decode only the reference in the spelling set

Decoding the composed filer path also decoded the table directory, so a
table at a literal percent-escaped path produced a kept set missing the
spelling the resolver returns, and expiry could delete a live file.

* test: gofmt the s3tables cluster helpers

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 20:49:49 -07:00
Chris LuandDevin 8541f959a9 filer.conf: merge fsync on presence like worm (#11953)
* filer.conf: merge fsync on presence like worm

fsync is a pure preference, so OR-merging it meant a nested rule could
never turn a parent's fsync off. Make PathConf.fsync optional and merge
it most-specific-wins, matching worm: an unset rule inherits, an explicit
value overrides in either direction. readOnly and disableChunkDeletion
stay OR'ed — they are fail-closed locks quota enforcement depends on.

Pre-version-2 configs were written with EmitUnpopulated, so every rule
carries an explicit fsync=false that meant nothing; it is dropped to
unset on load so it does not read as an opt-out.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer.conf: test fsync override survives a save/reload round trip

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 20:45:46 -07:00
Chris LuandDevin 673d1a02e0 operation: euthanize pooled upload conns that stall far below a fast host (#12003)
* operation: euthanize pooled upload conns that stall far below a fast host

A pooled keep-alive conn can wedge and drain the request body at a trickle
while fresh conns to the same volume run at full rate; every later upload
routed onto it stalls for tens of seconds, indefinitely. Track per-host
healthy-rate uploads (>=1MiB/s on a >=256KiB body) and, on a host that was
fast within the last minute, cancel an upload still in flight past
max(10s, size/256KiB/s): the transport discards the wedged conn and the
existing retry dials a fresh one. A watchdog kill also clears the fast mark
so the retried attempt runs unbounded, and hosts that never prove fast keep
today's unbounded behavior - a chronically slow link is never penalized.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* operation: keep the stall watchdog armed through the response read

uploadDone was set as soon as Do() returned, so a conn that accepted the
request body then wedged on the reply escaped the watchdog and could
park the upload for hours — the same failure this watchdog exists to
kill. Deferring the done-store keeps the timer live until the function
returns, so a stalled io.ReadAll is cancelled and retried like a
stalled write. The redundant post_err recheck goes with it.

* operation: pin watchdog coverage of a stalled response read

* operation: disarm the stall watchdog for an upload's remaining attempts

Once the watchdog has cancelled an attempt, a concurrent healthy upload
to the same host can restore the fast mark during the retry backoff.
The re-armed watchdog would then cancel the retry too, and a large
upload that merely exceeds the floor could burn every attempt. Track
the watchdog kill on uploadStatusError and carry a
stallWatchdogDisarmed flag through retriedUploadData so the rest of
that upload's attempts run unbounded; other uploads to the host keep
their watchdog.

* operation: pin watchdog disarm on retry, fresh-dial, and slow-upload cases

* operation: drop the send-stall watchdog once the request body drains

A wedged pooled conn is only blameworthy while it is still trickling the
request body; once the body is on the wire the wait is server-side commit,
which the transport's response header timeout already bounds. Cancelling
then would fail a healthy write whose replication or fsync is simply slow.
Track body EOF through a wrapping reader, gate the send-side timer on it,
and re-arm the watchdog only around the response-body read, which the
header timeout does not cover.

* operation: prove a host fast only on a verified upload

A quick rejection moves bytes no faster than the wire did, so mark the
host from the body-send duration and only after the response checks out.

* operation: count an abandoned request body as sent

The transport closes a request body it never finished draining when the
server answers early, so EOF alone under-reports the send boundary and
the stall watchdog could still blame the conn during the reply wait.

* operation: scope the body-sent mark to the current attempt

A lagging Close or EOF from the first attempt's body could stamp the
shared mark after the inner retry reset it, disarming the watchdog for a
send that never happened; each attempt now points the watchdog at its
own sent timestamp.

* operation: drop only the fast mark the stall timer observed

A concurrent upload can re-prove the host while a stalled attempt is in
flight; deleting the key outright would erase that newer mark, so the
watchdog now removes only the value it armed against.

* operation: keep the stall disarm on a per-call upload option

retriedUploadData marked the caller's UploadOption, so a caller reusing it
across UploadData calls would start the next upload with the watchdog off
and lose the cancellation this adds.

* test: stall watchdog retry must fail if the conn gets cancelled

The fake retry slept through any cancellation, so a re-armed watchdog
firing mid-retry still let the upload succeed — the guard was untested.
Park on the request context like a real wedged conn so a re-armed
watchdog cancels the attempt and fails the test.

* operation: prove the fast-host mark with elapsed-to-response

Body-drain time only measures when the transport consumed the reader:
kernel socket buffers can swallow a small body instantly on a slow link,
inflating the rate and marking the host fast. The next upload then gets
its conn cancelled mid-send despite making progress. Elapsed-to-response
bounds remote speed from below — the server cannot answer before the
body lands — so the mark can only under-mark, never over-mark.

* operation: scale the stall budget off the host's measured rate

Fixed bps limits fit no link well: 256KiB/s never fires on a slow link
and waits far too long on a fast one. Record the wire rate of each
sizeable upload and let a later upload run for slowFactor times the
body's span at that rate — a wedged conn shows as an order-of-magnitude
drop on any link speed. The recorded rate uses the wire body size so a
pre-compressed payload no longer inflates the mark.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 20:34:31 -07:00
Chris LuandDevin 6e8a2cdb14 filer.backup: process metadata events through the concurrent processor (#11951)
* filer.backup: process metadata events through the concurrent processor

Each event was handled inline on the subscription loop, so one
fetch+write at a time capped the stream at a few entries per second and
a busy source kept building lag. Route events through the same
MetadataProcessor filer.sync uses: it fans jobs out behind -concurrency
while serializing same-path and subtree-barrier conflicts, and the
checkpoint now advances on the processor's contiguous watermark via
GetResumeTsNs, with Resubscribe replaying anything a failure pins.
-disableErrorRetry keeps its log-and-skip meaning by returning nil after
logging, since the processor would otherwise pin the checkpoint forever.

* filer.backup: drain in-flight jobs before the stream retry loop reconnects

FollowMetadata returns as soon as the stream ends while admitted jobs keep writing; the next doFilerBackup builds a fresh processor that cannot order its replayed writes against them, so an older upload could finish last and clobber a newer file.

* filer.backup: checkpoint the completed watermark on a ticker and at stream end

The offset callback only fires while events arrive, so work draining on a quiet stream stayed uncheckpointed until the next event and a restart recopied it. Persist the processed watermark every 3s when it advances, and once more after the post-stream drain.

* filer.backup: bound the checkpoint wait before reconnecting

setOffset has no context, so a stalled KvPut would hold persistDone and block the retry loop forever. The persist goroutine now performs the final write itself on stopPersist, and the caller waits at most 30s.

* filer.backup: carry the processed position across retry attempts

A follow attempt that ended with a pinned failure recomputed startFrom on retry, so a -timeAgo window could move past the failed event and skip it. doFilerBackup now returns the failure pin (or processed watermark) and the retry loop clamps startFrom to it.

* filer.backup: resume retries at the previous attempt's pinned position

The retry loop threaded resumeTsNs through but never consulted it, so a
-timeAgo run recomputed startFrom as now-minus-the-window on every attempt
— a failed event older than the window was skipped on reconnect. Use the
pinned position when one exists; -timeAgo and the persisted checkpoint
only choose the first starting point.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer.backup: resume below the failure pin so the failed event replays

The metadata reader treats the resume position as exclusive — events at
or below it are skipped — so resuming at the failed event's own timestamp
would skip it. Back one tick off the pin.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer.backup: label the progress line as the received-event rate

The counter increments when a job is admitted, not when its copy
finishes, so the old wording showed backup throughput during a burst of
slow copies while the processed watermark had not moved.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* replication: bound the offset checkpoint RPC

setOffset called KvPut on the background context, so a stalled write
could pin a caller's drain or persistence loop indefinitely.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer.backup: keep the resume pin when filer-source setup fails

A retried attempt carries resumeTsNs so the failed event replays from the
same position, but filer-source and SSE initialization failures returned 0,
so the next attempt recomputed -timeAgo and could move the window past the
pinned event.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* replication: serialize checkpoint writes per key

A checkpoint write can outlive its caller's wait and land beside the next
subscription attempt's writes, moving saved progress backward. Serialize
writes per key inside setOffset and drop any value older than one already
written from this process.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer.backup: gofmt

* replication: serialize checkpoint writes per filer and key

A single process-wide lock made a stalled checkpoint RPC block every
stream's checkpoint, including the opposite direction of a bidirectional
filer.sync. Keeping the last-written watermark per persisted key also let
one stream's offset shadow another stream checkpointing the same key to
a different target filer. Lock and track progress per (filer, key).

* replication: keep the resume position behind a cleared failure pin

A redelivery that clears failedTs lets the processed watermark advance
past the earliest failure, but events dropped while the processor was
stopped — possibly sharing the failure's timestamp — have to replay.
Latched earliestFailedTsNs floors the resume position one tick below
that failure so reconnects still cover them.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* replication: persist floored checkpoints through the write ordering

setOffset drops a value older than the last written, which also swallows
a deliberate regression: the resume floor, an operator-seeded offset, or
a snapshot high-water mark. allowRegress marks those callers so the
checkpoint can move backward when a failure forces replay.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* replication: order checkpoint writes by issuance, not value

The value-based stale guard could never fire — every caller passed
allowRegress because floors and seeds are deliberate regressions — and
ordering by value cannot tell a stale high watermark apart from real
progress: a write that outlives its attempt could still land after a
floored resume and skip events owed a replay. Order by issuance instead:
a write issued before the last one to land under the key is dropped, so
late writes from a superseded attempt can no longer move the checkpoint
in either direction while deliberate regressions still apply.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* replication: keep the resume position one tick below the frontier

* remote.sync: resume and persist through the floored watermark

* test: concurrent metadata jobs fan out up to the limit

Distinct-path events must all be in flight before any finishes — a
serial loop could only ever have one running — while the limit still
bounds the pool. Pins the property behind the backup-throughput fix.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 20:02:55 -07:00
github-actions[bot] cd562b8514 docs: regenerate star history chart 2026-10-24 02:56:42 +00:00
Chris LuandDevin 617599859b docker: build RocksDB v11.1.2 for grocksdb v1.11.1 (#12002)
The rocksdb release image fails to compile: grocksdb v1.11.1 calls
C.rocksdb_block_based_options_set_block_align,
C.rocksdb_block_based_options_set_index_block_search_type, and
C.rocksdb_block_based_options_set_separate_key_value_in_data_block, all
of which only exist in RocksDB v11.x headers. grocksdb v1.11.1's own
build script pins 11.1.2, so move every default from v10.10.1 to
v11.1.2 to match.

Generated with [Devin](https://devin.ai)

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 18:50:27 -07:00
Chris Lu ce4c1fe31e ci: install fuse3 when runners lack fusermount3 (#12013)
Self-hosted runners do not ship fuse3, so the setuid repair step
failed with 'no fusermount3 in PATH' before the FUSE jobs could run.
Install the package instead of failing outright.
2026-10-23 18:49:04 -07:00
Chris Lu 462e4445e2 test: skip catalog client images that cannot execute on the host CPU (#12014)
New upstream releases build for x86-64-v3 (AVX2) - ClickHouse since
26.5, plus the DuckDB iceberg extension and pylance wheels - so on a
runner whose CPU lacks AVX2 the container dies on SIGILL before the
test reaches any SeaweedFS code. Probe each third-party image first
and skip on the illegal-instruction exit instead of failing; other
errors still surface normally.
2026-10-23 18:46:26 -07:00
Chris Lu 0d4e11a6e5 filer: bootstrap a timestamp-less peer entry onto a missing local path (#11975)
A zero local commit time conflated 'no local state' with 'state at the
epoch', so a missing entry was skipped whenever the sender reported no
commit time and the entry mtime was at or before the epoch. Only hold
the copy off when a local entry actually exists.
2026-10-23 18:45:09 -07:00
Chris Lu 85f2f9d2d4 ci: make docker registry and compose steps idempotent on self-hosted runners (#12012)
The registry container is started with --restart=always but never
removed during teardown, so once a machine runs the job, every later
run on that machine fails with a container name conflict. Reuse the
running container when it exists and recreate it only when missing or
broken.

Some self-hosted runners (e.g. seaweedfs-server03) also lack the
Docker Compose plugin, so install it via docker/setup-compose-action,
which is a no-op where compose is already available.
2026-10-23 18:37:36 -07:00
Chris LuandDevin 679835e4fc volume.fsck: run all filer-entry purges through the shared worker pool (#11960)
* volume.fsck: queue per-volume orphan deletes through the shared purge pool

The worker pool added for -reallyDeleteFilerEntries only covered entries
whose volume was missing entirely; the per-volume check that finds filer
entries pointing at absent needles still issued one synchronous HTTP
DELETE per orphan, which is the bulk of the work the report measured.
One pool now serves both purge phases and is drained before the empty-
directory sweep.

* volume.fsck: keep hard link ids in the filer fid file

Orphans queued by the per-volume check must hash into the pool by their
hard link id just like the missing-volume path, so two paths sharing an
id still serialize on one worker. The fid file now stores the id beside
each path and readFilerFileIdFile hands it to the caller.

* volume.fsck: cover orphan purge queuing and hard link id round-trip

* volume.fsck: fail on a truncated hard link id in the fid file

A .fid record cut short inside the hard link id length was emitted with a
nil id, so its purge could hash to a different worker than the entries it
shares the link with. Read the length strictly like the header and path.

* volume.fsck: drain the save consumer before returning a root genFn error

The early return left the save goroutine blocked on outputChan while the
caller ran finishPurge, so a late queuePurge could send on a closed
channel and panic.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* storage: skip EC shard-size encoding tests when disk space is low

TestCalculateExpectedShardSizeWithRealEncoding writes a .dat file plus
14 EC shards per subtest (~1.3GB peak for the 512MB case) into the temp
dir. Shared CI runners can run low on space while other test packages
write to /tmp in parallel, and the encode then dies with ENOSPC
mid-write, surfacing as a test failure. Check free space up front and
skip the subtest instead.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 18:33:51 -07:00
3810fb1123 s3: url-encode ListMultipartUploads results for encoding-type=url (#11998)
* s3: url-encode ListMultipartUploads results for encoding-type=url

The response echoed EncodingType=url but returned raw upload keys, and
XML 1.0 cannot carry control characters, so the encoder replaced them
with U+FFFD. The listed key then matched no upload: AbortMultipartUpload
with it answered 404 NoSuchUpload, leaving such uploads uncleanable.

Encode the key-named elements AWS encodes for this API (Key, Prefix,
Delimiter, KeyMarker, NextKeyMarker) the way ListObjects encodes its
keys when encoding-type=url is requested. Upload ids stay opaque, and
responses without encoding-type=url are unchanged.

Fixes #11991

* s3: echo EncodingType only when encoding-type=url is honored

A request with an unrecognized encoding-type echoed the value back in
<EncodingType> while the keys stayed raw, matching neither AWS (which
omits the element unless url encoding applies) nor the ListObjects
handlers, which gate the echo on the parsed url flag.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 18:21:26 -07:00
Chris LuandDevin 12d0537215 test: gofmt s3tables test files (#12011)
Unformatted test files leave the S3 Tables Format Check red on every
branch until they are re-formatted.

Generated with [Devin](https://devin.ai)

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 18:18:32 -07:00
Chris LuandDevin b2817e853d test: raise mini-server startup wait from 30s to 90s in Makefiles (#12005)
* test: raise mini-server startup wait from 30s to 90s in Makefiles

Self-hosted/persistent runners under load regularly need >30s for
weed mini (master+volume+filer+s3+iceberg+lance+webdav+worker) to
become healthy; hosted runners start in <10s. Siblings already wait
90s (s3/checksum, s3/cors). Fixes the S3 Filer Group first-run flake
on busy self-hosted runners.

* test: align Makefile startup guards and progress totals with 90s bound

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 18:15:01 -07:00
Chris Lu 1e0f571e75 ci: set up Go for CodeQL autobuild (#12007)
The workflow relied on a Go toolchain preinstalled on the runner
image; self-hosted runners without setup-go fail at autobuild with
'go: executable file not found'.
2026-10-23 17:52:50 -07:00
Chris Lu baf0dd680e ec.encode: require shard placement to meet -shardReplicaPlacement (#11947)
* ec: measure shard placement against replica-placement caps

ecShardPlacementViolations reports where a volume's newest-generation
shard placement exceeds the per-node and per-rack caps a
-shardReplicaPlacement requests, using the same digit mapping the
balance planner enforces (SameRackCount = shards/node, DiffRackCount =
shards/rack, dc-qualified rack keys).

* ec.encode: -strictPlacement keeps originals when placement breaches caps

-shardReplicaPlacement tells the balancer the requested per-node and
per-rack shard limits, but nothing verifies the outcome. With
-strictPlacement the pre-delete shard verification also measures the
final placement against those caps: a volume still over the node or
rack limit aborts before its source is deleted, so the encode only
completes when the requested failure-domain spread is actually met.

* ec.encode: cover the strict-placement delete gate with a test

The gate in verifyEcShardsBeforeDelete now runs against a stubbed
topology: a compliant spread passes while an over-cap placement is
refused before source deletion. The retry knobs become package vars so
the test does not wait out the heartbeat-propagation backoff.

* ec.encode: enforce -shardReplicaPlacement without a flag

Drop the -strictPlacement option: the requested per-node and per-rack
caps are always requirements at the pre-delete verification, so the
encode only completes when the placement actually honors them.
2026-10-23 17:29:36 -07:00
f5e319ae8e iam: GetUserPolicy returns NoSuchEntity for a policy name the user does not have (#11944)
* iam: GetUserPolicy returns NoSuchEntity for a policy name the user does not have

When the user has stored inline policies, an unknown name fell through to the
policy rebuilt from the user's actions, so a deleted policy still read back.
The standalone IAM server already returns NoSuchEntity here.

* iam: return service failure when listing inline policies fails

A ListUserInlinePolicies error in GetUserPolicy fell through to
action-based reconstruction, masking a store failure as a policy
document. Propagate it as ServiceFailureException instead, matching
the standalone iam server and the other callers in this file.

* iam: return service failure when the inline policy store read fails

---------

Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 17:08:49 -07:00
Chris Lu f8579bf728 s3: gate POST Object acl fields behind PutObjectAcl (#11939)
* s3: gate POST Object acl fields behind PutObjectAcl

The POST Object handler promotes acl and x-amz-grant-* form fields onto
the upload's ACL headers, and resolveFileMode honors the promoted canned
ACL. The PUT path only applies such headers after an s3:PutObjectAcl
grant; mirror that on POST by running the shared per-key authorization
with the WriteAcp action whenever the request carries ACL headers.

* s3: cover the POST Object acl gate

Signed POST requests carrying an acl form field are denied for a writer
without the WriteAcp action and for a writer a bucket policy denies
s3:PutObjectAcl on the target key; the authorization boundary is
exercised directly as well.

* s3: cover x-amz-grant-* POST fields and the no-ACL case

* s3: encode POST policy conditions as JSON in the acl-gate test
2026-10-23 17:05:44 -07:00
Chris LuandDevin 168789d5f3 master: cap writable volumes per collection via -maxWritableVolumes (#11923)
* master: cap writable volumes per collection via -maxWritableVolumes

Some deployments serialize disk writes (e.g. an Unraid array where
concurrent writes to multiple disks collapse throughput). With more
than one writable volume in a collection, the master's weighted pick
spreads assignments over all of them, so several disks get written at
once.

-maxWritableVolumes=N narrows the pick candidates to the N fullest
writable volumes, so a collection's writes concentrate on one (or few)
volumes at a time. 0 keeps the current spread-everywhere behavior.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* master: give master.follower a maxWritableVolumes default

* master: cap writable volumes inside the requested location

* master: pick the capped volumes without sorting the writable list

* master: assert the capped candidate set instead of random pick outcomes

* master: describe -maxWritableVolumes as a per-layout cap

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 17:04:36 -07:00
Chris Lu 8b26da8c53 ci: gate go/java/python tool caches to github-hosted runners (#12001)
Follow-up to #11985/#11986: setup-go/setup-java/setup-python steps
added or restructured since (e.g. by #11981's shared-artifact layout)
were missed. On self-hosted runners setup-go restores/saves a ~5GB
module cache archive per job — pure waste when the module cache is
already on local disk — and the restore blocks the job for minutes.
2026-10-23 16:25:23 -07:00
Chris Lu cd97c93deb command: keep filer, s3, and sftp up when the local unix socket cannot bind (#11999)
* pb: share the stale-socket tolerant unix listener

ServeGrpcOnLocalSocket already treats the local socket as optional; pull the remove-and-listen block out so the filer, s3, and sftp http sockets can use it too.

* command: keep filer, s3, and sftp up when the local socket cannot bind

A stale socket file owned by another uid (e.g. left in /tmp by a containerized job) made os.Remove fail with EPERM and the Fatalf took the whole server down. The gRPC local socket already treats the unix listener as optional; do the same for the http/sftp sockets.

* filer: clear shared localSocket flag when the unix socket cannot bind

s3.localFilerSocket and (in weed server) the sftp local socket share the
filer's -localSocket flag pointer. If the filer cannot bind the socket it
now keeps running without it, so consumers still holding the configured
path would dial a dead or foreign-owned socket.

* command: keep sftp on its own localSocket flag under weed server

sftpOptions.localSocket was overwritten with the filer's -localSocket
flag, so -sftp.localSocket was ignored and an explicit filer socket path
made the sftp server try to bind the same socket the filer already held.

* s3: fall back to TCP when the filer local socket cannot be dialed

The embedded S3 server copies the filer's -localSocket flag before the filer tries to bind it, so a socket that never comes up left every filer HTTP request dialing a dead path. Try the unix socket first and fall back to the request's TCP address on dial failure.

* filer: keep the shared localSocket flag read-only after startup

Clearing *fo.localSocket raced with the embedded s3 goroutine reading the same flag pointer and could not help a reader that had already copied it. The s3 filer client's TCP fallback covers the unbound-socket case without mutating shared flag state.
2026-10-23 16:07:32 -07:00
Chris Lu 6c87fb70f9 server: skip local unix sockets that cannot be bound (#12000)
server: skip default local unix sockets that cannot be bound

weed server fatally exited during S3, filer, or SFTP startup when the default /tmp/seaweedfs-<svc>-<port>.sock path could not be removed or bound. On shared hosts (e.g. CI runners where another user's weed left a socket in the sticky /tmp directory) a stale socket file turns into 'operation not permitted' and kills the whole server, even though the TCP listeners are unaffected. Warn and skip the unix socket listener instead, matching how the gRPC local socket already degrades. An explicitly configured -localSocket path still fails fast, since a co-located S3 server may be configured to dial the filer through it.
2026-10-23 16:02:33 -07:00
Chris Lu 29b81a827d volume: also listen on localhost like other components (#11995)
* volume: also listen on localhost like other components

Master, filer and s3 bind both the detected host ip and localhost via
NewIpAndLocalListeners, but the volume server used a plain NewListener on
bindIp, so a volume bound to the host ip never answers on loopback. Any
tooling that polls localhost (e.g. the java test workflow's readiness
curl on localhost:8080/status) then waits out its full timeout even
though the volume is up and registered with the master.

Use NewIpAndLocalListeners for the volume grpc, public and cluster http
listeners, and add a small multiHttpdownServer composite so graceful
shutdown still covers both listeners.

* volume: stop and wait dual listeners concurrently

A serial multiHttpdownServer.Stop let the localhost listener keep
accepting requests while the bound-ip listener drained, and a serial
Wait hid a localhost serving failure until the bound-ip listener ended.
Run both waits and both stops in parallel: Wait reports the first error
promptly, Stop closes every listener when shutdown begins.
2026-10-23 15:44:18 -07:00
Chris Lu aeb5db1b13 tests: pin weed mini admin ports and disable unused aux services (#11997)
* tests: pin weed mini admin ports and disable unused aux services

Every weed mini that does not configure them still binds the fixed
default ports for the admin server (12535 + derived 22535 gRPC), its
legacy compat listener (23646 + 33646), WebDAV (7333), Lance (9101) and
the Iceberg REST catalog (8181). Sequential or concurrent minis on one
host then collide on them: the port check warns and shifts, and the
admin's late bind can land on a stolen ephemeral port, adding noise and
flakiness to test logs (e.g. run 40017181958, job 120659267479).

Add testutil.MiniAdminArgs to pin the admin server to an allocated
mini-style port (HTTP + port+10000 gRPC, both below the Linux ephemeral
port floor since admin binds late) and disable the legacy compat
listener. Every test that spawns weed mini now also disables the aux
services it does not use: WebDAV, Lance and the Iceberg REST endpoint
get port=0 unless the test actually exercises them, in which case their
ports stay pinned as before.

* tests: take mini admin ports from the below-ephemeral range

* tests: reuse MiniAdminArgs for the remote-s3 fetch mini

* tests: fully disable webdav and co-allocate mini admin ports

- -webdav.port=0 only makes the service bind a random port; add
  -webdav=false so minis that do not test WebDAV skip it entirely.
- Spark, RisingWave and s3 Spark launchers allocated the admin port in
  a second MustFreeMiniPort call, which could pick a port already
  handed out by the first batch. Fold 'Admin' into the same
  MustFreeMiniPorts call.
- test/fuse_integration is a separate Go module and cannot import
  testutil; allocate the admin ports with its local freePort helper
  instead.

* test/fuse: derive the mini admin gRPC port from its HTTP port

Two independent freePort calls could return the same port, making admin.port equal admin.port.grpc (or collide with the filer port). Allocate once and derive the gRPC port with the +10000 convention.

* test/fuse: pin the mini admin port in the fuse_test framework copy

The fuse_test package framework spawns weed mini too; without a pinned admin port it still binds the fixed defaults 12535/23646 and collides with other minis.

* tests: drop the redundant -webdav.port=0 flag

-webdav=false alone disables the service; the port flag is dead config.
2026-10-23 15:38:09 -07:00
Chris Lu c925869dcc shell: retry filer discovery until a filer registers (#11994)
* shell: retry filer discovery until a filer registers

When -filer is not given, the shell discovers a filer with a single
ListClusterNodes call right after connecting to the master. A filer can
take several seconds to register after its port reports ready, so the
list routinely comes back empty and the shell proceeds with an empty
FilerAddress. Commands then fail with "dial tcp: missing address", which
is how the s3tables Dremio integration test flaked in CI.

Treat an empty list as retryable and keep polling ListClusterNodes until
a filer appears or -connectTimeout expires. On timeout, print a clear
"no filer registered" notice instead of failing later inside a command;
master-only shell commands keep working either way.

* shell: report the last filer discovery attempt failure

A successful empty ListClusterNodes followed by a master outage left the
diagnostic claiming no filer registered, hiding the real RPC failure.
Track the outcome of the latest attempt instead: if the final list call
failed, report that error; only when the last response was genuinely
empty report that no filer registered within the timeout.

* shell: test filer discovery retries against a fake master

Extract the discovery loop into listFilers and cover the retry behavior
with a fake master_pb.SeaweedServer: an empty cluster turning populated,
an empty cluster exhausting the window without error, the last attempt's
RPC failure being reported, and master-only commands still running after
the discovery timeout.
2026-10-23 15:21:41 -07:00
34f06ee92f test/fuse: fix cluster startup in the fuse test framework (#11977)
* test/fuse: fix cluster startup in the fuse test framework

weed volume Fatalfs when -dir does not exist, so every cluster setup
failed its volume wait after 30s. The mount also exited on the unknown
-cacheSizeMB flag (it is -cacheCapacityMB now).

* test/fuse: manage cluster processes and allocate free service ports

Each service now runs through a managed process that captures its log and
reports an early exit from the port wait, instead of a bare 30s timeout.
Ports are allocated free pairs (http + grpc sibling) rather than fixed
19333/18080/18888 so the framework no longer collides with weed mini's
fixed ports or another framework instance.

* test/fuse: bail out of the mount wait when the mount process exits early

The port waits already report an early exit with the captured log tail; the
mount wait still polled the device id for the full timeout, so a mount that
died at startup (bad flag, missing filer) surfaced as a bare 30s timeout
with no diagnosis. Check the managed process the same way and include its
log tail.

* test/fuse: deduplicate service ports handed out by freeServicePort

A probed port is released before the service binds it, so the three
sequential allocations for master, volume and filer could all return the
same still-unbound port and leave later services dying at startup. Track
handed-out ports so each allocation is unique.

* test/fuse: allocate service ports in Setup after the external-mount branch

* test/fuse: verify service readiness through its own /healthz endpoint

* test/fuse: bound the readiness probe so a silent listener cannot hang setup

* test/fuse: only accept a ready response while the service process still runs

* test/fuse: include the service log tail on readiness timeout

A startup that stalls instead of exiting still failed with a bare
"not ready within timeout" — the same opaque message that made the
original flake hard to diagnose. The process-exit path already tails
the captured service log; the timeout path now does too.

* test/fuse: report pjdfstest category results without failing the suite

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: allow allow_other mounts in posix-compliance

The job mounts with -allowOthers but never sets user_allow_other in
/etc/fuse.conf, so fusermount rejects the option and the mount exits
before the suite runs. The other FUSE workflows set it; do the same.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test/fuse: keep the standalone-framework service logs on failure

Cleanup removed the temp dir holding master/volume/filer/mount logs, so
a CI failure left nothing to diagnose. Dump the tails on failure and
copy them under /tmp/seaweedfs-fuse-logs/standalone/ for the artifact
upload, next to the mini framework's logs.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test/fuse: gate setup on a real write through the mount

Mount readiness only proves the daemon is serving; a fresh cluster still
cannot assign a file id until a volume heartbeats in and a grow round
finishes, so the first ~10-30s of writes surface as EIO on close. Probe
with a real write/read/remove before handing the mount to the suite.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test/fuse: gate the mini framework setup on a real write too

Same cold-start window as the standalone framework: mini's volume has to
register and grow before a write can be assigned. Keep the two
frameworks consistent so later suites do not flake on the same stall.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test/fuse: save each framework's CI logs under its test name

Fixed-name mini.log/mount.log in /tmp/seaweedfs-fuse-logs let a later
framework instance overwrite an earlier test's logs, so an uploaded
artifact could describe a passing run instead of the failure under
investigation. Apply to both framework copies since they share the
directory.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test/fuse: bound each waitForWritable probe attempt

A wedged mount can hold a file op open past the 90s deadline, so the
loop could not check the deadline or a mount exit and Setup hung until
the overall test timeout. Each attempt now runs off the loop and is
abandoned when the remaining time runs out. Same fix in both framework
copies.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test/fuse: keep one write probe in flight until it finishes or the deadline

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: chrislusf <chris.lu@seaweedfs.com>
2026-10-23 08:10:31 -07:00
Chris Lu 2c8079625c test: stop EC integration tests from pinning default master port 9333 (#11988)
The EC integration tests failed in CI with "raft.Server: Not current
leader" on every assign for tens of seconds — longer than any goraft
election takes. A stale weed process on the self-hosted runner was
still holding 9333, so each test's fresh master died at bind time and
the tests kept talking to a raft non-leader. Only the 9333 cluster
(startSeaweedFSCluster) failed; tests on 9334-9338 all passed.

Allocate mini-safe ports (http + grpc+10000 both verified free) for the
master and volume servers instead of hardcoding 9333/8080-8085, wait
for the master to actually report raft leadership via /cluster/status
before issuing raft-gated RPCs, and make waitForServer a real TCP dial —
grpc.NewClient never connects, so the old check always passed
instantly.
2026-10-23 08:08:49 -07:00
bati-valensasandChris Lu c4002fafc7 helm: fix bucket hook treating an existing bucket as missing (#11982)
* helm: fix bucket hook treating an existing bucket as missing

The skip check pipes awk into `grep -q`, which exits on the first match and
can leave awk writing into a closed pipe. awk then dies of SIGPIPE and
`set -o pipefail` makes the pipeline report that, so a bucket that exists
reads as missing: the hook runs `s3.bucket.create`, it errors with "already
exists", and `sh -ec` aborts the hook - failing the release.

Match in a single awk instead, which reads to EOF and cannot be killed by a
closing reader.

* helm: force string compare in bucket hook existence check

---------

Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-23 06:43:35 -07:00
Chris LuandDevin 384523761b ci: skip actions/cache on self-hosted runners (#11986)
cargo registry/target and docker-image caches are already local on a
persistent self-hosted runner; the save/restore pass is pure overhead
(the setup-go cache-save stall on awscli-tests was the same pattern).
Gate every actions/cache step on runner.environment == 'github-hosted'.

Generated with [Devin](https://devin.ai)

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 06:19:56 -07:00
Chris LuandDevin 277c9809c8 ci: skip setup-go cache on self-hosted runners (#11985)
* ci: skip setup-go cache on self-hosted runners

The module cache is already local on a persistent self-hosted runner;
archiving ~/go/pkg/mod into cache.tzst at every job's end is pure cost
and can stall (a ~4-minute zstdmt hang was observed on awscli-tests).
runner.environment gates the cache to GitHub-hosted runners only.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: gate the remaining setup-go caches too

The first pass only covered go-version-file steps; go-version steps in
nine workflows (including awscli-tests' own file) kept uploading the
cache archive on self-hosted runners.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 06:05:33 -07:00
Chris LuandDevin 728fc69976 ci: cache docker build layers for container_dev (#11980)
* ci: cache docker build layers for container_dev

The Go-build image rebuilds its layers on every run; type=gha backend
caching reuses them across runs and platforms.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: pin container build to the pushed sha under layer caching

Dockerfile.go_build clones master inside the image; a cache-from layer
would otherwise keep serving the previous commit's clone and publish
stale code under the dev tag. Passing BRANCH=github.sha makes the clone
layer's key commit-specific while toolchain layers still hit.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: pass BRANCH via build-args so the per-commit cache key reaches buildx

docker/build-push-action has no `args` input — the BRANCH override was
silently dropped, leaving the build arg at its `master` default and the
clone/checkout layers serving a stale commit under the gha cache.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 06:02:08 -07:00
Chris LuandDevin 9b59bc6b0d ci: build test binaries once, share them across shard jobs (#11981)
* ci: build test binaries once, share them across shard jobs

Each of the 6 rust-volume-go-tests shards, the rust-integration-tests
job, and the 6 volume-server-integration-tests shards rebuilt the same
weed and weed-volume binaries — 13 duplicate builds per run. A
build-test-binaries / build-weed job now compiles them once under the
existing cargo cache and publishes them as artifacts; the consumers
download and chmod them instead of building.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: preserve binary paths through the artifact round-trip

upload-artifact stores single files flat, so weed/weed would come back
at the workspace root and the consumers' expected paths break. Pack the
binaries in a tar before upload and extract on download.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: extract the shared test binaries in the rust go-test shards

The download step leaves rust-test-binaries.tar at the workspace root;
without untarring it the chmod fails on missing files and every shard
fails before running a test.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 04:50:29 -07:00
Chris LuandDevin f7af816ab4 ci: cancel superseded workflow runs on PR pushes (#11979)
* ci: cancel superseded workflow runs on PR pushes

PR-triggered workflows without a concurrency group keep running (and
holding runner slots) after a force-push or quick follow-up commit
supersedes them. Add the same group/cancel-in-progress block the go
workflow already uses, keyed on head_ref, to the remaining PR workflows.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: key concurrency groups on PR number or ref, not head_ref

github.head_ref is empty on tag pushes and manual runs (every run then
shares one group and cancels the last) and collides across forks when
two PRs share a branch name. pull_request.number is unique per PR;
github.ref covers push/tag/dispatch. Applied to all workflows using the
pattern, including pre-existing ones.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: restrict cancel-in-progress to PR and branch-push runs

* ci: serialize container_latest manual runs by destination tag

Two workflow_dispatch runs sharing an image_tag could otherwise build
concurrently: their per-arch images publish to the same platform tags,
so one run's create-manifest job can combine platforms from different
source refs. Group dispatch runs by the destination tag (with
cancel-in-progress already disabled for non-PR/non-branch events) so
same-tag publishes queue instead of interleaving.

* ci: serialize manual publish runs by destination tag in remaining container workflows

* ci: group foundationdb release dispatch runs by release tag

A manual run now shares the tag-push concurrency group for the same
release, so a re-publish of a tag queues behind (or ahead of) the push
run instead of racing it on the same image tag.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: serialize foundationdb_version dispatch runs by destination tag

Two manual runs resolving to the same docker tag now share a concurrency
group (explicit image_tag, else the computed
foundationdb_<fdb>_seaweedfs_<ref> default), so they queue instead of
racing pushes to the same tag.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: drop unsupported toLower() from foundationdb_version concurrency group

* ci: key rocksdb_version dispatch group on its published tag

* ci: queue publish runs via turnstyle instead of cancel-free shared groups

* Revert "ci: queue publish runs via turnstyle instead of cancel-free shared groups"

This reverts commit 039148f8d5.

* ci: queue same-tag publish runs instead of replacing the pending one

Dispatch/tag-only workflows can never cancel in progress, so opt their
concurrency groups into queue: max — up to 100 same-tag publishes wait
in line rather than each new dispatch cancelling the pending one.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 04:49:52 -07:00
Chris Lu 7c478fd89a fuse integration: run posix suites on weed mini and allow allow_other mounts (#11984)
The separate master/volume/filer processes in framework.go bound fixed
ports, discarded child output, and never noticed an early process exit,
so a slow or dead volume server burned the whole 30s timeout with no
logs. Switch to the same weed-mini approach the fuse_test package
framework already uses: dynamic filer ports, child output captured to
log files, and readiness waits that bail out the moment the process
dies. The external-mount path (TEST_MOUNT_POINT) is unchanged.

The posix-compliance workflow mounts with -allowOthers, but fusermount
refuses allow_other unless user_allow_other is set in /etc/fuse.conf,
mirroring the fuse-integration workflow FUSE setup.
2026-10-23 04:33:14 -07:00
ad89f003a5 s3: end a catalog-vended session with the access key it was vended for (#11945)
* s3: end a catalog-vended session with the access key it was vended for

A table credential the Iceberg catalog vends is a self-contained STS session:
it stays valid until it expires (an hour by default) even after the access key
that loaded the table is deleted or deactivated, or its identity is removed
or disabled. An operator revoking a leaked key cannot cut off the table
credentials that key already obtained.

Sessions are stateless JWTs, so there is no revocation list to consult. This
change follows the same pattern as role sessions, which are already refused
once their role's ID no longer matches (checkSessionRoleBinding): a vended
session carries the access key it was vended for (src_ak, recorded by the
previous change), and validateSTSSessionToken checks it against the identity
table held in memory. A session of provider iceberg-catalog that carries a
source key is refused (InvalidAccessKeyId, as for an unknown key) once its
identity is gone or disabled, or the key is no longer one of that identity's
active, unexpired credentials.

Scope:
- Only catalog-vended sessions are checked; web-identity, AssumeRole and
  every other STS session are validated as before.
- The vendor binds a session to the caller's key only when the caller is a
  stored identity holding that key. A caller that signed with another STS
  session's key, an OIDC user, or a request that recorded no key gets an
  unbound session, as today; binding it would refuse every request made with
  it. A session minted before this change carries no key and is unbound.
- The check is one map lookup under the read lock and a scan of one
  identity's keys (about 11 ns, 0 allocations, with 1,000 identities).

Tests (s3api_iceberg_vended_session_test.go) mint real sessions through the
catalog vendor and validate them the way a data-path request is validated:
deactivating, expiring or deleting key A ends A's session while the session
vended through key B on the same identity keeps working; disabling or
deleting the identity ends it; callers that are not a stored identity holding
the key get unbound sessions that keep working; other STS sessions are not
checked. The first two tests and the third fail without this change.

* s3: keep the vended-session key binding when the key dies before vend

vendedSessionKeyLive gated the binding, so a key revoked between the
request's authentication and the vend skipped it and minted an unbound
session that stayed usable until it expired. Every catalog auth path
pairs the identity name in context with the key that authenticated, so
a stored-identity principal always means the request signed with that
identity's own key. Bind whenever the principal is a stored identity: a
session bound to a now-dead key is refused on its first request, which
is what the binding is for.

* s3: gate the vended-session key binding on the authenticated caller type

lookupByIdentityName(principal) named only that some identity exists, not
that the request signed with a stored key: an STS session or OIDC principal
that collides with an identity name got a session bound to its temporary
key, which vendedSessionKeyLive then refused on every request. Verifiers
now record the verified key as a stored credential, and the vendor binds
only when that marker is present — keeping the binding even when the key
was revoked between authentication and the vend.

* s3: enforce the vended key binding under a dedicated claim

Sessions minted before the binding existed carry the caller's key in the
audit-only SourceAccessKeyId — including temporary STS keys that can never
resolve in the identity table. Enforcing on that field would end live table
credentials at upgrade; the binding now travels under BoundAccessKeyId,
leaving legacy sessions unbound as before.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: bind vended sessions to a credential fingerprint

The key id alone cannot identify a credential: an operator can delete a
key and re-create it under the same id with a different secret, reviving
sessions minted for the original. Verifiers now record a fingerprint of
the secret at authentication; the session dies unless it still matches.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: keep recording the caller's key on every catalog session

SourceAccessKeyId was unconditional audit metadata before the binding
existed; gating it on the stored-credential marker dropped the record
for callers that authenticated another way. With the binding under its
own claim there is no ambiguity left, so the field records the caller's
key again regardless of how it authenticated.

* s3: key the vended-session credential fingerprint with the signing key

The bound-credential fingerprint travels inside the vended session
token. As a plain SHA-256 of the secret it gave any token holder an
offline guessing oracle for weak secrets, and CodeQL flags it twice.

Store the verified secret in the request marker instead and HMAC it
with the STS signing key at vend and at validation. The claim can no
longer be mined, and the per-request auth path stops hashing secrets.

---------

Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 03:58:06 -07:00
Chris LuandDevin bf3081e23a s3 lifecycle: route version-file writes to current-version expiration (#11956)
* s3lifecycle: schedule ExpirationDays from version-file writes

On a versioned bucket a PUT lands in the meta-log as the version file
plus the .versions/ directory update — there is no bare-key event, so
the version-folder skip dropped every version-file event for all kinds
but ExpiredDeleteMarker. ExpirationDays therefore only fired via the
recovery walker's cold-start / rule-change pass: once a shard had a
saved cursor, no daily run ever scheduled another delete marker.

A version-file write is the latest for its logical key at write time,
so route it through the same event-driven evaluation the regular path
runs. The match carries the file's identity with an empty VersionID —
the dispatcher resolves the current latest at due time, and a fresher
PUT meanwhile resolves to STALE_IDENTITY there.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* lifecycle test: versioned Expiration fires on the second daily run

Establishes the shard cursor with a first pass, then PUTs and backdates
a new versioned object and runs again. The delete marker must appear
without a rule change forcing a recovery walk.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: run TestLifecycleVersionedBucketExpiresOnSecondRun in the lifecycle matrix

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 03:19:41 -07:00
Chris LuandDevin 2e7e691bed volume: drop stale index entries that resolve to a different needle (#11948)
* volume: drop index entries that resolve to a different needle

An index row whose offset lands on a needle with a different id and a
different size cannot have been produced by any write path: every appender
indexes a key at the offset and size of the record it wrote, and
WriteNeedleBlob keeps its indexed size consistent with the blob header.
Such rows are corruption (for example an offset replayed with stale high
bytes) and until now made the volume reject every write for the key with
"mismatching cookie" while reads resolved to the wrong needle.

When the duplicate check finds the indexed record belongs to a different
key and size, drop the stale entry and let the append re-index the key.
Same-key conflicts still fail the cookie check, and entries staying
consistent with a re-keyed blob keep their existing semantics.

* volume: verify the indexed needle header before streaming reads

The buffered read path re-parses the record and rejects a needle whose
stored size disagrees with the index, but the streamed path serves data
bytes without ever looking at the header, so a stale index row would
stream another needle's payload. Probe the header once up front and
report a miss when the record is neither the requested key nor the
indexed size, the same staleness test the write path applies.

* rust volume: drop index entries that resolve to a different needle

Mirror of the Go volume change: when the duplicate check finds the
indexed record belongs to a different key and size, delete the stale
entry and let the append re-index the key.

* volume: overwrite a stale index entry on commit, not up front

Deleting the stale row during the duplicate check published a tombstone
before the replacement append and, for durable writes, before its fsync.
A failed append or sync then rolled back the data but left the key
deleted. The publish step already overwrites any older offset, so leave
the entry in place and let it be replaced once the record commits.

* volume: report streaming header-probe failures to the io error tracker

A failed header read returned before the tracked body-read loop, so
repeated EIO on the probe never reached checkReadWriteError. Skip io.EOF
the way the loop does: it is a truncated record, not a disk fault.

* rust volume: overwrite a stale index entry on commit, not up front

Mirror of the Go volume change: the staged delete published a tombstone
before the replacement append and its .dat flush, so a failed append or
sync left the key deleted. publish_write already overwrites any older
offset once the record commits.

* rust volume: report needle meta read failures to the io error tracker

read_meta is the header probe ahead of a streamed body: an EIO there
escaped check_read_write_error entirely, so a failing disk could not
trip the tolerance. Report the outcome the way read_full already does.

* volume: retry the wrapped offset before judging an index entry stale

A 4-byte index value can index past the 32 GiB wrap: the low offset may
hold another needle while the real record sits at +MaxPossibleVolumeSize,
which is where reads already retry on a size mismatch. Skipping the
cookie check on the low-offset record alone would accept a foreign-cookie
write over a live key. Retry the wrapped header read first, the same
condition the read path uses; a wrapped read that fails for reasons other
than EOF stops the write like the low-offset read does.

* volume: probe the streamed needle header under the access lock

With HasSlowRead the access lock is dropped after the index lookup, so a
compaction commit could swap the backend between the lookup and the
header probe and fail a valid download on the old offset. Take the read
lock for the probe and resolve the compaction revision first, the way
the streaming loop re-resolves it on every page.

* rust volume: retry the wrapped offset before judging an index entry stale

Mirror of the Go volume change: probe offset + MAX_POSSIBLE_VOLUME_SIZE
under the 4-byte-offset build before treating a mismatched record as
stale, so a live wrapped entry still gets its cookie checked.

* rust volume: do not clear the io error streak on a successful meta read

read_meta runs before the payload read in read_needle_for_get, so recording success there reset the streak before every failing download and the volume could never reach quarantine. Report failures only; read_full still records success after a complete read.

* volume: record a meta-only read's success against the io error streak

A meta read no longer clears the streak because a payload read still
follows on GETs, but requests the meta read ends outright — HEAD answers
and cookie mismatches — never reached the body path, so their success
left earlier failures counted. Note the success on those terminal paths;
mirrored in the Rust volume server.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: report streamed reads to the io error streak

Streamed GETs never touched the tracker: a failing chunk read went
uncounted and a completed download never cleared the streak, unlike
readNeedleDataInto which does both.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: reset the io error streak only at terminal meta reads

A streamed GET's metadata read runs before its payload read, so clearing
the streak there erased the count before every failing download and the
volume could never reach quarantine. Record the meta-read success only
where the request actually ends: must-meta-only reads, HEAD responses,
and cookie mismatches — matching the Rust volume server's placement.

* rust volume: record the streamed read's success at the final frame

Hyper can complete a Content-Length response without polling the body
again, so a success recorded only when the stream is next polled could
be skipped, leaving an earlier io error streak in place after a
successful download.

* volume: record a meta-only read's success on conditional 304 replies

* rust volume: record a meta-only read's success on conditional 304 replies

* volume: reset the io error streak only after the checksum passes

A streamed read that ends in a CRC mismatch still cleared the EIO
streak, so corruption could mask progress toward quarantine. The Rust
side records success only after the final frame's checksum verifies;
move the reset below the CRC check to match.

* volume: satisfy codespell in the meta-read comment

* rust volume: report range reads to the io error streak

* rust volume: cover the range read's io error reporting

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 02:30:53 -07:00
Chris LuandDevin 27fa285d48 filer: do not stall startup waiting out the leader-settle window (#11972)
* filer: do not stall startup waiting out the leader-settle window

ListExistingPeerUpdates re-polled the leader for up to 10s when it
reported zero filers. A fresh single-filer cluster can never gain peers,
so every filer start paid the full settle timeout before listening —
past the 6s health window in weed mini and short probe budgets.

Take the leader's first answer instead; the fallback fullest-answer
scan below still covers a leader that is mid-convergence.

* filer: bound leader-settle re-poll below the startup health window

* filer: update ListExistingPeerUpdates doc for empty-leader fallback

* filer: re-check empty membership at the announce gate before registering

A fresh filer that scans empty membership during a master restart or
re-election skipped bootstrap and announced immediately, serving missing
files until a peer happened to reconnect — and a late peer only got an
incremental subscription, so pre-existing metadata was never backfilled.

The announce gate now re-polls ListExistingPeerUpdates while the scan can
still be disproved (10s), backfills from any late peers before their
streams start, and then waits for their catch-up like startup peers. A
genuinely empty cluster pays a delayed master listing rather than a
stalled startup, since this runs after the ports already serve.

* filer: overlap the late-peer copy cutoff and stream start

A peer event stamped just behind this filer's clock could land after
the traversal read but carry a timestamp below the stream start —
the startup scan margins both cutoffs by LogFlushInterval for exactly
this skew; mirror it for peers discovered at the announce gate.

* filer: bound the late-peer bootstrap copy by the shared announce deadline

MaybeBootstrapFromOnePeer now takes the caller's context so a stalled
peer metadata copy cannot out-wait the announce cap, and the bootstrap
re-check shares one announceCatchUpCap deadline with the catch-up wait
instead of each spending a full minute.

* filer: merge bootstrap entries by commit time instead of blind insert

The late-peer copy runs while gRPC already serves, so a blind InsertEntry
could replace a write the local filer just accepted. Routing each entry
through applyTraversedEntry orders it against the local commit time under
the replay locks — a fresher local write survives the backfill.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 02:04:27 -07:00
Chris LuandDevin b65832bdc9 iam/oidc: reject reused jti claims with a bounded replay cache (#11936)
* iam/oidc: reject reused jti claims with a bounded replay cache

* iam/sts: pass jti replay settings through convertToOIDCConfig

* iam/oidc: keep consumed jtis and stop the cleanup worker across re-init

Re-initializing a provider reset the JTI store, so an already-used token
could be presented again after a config refresh; the store is now
allocated once and survives Initialize. The cleanup worker is stopped on
every re-init (previously only when protection stayed enabled), is
started only after the rest of Initialize has succeeded, and Shutdown no
longer consumes a permanent sync.Once, so a re-initialized worker can be
stopped again.

* iam/oidc: enforce the jti replay cache bound for every token

Tokens without a usable jti claim skipped the cache entirely and stayed
reusable; they are now keyed by the token's own fingerprint so a repeat
presentation is still a replay. Tokens without exp previously failed as
expired on first use; their entry now gets a bounded retention. The
replay test-and-set now runs before any capacity work so replayed tokens
cannot each trigger a full-cache scan, the entry count is incremented as
part of the insert so concurrent validations cannot overshoot the bound,
and a guaranteed serialized sweep runs before a capacity rejection while
opportunistic sweeps are rate limited and confined to one at a time. Raw
token identifiers are no longer written to logs or errors, a stable
fingerprint is logged instead.

* iam/sts: shut down providers dropped by the managed-provider swap

Each managed OIDC provider runs a background JTI cleanup worker, but
SetIAMManagedOIDCProviders replaced the provider map without stopping
the old providers, so every refresh leaked goroutines. Providers absent
from the new map are now shut down; providers that stay keep running.

* iam/integration: reuse providers across refreshes for unchanged records

Every refresh rebuilt providers from their records, so a refreshed
provider forgot the JTIs it had already consumed and an already-used
token could be presented again. Records are now fingerprinted and an
unchanged record reuses its provider, keeping the consumed-token cache;
changed records are rebuilt, removed records are dropped, and the swap
in STS releases the workers of whatever is no longer referenced.

* iam/oidc: consume replay entries only on the single-use exchange path

Bearer authentication validates the same token on every request, so
consuming it in shared ValidateToken/Authenticate made every token
single-use and rejected all but the first request. The replay check now
runs only for the STS exchange via AuthenticateSingleUse; bearer paths
validate without consuming. A token without exp also no longer gets a
bounded replay entry that ages out while the token stays valid: its
entry is kept until the capacity bound evicts it, so it can never become
usable again.

* iam/oidc: keep the mock provider on the mock path for single-use calls

* iam/oidc: require exp on single-use tokens

A cache slot frees only by expiry, so an expiry-less entry would hold it
forever; a burst of distinct tokens could exhaust the bound permanently
and block every later exchange.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: give each web-identity test token a unique jti

Subtests minted tokens with identical claims inside one second, so the
exchange saw the same token fingerprint twice and the second call read
as a replay. Each exchange now carries its own jti.

* iam/oidc: carry the consumed-token replay store across provider rebuilds

A record change rebuilds the managed provider with a fresh JTI store, so a token consumed before the rebuild could be presented again until it expired. The new provider adopts the replaced provider's replay store before it is published, and the replaced provider is shut down so only one cleanup worker sweeps the shared store.

* iam/oidc: stop the cleanup worker of providers dropped from the store

A refresh removed the provider object from the managed cache but left its JTI cleanup goroutine running for the life of the process.

* iam/oidc: merge a provider's consumed replay entries into another's store

MergeReplayStateFrom folds one provider's consumed-token entries into
another's, skipping already-expired ones. Unlike AdoptReplayStateFrom it
swaps no pointers, so it is safe on a provider still serving validators.

* iam/integration: share one provider and replay store across identical issuer records

Records for the same issuer URL with identical validation settings now
reuse a single provider, so a token consumed through one account's
record is rejected through the other's — previously each record owned a
separate jti store and a spent token could be replayed cross-account.

A superseded provider's consumed entries merge into the shared store so
no rejection is lost, and a removed or rebuilt provider is only shut
down once no remaining record references it.

* iam/oidc: hand the superseded provider's live replay store to the shared one

* iam/oidc: carry merged replay entries without charging the capacity bound

* iam/oidc: key the jti-less replay fingerprint on the signed input

Hashing the whole token lets ECDSA signature malleability (s -> N-s)
mint a second cache key for identical claims, so a consumed token could
be replayed for another exchange. Key the fallback on header.payload
instead.

* iam/oidc: publish the jti cache bound before the enabled flag

Initialize stored jtiEnabled ahead of jtiMaxSize, so a provider flipped
from disabled to enabled exposed enabled=true with a zero bound for a
window; a single-use validation landing there would compute
newCount > maxSize and reject with "JTI cache capacity exceeded". Store
the bound first so the flag is the last thing published.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 00:46:14 -07:00
Chris LuandDevin 186bc5cf7f s3: bound ListMultipartUploads memory to the page size (#11971)
* s3: bound ListMultipartUploads memory to the page size

listMultipartUploads loaded every open upload in the bucket into a slice
and selectMultipartUploads filtered and sorted a second full copy, so one
request cost O(total open uploads) heap regardless of prefix, key-marker,
or max-uploads; concurrent calls multiplied it and could OOM the gateway.

The response is ordered by the object key kept in extended attributes, not
the upload id the filer sorts by, so every entry still has to be seen.
Streaming the listing into a bounded top-K collector keeps at most
maxUploads (capped at maxUploadsList) candidates, bounding heap by the
page instead of the folder. A caller-supplied max-uploads above the
server cap now also paginates rather than accumulating unboundedly.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: cover bounded ListMultipartUploads accumulation

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: report the effective max-uploads cap in ListMultipartUploads

* s3: cover ListMultipartUploads retry through the collector

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 00:46:10 -07:00
Chris LuandDevin 7154732b84 worker/ec: retry the pinned volume copy after a mid-copy compaction (#11929)
* worker/ec: retry the pinned volume copy after a mid-copy compaction

The EC task pins .idx/.dat copies to the CompactionRevision read from
ReadVolumeFileStatus so the pair comes from one volume generation. When
vacuum compaction lands between that read and either CopyFile, the
server rejects the copy with "volume N is compacted" and the whole task
fails — on clusters with queued maintenance this makes most EC tasks
die to a race they cannot avoid. On a compaction mismatch the copy now
restarts under a freshly read revision, up to a bounded number of
attempts; other errors still fail immediately.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* worker/ec: read the retried copy's expiry from its own file status

A mid-copy compaction makes the source reload the volume, and the fresh
ReadVolumeFileStatus can carry a different expireAtSec than the
pre-compaction read. Keeping the attempt-0 expiry could let the EC
volume outlive its source TTL.

* worker/ec: cover the compaction retry with a stub volume server

* worker/ec: also observe revised .idx bytes in the compaction retry test

* worker/ec: read the fake source under its mutex in the retry test

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-22 21:32:57 -07:00
Chris LuandDevin f2d5c27f65 operation: renew the write JWT when a fan-out rollback delete is rejected (#11970)
* operation: renew the cleanup JWT when a fan-out rollback delete is rejected

A stalled holder can out-wait the assignment JWT's 10s expiry, so the
type=replicate delete that rolls back landed copies comes back 401 and the
needle leaks. Retry the delete once with a fresh token minted through the new
LookupJwtFunc hook when a holder answers unauthorized.

* s3api: wire a cleanup-JWT renewer into chunked uploads

Keep the master client on the server and use it to mint fresh volume JWTs for
rollback deletes that run past the assignment token's expiry.

* admin: wire a cleanup-JWT renewer into file-browser uploads

* operation: bound the cleanup-JWT renewal with a per-attempt deadline

A rollback delete that hits 401 renewed the token through a background
context, so a master outage could park the wait for a connection and the
lookup RPC inside the failed upload's cleanup path. Thread a context
through LookupJwtFunc and LookupJwt, and bound each renewal with a 10s
deadline covering both the master wait and the lookup call.

* operation: let every holder's 401 retry with a renewed cleanup JWT

A single renewed flag meant only the first rejecting holder refreshed the
token; a holder that stalled past the renewed JWT's lifetime left later
copies stranded with a 401 they could not resolve. The retry limit is now
per holder — each delete is attempted once with the current token and once
more after a bounded renewal — so a slow middle holder cannot strand the
rest.

* operation: log the volume server's error body on a failed cleanup delete

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* operation: count test upload attempts atomically

The chunk upload loop fans AssignFunc out to parallel goroutines, so the
plain int counter in the partial-results test raced under -race.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-22 21:32:53 -07:00
github-actions[bot] b97ed0f101 4.49 2026-10-23 03:49:00 +00:00
github-actions[bot] 13cbe78537 docs: regenerate star history chart 2026-10-23 02:28:25 +00:00
Mohd Quamar Tyagi 94dd96a322 weed filer.remote.sync: ignore .system directory under buckets (#11966) (#11969) 2026-10-22 18:40:30 -07:00
Chris Lu 00e64441c6 fix: clone protobuf snapshots without copying internal locks (#11968)
* fix: clone protobuf snapshots without copying internal locks

* fix: reuse cloned snapshot fields when filtering
2026-10-22 18:38:33 -07:00
Chris Lu f08f075d23 deps: patch two open dependabot alerts (klauspost/compress, aws-smithy-json) (#11959)
* test/kafka: bump klauspost/compress to v1.20.3 in simple-publisher

Patch an s2 integer-overflow / out-of-bounds write affecting
klauspost/compress <= 1.18.6; matches the version the main module
already uses. The go directive rises to 1.25, still under the ^1.26
toolchain the kafka test workflows run with.

* seaweed-volume: bump aws-smithy-json to 0.62.7

Patch an uncontrolled-recursion denial of service in the unknown-key
skip path of aws-smithy-json <= 0.62.6 (reachable over the S3 SDK's
response parsing). The smithy runtime family moves together:
aws-smithy-runtime-api 1.12.3 and aws-smithy-types 1.5.0, all selected
within the workspace's Rust 1.91.1 compatibility bound.
2026-10-22 14:38:16 -07:00
Chris LuandDevin 7c980774e7 FUSE mount: add posix compliance reports (#7183)
* fuse integration: add POSIX compliance test suite

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: add posix-compliance workflow

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fuse integration: skip non-executable weed path candidates

* fuse integration: size readv buffer to the written segment

* fuse integration: reference posix_Makefile explicitly in recursive make calls

GNU make does not propagate -f through MAKEFLAGS, so bare $(MAKE) re-read the default Makefile in this directory.

* fuse integration: fail posix test targets when go test fails

Every suite pipes go test through tee under /bin/sh, so the recipe status was tee's and test failures stayed green. Run recipes under bash -o pipefail and emit the color escapes with printf since bash echo does not expand them.

* fuse integration: run only benchmarks in benchmark-posix

go test -bench=. without -run also executes the whole suite, mixing compliance output into the benchmark log.

* fuse integration: emit single-number test counts in reports

grep -c prints 0 and exits nonzero when nothing matches, so '|| echo 0' produced two lines that broke the expr arithmetic and JSON fields. awk prints exactly one count either way.

* fuse integration: skip the sendfile test on macOS

macOS sendfile(2) only accepts a socket destination; the test copies into a regular file and always gets ENOTSOCK.

* fuse integration: honor TestConfig and own only self-started resources

NewFuseTestFramework dropped its config argument and Cleanup read a fresh
DefaultTestConfig, so SkipCleanup never did anything. Cleanup also skipped
teardown whenever TEST_SKIP_CLUSTER_SETUP was set, even when TEST_MOUNT_POINT
was empty and Setup had started the cluster itself. The framework now keeps
the config it was built with, runs external-mount tests inside a per-run work
dir under the mount, and only skips teardown when an external mount was
actually used.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fuse integration: pass the debug flag before the subcommand

-v is a weed global flag; appended after master/volume/filer/mount it lands
on the subcommand flagset, which rejects it and the process dies at flag
parsing whenever EnableDebug is on.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fuse integration: require a real mount before running tests

Setup creates the mount point before weed mount starts, so statting and
listing it succeeded on the plain directory and tests ran against local
disk whenever the mount was slow or died. waitForMount now waits until the
mount point's device id differs from its parent's.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fuse integration: resolve the weed binary to an absolute path

The started processes run with cmd.Dir at the framework's temp dir, so a
relative candidate like ../weed no longer resolves. Prefer PATH, then probe
the usual build outputs and return absolute paths only.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fuse integration: build the external test work dir with MkdirTemp

The suite staged cloned tools and generated scripts under a predictable
/tmp path another user could pre-create and swap mid-run.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fuse integration: use the pjdfstest tree setup-pjdfstest installs

The test gated on a pjdfstest binary in PATH but then proved a suite it
cloned itself, so a tree installed by make setup-pjdfstest was never found.
Look for the built suite under external_tools first, then clone into the
work dir as before.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: require a real mount before posix tests

The wait loop fell back to listing the mount point, which passes on the
directory created before weed mount runs — a dead or slow mount meant the
suite exercised local disk. Only mountpoint counts now, and a dead mount
process fails fast instead of timing out.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: collect failure logs before cleanup deletes them

The always-on cleanup step removed /tmp/seaweedfs before the failure-only
debug step could copy the service logs, so failed runs uploaded nothing.
Run cleanup after log collection.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fuse integration: point MOUNT_POINT at TEST_MOUNT_POINT

Targets that work against MOUNT_POINT (pjdfstest, nfstest, fio configs)
ignored TEST_MOUNT_POINT and ran against the local-disk default while the
go tests used the external mount.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fuse integration: skip the permission-denied check as root

Root bypasses mode-bit checks, so writing the read-only file succeeded and
the assertion failed in root-run environments like containers.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fuse integration: align the posix docs with the suite

The doc listed a docker-test-posix target and make variables that do not
exist, and ran go test against single files that cannot compile alone. The
roadmap also still claimed the platform-specific tests this suite now
implements were all skipped.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fuse integration: honor SkipCleanup in the mini-based framework too

Same gap as the fuse-package framework: Cleanup read a fresh
DefaultTestConfig, so a test's SkipCleanup setting was ignored.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fuse integration: normalize config in Setup and clean the work dir too

Setup(nil) panicked on config.EnableDebug and external-mount Cleanup left
the temp dir behind; use the stored config for the start* calls and remove
tempDir alongside the external work dir.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: point external posix suites at the workflow FUSE mount

* fuse integration: create the fio target directory before running

* fuse integration: use unix.O_DIRECT instead of a hard-coded x86 value

0x4000 is O_DIRECT only on x86 and amd64; on arm64 it is O_DIRECTORY.
The test module already depends on golang.org/x/sys.

* fuse integration: run autoreconf and configure before building pjdfstest

A fresh pjdfstest clone is autotools-based and has no Makefile, so the
Go-side setup fallback always failed and the suite silently skipped.

* ci: require external posix tools only when setup installed them

setup-external-tools always exits 0 because each sub-target tolerates
failure, so POSIX_REQUIRE_EXTERNAL_TOOLS was set even with no tools
installed and the suite then hard-failed on the missing suites.

* ci: fail the posix workflow when the mount point is not a FUSE mount

The file write/read check passes on the plain directory created before
weed mount, so a dead mount let the whole suite grade the local disk.

* fuse integration: skip when the host has no FUSE device node

Setup failed the test outright on machines without /dev/fuse instead of
skipping; cover every suite entry point by skipping inside Setup.

* fuse integration: drop the chmod 666 /dev/fuse troubleshooting tip

World-writable device access is a security risk; the fuse group
membership on the line below is the safe fix.

* fuse integration: require go 1.21+ in check-go, not 1.20+

* fuse integration: refuse external suites on an unmounted path

* fuse integration: verify TEST_MOUNT_POINT is a real mount

* fuse integration: clean TEST_MOUNT_POINT before the parent-device check

filepath.Dir of a trailing-slash path returns the mount itself, so a mounted TEST_MOUNT_POINT ending in / was refused.

* fuse integration: detect WinFsp mounts on Windows in mountedAt

A directory mount is a reparse point and a drive mount is a volume root; a plain directory is now refused instead of accepted.

* fuse integration: strip a trailing slash off MOUNT_POINT in require-mounted

The mount-table grep fallback on hosts without mountpoint(8) would not match a slash-suffixed entry.

* fuse integration: require an external mount to be a FUSE filesystem

A device-ID boundary alone does not prove FUSE: a tmpfs mount would pass mountedAt and report its results as FUSE compliance. Check the filesystem type (statfs magic on linux, fstypename on the BSDs); platforms without a check keep the mount and write verification.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-22 14:37:03 -07:00
Chris Lu 8f6121694a test: fix HEAD bucket assertions and FUSE failover startup timing (#11967)
test: fix HEAD bucket assertions and warm up failover clusters
2026-10-22 14:34:52 -07:00
Chris Lu 137d29633b s3: fix prefix filtering and delimiter pagination (#11962) 2026-10-22 09:31:20 -07:00
github-actions[bot] ca0314ea4f docs: regenerate star history chart 2026-10-22 12:49:43 +00:00
Chris LuandJavier de la Torre 82f0a82564 filer+volume: make acknowledged writes durable under the fsync rule (#11943)
* volume: never compact a volume tiered to remote storage

A tiered volume's data backend is the remote object, but compaction reads
the local .dat path. When tiering removed the local .dat, every vacuum of a
tiered volume with garbage fails with "no such file or directory", and the
master's sweep and the worker's vacuum task keep selecting it. When the
local .dat was kept, the compaction succeeds against the stale copy and the
commit reloads the volume to read the remote object (old layout) through the
compacted index, so live needles stop reading.

Mirror the Rust volume server (#11545):
- CompactByIndex and CompactByVolumeData refuse a volume with a remote file
  before creating .cpd/.cpx ("volume N is tiered to remote storage, cannot
  compact").
- CommitCompact refuses under the data-file locks when the volume became
  tiered during the copy, and removes .cpd/.cpx.
- Store.CheckCompactVolume reports no garbage for a tiered volume, so the
  master's vacuum and volume.vacuum do not select it.
- The worker's vacuum detection skips a volume with a remote copy, as balance
  and EC detection already do.

Tests (each fails on master): TestCompactRefusesTieredVolume,
TestCommitCompactRefusesVolumeTieredDuringCopy (on master the commit
succeeds and a surviving needle no longer reads),
TestPickPrimaryReplicaSkipsTieredVolumes.

* volume: skip tiered replicas in vacuum selection, keep their real ratio

Reporting a zero garbage ratio for a tiered volume hid it from
volume.tier.compact, whose preflight reads that ratio while the volume is
still remote, and did not exclude it from a vacuum with a zero threshold.
Report the real ratio again, as the Rust volume server does, and have the
master's vacuum check skip a replica whose volume info names remote
storage, whatever the threshold or request.

When CommitCompact refuses a volume tiered during the copy, also close the
temporary needle map and remove .cpldb: nothing else would, since no later
vacuum selects the volume.

The tiered-compaction test no longer depends on random needle sizes
reaching a 0.3 garbage ratio.

* volume tier: do not attach a copy of a .dat a compaction replaced

The upload copies the .dat, then records the remote file in the .vif and
switches the backend. A compaction commit in between swaps in a compacted
.dat and index; the upload then served the old-layout remote object through
the new index, so live needles stopped reading. CommitCompact's tiered
check cannot see this: the volume is not tiered until the switch.

AttachRemoteDatFile records the remote file and switches the backend under
the data-file locks, and refuses when the backend is no longer the file
that was copied. The upload then removes the unused remote copy.

* volume tier: keep a committed .vif entry when only the dir fsync fails

AttachRemoteDatFile rolled back the in-memory remote entry on any
SaveVolumeInfo error. A NotCrashDurableError comes after the rename, so
the .vif on disk already names the copy; with the entry gone, the upload
handler deleted that object, and a reload would serve from a missing
remote file. Keep the entry and switch to the copy as the .vif says,
and still return the error so the caller keeps the local .dat.

* vacuum worker: filter tiered replicas instead of skipping the volume

A partially tiered volume is the leftover of an interrupted
volume.tier.upload; skipping the whole volume left garbage on its local
replicas unreclaimed. Per-replica filtering matches the master's
batchVacuumVolumeCheck and is safe: a tiered replica refuses both
compaction and commit.

* filer: sync leveldb entry writes when the path rule asks for fsync

The fsync storage rule only reached the volume upload; the metadata
writes behind an acknowledged create, update or delete still went to
the leveldb journal with Sync:false, so a power loss could roll back
an acknowledged bucket or object entry.

A write-side flag on the entry context carries the request: the filer
sets it from the path's storage rule in CreateEntry, UpdateEntry,
EnsureDirectoryEntry and DeleteEntryMetaAndData, and the HTTP write
path sets it for an explicit ?fsync=true. The leveldb stores write
with Sync:true when the flag is present.

* volume: flush the .idx before acknowledging a durable write

A write with fsync synced only the .dat: the index row written on
publish stayed in the page cache, so a power loss could leave the
needle's bytes durable with nothing indexing them.

The durable paths now run one .idx sync after publish, before the
requests complete — one per syncWrite and one per batch run, paired
with the .dat sync, the same split the Rust volume server makes
between flush_dat and flush_idx. A failed .idx sync cannot be rolled
back, so it stops the volume taking writes and fails the requests
instead of acknowledging an index entry that may not survive.

---------

Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
2026-10-21 23:57:09 -07:00
7603528aff s3api: support s3:RequestObjectTag/<key> in bucket policies (#11910)
* s3api: support s3:RequestObjectTag/<key> in bucket policies

The tags come from X-Amz-Tagging only where the handler stores them from
there: not on PutObjectTagging, which reads the body, and not on a copy
that keeps the source's tags. For a repeated key the last value counts,
since that is what CopyObject stores.

* s3api: extract RequestObjectTag only where the handler stores it

The header was honored on any request carrying it, so a tag attached to
a GET, DELETE, part upload, multipart completion, directory marker or a
PUT subresource satisfied a tag-conditioned policy the object never
receives. Extraction now follows the handlers: PutObject, a copy under
X-Amz-Tagging-Directive: REPLACE, and CreateMultipartUpload store the
header; a plain object PUT is identified by carrying only X-Amz-*
signing parameters.

* s3api: exclude multipart part PUTs before the copy-source check

A PUT carrying X-Amz-Copy-Source plus a numeric partNumber and uploadId
routes to CopyObjectPartHandler, which stores a part and never reads
X-Amz-Tagging. Checking the copy source first let such a request satisfy
an s3:RequestObjectTag condition under REPLACE with tags that are never
stored. Mirror the router: the part handlers match only when both
partNumber (digits only) and uploadId are present, so a lone or
non-numeric partNumber still falls through to the storing handlers.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3api: gate tag extraction on the routed subresource, not query key shape

Rejecting every PUT whose query held a key outside X-Amz- dropped
X-Amz-Tagging from ordinary object PUTs that carry arbitrary parameters —
Signature V2 presigns (AWSAccessKeyId/Signature/Expires) among them — so a
tag-conditioned Deny could be bypassed by appending an unrelated query
parameter while putToFiler still stored the tags. Exclude the object PUT
subresources the router actually dispatches (tagging, renameObject, acl,
retention, legal-hold) ahead of the copy-source check instead.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3api: cover the routed-subresource and part-copy tag extraction cases

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Chris Lu <chris.lu@gmail.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 23:55:01 -07:00
Chris LuandDevin cec96c02ea s3: reject requests carrying more than one routed subresource (#11942)
* s3: reject requests carrying more than one routed subresource

Routing picks the first matching subresource route while authorization
resolves a single action from the query, and the two can disagree. A
bucket-level PUT ?tagging&policy is authorized as s3:PutBucketTagging but
served by PutBucketPolicyHandler, letting a tagger rewrite the bucket
policy; the same shape lets a list grant read the policy via
?list-type=2&policy or ?versions&policy, and lets an object ACL writer
retag via PUT ?acl&tagging. Declining ambiguous requests keeps the
authorized action and the served handler identical.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: reuse hasAmbiguousSubresource in the routed-subresource guard

The standalone key list dropped versioning, so ?versioning&policy still
reached route matching and auth with two operations. The bucket-level
check already keys off operationSubresources plus bucketQueryActions;
running the same predicate keeps the lists from drifting apart.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 22:45:53 -07:00
Junker der Provinzandzouyixiong ee2718b14c filer: reduce forwarded lock log noise (#11933)
Forwarding a lock to its owner filer is routine in a multi-filer setup,
so the forward and acquire messages move from V(0) to V(3). The acquire
message is only logged when the forwarded response holds a renew token
and no application error. Transport failures still log at V(0).

Co-authored-by: zouyixiong <3390062+zouyixiong@users.noreply.github.com>
2026-10-21 20:46:23 -07:00
Chris LuandDevin 7ff77a4f3b s3api: skip error response body when http forbids it (#11931)
WriteResponse wrote the payload even for HEAD requests and statuses
that disallow a body (1xx, 204, 304), which made net/http return
ErrBodyNotAllowed and logged 'write err: http: request method or
response status code does not allow body' for ordinary requests.

Generated with [Devin](https://devin.ai)

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 20:43:32 -07:00
Chris LuandDevin 0b197867a1 filer leveldb2: tunable cache, buffer, block and table sizes (#11927)
* filer leveldb2: read cache and buffer sizes from filer.toml

The leveldb2 store hardcoded its block cache (32MiB) and write buffer
(16MiB) and offered no way to tune the table block size or compaction
table size. Optional block_cache_size_mb, write_buffer_size_mb,
block_size_kb and compaction_table_size_mb keys under [leveldb2] now
override them; zero or negative values keep the previous defaults.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer leveldb2: document tuning options in scaffold filer.toml

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer leveldb2: saturate size options instead of overflowing int

The option fields take int, so a MiB config value past math.MaxInt/MiB
(e.g. 2048 on 32-bit) wrapped negative and goleveldb would panic sizing
the memtable. Clamp the product instead of the input.

* filer leveldb2: reject size options too large for int bytes

Clamping a MiB setting at math.MaxInt silently hands goleveldb a
different size than configured; fail Initialize on values that cannot
fit instead.

* filer leveldb3: same tunable cache, buffer, block and table sizes

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer leveldb2/3: exercise oversized-option rejection on 64-bit too

The tests skipped whenever int was 64-bit, but write_buffer_size_mb
values above MaxInt/1MiB still overflow there, so the rejection path
had no coverage on the primary platform. Use MaxInt/1MiB + 1 so the
rejection triggers on every platform.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 20:39:06 -07:00
Chris LuandDevin 1451096c17 volume.fsck: delete missing filer entries through a worker pool (#11924)
* volume.fsck: delete missing filer entries through a worker pool

-reallyDeleteFilerEntries issued one synchronous HTTP DELETE per orphan
path inside the BFS consumer, so a million-entry purge ran one request
at a time. Queue the confirmed paths onto a bounded pool (default 16,
-tunable via -purgeConcurrency) so deletes run in parallel while the
traversal continues.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume.fsck: serialize command output across purge workers

* volume.fsck: serialize purges that share a hard link id

* volume.fsck: cover the purge worker pool

* volume.fsck: take the purge shard remainder in unsigned arithmetic

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 20:37:57 -07:00
Junker der Provinz 4854c6a0f4 s3: skip the local filer socket on Windows (#11921)
The filer only listens on its unix socket when GOOS is not windows, so
with a filer socket path configured on Windows the S3 server dialed a
socket that never exists.

Fixes #7278
2026-10-21 20:35:19 -07:00
74da88fc29 s3, iceberg: record the access key a request was signed with (#11901)
* s3, iceberg: record the access key a request was signed with

Several access keys can belong to one identity, but nothing downstream of
authentication says which one signed a request: the audit log carries the
identity name only, and every table credential the Iceberg catalog vends for
that identity gets the same session name. When an identity is shared (one key
per person or per service), the log cannot tell the holders apart.

The verifier that matched a credential now records that credential's access
key id in the request's identity holder (verifyV4Signature for header,
presigned, streaming and STS requests; both SigV2 verifiers). Nothing re-reads
the request for a key, so a second, unverified key elsewhere in the request
(a junk X-Amz-Credential on a SigV2 request, a junk SigV4 header on a SigV2
presigned URL) is never recorded.

- s3err.AccessLog gains access_key_id (omitempty): a static key's id, or an
  STS session's temporary key id. Every other field is unchanged.
- The Iceberg and Lance catalogs install the holder before authenticating a
  SigV4 request; a Bearer token records the key it was minted from (the token
  is verified with a key derived from it); a Lance x-api-key records its key.
- VendTableCredentials ends the session name with "-k" and the first 12 hex
  digits of SHA-256 of the key id (the table part gives way within the
  64-character limit; the tag is never cut), and carries the key id in the
  session token (src_ak, SessionInfo.SourceAccessKeyId).

Tests: auth_access_key_attribution_test.go and auth_access_key_shapes_test.go
(two keys on one identity, the deny path, SigV2 and presigned SigV2 with a
junk key elsewhere, streaming seed, STS session, failed signatures record
nothing), s3api_iceberg_session_name_test.go, iceberg and lance Auth tests,
sts source_access_key_test.go.

* s3: record the access key of a verified POST-policy form and of a retargeted signature

Two verified signatures left access_key_id empty in the audit log:

- A browser POST upload is verified in PostPolicyBucketHandler from its form
  fields by doesPolicySignatureV2Match / doesPolicySignatureV4Match, which did
  not record the key. Both now take the request context and record the
  credential's key once its signature verifies; a form that fails
  verification records nothing.
- verifyV4Signature refuses a signature that verified for one virtual-hosted
  bucket but was routed to another. The key was recorded only after that
  check, so the refusal carried none. It is now recorded as soon as a host
  candidate verifies.

Test: TestPostPolicyUploadRecordsTheSigningKey (SigV4 and SigV2 forms, and a
form with a wrong signature) and TestSignedVirtualHostRetargetRecordsTheSigningKey;
both fail before this change.

* s3: test VendTableCredentials end to end for the caller's access key

TestVendTableCredentialsRecordsTheCallersKey mints sessions through
VendTableCredentials with a configured STS service, the key recorded the way
the catalog's authentication records it, then validates each session token:
SourceAccessKeyId and the session name's key tag match the key, and a caller
without a key gets an untagged session. It fails if the vendor stops passing
either field.

---------

Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 20:33:08 -07:00
Jaehoon KimandChris Lu 744ab586c5 fix: volume.fsck purges live chunks of files renamed during its filer walk (SQL stores) (#11883)
* shell: volume.fsck reads the filer from one store snapshot when it can

volume.fsck collects the filer's chunk references by walking directory
by directory, so each directory is seen at a different time. A file
renamed from a directory not yet listed into one already listed is never
seen, its chunks look like orphans, and -reallyDeleteFromVolume deletes
live data. Checkpoint rotation and write-to-tmp-then-rename hit this.

Add an optional filer store capability, EntryScanner, that visits every
entry below a directory from one consistent snapshot. The SQL stores
implement it with one REPEATABLE READ read-only transaction read in
primary-key pages, so a concurrent rename is seen under exactly one of
its names. Key-value rows are skipped by their dirhash, a bucket
directory whose table is missing fails the scan (from the snapshot it
cannot be told from one dropped a moment ago), and hard links are
resolved in the same transaction. SQLite offers it only in WAL mode, on
its own connection.

TraverseBfsMetadata serves the snapshot when asked via request metadata
sw-traverse-mode=snapshot and reports the mode in a response header;
without the metadata it walks as before, so peer bootstrap and admin
export are unchanged. Snapshot responses carry no entry_ts_ns, whose
per-entry lookup would cost more than the scan.

volume.fsck requests the snapshot and falls back to the walk on an older
filer or a store that cannot scan.

* filer, shell: fail a scan that cannot start, not fall back to the walk

A transient BeginTx failure and a preliminary GetEntry error were treated
like a missing capability or missing root, so fsck would take the
directory walk - the very path that misses renamed-away entries. Only a
genuinely unsupported scan or a confirmed-absent root may fall back.

* filer: only snapshot-listed bucket names keep a missing table fatal

A cached-only name, whose bucket may have been deleted between copying
the cache and opening the snapshot, is skipped when its table is gone.

* filer: roll back to a savepoint before skipping a cached-only table

On PostgreSQL a failed statement aborts the snapshot transaction, so
ignoring a missing-table error without the savepoint still left every
later cached-only table unreadable.

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-10-21 20:31:42 -07:00
364e19a493 remote_storage/s3: use the AWS default credential chain when a source is configured (#11881)
* remote_storage/s3: use the AWS default credential chain when a source is configured

Type s3 forced anonymous credentials whenever no access/secret key was
set, so IRSA, container credentials, profiles and env keys were never
consulted. Keep anonymous (public bucket) access only when the environment
names no AWS credential source.

* remote_storage/s3: also recognize shared-config and default-profile env sources

* remote_storage/s3: resolve env credentials on their own transport

A guarded HTTP client is for the S3 endpoint; the remote credential
providers it would also serve reach link-local addresses that guard must
not dial. Profiles load shared config, and a test resolves env keys.

* remote_storage/s3: clear every detected credential source in the test

* remote_storage/s3: resolve the full default chain on the guarded client

The eager credential session used NewSession, whose chain covers only
env, shared-credentials and ECS/EC2 providers. Web identity and
assume-role profiles resolve through shared config, so IRSA never
worked when a custom HTTP client was pinned. Enable shared config and
carry the configured region so STS can sign assume-role requests.

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 20:30:28 -07:00
df33e81915 volume: report the needle read error with a failed read (#11934)
* volume: report the needle read error with a failed read

A read that fails on the volume server, for example on a CRC mismatch,
returned a bare 500, so mount and filer logs only showed the status.
The volume server now writes the error into the response body, and the
read helpers add it to the error they return.

Co-authored-by: Bruce <82158894+half-life666@users.noreply.github.com>

* volume: keep the 500 bare on HEAD requests

writeJsonError's body write fails with ErrBodyNotAllowed under net/http
when the request method is HEAD, adding an error-writing log line on top
of the failed read. Send only the status, matching the no-error path.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* util/http: decode gzip-encoded error bodies in statusError

Full-chunk reads set Accept-Encoding: gzip themselves, so the
transport does not transparently decompress; a gzip-encoded error
body reached statusError as compressed bytes and produced an
unreadable message. statusError now takes the response and decodes
Content-Encoding before extracting the bounded message.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: log the read error when the 500 stays bare

On HEAD requests the body write is skipped, which also skipped the
V(0) response log inside writeJson — a failed HEAD read left no
cause in normal volume-server logs. Log the error in the same
format before the bare 500.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* util/http: check the status before decoding gzip in GetAuthenticated

The gzip reader eagerly consumes the header when it is created, so
statusError saw a drained or corrupt body on a gzipped error response
and the server message was lost. Match the other read helpers and check
the status first.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* util/http: export StatusError

Other volume read paths outside this file need the same error body.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: redact jwt query param when logging failed-read URLs

The new read-failure logs print the request URL, and volume reads accept
?jwt= as a credential, so a failed authenticated read would write the
bearer token to the volume server log.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* util/http: surface the error body on the remaining chunk-read paths

* volume: redact duplicate jwt params in failed-read logs

* operation: pass the lookup jwt on the chunk needle read

---------

Co-authored-by: Bruce <82158894+half-life666@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-10-21 20:24:48 -07:00
Chris LuandDevin 867a537802 volume.configure: keep the superblock in sync with the .vif (#11930)
* volume.configure: keep the superblock in sync with the .vif

ConfigureVolume only rewrote the .vif's replication field, leaving the
.dat superblock stale. A mounted volume also kept reporting the old
placement in heartbeats, and its next compaction would rewrite the stale
placement into a fresh .dat.

Parse the placement once, then update the live volume when mounted: the
in-memory .vif copy and superblock, and the .dat header through the
volume's own backend under the data-file locks. For unmounted volumes
rewrite the header directly; remote-only volumes skip it since the .vif
carries the placement.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume.configure: update the .dat superblock in the Rust volume server

Mirror the Go ConfigureVolume change: after rewriting the .vif, rewrite
the replica placement in the .dat header too, so a remounted volume's
superblock, heartbeats, and later compactions agree with the .vif. A
missing .dat (remote-only volume) is skipped since the .vif carries the
placement.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 20:18:56 -07:00
Chris LuandDevin 85c1b990b1 shell: bound the initial master connection wait behind -connectTimeout (#11922)
* shell: bound the initial master connection wait behind -connectTimeout

weed shell waited on WaitUntilConnected with a background context, so
an unreachable master hung the command forever. The wait now runs on
a 30s (configurable) timeout and reports the failure instead.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* shell: stop the master reconnect loop when RunShell returns

KeepConnectedToMaster ran on context.Background(), so once RunShell
returned — including the new connect-timeout early return — the
reconnect goroutine kept dialing forever, and repeated RunShell calls
accumulated one each. Give it its own cancellable context, separate
from connectCtx whose deadline must not cut reconnects under a live
shell.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* shell: bound filer discovery by -connectTimeout too

The discovery WithClient and ListClusterNodes ran on the unbounded
background context, so a master disconnect after the initial connect —
or a persistently failing ListClusterNodes — could block startup past
the deadline. Reusing connectCtx bounds the whole startup phase.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* shell: report filer discovery failures instead of silently losing the filer target

* shell: give filer discovery its own budget after master connects

Discovery shared connectCtx, so a connection that landed near the
-connectTimeout deadline left ListClusterNodes almost no time and no room
for WithClient to retry. A fresh timeout keeps discovery bounded without
starving it.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* shell: cover the -connectTimeout master wait with a test

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* shell: note filer discovery in the -connectTimeout help text

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 20:17:51 -07:00
ac319fe318 volume: never compact a volume tiered to remote storage (#11895)
* volume: never compact a volume tiered to remote storage

A tiered volume's data backend is the remote object, but compaction reads
the local .dat path. When tiering removed the local .dat, every vacuum of a
tiered volume with garbage fails with "no such file or directory", and the
master's sweep and the worker's vacuum task keep selecting it. When the
local .dat was kept, the compaction succeeds against the stale copy and the
commit reloads the volume to read the remote object (old layout) through the
compacted index, so live needles stop reading.

Mirror the Rust volume server (#11545):
- CompactByIndex and CompactByVolumeData refuse a volume with a remote file
  before creating .cpd/.cpx ("volume N is tiered to remote storage, cannot
  compact").
- CommitCompact refuses under the data-file locks when the volume became
  tiered during the copy, and removes .cpd/.cpx.
- Store.CheckCompactVolume reports no garbage for a tiered volume, so the
  master's vacuum and volume.vacuum do not select it.
- The worker's vacuum detection skips a volume with a remote copy, as balance
  and EC detection already do.

Tests (each fails on master): TestCompactRefusesTieredVolume,
TestCommitCompactRefusesVolumeTieredDuringCopy (on master the commit
succeeds and a surviving needle no longer reads),
TestPickPrimaryReplicaSkipsTieredVolumes.

* volume: skip tiered replicas in vacuum selection, keep their real ratio

Reporting a zero garbage ratio for a tiered volume hid it from
volume.tier.compact, whose preflight reads that ratio while the volume is
still remote, and did not exclude it from a vacuum with a zero threshold.
Report the real ratio again, as the Rust volume server does, and have the
master's vacuum check skip a replica whose volume info names remote
storage, whatever the threshold or request.

When CommitCompact refuses a volume tiered during the copy, also close the
temporary needle map and remove .cpldb: nothing else would, since no later
vacuum selects the volume.

The tiered-compaction test no longer depends on random needle sizes
reaching a 0.3 garbage ratio.

* volume tier: do not attach a copy of a .dat a compaction replaced

The upload copies the .dat, then records the remote file in the .vif and
switches the backend. A compaction commit in between swaps in a compacted
.dat and index; the upload then served the old-layout remote object through
the new index, so live needles stopped reading. CommitCompact's tiered
check cannot see this: the volume is not tiered until the switch.

AttachRemoteDatFile records the remote file and switches the backend under
the data-file locks, and refuses when the backend is no longer the file
that was copied. The upload then removes the unused remote copy.

* volume tier: keep a committed .vif entry when only the dir fsync fails

AttachRemoteDatFile rolled back the in-memory remote entry on any
SaveVolumeInfo error. A NotCrashDurableError comes after the rename, so
the .vif on disk already names the copy; with the entry gone, the upload
handler deleted that object, and a reload would serve from a missing
remote file. Keep the entry and switch to the copy as the .vif says,
and still return the error so the caller keeps the local .dat.

* vacuum worker: filter tiered replicas instead of skipping the volume

A partially tiered volume is the leftover of an interrupted
volume.tier.upload; skipping the whole volume left garbage on its local
replicas unreclaimed. Per-replica filtering matches the master's
batchVacuumVolumeCheck and is safe: a tiered replica refuses both
compaction and commit.

---------

Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-10-21 20:16:02 -07:00
Chris LuandDevin 164a7d8b56 upload: upload -dir files concurrently behind a new -c flag (#11918)
* upload: upload -dir files concurrently behind a new -c flag

Each file under -dir was submitted serially inside filepath.Walk, so
every small file paid a sequential master-assign plus upload round
trip. A bounded pool now runs those submissions concurrently while
keeping results in walk order; failures are reported per file and the
command exits non-zero when any upload failed.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* submit: close opened readers when a batch assign fails

SubmitFiles returns early when the assign RPC fails, before
FilePart.Upload gets a chance to close each opened file. Close the
readers on that path so repeated assign failures do not leak handles.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* upload: stream -dir results as each file completes

Batching the output until every upload finished meant an interrupted
run lost the file ids of completed uploads. A per-index ready channel
lets a printer emit each result in walk order as soon as it lands.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* submit: close the opened file when Stat fails

newFilePart returned the still-open handle with the error, and the
concurrent upload path records the error without ever reaching
SubmitFiles, leaking one descriptor per failing file.

* upload: still upload files collected before a walk error

A walk that aborts partway used to leave the already-uploaded files
behind; collecting paths first must not drop them, so upload what was
gathered and still exit non-zero.

* upload: submit batched files concurrently

SubmitFiles fetched all file ids in one assign RPC but then uploaded
the files one after another, so a multi-file list still spent its time
in serial per-file round trips even with -dir concurrency. Uploads now
run through a small worker pool while results keep their input order.

Verified end to end: a 5-file list uploads with consecutive file keys
and each stored needle matches its source bytes.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* upload: print per-file results before exiting on a partial batch failure

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 20:08:57 -07:00
chuanying chenandChris Lu d35f4c1c6b Missing data read in super block extra parsing (#8392)
* storage: fail on short super block extra read even without an error

Some backends (e.g. s3_backend) clear io.EOF after a short read, returning
n < len(p) with a nil error. Checking n only when the error is non-nil
lets truncated extra data reach proto.Unmarshal with a zero-filled tail.

* storage: apply the same short-read check to the super block header

A backend that returns n < SuperBlockSize with a nil error previously
slipped through and parsed a partially zero-filled header.

* storage: test super block extra reads

Covers a round trip through ReadSuperBlock, a truncated extra region, a
backend returning io.EOF alongside a full read, and a backend returning
a short read with a nil error.

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-10-21 20:07:52 -07:00
1aad4cfe70 iam: optionally require an iam: grant for self-service access-key actions (#11937)
* iam: optionally require an iam: grant for self-service access-key actions

AuthorizeIamAction lets every authenticated identity run CreateAccessKey,
DeleteAccessKey, UpdateAccessKey, ListAccessKeys and GetUser on itself with
no grant. On AWS these actions need an IAM policy (the usual one grants them
on arn:aws:iam::*:user/${aws:username}); nothing is allowed implicitly.

Where identities hold data-plane rights only, the implicit carve-out lets a
leaked key mint further keys for its own identity, with no cap, that keep
working after the operator deletes the key that leaked; it can also
deactivate or delete the keys the operator issued.

A new option, off by default, makes these actions need the iam:<Action> grant
like every other IAM action: -s3.iam.selfServiceRequiresGrant on weed
server, filer and mini, -iam.selfServiceRequiresGrant on weed s3. With it
set, the grant is authorized against the target user's ARN, the caller's own
when UserName is empty, so a policy can confine it to the caller's own keys
(Resource arn:aws:iam:::user/${aws:username}), the same resource scoping
CreateServiceAccount already uses. The admin and iam:* grants are unchanged;
with the option off nothing changes.

Tests (s3api_embedded_iam_authz_test.go): with the option set, an identity
holding only s3:* and a bare identity are refused all five actions on an
empty, own and other target (fails without the change); a grant on the
caller's own user ARN, literal or ${aws:username}, allows them on itself and
not on another user; the admin and an iam:* grant keep every action; with the
option off self-service is allowed as before.

* iam: wire selfServiceRequiresGrant on the standalone iam server

Every other command that runs the embedded IAM API accepts the flag; the
deprecated weed iam could not enable the enforcement at all.

* iam: wire selfServiceRequiresGrant on weed server and filer -iam

The IamOptions used by 'weed server -iam' and 'weed filer -iam' never
received the flag, so startIamServer dereferenced a nil pointer at
startup. Share the -s3.iam.selfServiceRequiresGrant pointer with them.

---------

Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-10-21 20:06:30 -07:00
Chris LuandDevin ea859a0844 sftp: reject empty passwords in PasswordAuthenticator (#11938)
* sftp: reject empty passwords in PasswordAuthenticator

FileStore.ValidatePassword already refuses a zero-length stored or
supplied password, but the guard belongs at the authenticator too: an
empty password is never valid, and checking before the store call keeps
any user.Store implementation without its own empty-secret check from
matching on zero-length input.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* sftp: keep the brute-force delay on empty-password failures

The early rejection returned before the shared failure path, so an empty
password failed instantly while other failures slept 100-199 ms. Folding
the length check into the store call keeps the store untouched and every
failed attempt equally slow.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 20:03:14 -07:00
Chris LuandDevin c0152e6be1 s3: keep '\' a literal object-key character behind -s3.allowBackslashInKey (#11911)
* s3: keep '\' a literal object-key character when AllowBackslashInKey is set

NormalizeObjectKey rewrites every S3 object key: '\' becomes '/' and
duplicate slashes collapse. On AWS S3 both are ordinary key characters,
so keys that are distinct there alias one object here — writing
probe/back/slash.txt silently overwrites probe/back\slash.txt.

The filer only treats '/' as a path separator, so a verbatim '\' stays
inside one entry name. Gate the backslash rewrite and the matching '\'
handling in IsValidObjectKey behind a process-wide mode the S3 server
sets from the new S3ApiServerOption. Duplicate-slash collapsing stays:
an entry name cannot contain an empty segment.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* command: add -s3.allowBackslashInKey flag for AWS-compatible key handling

Registers the flag on all four S3 entry points (s3, server -s3,
filer -s3, mini) and passes it through S3Options into
S3ApiServerOption. Default off, so existing normalization is
unchanged.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: correct IsValidObjectKey/NormalizeObjectKey docs under allowBackslashInKey

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: normalize DeleteObjects body keys the same as path keys

DeleteObjects keys arrive in the XML body, so they bypassed the
normalization applied to route-var keys. The lock, the authorization
checks, the table-bucket layout check and the filer delete itself all
addressed the raw key while the conditional check resolved the
normalized one, so a key spelled with '\' or '//' could be reported
deleted while the stored entry survived. Echo the client-sent key in
the result while operating on the normalized form.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: report a normalized-empty DeleteObjects key as a delete no-op

A multi-delete key of only slashes normalizes to the empty string. Left
to reach the filer delete, the empty name resolves to the bucket entry
itself, so an unauthorized caller could remove a bucket through the
body-key path. Such a key can never name a stored entry, so like a
missing key the delete is an idempotent no-op reported before any
permission check or filer call.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: drop unused AllowBackslashInObjectKey getter

* s3: answer conditional delete of a normalized-empty key with NoSuchKey

* s3: normalize the probed key in authorizeAnonymousBucketList

The masked-miss probe read the raw mux var, so a spelling like folder//
or folder\ was authorized against a key that can never exist while the
request itself targets the normalized folder/. Probe the normalized key
so the answer matches what the read path authorizes.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: cover anonymous authz of a literal-backslash key under allowBackslashInKey

A public grant on the slash key must not authorize an anonymous read of
the distinct backslash object when the flag keeps '\' literal; grant the
backslash key's own ACL and it reads back. Default mode still folds the
spelling onto the covered slash key.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: publish the key mode only after the S3 server is constructed

A failed NewS3ApiServerWithStore could previously flip the process-wide
key mode out from under an already-running gateway.

* s3: normalize the copy-source key before validation and authorization

pathToBucketObjectAndVersion returned the raw header spelling, so an
equivalent spelling (a//b, a\b) was authorized and compared against the
normalized destination while resolveCopySourceEntry looked up the
normalized key — a deny on the canonical key could be bypassed and
self-copy detection missed. Normalize once at parse so the CopyObject and
UploadPartCopy paths agree.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 20:02:04 -07:00
e057244b0a storage: read the super block extra before unmarshaling it (#11932)
* storage: read the super block extra before unmarshaling it

ReadSuperBlock allocated a buffer of ExtraSize bytes but never filled it,
so any volume with a super block extra failed to load with an invalid
wire-format error.

Co-authored-by: chuanying chen <18412672+ccy00808@users.noreply.github.com>

* storage: test super block extra read error paths

---------

Co-authored-by: chuanying chen <18412672+ccy00808@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-10-21 19:55:27 -07:00
Javier de la TorreandJavier de la Torre 763e2a2090 iceberg: record geometry bounds on the data files compaction writes (#11898)
* iceberg: record geometry bounds on the data files compaction writes

Compaction collected column statistics for every primitive column except
geometry: the bound converter has no geometry case, so a compacted file's
manifest entry carried value and null counts for a geometry column but no
lower or upper bound. Readers that skip files by these bounds for a spatial
filter (for example DuckDB for `geom && <box>`) had to open every compacted
file, where the files before compaction could be skipped.

The bound follows the spec's single-value serialization for geometry: the
lower bound is the point (xmin, ymin), the upper bound (xmax, ymax), each an
8-byte little-endian double; z and m are not written. It is the union of the
written file's row-group geospatial statistics, which parquet-go computes
from every non-null value.

A bound that is too small makes a reader skip rows, so any doubt gives none:
a row group with non-null values but no geospatial statistics (parquet-go
omits them when a value does not parse as WKB), a box that is not finite, an
inverted box, or a chunk whose statistics cannot be read. A row group of
nulls only constrains nothing. Geography gets no bound, as iceberg-go writes
none for it (geodesic edges, boxes across the antimeridian).

Tests: points and polygons far apart through a bin-packed compaction (the
bound equals the union of every geometry), a sorted compaction (any bound
covers every geometry), a column of nulls only, an unparseable value in a
second row group (no bound; without the rule the file got the bound of the
first row group only), geography, and the aggregation rules.

* iceberg: do not order compaction bins by a geometry column

When a table declares no usable sort order, compaction orders a bin's files
by the first column every file has bounds for. A v3 table whose first column
is a geometry qualifies as soon as its files carry geometry bounds (DuckDB
writes them, and compacted files now do too). Those bounds decode into geo
literals that compareLiterals cannot compare, so the order sorted nothing,
yet an oversized bin was still split into "ordered" runs by splitOrderedBin.

Geometry and geography columns are now skipped in both the declared sort
order and the inferred one, so such a table is ordered by its next column
with bounds.

Test: TestResolveCompactionOrderSkipsGeometry, a geometry column first and
an id column second; on master the order is the geometry column.

---------

Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
2026-10-21 19:28:26 -07:00
917b6f4afd iceberg: authorize a commit before staging its metadata file (#11891)
* iceberg: authorize a commit before staging its metadata file

Both commit paths (CommitTable and CommitTransaction) write the new
vN.metadata.json into the table's metadata/ directory and only then reach
UpdateTable, whose authorization refuses a caller that may read the table but
not commit to it. The refused commit then deletes the staged file on the way
out, so such a caller could make the server write and delete a file under the
table's prefix on every attempt. The same shape was closed for creates in
#10991.

Both paths now ask Manager.AuthorizeUpdateTable right after loading the table
and answer 403 ForbiddenException before anything is staged. UpdateTable still
decides under the policies in force when the pointer flips, so a policy changed
mid-commit is caught as before.

Test: TestCommitDeniedBeforeStagingMetadata records every file the handlers
create; on master both the single-table and the transaction commit by a
GetTable-only identity write v2.metadata.json before answering 403, with the
change nothing is written. TestCommitAllowedStagesMetadata is the positive
control (the owner, and an identity whose own actions include UpdateTable).

* iceberg: authorize a view commit before staging its metadata

handleUpdateView has the same write-before-authorize gap the table commit
paths had: it stages the new metadata file and only reaches the UpdateView
authorization at the pointer flip, so a caller who may only load the view
could still make the server write and delete files under it. Gate it on
AuthorizeUpdateView, mirroring authorizeCommit.

* iceberg: seed committed view metadata in the denied-commit test

Without stored view metadata the request fails on the missing-metadata
check before it could stage a file, so the zero-write assertion could
not catch a write-before-denial regression. Seed the same metadata the
allowed-path test uses so the denied request would otherwise stage.

---------

Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-10-21 19:13:02 -07:00
Chris LuandDevin 106fd31af1 s3: resolve versionId ACL and attributes requests to the version-scoped actions (#11916)
* s3: resolve versionId ACL requests to the version ACL actions

A request for an object's ACL carrying versionId acted on the historical version but authorized against the current-object action (s3:GetObjectAcl/s3:PutObjectAcl), because the acl block in resolveFromQueryParameters returned before versionId was considered - the same gap closed for tagging. An identity granted only current-object ACL access - or explicitly denied s3:*ObjectVersionAcl - could still read and modify ACLs on noncurrent versions.

?acl&versionId=<id> on an object now resolves to s3:GetObjectVersionAcl/s3:PutObjectVersionAcl; ?acl alone keeps the current-object actions and bucket ACL is unaffected. s3:PutObjectVersionAcl is mapped onto the WriteAcp identity action so policies can name it.

* s3: resolve versionId attributes requests to the version attributes action

GET /bucket/key?attributes&versionId=<id> fetched the historical version's attributes but authorized against s3:GetObjectAttributes, so a grant limited to the current-object attributes action could reach noncurrent versions. The request now resolves to s3:GetObjectVersionAttributes, and both attributes actions map onto the Read identity action so policies can name them (s3:GetObjectAttributes was previously rejected as an unmapped action).

* s3api: recheck the version action on versioned ACL/attributes requests

The phase-2 policy recheck inside GetObjectAclHandler,
PutObjectAclHandler, and GetObjectAttributesHandler passed the
current-object action even when the request carried versionId and phase 1
had resolved the version action. A Deny scoped to s3:*VersionAcl or
s3:GetObjectVersionAttributes could never fire in the recheck, and an
Allow limited to the version action went unevaluated. Select the action
the same way the resolver does — versioned whenever versionId is present.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3api: recheck the version action on versioned tagging requests

The same phase-2 gap exists in the tagging handlers left by the earlier
version-tagging resolution change: versioned GetObjectTagging /
PutObjectTagging / DeleteObjectTagging rechecks evaluated the
current-object tagging action, so a Deny on s3:*VersionTagging could not
fire.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3api: resolve HEAD ?acl as the head-object action, not an ACL action

No HEAD acl route exists, so HEAD /bucket/key?acl lands on
HeadObjectHandler — which serves object headers, not the ACL. Resolving
it to an ACL action denied readers who hold only GetObject(Version) and
let ACL grants reach a head-object read. Dropping HEAD from the acl
branch lets the request fall through to GetObject/GetObjectVersion,
matching how the tagging branch already treats HEAD.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3api: require the version read grant for versioned attributes

AWS asks for s3:GetObjectVersion alongside s3:GetObjectVersionAttributes
when a GetObjectAttributes request names a versionId; the resolved
version-scoped action on its own left a lone s3:GetObjectVersion Deny
unable to cover the attributes read. Check the version read grant too:
the identity side through VerifyActionPermission, the bucket-policy side
through the same recheck that evaluates the scoped action.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3api: check the version read grant on a synthesized versioned GET

The versioned attributes check passed the literal s3:GetObjectVersion
action, but both policy paths re-resolve the action from the request, so
the attributes query parameter mapped it back to
s3:GetObjectVersionAttributes and a lone version-read Deny never fired.
The literal action also reached the legacy CanDo path, where a plain
Read grant no longer matched. Evaluate the grant on a request shaped
like a plain versioned GET instead.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3api: reject a named versionId on buckets without versioning

Tagging, ACL, and attributes handlers answered versionId requests on
never-versioned buckets as the current object while evaluating the
version-scoped action, so a grant limited to the version action could
authorize changes to the live entry. Reject the request up front through
the existing checkVersionIdConfigured gate.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3api: carry presign params onto synthesized policy requests

objectKeyPolicyRequest rebuilt the query with only versionId and the STS
token, so a presigned request's s3:authType condition value silently
changed from REST-QUERY-STRING to unset during phase-2 rechecks (version
read grant, copy source, delete keys, post-policy writes). A policy
conditioned on s3:authType would then deny a request it already allowed.
Carry the SigV2/SigV4 presign keys through; none of them steer
ResolveS3Action.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 16:50:34 -07:00
Chris LuandDevin 137f23a229 ec distribution: implement the documented shard distribution API (#11915)
* ec distribution: parse EC and replication configs into a shard plan

NewECConfig/DefaultECConfig and NewReplicationConfig(FromString) turn an
EC ratio and a replica placement into ECConfig/ReplicationConfig values,
and CalculateDistribution derives the target and maximum shard counts per
data center, rack, and node. ECDistribution.Summary and
FaultToleranceAnalysis describe, per topology level, whether losing one
failure domain leaves enough shards to reconstruct the volume, so the EC
durability guarantees of a given setup can be answered directly instead of
only documented informally.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ec distribution: analyze topology and plan shard moves

TopologyAnalysis records nodes and shard locations per DC/rack/node, and
Rebalancer.PlanRebalance emits the moves that level the layout: data
centers first, then racks within each DC, then nodes within each rack,
moving parity shards before data shards and honoring per-node free slots.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ec distribution: cover the plan math and rebalancing with tests

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ec distribution: keep AddShardLocation and node ShardIDs in sync

AddShardLocation updated the count maps but never the node's ShardIDs —
yet every rebalancer count reads len(node.ShardIDs), so the documented
AddNode-then-AddShardLocation flow produced an empty rebalance plan. It
now appends to the node's shard list and skips shards the node already
has, making the two registration styles equivalent and idempotent.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ec distribution: key rack-scoped maps by dc/rack

Rack names are only unique within a data center — two DCs can both hold a
"rack1" — so ShardsByRack, RackToShards, and RackToNodes keyed by the bare
rack name merged distinct domains and corrupted both the DC-level shard
counts and rack-level balancing. Rack lookups now use the dc/rack pair;
DCToRacks still lists bare rack names.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ec distribution: never move a shard onto a node already holding it

The move picker selected the destination node before the shard, so a node
already holding the chosen shard could be targeted. Pick the shard first
(parity-first) and choose among destination nodes that lack it.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ec distribution: tolerate zero-value replication configs

A ReplicationConfig{} with zeroed minimums made maxShare divide by zero;
clamp each level's unit count to at least one. Covers the new behaviors
with tests.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ec distribution: keep rack keys distinct when names contain "/"

rackKey joined data center and rack with "/", so dc "a"/rack "b/c" and
dc "a/b"/rack "c" shared one key and were counted as the same failure
domain. Separate the parts with a NUL byte, which cannot appear in a
name.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ec distribution: plan rebalances on a copy of the topology

recordMove mutated the caller's TopologyAnalysis while planning, so an
interrupted or partial execution left the input describing moves that
never happened, and replanning on it returned nothing. PlanRebalance
now simulates on a cloned analysis; plan.Analysis reports the layout
the moves would produce and the caller's input is left untouched.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ec distribution: balance nodes against the rack's current shard count

The node-level target still read the original analysis, so a rack that
had just sent or received shards during DC/rack balancing aimed at a
stale total and left its nodes unbalanced.

* ec distribution: register each node id once

Adding the same node twice appended it to RackToNodes twice, so every
count that walks rack nodes double-counted its shards.

* ec distribution: normalize replication counts in the stored plan

A zero-value ReplicationConfig computed targets against one domain per
level while Summary still reported zero DCs, racks, and nodes.

* ec distribution: cap a survivable level's max at the loss budget

* ec distribution: skip shards already inside the destination domain

* ec distribution: register nodes implied by shard locations

* ec distribution: merge repeat node registrations

A shard location now auto-registers an unknown node with no free slots;
a later AddNode carrying the real capacity was then dropped entirely.
Repeat registration now updates the node's capacity and adopts any new
shards, while the first registration keeps the topology placement.

* ec distribution: try other overloaded sources before stopping

When the most overloaded domain could not shed any shard, balance
returned early and skipped every other source that still had movable
shards. A source that cannot move now gets set aside for the rest of
the pass.

* ec distribution: collapse chained shard moves to the final destination

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ec distribution: schedule collapsed moves behind the departures that free slots

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 16:50:01 -07:00
Junker der ProvinzandDan Foster e645449872 fix(storage): count preallocated space when auto-sizing max volumes (#11919)
The max volume count subtracted each volume's remaining room from the
free disk space, except when the master had preallocation turned on,
in which case it subtracted nothing. That is wrong for a disk that
mixes preallocated and plain volumes, and on platforms where
preallocation is not supported.

Use the larger of the .dat file's size and its allocated blocks, in the
Go and the Rust volume server, so a preallocated volume counts as full
and a plain one keeps its remaining room reserved.

Fixes #5778

Co-authored-by: Dan Foster <4570306+danfoster@users.noreply.github.com>
2026-10-21 15:11:06 -07:00
Junker der Provinz 9b642d4636 volume: use FALLOC_FL_KEEP_SIZE and log preallocation failures (#11920) 2026-10-21 15:07:55 -07:00
Javier de la TorreandJavier de la Torre a882565cd7 iceberg: keep a default-CRS geometry column without a CRS after compaction (#11899)
* iceberg: keep a default-CRS geometry column without a CRS after compaction

Writers such as DuckDB write a GEOMETRY column whose Iceberg type has the
default CRS (OGC:CRS84) with no CRS in the Parquet logical type. parquet-go
spells the default out when it writes such a column (geometryType's
LogicalType), even when the column it read declared none, so every
compacted file declared GEOMETRY(crs=OGC:CRS84) where its inputs declared
GEOMETRY. Both mean the same to the Parquet spec, but a reader may treat the
file column as another type than the table column: DuckDB casts it to the
table's GEOMETRY, and a cast column is no longer pruned by its row-group
statistics, so a spatial filter read every row group of a compacted file.

After a bin is merged, the footer's schema elements of every GEOMETRY
column whose Iceberg type has the default CRS (top-level or nested) are
rewritten to declare no CRS. The data, the page indexes and every offset
stay byte for byte; only the footer is re-encoded. An explicit CRS is left
as it is. Retyping the writer's column instead does not work: parquet-go
refuses to convert between two GEOMETRY types that both mean the default,
and its sorting writer converts.

Tests: geometry_crs_test.go, a bin-packed and a sorted compaction of
default-CRS inputs (one of them spelling the default out) write no CRS,
with a control showing parquet-go spells it out on its own; an explicit CRS
is kept; a nested geometry. On master the binpack and sort cases fail with
CRS "OGC:CRS84".

* iceberg: compress the row lineage columns of compacted v3 files

A v3 compaction appends _row_id and _last_updated_sequence_number to the
merged file. The two leaves were built without a codec, so they were
written uncompressed, 16 bytes a row, whatever codec the data columns use;
on a table of points the compacted files grew by about a third.

The lineage leaves now take the codec of the reference file's first leaf
that declares one, ZSTD when none does.

Test: TestCompactCompressesLineageColumns; on master both columns are
written UNCOMPRESSED next to ZSTD data columns.

* iceberg: match list and map geometry by Iceberg name in files without field ids

A Parquet footer path runs through the repeated group of a LIST or MAP
(shapes.list.element, attrs.key_value.value), which Iceberg leaves out of
its field names (shapes.element, attrs.value). The name lookup for a file
without field ids therefore missed every geometry inside a list or a map
and left the default CRS spelled out. Name the paths as Iceberg does,
including the legacy two-level list forms.

* iceberg: rewrite the geometry CRS footer inside the merged file's buffer

Removing a CRS shortens the footer, so the new one fits where the old one
was. Copying the whole merged file to replace a few hundred bytes of
footer doubled the memory of a compaction near the target file size.

* iceberg: take the lineage codec from the data columns, not an input's _row_id

An input that already carries _row_id may list it before its data
columns, and write it uncompressed; the new lineage columns then copied
that codec instead of the data columns'. Choose from the columns the
layout keeps.

---------

Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
2026-10-21 14:50:45 -07:00
Javier de la TorreandJavier de la Torre de2fc03c7d s3tables: accept the external-volume verification object at a table bucket's root (#11900)
* s3tables: accept the external-volume verification object at a table bucket's root

Snowflake verifies an S3-compatible external volume before it creates or
writes Iceberg tables in it (SYSTEM$VERIFY_EXTERNAL_VOLUME, also run
implicitly): it writes, reads, lists and deletes one small object,
<base>/verify_<epoch milliseconds>_<number>, a 21-byte body. A table bucket
refuses every key outside the Iceberg layout, so the check failed and
Snowflake could not use a table bucket as an external volume.

PUT, GET, HEAD, DELETE and DeleteObjects now accept exactly that key in a
table bucket, narrowly enough that it cannot be used to store data:
- at the bucket root only, so nothing is encoded in a prefix, written into
  a hidden tree or placed on a namespace or table name;
- one shape: verify_<13 digits of epoch ms within one hour of the server's
  clock>_<1-10 digits>;
- a PUT must declare a Content-Length of at most 1 KiB (no chunked or
  undeclared body) and carry no user metadata, tagging, Content-Disposition,
  Cache-Control, Expires or website redirect.
Multipart, copy, rename, ACL, tagging and POST policy writes keep the plain
layout rule. Buckets that are not table buckets are unaffected.

Tests: table_bucket_verify_object_test.go, the key accepted (with the
Iceberg layout as the positive control and the plain rule still refusing
it), 15 look-alikes refused (prefixed, hidden tree, wrong shape, outside the
window, ordinary keys), and seven PUT refusals through PutObjectHandler.

* s3tables: keep the verification key off namespaces and narrow its PUT

A namespace may be named like a verification key (verify_<ms>_<n>), and
the key then names the namespace's directory. A PUT replaced that
directory with a prefix object, dropping the namespace's catalog
metadata; a DELETE removed an empty namespace. The key now gets the plain
layout rule when it names a directory or its entry cannot be read, and a
PUT of it never falls back to a prefix object, so a namespace created
after the check is not replaced either.

The time window applies to PUT only: GET, HEAD and DELETE take a key of
any date, so an object a check left behind for more than an hour can
still be read and removed. A verification PUT also refuses an extraction
request and the stored Content-Language, Response-Content-Disposition and
Content-Encoding headers (aws-chunked framing alone is still accepted).

The integration test runs the check's PUT, HEAD, GET, list and DELETE
against a table bucket, and the PUT and DELETE over a namespace of the
key's shape.

---------

Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
2026-10-21 14:49:36 -07:00
Javier de la TorreandJavier de la Torre 21add33b6e s3 backend: optional HTTP client settings for the cloud tier (#11897)
* s3 backend: optional HTTP client settings for the cloud tier

The S3 tier backend builds its client from a bare aws.Config, so it always
uses Go's default transport: HTTP/2 wherever the endpoint offers it over TLS,
no cap on connections per host, and a 90s idle timeout. Against some
S3-compatible services that is a poor fit. Over HTTP/2 every upload and
range read of a tiered volume shares one connection and its flow-control
window; with a small window, tier uploads and concurrent reads of tiered
volumes become throughput-bound on that one connection, while HTTP/1.1 opens
a connection per concurrent request. Some services also limit parallel
connections per client address, or drop idle connections before 90s.

Add three optional settings to [storage.backend.s3.<id>], all off by
default so existing deployments keep the SDK's default client:
- disable_http2: keep the client on HTTP/1.1 (ForceAttemptHTTP2=false and an
  empty TLSNextProto, the documented opt-out);
- max_connections: MaxConnsPerHost, 0 = no cap;
- idle_connection_timeout_seconds: IdleConnTimeout, 0 = Go's default.

The session cache key includes the options when any is set, so two backends
on the same endpoint with different settings do not share a client.
ToProperties reports the settings; the master.toml scaffold documents them.

Tests: TestS3BackendHTTPOptionsReachTheClient fails without the change (the
settings never reach the client); TestS3BackendDefaultHTTPClientUnchanged,
TestS3BackendSessionsDifferByHTTPOptions, TestParseNonNegative.

* s3 backend: reject an idle timeout too large for time.Duration

An idle_connection_timeout_seconds above math.MaxInt64/1e9 overflowed when
multiplied by time.Second. The negative result made the transport keep its
default timeout and ToProperties export a negative value. Warn and use the
default instead.

* s3 backend: parse the idle timeout as 64-bit seconds

parseIdleConnTimeout went through strconv.Atoi, so on 32-bit platforms
any value above 2147483647 seconds was rejected while the same value was
accepted on 64-bit, and the overflow test failed under GOARCH=386. Parse
with ParseInt(…, 64) and export the effective value with FormatInt so the
accepted range is the same on every platform.

---------

Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
2026-10-21 14:44:39 -07:00
Chris LuandDevin 9a03c09396 s3api: cover the inline-content UploadPartCopy route at handler level (#11914)
s3api: exercise the inline-content route through CopyObjectPartHandler

Generated with [Devin](https://devin.ai)

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 14:15:04 -07:00
Javier de la TorreandJavier de la Torre f40267db44 s3: answer InvalidBucketName to a denied request on a malformed bucket name (#11893)
* s3: answer InvalidBucketName to a denied request on a malformed bucket name

The store authorizes a request before it looks at the bucket, so a caller
denied on a bucket whose name breaks the S3 naming rules (upper case, an
underscore, too short, two adjacent periods, a non-ASCII letter) gets 403
AccessDenied, where AWS S3 answers 400 InvalidBucketName. Clients and
compatibility suites that check the error code read the 403 as a permission
problem.

Auth and AuthPostPolicy now turn an AccessDenied for a resolved identity into
InvalidBucketName when the bucket name fails VerifyS3BucketName or uses a
character outside a-z, 0-9, '.' and '-' (VerifyS3BucketName checks with
unicode.IsLower/IsNumber, which accept letters such as 'ü'). The answer depends
on the name alone, so it reveals nothing about which buckets exist. An
authentication failure, an allowed request and a denial on a well-formed name
answer as before.

Test: TestAuthDeniedInvalidBucketName drives iam.Auth with signed and anonymous
requests; on master every malformed name answers 403, with the change 400,
while the identity's own bucket stays allowed and a well-formed foreign name
stays 403. TestRefineDenialForBucketName covers the rule table.

* s3: refine POST-policy and deferred anonymous-read denials on a malformed bucket name

Two denial paths bypass the refinement in Auth and AuthPostPolicy:

- A browser POST upload carries its credentials in the form, so
  AuthPostPolicy delegates it to PostPolicyBucketHandler, which verifies the
  form and writes AccessDenied itself when the signer may not write the bucket
  or the per-key authorization denies the object. Both denials now go through
  refineDenialForBucketName with the verified identity.

  The coarse Write check used to run inside doesPolicySignatureV2Match and
  doesPolicySignatureV4Match before the signature was compared, so a form with
  a wrong signature was told whether the key may write the bucket. It now runs
  in doesPolicySignatureMatch after the signature verifies, as on the
  header-signed path; a form that fails verification answers
  SignatureDoesNotMatch (or MissingFields) and is never refined.

- Unsigned GET/HEAD object reads are deferred by AuthWithPublicRead to the
  object handlers, where writeAnonymousAwareNotFound masks a miss as
  AccessDenied. That denial is refined with the anonymous identity, so a
  missing malformed bucket answers InvalidBucketName when an anonymous
  identity is configured, as Auth does. A denial on an entry that was found
  is left alone: the bucket exists.

Test: TestPostPolicyDeniedInvalidBucketName drives AuthPostPolicy with signed
forms (Write denied, per-key policy deny, bad signature, well-formed name);
TestAnonymousReadDeniedInvalidBucketName drives AuthWithPublicRead with GET and
HEAD, with and without an anonymous identity. Both fail before this change.

* s3: refine a denied anonymous read of an existing key on a malformed bucket name

A deferred anonymous GET/HEAD refined only its masked miss to
InvalidBucketName, so in a bucket whose name breaks the S3 rules (one
VerifyS3BucketName still accepts, such as a non-ASCII name) a missing key
answered 400 and an existing private key 403, telling the caller which
keys exist. The entry check now refines its denial the same way, so both
answer alike. Without an anonymous identity no identity resolves and both
stay 403, as in Auth.

The deny-precedence case in TestAnonymousObjectACLConfiguredIdentity runs
on the fixture bucket "b", shorter than S3's three-character minimum, so
its denial now answers InvalidBucketName.

---------

Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
2026-10-21 14:08:33 -07:00
Chris LuandDevin 6ee7c4c5c7 s3: no empty-directory marker in a "<dir>/" self-prefix listing (#11906)
An out-of-band empty directory listed under its own trailing-slash
prefix answered the probe with a fake entry of itself: a Contents key on
4.48 and, since recently, a CommonPrefix. That listing enumerates
children, where S3 returns nothing for an empty prefix, and clients that
read every returned key (Pulumi lock listing) choked on the marker.

The prefix object and directory key object paths are unchanged: a real
"dir/" object still lists as its own key.

Generated with [Devin](https://devin.ai)

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 14:03:02 -07:00
9c427b265f volume tier: accept deletes on a volume as soon as it is tiered (#11894)
* volume tier: accept deletes on a volume as soon as it is tiered

volume.tier.upload marks the volume read-only without deletes before the
upload (markVolumeReplicasWritable(..., false, false), i.e. noWriteOrDelete),
and nothing clears that until the volume server restarts, when load() gives
every tiered volume noWriteCanDelete. Every delete in between is refused with
"volume N is read only": the needle stays live in the index, and its bytes in
the remote object, until someone notices.

After the remote file is loaded, the upload handler now gives the volume the
flags a restart would (MarkTieredDeletable: writes refused, deletes accepted),
with two exceptions that keep the current behaviour:
- a replicated volume: the shell copies the uploaded replica's .idx and .vif
  to the other replicas after the upload, and a tombstone written before that
  copy would reach one replica only;
- a volume in I/O quarantine: that flag is not the upload's to lift.

Tests: TestTierUploadAcceptsDeletesOnASingleCopyVolume fails without the
change ("volume 1 is read only") and checks writes stay refused;
TestTierUploadKeepsRefusingDeletesOnAReplicatedVolume; TestMarkTieredDeletable
(quarantine kept).

* volume tier: check the I/O quarantine under noWriteLock

MarkTieredDeletable read the quarantine state before taking noWriteLock.
CollectHeartbeat could quarantine the volume and set noWriteOrDelete in
between, and the upload then cleared that refusal, so a quarantined volume
accepted deletes until the next heartbeat. Take noWriteLock first: the
heartbeat marks the quarantine before it takes noWriteLock, so either the
check sees it or the heartbeat's refusal lands afterwards and stays.

Also leave alone a volume past IoErrorTolerance that the next heartbeat
will quarantine, as the heartbeat itself does.

* volume: read noWriteOrDelete under noWriteLock in DeleteVolumeNeedle

DeleteVolumeNeedle checked v.noWriteOrDelete without the lock that
MarkVolumeReadonly, CollectHeartbeat and now MarkTieredDeletable hold when
they write it, so a delete racing a tier upload was a data race. Copy the
flag under the read lock; the delete path is otherwise unchanged.

* volume tier: mirror the delete acceptance rules in the Rust volume server

The Rust tier upload already promoted a freshly tiered volume to
no-write-can-delete via refresh_remote_write_mode, and did so for every
volume -- including the replicated and I/O-quarantined ones the Go
MarkTieredDeletable deliberately refuses. A live tier-up now goes
through mark_tiered_deletable (replicated volumes and quarantines keep
their upload-time mark until the next load), while load() keeps
promoting every tiered volume unconditionally, as Go does.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: keep a tiered volume delete-only when marked writable

MarkVolumeWritable preserved noWriteCanDelete on a remote-tiered volume
only when it was already set. A tiered volume still holding the
upload-time mark (a replicated volume, or one whose tier upload ran
before MarkTieredDeletable existed) had it clear, so marking writable
published the volume as fully writable while its .dat lives remotely --
writes then failed against a backend that cannot append, after the
master was told the volume is writable again.

load() gives every remote-tiered volume noWriteCanDelete; deriving the
same flag here keeps mark-writable consistent with a restart: "writable"
on a remote volume resolves to can-delete.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: keep a tiered volume delete-only when marked writable (rust)

set_writable preserved the remote delete-only mode only when the flag was
already set, so a tiered volume holding the upload-time read-only mark
(replicated, or quarantined under mark_tiered_deletable) was published
fully writable with its .dat remote. Derive no_write_can_delete from
has_remote_file() instead, matching load() and the Go MarkVolumeWritable.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume tier: serialize the quarantine check with the flag swap (rust)

A heartbeat refusal landing between should_quarantine() and the store
could be clobbered until the next heartbeat re-asserted it — the same
race the Go side closed by checking under noWriteLock. A
compare-exchange gives the same ordering: the swap fails, and the retry
sees the quarantine mark the heartbeat sets first.

* volume tier: MarkTieredDeletable refuses a non-tiered volume

A stray call on a local single-copy volume would pin it
read-only-can-delete. The only caller promotes right after
LoadRemoteFile, but the guard makes the method safe to reuse; the Rust
mark_tiered_deletable gets the same check, though its caller already
only reaches it under has_remote.

---------

Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
Co-authored-by: chrislusf <chris.lu@gmail.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 14:01:10 -07:00
Javier de la TorreandJavier de la Torre 97f8ca13ff s3tables, iceberg: log client refusals at V(1), not ERROR (#11896)
* s3tables, iceberg: log client refusals at V(1), not ERROR

Requests the S3 Tables and Iceberg REST APIs correctly refuse with a 4xx
(access denied, not authorized, no such namespace, already exists, not
empty, a commit conflict) were logged with glog.Errorf, so a client probing
for what it may do, or retrying an optimistic commit, filled the ERROR log
and hid real faults from anyone alerting on it.

- s3tables: HandleRequest wraps the ResponseWriter in a statusRecorder (with
  Unwrap for http.ResponseController) and logs a handler error by the status
  it answered: 4xx at V(1) as "<op> refused (<status>)", 5xx or no status at
  ERROR as before.
- s3tables: the "handleCreateNamespace called" trace moves from ERROR to V(3).
- iceberg: writeManagerError's mapping is split into classifyManagerError,
  shared with a new logManagerError that logs 4xx-mapped errors at V(1) and
  everything else at ERROR; used by CreateNamespace, CommitTable and
  CommitTransaction.
- storage: loading a remote-tier volume logged
  "readSuperBlock remote volume N: <nil>" at ERROR even when the read
  succeeded; it is now logged only when there is an error.

Tests: TestRequestRefusedWith4xxIsNotLoggedAsError, TestLogRequestErrorByStatus,
TestStatusRecorderKeepsTheFirstStatus, TestStatusRecorderUnwrapsForResponseController,
TestNoTraceLoggedAtErrorLevel (fails on master on the CreateNamespace trace),
TestLogManagerErrorRefusalsAreNotErrors, TestRemoteSuperBlockLogOnlyOnError.

* s3tables: do not log a malformed CreateNamespace body at ERROR

handleCreateNamespace logged a body it could not decode at ERROR before
answering 400; the dispatcher already logs that refusal once at V(1).

Also drop statusRecorder.Flush: it made the recorder an http.Flusher even
over a writer that cannot flush, so http.NewResponseController reported
success instead of ErrNotSupported. Unwrap already lets the controller
reach the wrapped writer.

* iceberg: require exactly one log line per classified manager error

The test accepted a refusal that produced no log, or both logs. Require
one V(1) refusal and no ERROR for a 4xx, and the reverse for a fault.

---------

Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
2026-10-21 14:00:33 -07:00
Chris LuandDevin 780760fccc s3: report the copied bytes' MD5 in UploadPartCopy ETags (#11884)
* s3api: regression test for UploadPartCopy ETag on ranged copies

Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3api: hash the copied bytes for a partial-chunk UploadPartCopy ETag

A ranged UploadPartCopy that lands inside a source chunk stored the source chunk's ETag on the destination chunk, so the part reported the MD5 of the wrong bytes. streamCopyChunkRange now tees the forwarded plaintext into an optional digest writer; copySingleChunkForRange feeds it an md5 only when the range is not the whole source chunk, since a whole-chunk copy's stored bytes (and ETag) are unchanged.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3api: stamp the copied range's MD5 on UploadPartCopy part entries

The part entry got no Attributes.Md5, so copyEntryETag fell back to filer.ETagChunks: a single-chunk part inherited the source chunk's MD5 and a multi-chunk part reported a <md5>-N composite of source chunks. copyChunksForRange now returns the MD5 of the copied bytes — a single chunk's own digest, or for multi-chunk parts a second ordered stream of the source range hashed beside the concurrent chunk copies (md5 cannot combine per-chunk digests). The handler stores it on Attributes.Md5 and mirrors putToFiler's ExtETagKey, so CopyPartResult, ListParts and the completed object's composite ETag use each part's own MD5 (#11855).\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3api: drop the source chunk's gzip flag on partial-range copies

A ranged UploadPartCopy fetches raw (decompressed) bytes from the source volume and stores them uncompressed, but the destination chunk kept the source's IsCompressed flag, so reads of a partial copy of a gzip-stored chunk would try to decompress raw bytes.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3api: hash the chunk range, not the entry, for the part MD5

getEncryptedStreamFromVolumes prefers inline Content over chunks, but copyChunksForRange copies the chunks — on an entry carrying both, the hash would describe different bytes than the copy.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3api: cover sparse ranges in the part MD5 and fail on hash errors

A single chunk's digest is the part MD5 only when it supplies the whole range; a sparse source zero-fills the rest, so gaps are hashed via the ordered range stream, and a fully chunkless range hashes zeros locally. A failed hash pass now fails the part copy instead of storing a chunk-composite ETag that describes the wrong bytes.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3api: rank copied-range destination chunks by source timestamps

The concurrent chunk copies stamped each destination chunk with an arbitrary finish time, but reads resolve overlapping chunks by ModifiedTsNs — a copy of overlapping source chunks could expose different bytes than the source (and than the part MD5 just hashed). Destination chunks now get timestamps ranked by the source's, preserving its overlap precedence.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3api: cancel the part-MD5 stream when a chunk copy fails

The hashing goroutine ran on context.Background(), so an early chunk-copy error return left it streaming and hashing the whole source range for a discarded result. A cancellable context now aborts it on every exit path.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3api: hash the copied range when the covering chunk has no usable ETag

TUS and mount writes leave chunks without an ETag, so a whole-range
UploadPartCopy would otherwise save the part with an empty Md5 and no
ExtETagKey, and multipart completion rejects the part.

* s3api: build the md5 source entry without copying the proto message

* s3api: drop the issue reference from the part-copy etag test

* s3api: route inline-content sources through the plaintext part copy

The raw-byte path walks entry.GetChunks() only, so a ranged
UploadPartCopy of an inline-Content source (filer HTTP or mount writes)
produced a zero-filled part. Send those through
copyObjectPartViaReencryption, which streams the real bytes and stamps
their MD5 on the part.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3api: store the empty-part MD5 on writeEmptyCopyPart entries

The slow path's empty part returned the right CopyPartResult ETag but
saved no Attributes.Md5 or ExtETagKey, so ListParts and the completed
object's composite ETag still fell back to a chunk-derived value.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 13:46:11 -07:00
Jay HemnaniandChris Lu a3e97ec709 s3: apply Range to HeadObject (#11856)
* s3: apply Range to HeadObject

* s3: apply Range to a HEAD of a stored directory key

Directory-key entries answer through serveDirectoryContent before
HeadObjectHandler's range branch, so a ranged HEAD returned 200 and the
full length.

* s3: keep the missing-directory check for plain directories only

A stored directory-key marker is an object, so a range on its empty body
answers 416 like on any other zero-byte object.

* s3: let partNumber keep its priority over Range on a directory key HEAD

* s3: serve ranged GETs of a stored directory key

The same branch now applies the validated range to the inline content on
GET, so a directory-key object answers 206 with the selected bytes like
any other object.

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-10-21 13:37:00 -07:00
23d8c62844 volume: route lookups to the leader master only (#11888)
* volume: route lookups to the leader master only (#11887)

After a leader change, the volume server kept sending volume and EC
shard lookups to masters that were not the leader:

* currentMaster was set as soon as a heartbeat stream opened, before
  the peer said whether it was leader, and kept a follower through the
  redirect and its 3s sleep. A follower answers LookupVolume from its
  own vid cache, which right after an election can be empty, so reads
  went 404 rather than failing over.
* EC lookups used Store.MasterAddress, a second, unlocked copy that the
  heartbeat loop set before each dial and cleared on errors, so EC and
  plain reads could disagree on the master or have none.
* the receive goroutine called stream.Send itself when the volume
  options changed, racing the send loop, and exited silently on
  failure, after which leader redirects were no longer read.

Publish currentMaster only once a master is known to be leader: the
peer answered a new stream without naming a leader, which the leader
always does, or a follower named the leader. Compare redirects against
the connected peer. Replace Store.MasterAddress with MasterFn, backed
by getCurrentMaster. Have the receiver ask the send loop for the extra
heartbeat instead of sending it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: ask the masters for the leader after losing the heartbeat

When the heartbeat stream to the leader failed, currentMaster kept the
lost master until a new stream answered, so lookups on a cache miss
kept dialing it through the retry delay even with a new leader already
elected.

On a regular heartbeat error, ask the configured masters for the raft
leader and publish it, within the retry delay rather than on top of it.
Without an answer the last known leader stays: clearing it would only
trade a dead address for an empty one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: stop the heartbeat on a failed assertion

The leader test closed stopChan only once every assertion had passed,
so a failure left the stream open and the fake master's GracefulStop
waited for the whole test run's timeout.

Start the heartbeat through a helper whose cleanup stops it, safe to
call twice, and waits for it to exit. It runs before the cleanups
registered earlier, so the stream is closed before the store and the
fake master go away. Both heartbeat tests use it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: clear the leader when no master names one

When the heartbeat lost its master and no configured master named a
leader, refreshLeaderFromPeers kept the lost one. That master is dead
or no longer leader, and a follower answers LookupVolume from its own
cache instead of refusing, so lookups could return locations no leader
vouched for.

Clear currentMaster instead. Lookups then fail fast ("missing address")
until a leader is known again, rather than reading a follower's cache
or waiting on a dial to a dead master.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: keep volumeSizeLimit 64-bit aligned on 32-bit builds

MasterFn (a func, 4 bytes on 386) replaced MasterAddress (a string, 8
bytes) ahead of volumeSizeLimit and moved it to a 4-byte-aligned
offset, where atomic.LoadUint64 panics with an unaligned 64-bit
operation. The atomic.Uint64 type carries the alignment itself.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: publish the answering leader before a duplicate-uuid retry

Read the leader field first on each heartbeat reply. A follower reply
carries nothing else, so it no longer runs through the option updates;
and a leader that rejects the registration over duplicate uuids still
answers lookups, so it stays the lookup target through the retry wait
instead of leaving the previous master in place.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: publish the master only once it is known leader (rust)

* volume: ask the masters for the leader after losing the heartbeat (rust)

* volume: read the leader field first on each heartbeat reply (rust)

* volume: drop the seed fallback from ec shard lookups (rust)

cached_lookup_ec_shard_locations fell back to the static first seed when
no leader was published, so a mid-election EC lookup could land on a
follower and read its stale shard map. Error out like Go's
masterAddress() returning "" instead, via a current_master() getter
matching getCurrentMaster.

* volume: send volume lookups to the confirmed leader, not the seed (rust)

Every HTTP lookup path — proxy/redirect targets, replicated writes and
deletes, batch chunk deletes, remote chunk reads — dialed the static
first seed, so after a failover they kept asking a follower whose vid
map can be stale. Go routes all of these through GetMaster; switch the
lookup calls and their leader-known gates to current_master().

* volume: mark volumes readonly on the leader, retrying the named one (rust)

notify_master_volume_readonly fell back to the seed when no leader was
published, which the peer answers with "Not current leader". It now
uses only the confirmed leader, skips the call in masterless mode, and
on NotLeader asks the seed masters who leads and retries there — the
same path Go's notifyMasterVolumeReadonly takes. The peer leader query
is shared with refresh_leader_from_peers as lookup_leader_from_peers.

* volume: ask the confirmed leader for copy prealloc options (rust)

VolumeCopy's preallocate query went to the static seed; Go issues it
through GetMaster. Route it through current_master() too — when no
leader is known the lookup fails fast and the copy proceeds without
preallocating, as before.

* volume: keep the announced master port forms, not just http (rust)

publish_current_master stripped to http form, so a leader configured
with an explicit gRPC port (host:9333.19999) was redialed on the
default offset. Store the announced host:port[.grpcPort] form like Go's
currentMaster; callers pick to_http_address / to_grpc_address /
parse_grpc_address per use, and Ping admission compares on the http
form.

* volume: fail replicated writes and deletes when no leader is known (rust)

Gating replication on a known leader meant a write could be
acknowledged on fewer copies than required, and a delete could leave
the data readable on the other replicas. Go fails the request when the
replica-location lookup cannot run; drop the gate so the empty-leader
lookup errors the same way.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-10-21 13:36:05 -07:00
Chris LuandDevin 4bdbfbaaa8 s3tables: judge an absent tag target by its canonical ARN (#11912)
* s3tables: judge an absent tag target by its canonical ARN

authorizeAbsentTarget matched the bucket policy's Deny statements
against the request's raw resource ARN. The ARN parsers accept any
account and region but resolve the target by name, so a caller with an
identity grant could reformat the ARN, miss the Deny match, and learn
whether a denied name exists — the oracle the absent-target check was
meant to close. Rebuild the ARN from the resolved names and the real
bucket owner and evaluate the Deny against that, matching what the
existing-target path checks.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3tables: cover a foreign-account ARN on an absent denied table

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3tables: keep the absent-target deny check behind a supplied ARN

Callers that pass no resource ARN (ListTables, ListViews,
CreateNamespace) must reach CheckTableOrBucketPermission's admin
shortcut unchanged; running the bucket-policy Deny match first would
deny the admin lists those callers take.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 13:31:04 -07:00
Mohd Quamar TyagiandChris Lu e73c23a4ee fix(s3): set Access-Control-Allow-Credentials on CORS responses (#11886)
* fix(s3): set Access-Control-Allow-Credentials on CORS responses (fixes #11885)

* fix(s3): expand wildcard exposed headers for credentialed CORS responses

* fix(s3): keep wildcard in exposed headers for non-credentialed CORS requests

* fix(s3): expose x-amz-tagging-count on credentialed CORS responses

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-10-21 13:10:49 -07:00
Chris LuandDevin 8c3d8fcea0 s3tables: keep absent-target answers identical to denied ones (#11905)
* s3tables: authorize before revealing a list or tag target exists

ListTables, ListViews and the tag operations read the target's catalog
metadata before any permission check, so a denied caller could tell an
existing name (403 or the resource's own error) from an absent one (an
empty list, or 404). Evaluate the operation's permission with whatever
bucket context could be read and answer a denied caller the same way for
absent, damaged and denied targets.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3tables: answer bucket reads uniformly when the entry is not a table bucket

A plain S3 bucket directory answers GetTableBucket and ListNamespaces with
500 attribute-not-found while a missing name gets 404, and a denied
CreateNamespace is 403 while a missing bucket is 404. Run the permission
check before the absence answer and read not-a-table-bucket as
NoSuchTableBucket.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3tables: cover absent-target answers matching denied ones

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 13:07:47 -07:00
Chris Lu bb1ba8b247 iam: take the implicit user name from the verified identity (#11903)
* iam: take the implicit user name from the verified identity

For self-service actions sent without UserName, the implicit user was
resolved by re-parsing the raw Authorization header for a SigV4
credential and looking that access key up. The header is not what
authentication verified: a request can be authenticated by one
mechanism while carrying a header that names another user's access
key, and the action then ran against that other user.

The auth middleware already records the verified identity in the
request context, so use it, on both the standalone iam server and the
embedded IAM surface. When no verified identity is present the name is
left unset rather than guessed.

* iam: test implicit user name resolution

Cover a request authenticated by presigned credentials that also
carries a junk SigV4 header naming another user: the self-service
actions must run as the verified signer. Update the unit tests for the
context-based lookup.

* iam: resolve the implicit user for standalone GetUser too

The embedded surface fills UserName for GetUser; the standalone server
skipped it, so an implicit GetUser on `weed iam` answered NoSuchEntity.
Same verified-identity source as the other self-service actions.

* iam: cover all implicit-username actions in the regression test
2026-10-21 12:21:04 -07:00
Javier de la TorreandJavier de la Torre 593cfee85b s3tables: accept Puffin statistics files in a table's metadata/ (#11892)
Table and partition statistics are Puffin files that the table metadata lists
in its statistics entries; orphan cleanup keeps them (#11651), and its test
places one at metadata/stats-1.puffin. The table bucket layout rule accepted
only *.stats among them, so an engine that names a statistics file *.puffin
had the upload refused with 403 and could not write statistics at all.

The metadata patterns now admit a *.puffin file directly under metadata/,
with the same safe-character rule as the other catch-all patterns. Nothing
else widens: a Puffin file in a subdirectory of metadata/, with a trailing
extension, or outside <namespace>/<table>/ is still refused.

Test: the new metadata/*.puffin cases in
TestIcebergLayoutValidator_AcceptsRealWorldManifestNames and
TestTableBucketFileValidator_PuffinStatisticsOnlyInTableMetadata fail on
master ("invalid metadata file format") and pass with the change; the
rejected cases pass on both.

Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
2026-10-21 12:19:53 -07:00
Daniel García 03065155db filer.remote.sync: do not upload an entry whose only content is the remote object (#11890)
* filer.remote.sync: do not upload an entry whose local copy was dropped

remote.uncache and the filer's eviction under storage pressure drop a
synced entry's chunks and clear LastLocalSyncTsNs; remote.meta.sync does
the same when it pulls a newer remote version. When the remote mtime
falls in the entry's mtime second, which is the common case for a small
file synced promptly, shouldSendToRemote falls back to the cleared stamp
and reads the entry as dirty. The dropped chunks make the update a
content change, so the chunkless entry was uploaded and a zero-byte
object replaced the remote one, the entry's only content.

Skip an update that leaves the entry at the same key with no local data
and a sized RemoteEntry: the object at that key already is the content.
The stamp stays cleared, so remote.cache and eviction still see the
entry as uncached. Renames keep going through completeRemoteOnlyRename,
metadata-only edits of a remote-only entry still reach the remote, and a
rewrite in the remote mtime's second holds local data and is uploaded.

* filer.remote.sync: upload a truncate that leaves the entry remote-only

A FUSE truncate of a cached, remote-backed file to zero (truncateEntry,
or SetAttr with size 0) drops the chunks and keeps the sized
RemoteEntry, so the entry reads as remote-only and the dropped-copy skip
left the remote object with its old content.

Skip only while the entry's size is still the remote object's. Uncache
and eviction keep the size and remote.meta.sync sets it from the
object, so those stay skipped; a truncate changes the size and is
uploaded, in the remote mtime's second too.
2026-10-21 11:32:20 -07:00
jsasandjames f8cfea0dcf gcs: size the upload buffer from the entry instead of reserving the default 16 MiB chunk per writer (#11854)
Co-authored-by: james <james@medable.com>
2026-10-21 00:03:38 -07:00
Jay Hemnani c90da999f2 s3: answer a conditional write on a missing object with 404, not 412 (#11882) 2026-10-20 23:43:30 -07:00
Chris LuandDevin 379abbe278 volume: repair torn tails on canDelete volumes; reject damaged 1-3 byte records (#11876)
- Go/Rust: run the unindexed .dat tail repair on noWriteCanDelete volumes
  too — deletes still append tombstone records, so the same crash window
  applies and they previously degraded to fully read-only.
- Go readTailNeedle: reject a positive size of 1-3 for v2+ as corrupted
  before ReadBytes slices the body's 4-byte data-size field and panics;
  a torn last record is now cut instead of crashing the load.
- Tests: TestVolumeLoadRepairsTailOnCanDeleteVolume and
  TestVolumeLoadCutsDamagedSmallSizeTailOnCanDeleteVolume (Go), and the
  Rust mirrors volume_load_repairs_tail_on_can_delete_volume /
  volume_load_cuts_damaged_small_size_tail.

Generated with [Devin](https://devin.ai)

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-20 23:40:51 -07:00
GAURIandGAURI c601712cc4 README: describe OLake Go and fix its wiki link (#11880)
README: describe OLake Go and point its link at the renamed wiki page

Co-authored-by: GAURI <174253719+gauriiiiiiiiiiii@users.noreply.github.com>
2026-10-20 23:31:18 -07:00
Chris LuandDevin ab738bfd33 rust volume: repair an unindexed .dat tail at load (#11878)
* rust volume: repair an unindexed .dat tail at load

Mirror Go's repairUnindexedDatTail (#11870): a crash between the .dat\nappend and the .idx append leaves a .dat tail no index entry covers,\nwhich sent the volume read-only on the next load and left acknowledged\nneedles unreadable. Before the index loads, re-index the complete\nneedles in the tail (a size 0 record as the tombstone or empty write\nresolveEmptyTailRecords reads it as) and cut only an incomplete last\nrecord. A tail an interrupted append does not explain is left as it is\nand the integrity check quarantines the volume read-only as before.

Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: tests for the unindexed .dat tail repair

Reload volumes through the real load path after dropping .idx entries\nand cutting or damaging the .dat, mirroring Go's\nvolume_checking_tail_repair_test.go: complete unindexed needles are\nre-indexed (deletes as tombstones, empty writes as empty needles), an\nincomplete last record is cut, and a tail an interrupted append does\nnot explain is left untouched with the volume read-only.

Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: re-index a torn size 0 record as a live empty needle

A delete and a write with no data append the identical size 0 record, so a
recovered one cannot be told apart from the prior index state alone: the
write path accepts an empty overwrite of a live key and indexes it with
size 0, while a delete of the same key publishes a tombstone. Inferring a
tombstone from prior live state turned a lost index row after an empty
overwrite into a deletion -- the object read back not-found although the
write was answered.

Every re-indexed record is now indexed at its on-disk size, the entry
publishWriteRequest writes for an empty write. The other misread is
bounded: a recovered delete reads its key back as an empty object rather
than not-found, and the size 0 record carries no data to resurrect.

Review feedback on the Rust mirror (#11878): the same bug exists here;
fixing both sides identically.

Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: re-index a torn size 0 record as a live empty needle

Mirrors the Go fix in this pull request. A delete and a write with no data
append the identical size 0 record, so resolve_empty_tail_records could
not tell them apart from the prior index state: it treated a zero-size
record of a live id as a tombstone, although the write path indexes the
same bytes as a live size 0 entry. A crash after an empty overwrite of a
live key therefore recovered the object as deleted.

Every re-indexed record is now indexed at its on-disk size, the entry
publish_write writes for an empty write. A recovered delete reads its key
back as an empty object rather than not-found; the size 0 record carries
no data to resurrect.

Review: greptile-apps[bot] and devin-ai-integration[bot] on #11878.

Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: repair a .dat tail left past an empty .idx

The writable index loader creates the .idx before the first needle append,
so a crash after that append leaves a volume whose data nothing indexes.
CheckVolumeDataIntegrity skipped an empty index outright, so the volume
loaded writable and the surviving first needle stayed invisible while
later writes appended past it.

A zero-size index now runs the same repair with tailEntryPos -1: the walk
starts at the first record past the superblock with no anchor needle and
no append timestamp to compare against, gets the same complete-record
validation and incomplete-tail policy, and re-verifies through the normal
tail check once it wrote entries. A tail the repair cannot explain fails
the check, so the volume loads read-only instead of writable on an
unaccounted-for .dat.

Review feedback on the Rust mirror (#11878): the same bug exists here;
fixing both sides identically.

Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: repair a .dat tail left past an empty .idx

Mirrors the Go fix in this pull request. The writable index loader creates
the .idx before the first needle append, so a crash after that append
leaves a volume whose data nothing indexes. Both the repair and the
integrity check skipped an empty index, so the volume loaded writable and
the surviving first needle stayed invisible while later writes appended
past it.

An empty .idx now runs the same repair with no anchor: the walk starts at
the first record past the superblock, and a tail it cannot explain fails
the load like any other repair failure, since every later check skips an
empty index.

Review: devin-ai-integration[bot] on #11878.

Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: sync the .dat before publishing re-indexed .idx rows

A power loss after the repaired .idx rows become durable but before the
.dat tail records they name are flushed leaves the index pointing past
the end of the data file; the next load then keeps the volume read-only
until an operator steps in. Sync the .dat first so the rows never
outlive the records they reference.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: sync the .dat before publishing re-indexed .idx rows

Mirrors the Go fix: a power loss after the repaired .idx rows become
durable but before the .dat tail records they name are flushed leaves
the index pointing past the end of the data file. Sync the .dat first.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-20 23:13:51 -07:00
Chris LuandDevin 222e96d6ab iceberg maintenance: merge bin files across parquet schema spellings (#11875)
* iceberg maintenance: compare parquet schemas by field id in bin-pack merge

parquet.EqualNodes ignores Iceberg field ids, so a column dropped and\nre-added under the same name and type compared equal and was merged,\nfiling the new column's values under the dropped field id. Require the\nfield ids to agree wherever both sides carry one; the v3 layout's\nper-field check uses the same test.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* iceberg maintenance: merge bin files whose schemas differ by field id

A bin used to take the first file's Parquet schema as the writer's and\nrefuse every file that was not structurally identical, so a table never\ncompacted once files had a different column set or REQUIRED-vs-OPTIONAL\nrepetition. A file whose flat columns the writer's schema covers by\nfield id is converted into it (a column the file lacks reads as null);\na file that covers the writer's schema restarts the bin with it, so no\ncolumn is dropped. Equality-delete columns resolve on each file's own\nschema. Renames, type changes, nested columns and files without field\nids stay refused. The sorted and spatial merges are unchanged.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* iceberg maintenance: test bin-pack merge across parquet schema spellings

Regression coverage for the field-id merge: an added column in either\norder, REQUIRED vs OPTIONAL, equality deletes resolved per file, the\nre-added column refused on v1/v2 and v3, and the refusals that stay\n(rename, type change, one-sided columns, an added required column, no\nfield ids).\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* iceberg maintenance: resolve missing delete columns as null in bin-pack merge

An equality delete on an optional column added after a data file was\nwritten no longer refuses the bin: the file reads the column as null,\nso a real delete key cannot match its rows and a null key deletes them.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* iceberg maintenance: write bin-pack merges under the files' union schema

The writer schema is now the union of the bin's file schemas by field\nid, computed in a schema-only pass before any row is written: a column\nstays REQUIRED only where every file has it and spells it so, and the\nmerge refuses only true name/type/id conflicts. Files that need a union\nneither side covers (an added column plus a REQUIRED/OPTIONAL\ndifference, disjoint added columns) now merge in a single pass instead\nof refusing, and progressively wider bins no longer reread and rewrite\nevery earlier file per restart.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* iceberg maintenance: spell the re-added column's test file for codespell

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* iceberg maintenance: resolve equality-delete columns by field id

A column renamed after its data and delete files were written still carries
its field id there; resolving it by id instead of the current name keeps a
rename from turning every equality key null. The missing-column null-key
fallback is now for data files only: an equality-delete file that lacks its
equality column is refused, as the spec requires it to carry them.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* iceberg maintenance: read only the parquet footer choosing a bin's writer schema

The union-schema pass downloaded every input in full, then the merge
downloaded each again. The footer suffices for the schema, and a chunked
file's ReadAt pulls only the chunks the footer spans.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-20 23:13:43 -07:00
Chris Lu a724f4296c s3, s3tables: create buckets and namespaces exclusively (#11872)
* filer_pb: add MkdirExclusive

A taken name leaves the callback as a result, not an error, so filer
failover does not count a lost race as a filer failure.

* s3tables: create the entry with its attributes in one exclusive create

CreateTableBucket and CreateNamespace checked that the name was free and
then wrote the entry in several calls, so a second create in the window
replaced the first caller's entry including its owner. The create now
carries the marker, metadata and tags in one O_EXCL CreateEntry that
answers ErrEntryAlreadyExists with the same 409.

* s3: PutBucket uses an exclusive mkdir

The same window let a plain PutBucket replace a table bucket created
concurrently. The already-exists branch now answers from the entry itself,
since the registry just cached the name as absent, and the admin
auto-create path is exclusive too.

* s3: read the winning entry when a create loses the race

The concurrent-create branch called existingBucketError on a name the
registry had just cached as absent, answering NoSuchBucket, and the
auto-create fallback could assign an S3 owner to a table bucket it met
there. The negative cache is cleared before the existing-bucket error is
built, and a table bucket keeps its owner-less shape.

* s3: fail an auto-create that met a table bucket

A table bucket carries no S3 owner and must not gain one, and the upload
that auto-create ran for must not continue into it either.

* s3: fail an auto-create when the winning entry cannot be read

An unreadable existing entry fell through to success, so the write
continued into an unvalidated entry, possibly a table bucket.
2026-10-20 19:01:08 -07:00
Chris Lu f05d743459 s3: resolve versionId tagging requests to the version-tagging actions (#11874)
* s3: resolve versionId tagging requests to the version-tagging actions

A request for an object's tags with versionId acted on a historical
version but authorized against the current-object tagging action, so an
explicit Deny on s3:*ObjectVersionTagging never applied.

* iam: map PutObjectVersionTagging and DeleteObjectVersionTagging to the tagging action

* s3: only resolve tagging when the query value is empty

The tagging routes match ?tagging with an empty value, so a non-empty one
is not a tagging request and would resolve a tagging grant onto the plain
read or version read that actually serves it.

* s3: keep resolving tagging for any query value

The routes match ?tagging whenever the key is present, so tagging=x is a
tagging request served by the tagging handlers and must resolve to the
tagging actions.
2026-10-20 18:59:19 -07:00
Chris Lu 03a75cbc52 s3tables: only the admin repairs a table bucket that has no owner metadata (#11873)
* s3tables: only the admin repairs a table bucket that has no owner metadata

CreateNamespace wrote bucket metadata naming the caller as owner before any
permission check ran, so an ordinary caller that found the marker without
metadata was written in as the bucket's owner and passed every check after.
A bucket in that state is answered with the operation's own denial; the
admin or a trusted internal handler may still repair it.

* s3tables: repair the marker-only bucket under the entry lock

A creator finishing its own create between the missing-metadata check and
the repair write would have its owner overwritten. The mutation now adopts
whatever metadata is there and writes only while it is still absent.

* s3tables: gate the repair on the Admin action, not an operation grant

hasAdminAction also answers s3tables:*, so a non-admin identity holding
the wildcard operation grant could pass the repair check and write itself
in as the bucket's owner.
2026-10-20 18:58:54 -07:00
dependabot[bot] efa9d304d2 build(deps): bump github.com/klauspost/compress from 1.18.0 to 1.18.7 in /test/kafka/kafka-client-loadtest (#11879)
build(deps): bump github.com/klauspost/compress

Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.18.0 to 1.18.7.
- [Release notes](https://github.com/klauspost/compress/releases)
- [Commits](https://github.com/klauspost/compress/compare/v1.18.0...v1.18.7)

---
updated-dependencies:
- dependency-name: github.com/klauspost/compress
  dependency-version: 1.18.7
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-20 17:49:47 -07:00
github-actions[bot] b7f5d04428 docs: regenerate star history chart 2026-10-21 00:43:44 +00:00
Chris LuandDevin ece2f7a7c2 iam: targeted writes for single-user IAM actions (#11877)
* s3api: persist single-identity IAM changes through the credential manager

CreateAccessKey, DeleteAccessKey, UpdateAccessKey, PutUserPolicy, DeleteUserPolicy, SetUserStatus, UpdateUser and DeleteUser used to load the whole configuration, change one identity and then SaveConfiguration, rewriting and re-propagating every identity. They now write only the identity (and, for rename/delete, the groups and service accounts that referenced it) through the credential manager, which propagates that identity alone. Static identities and skipPersist keep the previous full-save/no-write contract.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* iamapi: persist single-identity IAM changes through the credential manager

Same targeted-write change as the embedded IAM API for the standalone weed iam dispatcher: UpdateUser, DeleteUser, CreateAccessKey, DeleteAccessKey, UpdateAccessKey, PutUserPolicy and DeleteUserPolicy write one identity through the credential manager instead of saving the whole configuration. CreateAccessKey keeps the full save when the handler created a user the store does not know.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* credential/memory: re-key identity on UpdateUser rename

UpdateUser stored the identity under the old username key even when identity.Name changed. Re-key it the way filer_etc does (delete old, save new) so the targeted rename path sees the moved record.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test: cover targeted single-identity IAM writes

Recording-store tests assert each rerouted action uses the single-identity credential store methods and never SaveConfiguration, in both the embedded S3 IAM dispatcher and the standalone iamapi dispatcher, plus the skipPersist and static-identity contracts and a memory-store rename case.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* credential: keep not-found and rename-collision semantics in stores

IamGrpcStore.DeleteAccessKey translates gRPC NotFound to ErrAccessKeyNotFound so a repeat delete stays the no-op the IAM handlers advertise. MemoryStore.UpdateUser rejects a rename onto an existing user instead of overwriting it, matching RenameUser.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-20 17:41:37 -07:00
Chris LuandDevin 610346ae8c iceberg: keep version-hint.text out of remove_orphans (#11867) (#11871)
Generated with [Devin](https://devin.ai)

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-20 13:05:38 -07:00
Javier de la TorreandJavier de la Torre df89afbef4 volume: repair an unindexed .dat tail at load instead of loading read-only (#11870)
* volume: repair an unindexed .dat tail at load instead of loading read-only

A needle write appends the needle to the .dat, then its entry to the .idx,
then answers the client. A crash between the two appends leaves a .dat that
ends past the last indexed needle; on the next start CheckVolumeDataIntegrity
reports "actual N bytes expected M bytes" and the volume loads read-only, and
a complete needle in that tail cannot be read because nothing indexes it.
After a process crash those bytes belong to a write that was never answered;
after a power loss a complete needle can survive in the .dat while its .idx
entry did not, although the client was answered. So complete needles are
re-indexed rather than cut.

CheckVolumeDataIntegrity now repairs the tail before the existing check,
which then runs on the repaired files as before. The repair:

- acts only on a local, version 3 volume within MaxPossibleVolumeSize, with a
  whole number of .idx entries, that is about to load writable (neither
  noWriteOrDelete nor noWriteCanDelete is set);
- requires the needle at the highest indexed offset to match its entry (id,
  size, whole record, CRC);
- walks from the end of that needle to the real end of the file and gives
  every complete needle (header, body, checksum, timestamp and padding inside
  the file, CRC valid, append timestamp later than the needle before it) the
  .idx entry the volume server would have written, in .dat order; a size 0
  record, which is how a delete is appended, is indexed as a tombstone;
- cuts only an incomplete last record: fewer bytes than a header, a record
  running past the end of the file, or a last record whose CRC fails;
- changes nothing otherwise (a damaged record with more data after it, a
  header with id 0 or a negative size, append timestamps going backwards), so
  the volume loads read-only exactly as before.

The .idx entries are written and synced before the .dat is truncated and
synced, so a crash during the repair leaves the needles indexed and the
remainder still past them; the next load cuts it. One warning per repaired
volume names the needles re-indexed and the bytes cut.

Tests: volume_checking_tail_repair_test.go reloads volumes through the normal
load path. Repaired: one and two complete unindexed needles, an unindexed
delete, a partial header, a partial body, a complete needle followed by a
partial one, a last needle with a bad CRC, a partial needle after an indexed
tombstone; each reloads clean afterwards. Left read-only and unchanged: the
last indexed needle with a wrong id or a bad CRC, a bad CRC followed by more
data, zero bytes past the last needle, an index past the data, a volume
persisted read-only, a partial .idx entry, and a clean volume.
TestCheckVolumeDataIntegrityVerifiesDeletionTail now also checks that on a
writable volume its 64 zero bytes are reported and kept.

Fixes #11864

* volume: fail the integrity check when a tail repair cannot complete

A repair that wrote the .idx or cut the .dat and then failed to sync only
logged the error; the check that followed saw the repaired files and the
volume loaded writable on a repair that may not be on disk. Any repair
error other than an unexplained tail now fails the check, so the volume
loads read-only, and the next load repairs again.

A failed .idx append could leave part of an entry behind, which made every
later load reject the .idx as a partial entry. The appended bytes are
truncated away again before the error is returned.

* volume: do not cut a tail record that could not be read

The walk cut a last record whose read failed for any reason, so an I/O
error (EIO) on a complete record erased it. Only a record whose bytes are
damaged (needle.ErrorCorrupted: a CRC mismatch or a body that does not
parse) is cut now; any other read error is returned with both files left
as they are, and the volume loads read-only.

* volume: treat a sized tail record with no data as damaged

A needle with no data is written with size 0. A tail record whose header
gives a body but whose data size field reads 0 parsed as an empty needle,
and ReadBytes checks the CRC only when there is data, so the repair indexed
it with its positive size and reads returned an empty body. Such a record
is now damaged like one with a bad CRC: cut when it is the last record,
otherwise the tail is left alone and the volume loads read-only.

* volume: keep an unindexed empty write as an empty needle

A delete and a write with no data both append a size 0 record, and the
repair indexed every size 0 record in the tail as a tombstone, so an
acknowledged empty write whose .idx entry was lost read back as deleted.
A delete is only appended for a needle that is live, so a size 0 record
of a needle that is absent or deleted (in the .idx and in the tail before
it) is an empty write and is re-indexed with size 0. One of a live needle
stays a tombstone, the reading weed fix and the backup's index
regeneration already give it. The .idx is scanned only when the tail holds
a size 0 record.

* volume: cut torn-tail test records inside their body, not at +40

The partial-record cases cut the .dat 40 bytes into a record. A needle
with up to 7 bytes of data is a 40-byte record, so newRandomNeedle's
random length (1..1023) left it whole about once in 150 runs per case:
the repair then correctly re-indexed it and cut nothing, and the test
failed on the .dat size. The cut now lands halfway through the record's
body, which is always past its header and short of its end.

---------

Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
2026-10-20 13:03:00 -07:00
dependabot[bot] 1253b97303 build(deps): bump github.com/klauspost/compress from 1.15.9 to 1.18.7 in /test/kafka/kafka-client-loadtest/tools (#11866)
build(deps): bump github.com/klauspost/compress

Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.15.9 to 1.18.7.
- [Release notes](https://github.com/klauspost/compress/releases)
- [Commits](https://github.com/klauspost/compress/compare/v1.15.9...v1.18.7)

---
updated-dependencies:
- dependency-name: github.com/klauspost/compress
  dependency-version: 1.18.7
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-20 13:02:26 -07:00
dependabot[bot] a1d8739464 build(deps): bump github.com/klauspost/compress from 1.17.0 to 1.18.7 in /test/kafka/simple-consumer (#11869)
build(deps): bump github.com/klauspost/compress

Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.17.0 to 1.18.7.
- [Release notes](https://github.com/klauspost/compress/releases)
- [Commits](https://github.com/klauspost/compress/compare/v1.17.0...v1.18.7)

---
updated-dependencies:
- dependency-name: github.com/klauspost/compress
  dependency-version: 1.18.7
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-20 13:01:44 -07:00
Chris LuandDevin 1cfd9bbb4c volume: run the durable sync outside the reader-visible lock (#11863)
* volume: run the durable sync outside the reader-visible lock

Reads of a volume waited on that volume's fsync because the write lock
was held across Sync()/sync_all. A new per-volume lock (Go
dataFileSyncLock, Rust data_file_sync_lock) now serializes every .dat
mutation and lifecycle op across append+publish+sync+rollback, while the
fsync itself runs with dataFileAccessLock/the io write guard dropped —
so readers proceed during a sync, and a failed sync's rollback still
cannot clobber another write or a compaction swap.

Same change in Go (volume_write.go syncWrite/processBatch, SyncToDisk)
and Rust (write_needles_grouped/write_synced_run).

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: publish after sync; fix Close/CommitCompact lock order

Address review on #11863:

- Close took dataFileSyncLock before waiting on isCompactionInProgress
  while CommitCompact claims the flag then takes the lock — a deadlock.
  Close now claims the flag before taking the locks (flag -> sync -> access).

- Go now publishes needle-map entries only after the .dat sync, matching
  Rust's ordering: appendWriteRequest/appendDeleteRequest stage the bytes,
  publishWriteRequest/publishDeleteRequest index them after Sync. A failed
  sync rolls back bytes no reader ever resolved — the streamed-read
  offset-invalidation window is gone. processBatch splits runs at repeated
  needle ids (Rust parity) so dedup/cookie checks stay sequential; the old
  snapshot/restore rollback machinery is removed.

- Rust commit_compact (store-lock caller) takes data_file_sync_lock via
  try_lock -> WouldBlock before the write lease; commit_compact_held and
  the tier-down _held paths document/require the caller-held sync lock,
  and the two vol.instance().write_lock() grpc sites take it first. The
  swap can no longer land between a durable write's append and publish.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: restore write watermarks when an inline fsync fails

The syncWrite rollback truncated the .dat but left lastAppendAtNs,
lastWriteAppendAtNs, lastWriteNeedleKey, lastWriteDeleted and
lastModifiedTsSeconds describing the rejected write; recovery and
compaction could read a watermark for a record that no longer exists.
Snapshot and restore them on failure, as processBatchRun already does.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-20 12:58:02 -07:00
Chris LuandDevin 0de6347b11 s3: mask anonymous misses behind AccessDenied unless the bucket is listable (#11857)
* s3: mask anonymous misses behind AccessDenied unless the bucket is listable

Deferred anonymous GET/HEAD authorization only runs once an entry is
selected, so misses returned 404 while existing keys returned 403 — an
unauthenticated existence oracle for bucket names and object keys.

Route not-found responses in Get/HeadObjectHandler through a check that
only discloses 404 when anonymous listing is allowed (policy, identity or
a public-read bucket); otherwise the miss is indistinguishable from a
denied read, matching AWS and 4.48 behavior.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: mask missing-directory misses for anonymous reads too

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: check a fixed request shape when masking anonymous misses

The miss check passed the caller's request through, so query parameters
could retarget the resolved action (?policy) or satisfy an s3:prefix
condition on an unrelated subtree. Each check now runs on a request with a
fixed shape: a ListBucket scoped to the probed key and a plain read of the
key, so the miss only shows when the caller could learn the same fact
directly.

* s3: mask conditional-header misses for anonymous reads

An If-Match conditional on a missing key wrote NoSuchKey directly from
processConditionalHeaders, before the handlers reached the masked miss
path, so anonymous callers could still distinguish a missing key from a
denied read.

* s3: keep the versionId when masking anonymous misses

The read leg must evaluate the action the caller's request resolves to:
a versioned miss is s3:GetObjectVersion, and dropping versionId would let a
plain GetObject grant disclose versions the caller may not read.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-20 12:52:58 -07:00
Chris Lu 6936bbdc3e rust volume: score EC shard placement with the volume's own ratio (#11862)
The auto-select branches of ReceiveFile (is_ec_volume) and
VolumeEcShardsCopy passed the build-default DATA_SHARDS_COUNT into
find_ec_shard_target_location, so its free-slot maths scored every
volume as 10+4 even when the volume's .vif declares a custom ratio
(e.g. 6+2). A wider-ratio volume's shards could be steered onto a disk
too full to hold them all, and a narrower one would be refused slots
it actually has.

Add Store::lookup_ec_data_shards(collection, vid), which answers from
the mounted EC volume first and then from the .vif on whichever disk
holds it (via read_ec_shard_config_across_dirs, so split -dir/-dir.idx
and rebuild-staging layouts are covered), falling back to the default
when no metadata exists yet.

Mirrors the enterprise fix (seaweed-mono c22fa8a15).
2026-10-20 07:40:43 -07:00
Chris Lu 4d06742664 Detect and quarantine duplicate volume ids (#11763) (#11828)
* master: detect and quarantine duplicate volume ids across layouts (#11763)

* shell: volume.check.duplicate_ids audit command (#11763)

* master: persist duplicate-vid quarantine across heartbeats (#11763)

* master: quarantine all copies of a duplicated volume id

Near-simultaneous registrations let each side quarantine only the other,
so arrival order decided which copy stayed writable. A confirmed
cross-layout collision means either copy could be the impostor, so flag
every colliding copy read-only regardless of registration order.

* topology: return a copy from VolumeLayout.Lookup

Callers iterating the returned slice raced with registrations and
removals replacing the location list in place.

* master: release duplicate-vid quarantine when the conflict resolves

A cross-node volume.move briefly shows the same id on two nodes under
different disk layouts; permanent quarantine would pin a valid target
read-only. Keep the first-reported state per quarantined copy and lift
the marks once the conflicting copy unregisters, so only a persistent
collision stays read-only.

* shell: include ec shards in volume.check.duplicate_ids

Ec-only volumes share the global id space. Compare shard collections
against regular ones while allowing shards and regular copies of the
same encoded volume.

* master: keep quarantine release current and cover all removal paths

Track the latest server-reported read-only flag while quarantined so a
release restores the server s true state, and run the release check on
collection deletion and data-node removal too, not only explicit volume
unregistration.

* shell: do not flag ec-only volumes as duplicates

An ec-only volume has no regular copy, so its shard collections had
nothing to compare against and always read as foreign.

* master: quarantine every conflicting layout at the layout level (#11763)

- collect all conflicting layouts, not just the first, so a vid spanning
  three layouts does not leave one writable
- mark quarantine on the location list only: folding it into the stored
  report made the volume digest permanently disagree with the server and
  forced a full volume list every heartbeat
- isAllWritable now honors the location read-only mark so a quarantined
  copy stays out of writables through resyncs and re-registration
- re-check the conflict before keeping marks, and do not clear a node's
  quarantine while it still holds the vid in another layout
- log the full layout keys on both sides of the conflict

* master: push duplicate-vid quarantine to the volume server (#11763)

A quarantined copy stayed writable on the volume server itself for a
client holding an existing file id, since the master-only mark never
reached the server's local write path. Quarantine now also calls
VolumeMarkReadonly (can_delete so expiry still drains), is re-pushed if
the server comes back reporting it writable, and VolumeMarkWritable
runs when the conflict resolves and the server had reported writable.

* master: retry unenforced quarantine marks and restore server writability on release (#11763)

Dial the advertised grpc port, return mark errors, and keep a
quarantine entry until the writable push succeeds so a released
copy does not stay read-only on the server.

* master: treat rp-only layout differences as an in-flight replication change (#11763)

volume.configure.replication rolls through nodes one at a time, so a
volume legitimately reports the old and new replica placement while it
settles. Layouts differing only in rp within the same collection are a
supported transition, not a reused id.

* shell: group ec shards by collection in the duplicate id audit (#11763)

EC shards of one volume legitimately span disk types when
DiskTypePrefer spills, so grouping by disk type reports healthy
volumes as duplicates.
2026-10-20 07:37:33 -07:00
Chris Lu 2874bd05fb Freeze volume growth on topology id mismatch (#11763) (#11820)
* pb: ResumeVolumeGrowth rpc for operator-driven growth resume (#11763)

* master: freeze volume growth on topology id mismatch (#11763)

* shell: volume.growth.resume command (#11763)

* master: replicate the growth freeze and gate it on veteran heartbeats (#11763)

* master: block allocation while the topology id is being minted

DeferAllocation only armed after the topology id persisted, so growth
requests arriving during init ran ahead of the warmup window. Gate
NextVolumeId on an init-pending flag spanning the whole mint.

* master: stash reported topology ids until the local id is minted (#11763)

* drain stashed topology id reports when the id arrives outside the mint path

* persist the growth-freeze flag in the mirror file and raft snapshots

* log hashicorp raft marshal and apply errors in applyGrowthFreeze
2026-10-20 07:34:31 -07:00
Chris LuandDevin 1144bf15d4 rust volume: group-commit fsync uploads on the direct write path (#11860)
* rust volume: group-commit fsync uploads on the direct write path

Each fsync=true upload used to run its own .dat/.idx sync when the write
queue was off, which is the default. Route durable uploads through the
per-volume write workers so a batch shares one sync, matching Go's
writeNeedle2: non-fsync writes, and writes while the server is stopping,
stay on the synchronous path. The queue is now always constructed and
SEAWEED_WRITE_QUEUE is gone.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: document post_handler write routing

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-20 07:32:58 -07:00
Chris Lu 8d2b417c69 iceberg, lance: vend read-only credentials to callers that may only read (#11859)
* s3tables: share the UpdateTable authorization gate

Extract the validation, load, and permission check from handleUpdateTable
into authorizeUpdateTable so the table an update was authorized against is
the same snapshot the mutation writes to. Manager gains
AuthorizeUpdateTable, mirroring AuthorizeCreateTable, so callers can ask the
question without updating anything.

* s3tables: share the UpdateView authorization gate

Extract the lookup, policy load, and permission check from handleUpdateView
into authorizeUpdateView, mirroring authorizeUpdateTable, and expose it as
Manager.AuthorizeUpdateView for callers that need the answer without the
update.

* s3api: scope vended table credentials to read-only on request

VendTableCredentials takes a readOnly flag and tablePrefixSessionPolicy
emits only s3:GetObject for the table prefix when it is set, keeping the
conditioned ListBucket and unconditioned GetBucketLocation statements that
reads still need.

* iceberg: vend read-only credentials when the caller may not write

LoadTable and friends vend storage credentials only for the authority the
caller holds: the vend now checks UpdateTable/UpdateView permission and a
caller that may only read the catalog entry gets a read-only session. Paths
that just passed a write gate (stage-create, view commit) keep the full
session.

* lance: vend read-only credentials when the caller may not write

storage_options vends the same read-only session as the Iceberg catalog
when the caller lacks UpdateTable on the table.

* lance: keep write credentials for callers authorized to declare

A Lance commit is a direct S3 write that never revisits the catalog, so a
caller that just passed CreateTable still gets a writable session; the
UpdateTable gate applies to describe, the read path.

* iceberg, lance: log a failed write check instead of only reading it as denied
2026-10-20 07:16:27 -07:00
7ed26deaae s3: list keys in byte order around directory markers (#11832)
* s3: list keys in byte order around directory markers

* s3: order deferred dir blocks by slash name and hold quota

* s3: fix deferred dir blocks at page and marker boundaries

Review follow-ups on the marker byte-order change:

- A marker inside an emitted "<name>/" block re-descended and re-listed its
  folded contents under delimiter "/" instead of skipping the block.
- A prefix object's "<name>/" common prefix was emitted eagerly at the name
  position, outranking siblings like "a.txt" and repeating across pages; the
  plain key now emits alone at the name position and the block emits the
  common prefix when children exist.
- A recursion starved on quota before listing a deferred subtree returned
  without truncation, dropping keys silently; the block now reports a
  resumable truncation when the subtree still holds entries.
- Clone prefix-object entries with proto.Clone rather than copying the
  protobuf message by value.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: close the hasEntries probe stream on return

* s3: resume delimited listings at the dir block boundary

A deferred directory block emits at its "<name>/" position, but emitDir
left nextMarker at the bare name, which sorts inside the (name, name/)
band and reads as "block pending". Every pass of the listing loop then
re-emitted the same common prefix until maxKeys ran out (999 prefixes,
IsTruncated), and a "<name>/" continuation token trimmed to "<name>" by
adjustMarkerForDelimiter reopened the block and paged forever.

Advance nextMarker to the block boundary the same way the recursive
branch does, and keep a "/" -ending marker intact — it already names the
bound past the block.

---------

Co-authored-by: Devin <devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-20 05:36:22 -07:00
GAURIandGAURI da0d423bd9 Add OLake to README and Admin UI; fix catalog_olake path-style docs (#11858)
* README: mention OLake in the Iceberg lakehouse section

* catalog_olake: fix path-style prose; admin UI: list OLake as a REST catalog client

* admin UI: restore generated file import block, keep only the string change

---------

Co-authored-by: GAURI <174253719+gauriiiiiiiiiiii@users.noreply.github.com>
2026-10-20 04:57:55 -07:00
Chris LuandDevin 2095c905ba server: give heartbeat-options test a topology (#11853)
heartbeatResponse now reads MaxVolumeId and TopologyId off ms.Topo, but
the test built a MasterServer without one and panicked, failing the
package for every PR.

Generated with [Devin](https://devin.ai)

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-19 21:01:31 -07:00
Chris Lu 6b6a268903 volume balance: stop replica-fallback moves between equally full servers (#11851)
* volume balance: unify the per-move convergence guard

Evaluate the post-move utilization comparison through serverUtilization so
the guard also applies when servers lack MaxVolumeCount info (raw-count
mode), where it previously did not run at all.

* volume balance: restrict fallback destinations to improving servers

The score-based fallback inside createBalanceTask could redirect a move to
any valid server, including ones as full as the source — between equally
full servers the move only relocates the imbalance and the next detection
cycle reverses it (#11839). Exclude from eligibleTargets any server whose
post-move utilization would exceed the source's, so a fallback destination
satisfies the same convergence guard as the loop-chosen target.

* volume balance: skip a volume when the chosen target violates placement

When the detection loop's target resolves but cannot host the replica (it
already holds one, or is out of the filtered scope), createBalanceTask now
returns nil so the loop advances to the next volume — matching shell
volume.balance, which continues to the next candidate instead of
redirecting the move to a different server.

The score-based fallback is kept for the case where the target cannot be
resolved at all; its allowed set is already limited to servers the move
would improve.

* volume balance: regression test for replica-constrained targets (#11839)

Reproduces the reported loop: the emptiest server holds the second rp=001
replica of the lowest-ID volumes on the fuller servers, so it is never a
legal target for them. Detection must skip those volumes and send the
movable ones to the under-utilized server, and re-running detection after
applying the plan must never move the same volume twice.
2026-10-19 20:52:30 -07:00
Chris LuandDevin 79daffb2d2 test: retry retryable 503s in cold-start assign test (#11852)
The master intentionally sheds an assign that triggered volume growth
once its wait window elapses (503 + Retry-After); on a slow CI box
cold-start growth can outlast it, so the single-shot assertion flaked.
Honor the documented retry contract like a real client, within a 60s
deadline.

Generated with [Devin](https://devin.ai)

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-19 20:51:13 -07:00
github-actions[bot] 207d06863a docs: regenerate star history chart 2026-10-20 02:55:29 +00:00
Chris LuandDevin 7e348951c2 s3: strip internal metadata headers on auth entry (#11850)
* s3: strip internal object-lock and version headers on auth entry

putToFiler persists Seaweed-X-Amz-Object-Lock-Mode, -Retention-Until-Date,
-Legal-Hold and -Version-Id request headers into authoritative entry
metadata without a permission check, so an s3:PutObject-only writer could
pin COMPLIANCE retention and legal hold on its own uploads, bypassing the
dedicated-permission gate on the standard x-amz-object-lock-* headers.
The server only sets these headers from validated inputs, so drop them in
authenticateRequestInternal alongside the identity headers already
scrubbed there.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: strip internal SSE headers on auth entry

X-SeaweedFS-SSE-S3-Key-Data, -SSE-S3-Base-IV and -SSE-KMS-Base-IV are read
from the request to drive multipart encryption, so a client could inject
its own key material and IVs. The server sets them itself between handler
stages of a request; drop inbound values in authenticateRequestInternal.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: scrub internal headers before the auth-enabled early return

Auth and AuthPostPolicy skip authenticateRequestInternal when IAM is
disabled, so client-supplied internal headers still reached putToFiler.
Hoist the scrub into scrubInternalHeaders and run it unconditionally in
both wrappers as well as the shared auth entry point.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: test internal header scrub through the auth-disabled wrapper

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-19 18:35:20 -07:00
Chris LuandDevin 8b09c96cec s3: keep a signed content-length in the canonical request (#11849)
Co-authored-by: Devin <devin-ai-integration[bot]@users.noreply.github.com>
2026-10-19 18:32:44 -07:00
Chris Lu 0fb6c0e77d filer.sync: unstick sync on chunks the source no longer has (#11845)
* filer.sync: keep the needle-404 clock from resetting on live volumes

The give-up clock for a source chunk that cannot be produced is kept per
volume so a vacuumed volume's files all inherit one wait. But a needle
404 and a volume lookup failure shared that clock, and sourceServed
cleared it whenever any needle on the same volume read fine, so on a
busy cluster the grace never elapsed for a single deleted needle on a
healthy volume and filer.sync retried it forever (#11841).

Give needle not-found answers their own per-volume clock that serving
siblings does not reset.

* filer.sync: an answered 404 needs no liveness probe before skipping

After the missing-chunk gate gives up, onReplicateChunkError only skips
the entry when a re-read of the most recently served chunk succeeds.
That probe can never have been armed on a fresh sync, and its single
stored file id can itself be vacuumed since — either way the entry stays
pinned and the offset never advances.

A needle 404 was answered by a live volume server, which is itself proof
the source is serving requests, so skip on it directly. Keep the probe
for the lookup-failure flavor, where no server answered and the silence
can still mean a source outage.

* filer.sync: skip a replayed version the source rewrote in place

Supersession was decided by entry mtime alone, so a source entry whose
chunks were swapped without an mtime bump — remote cache eviction and
re-cache, an in-place repair — never read as superseded, and its
vacuumed chunks were retried indefinitely.

When a chunk is missing, compare the replayed chunk's file id against
the live entry's resolved chunk set: absent means the replayed version
is obsolete and its event is skipped on the first failure instead of
after the give-up window. The same live-entry comparison now also
decides the entry-level skip in onCorruptChunk/onReplicateChunkError.

* filer.sync: give each missing needle its own grace clock

A 404 is one needle's answer, but the clock was kept per volume: once the
first needle's wait expired, every other needle on that volume inherited an
already-elapsed clock and could be written off on its first sighting. Track
the wait per file id and clear it only when that needle itself serves again,
so a live volume's healthy needles neither shorten nor restart it.

* filer.sync: keep the probe for filer-proxied 404s

ReadPart proxied through the source filer collapses "no volume locations"
into the same 404 a missing needle returns, so that answer cannot stand in
for the liveness probe the way a direct volume read's can. Only treat 404 as
an answered needle when the sink reads volume servers directly.

* filer.sync: only a not-older live entry can supersede a replay

Chunk-list comparison must respect the same ordering the mtime check does:
a live entry older than the replayed version means the source was rolled
back, so its differing chunks say nothing about whether the replayed data
is still needed. Gate the content check on live mtime >= replayed mtime in
both sourceMovedOn and sourceDroppedChunk.

* filer.sync: compare chunk ids without resolving manifests

resolvedChunkFileIds downloaded manifest files to expand chunk lists, so a
manifest sitting on a dead volume made the supersession check itself fail
and left the offset pinned behind a replica the source cannot read. File
ids are immutable needle identities — the unresolved sets already compare
chunk content exactly, and the check no longer performs I/O.

* filer.sync: judge nested chunks by their top-level chunk

A manifest child is not in the entry's chunk list even when the manifest
still references it, so checking the live entry for the child id reported
it dropped and returned the raw 404 — unwrapped, so the offset still
pinned. Thread the entry-level chunk id down the manifest recursion and
consult it for the dropped-chunk check instead.

* filer.sync: let the manifest-child test exit through the gate

Run the worker with a shortened grace period and assert the returned error
is marked errSourceChunkMissing, instead of racing the worker still running
against a restored RetryWaitTime.
2026-10-19 18:11:51 -07:00
Jay HemnaniandChris Lu 60b06b6881 s3: sort ListMultipartUploads by key and honor key-marker (#11833)
* s3: sort ListMultipartUploads by key and honor key-marker

* s3: skip upload folders without an object key in ListMultipartUploads

* s3: order same-key multipart uploads by initiation time

The upload id suffix is now a time-ordered UUIDv7, so sorting by upload id lists a key's uploads oldest first, like S3. Pagination is unchanged: ordering and the upload-id-marker comparison share the same key.

selectMultipartUploads now builds the whole result, including NextKeyMarker and NextUploadIdMarker, and the pagination test pages with the response markers.

---------

Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-19 18:10:06 -07:00
Chris LuandDevin dbcf6850b6 ci: fail the awscli multipart retry loop on the last attempt (#11848)
Co-authored-by: Devin <devin-ai-integration[bot]@users.noreply.github.com>
2026-10-19 17:59:50 -07:00
Chris LuandDevin 04eff7d613 ci: retry the awscli multipart download on transient failures (#11846)
The HEAD on the completed multipart object occasionally gets a 403 from a
signature verification mismatch before the object read settles; retry the
download instead of failing the run.

Co-authored-by: Devin <devin-ai-integration[bot]@users.noreply.github.com>
2026-10-19 17:09:58 -07:00
Chris Lu 4feb30deff Federated volume-id bound via heartbeat stamps (#11763) (#11819)
* pb: heartbeat fields for the federated volume-id bound (#11763)

Heartbeat carries max_volume_id_seen and last_seen_topology_id up, and
HeartbeatResponse echoes max_volume_id and topology_id down, so the
allocation bound is replicated to every volume server.

* master: echo the volume-id bound and fold reported bounds in (#11763)

Every heartbeat response carries the leader current max volume id and
topology id; every heartbeat reported max_volume_id_seen is folded into
the topology bound via UpAdjustMaxVolumeId.

* volume: persist a per-dir vid stamp of the highest id ever seen (#11763)

The store keeps max_volume_id_seen + last_seen_topology_id in a
.vid_stamp file per volume dir, fed by heartbeat echoes, AllocateVolume,
and local volume/EC creation, seeded at boot from stamps and local
holdings, and reported in every heartbeat. A rebuilt master recovers
the bound from the first surviving veteran report.

* volume-rust: mirror the vid stamp and heartbeat echo handling (#11763)

Same mechanism as the Go volume server: .vid_stamp per dir, bound folded
in from add_location scans and allocate/mount calls, echoed fields
applied on heartbeat responses, reported on every heartbeat.

* volume: serialize vid stamp writes and fsync the directory (#11763)

* volume-rust: serialize vid stamp writes and fsync the directory (#11763)

* master: echo the allocation bound to connected servers on change

The bound was only sent at connect time or on digest mismatch, so a
healthy stream never learned later increases. If the high-id server then
went offline and the master lost state, survivors restored a stale bound
and the id could be reissued. Resend the heartbeat response whenever the
bound or topology id moved.

* volume: retry failed vid stamp saves on later updates

A transient write failure left the in-memory bound raised but the stamp
stale, and repeat updates at the same bound never retried. Track a dirty
flag so any later update re-attempts the save.

* volume: cover stamp bootstrap for pre-stamp dirs in test

* volume-rust: retry failed vid stamp saves, mirroring Go

* volume-rust: stamp every dir with the combined bound on add_location

A dir whose own stamp sat below the combined bound kept its stale value
after fold; losing the higher-stamped disk then restarted the server
below ids it had already learned. Rewrite all stamps after each location
folds in.

* volume: stamp mounted volume and ec shard ids (#11763)

* volume-rust: stamp mounted volume ids, mirroring Go (#11763)

* volume: retry dirty vid stamp save on unchanged topology id (#11763)

* volume-rust: retry dirty vid stamp save on unchanged topology id (#11763)

* master: record echoed bound from the sent heartbeat response (#11763)
2026-10-19 17:01:19 -07:00
Chris Lu 117ea58a83 s3tables: accept .puffin deletion vectors in table data/ (#11844)
* s3tables: accept .puffin data files for Iceberg v3 deletion vectors

Iceberg v3 writes deletion vectors as Puffin files under a table's data/
directory, so merge-on-read DELETE and UPDATE on v3 tables were refused
with AccessDenied by the table-bucket upload validator.

Fixes #11842

* s3tables: name all accepted data formats in rejection error

The message stopped at .avro even though .lance and now .puffin are
accepted.

* s3tables: cover deletion-vector filenames in data layout test

Uses the names real engines emit: Spark/Iceberg Java
{seq}-{task}-{uuid}-{counter}-deletes.puffin, UUID-suffixed names, and
the same under a partition directory.
2026-10-19 16:59:08 -07:00
dependabot[bot] ce93660abb build(deps): bump github.com/linxGnu/grocksdb from 1.10.8 to 1.11.1 (#11836)
Bumps [github.com/linxGnu/grocksdb](https://github.com/linxGnu/grocksdb) from 1.10.8 to 1.11.1.
- [Release notes](https://github.com/linxGnu/grocksdb/releases)
- [Commits](https://github.com/linxGnu/grocksdb/compare/v1.10.8...v1.11.1)

---
updated-dependencies:
- dependency-name: github.com/linxGnu/grocksdb
  dependency-version: 1.11.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-19 16:28:08 -07:00
dependabot[bot] 73391a2e2f build(deps): bump github.com/tidwall/gjson from 1.19.0 to 1.20.0 (#11834)
Bumps [github.com/tidwall/gjson](https://github.com/tidwall/gjson) from 1.19.0 to 1.20.0.
- [Commits](https://github.com/tidwall/gjson/compare/v1.19.0...v1.20.0)

---
updated-dependencies:
- dependency-name: github.com/tidwall/gjson
  dependency-version: 1.20.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-19 16:27:54 -07:00
dependabot[bot] 3a30ee177a build(deps): bump github.com/klauspost/compress from 1.20.1 to 1.20.3 (#11837)
Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.20.1 to 1.20.3.
- [Release notes](https://github.com/klauspost/compress/releases)
- [Commits](https://github.com/klauspost/compress/compare/v1.20.1...v1.20.3)

---
updated-dependencies:
- dependency-name: github.com/klauspost/compress
  dependency-version: 1.20.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-19 16:27:28 -07:00
dependabot[bot] 109dd24f52 build(deps): bump github/codeql-action from 4.38.3 to 4.39.0 (#11840)
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.38.3 to 4.39.0.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/v4.38.3...v4.39.0)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-version: 4.39.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-19 16:05:07 -07:00
dependabot[bot] a2f8f4943c build(deps): bump gocloud.dev/pubsub/rabbitpubsub from 0.46.0 to 0.47.0 (#11835)
---
updated-dependencies:
- dependency-name: gocloud.dev/pubsub/rabbitpubsub
  dependency-version: 0.47.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-19 16:03:18 -07:00
dependabot[bot] 4909843748 build(deps): bump modernc.org/sqlite from 1.57.0 to 1.60.1 (#11838)
Bumps [modernc.org/sqlite](https://gitlab.com/cznic/sqlite) from 1.57.0 to 1.60.1.
- [Changelog](https://gitlab.com/cznic/sqlite/blob/master/CHANGELOG.md)
- [Commits](https://gitlab.com/cznic/sqlite/compare/v1.57.0...v1.60.1)

---
updated-dependencies:
- dependency-name: modernc.org/sqlite
  dependency-version: 1.60.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-19 16:03:12 -07:00
Chris Lu 0a96972a99 s3: verify X-Forwarded-Prefix only from trusted proxies (#11843)
* s3api: check the direct peer against the trusted-proxy allowlist

* s3: verify X-Forwarded-Prefix only from trusted proxies

* s3: note that trusted proxies must sanitize forwarded headers
2026-10-19 15:55:18 -07:00
Jay Hemnani 4e5fed9e9e s3: list the size of a directory marker that has a body (#11830) 2026-10-19 04:35:23 -07:00
Chris LuandDevin e7923c14d1 s3tables: honor resource policy Deny on remaining operations (#11825)
* s3tables: honor a table policy Deny for maintenance operations

authorizeMaintenanceTable evaluated only the bucket policy, so a Deny in the table policy never vetoed PutTableMaintenanceConfiguration, GetTableMaintenanceConfiguration or GetTableMaintenanceJobStatus even though they act on the table ARN. The stored table policy is now checked for a matching Deny first via deniedByPolicy, a veto helper that keeps the admin exemption, and resource and bucket tags are loaded into the policy context so tag-conditioned Denies evaluate. readBucketPolicy is renamed to readPolicy and reused for the table path.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3tables: honor the stored policy's Deny on policy management operations

PutTablePolicy, GetTablePolicy and DeleteTablePolicy only evaluated the bucket policy, so a Deny in the table's own policy was ignored and a bucket-level Allow could strip or replace the policy guarding a table. The existing table policy is now loaded and a matching Deny vetoes the call. PutTableBucketPolicy likewise consults the existing bucket policy for a Deny before replacing it. Resource and bucket tags populate the policy context so tag-conditioned Denies evaluate, and policy writes assert the policy authorized against is still in force, returning a conflict when it changed mid-request.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3tables: apply a table policy Deny to tag operations

TagResource, ListTagsForResource and UntagResource evaluated only the bucket policy; a Deny in the table's own policy is now honored when the target ARN is a table. ListTagsForResource and UntagResource also returned early when the resource carried no tags attribute, skipping the permission check entirely; the check now always runs. Policy evaluation uses the canonical resource ARN rebuilt from stored metadata, so a request ARN with a forged account or region cannot dodge the Deny.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3tables: regression tests for resource policy denies

Cover the case where a bucket policy grants an identity s3tables:* while the table policy denies it: the policy, tag and maintenance operations must stay denied so the protection cannot be stripped. Also cover a bucket policy that denies PutTableBucketPolicy under the open fallback, and a tag-conditioned Deny, which vetoes only while the stored tag matches.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-19 03:59:18 -07:00
Chris Lu 51a4c092c8 rust volume: release the store write lock during needle I/O (#11823)
* volume: give Volume interior locks mirroring Go's

Volume's mutable state moves behind interior locks matching Go's
per-volume lock domains, so the store can hand out shared handles
instead of requiring exclusive &mut access:

- io: RwLock<VolumeIo> owns the .dat/remote handles, needle map,
  append cursor, compaction bookkeeping and write watermarks —
  Go's dataFileAccessLock-adjacent mutable set; helpers take the
  guard as &mut VolumeIo so nothing re-locks.
- super_block and volume_info each get a RwLock
  (superBlockAccessLock, volumeInfoRWLock); dir_idx gets its own
  so path building never contends with I/O.
- no_write packs Go's noWriteLock pair into AtomicU8;
  last_modified_ts_seconds and compaction_byte_per_second become
  atomics.

The data-file write lease (dataFileAccessLock) now covers every
mutating entry that previously relied on borrow exclusivity:
writes, deletes, blob writes, set_writable's flag/index swap,
remote-file updates, local-backend swaps, index relocation,
replica-placement rewrite, put_needle_index, close and destroy.
set_read_only_persist leases only its can_delete idx-attach branch
because mark_io_unavailable reaches it under a held lease.
commit_compact takes try_write_lock — callers hold the store lock,
so blocking would stall every volume behind one slow reader;
WouldBlock surfaces as COMMIT_WAITS_FOR_READS for the caller to
retry.

Lock order is fixed: lease -> io -> volume_info/super_block/dir_idx.

* storage: mount volumes as shared Arc<Volume> handles

DiskLocation's map owns Arc<Volume> so the store can hand a caller a
handle that stays valid across store-guard releases, instead of
borrowing &mut Volume under the store write lock.

- volumes: HashMap<VolumeId, Arc<Volume>>; find_volume returns a
  cloned Arc.
- find_volume_mut and volumes_mut are gone — every Volume method is
  &self now, so shared handles cover what exclusive borrows did.
- writable_volume_mut -> writable_volume(&self) -> Result<Arc<Volume>>
  keeps the same admission checks (exists, disk-space-low).
- write_volume_needle/write_volume_needles/delete_volume_needle are
  &self; lifecycle methods that mutate the map or mounted state stay
  &mut self.

* server: run needle I/O with the store guard released

The store write lock previously wrapped needle writes and deletes
end-to-end: process_batch held it across every grouped fsync, and the
HTTP/gRPC fallbacks, tail receiver and fetch-and-write paths did the
same. One volume's slow I/O then stalled admission and metadata on
every other volume.

- write_queue resolves each group's handle via writable_volume under
  a brief store read, then runs write_needles_grouped after the guard
  drops.
- HTTP write/DELETE fallbacks, the gRPC batch-delete EC recheck +
  delete, tail receiver and fetch_and_write local writes, blob writes
  and the quarantine/sync/set_writable/remote-file paths all run
  under store.read() now that every volume method they reach is
  &self.
- Lifecycle calls that mutate the volume map or mounted state
  (mount/unmount/delete, EC shard ops, compaction commits,
  load_new_volumes, heartbeat volume actions) keep store.write().

* test: cross-volume admission proceeds during another volume's write

Regression coverage for the store-lock fix:

- write_queue's test parks a volume-1 write inside its .dat append via
  a new needle_write_hook seam (mirroring needle_read_hook), then
  proves the store still hands out a lifecycle writer and a volume-2
  handle — while volume 1's data-file lease still excludes same-volume
  writers.
- tests/store_lock_repro.rs exercises the public API: concurrent
  read-guarded writes on two volumes, and a stale Arc<Volume> handle
  failing closed after unmount instead of writing into torn-down
  files.

* test: drop store write guards that shared handles made read-only

Test setup sites held store.write() purely to reach find_volume_mut;
with Arc<Volume> and &self methods the guard is read-only (clippy
readonly_write_lock). The volume lease still serializes the writes
inside.

* volume: drop a windows-only read_exact_at that shadows the io helper

cargo's windows build failed E0255: the free fn copied with the
interior-lock conversion collided with the unconditional
crate::storage::io::read_exact_at import, which already carries the
platform branches internally.

* server: release the store read guard before needle I/O

The read guard still ran through write_needle_blob_and_index, the tail
receiver write, the fetch-and-write local write, and the HTTP write and
DELETE fallbacks, so a slow append blocked mount/unmount writers (and,
on writer-preferring locks, readers on other volumes). Resolve the
Arc<Volume> under a short guard and write through the handle after it
drops — the same scoping process_batch already used.

* volume: serialize the read-only mark and its .vif save under the lease

set_read_only_persist only leased the idx-attach branch, so concurrent
mark-readonly and mark-writable calls could interleave flag updates and
.vif writes — including two save_vif calls truncating and writing the
file at once. The lease now spans the whole flag change and save, with
a _held variant for mark_io_unavailable, which reaches this from write
paths already holding the lease (Go: markVolumeReadonly runs under
dataFileAccessLock).

* volume,server: span the tier-down transition under one write lease

update_remote_files cleared the remote-only write restriction and
released its lease before open_local_dat_backend attached the local
file, so a write could slip into the gap — pass check_writable, then
fail appending to a missing dat_file. The handler now holds the lease
across the reference trim, .vif save, and backend swap (and the rollback
re-insert), via _held variants; the store read guard is scoped to the
handle lookup. The swap still lands after the .vif trim — the ordering
that keeps a crash from pointing the volume at a deleted remote object.

* server: run a batch's volume groups on separate scoped threads

A single batch can hold writes for several volumes; running the groups
sequentially made every later volume wait behind a stalled append on an
earlier one — the same head-of-line blocking the store guard release
fixed, one level up. Each group now gets a scoped thread, preserving
order within a group while independent volumes progress in parallel.

* storage: close a volume handle a mount slot replaces

With shared handles, inserting a new Volume over an existing map entry
left the old handle alive in callers — two Volume instances could then
write the same files with separate in-memory indexes. set_volume and
create_volume now close() the replaced handle so its writers fail
closed, matching what unload_volume already does.

* test: key the write hook off a reserved needle id

The hook registry is process-global and keyed by needle id, and other
parallel tests write NeedleId(1) — an unrelated write could enter the
barriers and leave the test's real writer parked forever.

* server: wait out readers for vacuum commit without the store guard

commit_compact try-locked the write lease because callers held the store
write lock, returning COMMIT_WAITS_FOR_READS while a download was in
flight — and the master reads that as a failed vacuum, discarding
completed compaction work. The commit RPC now resolves the handle under
a brief store read and blocks on the volume's lease, matching Go's
CommitCompact under dataFileAccessLock: a slow reader stalls only this
volume's vacuum. The try-lock path stays for callers that still hold the
store lock.

* server: scope the remaining store read guards to volume lookup

Same class as the write-path fix: the batch-delete cookie check and
tombstone, the query read, volume_needle_status, and the local chunk
read all kept the read guard through the needle operation, so slow I/O
there could still stall store writers. Each now resolves the Arc<Volume>
under a short guard; the EC rechecks keep their guards.

* seaweed-volume: fail closed on detached volume handles

A handle resolved under a brief store read can outlive its mount: an
unmount+remount between lookup and lease leaves the Arc pointing at a
closed Volume while a new instance owns the files. Mark the handle in
close() and reject it under the write lease in commit_compact_held,
delete_needle, check_writable, set_read_only_persist and
update_remote_files_held; the tier-down handler also checks the flag
before deleting a remote object its replacement still reads.

* seaweed-volume: drop the store guard before mark-readonly waits on the lease

mark_volume_readonly held a store read across set_read_only_persist, which waits out the volume's leased readers; a slow download would stall every lifecycle operation behind the guard. Resolve the handle under a short read, then mark off the guard; a detached handle now maps to not_found.

* seaweed-volume: retry batch deletes that raced an EC mount

The store guard now drops before delete_needle, so ec.encode can mount shards between the pre-delete recheck and the tombstone; the encode then removes the .dat and the needle resurrects from pre-tombstone shards. Re-check has_ec_volume after each successful delete and return the retriable 503 so the filer requeues onto the EC path.

* seaweed-volume: one write-queue worker per volume

A stalled batch kept later batches for other volumes queued behind it: the single worker awaited process_batch before receiving again. Go's volume_write.go keeps one batching worker per volume, so give each volume its own FIFO worker spawned lazily, bounded by a global concurrent grouped-I/O semaphore and a global queued-request bound; workers exit only when their channel closes, and submit refuses unknown volume ids before spawning.

* seaweed-volume: move the queued-write permit inside the request

submit held the global queued-slot permit across the send and the response wait, so a canceled submit freed the slot while the worker still held the needle; repeated cancels could overflow the bound. The request now owns the permit until the worker drops it.

* seaweed-volume: retire write workers whose volume left the store

worker_for kept a sender per volume forever, so servers that mount and retire volumes accumulated idle workers. A worker now polls while idle and retires once its volume leaves the store: close the channel so racing senders fail fast, drain accepted writes, then release the map slot — a replacement worker can never start while this one still holds queued writes, so same-volume FIFO survives retirement.
2026-10-19 03:00:26 -07:00
Chris Lu 0b3ed4502f Design + master mirror of max volume id outside raft (#11763) (#11818)
* design: durable volume-id allocation boundary (#11763)

* master: mirror max volume id to a file outside raft (#11763)

The allocation bound lived only in raft state, so a master that lost it
rebuilt the bound from whatever volume servers were connected and could
reissue ids held by offline servers.

NextVolumeId now persists the bound to <mdir>/max_volume_id before the
raft apply, and both raft implementations persist on apply and recovery
so follower mirrors stay current. The file also carries the topology id,
which survives raft state cleanup the same way the snapshot recovery did.

* master: serialize and fully sync the max volume id mirror (#11763)

* master: rewrite the mirror when the topology id changes at the same bound

A same-bound raft apply carrying a new topology id was skipped by the
lastPersistedMaxVolumeId check, leaving the mirrored topology id empty
until the next allocation. Track the mirrored topology id too and
rewrite when either value differs.

* drop the design doc from the PR stack
2026-10-19 02:42:42 -07:00
Chris Lu ab559bca76 storage: hash index-metric bloom filter with xxhash
The reverse index walk dedupes seen NeedleIds with a BoomFilters bloom
filter. Its default kernel is FNV-1, whose final step only XORs the last
input byte into the hash's low bits, so keys sharing the high 7 bytes of
a big-endian NeedleId land on correlated probe positions and false
positives arrive in bursts of hundreds instead of the ~0.001 target
rate. Each false positive hides a live-latest key and inflates the
rebuilt deletion counters, which is what made
TestFastLoadingNeedleMapMetrics flake past its +256 bound in CI
(observed DeletedCount drift of 267).

Swap the kernel for xxhash, which fully diffuses the key bytes; the
observed false-positive count in 300 local runs of the test drops to
max 2 (avg 0.23).
2026-10-19 01:47:25 -07:00
d04d838ebd ec: compare live idx NeedleId sets before encode (#11812)
* ec: compare live idx NeedleId sets before encode

Refuse encode when replicas' live .idx NeedleId sets differ. Walk only
live rows (skip deleted and Size<=0) and compare NeedleId, ignoring Size
so MIME/flags drift does not abort a matching id set. Mark the volume
read-only for encode and reject writes after that.

* fix(ec): compare all replicas then pick capacity source

Log matching live idx sets with "replicas are consistent" for the
shell CI check. After readonly, compare every replica's live NeedleId
set, then require the encode source to have FreeVolumeCount>=2.
Empty live-idx vs non-empty refuses encode instead of skipping
superblock stubs. Drop the FileCount rebind after the idx gate.

* volume: mark readonly under the write lock so checked appends finish first

* ec: fix readonly deadlock and restore worker-host source preference

MarkVolumeReadonly held dataFileAccessLock while calling
reopenIdxForWrite, which re-acquires the same mutex and deadlocks
TestMarkVolumeReadonlyCanDelete. Split out reopenIdxForWriteLocked for
callers already holding the lock, matching the existing Locked pattern.

The worker task lost its worker-host source preference when the idx gate
replaced the replica union. Late-bind the source to a replica on the
worker host using only the checked location set, with no FileCount
rebinding. Drop the now-unused helpers, trim comments added by the
earlier commits, and rename tests whose names still claimed convergence
while asserting refusal.

---------

Co-authored-by: Chepurnov Simon <195125934+CHEPURNOV-S-S@users.noreply.github.com>
Co-authored-by: Devin <devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-19 00:56:17 -07:00
Chris Lu 10ce19cb17 test: poll for TopologyId convergence in multi-master failover tests
A restarted goraft master wipes its raft state and only learns the
TopologyId after the leader re-admits it and replays the entry. That
trails HTTP readiness by several seconds, so the one-shot check in
assertTopologyIdConsistent raced the propagation (CI run 39199033687:
TopologyId applied on the follower ~2s after the test gave up).
2026-10-19 00:55:41 -07:00
Mohd Quamar TyagiandMohd Quamar Tyagi e20640acb4 fix(admin): improve Total Volumes count contrast (#11817)
fix(admin): improve Total Volumes count contrast on dashboard

Co-authored-by: Mohd Quamar Tyagi <Tyagiquamar@users.noreply.github.com>
2026-10-18 21:34:09 -07:00
Chris Lu 8ac5133abe admin: default to port 12535, keep 23646 as a best-effort compat listener (#11816)
* admin: default to port 12535, keep 23646 as a best-effort compat listener

The default admin HTTP port 23646 derives a worker gRPC port of 33646
(port + GrpcPortOffset), inside the Linux default ephemeral range
(32768+). One of our own startup dials can claim 33646 before the admin
binds it, killing startup. This is the same failure #10209 fixed for
the test port allocator, still reachable via the mini default (#10838)
and via standalone `weed admin` defaults.

Move the default admin port to 12535 (gRPC 22535) everywhere, keeping
the http+10000 convention so worker -admin derivation, the Rust worker
and host:http.grpc addressing keep working:

- weed admin -port, weed mini -admin.port
- weed worker -admin, the plugin worker fallback, Rust worker and
  weed-lance-worker defaults
- help text, README, helm values, docker compose, tests

So old workers, bookmarks and scripts keep working, admin also serves
the same handler and the worker gRPC server on the previous default
(-port.legacy / -admin.port.legacy, default 23646, 0 disables). The
legacy gRPC port binds best-effort: it lives in the ephemeral range, so
a failed bind only warns instead of reintroducing the startup failure.
weed mini reserves it early like the primary port and keeps it out of
its own port relocation. grpc.Server tracks every listener handed to
Serve, so the compat listener stops with the primary.

Also warn when an explicitly configured admin port puts the derived
gRPC port at or above the ephemeral floor, the other fix suggested in
the issue.

Supersedes the pinned-grpc approach in #11814: pinning -admin.port.grpc
off the +10000 convention breaks external workers that derive the admin
gRPC port from the HTTP address. ClickHouse/ClickHouse#115512 pins
-admin.port on their side; this makes that pin unnecessary.

Fixes #10838

* admin: let services win the legacy ports, close compat listener on failure

Review findings on the compat listeners:

- Do not put the legacy ports in mini's reservedPorts: an explicit
  -s3.port=23646 or -master.port.grpc=33646 must be honored, not rejected
  or stolen by the early reservation. Mini now checks the finalized port
  set and skips only the conflicting compat listener, so HTTP and gRPC
  compat degrade independently and the service keeps its port.
- Also skip the legacy HTTP listener when a mini service owns the port:
  a wildcard-bound service can still leave loopback bindable, which would
  let the compat listener shadow it for loopback clients.
- Close legacyServer's raw listener in the deferred cleanup: Serve may
  not have registered it yet when startAdminServer returns early.
- Helm: add admin.legacyPort (default 23646) and expose the legacy
  http/grpc ports through the admin Service, container ports and
  NetworkPolicy so older workers resolved via the Service keep working.

* helm: only pass -port.legacy when it differs from the binary default

ct install deploys the released image with the new chart, and 4.48 does
not know -port.legacy, so the admin pod crash-loops. Omitting the flag at
the default value is behavior-identical on new binaries (their default
is also 23646) and keeps older releases installable; legacyPort: 0 or a
custom port still renders the flag.
2026-10-18 21:21:05 -07:00
github-actions[bot] d3ee817b2e docs: regenerate star history chart 2026-10-19 00:47:21 +00:00
863efb3d35 webdav: delete a folder with its contents (#11811)
* webdav: delete a folder with its contents

* webdav: guard the recursive delete against stale and confined paths

- propagate stat lookup errors instead of returning an error-bearing
  FileInfo: OpenFile and DELETE would otherwise take the recursive
  removeAll on a path whose type was never confirmed, wiping a
  directory on a transient filer error
- refuse wrappedFs.Rename from or to the served root, matching the
  RemoveAll guard and webdav.Dir
- normalize trailing slashes in removeAll so the delete names the
  entry itself, and refuse the root for every caller

* webdav: keep file-only deletes shallow and refuse ancestor destinations

- removeAll used by OpenFile and failed-upload cleanup now deletes
  non-recursively, so a folder swapped in after the stat check is not
  emptied by a pending file delete
- MOVE and COPY whose destination contains the source are refused
  before the handler removes the destination, which would otherwise
  take the source down with it

* webdav: also refuse a destination inside the source

a MOVE or COPY into its own subtree recurses into the destination it
just created; refuse it with the same guard instead of burning the
recursion budget for a 500

* webdav: allow a shallow copy into the source subtree

a COPY with Depth: 0 creates only the destination collection and never
reads children, so it is safe inside the source; only deep copies and
moves are refused there

* webdav: refuse a copy/move whose source is unreadable

* webdav: cover the missing-source refusal in the guard test

* webdav: 500 on a source lookup failure, 404 only when absent

---------

Co-authored-by: Chris Lu <chris.lu@gmail.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-10-18 12:57:10 -07:00
Chris Lu 8cb42cd69f build(deps): bump github.com/seaweedfs/raft from 1.2.1 to 1.2.2 (#11815)
Brings in the raft snapshot-recovery fixes: a recovered follower returns
to Follower so it can campaign when the leader goes away (masters on
ephemeral -mdir no longer wedge leaderless), and currentTerm stays
monotonic and persisted through recovery.
2026-10-18 12:56:29 -07:00
Chris Lu 6a594da739 filer: free orphaned hard-link chunks and remote objects in delete sweeps (#11809)
* filer: free shared chunks when a folder delete drops the last hard link

doBatchDeleteFolderMetaAndData skipped chunk collection for every hard-linked child, relying on maybeDeleteHardLinks to retire the shared inode record. When the delete removes the last link the record is gone and nobody else will ever free those chunks. After the link callback runs, check whether the record survived; if it did not, queue the entry's chunks for deletion.

* filer: free orphaned hard-link chunks in the replayed delete sweep

The subtree sweep on a replayed directory delete skipped every hard-linked child, so a link that was the inode's last owner leaked its shared chunks. Check the inode record after the store delete; when it is gone, queue the child's chunks like any other swept entry.

* filer: mirror remote-object cleanup in the replayed delete sweep

Children a lagging peer alone still holds under a replayed directory delete were dropped with plain store deletes: their remote objects leaked and lazy remote fetches could resurrect them. The sweep now records event-stamped tombstones and deletes swept non-directory children through maybeDeleteFromRemoteIfUnchanged, which carries the same stat guard the write-back daemon's processDeleteEvent applies plus a modification-time check — a delayed replay cannot erase an object another writer replaced, even a same-size one. A failed remote delete fails the replay so the event is retried instead of orphaning the object. The directory's own tombstone comes from the event pipeline at the event timestamp and covers the subtree. Metadata-only deletes and cross-cluster events keep the origin's semantics.

* filerstore: return real errors from KvGet on MongoDB, ArangoDB, Elastic

These stores mapped every lookup failure to ErrKvNotFound, so a database error looked identical to a missing key: hard-link records could appear gone while links still owned their chunks, and path timestamp reads could treat committed state as absent. Return the underlying error and keep ErrKvNotFound for genuinely missing keys.

* filer: bind remote-object deletes to the observed object version

A guarded stat-then-delete still lets a replacement landing between the two requests go down with the deleted object. Add an optional conditional-delete operation to remote storage clients — S3 If-Match on the live ETag, GCS generation match, Azure IfMatch — and use it everywhere a delete is guarded by a stored stamp: the write-back daemon's delete, rename-source, and pending-rename paths, and the replayed-delete sweep. Backends without it keep the guarded stat-then-delete.

* filer: restamp the remote entry after a metadata-only remote update

Changing Content-Encoding rewrites the object in place on S3 and advances its LastModified, while the recorded stamp keeps the old mtime. A version-checked delete would then mistake the object for a replacement and keep it forever. Re-stat after UpdateFileMetadata so the saved stamp describes the live object.

* filer: mirror the remote-delete ownership checks in the replayed sweep

The sweep deleted swept children's remote objects on the listing's snapshot
alone. Two gaps: a remote-only entry that took the path since — one a commit
record may not cover, such as a lazy fetch materialized from the remote —
reads the object as its only copy, and an entry removed while its rename
copy was pending leaves the object at the source key orphaned. The per-child
delete now runs under the child's entry lock with the commit-time check
repeated, keeps a remote-only entry that holds the path, and releases the
pending rename's source under the same ownership and stamp conditions the
write-back daemon's releasePendingSource applies.

* remote.sync: restamp only the object the metadata update touched

The live stat after UpdateFileMetadata was saved unconditionally, so a
replacement written between the update and the stat stamped the entry with
an object it never described — and a later delete would then pass the
freshness guard on the replacement's data. The restamp now requires the live
object to still match the stored stamp on size and ETag; a different object
keeps the old stamp, which the delete guard reads as replaced and keeps.

* filer: skip unbudgeted reader-cache test on 32-bit

TestReaderCacheUnboundedWithoutBudget proves the reader cache has no
implicit memory budget by keeping 96 concurrent 4 MiB downloads in flight
(>256 MiB). On linux/386 the weed/filer test binary already runs near the
4 GiB address-space limit, so the burst reliably crashes it with
"out of memory" mid-suite.

* filer: bail out of a swept child lock instead of waiting on it

Replay already holds the event's path locks when the sweep reaches a
descendant. A local rename can hold that descendant while waiting on one
of the replayed paths, so blocking on the child lock deadlocks both. The
sweep now tries the lock and reports ErrLockContended — transient, so the
replication retry replays the event once the local write settles.

* remote: report the stamp a metadata update produced

A metadata write can rewrite the object it touches — S3 changes
Content-Encoding through an in-place copy, which can mint a new ETag —
so restamping from a fresh stat can reject the very object the update
produced, or describe a replacement that landed in between.
UpdateFileMetadata now returns the RemoteEntry of the object it wrote
(S3 from the copy result, GCS from the update attributes, Azure from a
post-update stat); nil means the object was untouched and callers keep
the stored stamp.

* remote: restamp Azure from the metadata write's own etag
2026-10-18 12:23:30 -07:00
eea3f232da iam: scope CreateServiceAccount's ParentUser by policy resource (#11736)
* iam: scope CreateServiceAccount's ParentUser by policy resource

AuthorizeIamAction evaluated the iam:CreateServiceAccount grant against a
hardcoded arn:aws:iam:::* resource, so the policy's Resource never had a
say; #11218 then bound the target to the caller outright. Evaluate
self-target actions against the target identity's ARN instead: a wildcard
resource keeps the deliberate any-target grant, a scoped resource confines
issuance to the named parent identity, anything else is denied.

buildResourceARN gains the ARN pass-through buildS3ResourceArn already has,
so iam: ARNs reach the policy engine intact on the attached-policies route.

Closes #11735

* ci: retrigger after unrelated weed/storage metrics flake

* iam: fold IAM-ARN matching into awsIAMMatch, drop debug test

- remove zz_debug_test.go (local debugging leftover)
- fold awsIAMMatchResource into awsIAMMatch: the arn:aws:iam:: value
  prefix selects case-sensitive matching, so no separate name is needed
- trim comments

---------

Co-authored-by: batonogov <batonogov@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-18 11:48:01 -07:00
Chris Lu 4a57432488 s3: list children of the directory a ListObjects marker names (#11805)
* s3: resume into the directory a listing marker names

* s3: keep delimited marker resumes exclusive

* s3: descend a marker-named directory without repeating its key

* s3: emit marker-named dir objects but keep internal folders hidden
2026-10-18 11:44:29 -07:00
Jay HemnaniandChris Lu 006d4150f0 webdav: create a folder under its own name (#11810)
* webdav: create a folder under its own name

* webdav: reuse NormalizePath for the MKCOL entry split

---------

Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-18 11:43:19 -07:00
Chris Lu ab39e3a51b Rust volume: stamp ExpireAtSec into the live volume_info on .vif persist (#11813)
The worker fix reads the source's TTL destroy time through
ReadVolumeFileStatus, which returns the in-memory volume_info. Go's
SaveVolumeInfo stamps volumeInfo.ExpireAtSec before writing; Rust's
save_vif wrote it only to the serialized file, so a Rust source kept
reporting the stale creation-time value and the EC .vif would carry a
wrong expiry. Stamp the live volume_info first, mirroring Go and
save_volume_info.
2026-10-18 04:29:48 -07:00
Chris Lu 8c55425d55 s3tables: tighten legacy OIDC subject re-anchoring (#11808)
* sts: recognize issuer-scoped parent user ids

ComputeParentUser output has a fixed shape (base64url SHA-256); let
callers distinguish it from bare subject strings.

* s3tables: keep legacy owner re-anchoring off local and scoped account ids

Re-anchoring a stored bare subject under the single trusted issuer must
only apply to strings that could actually be a legacy OIDC subject. A
stored owner naming the reserved or a configured local account, or an
already issuer-scoped id, can otherwise be claimed by any caller whose
token asserts that string as sub.

* s3tables: re-anchor legacy policy principals in map and array forms

A stored bare subject can appear as {"AWS": sub} or inside a principal
list, not only as a top-level string; re-anchoring only the string form
left legacy Deny statements unable to match the caller.

* s3tables: keep built-in legacy exclusions independent of the account checker

A missing local-account predicate must not disable re-anchoring entirely:
the reserved accounts, ARN-shaped values, and scoped-id shapes are never
OIDC subjects regardless, while legacy Deny statements for plain subjects
still need to resolve so a denied caller cannot fall through to an
identity grant.

* s3tables: exercise legacy Deny against an owner grant

The previous case denied the caller regardless of whether the statement
principal matched; make the caller the owner so a missed match shows.
Cover the array Deny form alongside the AWS map.
2026-10-18 02:43:03 -07:00
Chris Lu 88d9926b35 s3api: recheck suspended-version condition under the lock when the filer ignores condition_alt_key (#11806)
* s3api: recheck condition under lock when routed filer ignores condition_alt_key

A filer that predates condition_alt_key silently drops the field and
evaluates a suspended-versioning condition against the empty null slot
alone, so a matching If-Match still gets a false 412. Recheck under the
object lock instead of trusting that response: the lock path resolves the
same entry the read path serves. Genuine precondition failures still
return 412 from the recheck.

* s3api: cover suspended If-Match against a filer ignoring condition_alt_key

* s3api: resend against the resolved entry instead of the lock-path recheck

Rechecking under the gateway object lock does not serialize against
routed writers, which commit under the filer's path lock; a concurrent
routed PUT could land between the recheck and the create and be
overwritten with a stale precondition. Resend the same transaction with
condition_key set to the resolved version file so the condition and the
write stay under the same filer lock. When the resolved entry is the
null slot, a delete marker, or unresolvable, the resend targets the
object path and fails closed.
2026-10-18 02:23:00 -07:00
Chris Lu e596be10f0 s3api: evaluate object-tag policy conditions on delete, ACL and lock operations (#11807)
* s3api: resolve the target version in tag-condition policy lookups

* s3api: apply tag-conditioned policy to object deletes

* s3api: apply tag-conditioned policy to object ACL reads and writes

* s3api: apply tag-conditioned policy to retention and legal hold

* s3api: cover tag-conditioned delete, ACL and object-lock rechecks

* s3api: let tag-conditioned grants redecide denied batch deletes

* s3api: resolve the tagged version in delete policy tests
2026-10-17 22:30:32 -07:00
Chris Lu 2ff7888546 filer: honor a storage rule's rack/data-center constraint instead of falling back (#11804)
filer: stop widening a storage rule's location on assign failure
2026-10-17 22:28:53 -07:00
Nabil RagabandClaude Opus 5.5 6ab6926c94 s3: read chunks from HTTPS volume servers with the global HTTP client (#11800)
readChunkRange sent the volume lookup's http:// URL through a bare
http.Client, so with [https.client]/[https.volume] configured every read of
an SSE-KMS object larger than one chunk (and every ranged read through this
path) failed with "client sent an HTTP request to an HTTPS server" and the
GET returned 500. Normalise the URL's scheme and use util_http's global
client, which carries the [https.client] TLS configuration, as the other
volume reads already do. Drop the now-unused volumeServerHTTPClient.

Fixes #11799

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 21:37:23 -07:00
Chris Lu e6a0534e3b s3api: make ambiguous routed-PUT recovery atomic and stop transaction replay (#11787)
* filer_pb: map PRECONDITION_FAILED to a sentinel error

A conditional CreateEntry that the owner's path lock refuses comes back as
FilerError_PRECONDITION_FAILED; mapping it to ErrPreconditionFailed lets the
S3 gateway tell a rejected precondition apart from a transport failure and
report 412 instead of retrying the write.

* s3api: make the ambiguous-PUT recovery create conditional under the lock

createAfterAmbiguousRoute verified state and then issued an unconditional
create, so a write landing between the lookup and the create was still
clobbered by the stale re-commit (issue #11631). The recovery create now
carries a WriteCondition — IF_NOT_EXISTS for a proven-absent entry,
IF_ENTRY_EQUAL for a directory it may need to write onto — which the owner
filer evaluates under its per-path lock, making the check and the write
atomic. A refused precondition loops back to re-read the state instead of
overwriting whatever won the race.

* s3api: do not fail over a routed transaction past a possibly-sent request

withFilerClientFailover retried the callback on the next filer on any
transport error, so an ObjectTransaction whose response was lost could be
re-applied over a write that committed in between (issue #11631). A gRPC
error cannot distinguish "never sent" from "committed", but channel
readiness can: a non-replayable callback now runs only on a connection that
just reached READY, and failover continues only over channels that never
came up — the one failure mode that provably sent nothing.

* s3api: run ambiguous-route recovery for empty PUTs too

The recovery was gated on a nonempty chunk list, so a zero-byte PUT whose
routed transaction lost its response still fell back to an unconditional
create that could overwrite a newer write and delete its chunks. Chunk-count
mismatch on an empty upload list is enough to distinguish a superseding
entry from this PUT's — there is nothing to save by skipping the check.

* s3api: keep conditional recovery creates off the replay path

The conditional create in the ambiguous-PUT fallback rode the replayable
WithFilerClient failover, so a committed create whose response was lost
could be re-sent to another filer — where a delete in between would let
IF_NOT_EXISTS pass again and resurrect an entry pointing at reclaimed
chunks. Conditional creates now run through the non-replayable failover;
an ambiguous result falls through to confirmCreateLanded, which resolves
the committed entry rather than resending.

* s3api: do not mark a filer unhealthy for an answered request

A reply that reached the callback — including an application-level failure
like a refused write precondition — proves the filer is alive. Checking
received before the health bookkeeping keeps definitive application errors
from tripping the failure circuit or flagging a live routed owner as
unreachable.

* s3api: do not claim an empty stored entry as an ambiguous PUT's own

Two chunkless entries compare equal under sameFileChunks, so an older
empty object satisfies "the route landed" even though it predates this
write — recovery would return success while leaving the request's
Content-Type and metadata unapplied. A landed PUT stores the entry
exactly as built, so require mime/size/md5/content and the extended
attributes to match before claiming it; anything else stays uncertain
and the client retries.

* s3api: prove ambiguous empty PUTs by write id and pin recovery to the owner

- Conditional recovery creates are sent to the routed owner alone instead of failing over: the condition only serializes with routed writes under the owner's per-path lock, and another filer's forwarding is only as correct as its ring view. An unreachable owner now yields an uncertain result rather than a create under a different serialization boundary.
- A proven-absent path no longer satisfies an empty, chunkless PUT: the routed commit may have landed and been deleted, and there are no chunks to verify, so IF_NOT_EXISTS could resurrect a removed object. Recovery returns uncertain for the client to retry.
- Chunkless identity moves from metadata equality to a per-write id (x-seaweedfs-write-id) stamped into Extended: an identical older object can match mime/md5/content but not the id. Entries without this write's id are not claimed, and confirmCreateLanded can now positively confirm a committed empty object instead of always reporting failure.
2026-10-17 21:36:39 -07:00
Chris Lu e6633146d8 filer: shrink zip spool threshold in tests to fix 32-bit OOM (#11801)
* filer: shrink zip spool threshold in tests to fix 32-bit OOM

TestSpoolZipReaderTempFile and TestSpoolZipReaderLimit each allocate
~33-70 MiB to cross the 32 MiB in-memory spool threshold. On linux/386 the
weed/filer test binary already runs near the 4 GiB address-space limit, so
these allocations reliably crash it with "out of memory" mid-suite.

Make zipSpoolMemoryLimit a var and drop it to 1 KiB in those tests, which
exercises the same memory-vs-temp-file spool paths with KiB-sized buffers.

* s3 lifecycle: update stale runShard and WalkerFunc call sites in tests

Recent master changes added a walks param to runShard (#11797) and made WalkerFunc take []int, but replay_view_test.go and run_bounded_test.go still use the old signatures, breaking go vet and test builds on merge.
2026-10-17 20:30:21 -07:00
Chris Lu fe8221c453 fs.meta.load: wait for in-flight entry creation before returning (#11776)
* fs.meta.load: wait for in-flight entry creation before returning

On io.EOF and error paths the command returned while CreateEntry
goroutines were still running, so trailing entries could be silently
dropped once the shell exited. Wait for them on every exit path and
guard the shared error with a mutex.

* fs.meta.load: surface worker errors on read/unmarshal failure paths

The ReadFull and proto.Unmarshal error paths waited for in-flight
CreateEntry workers but returned only the local error, hiding a failed
create from the caller. Check loadErr after wg.Wait() like the other
exit paths so the worker error is reported.

Extract the load loop into loadEntries so the concurrency and error
propagation can be exercised by unit tests, and add regression tests
covering delayed creates at EOF and on corrupt input plus worker error
reporting.
2026-10-17 20:28:43 -07:00
github-actions[bot] e32d8ca2a6 docs: regenerate star history chart 2026-10-18 00:47:40 +00:00
Chris Lu 602ddb31f2 s3: count stored versions as bucket contents on delete (#11775)
* s3: count stored versions as bucket contents

A <key>.versions directory holding version entries is user data; skipping
it let bucketHasUserObjects report an empty bucket while the recursive
delete dropped every stored version.

* test: rename stale reserved-folders case in bucket delete test

.versions directories are now descended into, so the case only holds
because the folder is empty.
2026-10-17 15:25:56 -07:00
Chris Lu 44bb697495 s3api: hydrate runtime policies on propagated IAM updates (#11784)
* s3api: hydrate runtime policies on pushed identity upserts

PutIdentity propagation (used on every IAM write when the credential
store has a master client) carries the persisted identity proto. That
proto records inline policies only as flattened legacy Actions; their
runtime __inline_policy__ names live in the policy store and were
dropped by the merge. For the interval until the next full reload, an
identity whose policy denied e.g. DeleteObject routed to legacy
CanDo, where "Write:bucket" authorizes deletes.

Hydrate the pushed identity the same way a full load does, so the
merged state always carries its runtime policy names and documents.

* s3api: hydrate runtime policies on pushed group upserts

PutGroup propagation has the same gap as identity pushes: the wire
proto does not carry the group's runtime __inline_group_policy__ names,
so members' attached-policy check lost them until the next reload.

* s3api: attach only runtime inline policy names on pushes

Listing the pushed identity's or group's own inline policies is enough
to keep it on the attached-policy route; a whole-store hydration per
pushed proto is not. Pushes now never write iam.policies, so a stale
read can no longer overwrite a newer committed policy document.

* s3api: carry inline policy documents on pushed upserts

Attaching only runtime policy names left a window where a pushed identity or group referenced a document the local cache had never loaded, denying allowed requests until the next reload. Fetch the pushed entity own inline documents and publish them with the update; reads stay scoped to the pushed entity rather than the whole store, and inline runtime names are only ever populated from this store so a read cannot install a document older than the committed one. Resync the IAM manager only when a push actually installs documents.

* s3api: order pushed inline policy reads against their installs

A pushed UpsertIdentity or PutGroup read inline policy documents from
the credential store and installed them without holding the ordering
guard that full reloads already take, so a delayed push could overwrite
a newer committed document with its stale read and restore a revoked
grant until the next update or reload.

Pushes now hold reloadMu across the store read and the install, the same
snapshot-through-commit serialization the credential-manager load uses.
MergeS3ApiConfiguration also applies config.Policies to the live
iam.policies at commit instead of swapping in its earlier snapshot, so a
policy installed while a merge prepared (a pushed inline document or a
PutPolicy update) is no longer discarded.
2026-10-17 15:25:00 -07:00
74ed12d6b3 s3 lifecycle: replay only the rules that fit the retention window (#11789)
* s3 lifecycle: replay only the rules that fit the retention window

runShard handed drainShardEvents the full snapshot, so rules the retention
window assigns to the walk were also matched during meta-log replay. Their
matches are not due yet, and drainShardEvents freezes the cursor at the first
event with a not-yet-due match. With a 3-day and a 30-day expiration rule on one
bucket and RetentionWindow of 8 days, every shard's cursor stopped at the first
30-day object's PUT and stayed there until that object came due. Each pass
then re-read the log from that point and re-dispatched every 3-day expiration
already handled, which surfaces as NOOP_RESOLVED dispatches, a growing
cursor_lag_max, and passes that still report status=ok.

Snapshot.ReplayView drops the replay-kind actions promoted to the walk and
keeps every other action as compiled, and the drain uses it. Promoted rules
are served by the walk as before. RulesForShard, PromotedHash and ReplayView
share one isPromoted predicate. With RetentionWindow unset no active rule is
promoted, so behaviour without a configured window is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: scale the replay-view test with DaysToDuration

The test took its retention window from DaysToDuration but dated its PUTs
in hours, so under the s3tests build tag (one lifecycle day = 10s) every
event fell before the seeded cursor and replay never ran. Express the
event ages in DaysToDuration too, so the test exercises replay at both
scales.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: rehabilitate scan_only mode in the replay drain view

ReplayView shared the base CompiledAction pointers, so a replay-kind
rule locked into ModeScanOnly by a prior compile stayed invisible to
router.Route's event_driven gate even when the runtime retention window
kept it in the replay partition - and it is absent from the walk view
too, so the rule would be served by neither.

Clone non-promoted replay-kind actions the way RulesForShard does
(Mode forced to event_driven), reactivating the scan_only case whose
bootstrap rendezvous already completed, while preserving the base
active bit for genuinely pre-bootstrap actions. Disabled actions are
excluded for parity with the other views, and DESIGN.md now documents
the view.

* s3 lifecycle: keep drain activation observable, require walker with retention

ReplayView now shares pointers for event-driven actions so a mid-pass
MarkActive flip remains visible to the drain; only a scan_only-locked
action is cloned to event_driven, and reactivated only when its
bootstrap rendezvous completed (tracked via a new bootstrapComplete
field on CompiledAction).

validate() rejects RetentionWindow without Walker: promoted rules
would lose their only dispatch path while the drain cursor still
advanced past their events.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-17 15:24:35 -07:00
Chris Lu 443ae1cd83 s3 lifecycle: share daily bucket walk and prune listings (#11797)
* s3 lifecycle: share the daily bucket walk across shards

The daily pass invoked the walker once per shard, and every shard's
walk listed the full bucket tree while keeping only its own 1/16 of
the entries — 16 full listings per pass by default (#11791).

Run now groups the shards by the walk branch they'll take (recovery
vs steady, using the same cursor data and hash checks runShard applies)
and starts one shared walk per group, so a pass lists each bucket at
most twice. WalkerFunc and WalkBuckets take a shard set instead of a
single shard. runShard keeps its own decision and waits on its group's
walk; a shard whose freshly-loaded cursor disagrees with the plan falls
back to a per-shard walk, so mid-pass cursor rewrites can't under-cover
a shard.

* s3 lifecycle: skip the .versions/ pre-pass on non-versioned buckets

walkBucketTree listed every directory twice — once looking for
.versions/ dirs, once for regular entries. On a bucket that never had
versioning enabled the first pass can only find coincidentally-named
user folders, so the extra listing is pure cost (#11791).

FilerListFunc now takes the walk view and consults
Snapshot.BucketVersioned; non-versioned buckets skip pass 1 entirely.
A *-suffixed .versions user folder on such a bucket is recursed into
as a regular directory so real objects inside still surface; a nil
view keeps the two-pass behavior.

* s3 lifecycle: prune walk directories to active rule prefixes

The daily walk listed every directory under a bucket even when no
rule prefix could reach it. Collect the active rule prefixes of each
bucket from the walk view (new Snapshot.BucketPrefixes) and skip
recursing into directories whose keys can never match; a bucket
with no active actions is skipped entirely, saving the listing.

.uploads stays in scope regardless of prefixes because AbortMPU
rules match DestKey, and .versions expansion is pruned only
conservatively (the logical key is a string-prefix of the dir key).

* s3 lifecycle: join planned walks before Run returns

planWalks starts a shared walk goroutine per group, but a shard whose
authoritative cursor load then fails exits runShard without waiting on
it. Run returned while the walk still listed or dispatched deletes
against connections the caller closes on return. Join every planned
walk after the shard waitgroup.

* s3 lifecycle: treat object-locked buckets as versioned in the walk

Object Lock requires versioning and every writer sets both extended
attributes, but IsBucketVersioned only read ExtVersioningKey. A locked
bucket missing that attribute would have its .versions dirs walked as
ordinary folders, hiding version siblings from lifecycle. Classify
ExtObjectLockEnabledKey as versioned too.
2026-10-17 15:21:53 -07:00
Chris Lu bc29ffbfbd helm: apply placement values to the chart's hook Jobs (#11795)
* helm: honor placement values in the bucket creation hook

The post-install/post-upgrade bucket Job ignored nodeSelector,
tolerations and affinity, so it could land on nodes the operator
kept clear, or fail to schedule at all on a fully tainted cluster.

The hook now accepts nodeSelector, tolerations and affinity under
each createBucketsHook (filer.s3, s3, allInOne.s3), read from the
same values section that supplies createBuckets. Unset nodeSelector
and tolerations fall back to the workload the hook already borrows
its security context from; affinity is explicit-only since the
anti-affinity rules of a long-lived server do not suit a one-off
Job.

* helm: honor placement values in the volume resize hook

The pre-install/pre-upgrade resize Job inherited nothing from the
volume server, so on a cluster whose storage nodes are all tainted
the Job could not be scheduled and the upgrade timed out.

The hook now falls back to the volume server's nodeSelector and
tolerations, and accepts explicit nodeSelector, tolerations and
affinity under volume.resizeHook. Affinity stays explicit-only: the
default podAntiAffinity against component=volume would keep the Job
off every node already running a volume pod, which is most of them
during an upgrade. Placement strings are rendered through tpl with
$volumeName bound, same as the StatefulSet.
2026-10-17 12:40:30 -07:00
Chris Lu 01d8160323 mini/server/filer: let embedded s3/sftp gateways use their own -dataCenter (#11798)
* mini: stop forcing -dataCenter onto the embedded S3 gateway

-dataCenter names the embedded volume server's data center, but it was
also copied into the S3 gateway options, overwriting an explicit
-s3.dataCenter and making every S3 write carry an explicit dataCenter
pin in its assign request. That pin outranks fs.configure dataCenter
rules, so a mini node in one DC could never steer S3 writes to volume
servers in another.

With the copy removed, assign requests carry no DC by default: the
filer's own -dataCenter still provides the write fallback, and
fs.configure rules now apply to S3 traffic. -s3.dataCenter still pins
reads and writes when set explicitly.

* server: give embedded s3/sftp gateways their own -dataCenter flags

-dataCenter names the embedded volume server's data center, but it was
also copied into the S3 and SFTP gateway options, and neither gateway
had a flag of its own to override it. The injected DC lands in every
S3 assign request as an explicit pin, outranking fs.configure
dataCenter rules.

Add -s3.dataCenter and -sftp.dataCenter (same names as weed filer) and
stop copying -dataCenter into them, so gateway writes go through the
filer's rules and defaults unless a gateway DC is set explicitly.

* filer: stop defaulting embedded gateway dataCenter to -dataCenter

The embedded S3 and SFTP gateways inherited -dataCenter whenever their
own flag was unset. That made every S3 assign request carry the DC as
an explicit pin, outranking fs.configure dataCenter rules, and left no
way to let rules steer writes while -dataCenter was set.

Let -s3.dataCenter and -sftp.dataCenter stay empty unless set, matching
standalone weed s3 and weed sftp. The filer's own -dataCenter still
provides the write fallback.
2026-10-17 12:38:35 -07:00
Chris LuandDevin 8d9c7e76a5 sftp: enforce file ACLs on rename targets (#11770)
* sftp: check delete permission on an existing rename target

renameEntry authorized only the source path and the target directory,
so renaming over a file with a more restrictive ACL destroyed it even
though Remove on the same file was denied.

* sftp: deny a rename that would strand a carved-out ACL

A rename moves data without moving its path-based ACL: renaming an
ancestor directory left a more specific read-only entry behind, so the
relocated file fell under a permissive ancestor ACL and became
writable. Deny the rename when a configured ACL under the source would
grant less write/delete access than the relocated path would get.

* sftp: cover inherited and destination-side ACLs on rename

The rename ACL check compared only configured paths under the source,
so it missed restrictions inherited from ancestors above the source and
deeper grants configured under the destination. It now compares the
moved root plus every configured path under either tree mapped to its
source counterpart, and skips paths whose entries do not actually move.
The target delete check no longer depends on an existence lookup, so a
protected target created during a rename cannot be overwritten; a no-op
rename skips it. Root bypasses the comparison like every other check.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* sftp: fail closed on rename ACL lookups and restore new-name renames

- checkRenameKeepsAcl returns lookup errors instead of treating them as a missing entry, so a transient filer failure cannot skip the ACL comparison; it also compares read and list gains, not only write and delete, and evaluates each pair with the moved entry's own IsDirectory.

- The target delete check now distinguishes an existing target (still requires delete) from a fresh name (no delete needed) while still enforcing a configured ACL on the target name itself, so a concurrent create cannot slip a protected target under the rename.

* filer: add fail_if_target_exists option to AtomicRenameEntry

Callers that authorize a rename while the target is absent cannot see a
target created in the gap before the move runs. The flag makes the filer
reject the rename when the target exists, checked under the rename path
locks that CreateEntry on the same filer also takes.

* sftp: guard rename against concurrent targets and file-child ACL gains

A rename onto a name that inherits its directory's ACL still needs no
delete access, but now asks the filer to fail if a target appears before
the move, so a concurrently created file cannot be overwritten without
delete permission. Moved directories are also compared in the file ACL
shape, where list no longer counts as read, so their file children cannot
gain read access through the relocation.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-17 12:36:20 -07:00
Chris Lu 887b1aac76 ec: carry the source volume's TTL expiry into the worker-written .vif (#11777)
* ec: carry the source volume's TTL expiry into the worker-written .vif

ReadVolumeFileStatus now reports the volume's ExpireAtSec (last write +
TTL), and the admin worker stamps it into the .vif it distributes with
the shards. Previously every worker-encoded .vif had expireAtSec 0, so
EC volumes of TTL collections never expired.

* ec: report expiry from the Rust volume server and cover the .vif carry

The seaweed-volume drop-in server omitted expire_at_sec from its
ReadVolumeFileStatus response, so worker-encoded volumes sourced from it
lost the TTL. Fall back to the nested VolumeInfo expiry for servers that
predate the top-level field, and pin the nonzero/zero carry with a test.
2026-10-17 12:34:55 -07:00
Chris Lu ac996c0ad1 filer: shrink zip spool threshold in tests to fix 32-bit OOM (#11796)
TestSpoolZipReaderTempFile and TestSpoolZipReaderLimit each allocate
~33-70 MiB to cross the 32 MiB in-memory spool threshold. On linux/386 the
weed/filer test binary already runs near the 4 GiB address-space limit, so
these allocations reliably crash it with "out of memory" mid-suite.

Make zipSpoolMemoryLimit a var and drop it to 1 KiB in those tests, which
exercises the same memory-vs-temp-file spool paths with KiB-sized buffers.
2026-10-17 11:43:55 -07:00
Chris Lu 1155e65fc5 admin: regenerate the gzipped static mirror
admin.js changed in 161310d83 without rerunning go generate, leaving
static_gz/js/admin.js.gz stale and TestStaticGzMirror failing on master.
2026-10-17 10:49:51 -07:00
joma1022 161310d834 admin: fix Create Bucket sending POST /api/s3/buckets twice (#11790)
admin: fix Create Bucket submitting twice

   admin.js and the Buckets page template both bound a submit listener to #createBucketForm, so one click sent POST /api/s3/buckets twice. Keep only the page handler.
2026-10-16 23:47:48 -07:00
Peter Dodd 42ff535511 vacuum: bound concurrent commits per volume server (#11658)
CommitCompact holds the volume's write lock for the whole commit, and
reads of that volume wait on it. Volumes compacted together tend to
finish together, so their commits land at once and their fsyncs
contend: each commit's lock hold grows with how many run alongside it,
even when the disk is far from its throughput limit.

Add -compactionMaxParallelCommits (volume.compactionMaxParallelCommits
for server/mini). A commit waits for a slot before CommitCompactVolume,
so a queued commit holds no volume lock and reads keep flowing. The
wait counts against the caller's deadline and returns the context
error if it expires; a slot granted to an already-expired context is
handed back, so a request the caller gave up on never starts a commit,
and maintenance mode is checked again once the slot is held.
The slot is released whether the commit succeeds or fails. Compaction
itself is not gated, so -maxParallelVacuumPerServer still sets reclaim
throughput. 0, the default, keeps today's behaviour.

The limit sits in the volume server so it covers every commit caller:
the master's vacuum sweep, the vacuum worker task and volume.tier.compact.

The Rust volume server mirrors the flag and behaviour: queued commits
wait on a semaphore keyed to the grpc-timeout deadline, hold no store
lock, recheck maintenance mode once a permit is held, and release it
on every exit path.
2026-10-16 23:46:36 -07:00
Chris Lu 42ff7f2b8e s3: report an empty probed directory as a common prefix, not a key (#11779)
The "<dir>/" existence probe emitted the empty directory as a
FolderMimeType Contents entry, so delimited listings returned a fake
readable key that breaks clients treating every key as an object.
Delimited probes now report the directory as a CommonPrefix -- which
the probe accepts for existence -- while the recursive (no delimiter)
probe keeps the directory marker.
2026-10-16 23:45:37 -07:00
Chris Lu 3e55971e0e ec.shard.unmount: treat same-node shard copies as one target (#11778)
A shard held on two disks of one server produced two identical live
ShardRefs, so resolution reported "ambiguous" even for a fully
node-qualified -shardId. Ambiguity now means the shard lives on more
than one node; duplicate refs on the same node resolve to it.
2026-10-16 23:45:10 -07:00
3c6700c310 volume balance: stop planning moves onto disk types a server does not have (#11762)
* fix(topology): capacity queries no longer create empty disk-type entries

AvailableSpaceFor and CapacityFor called getOrCreateDisk, so asking a
node whether it had room for a disk type it lacks created a zeroed
DiskUsageCounts entry for that type. Volume growth walks every data
center, rack and data node, and ToDataNodeInfo publishes diskUsages, so
every node ended up advertising an empty disk (MaxVolumeCount 0,
DiskId 0) of every type ever grown: HDD-only servers showed an "ssd"
disk and SSD-only servers an "hdd" disk.

Read the counters through a non-creating getDisk; a missing entry
answers 0, the same numbers as before.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq

* fix(admin): don't let an empty disk-type entry shadow a node's real disk

UpdateTopology keys disks by node and disk id. The master published an
empty entry (no capacity, nothing on it, DiskId 0) for every disk type a
capacity check had asked about, and it collided with the node's real
disk 0; map iteration order decided which survived. An SSD-only node
intermittently showed an empty HDD disk, and an HDD node lost its disk
-- and with it its volumes from the volume index, hence "Could not find
volume N on source server" in balance detection.

IsShadowedEmptyDiskEntry matches an empty entry whose disk id another
DiskInfo of the node, one with capacity or data, already uses.
UpdateTopology and CountTopologyResources skip it. A lone empty entry
is still an empty disk of unknown capacity and is kept, as
GetDisksWithEffectiveCapacity expects.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq

* fix(balance): don't seed balance candidates from shadowed empty disk entries

detectForDiskType registered every node with a DiskInfos entry of the
type as a zero-volume candidate. An empty entry the master made up for
a type the node lacks made that node the least-loaded destination, and
its zero max switched the whole disk type to raw-count balancing. The
volume server then refused every move: "no space left hdd" / "no space
left ssd".

Skip entries IsShadowedEmptyDiskEntry matches when seeding candidates.
A server that holds volumes of the type but reports no max still enters
through the metrics and still triggers the raw-count fallback.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq

* fix(admin,balance): ignore made-up disk-type entries at any disk id

The made-up entry is published with DiskId 0, but it need not collide
with a real disk: when location 0 is unavailable (not reported) the
node's usable disk is location 1. IsShadowedEmptyDiskEntry required a
collision, so such an entry survived both UpdateTopology -- where the
unknown-capacity fallback offers it to planners -- and balance
detection, which planned moves onto it again.

Its disk id cannot tell it apart: a DiskInfo with no volumes, no EC
shards and no per-disk max is published with DiskId 0 whichever disk
it stands for. IsPhantomDiskTypeEntry (renamed, since nothing needs to
be shadowed now) matches an empty entry on any node that has a disk of
another type with capacity or data. A real disk looks the same only
while it offers no volume slot, when it can take no volume either. A
node whose entries are all empty keeps them as disks of unknown
capacity, as before.

Reported by review on #11762.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq

* test(balance): assert where the real disks land in the empty-entry test

Check through GetNodeDisks that each server's real disk sits at the
subtest's disk id before planning. The id-1 case already landed on
physical disk 1 (SplitByPhysicalDisk folds all-zero record ids onto the
entry's DiskId), so this only makes the setup explicit; on the first
revision of the fix it shows the real disk at 1 beside the made-up
entry at 0.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq

* fix(topology,balance,ec): don't publish or trust usage-only disk-type entries

ToDataNodeInfo started from the raw disk-usage counters, so a node that
was asked for capacity of a disk type it lacks kept advertising an empty
DiskInfo for that type even though no such disk was ever reported. Skip
usage entries that have no disk child and hold all-zero counters.

The same stale entries still misled planners that read DiskInfos
directly: the balance plugin's move precondition treated a phantom entry
as proof a target has that disk type, and the EC balancer counted it as
a matching disk. Filter them with IsPhantomDiskTypeEntry.

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-16 23:41:26 -07:00
Chris LuandDevin 5a0be275cc s3: keep the gateway up when an auxiliary listener fails to bind (#11768)
* s3: keep the gateway up when the Iceberg or Lance listener fails to bind

The optional catalog servers ran glog.Fatalf on a bind error, so an
occupied port (e.g. the default Lance port 9101 taken by another process)
killed the entire S3 server. Log the error and run without the auxiliary
listener instead; the S3 API itself still fails hard on bind errors.

* mini: only probe and advertise catalogs whose listeners bound

With aux listener failures no longer fatal, the configured catalog port may
answer as an unrelated process; mini's readiness probe accepts any HTTP
response and would announce that foreign service as the catalog. Record the
bind outcome on S3Options (final by the time the S3 port serves, since the
aux binds happen first) and gate mini's readiness wait and admin UI
advertisement on it.

* s3: test that a failed aux listener bind is non-fatal

Hold the Iceberg and Lance ports open and assert both starters return no
server and leave the bound flags false, covering the new nonfatal path.

* command: range the endpoint test table by index

S3Options now holds atomic.Bool flags, so copying a test case trips
go vet's lock-copy check.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-16 23:10:09 -07:00
Chris Lu cabaf9c38d iceberg: collapse doubled /v1 prefix in catalog requests (#11767)
* s3: verify SigV4 against the request path the client sent

Internal routers that rewrite URL.Path (the Iceberg catalog doubled-/v1
collapse) broke signature verification: verifyV4Signature re-derived the
canonical URI from the rewritten path while the client had signed the
original one, so valid signed catalog requests were rejected. Stash the
client-sent path in the request context at rewrite time and prefer it when
computing pathForSignature.

* iceberg: collapse repeated /v1 prefixes before route matching

Move the doubled-prefix collapse from the catch-all into a middleware so it
also covers paths that match the {prefix} routes (/v1/v1/namespaces would
otherwise resolve "v1" as the table bucket; the name is unreachable anyway
since bucket names require 3-63 chars). Collapse all leading repetitions in
a single pass and re-dispatch once instead of recursing per segment, and
record the original path in context so signed requests still verify.

* iceberg: slice off repeated /v1 prefixes instead of concatenating

Stripping path[len("/v1"):] per iteration reuses the original string
instead of allocating a fresh copy of the remaining path each pass.
2026-10-16 23:01:57 -07:00
ad2486919c iam: accept AWS action names for listing multipart uploads and parts (#11765)
* iam: accept AWS action names for listing multipart uploads and parts

PutUserPolicy rejected s3:ListBucketMultipartUploads and
s3:ListMultipartUploadParts as invalid actions, so no inline policy could
grant these operations by their AWS names. Map both to Read, which is the
action their routes are registered under.

Fixes #11738

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y1b55sEL1gA1c3FCMcQCbm

* test: check each multipart listing action separately

Granting both actions in one policy let a correct mapping for one hide a
wrong mapping for the other. Test each action on its own and assert it
grants only Read.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y1b55sEL1gA1c3FCMcQCbm

* iam: scope multipart listing grants to their resolved action

A policy naming s3:ListBucketMultipartUploads or s3:ListMultipartUploadParts
projected to a Read:<bucket> grant, which the legacy route check honored for
every object in the bucket. Map the AWS names to dedicated action constants
and let VerifyActionPermission fall back to the resolved request action, so
the grant covers only the listing operation it names.

* iam: narrow legacy ListParts and ListMultipartUploads mappings

Both policy names projected to Write:<bucket>, granting full object writes
on any bucket where listing was intended. They now map to the same scoped
listing actions as their AWS names.

* iam: bind the multipart listing fallback to the listing routes

The legacy-action fallback in VerifyActionPermission resolved the
operation with ResolveS3Action, which reads the promoted authorization
object. For GET /bucket?uploads&prefix=foo/ the prefix becomes the
object, so the request resolved to GetObject and a
ListBucketMultipartUploads grant was denied. Resolving from query
priority also did not match how the router picks a handler.

Only fall back for the two listing routes, identified from the route
vars the same way the router dispatches them, and add the scoped
actions to MapToIdentitiesAction so GetUserPolicy in the standalone
IAM API rebuilds valid policy actions.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y1b55sEL1gA1c3FCMcQCbm

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-10-16 21:11:10 -07:00
Chris LuandDevin 2521c202aa s3: accept signed streaming uploads when IAM is disabled (#11766)
* s3: skip seed-signature verification in chunked reader when IAM is off

When no credentials are configured there is nothing to verify streaming
chunk signatures against. Leave the credential unset so the reader decodes
the aws-chunked body and skips per-chunk verification, matching the
unsigned-streaming path. This also fixes SigV4-signed unsigned-streaming
uploads on open gateways, which previously failed in calculateSeedSignature.

setupIam() now sets isAuthEnabled to match the credentials it installs, so
the signature-verification tests still exercise the enabled path.

* s3: decode signed streaming uploads when IAM is disabled

AWS SDK v2 clients (Trino, Spark, modern aws-cli) send PutObject bodies as
STREAMING-AWS4-HMAC-SHA256-PAYLOAD. On a gateway without S3 auth this was
rejected with ErrAuthNotSetup, breaking writes to table buckets and regular
buckets alike. The auth check already short-circuits when IAM is off, so
route both streaming payload types through the chunked reader.

* s3: cover signed trailer streaming uploads without IAM

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-16 21:10:38 -07:00
Chris Lu b0ab4c4e1e shell: make every -collection flag parse the same way (#11794)
* shell: make every -collection flag parse the same way

The filter vocabulary (comma-separated names, wildcards, regexes,
"_default" for the empty-named collection) already existed in
wildcard.CompileCollectionMatcher, but several commands still did their
own exact-match comparison or sent the raw flag value to servers.

fs.mergeVolumes used a literal string compare, so -collection=_default
silently produced an empty merge plan; it now uses the shared matcher
while keeping "*" as the match-all default. volume.fsck and volume.mark
filter through the matcher too, and volume.vacuum resolves the filter
client-side into per-collection and per-volume requests, since no
Collection value scopes the VacuumVolume RPC to the unnamed collection
alone.

Commands that put one concrete collection name on the wire normalize
"_default" to "" (volume.grow, fs.configure, mq.topic.compact,
collection.delete, ec.rebuild, ec.balance), and the tier -volumeId
commands resolve the volume's actual collection from topology first so
patterns and named-collection requests pass the volume server's
collection check.

* shell: spell the default-collection keyword via CollectionDefault

* shell: address review findings on collection filter handling

volume.fsck keeps a set of every volume the master knows before any
filtering, so a filer entry whose volume was excluded by -collection or
-volumeId is skipped instead of reported "volume not found" and deleted
by -reallyDeleteFilerEntries. The collection filter also no longer
scopes the filer walk to /buckets/<collection>: fs.configure can bind a
collection to any path prefix, and a narrower walk could mark live
chunks as orphans under -reallyDeleteFromVolume.

ec.rebuild and ec.balance resolve -collection through the new
wildcard.ExpandCollectionFilter, which passes literal names through even
when the master does not list them — ec.rebuild must be able to recover
a collection whose shards all lost their index and never registered.

fs.configure and mq.topic.compact now reject _default outright: an empty
Collection means "unset" in both the path-rule merge and the filer's
storage-option fallback chain, so the keyword would have been silently
ignored.

The unnamed-collection vacuum sweep keeps the master's low-disk-space
exception: a volume whose every replica is read-only is probed with
VacuumVolumeCheck and stays eligible when a replica reports
DiskSpaceLow, instead of being dropped outright.
2026-10-16 21:07:51 -07:00
Chris Lu 268424a5e1 s3: extract zip members on PutObject (#11786)
* filer: add zip archive extraction helpers

SpoolZipReader makes an upload body seekable for archive/zip,
SanitizeZipMemberName rejects zip-slip and non-regular members, and
CheckZipLimits caps member count and total uncompressed size.

For seaweedfs/seaweedfs#11722

* filer: extract zip members on ?extract=zip upload

POST/PUT to a directory path with ?extract=zip (or X-Seaweedfs-Extract:
zip) writes each archive member as its own entry, so a bundle of small
files lands in one request instead of one roundtrip each. Members keep
their mtime, mode and mime-by-extension; directories are created;
unsafe names are skipped and reported. extract.keep=true also stores
the archive itself.

For seaweedfs/seaweedfs#11722

* filer: bound zip spool memory by actual bytes

A chunked upload has no Content-Length, so the old size hint let an unbounded body go through io.ReadAll. Read up to the memory threshold plus one byte first, then spill the prefix and the rest to a temp file, capped at ZipExtractMaxBytes total.

* filer: guard zip member size total against overflow

Declared member sizes summing past math.MaxUint64 wrapped the running total and slipped past the 1 TiB cap.

* server: only take the inline fast path at a short first read

uploadReaderToChunks read at most chunkSize bytes, then treated any read below SaveToFilerLimit as the whole file. When SaveToFilerLimit exceeded chunkSize, longer uploads were stored truncated.

* server: resolve storage options per extracted zip member

A member can land under a storage rule more specific than the request path. Re-running detectStorageOption0 per destination applies that rule's collection, replication, TTL, filename limit, and read-only guard to files and directories alike, while explicit query overrides still win.

* server: accept raw request body for POST extract uploads

Every POST was forced through MultipartReader, so a raw zip body always got a 400. Only parse multipart when the Content-Type is actually multipart.

* server: extract zip directories before files and stop on cancel

Processing directory members first keeps their stored mtime/mode from being overwritten by parents auto-created for an earlier file member. Checking ctx.Err() between members keeps a disconnected client from driving thousands more writes.

* server: assert stored mtime and mode for extracted members

* s3: add zip extraction header constants

* s3: extract zip members on PutObject with extract header

x-amz-meta-snowball-auto-extract or x-seaweedfs-extract: zip on a PutObject fans out the archive to one object per member under the object's directory, reusing putToFiler so chunking, collections, storage rules, versioning, and bucket-default encryption apply to each member. The archive itself is not stored. Explicit SSE request headers are rejected since one encryption config cannot represent independently created member objects.

* s3: bound member fan-out by request cancellation

Stop starting new member writes once the request context is done, matching the filer extract loop.

* filer: bound compressed zip spool and reject non-local member names

The 1 TiB uncompressed limit was also capping the compressed bytes
spooled to the host's temp directory, so a handful of concurrent
extractions could fill the filer's disk before any ZIP validation ran.
SpoolZipReader now takes a separate compressed-size limit (the filer
passes its TUS max upload size) with a 5 GiB default.

SanitizeZipMemberName gains a filepath.IsLocal check: the segment loop
already rejects "..", but CodeQL only treats member names as sanitized
through a recognized guard, and the extra check also covers reserved
Windows names.

* server: honor destination rules for zip members and clamp maxMB

An x-seaweedfs-destination header picks the storage rules for a logical
destination while members still land under the request path, matching
PostHandler. Directory members now also enforce the actual path's
read-only rule instead of relying on option resolution to fail.

maxMB up to 2048 overflowed the int32 chunk size, so an oversized query
value made extraction (and ordinary autoChunk uploads) store members
empty yet report success. The parsing is shared and clamped at 2047 MB.

* s3: scope zip member writes through the per-key checks PutObject applies

The extraction loop bypassed most of what a standalone PutObject would
face: members were written without IAM authorization for their key,
directory members replaced markers without entry lock checks, and lock
headers persisted retention without the s3:PutObjectRetention or
s3:PutObjectLegalHold permissions. authorizeMember now runs IAM,
table-bucket path rules, bucket policy, lock-header validation, and
non-versioned lock enforcement per member key, and directory members
take the same guarded mkdir path as an explicit dir marker.

The archive's own Content-Md5 and x-amz-checksum-* values are verified
while the body spools instead of being silently dropped, members stop
inheriting the archive's Content-Encoding, and non-versioned members
resolve their lifecycle TTL like normal uploads.

* s3: fail malformed lock headers before spooling the archive

Per-member re-validation turned a bad object-lock header into one error
entry per member inside a 200 response. Validate once up front like
PutObjectHandler does, and keep only the dedicated retention/legal-hold
permission checks scoped to each member key.

* filer: return 413 for oversized archives and honor saveInside in extract

Map filer.ErrZipTooLarge to 413 instead of 500 for client-side archive
size violations. Extracted members written under saveInside now store
as inline entry content instead of unreachable volume chunks.

* s3: write extracted members under their own request context

Point member writes at the member key via mux.SetURLVars so SSE-KMS
builds the encryption context from the member ARN, not the archive's.
Map the spool limit error to ErrEntityTooLarge.
2026-10-16 21:07:05 -07:00
Chris Lu a8affc851d filer: extract zip members on ?extract=zip upload (#11782)
* filer: add zip archive extraction helpers

SpoolZipReader makes an upload body seekable for archive/zip,
SanitizeZipMemberName rejects zip-slip and non-regular members, and
CheckZipLimits caps member count and total uncompressed size.

For seaweedfs/seaweedfs#11722

* filer: extract zip members on ?extract=zip upload

POST/PUT to a directory path with ?extract=zip (or X-Seaweedfs-Extract:
zip) writes each archive member as its own entry, so a bundle of small
files lands in one request instead of one roundtrip each. Members keep
their mtime, mode and mime-by-extension; directories are created;
unsafe names are skipped and reported. extract.keep=true also stores
the archive itself.

For seaweedfs/seaweedfs#11722

* filer: bound zip spool memory by actual bytes

A chunked upload has no Content-Length, so the old size hint let an unbounded body go through io.ReadAll. Read up to the memory threshold plus one byte first, then spill the prefix and the rest to a temp file, capped at ZipExtractMaxBytes total.

* filer: guard zip member size total against overflow

Declared member sizes summing past math.MaxUint64 wrapped the running total and slipped past the 1 TiB cap.

* server: only take the inline fast path at a short first read

uploadReaderToChunks read at most chunkSize bytes, then treated any read below SaveToFilerLimit as the whole file. When SaveToFilerLimit exceeded chunkSize, longer uploads were stored truncated.

* server: resolve storage options per extracted zip member

A member can land under a storage rule more specific than the request path. Re-running detectStorageOption0 per destination applies that rule's collection, replication, TTL, filename limit, and read-only guard to files and directories alike, while explicit query overrides still win.

* server: accept raw request body for POST extract uploads

Every POST was forced through MultipartReader, so a raw zip body always got a 400. Only parse multipart when the Content-Type is actually multipart.

* server: extract zip directories before files and stop on cancel

Processing directory members first keeps their stored mtime/mode from being overwritten by parents auto-created for an earlier file member. Checking ctx.Err() between members keeps a disconnected client from driving thousands more writes.

* server: assert stored mtime and mode for extracted members

* filer: bound compressed zip spool and reject non-local member names

The 1 TiB uncompressed limit was also capping the compressed bytes
spooled to the host's temp directory, so a handful of concurrent
extractions could fill the filer's disk before any ZIP validation ran.
SpoolZipReader now takes a separate compressed-size limit (the filer
passes its TUS max upload size) with a 5 GiB default.

SanitizeZipMemberName gains a filepath.IsLocal check: the segment loop
already rejects "..", but CodeQL only treats member names as sanitized
through a recognized guard, and the extra check also covers reserved
Windows names.

* server: honor destination rules for zip members and clamp maxMB

An x-seaweedfs-destination header picks the storage rules for a logical
destination while members still land under the request path, matching
PostHandler. Directory members now also enforce the actual path's
read-only rule instead of relying on option resolution to fail.

maxMB up to 2048 overflowed the int32 chunk size, so an oversized query
value made extraction (and ordinary autoChunk uploads) store members
empty yet report success. The parsing is shared and clamped at 2047 MB.

* filer: return 413 for oversized archives and honor saveInside in extract

Map filer.ErrZipTooLarge to 413 instead of 500 for client-side archive
size violations. Extracted members written under saveInside now store
as inline entry content instead of unreachable volume chunks.
2026-10-16 21:04:54 -07:00
Chris LuandDevin 6da432b8bf filer: make a retried ObjectTransaction replay its recorded outcome (#11788)
* filer_pb: add request_id to ObjectTransactionRequest

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: replay recorded outcome for a retried ObjectTransaction

A failover replay carries the same request_id; under the same path lock
the owner replays the recorded outcome instead of re-applying the
mutations over whatever superseded them. Only application-level
outcomes are recorded — a lookup error leaves the request free to
re-run.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: stamp routed object transactions with a request id

One id per logical transaction so failover retries replay the recorded
outcome instead of re-applying the mutations.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: scope transaction replay tokens per path and keep them for their TTL

The dedup key includes the lock path so a token reused on a different
object cannot replay this outcome. Records are never evicted early —
an early eviction would let a replay re-apply — so the expiry sweep is
amortized and the map is bounded by the insert rate over the TTL.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: bound the transaction replay cache at capacity

* filer: do not record indeterminate transaction outcomes

A mutation error can be transient or follow a partial apply; recording
it replays that failure to a retry for the whole TTL instead of letting
the same idempotent mutations converge. Only successful transactions
and precondition failures are definitive outcomes worth replaying.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: replay recorded mutation failures instead of re-applying

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-16 20:05:57 -07:00
Chris LuandDevin da3b7feeae admin: move dashboard progress labels beside the bar (#11781)
* s3: keep suspended-versioning conditional writes off the routed condition

A suspended PUT writes the null-version slot at the regular path while the
entry the conditional headers evaluate lives under .versions/. The routed
WriteCondition checked the request's etag against that empty slot, so a
correct If-Match failed with 412. Passing uniqueWritePath sends conditional
requests through the lock path, whose precondition resolves the same entry
the read path serves; unconditional writes still route.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* admin: keep the volumes progress label readable at low fill

The dashboard's Volumes and tier-usage cells rendered their text inside the
progress bar, so a low fill left white label text on the light track and the
count was unreadable. Render the label beside the bar instead.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* admin: keep a minimum width on dashboard progress tracks

A nonwrapping label beside a flex track could squeeze it to zero in a
narrow table, hiding the bar entirely.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-16 20:03:23 -07:00
Chris LuandDevin 00804b32fa s3tables: let an explicit policy Deny veto fallback and owner allows (#11772)
* s3tables: let an explicit policy Deny veto fallback and owner allows

Table operations evaluated the table or namespace policy and the bucket
policy as an OR, so an explicit Deny in one policy was overridden by an
Allow in the other. Owner and identity grants also short-circuited
before the resource policy was read. Deny statements are now evaluated
first at both levels, matching AWS explicit-deny semantics.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3tables: fail closed when a resource side has no owner

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-16 19:54:46 -07:00
Chris LuandDevin 44657988ea s3: apply destination retention on copies instead of inheriting it (#11773)
* s3: apply destination retention on copies instead of inheriting it

CopyObject carried the source object's retention mode, retain-until date
and legal hold into the destination entry, so a copy onto a bucket with
default COMPLIANCE retention produced an unlocked object that could be
deleted early. Copied objects now take their retention from the copy
request's lock headers or the destination bucket's default, matching
PUT semantics; a versioned self-copy gets the same treatment while an
unversioned metadata-only self-copy keeps the object's existing lock.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: validate lock headers on metadata-only self-copies

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-16 19:52:10 -07:00
Chris LuandDevin 2d06e2d6c1 s3: evaluate tag conditions on copy-source and tagging subresources (#11771)
* s3: evaluate tag conditions on copy-source and tagging subresources

Policy evaluation ran without the object entry on several paths, so a
Deny conditioned on s3:ExistingObjectTag never matched:

- CopyObject, UploadPartCopy and RenameObject checked the source key
  before its entry was fetched; repeat the check on a synthesized GET
  once the entry resolves.
- GetObjectAttributes and Get/Put/DeleteObjectTagging never supplied
  the entry; recheck with the resolved entry.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: fail closed on tag-policy lookups and honor tag-conditioned grants

- recheckTagConditionedPolicyWithLookup returned success when the entry
  lookup errored, letting a tag-gated Deny slip past a transient miss;
  only ErrNotFound still falls through to the handler's own NoSuchKey.
- A read or delete denied in the tagless first phase now gets one more
  evaluation with the resolved entry's tags in hand, so an
  s3:ExistingObjectTag-conditioned Allow can still grant a copy-source
  read or rename. The deferred recheck requires an explicit Allow;
  indeterminate results keep the original verdict, and the existence of
  the key never leaks past a deny.
- RenameObject now applies the same tag-aware recheck to the source
  removal, so a tag-gated Deny on s3:DeleteObject blocks a rename the
  way it blocks a direct DELETE.
- GetObjectAttributes evaluates the object-entry policy recheck before
  conditional headers, so a 304/412 response can no longer bypass a
  tag-based Deny.

* s3: honor identity denies in the tag-aware source recheck

The deferred recheck re-evaluates only the bucket policy, so a bucket
Allow could lift an explicit Deny in the caller's attached or session
policies on a copy/rename source. Apply the same applicable-policy Deny
check the first phase uses.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-16 19:50:36 -07:00
Chris LuandDevin 446e6f3444 s3tables: scope OIDC-derived account IDs to the token issuer (#11774)
* s3tables: scope OIDC-derived account IDs to the token issuer

Table ownership keyed off the bare sub claim, so two identity providers
issuing the same sub collided on one account - a caller could act on
tables owned by a different provider's user. OIDC bearer tokens now
carry the same issuer-namespaced (sub, iss) hash STS sessions already
expose as aws:userid, and getAccountID prefers it over the raw claims.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3tables: keep legacy owners working and scrub client-subject claims

- OIDC bearer tokens without a subject are rejected, and a token-supplied
  aws:userid claim is always overwritten with the verified (sub, iss)
  hash, so a forged claim can no longer stand in for the principal.
- Owners written before issuer scoping hold a bare subject; permission
  checks now re-anchor that subject under the caller's issuer, so the
  same (sub, iss) pair keeps its ownership while scoped owners remain
  issuer-isolated. The token issuer is asserted into claims alongside
  sub and carried through PolicyContext.

* s3tables: anchor legacy subjects to the configured issuer set

* s3: enumerate trusted issuers from registered providers too

* s3: fail closed when the OIDC provider store cannot be listed

trustedOIDCIssuers kept the registered issuers when ListOIDCProviders
failed, presenting a partial set as complete. A deployment trusting one
registered issuer A plus a stored issuer B would, on a listing error,
report only A — the single-issuer shape under which a legacy bare
subject re-anchors — letting a caller from A claim a stored subject
that belongs to B and gain owner access.

With no provider store the registered providers are provably complete,
so keep returning them; with a store, a failed listing means the real
set is unknown, so return none.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-16 19:48:32 -07:00
Chris LuandDevin 348116b2c4 filer: sweep the local subtree on a replayed directory delete (#11785)
* filer: sweep the local subtree on a replayed directory delete

A recursive delete emits only the directory's own key, so peers applied
it with a single DeleteEntry. Children the peer held that the origin had
not listed stayed behind unlisted but still readable, and their needles
were never deleted. Sweep the local children on a replayed directory
delete and drop their chunks when the origin deleted chunks.

* filer: walk the subtree on a replayed directory delete

DeleteFolderChildren removes only direct children in stores keyed by
immediate parent, so the sweep now lists and deletes each descendant
entry itself. Entries the local filer committed after the event survive
— with the directories that hold them — and hard-linked files keep
their chunks for the origin's own events to settle.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: key replayed subtree chunk deletes to each entry's path

The sweep queued every descendant's chunks under the deleted directory's
storage rule and skipped directory entries entirely. Chunks now collect
keyed by their owning path, so per-path rules apply to each file — and a
directory carrying its own chunks (a file promoted by an S3 child) frees
them too.

* filer: fence kept replayed dirs and flush collected chunks on error

A directory kept because a newer descendant survived still needs its
delete timestamp recorded, or a stale replayed create can resurrect
another child under it. And chunks collected for deleted children must
be flushed even when the sweep fails partway — the entries are gone and
a retry cannot list them again. Collection now also runs only after the
entry delete succeeds.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: flush swept chunk references in batches

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-16 19:47:20 -07:00
Chris LuandDevin 5b4a707c43 shell: keep volume.tier.compact replicas on the same remote object (#11783)
* shell: keep volume.tier.compact replicas on the same remote object

The compact cycle ran on a single replica: it deleted the shared remote
object up front and never touched the siblings, leaving their .vif/.idx
pointing at a removed object. It also remounted the working replica into
a writable local volume mid-operation.

Freeze all replicas first, keep the shared object while siblings still
reference it, replicate the new metadata to every replica, and let the
last tiered replica retire the old object.

* storage: surface remote object delete failures from volume destroy

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* shell: keep tier.compact replicas frozen until every copy lands

A delete between the source unfreeze and a sibling's index copy could
land past the copied snapshot, leaving the replica serving deleted
data. Unfreeze all replicas only after every resync completes; a remote
volume's readonly mark is inert, so the freeze adds no extra cost.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* storage: abort volume destroy when remote delete fails

destroyLocked removed local files even when the backend delete errored,
leaving a registered replica with no .dat/.idx and a leaked remote
object. Run the remote delete before local cleanup and bail out on
failure so the volume stays fully usable and the delete can be retried.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: mirror destroy ordering for remote delete failures

Propagate a failed remote-object delete instead of warning past it, and
reinsert the volume into the disk location registry so a still-intact
replica is not unregistered.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* shell: recover partial tier.compact freezes and report held replicas

A failed freeze left already-marked replicas readonly with nothing
changed, and later stage failures returned without saying the replicas
stay frozen or how to resume. Roll back the partial mark when freezing
fails, note in every mid-flight error that replicas remain readonly and
rerun is the recovery path, and report how many replicas a partial
unfreeze released.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* shell: retire abandoned remote objects during tier.compact resync

A resync overwrote each replica's .vif, orphaning its previous remote
object; an interrupted run could leave replicas on different keys, and
only the last tiered replica's object was deleted. Read every replica's
remote file list up front, count holders per object, and delete-before-
copy whichever replica is an object's last holder, so shared and
stranded objects alike are retired instead of leaked.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: keep collection members registered when destroy fails

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-16 19:45:17 -07:00
Chris LuandDevin d3a97f76bd s3: fix PutObject If-Match under suspended versioning (#11780)
* s3: keep suspended-versioning conditional writes off the routed condition

A suspended PUT writes the null-version slot at the regular path while the
entry the conditional headers evaluate lives under .versions/. The routed
WriteCondition checked the request's etag against that empty slot, so a
correct If-Match failed with 412. Passing uniqueWritePath sends conditional
requests through the lock path, whose precondition resolves the same entry
the read path serves; unconditional writes still route.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer_pb: add condition_alt_key to ObjectTransactionRequest

* filer: evaluate a transaction condition against a fallback entry

condition_alt_key names the entry consulted when the primary is absent,
so a suspended-versioning write can check 'the null slot if present,
else the latest version' atomically under the lock.

* s3: route suspended-versioning conditional writes with a fallback key

The gateway names the latest version file as condition_alt_key, so the
transaction checks the entry the read path serves — null slot first,
latest version when it is absent — under the same lock that applies
the write. Conditional suspended PUTs route again instead of splitting
check and write across a gateway lock that routed writes skip.

* filer: resolve the condition fallback under the transaction lock

The gateway named the pointed-at version file as condition_alt_key,
resolved before the transaction ran: a concurrent version delete could
move the latest pointer between the gateway read and the locked check,
and a delete-marker latest read as an existing object. The alt key is
now the .versions directory and the filer resolves the latest-version
pointer under the same lock that serializes the write — a delete marker
resolves to absent, and unresolvable pointer state fails closed.

* filer: fail closed on alt-key lookups and skip directory primaries

A transient FindEntry error returning a nil entry read as "versions dir
absent" and let If-None-Match writes pass. Check the error before the
entry. And a bare directory at the object path satisfied the primary
lookup, so the .versions fallback never ran — directories are not the
object, treat them as an absent slot.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: scope directory-as-absent conditions to alt-key callers

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-16 19:44:26 -07:00
github-actions[bot] cdc6976e3a docs: regenerate star history chart 2026-10-17 00:45:25 +00:00
Chris Lu 921535411b s3: default allowDeleteBucketNotEmpty to false (#11769)
* s3: default allowDeleteBucketNotEmpty to false

DeleteBucket on a non-empty bucket silently destroyed every object,
while AWS returns 409 BucketNotEmpty. Clients like 'aws s3 rb' rely on
that refusal as the 'delete only if empty' safety net.

Deployments that need the recursive delete can opt back in with
-allowDeleteBucketNotEmpty=true.

* test: pin allowDeleteBucketNotEmpty=true on versioning config1

The config1 run relied on the flag's old true default to contrast with
config2's false. Pin it so the two modes stay distinct.

* test: drop redundant allowDeleteBucketNotEmpty=false in s3 integration test

False is the default now, so TestS3DeleteBucketNotEmpty exercises the
out-of-box behavior.
2026-10-15 23:45:36 -07:00
Chris LuandDevin 651546ae74 mount: release flock locks before the close-time flush (#11726)
* mount: release flock locks before the close-time flush

Release ran the dirty-handle flush before dropping the file
description's flock locks. A flush parked on a dead filer kept a dead
holder's lock in the table, so a restarted process taking flock on the
same file saw "file already locked" until the mount restarted.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* mount: finish the blocked-flush release test deterministically

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* mount: let the release test's parked flush talk to a live filer

Marking the handle deleted raced with the flush's unlocked early read
of isDeleted. Pointing the fixture at a live fake filer lets the flush
complete on its own once the handle lock frees, no flag write needed.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-15 22:38:03 -07:00
dependabot[bot]andChris Lu 82f0a37efe build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob from 1.8.0 to 1.8.2 (#11740)
* build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob

Bumps [github.com/Azure/azure-sdk-for-go/sdk/storage/azblob](https://github.com/Azure/azure-sdk-for-go) from 1.8.0 to 1.8.2.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.8.0...sdk/azidentity/v1.8.2)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
  dependency-version: 1.8.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* test: restore loadBucketMetadataFromFiler stub after TestGetBucketMetadata

TestGetBucketMetadata replaced the package-level loadBucketMetadataFromFiler
with a stub that sleeps one second per load and never restored it. Every
test running after it in the same binary paid that sleep on each bucket
metadata cache miss: TestPutObjectUploadACL (~245s) and
TestCopyObjectDestinationACL (~46s) each created a fresh BucketRegistry per
subtest, so the stub fired hundreds of times. In CI this stretched the
Test job past 11 minutes and left it exposed to runner cancellation.

Restore the real loader via t.Cleanup so the slow stub is scoped to the
test that needs it.

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-15 21:53:16 -07:00
dependabot[bot] 9aae0bb702 build(deps): bump helm/kind-action from 1.15.0 to 1.15.1 (#11747)
Bumps [helm/kind-action](https://github.com/helm/kind-action) from 1.15.0 to 1.15.1.
- [Release notes](https://github.com/helm/kind-action/releases)
- [Commits](https://github.com/helm/kind-action/compare/v1.15.0...v1.15.1)

---
updated-dependencies:
- dependency-name: helm/kind-action
  dependency-version: 1.15.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-15 21:53:01 -07:00
dependabot[bot] 80060e1bd6 build(deps): bump github.com/a-h/templ from 0.3.1020 to 0.3.1070 (#11742)
Bumps [github.com/a-h/templ](https://github.com/a-h/templ) from 0.3.1020 to 0.3.1070.
- [Release notes](https://github.com/a-h/templ/releases)
- [Commits](https://github.com/a-h/templ/compare/v0.3.1020...v0.3.1070)

---
updated-dependencies:
- dependency-name: github.com/a-h/templ
  dependency-version: 0.3.1070
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-15 21:52:43 -07:00
dependabot[bot] 72aa17ff02 build(deps): bump github.com/aws/aws-sdk-go-v2 from 1.47.1 to 1.47.3 (#11741)
Bumps [github.com/aws/aws-sdk-go-v2](https://github.com/aws/aws-sdk-go-v2) from 1.47.1 to 1.47.3.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.47.1...v1.47.3)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2
  dependency-version: 1.47.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-15 21:52:20 -07:00
Chris Lu e6cf190596 ec: re-plan balance moves when a target reports no space left (#11759)
* ecbalancer: add Node.SetNoReceive to exclude a node as a move target

A node that has refused a shard copy at runtime (e.g. 'no space left' on the actual disk, issue #11753) can no longer be trusted as a destination for the remainder of a balance run, but it may still hold shards that must move away. SetNoReceive excludes the node from target selection and from receiving-side capacity accounting, while leaving it eligible as a move source.

* ec: re-plan balance moves when a target reports no space left

Moves are planned from a heartbeat snapshot, but concurrent copies let the volume server's real disk state diverge from it, so a planned target can refuse the copy with 'no space left' mid-run and abort the whole balance (issue #11753). Track the refusing targets during execution, mark them no-receive, re-collect the topology, and plan the remaining moves again. The loop terminates because a marked node can never be picked as a target again.

* volume server: return ResourceExhausted when no disk can take an EC shard

A failed EcShardsCopy carried code=Unknown even though the cause is exhausted local capacity; ResourceExhausted describes the refusal and lets clients distinguish it from internal errors. The message text is kept so message-based matching still works (the Rust volume server reports the same text under Internal).

* ec: replay applied moves onto the re-collected snapshot and surface failures

Addresses review on the re-plan loop: a fresh topology read can still describe an applied move while heartbeats lag, so journal each applied move and replay it onto the next snapshot; retry only when every failure was a 'no space left' refusal (an unrelated failure aborts instead of being hidden); and exclude refused nodes' slots from the free-space precheck so a balance with no remaining receiver fails instead of reporting success for an empty plan.

* ec: carry per-disk state when replaying applied moves

A move that landed on an explicit TargetDisk loses that attribution when replayed onto a stale snapshot, leaving the disk's free slots and shard count unchanged; the next plan could pick the same disk, hit another refusal, and over-exclude the node. Replay the shard onto the target disk's entry and move the disk-level counters, and restore the source disk's counters on removal.

* ecbalancer: add Node.SetDiskNoReceive to exclude a single disk as a target

A copy with an explicit disk_id can be refused while the node's other disks still have room; a disk_id 0 refusal already means the server searched every disk, so the scopes differ. disk.noReceive removes that disk from receiving capacity and disk selection while leaving the node and its sibling disks eligible.

* ec: record explicit-disk refusals as disk-scoped no-receive

recordMoveResult now distinguishes the two refusal scopes: TargetDisk>0 marks only (node, disk), while TargetDisk==0 marks the whole node since the server's auto-pick found no usable disk. The free-slot precheck subtracts refused disk slots, and the replan log names refused destinations as node:diskN.

* ecbalancer: skip no-receive disks in pickBestDiskOnNode

The balance phases pick the destination disk through pickBestDiskOnNode, which ranked disks by score only; a refused disk with a low shard count would be re-picked, fail again, and abort the run since no new destination was rejected.

* volume_move: export IsNoSpaceLeftError for move callers

Both the ec.balance replanner and the worker ec_balance task need to recognize the volume server's no-space refusal; keep one matcher next to the move RPCs instead of a private copy in weed/ec.

* worker: retry pinned-disk EC moves with a server-picked disk on no space

ec_balance detection always pins TargetDisk, so a full disk fails the job even when a sibling disk has room. On a no-space refusal to the pinned disk, retry the move once with TargetDisk 0 so the volume server picks a disk on the same node; the planner's node choice stands.
2026-10-15 18:47:07 -07:00
Youngju KIMandChris Lu ee2d98f664 mini: note mini.options and Docker filer.toml pitfalls in help (#11755)
* mini: note mini.options and Docker filer.toml pitfalls in help

Two behaviors of "weed mini" surprise users and are not mentioned in
its help text.

Flags are saved to <dir>/mini.options and applied again on the next
start to every flag that still has its default value. Removing a flag,
or passing it with its default value, therefore keeps the saved value.

The official Docker image ships /etc/seaweedfs/filer.toml, which sets
the leveldb2 store to /data/filerldb2. When filer.toml is found, the
filer ignores the default store dir derived from -dir, so moving -dir
away from /data leaves the metadata in /data. If /data is not
persisted, buckets look empty after the container is recreated.

Only the help text changes.

Signed-off-by: YoungjuKim <fjvbn2003@gmail.com>

* mini: update help notes for computed-only mini.options

---------

Signed-off-by: YoungjuKim <fjvbn2003@gmail.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-15 18:44:31 -07:00
Chris Lu 18b235da0d mini: surface saved options and warn when filer store escapes -dir (#11758)
mini: persist only computed options, and warn when filer store escapes -dir

- saveMiniConfiguration no longer writes command-line flags to
  mini.options; the file now only pins values mini computed itself
  (currently master.volumeSizeLimitMB), so a dropped flag can no longer
  resurrect on the next start. Existing files still apply once and are
  then rewritten without the saved flags.
- applyConfigFileOptions skips any flag explicitly provided on the
  command line or via WEED_* env vars, so passing a flag always wins
  over a saved value, including at its default value.
- mini prints the saved options it applies at startup (secret values
  redacted), since quietMiniLogs hides the previous V(1)/V(2) messages.
- warn once at startup when a filer.toml points a local filer store
  (leveldb/leveldb2/leveldb3, including path-specific <store>.<id>
  sections) outside -dir and -master.dir: the filer skips the
  -dir-derived store dir whenever a filer.toml exists, and the metadata
  is lost on container recreate if that path is not persisted.
2026-10-15 18:36:46 -07:00
github-actions[bot] 4c81015bd3 docs: regenerate star history chart 2026-10-16 00:43:27 +00:00
Chris Lu a5ccefb661 filer: escape LIKE wildcards in SQL list prefixes (#11757)
* filer: escape LIKE wildcards in SQL list prefixes

ListDirectoryPrefixedEntries bound prefix+"%" straight into a LIKE
clause, so '_' and '%' in object names (e.g. Langfuse keys like
call_03_...) acted as wildcards: wrong results and a ~40x slower
range scan because (dirhash, name) can no longer be narrowed.

Escape '!'/'%'/'_' in the prefix and declare ESCAPE '!' in the
generated list SQL so every abstract_sql backend (mysql, mysql2,
postgres, postgres2, sqlite) matches literally with identical
semantics. '!' is used instead of '\' because SQLite shares
SqlGenMysql but would reject the MySQL-style '\\' literal, and
'\' would still need escaping itself.

* filer: escape LIKE wildcards in YDB list prefixes

Same wildcard-in-prefix bug as the abstract_sql backends: YQL LIKE
treated '_' and '%' in the prefix as wildcards. Apply the shared
LikePattern helper and declare ESCAPE '!' (YQL allows any escape
char except '%', '_' and '\').

* test: cover literal '_'/'%'/'!' prefixes in sqlite filer listing

End-to-end regression test through SqliteStore against a real
database; runs in CI under the sqlite build tag.
2026-10-15 16:51:15 -07:00
103328e70f filer: create parent directories for lazily fetched remote entries (#11727)
* filer: create parent directories for lazily fetched remote entries

A remote-mounted object first read through the filer was persisted without
its parent directories, so remote.uncache and the remote cache evictor, which
walk down from the mount directory, never reached it: its cached chunks could
not be evicted.

Persist the missing parents with the entry. With those directories in place,
the empty-folder cleaner leaves folders under a remote mount alone: deleting
one removes its whole prefix on the remote, which can still hold objects
that were never read locally.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: keep remote objects at parent keys and remote prefixes of empty folders

Look up a lazily fetched entry's missing parents through lazy fetch before
creating them, so a remote object stored at a parent's key is materialized
and promoted to a prefix object rather than hidden behind a new directory.

Deleting a folder that is empty in the filer is metadata-only under a remote
mount, both for a non-recursive delete and for the expiry cleanup of empty
parents: the mount is populated lazily, so the folder can still hold remote
objects never read, and removing its remote directory deletes every object
under that prefix. This replaces the empty-folder cleaner's own guard, which
covered only one of those paths.

A metadata-only delete records no remote-deletion tombstone, locally or from
its metadata event: nothing is owed to the remote, so lazy reads stay open.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: stop the lazy parent lookup at a store error

Only a parent the store reports as not found is missing; a parent it cannot
read ends the walk, so the lookup does not stat ancestors above it on the
remote.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: stat each missing remote parent once

A lazy fetch that finds k missing parents asked FindEntry for each, and every nested fetch re-walked its own still-missing ancestors, re-stating remote keys already proven absent: 1+k(k+1)/2 remote stats for one first read. The flagged parent lookups now skip their own walk; their entries are created by the same CreateEntry that already persists the chain.

* filer: materialize parents for lazily listed remote entries

maybeLazyListFromRemote persisted children and the listing directory with skipCreateParentDir, so a first access that goes through the list path still left entries unreachable to a walk from the mount directory. Materialize the missing ancestors the same way the lazy fetch does, and let the synced_at directory create its parent chain.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-15 16:23:33 -07:00
Jay HemnaniandChris Lu 1562621c5e filer: record the size of a multipart upload saved inside the entry (#11734)
* filer: record the size of a multipart upload saved inside the entry

* filer: guard save-inside multipart append and check Content-MD5

Addressing review feedback:

- Reject a save-inside multipart append when the target is chunk
  backed. The buffered content cannot be attached to existing chunks,
  so previously the request grew FileSize without storing the appended
  bytes, and reads returned zeros for the added range.
- Compare the request Content-MD5 against the buffered content before
  saving, matching the regular multipart path, so a bad digest no
  longer commits the file.
- Extend the size test to check the saved entry MD5, and cover the
  rejected append against chunk-backed and inline targets plus the
  bad-digest rejection.

* filer: return 400 for unsupported append modes

The save-inside append rejection and the existing inline append and offset write rejections are deterministic client errors, but the autoChunk default case mapped them to 500. Give them shared constants and return 400 instead, and assert the status in the tests.

---------

Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-15 14:20:42 -07:00
Jay Hemnani 3659c369e0 webdav: read small files stored inline in the entry (#11733) 2026-10-15 06:38:53 -07:00
Robin Tweedie e12ec94747 fix(shell): compare fractional evacuation destination ratios (#11752) 2026-10-15 06:36:37 -07:00
dependabot[bot] 9e56651808 build(deps): bump github.com/go-redsync/redsync/v4 from 4.17.0 to 4.18.0 (#11743)
Bumps [github.com/go-redsync/redsync/v4](https://github.com/go-redsync/redsync) from 4.17.0 to 4.18.0.
- [Commits](https://github.com/go-redsync/redsync/compare/v4.17.0...v4.18.0)

---
updated-dependencies:
- dependency-name: github.com/go-redsync/redsync/v4
  dependency-version: 4.18.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-15 06:22:09 -07:00
dependabot[bot] d167521270 build(deps): bump helm/chart-testing-action from 2.8.0 to 2.9.0 (#11745)
Bumps [helm/chart-testing-action](https://github.com/helm/chart-testing-action) from 2.8.0 to 2.9.0.
- [Release notes](https://github.com/helm/chart-testing-action/releases)
- [Commits](https://github.com/helm/chart-testing-action/compare/v2.8.0...v2.9.0)

---
updated-dependencies:
- dependency-name: helm/chart-testing-action
  dependency-version: 2.9.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-15 06:20:36 -07:00
dependabot[bot] 8608a4e75d build(deps): bump github.com/ydb-platform/ydb-go-sdk/v3 from 3.151.1 to 3.155.1 (#11739)
build(deps): bump github.com/ydb-platform/ydb-go-sdk/v3

Bumps [github.com/ydb-platform/ydb-go-sdk/v3](https://github.com/ydb-platform/ydb-go-sdk) from 3.151.1 to 3.155.1.
- [Release notes](https://github.com/ydb-platform/ydb-go-sdk/releases)
- [Changelog](https://github.com/ydb-platform/ydb-go-sdk/blob/master/CHANGELOG.md)
- [Commits](https://github.com/ydb-platform/ydb-go-sdk/compare/v3.151.1...v3.155.1)

---
updated-dependencies:
- dependency-name: github.com/ydb-platform/ydb-go-sdk/v3
  dependency-version: 3.155.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-15 06:19:29 -07:00
dependabot[bot] 896e4dd192 build(deps): bump github/codeql-action from 4.38.2 to 4.38.3 (#11746)
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.38.2 to 4.38.3.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/v4.38.2...v4.38.3)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-version: 4.38.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-15 06:18:54 -07:00
github-actions[bot] 84512927ab docs: regenerate star history chart 2026-10-15 00:43:00 +00:00
github-actions[bot] 590a75b9d6 docs: regenerate star history chart 2026-10-14 00:42:11 +00:00
github-actions[bot] 8093ecf4c8 docs: regenerate star history chart 2026-10-13 00:52:42 +00:00
6840371774 sftp: keep a file's existing bytes when it is opened for write without truncate (#11723)
* sftp: keep a file's existing bytes when it is opened for write without truncate

* sftp: preserve sparse gaps and refuse unreadable entries when copying existing content

The copy used a single chunk-stream read, which ends at the first sparse
gap: the tail past a hole was dropped, and a remote-only entry with no
local chunks copied zero bytes, so closing the handle uploaded an empty
file. Read through PrepareStreamContent, which fills unstored regions
with zeros, and refuse entries that carry no local bytes at all.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: report a failed destination write from a completed chunk fetch

retriedStreamFetchChunkData broke out of the retry loop on a clean HTTP
end before checking writeErr, so a destination write that failed mid-
stream returned a short write and nil error. Check writeErr first.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* sftp: cover remote and all-holes sizes when copying existing content

Take the declared size from RemoteEntry when it exceeds FileSize, pad
inline content to the declared size, and treat a chunkless local file as
all holes rather than unreadable. A remote-only entry still refuses the
copy, now checked with a zero FileSize as well.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* sftp: size the existing-content copy the way filer.FileSize does

A locally shortened file still names its older, larger remote object;
taking RemoteSize unconditionally padded the copy back out to it.
filer.FileSize only honors the remote size when the remote is newer,
and also covers the chunk-extent size. A remote-only entry still
refuses the copy whenever it points at remote bytes.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-13 07:20:20 +08:00
dependabot[bot] d67fc47a61 build(deps): bump rustls from 0.23.37 to 0.23.45 in /seaweed-volume (#11707)
Bumps [rustls](https://github.com/rustls/rustls) from 0.23.37 to 0.23.45.
- [Release notes](https://github.com/rustls/rustls/releases)
- [Changelog](https://github.com/rustls/rustls/blob/main/CHANGELOG.md)
- [Commits](https://github.com/rustls/rustls/compare/v/0.23.37...v/0.23.45)

---
updated-dependencies:
- dependency-name: rustls
  dependency-version: 0.23.45
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-13 05:57:00 +08:00
mortezaandMorteza d0f3fad133 ec: avoid redundant needle.GetActualSize call in LocateEcShardNeedle (#11629)
LocateEcShardNeedle passed needle.GetActualSize(size, version) to
LocateEcShardNeedleInterval, which calls needle.GetActualSize internally.
This caused header and padding to be added twice to needle size.

Fixes #3864

Co-authored-by: Morteza <m.mohammadi@rahkar.team>
2026-10-13 05:50:09 +08:00
Чепурнов СемёнandChepurnov Simon 624a13ecc6 EC worker: pin jobs to replica-host workers, late-bind local source (#11718)
* feat: prefer same-host worker and local EC replica

Admin executor pick ranked idle workers only, so a job whose volume
lived on full-03 could still run on full-01 and pull the whole .dat
across the LAN. Rank workers whose Hello host matches any replica
first (fallback to others if the local worker is full).

Detection named Sources[0] / the lowest server id; execute then
SyncAndSelectBestReplica and could switch the copy source to another
rack. After sync, if a replica on this machine has a complete file
count, copy from that URL so affinity and pull stay on one host.

* feat: pin scheduler jobs to replica-host workers until load K

Reserve prefers the least-loaded replica-host worker and waits while
that load is below 5 instead of spilling to a free remote worker.
K=0 restores immediate spill. Waterfill uses reservation load so four
idle preferred workers each get a job.

* feat: park scheduled jobs on per-worker admin queues

Assign a detection batch to the least-loaded replica-host worker in
one pass, then let ExecutionConcurrency goroutines pop a home-runnable
job or steal when that home queue reaches K. Dispatch no longer holds
a job in reserve-backoff, so a free worker is not blocked by another
home's pin. Tracked jobs stay pending/queued and record the home
worker for the admin UI.

* fix: rehome queued jobs when the home worker dies

A live home still pins below K. A home missing from live executors
is stolen immediately onto a free worker so a short orphaned queue
does not sit until the dispatch window ends.

* feat: wait before stealing a dead home worker queue

Orphaned queues are held for 30s so a worker reconnect blink does
not dump jobs onto other machines. After the grace, steal without K
as before. Zero disables the wait. A home that comes back during
grace keeps its queue.

* fix: late-bind EC from to worker hostname

After replicas are consistent (equal FileCount), rebind encode source
to a volume server on the same hostname as worker -address. Any port
on that host is local. Diverged FileCount still uses the max replica.

* feat: assign homeless jobs to global FIFO

Jobs whose replica hostname has no registered worker skip pin and orphan grace.
Free workers take own home, then steal at K/orphan, then leftover FIFO.

* fix: take homeless FIFO before steal for no-home workers

Workers without a replica-host match take leftover first; steal (K/orphan)
only when FIFO is empty. Home workers keep own-queue, steal, then leftover.

---------

Co-authored-by: Chepurnov Simon <195125934+CHEPURNOV-S-S@users.noreply.github.com>
2026-10-13 05:31:34 +08:00
492e700d1f Volume grow avg fill threshold (#11725)
* master: add conditional grow by average fill ratio to VolumeGrow gRPC

Callers that pre-grow volumes on fill level currently have to list all
volumes of a collection and sum sizes themselves, which is expensive on
clusters with millions of volumes. The master already tracks each
writable volume's effective size in memory, so it can answer the
question directly.

Add grow_avg_fill_threshold to VolumeGrowRequest: when set (e.g. 0.6),
growth only happens once the layout's writable volumes have reached
that average fill ratio. A layout with no writable volumes skips the
check, since that is the case growth exists to fix. VolumeGrowResponse
now reports the observed avg_fill and writable_count whether or not
growth was triggered.

* master: unit tests for WritableAvgFill and FindVolumeLayout

Covers the average-fill computation (mixed fills, no writables, a
writable volume without a size-tracking entry yet) and that
FindVolumeLayout resolves existing layouts without creating missing
collections.

* address review: report avg_fill and writable_count for every VolumeGrow

The stats were only read when grow_avg_fill_threshold was set, so a
request without a threshold got AvgFill=0 and WritableCount=0 even when
writable volumes existed. Read the layout stats unconditionally and use
the threshold only to decide whether growth happens.

* master: scope the grow fill check to the requested target

A VolumeGrow aimed at a data center, rack or node must not skip growth
because fill elsewhere keeps the layout average below the threshold.
Measure only writables serving the target; a target with none still
grows.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-13 05:30:03 +08:00
b5ee5b871e sftp: fail a directory listing on a stream error, and page past hidden entries (#11724)
* sftp: fail a directory listing on a stream error, and page past hidden entries

* sftp: test a fully hidden listing page

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 19:34:46 +08:00
6fb21c9ec0 filer: copy a remote-only entry on the remote when renaming it (#11709)
* filer: copy a remote-only entry on the remote when renaming it

Renaming a file under a remote.mount whose content exists only on the
remote (never cached, or remote.uncache'd) lost it: the new entry kept
the old RemoteEntry, deleting the old entry deleted the old object, and
with no chunks nothing ever wrote the new key. filer.remote.sync could
not finish the rename either, since the old object was already gone.

moveSelfEntry now copies such an entry's object to the new key before
it touches anything, and the new entry gets the RemoteEntry of the
copy. The copy is a new optional RemoteStorageObjectCopier interface,
implemented for S3 with CopyObject up to 5 GB and UploadPartCopy above
that; the multipart copy carries the source's metadata and, where the
remote supports tagging, its tags. When an existing target is
overwritten, its delete keeps the remote object (the copy replaced it)
and its event is marked metadata-only so the sync leaves it alone too.

A remote-only rename that leaves its mount, or whose remote cannot
copy, is refused instead of deleting the only copy. Empty entries and
entries with local data rename as before.

UpdateFileMetadata builds its copy source with the same helper, which
escapes each key segment and keeps the slashes.

* filer: delete a copied rename's old object only after the commit

Review follow-ups:

- The old key of a copied remote-only entry, and the old directory of a
  rename that copied one, are now deleted once the rename has committed.
  A rename that fails after the copy leaves the source entry and its
  object in place, plus an extra object at the new key.
- Every part of a multipart copy is conditioned on the ETag the copy
  started from, so an object overwritten mid-copy fails the copy
  instead of mixing versions.
- Both copy paths keep the source's server-side encryption settings.
- filer.remote.sync no longer deletes a key that a remote-only entry
  holds by the time the delete event is processed: a file deleted and
  then replaced by a remote-only rename has its content only there.

* filer: do not sweep the old prefix of a directory rename that copied

Removing the old directory on the remote deletes everything under its
prefix. Run after the commit, that could also take an object another
rename copied in under the same path meanwhile. When a child was copied,
leave the prefix alone and delete only the copied children's old keys.

* filer: keep an old key a remote-only entry has retaken by cleanup time

A directory rename does not lock its children's paths, so between the
commit and the deferred delete of a copied child's old key another
rename can put a remote-only entry there, whose content is then only in
that object. Check the store before deleting.

* filer: keep a renamed entry's old object when the path lookup fails

The deferred delete must only run once the lookup shows no remote-only
entry at the old path. A failed lookup shows nothing, and an extra
object loses nothing.

* filer: copy over a remote-only rename target only after the commit

Copying the source object onto the target's key before the transaction
committed destroyed the target's only content, which a rolled-back
commit cannot restore. When the target is remote-only the copy now runs
after the commit: the new entry keeps the old key's stamp until the copy
lands and re-stamps it, the target's object is kept meanwhile, and a
failed copy keeps the old object so remote.sync can finish it.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* remote.sync: match the delete event's etag before deleting the object

A rename can copy another entry's content onto a key between the filer's
check and the remote delete, and the delete then removes the copy. The
delete event carries the object's etag; delete only when the live object
still matches it.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: finish a rename's remote work outside the committed transaction

The SQL stores keep the transaction in the context and refuse every
call through it once it has committed. The work that runs after
CommitTransaction used that context, so on postgres2 the deferred
old-key delete could not look up the old path and kept the object, and
finishRemoteCopies could not read or stamp the entry. Both failed with
"sql: transaction has already been committed or rolled back" on every
copied rename; remote.sync cleaned up after them.

Keep the context from before the transaction for the work after the
commit. The rename test store gets a strict mode that fails calls
through a committed transaction the way database/sql does.

* filer: carry a deferred copy's stamp in the rename event; delete orphaned uploads

- The deferred copy stamps the entry with a plain UpdateEntry, which
  publishes no event, so the queued rename event still carried the old
  key's stamp to peer filers and subscribers. Give it the copy's
  RemoteEntry before it is sent. The two notify loops share the code now.
- remote.sync kept any object whose ETag changed since the delete. With
  no entry at the path that is an upload which finished after the
  delete, and keeping it leaves it on the remote for good. Look up the
  path first and keep a changed object only when an entry holds it.

* remote.sync: keep a changed object even with no entry at its path

Deleting a changed object when no entry holds the path also deletes a
copy a directory rename has written but not yet made visible, while a
delete of that path waits in the sync; the rename then removes the old
key and the content is gone. Keep any object whose ETag changed since
the delete, as before. An upload that finishes after the delete leaks
its object instead, which is the cheaper failure.

* filer: read a pending remote rename copy from the source key

A remote-only rename over a remote-only target commits before its copy
lands, so reads on the new path served the target's old object until the
copy stamped the entry — and indefinitely when the copy failed without
remote.sync running. The committed entry now records the renamed-from
path, and remote-only reads resolve that object's key while the stamp
is pending; the copy's stamp and the sync's completion stamp both clear
the marker.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: stamp the rename event with the deferred copy's result

A remote-only rename over a remote-only target queued the new entry's
event with the source's stamp before the deferred copy ran. Peers
replaying it saved a stamp the copy had already superseded, and the
sync serving such a peer rejected the destination as a match and tried
to copy the deleted source. The event now carries the copy's result
and drops the pending-source marker once it lands.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* remote.sync: delete only the object the event's stamp describes

An object at a deleted key is not always the one the delete meant: a
rename copy can land before its filer entry is visible, and a rewrite
can upload over it. When the event recorded no remote stamp, or the
live stat cannot confirm the recorded etag, the delete now keeps the
object instead of treating it as a late upload.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: tighten the pending rename source marker

Follow-ups to reading a pending remote rename copy from its source key:

- Only the filer sets the marker. The HTTP upload and tagging handlers
  store Seaweed-* request headers as extended attributes, so they skip
  this one, and a marker that leads out of the entry's own mount is
  ignored, so it cannot point reads at another remote.
- A pending entry renamed again copies from the marker's source, not
  from its own key, which still holds the old target's bytes. A further
  deferred rename keeps naming the original source; an immediate copy
  clears the marker.
- When remote.sync finds the destination already holding the content,
  it stamps the entry (clearing the marker) before deleting the source,
  and deletes the source only once the entry no longer reads it.
- A read that resolved the source just before the copy finished and
  the source went away retries with the entry as it is now.

* filer, remote.sync: follow a pending rename's source across renames

When a remote-only entry is renamed over another remote-only file and
the deferred copy fails, it reads the source key through its marker.
Renamed again the same way, the marker keeps naming that source, but:

- remote.sync completed each rename from the event's old key. For the
  second one that is the first target's old object, which it copied
  over the destination before clearing the marker. It now copies from
  the source the entry's marker names.
- For the first rename it found no entry at the new path and deleted
  the old key, which the later entry still reads. When the event's new
  entry carried a marker and the path is gone, it now leaves the key to
  the entry that moved on.
- When the filer copies a pending entry immediately on a later rename,
  it now also releases the source the copy read, after the commit and
  with the same checks as the old key.

* remote.sync: release a pending rename's source, the filer leaves it

The filer deleted a pending entry's source after copying it on a later
rename, but that rename locks only its own paths: another rename can
copy onto the source key before its entry is visible, and the cleanup
then deleted that copy. And deleting a pending entry never released its
source at all.

The filer now leaves marked sources alone. remote.sync releases one
when the pending entry is deleted, or when a later rename was copied
right away, and only while the live object is still the one the
entry's stamp describes and no remote-only entry holds the path: a
copy that landed there since has a different ETag and is kept.

* remote.sync: release a pending source when the renamed entry is cached

A rename copied right away left its pending source to remote.sync,
which released it only while the destination was still remote-only.
Cached before the sync got there, the destination took the other path
and the source stayed on the remote for good. Release it there too,
with the same checks.

* remote.sync: release a pending source only after the destination is safe

e6d6438 released the source a renamed entry's old version read before
the destination was uploaded. A cached destination that still carried
the pending marker then had nothing behind it: had its upload failed,
remote.uncache could drop its chunks and leave it reading a deleted
source. Release the source only after the upload is stamped, and only
when the entry no longer reads through a marker.

* remote.sync: delete a rename's old key only if it is still the renamed content

completeRemoteOnlyRename deleted the old key once the destination held
the content. Another rename onto the vacated path can copy its object
there before the sync reaches the first rename, and that copy was then
deleted with it. Delete the old key only while the live object is still
the one the renamed entry's stamp describes.

Also keep the pending-source marker out of DELETE ?tagging, which
clears every Seaweed-* attribute: the marker is not a tag, and losing it
points reads back at the old target's bytes.

* remote.sync: compare a rename's old key against the source's own stamp

A retry can find the renamed entry already stamped with the copy, and a
copy of a multipart source has another ETag, so the unchanged source
was kept and stayed on the remote. Carry the stamp the event recorded
for the source (or the pending entry's, for a marked source) and compare
against that.

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 19:21:54 +08:00
41260a55f6 master: record writableVolumeCount on HTTP /dir/assign like gRPC Assign (#11716)
* master: extract the periodic per-layout grow decision

Move the must-grow and crowded-step decision of the periodic growth scan
in ProcessGrowRequest into periodicGrowCount, a pure function of the
layout's lastGrowCount, writable and crowded counts, so it can be tested
without running the scan loop or growing real volumes.

No behavior change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* master: record writableVolumeCount on HTTP /dir/assign

The periodic growth scan (#5781) keeps a layout supplied ahead of
demand only when it knows how many writable volumes clients want:
lastGrowCount, which gRPC Assign records with SetLastGrowCount. HTTP
/dir/assign parsed writableVolumeCount but only used it to size the
emergency grow when no writable volume was left, so the scan always saw
lastGrowCount 0 for HTTP-only writers.

Since #5821 removed the per-assign threshold grow, such a layout only
grew by the rack-aware step of 2, and only once every writable volume
was crowded. When a batch of volumes filled in lockstep, those 2 took
every write and could fill before the next scan, leaving no writable
volumes until an assign-triggered emergency grow landed, with 503s for
every other assign meanwhile.

Record writableVolumeCount on HTTP assigns as gRPC Assign does. Clients
that do not send it are unaffected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* master: reject unknown rack/dataNode before recording grow targets

An assign naming a rack or data node outside the topology can never be
served, but lastGrowCount was recorded first. The periodic growth scan
drops the requested placement, so it kept topping the layout up with
volumes the requester could not use. Both assign paths now validate the
placement like they already validate the data center.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* master: scope placement existence checks to the requested path

A rack in another data center or a data node under another rack passed
validation yet could never serve the request, and the grow count was
still recorded. The rack and data-node lookups now take the requested
parents, matching the filters the placement engine applies.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 18:10:41 +08:00
3031ab5d97 EC placer: walk extra shards on freeSlots, keep 2+2 (#11714)
* fix: rotate EC extra shards on pickTarget count ties

pickTarget used to keep the first sorted key when shard counts tied,
so 10+4 onto 4 racks always put the two fours on full-01/full-02.
Tie-break on remaining freeSlots (same idea as pickNodeInRackEligible)
so the leftover shards walk across racks over a series of volumes.

* fix: pay within-rack EC shard debt on Place

Score nodes by live EcShards plus this Place's assignments, not
this-volume counts or EC-volume presence. Allow SameRackCount+1
only after a sibling already holds a shard, so 4-on-2 becomes 3+1
while 3-on-2 stays 2+1. Rack pickTarget tie-break is unchanged.

* fix: even-spread EC disks, pay debt with remainders only

Do not pack 3+1: three shards on one disk is one disk-loss from
unrecoverable. Place this volume evenly first; leftover shards
(2+1) go to the thinner live count. 4-on-2 stays 2+2.

* fix: score Place extras by freeSlots then liveShards

Even-per-volume and SameRackCount stay; remainder prefers capacity,
then shard debt. No 3+1.

* fix: keep shard-debt fixture capacity-consistent

freeSlots = cap-shards so extras keep paying the thinner disk
across volumes instead of oscillating after the first pair.

* refactor: document dropped mFree; switch EcBalance node pick

Keep host even (mCount). Remainder is disk freeSlots then liveShards so
thin disks fill first. SameRackCount / 2+2 unchanged. pickTarget unchanged.

* test: cover Place 10+4 extras on thin disks/racks

Seed liveShards/freeSlots imbalance on 1-1-{4,6,7,8,15}, 4-2-3, 5-2-2, 6-1-2, 7-1-2. Remainder tails must miss fat first ids; remainder 0 freezes the live gap.

* fix(ecbalancer): count only matching-type slots when picking a move target

Aggregate freeSlots let a rack, machine, or node whose free capacity sat
on another disk type win placement; pickBestDiskOnNode then found no
eligible disk and the move went out with TargetDisk 0, which the copy
endpoint reads as automatic selection onto any disk.

Score candidates on freeSlotsForType — per-disk metadata when present,
the node aggregate when absent — so a wrong-type destination is never
chosen, and cover pickBestNodeForVolume's tie-breaks and the wrong-type
plan case.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ec_balance: spread free slots per disk type, keeping the remainder

Dividing the node's free slots evenly over all its non-full disks floored
a leftover slot to zero on every disk — one free slot across two disks
made the node look full. It also gave disks of other types a share of
slots counted only for the requested type. Each type's slots now spread
over that type's disks with ceil division, so the total is preserved and
every usable slot stays visible to the planner.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Chepurnov Simon <195125934+CHEPURNOV-S-S@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 18:09:53 +08:00
acb44a68a9 fix(volume): skip unnecessary needle data reads on delete (#11711)
* fix(volume): do not read a needle's data to delete it

A delete read the needle's whole payload from disk only to discard it.
The delete handler and BatchDelete need the needle's cookie and its
chunk-manifest flag, which live in the header and trailer, but fetched
them with a full needle read, the same call a GET uses.

readNeedle now reads the header and trailer only when MustMetaOnly is
set, whatever the needle's size, and the delete paths ask for that. The
Rust volume server honours must_meta_only the same way for its delete
paths. A compressed needle or a chunk manifest still gets the full read,
as before, because their callers need the bytes.

HEAD is unchanged. It used to set MustMetaOnly, which nothing read; it
now asks for AttemptMetaOnly alone, so it keeps reading a needle up to
PagedReadLimit whole. On a spinning disk that one read is cheaper than a
header read followed by a trailer read.

Version 1 needles are always read in full. They have no data size field
and no metadata after the data, which ReadNeedleMeta relies on, so a
metadata-only read of one failed with "invalid needle meta size". That
already happened for version 1 needles above PagedReadLimit read with
AttemptMetaOnly.

A delete therefore no longer verifies the data CRC of the needle it
removes. The cookie check is unchanged.

* fix(volume): read transformed images fully on HEAD too

A HEAD carrying resize or crop params reports the transformed length,
which a metadata read cannot produce. Go and Rust both answered such
HEADs from the source header, returning the stored size instead of the
cropped or resized one.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 18:08:19 +08:00
Chris Lu 6927de6003 seaweed-worker: mint the admin bearer token like the Go worker
The admin control stream now requires mTLS or a token signed by
jwt.filer_signing.key; the Rust worker gets --filer-signing-key and
--filer-signing-key-expires-after-seconds, mirroring the Go worker's
config names, and attaches the token to the stream metadata.
2026-10-12 03:06:57 -07:00
Chris Lu 613a286cde worker: present the admin bearer token on control streams
Workers mint a SeaweedFilerAdminClaims token when jwt.filer_signing.key
is in their configuration, matching the authentication the admin server
now requires.
2026-10-12 02:55:50 -07:00
Chris Lu 55ac46f7a9 admin: authenticate worker and plugin gRPC streams
Worker registration previously ran unauthenticated on any interface the
listener bound. The stream now requires a verified mTLS client
certificate or a Bearer token signed by jwt.filer_signing.key, and falls
back to local clients only when no key is configured.
2026-10-12 02:55:50 -07:00
Chris Lu 3f8035deb7 security: share the admin gRPC peer authentication check
The local-peer fallback and bearer-token validation used by the filer IAM
gRPC path move into weed/security so every admin-plane gRPC service can
enforce the same contract. A client certificate verified by the server's
TLS config now also counts as authenticated.
2026-10-12 02:55:50 -07:00
Chris Lu a4c8d31cb8 s3api: ignore response header overrides on anonymous requests
AWS requires response-* query parameters to be signed; an anonymous
caller must not choose how a stored object is served.
2026-10-12 02:49:55 -07:00
Chris LuandDevin 6bac506842 sftp: terminate the missing-ancestor permission walk at the root
A path whose ancestors are all absent recursed on the root forever,
since Dir("/") is "/".

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 02:41:28 -07:00
Chris LuandDevin 2c031d9d63 sftp: check write permission on the target file, not only its parent
newFileWriter authorized uploads against the directory, so a file with
a more specific read-only ACL or mode could still be overwritten. The
same parent-directory fallback used for nonexistent paths now applies
through CheckFilePermission on the target itself.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 02:41:28 -07:00
Chris LuandDevin d69f45ee83 s3: validate multipart initiate ACLs like PUT
CreateMultipartUpload accepts canned and custom ACLs on initiation;
run them through the same authorization path as PutObject so an
explicit ACL requires s3:PutObjectAcl, and persist the validated
grants on the upload entry for completion to apply.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 02:41:28 -07:00
Chris LuandDevin 3aaf80b33b s3: stop trusting internal ACL headers in ParseS3Metadata
Multipart initiation stored client-supplied Seaweed-X-Amz-Acl and
Seaweed-X-Amz-Owner request headers onto the upload entry, and
CompleteMultipartUpload copied them to the finished object, letting a
caller forge ownership or a public ACL without s3:PutObjectAcl.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 02:41:28 -07:00
Chris LuandDevin 85f04b54fd s3tables: check data-path ancestors for catalog markers on DeleteTable
The decoupled-delete guard only inspects the resolved data path itself,
but the catalog marker lives on a table's name path, so a location that
resolves inside another table's directory (its marker-less data
subdirectory) slips through and the recursive purge destroys the other
table's data. Walk the data path's ancestors within the bucket and
refuse when any carries a table or view catalog entry.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 02:41:28 -07:00
Chris LuandDevin 6f93f87446 s3tables: confine caller-supplied metadataLocation to the table's namespace
CreateTable/RegisterTable/UpdateTable (and the view equivalents) validate
the location only for bucket and traversal, so a tenant can register a
table whose location resolves into another namespace's directory and
DeleteTable will purge it. Require the location to live under the
caller's namespace subtree, the boundary per-namespace delegation uses.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 02:41:28 -07:00
scooby87 73949daa98 filer: keep sequential prefetch within prefetchCount chunks (#11674)
ChunkReadAt.ReadAt asks the ReaderCache to prefetch after every read,
and MaybeCache counted only the downloads it newly started toward the
window, so each call reached prefetchCount chunks further than the last.
A reader consuming a chunk in several reads, as the S3 GET path does
with a 256 KiB copy buffer over 4 MiB chunks, ran the read-ahead up to
the ReaderCache downloader limit and kept most of a large object in
memory, enough to get an S3 gateway OOM-killed mid-response.

Count every chunk in the window, downloading or not, so the read-ahead
stays at prefetchCount chunks past the reader.


Assisted-by: LLM

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-12 17:41:01 +08:00
Chris LuandDevin 603a1221a7 filer: converge peer metadata replication on restart (#11698)
* filer: skip replayed peer events older than the local entry

Replay deleted and re-inserted every received event unconditionally, so
a restarted filer that rewound its subscription offset re-applied stale
peer writes over newer local state (values reverted, deleted entries
came back and pointed at already-vacuumed chunks). Compare the event's
TsNs against the local entry's Mtime and skip per-path, so a move event
can still remove its source while keeping a newer destination.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: add replay last-writer-wins regression tests

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: stop peer subscriptions and flush offsets on shutdown

Filer.Shutdown closed the store while subscription goroutines were still
mid-stream; their deferred offset save was never guaranteed to run, so a
restart rewound each peer's stream back to the last periodic flush and
replayed stale events. MetaAggregator.Shutdown now closes every peer's
stop channel and waits for the loops so the saved offset covers what was
replayed, and the reconnect sleep is interruptible by the stop channel.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: test that shutdown stops and waits for peer loops

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: order replay by persisted per-path commit timestamps

Review feedback found the mtime comparison still lets stale replay win:
a local delete removes the entry it would have been compared against, an
entry mtime is client-controllable, and the check ran unlocked so a local
write could land between it and the store mutation.

- Record the event timestamp per mutated path in the store KV on every
  emitted mutation, plus a delete record for sources of deletes and
  renames. The record survives the entry, so a replayed create older than
  a local delete no longer resurrects it; ancestor delete records cover
  children swept without per-child events.
- Replay compares the event timestamp against that record under the same
  per-path entry locks local RPC mutations hold, falling back to entry
  mtime only for entries written before records existed. Lookup errors
  now abort the event instead of being treated as an absent entry.
- MetaAggregator.Shutdown marks the aggregator stopped under peerChansLock
  so a master update cannot start a subscription that outlives the wait,
  and waits for the loops without the five-second cap now that every
  blocking stage — signature RPC, bootstrap traversal, retry sleeps, and
  per-event processing — cancels on the stop channel.

* filer: pin peer offset at failed replay, fix commit-record context and key size

A replayed event that exhausted its retries (or was interrupted by
shutdown) was still followed by advancing the peer watermark, so the
persisted offset skipped the unapplied event forever. replicateMetadataChange
now reports the outcome: errPeerSubscriptionStopped aborts the subscription
so the deferred save keeps the last completed event, and an exhausted retry
records a barrier the persisted offset and reconnect resume point cannot
pass while the stream keeps flowing.

Commit records also failed on two fronts: NotifyUpdateEvent runs after
CommitTransaction with the same context, so SQL stores wrote MpTs/MpDl
through an already-committed *sql.Tx; and metaPathKey embedded the whole
path, overflowing the 766-char name column SQL KV stores use, which the
KvPut fallback update then silently dropped. The writes now use a fresh
context and a sha256-hashed fixed-size key.

* filer: lock HTTP delete handler and replay against ancestor paths

The HTTP DeleteHandler called DeleteEntryMetaAndData without the shared
entry lock, so a peer replay could still interleave with a local delete and
its commit record. It now holds the same lock the gRPC DeleteEntry handler
does.

Replay also locked only the exact event paths, but its ordering check
consults ancestor delete records: a local recursive delete holds only the
ancestor's lock, so a stale child create could check before the tombstone
landed and then reinsert. lockReplayPaths now locks every ancestor of both
event paths in one lexicographic order — parents sort before children, as
the rename handler's two-path order already does — while never locking the
root, which cannot be deleted and would serialize all replays.

* filer: record replay commit before the replacement insert

If the same-path update DeleteEntry succeeded but InsertEntry failed, the path sat deleted with no commit record until the retry arrived, and a stale peer create could resurrect it in the gap. The commit record now lands immediately after the delete — without a delete record, since an update is not a delete — and again after the successful insert.

* filer: lock replay ancestors shared, not exclusive

Exclusive locks on every ancestor serialized all replays under a shared
parent — unrelated objects in the same bucket could no longer apply in
parallel. Ancestors only need to exclude an exclusive holder (a local
delete or rename of the ancestor), so they now take shared locks; a path
the event changes stays exclusive even when it is also the other path's
ancestor.

* filer: order peer bootstrap on the sender's commit timestamp

traversePeerMetadata compared localCommitTsNs — a commit-clock value —
against the entry's mtime, which a client can set freely: a peer change
committed recently with an old mtime was skipped, and a stale entry
mtime-stamped in the future overwrote a newer local commit.

TraverseBfsMetadataResponse now carries the sender's per-path commit
timestamp (recorded MpTs record, entry mtime as the unrecorded fallback).
The bootstrap applies an entry only when its commit is newer than the
local one and records the applied timestamp, so replayed events order
against it — closing the gap between bootstrap content and the event
stream. A peer without the field sends 0 and keeps the mtime fallback,
preserving rolling-upgrade behavior.

* filer: keep failed commit records pending until they land

notePathChange ran after the mutation committed and only logged a KvPut
failure: a delete whose MpTs/MpDl record never landed left no tombstone,
so a stale replayed create could resurrect the path while the store was
unreachable.

Failed records now stay in metaTsPending and a repair pass retries them
until they land. localCommitTsNs, deletedAfterTsNs and CommitTsNs
consult the pending record, so ordering holds in the window between the
committed mutation and the durable record. A per-path flag bounds the
repair to one goroutine, and respawns cover merges landing between its
last pass and the flag release.

* filer: keep only unwritten work in the pending commit record

The pending record merged every obligation into one timestamp and flag,
and a later successful write left it untouched: a repair pass could then
write an older timestamp over a newer one, and a merged delete inherited
the newer change's time, which the ancestor check reads as a newer
delete. The record now holds a change and a delete timestamp of its own,
both discharged under the record's lock alongside the store writes that
cover them, so a repair writes only what is still owed.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: fail traversal on a commit-record read error

CommitTsNs answered a store read failure with the entry mtime, a value
clients can set arbitrarily far ahead; a peer bootstrapping on it would
record the minted time and reject real updates below it. The lookup now
returns the error so the traversal restarts instead.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 17:39:34 +08:00
f11cecc697 helm: keep the S3 config checksum stable across the first upgrade (fixes #11719) (#11720)
* helm: keep the S3 config checksum stable across the first upgrade (fixes #11719)

checksum/s3config hashed the rendered s3-secret.yaml. That template
keeps its generated S3 keys with lookup, so rendering it again for the
checksum drew new keys whenever lookup found nothing: on install the
filer, S3 and all-in-one pods each carried a checksum of a different
set of keys than the Secret got, and the first upgrade with unchanged
values corrected it and restarted them.

Hash the values s3-secret.yaml is built from instead: the enable and
auth switches, existingConfigSecret, credentials, the legacy-secret
settings, the audit log configs and the Secret name. Only those, so
scaling the S3 gateway or changing its resources does not restart the
filer. A change to any of them still restarts the pods. Keys that the
chart generated stay out of the hash; they only change if the Secret
is deleted by hand.

The new render test fails on the old template (two checksums for one
Secret, different on every render) and passes now; it also checks that
the checksum follows the credentials and filer.s3.auditLogConfig and
ignores s3.replicas and s3.resources. On kind, an upgrade with unchanged
values no longer replaces the filer pod.

* helm: check the S3 checksum per workload, all-in-one included

Each workload's checksum/s3config is now read on its own (--show-only),
so a workload that loses the annotation fails the test instead of being
hidden by sort -u, and the all-in-one deployment gets the same checks.
Verified against two broken copies: one without the filer annotation,
one with the old all-in-one checksum; the test fails on both.

The helper comment now says what happens when the Secret is deleted to
rotate generated keys.

* ci(helm): fail the S3 checksum test on a missing or unrendered checksum

A failed helm template or a render without the annotation produced an
empty checksum, and the inequality assertions then passed against it.
s3_checksum now returns non-zero unless it found a checksum, and the
changed-value renders are assigned before comparing so failures cannot
hide inside test.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Ege Zambelli <egezamb@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 15:43:53 +08:00
1961 changed files with 259001 additions and 21162 deletions

No files matched your search

+2 -1
View File
@@ -20,4 +20,5 @@ ignore-regex = \b[a-z]+[A-Z]\w*\b|\b[A-Z][a-z]+[A-Z]\w*\b
# unknwon: GitHub username / Go module path (`github.com/unknwon/goconfig`)
# atleast: CLI mode literal string in test/benchmark/fuse_db/bin/sqlite_verify.py
# sme: local variable for a *streamMutateError in mount tests
ignore-words-list = visibles,fo,te,ser,bject,unparseable,keep-alives,tread,anc,ue,auther,thirdparty,unknwon,atleast,sme
# nin: AWS S3 Vectors "$nin" filter operator literal (API-defined, cannot rename)
ignore-words-list = visibles,fo,te,ser,bject,unparseable,keep-alives,tread,anc,ue,auther,thirdparty,unknwon,atleast,sme,nin
@@ -0,0 +1,11 @@
name: Clean up stale SeaweedFS test mounts
description: Detach disconnected FUSE test mounts left by earlier jobs on a shared runner.
runs:
using: composite
steps:
- name: Unmount disconnected SeaweedFS test mounts
if: runner.os == 'Linux' && runner.environment == 'self-hosted'
shell: bash
env:
CLEANUP_SCRIPT: ${{ github.action_path }}/cleanup.py
run: python3 "$CLEANUP_SCRIPT"
@@ -0,0 +1,89 @@
#!/usr/bin/env python3
"""Detach disconnected SeaweedFS integration-test mounts without touching live ones."""
import errno
import os
from pathlib import Path
import re
import shutil
import subprocess
import sys
import tempfile
def test_mounts(mountinfo):
"""Read the kernel table directly: traversing a stale mount may block."""
roots = {tempfile.gettempdir(), "/tmp", "/var/tmp"}
for line in mountinfo.splitlines():
fields, filesystem = line.split(" - ", 1)
if filesystem.split()[0].split(".")[0] != "fuse":
continue
fields = fields.split()
# mountinfo escapes whitespace and backslashes with octal sequences.
target = re.sub(r"\\([0-7]{3})", lambda m: chr(int(m[1], 8)), fields[4])
path = Path(target)
# Test mounts are a mount* leaf under a seaweedfs_fuse_* scratch dir
# (seaweedfs_fuse_test_*/mount, seaweedfs_fuse_failover_*/mount0, ...)
# or the posix-compliance workflow's fixed /tmp/seaweedfs/mount.
if (not path.name.startswith("mount")
or str(path.parent.parent) not in roots
or not (path.parent.name.startswith("seaweedfs_fuse")
or path.parent.name == "seaweedfs")):
continue
yield fields[0], target
def mounted(mount_id, target):
return (mount_id, target) in test_mounts(Path("/proc/self/mountinfo").read_text())
def cleanup():
failed = False
for mount_id, target in test_mounts(Path("/proc/self/mountinfo").read_text()):
try:
# A separate, bounded probe prevents a wedged FUSE daemon from
# hanging cleanup. Only ENOTCONN proves the daemon disconnected;
# a timeout or another error is not enough to detach a live mount.
probe = subprocess.run(
[sys.executable, __file__, "--probe", target], timeout=5,
capture_output=True, check=False,
)
except subprocess.TimeoutExpired:
print(f"::warning::FUSE probe timed out; leaving {target} mounted")
continue
if probe.returncode != errno.ENOTCONN or not mounted(mount_id, target):
continue
print(f"Unmounting disconnected FUSE test mount: {target}", flush=True)
# Lazy unmount detaches a dead mount even when an old process still
# holds it open. Prefer the unprivileged FUSE helper; sudo is fallback.
for command in (
["fusermount3", "-uz", "--", target],
["fusermount", "-uz", "--", target],
["sudo", "-n", "umount", "-l", "--", target],
):
try:
subprocess.run(command, timeout=10, check=False)
except (OSError, subprocess.TimeoutExpired) as error:
print(f"{command[0]}: {error}", file=sys.stderr)
if not mounted(mount_id, target):
break
else:
print(f"::error::Could not unmount disconnected FUSE test mount: {target}")
failed = True
continue
# The detached mount's parent still holds the crashed run's data and
# logs; test_mounts only yields targets under a SeaweedFS test dir in
# a temp root, so the parent is safe to reclaim here.
shutil.rmtree(Path(target).parent, ignore_errors=True)
return int(failed)
if __name__ == "__main__":
if len(sys.argv) == 3 and sys.argv[1] == "--probe":
try:
os.stat(sys.argv[2])
except OSError as error:
sys.exit(error.errno)
sys.exit(0)
sys.exit(cleanup())
@@ -0,0 +1,137 @@
import errno
import importlib.util
from pathlib import Path
import subprocess
import unittest
from unittest.mock import patch
spec = importlib.util.spec_from_file_location("cleanup", Path(__file__).with_name("cleanup.py"))
cleanup = importlib.util.module_from_spec(spec)
spec.loader.exec_module(cleanup)
TARGET = "/tmp/seaweedfs_fuse_test_123/mount"
MOUNTINFO = f"42 1 0:9 / {TARGET} rw - fuse.seaweedfs seaweedfs rw\n"
class CleanupTests(unittest.TestCase):
def run_cleanup(self, results, mounted=True):
with patch.object(cleanup.Path, "read_text", return_value=MOUNTINFO), \
patch.object(cleanup, "mounted", side_effect=mounted if isinstance(mounted, list) else None,
return_value=mounted) as check, \
patch.object(cleanup.shutil, "rmtree") as rmtree, \
patch.object(cleanup.subprocess, "run", side_effect=results) as run:
status = cleanup.cleanup()
return status, run, check, rmtree
def result(self, status):
return subprocess.CompletedProcess([], status)
def test_only_fuse_integration_test_mounts_are_candidates(self):
rows = MOUNTINFO
rows += MOUNTINFO.replace("fuse.seaweedfs", "ext4")
rows += MOUNTINFO.replace(TARGET, "/mnt/production")
rows += MOUNTINFO.replace(TARGET, "/tmp/seaweedfs_fuse_test_123/data")
rows += MOUNTINFO.replace(TARGET, "/home/user/seaweedfs_fuse_test_123/mount")
rows += MOUNTINFO.replace(TARGET, "/tmp/seaweedfs/data")
self.assertEqual(list(cleanup.test_mounts(rows)), [("42", TARGET)])
def test_all_test_mount_layouts_are_candidates(self):
targets = (
"/tmp/seaweedfs_fuse_test_123/mount",
"/tmp/seaweedfs_fuse_failover_456/mount0",
"/tmp/seaweedfs_fuse_failover_456/mount1",
"/var/tmp/seaweedfs_fuse_p2p_test_789/mount2",
"/tmp/seaweedfs/mount",
)
rows = "".join(
MOUNTINFO.replace("42", str(40 + i)).replace(TARGET, target)
for i, target in enumerate(targets)
)
self.assertEqual(
list(cleanup.test_mounts(rows)),
[(str(40 + i), target) for i, target in enumerate(targets)],
)
def test_mountinfo_paths_are_unescaped_once(self):
target = "/tmp/seaweedfs_fuse_test_space and\\slash/mount"
encoded = target.replace("\\", r"\134").replace(" ", r"\040")
rows = MOUNTINFO.replace(TARGET, encoded)
self.assertEqual(list(cleanup.test_mounts(rows)), [("42", target)])
def test_no_mounts_is_a_noop(self):
with patch.object(cleanup.Path, "read_text", return_value=""), \
patch.object(cleanup.subprocess, "run") as run:
self.assertEqual(cleanup.cleanup(), 0)
run.assert_not_called()
def test_healthy_mount_is_untouched(self):
status, run, _, _ = self.run_cleanup([self.result(0)])
self.assertEqual(status, 0)
self.assertEqual(run.call_count, 1)
def test_other_probe_errors_are_untouched(self):
for error in (errno.EACCES, errno.ENOENT, errno.EIO):
with self.subTest(error=error):
status, run, _, _ = self.run_cleanup([self.result(error)])
self.assertEqual(status, 0)
self.assertEqual(run.call_count, 1)
def test_probe_timeout_does_not_detach_a_possibly_live_mount(self):
status, run, _, _ = self.run_cleanup([subprocess.TimeoutExpired("probe", 5)])
self.assertEqual(status, 0)
self.assertEqual(run.call_count, 1)
def test_disconnected_mount_is_lazily_unmounted(self):
status, run, _, _ = self.run_cleanup([self.result(errno.ENOTCONN), self.result(0)], [True, False])
self.assertEqual(status, 0)
self.assertEqual(run.call_args_list[1].args[0], ["fusermount3", "-uz", "--", TARGET])
self.assertEqual(run.call_args_list[0].kwargs["timeout"], 5)
self.assertEqual(run.call_args_list[1].kwargs["timeout"], 10)
def test_replaced_or_removed_mount_is_untouched(self):
status, run, _, _ = self.run_cleanup([self.result(errno.ENOTCONN)], False)
self.assertEqual(status, 0)
self.assertEqual(run.call_count, 1)
def test_mount_identity_is_checked_without_stat(self):
with patch.object(cleanup.Path, "read_text", return_value=MOUNTINFO):
self.assertTrue(cleanup.mounted("42", TARGET))
self.assertFalse(cleanup.mounted("43", TARGET))
def test_missing_or_timed_out_helper_uses_fallback(self):
for error in (FileNotFoundError(), subprocess.TimeoutExpired("fusermount3", 10)):
with self.subTest(error=error):
status, run, _, _ = self.run_cleanup(
[self.result(errno.ENOTCONN), error, self.result(0)], [True, True, False])
self.assertEqual(status, 0)
self.assertEqual(run.call_args.args[0], ["fusermount", "-uz", "--", TARGET])
def test_failed_helpers_fall_back_to_sudo(self):
status, run, _, _ = self.run_cleanup(
[self.result(errno.ENOTCONN), self.result(1), self.result(1), self.result(0)],
[True, True, True, False])
self.assertEqual(status, 0)
self.assertEqual(run.call_args.args[0], ["sudo", "-n", "umount", "-l", "--", TARGET])
def test_cleanup_fails_when_stale_mount_remains(self):
status, run, _, rmtree = self.run_cleanup(
[self.result(errno.ENOTCONN), self.result(1), self.result(1), self.result(1)])
self.assertEqual(status, 1)
self.assertEqual(run.call_count, 4)
rmtree.assert_not_called()
def test_unmounted_test_dir_is_reclaimed(self):
status, _, _, rmtree = self.run_cleanup(
[self.result(errno.ENOTCONN), self.result(0)], [True, False])
self.assertEqual(status, 0)
rmtree.assert_called_once_with(Path(TARGET).parent, ignore_errors=True)
def test_probe_process_reports_errno(self):
for target, expected in ((str(Path(__file__)), 0), (str(Path(__file__) / "absent"), errno.ENOTDIR)):
with self.subTest(target=target):
result = subprocess.run([cleanup.sys.executable, cleanup.__file__, "--probe", target], check=False)
self.assertEqual(result.returncode, expected)
if __name__ == "__main__":
unittest.main()
@@ -23,8 +23,14 @@ runs:
bin=$(command -v fusermount3 || true)
if [ -z "$bin" ]; then
echo "no fusermount3 in PATH" >&2
exit 1
# Self-hosted runners are not guaranteed to carry fuse3.
sudo apt-get update
sudo apt-get install -y fuse3
bin=$(command -v fusermount3 || true)
if [ -z "$bin" ]; then
echo "no fusermount3 in PATH after installing fuse3" >&2
exit 1
fi
fi
if can_mount "$bin"; then
ls -l "$bin"
@@ -0,0 +1,57 @@
name: Set up a musl Rust target
description: Install the stable toolchain for a static Linux musl target, with the C toolchain and the link shims aws-lc-sys needs.
inputs:
target:
description: x86_64-unknown-linux-musl or aarch64-unknown-linux-musl
required: true
runs:
using: composite
steps:
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
with:
targets: ${{ inputs.target }}
- name: Install musl tools (amd64)
if: inputs.target == 'x86_64-unknown-linux-musl'
shell: bash
run: |
sudo apt-get install -y musl-tools
# C deps (aws-lc-sys) must compile against musl headers, not the host
# gcc's glibc headers — glibc's C23 redirects (__isoc23_sscanf,
# __isoc23_strtol) don't exist in musl and fail the link.
echo "CC_x86_64_unknown_linux_musl=musl-gcc" >> "$GITHUB_ENV"
- name: Install cross-compilation tools (arm64)
if: inputs.target == 'aarch64-unknown-linux-musl'
shell: bash
run: |
sudo apt-get install -y gcc-aarch64-linux-gnu
echo "CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_LINKER=aarch64-linux-gnu-gcc" >> "$GITHUB_ENV"
# Disable glibc fortify source — its __memcpy_chk etc. symbols don't exist in musl
echo "CFLAGS_aarch64_unknown_linux_musl=-U_FORTIFY_SOURCE" >> "$GITHUB_ENV"
# aws-lc-sys compiles against glibc headers, which redirect
# sscanf/strtol to C23 __isoc23_* symbols musl does not provide.
# Link a tiny forwarding shim so those references resolve to the
# musl implementations. Prototypes are declared manually: including
# glibc headers here would re-redirect the calls back to __isoc23_*.
cat > /tmp/isoc23-shim.c <<'SHIM'
#include <stdarg.h>
extern int vsscanf(const char *s, const char *fmt, va_list ap);
extern long strtol(const char *nptr, char **endptr, int base);
int __isoc23_sscanf(const char *s, const char *fmt, ...) {
va_list ap;
va_start(ap, fmt);
int r = vsscanf(s, fmt, ap);
va_end(ap);
return r;
}
long __isoc23_strtol(const char *nptr, char **endptr, int base) {
return strtol(nptr, endptr, base);
}
SHIM
aarch64-linux-gnu-gcc -O2 -c /tmp/isoc23-shim.c -o /tmp/isoc23-shim.o
# link-args land after libc on the link line; rescan musl libc so
# the shim's own vsscanf/strtol references resolve, and libgcc for
# the fp128 builtins musl's vfscanf/floatscan pull in on aarch64.
echo "CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_RUSTFLAGS=-Clink-arg=/tmp/isoc23-shim.o -Clink-arg=-lc -Clink-arg=-lgcc" >> "$GITHUB_ENV"
+1 -1
View File
@@ -15,7 +15,7 @@ runs:
using: composite
steps:
- name: Install cosign
uses: sigstore/cosign-installer@v4.1.2
uses: sigstore/cosign-installer@v4.1.3
- name: Sign
shell: bash
+25 -1
View File
@@ -9,6 +9,8 @@ logs a warning rather than under-reporting.
import json
import os
import sys
import time
import urllib.error
import urllib.request
from datetime import datetime
@@ -22,6 +24,28 @@ REPO = "seaweedfs/seaweedfs"
TOKEN = os.environ["GITHUB_TOKEN"]
OUT = os.environ.get("OUT", "note/star_history.svg")
PAGE_CAP = 400 # GitHub's hard limit on stargazer pagination
REQUEST_ATTEMPTS = 5
RETRYABLE_STATUS_CODES = {429, 500, 502, 503, 504}
def open_url(req):
for attempt in range(REQUEST_ATTEMPTS):
try:
return urllib.request.urlopen(req, timeout=30)
except urllib.error.HTTPError as error:
if error.code not in RETRYABLE_STATUS_CODES or attempt == REQUEST_ATTEMPTS - 1:
raise
retry_after = error.headers.get("Retry-After")
try:
delay = min(float(retry_after), 60) if retry_after else 2**attempt
except ValueError:
delay = 2**attempt
error.close()
except urllib.error.URLError:
if attempt == REQUEST_ATTEMPTS - 1:
raise
delay = 2**attempt
time.sleep(delay)
def fetch_stargazers():
@@ -38,7 +62,7 @@ def fetch_stargazers():
"User-Agent": "seaweedfs-star-history",
},
)
with urllib.request.urlopen(req) as resp:
with open_url(req) as resp:
batch = json.load(resp)
if not batch:
break
+75
View File
@@ -0,0 +1,75 @@
import importlib.util
import os
from pathlib import Path
import sys
import types
import unittest
from unittest.mock import patch
from urllib.error import HTTPError
def load_star_history():
matplotlib = types.ModuleType("matplotlib")
matplotlib.use = lambda backend: None
pyplot = types.ModuleType("matplotlib.pyplot")
dates = types.ModuleType("matplotlib.dates")
dates.AutoDateLocator = object
dates.DateFormatter = object
modules = {
"matplotlib": matplotlib,
"matplotlib.pyplot": pyplot,
"matplotlib.dates": dates,
}
with patch.dict(sys.modules, modules), patch.dict(os.environ, {"GITHUB_TOKEN": "test"}):
path = Path(__file__).with_name("star_history.py")
spec = importlib.util.spec_from_file_location("star_history", path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
star_history = load_star_history()
class StarHistoryTests(unittest.TestCase):
def http_error(self, code):
return HTTPError("https://api.github.com", code, "error", {}, None)
@patch.object(star_history.time, "sleep")
@patch.object(star_history.urllib.request, "urlopen")
def test_retries_server_error(self, urlopen, sleep):
response = object()
urlopen.side_effect = [self.http_error(500), response]
result = star_history.open_url(object())
self.assertIs(result, response)
self.assertEqual(urlopen.call_count, 2)
sleep.assert_called_once_with(1)
@patch.object(star_history.time, "sleep")
@patch.object(star_history.urllib.request, "urlopen")
def test_does_not_retry_non_transient_http_error(self, urlopen, sleep):
error = self.http_error(404)
urlopen.side_effect = error
with self.assertRaises(HTTPError):
star_history.open_url(object())
urlopen.assert_called_once()
sleep.assert_not_called()
@patch.object(star_history.time, "sleep")
@patch.object(star_history.urllib.request, "urlopen")
def test_stops_after_retry_limit(self, urlopen, sleep):
urlopen.side_effect = self.http_error(500)
with self.assertRaises(HTTPError):
star_history.open_url(object())
self.assertEqual(urlopen.call_count, star_history.REQUEST_ATTEMPTS)
self.assertEqual(sleep.call_count, star_history.REQUEST_ATTEMPTS - 1)
if __name__ == "__main__":
unittest.main()
+30
View File
@@ -0,0 +1,30 @@
name: "actions: unit tests"
on:
push:
branches: [ master ]
paths:
- '.github/actions/**'
- '.github/workflows/action-tests.yml'
pull_request:
branches: [ master ]
paths:
- '.github/actions/**'
- '.github/workflows/action-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
jobs:
cleanup-stale-fuse-mounts:
runs-on: ubuntu-latest
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Run cleanup action unit tests
run: python3 .github/actions/cleanup-stale-fuse-mounts/test_cleanup.py
@@ -0,0 +1,37 @@
name: "Cancel runs of closed PRs"
# Merging or closing a PR leaves its pull_request runs going: concurrency
# groups cancel a run only when a newer push arrives. pull_request_target gives
# this job a token that can cancel runs, fork PRs included. It checks out and
# runs no PR code.
on:
pull_request_target:
types: [closed]
permissions:
actions: write
pull-requests: read
jobs:
cancel:
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: Cancel the PR's queued and running pull_request runs
env:
GH_TOKEN: ${{ github.token }}
GH_REPO: ${{ github.repository }}
PR: ${{ github.event.pull_request.number }}
HEAD_REF: ${{ github.event.pull_request.head.ref }}
run: |
set -euo pipefail
# Match the PR's own commits: forks reuse branch names such as master.
shas=$(gh api --paginate "repos/$GH_REPO/pulls/$PR/commits" --jq '.[].sha')
for status in queued in_progress waiting pending requested; do
gh run list --branch "$HEAD_REF" --event pull_request --status "$status" \
--limit 200 --json databaseId,headSha --jq '.[] | "\(.databaseId) \(.headSha)"'
done | while read -r id sha; do
grep -qx "$sha" <<<"$shas" || continue
echo "cancel run $id at $sha"
gh run cancel "$id" || true
done
+11 -5
View File
@@ -9,8 +9,8 @@ on:
- '.github/workflows/codeql.yml'
concurrency:
group: ${{ github.head_ref }}/codeql
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
jobs:
CodeQL-Build:
@@ -25,9 +25,15 @@ jobs:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@v4.38.2
uses: github/codeql-action/init@v4.39.3
# Override language selection by uncommenting this and choosing your languages
with:
languages: go
@@ -35,7 +41,7 @@ jobs:
# Autobuild attempts to build any compiled languages (C/C++, C#, or Java).
# If this step fails, then you should remove it and run the build manually (see below).
- name: Autobuild
uses: github/codeql-action/autobuild@v4.38.2
uses: github/codeql-action/autobuild@v4.39.3
# ℹ️ Command-line programs to run using the OS shell.
# 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun
@@ -49,4 +55,4 @@ jobs:
# make release
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v4.38.2
uses: github/codeql-action/analyze@v4.39.3
+4
View File
@@ -8,6 +8,10 @@ on:
pull_request:
branches: [master]
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
+36 -20
View File
@@ -8,11 +8,16 @@ on:
- 'seaweed-volume/**'
- 'seaweed-common/**'
- 'seaweed-worker/**'
- 'seaweed-vector/**'
- 'docker/**'
- 'go.mod'
- 'go.sum'
- '.github/workflows/container_dev.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -20,7 +25,7 @@ jobs:
# ── Pre-build the Rust binaries natively ────────────────────────────
build-rust-binaries:
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
strategy:
matrix:
include:
@@ -28,29 +33,17 @@ jobs:
arch: amd64
- target: aarch64-unknown-linux-musl
arch: arm64
cross: true
steps:
- name: Checkout
uses: actions/checkout@v7
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
- name: Set up the musl target
uses: ./.github/actions/rust-musl-target
with:
targets: ${{ matrix.target }}
- name: Install musl tools (amd64)
if: ${{ !matrix.cross }}
run: sudo apt-get install -y musl-tools
- name: Install cross-compilation tools (arm64)
if: matrix.cross
run: |
sudo apt-get install -y gcc-aarch64-linux-gnu
echo "CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_LINKER=aarch64-linux-gnu-gcc" >> "$GITHUB_ENV"
# Disable glibc fortify source — its __memcpy_chk etc. symbols don't exist in musl
echo "CFLAGS_aarch64_unknown_linux_musl=-U_FORTIFY_SOURCE" >> "$GITHUB_ENV"
target: ${{ matrix.target }}
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
@@ -58,7 +51,8 @@ jobs:
~/.cargo/git
seaweed-volume/target
seaweed-worker/target/${{ matrix.target }}/release
key: rust-docker-dev-${{ matrix.target }}-${{ hashFiles('seaweed-volume/Cargo.lock', 'seaweed-worker/Cargo.lock') }}
seaweed-vector/target/${{ matrix.target }}/release
key: rust-docker-dev-${{ matrix.target }}-${{ hashFiles('seaweed-volume/Cargo.lock', 'seaweed-worker/Cargo.lock', 'seaweed-vector/Cargo.lock') }}
restore-keys: |
rust-docker-dev-${{ matrix.target }}-
@@ -91,6 +85,12 @@ jobs:
cargo build --release -p weed-lance-worker --target ${{ matrix.target }}
cp target/${{ matrix.target }}/release/weed-worker ../weed-worker-${{ matrix.arch }}
- name: Build the vector engine
run: |
cd seaweed-vector
cargo build --release -p weed-vector-server --target ${{ matrix.target }}
cp target/${{ matrix.target }}/release/weed-vector ../weed-vector-${{ matrix.arch }}
- name: Upload artifacts
uses: actions/upload-artifact@v7
with:
@@ -98,6 +98,7 @@ jobs:
path: |
weed-volume-normal-${{ matrix.arch }}
weed-worker-${{ matrix.arch }}
weed-vector-${{ matrix.arch }}
build-dev-containers:
needs: [build-rust-binaries]
@@ -135,8 +136,17 @@ jobs:
echo "Placed pre-built Rust worker for ${arch}"
fi
done
mkdir -p docker/weed-vector-prebuilt
for arch in amd64 arm64; do
src="./rust-bins/weed-vector-${arch}"
if [ -f "$src" ]; then
cp "$src" "docker/weed-vector-prebuilt/weed-vector-${arch}"
echo "Placed pre-built vector engine for ${arch}"
fi
done
ls -la docker/weed-volume-prebuilt/
ls -la docker/weed-worker-prebuilt/
ls -la docker/weed-vector-prebuilt/
- name: Docker meta
id: docker_meta
@@ -170,14 +180,14 @@ jobs:
- name: Login to Docker Hub
if: github.event_name != 'pull_request'
uses: docker/login-action@v4.6.0
uses: docker/login-action@v4.8.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Login to GHCR
if: github.event_name != 'pull_request'
uses: docker/login-action@v4.6.0
uses: docker/login-action@v4.8.0
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
@@ -193,6 +203,12 @@ jobs:
platforms: linux/amd64, linux/arm64
tags: ${{ steps.docker_meta.outputs.tags }}
labels: ${{ steps.docker_meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max
# Key the checkout layer on the pushed sha so cached layers can't
# serve stale code; the Dockerfile fetch fallback picks it up.
build-args: |
BRANCH=${{ github.sha }}
- name: Sign
if: github.event_name != 'pull_request'
@@ -24,6 +24,10 @@ on:
required: false
default: ''
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.event.inputs.image_tag || format('foundationdb_{0}_seaweedfs_{1}', github.event.inputs.fdb_version || '7.4.5', github.event.inputs.seaweedfs_ref || github.ref_name) }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -64,6 +68,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: go.mod
cache: ${{ runner.environment == 'github-hosted' }}
- name: Run FoundationDB tagged tests
env:
@@ -136,7 +141,7 @@ jobs:
- name: Login to Docker Hub
if: github.event_name != 'pull_request'
uses: docker/login-action@v4.6.0
uses: docker/login-action@v4.8.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
+55 -26
View File
@@ -30,6 +30,11 @@ on:
type: boolean
default: false
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.event.inputs.image_tag || github.run_id }}
cancel-in-progress: false
queue: max
permissions:
contents: read
security-events: write
@@ -61,7 +66,7 @@ jobs:
# ── Pre-build the Rust binaries natively ────────────────────────────
build-rust-binaries:
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
strategy:
matrix:
include:
@@ -69,31 +74,21 @@ jobs:
arch: amd64
- target: aarch64-unknown-linux-musl
arch: arm64
cross: true
steps:
- name: Checkout
uses: actions/checkout@v7
with:
ref: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.source_ref || github.ref }}
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
# The checkout above is the source ref, which may predate these
# actions; take them from master, the ref this fallback runs from.
- name: Set up the musl target
uses: seaweedfs/seaweedfs/.github/actions/rust-musl-target@master
with:
targets: ${{ matrix.target }}
- name: Install musl tools (amd64)
if: ${{ !matrix.cross }}
run: sudo apt-get install -y musl-tools
- name: Install cross-compilation tools (arm64)
if: matrix.cross
run: |
sudo apt-get install -y gcc-aarch64-linux-gnu
echo "CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_LINKER=aarch64-linux-gnu-gcc" >> "$GITHUB_ENV"
# Disable glibc fortify source — its __memcpy_chk etc. symbols don't exist in musl
echo "CFLAGS_aarch64_unknown_linux_musl=-U_FORTIFY_SOURCE" >> "$GITHUB_ENV"
target: ${{ matrix.target }}
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
@@ -101,7 +96,8 @@ jobs:
~/.cargo/git
seaweed-volume/target
seaweed-worker/target/${{ matrix.target }}/release
key: rust-docker-${{ matrix.target }}-${{ hashFiles('seaweed-volume/Cargo.lock', 'seaweed-worker/Cargo.lock') }}
seaweed-vector/target/${{ matrix.target }}/release
key: rust-docker-${{ matrix.target }}-${{ hashFiles('seaweed-volume/Cargo.lock', 'seaweed-worker/Cargo.lock', 'seaweed-vector/Cargo.lock') }}
restore-keys: |
rust-docker-${{ matrix.target }}-
@@ -142,6 +138,17 @@ jobs:
cargo build --release -p weed-lance-worker --target ${{ matrix.target }}
cp target/${{ matrix.target }}/release/weed-worker ../weed-worker-${{ matrix.arch }}
- name: Build the vector engine
run: |
# An older source_ref predates the engine; its image goes without one.
if [ ! -f seaweed-vector/crates/server/Cargo.toml ]; then
echo "No vector engine in this source ref"
exit 0
fi
cd seaweed-vector
cargo build --release -p weed-vector-server --target ${{ matrix.target }}
cp target/${{ matrix.target }}/release/weed-vector ../weed-vector-${{ matrix.arch }}
- name: Upload artifacts
uses: actions/upload-artifact@v7
with:
@@ -150,6 +157,7 @@ jobs:
weed-volume-large-disk-${{ matrix.arch }}
weed-volume-normal-${{ matrix.arch }}
weed-worker-${{ matrix.arch }}
weed-vector-${{ matrix.arch }}
build:
needs: [setup, build-rust-binaries]
@@ -164,10 +172,13 @@ jobs:
uses: actions/checkout@v7
with:
ref: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.source_ref || github.ref }}
- name: Clean up stale FUSE mounts
uses: seaweedfs/seaweedfs/.github/actions/cleanup-stale-fuse-mounts@master
- name: Free Disk Space
run: |
echo "Available disk space before cleanup:"
df -h
timeout 15 df -h || true
# Remove pre-installed tools
sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /opt/hostedtoolcache/CodeQL
# Clean package managers
@@ -179,7 +190,7 @@ jobs:
[ -d ~/.cache/go-build ] && rm -rf ~/.cache/go-build || true
[ -d /go/pkg ] && rm -rf /go/pkg || true
echo "Available disk space after cleanup:"
df -h
timeout 15 df -h || true
- name: Configure variant
id: config
@@ -219,8 +230,17 @@ jobs:
echo "Placed pre-built Rust worker for ${arch}"
fi
done
mkdir -p docker/weed-vector-prebuilt
for arch in amd64 arm64; do
src="./rust-bins/weed-vector-${arch}"
if [ -f "$src" ]; then
cp "$src" "docker/weed-vector-prebuilt/weed-vector-${arch}"
echo "Placed pre-built vector engine for ${arch}"
fi
done
ls -la docker/weed-volume-prebuilt/
ls -la docker/weed-worker-prebuilt/
ls -la docker/weed-vector-prebuilt/
- name: Docker meta
id: docker_meta
@@ -250,13 +270,13 @@ jobs:
buildkitd-config: /tmp/buildkitd.toml
- name: Login to Docker Hub
if: needs.setup.outputs.publish == 'true'
uses: docker/login-action@v4.6.0
uses: docker/login-action@v4.8.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Login to GHCR
if: needs.setup.outputs.publish == 'true'
uses: docker/login-action@v4.6.0
uses: docker/login-action@v4.8.0
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
@@ -304,7 +324,7 @@ jobs:
fi
- name: Login to GHCR
if: needs.setup.outputs.publish == 'true'
uses: docker/login-action@v4.6.0
uses: docker/login-action@v4.8.0
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
@@ -344,8 +364,17 @@ jobs:
echo "Placed pre-built Rust worker for ${arch}"
fi
done
mkdir -p docker/weed-vector-prebuilt
for arch in amd64 arm64; do
src="./rust-bins/weed-vector-${arch}"
if [ -f "$src" ]; then
cp "$src" "docker/weed-vector-prebuilt/weed-vector-${arch}"
echo "Placed pre-built vector engine for ${arch}"
fi
done
ls -la docker/weed-volume-prebuilt/
ls -la docker/weed-worker-prebuilt/
ls -la docker/weed-vector-prebuilt/
- name: Create BuildKit config for local scan build
if: needs.setup.outputs.publish != 'true'
run: |
@@ -405,7 +434,7 @@ jobs:
output: trivy-results.sarif
exit-code: '0'
- name: Upload Trivy scan results to GitHub Security
uses: github/codeql-action/upload-sarif@v4.38.2
uses: github/codeql-action/upload-sarif@v4.39.3
if: always()
with:
sarif_file: trivy-results.sarif
@@ -472,12 +501,12 @@ jobs:
ghcr.io/chrislusf/seaweedfs
tags: type=raw,value=${{ github.event_name == 'workflow_dispatch' && github.event.inputs.image_tag || 'latest' }},suffix=${{ steps.config.outputs.tag_suffix }}
- name: Login to Docker Hub
uses: docker/login-action@v4.6.0
uses: docker/login-action@v4.8.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Login to GHCR
uses: docker/login-action@v4.6.0
uses: docker/login-action@v4.8.0
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
@@ -11,6 +11,11 @@ on:
required: true
default: ''
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || format('refs/tags/{0}', github.event.inputs.release_tag) || github.run_id }}
cancel-in-progress: false
queue: max
permissions:
contents: read
@@ -53,7 +58,7 @@ jobs:
-
name: Login to Docker Hub
if: github.event_name != 'pull_request'
uses: docker/login-action@v4.6.0
uses: docker/login-action@v4.8.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
+89 -34
View File
@@ -25,7 +25,7 @@ on:
rocksdb_version:
description: 'RocksDB git tag to use when variant=rocksdb'
required: false
default: 'v10.10.1'
default: 'v11.1.2'
permissions:
contents: read
@@ -43,11 +43,11 @@ concurrency:
jobs:
# ── Pre-build the Rust binaries natively ────────────────────────────
# The volume server and the Rust maintenance worker, cross-compiled for
# amd64 and arm64 without QEMU, turning a 5-hour emulated cargo build into
# ~15 minutes of native compilation.
# The volume server, the Rust maintenance worker and the vector engine,
# cross-compiled for amd64 and arm64 without QEMU, turning a 5-hour emulated
# cargo build into ~15 minutes of native compilation.
build-rust-binaries:
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
strategy:
matrix:
include:
@@ -55,29 +55,17 @@ jobs:
arch: amd64
- target: aarch64-unknown-linux-musl
arch: arm64
cross: true
steps:
- name: Checkout
uses: actions/checkout@v7
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
- name: Set up the musl target
uses: ./.github/actions/rust-musl-target
with:
targets: ${{ matrix.target }}
- name: Install musl tools (amd64)
if: ${{ !matrix.cross }}
run: sudo apt-get install -y musl-tools
- name: Install cross-compilation tools (arm64)
if: matrix.cross
run: |
sudo apt-get install -y gcc-aarch64-linux-gnu
echo "CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_LINKER=aarch64-linux-gnu-gcc" >> "$GITHUB_ENV"
# Disable glibc fortify source — its __memcpy_chk etc. symbols don't exist in musl
echo "CFLAGS_aarch64_unknown_linux_musl=-U_FORTIFY_SOURCE" >> "$GITHUB_ENV"
target: ${{ matrix.target }}
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
@@ -85,7 +73,8 @@ jobs:
~/.cargo/git
seaweed-volume/target
seaweed-worker/target/${{ matrix.target }}/release
key: rust-docker-${{ matrix.target }}-${{ hashFiles('seaweed-volume/Cargo.lock', 'seaweed-worker/Cargo.lock') }}
seaweed-vector/target/${{ matrix.target }}/release
key: rust-docker-${{ matrix.target }}-${{ hashFiles('seaweed-volume/Cargo.lock', 'seaweed-worker/Cargo.lock', 'seaweed-vector/Cargo.lock') }}
restore-keys: |
rust-docker-${{ matrix.target }}-
@@ -126,6 +115,12 @@ jobs:
cargo build --release -p weed-lance-worker --target ${{ matrix.target }}
cp target/${{ matrix.target }}/release/weed-worker ../weed-worker-${{ matrix.arch }}
- name: Build the vector engine
run: |
cd seaweed-vector
cargo build --release -p weed-vector-server --target ${{ matrix.target }}
cp target/${{ matrix.target }}/release/weed-vector ../weed-vector-${{ matrix.arch }}
- name: Upload artifacts
uses: actions/upload-artifact@v7
with:
@@ -134,6 +129,7 @@ jobs:
weed-volume-large-disk-${{ matrix.arch }}
weed-volume-normal-${{ matrix.arch }}
weed-worker-${{ matrix.arch }}
weed-vector-${{ matrix.arch }}
# One job per (variant, platform) on a native runner, pushed by digest;
# the merge job stitches the digests into one multi-arch tag.
@@ -179,6 +175,10 @@ jobs:
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Download pre-built Rust binaries
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
uses: actions/download-artifact@v8
@@ -206,8 +206,17 @@ jobs:
echo "Placed pre-built Rust worker for ${arch}"
fi
done
mkdir -p docker/weed-vector-prebuilt
for arch in amd64 arm64; do
src="./rust-bins/weed-vector-${arch}"
if [ -f "$src" ]; then
cp "$src" "docker/weed-vector-prebuilt/weed-vector-${arch}"
echo "Placed pre-built vector engine for ${arch}"
fi
done
ls -la docker/weed-volume-prebuilt/
ls -la docker/weed-worker-prebuilt/
ls -la docker/weed-vector-prebuilt/
- name: Free Disk Space
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
@@ -218,7 +227,7 @@ jobs:
sudo docker system prune -af --volumes
[ -d ~/.cache/go-build ] && rm -rf ~/.cache/go-build || true
[ -d /go/pkg ] && rm -rf /go/pkg || true
df -h
timeout 15 df -h || true
- name: Docker meta
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
@@ -253,7 +262,7 @@ jobs:
- name: Login to GHCR
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
uses: docker/login-action@v4.6.0
uses: docker/login-action@v4.8.0
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
@@ -270,8 +279,11 @@ jobs:
file: ${{ matrix.dockerfile }}
platforms: ${{ matrix.platform }}
labels: ${{ steps.docker_meta.outputs.labels }}
# Flat single-platform manifest so imagetools create assembles cleanly.
provenance: false
# mode=max keeps resolvedDependencies complete and records the
# builder identity; attestations ride inside each pushed index and
# survive the imagetools merge into the multi-arch manifest.
provenance: mode=max
sbom: true
outputs: type=image,name=${{ env.IMAGE }},push-by-digest=true,name-canonical=true,push=true
cache-from: type=gha,scope=${{ matrix.variant }}-${{ matrix.arch }}
# max only for rocksdb: its RocksDB compile is sha-independent and worth
@@ -281,7 +293,7 @@ jobs:
${{ matrix.build_args }}
BUILDKIT_INLINE_CACHE=1
BRANCH=${{ github.sha }}
${{ matrix.variant == 'rocksdb' && format('ROCKSDB_VERSION={0}', github.event.inputs.rocksdb_version || 'v10.10.1') || '' }}
${{ matrix.variant == 'rocksdb' && format('ROCKSDB_VERSION={0}', github.event.inputs.rocksdb_version || 'v11.1.2') || '' }}
- name: Export digest
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
@@ -307,6 +319,8 @@ jobs:
permissions:
contents: read
id-token: write
attestations: write
packages: write
strategy:
fail-fast: false
matrix:
@@ -338,7 +352,7 @@ jobs:
- name: Login to GHCR
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
uses: docker/login-action@v4.6.0
uses: docker/login-action@v4.8.0
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
@@ -360,7 +374,7 @@ jobs:
- name: Login to Docker Hub
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
uses: docker/login-action@v4.6.0
uses: docker/login-action@v4.8.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
@@ -414,6 +428,45 @@ jobs:
${{ env.IMAGE }}@${{ steps.manifest.outputs.digest }}
chrislusf/seaweedfs@${{ steps.manifest.outputs.digest }}
# Attest the merged multi-arch digest itself, so digest-pinning consumers
# can verify provenance against the exact index they pull. Both steps
# reuse the GHCR login above and push as OCI referrers.
- name: Attest build provenance for ${{ matrix.variant }}
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
uses: actions/attest-build-provenance@v4
with:
subject-name: ${{ env.IMAGE }}
subject-digest: ${{ steps.manifest.outputs.digest }}
push-to-registry: true
# SBOMs are generated and attested per platform manifest: an index-wide
# scan only describes syft's default platform, while the manifests it
# merges differ (some ship the real binary, others placeholders), so the
# SBOM has to be bound to the platform digest a consumer actually pulls.
- name: Install syft
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
uses: anchore/sbom-action/download-syft@v0
- name: Attest per-platform SBOMs for ${{ matrix.variant }}
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
working-directory: /tmp/digests
run: |
set -euo pipefail
for f in *; do
# Each saved digest is an OCI index (image + attestation manifests);
# scan and attest the image manifest a same-platform pull resolves.
ref="${{ env.IMAGE }}@sha256:${f}"
image=$(crane manifest "$ref" | jq -r '
[.manifests[]? | select((.annotations["vnd.docker.reference.type"] // "") != "attestation-manifest"
and (.platform.os // "unknown") != "unknown") | .digest][0] // empty')
ref="${{ env.IMAGE }}@${image:-sha256:${f}}"
syft "registry:${ref}" -o "spdx-json=/tmp/sbom-${f}.spdx.json"
cosign attest --yes \
--predicate "/tmp/sbom-${f}.spdx.json" \
--type spdxjson \
"$ref"
done
# Report-only trivy scan: uploads fixable HIGH/CRITICAL findings to GitHub
# Security for visibility, but never blocks the release. Releases (including
# `latest`) ship regardless — vulnerabilities are tracked, not gated, since
@@ -434,7 +487,7 @@ jobs:
variant: large_disk
steps:
- name: Login to GHCR
uses: docker/login-action@v4.6.0
uses: docker/login-action@v4.8.0
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
@@ -452,7 +505,9 @@ jobs:
scanners: vuln
vuln-type: os,library
severity: HIGH,CRITICAL
ignore-unfixed: true
# Keep unfixed findings in the report so digest-pinning consumers see
# the complete picture, not just what a patch could resolve.
ignore-unfixed: false
limit-severities-for-sarif: true
format: sarif
output: trivy-results.sarif
@@ -460,7 +515,7 @@ jobs:
- name: Upload Trivy scan results to GitHub Security
if: always()
uses: github/codeql-action/upload-sarif@v4.38.2
uses: github/codeql-action/upload-sarif@v4.39.3
with:
sarif_file: trivy-results.sarif
category: trivy-${{ matrix.variant }}
@@ -484,13 +539,13 @@ jobs:
latest_tag: latest_large_disk
steps:
- name: Login to Docker Hub
uses: docker/login-action@v4.6.0
uses: docker/login-action@v4.8.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Login to GHCR
uses: docker/login-action@v4.6.0
uses: docker/login-action@v4.8.0
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
@@ -4,9 +4,9 @@ on:
workflow_dispatch:
inputs:
rocksdb_version:
description: 'RocksDB git tag or branch to build (e.g. v10.10.1)'
description: 'RocksDB git tag or branch to build (e.g. v11.1.2)'
required: true
default: 'v10.10.1'
default: 'v11.1.2'
seaweedfs_ref:
description: 'SeaweedFS git tag, branch, or commit to build'
required: true
@@ -16,6 +16,11 @@ on:
required: false
default: ''
concurrency:
group: ${{ github.workflow }}-${{ github.event.inputs.image_tag || format('rocksdb_{0}_seaweedfs_{1}', github.event.inputs.rocksdb_version, github.event.inputs.seaweedfs_ref) }}
cancel-in-progress: false
queue: max
permissions:
contents: read
@@ -85,13 +90,13 @@ jobs:
echo "seaweedfs_ref=$seaweed" >> "$GITHUB_OUTPUT"
- name: Set up QEMU
uses: docker/setup-qemu-action@99012661954931238ded8c8b007157a8430204e1 # v1
uses: docker/setup-qemu-action@99012661954931238ded8c8b007157a8430204e1 # v4.4.0
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v1
uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0
- name: Login to Docker Hub
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v1
uses: docker/login-action@8aa5831fb8d48d8f76594c7ddde61ae58019f550 # v4.8.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
+1
View File
@@ -27,6 +27,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'telemetry/server/go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Build Telemetry Server
if: github.event_name == 'workflow_dispatch' && inputs.deploy
+4
View File
@@ -1,6 +1,10 @@
name: 'Dependency Review'
on: [pull_request]
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
+8 -3
View File
@@ -19,8 +19,8 @@ on:
- '.github/workflows/e2e.yml'
concurrency:
group: ${{ github.head_ref }}/e2e
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -32,7 +32,7 @@ defaults:
jobs:
e2e:
name: FUSE Mount
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
- name: Check out code into the Go module directory
@@ -42,6 +42,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Configure Docker Hub mirror
run: |
@@ -52,6 +53,7 @@ jobs:
uses: docker/setup-buildx-action@v4
- name: Cache Docker layers
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: /tmp/.buildx-cache
@@ -59,6 +61,9 @@ jobs:
restore-keys: |
${{ runner.os }}-buildx-e2e-
- name: Test APT mirror fallback
run: python3 test_apt_install.py
- name: Install dependencies
run: |
sudo rm -f /etc/apt/sources.list.d/azure-cli.list /etc/apt/sources.list.d/microsoft-prod.list
+6 -1
View File
@@ -18,18 +18,23 @@ on:
- 'go.sum'
- '.github/workflows/ec-integration-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
jobs:
ec-integration-tests:
name: EC Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
- name: Set up Go 1.x
uses: actions/setup-go@v7
with:
cache: ${{ runner.environment == 'github-hosted' }}
go-version: ^1.26
id: go
+4 -3
View File
@@ -19,8 +19,8 @@ on:
- '.github/actions/fix-fusermount-setuid/**'
concurrency:
group: ${{ github.head_ref || github.ref }}/fuse-dlm-integration
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -28,7 +28,7 @@ permissions:
jobs:
fuse-dlm-integration:
name: FUSE DLM Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
@@ -39,6 +39,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Configure FUSE
run: |
+7 -3
View File
@@ -24,8 +24,8 @@ on:
- '.github/actions/fix-fusermount-setuid/**'
concurrency:
group: ${{ github.head_ref || github.ref }}/fuse-failover
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -33,7 +33,7 @@ permissions:
jobs:
fuse-failover:
name: FUSE Volume Server Failover
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 40
steps:
@@ -42,10 +42,14 @@ jobs:
with:
persist-credentials: false
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Configure FUSE
run: |
+4 -3
View File
@@ -17,8 +17,8 @@ on:
- '.github/actions/fix-fusermount-setuid/**'
concurrency:
group: ${{ github.head_ref }}/fuse-integration
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -26,7 +26,7 @@ permissions:
jobs:
fuse-integration:
name: FUSE Integration Testing
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 50
steps:
@@ -37,6 +37,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Configure FUSE
run: |
+7 -3
View File
@@ -25,8 +25,8 @@ on:
- '.github/actions/fix-fusermount-setuid/**'
concurrency:
group: ${{ github.head_ref || github.ref }}/fuse-p2p-integration
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -34,17 +34,21 @@ permissions:
jobs:
fuse-p2p-integration:
name: FUSE P2P Peer Chunk Sharing
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Configure FUSE
run: |
+9 -2
View File
@@ -17,8 +17,8 @@ on:
- '.github/workflows/go.yml'
concurrency:
group: ${{ github.head_ref }}/go
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -35,6 +35,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Get dependencies
run: |
cd weed; go get -v -t -d ./...
@@ -57,6 +58,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Go Vet linux/386 (type-checks code and tests for 32-bit int overflows)
run: |
GOOS=linux GOARCH=386 go vet ./... 2>&1 | grep -v "MessageState contains sync.Mutex" | grep -v "IdentityAccessManagement contains sync.RWMutex" | tee vet-32bit-output.txt
@@ -72,6 +74,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Build
run: cd weed; go build -tags "elastic gocdk sqlite ydb tarantool tikv rclone" -v .
@@ -94,6 +97,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
# Nothing else on a PR compiles these, so per-OS syscall constants creep
# back into shared files unnoticed and only a release build catches it.
- name: Cross-compile
@@ -128,6 +132,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Test
env:
RUN_REDIS_TESTS: "1"
@@ -143,6 +148,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
# 386 test binaries run natively on the amd64 runner. This catches what vet
# can't: unaligned 64-bit atomics and arithmetic that wraps at runtime.
# -short skips the e2e suites already covered on amd64.
@@ -159,6 +165,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
# weed/kms/azure is excluded from every default build, so without this
# nothing compiles it and tag-gated code silently rots.
- name: Build and test with azurekms
+477 -3
View File
@@ -8,6 +8,10 @@ on:
branches: [ master ]
paths: ['k8s/**', '.github/workflows/helm_ci.yml', '.github/scripts/helm_jwt_keys.py']
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -33,13 +37,20 @@ jobs:
with:
version: v3.18.4
- uses: actions/setup-go@v7
with:
go-version-file: go.mod
- name: Test vector chart resources
run: go test ./k8s/charts/seaweedfs/tests -count=1
- uses: actions/setup-python@v7
with:
python-version: '3.10'
check-latest: true
- name: Set up chart-testing
uses: helm/chart-testing-action@v2.8.0
uses: helm/chart-testing-action@v2.9.0
- name: Run chart-testing (list-changed)
id: list-changed
@@ -93,8 +104,86 @@ jobs:
grep -A 4 "^ strategy:" /tmp/s3-rollout.yaml | grep -q "maxUnavailable: 0"
grep -A 4 "^ strategy:" /tmp/s3-rollout.yaml | grep -q "type: RollingUpdate"
grep -A 5 "^ lifecycle:" /tmp/s3-rollout.yaml | grep -q -- "- sleep"
test "$(grep -c -- '-preStopSeconds' /tmp/s3-rollout-defaults.yaml)" -eq 0
echo "S3 rollout settings render correctly"
echo "=== Testing S3 preStopSeconds ==="
if helm template test $CHART_DIR --show-only templates/s3/s3-deployment.yaml \
--set s3.enabled=true --set s3.preStopSeconds=20 > /tmp/s3-prestop-short.yaml 2>&1; then
echo "FAIL: s3.preStopSeconds=20 rendered with the default 10s grace period"; exit 1
fi
grep -q "must be at least 35" /tmp/s3-prestop-short.yaml
helm template test $CHART_DIR --show-only templates/s3/s3-deployment.yaml \
--set s3.enabled=true --set s3.preStopSeconds=20 \
--set s3.terminationGracePeriodSeconds=35 > /tmp/s3-prestop.yaml
grep -q -- "-preStopSeconds=20 " /tmp/s3-prestop.yaml
grep -q "terminationGracePeriodSeconds: 35$" /tmp/s3-prestop.yaml
helm template test $CHART_DIR --show-only templates/filer/filer-statefulset.yaml \
--set filer.s3.enabled=true > /tmp/filer-s3-prestop-defaults.yaml
test "$(grep -c -- '-s3.preStopSeconds' /tmp/filer-s3-prestop-defaults.yaml)" -eq 0
grep -q "terminationGracePeriodSeconds: 60$" /tmp/filer-s3-prestop-defaults.yaml
helm template test $CHART_DIR --show-only templates/filer/filer-statefulset.yaml \
--set filer.s3.enabled=true --set filer.s3.preStopSeconds=20 > /tmp/filer-s3-prestop.yaml
grep -q -- "-s3.preStopSeconds=20 " /tmp/filer-s3-prestop.yaml
grep -q "terminationGracePeriodSeconds: 80$" /tmp/filer-s3-prestop.yaml
helm template test $CHART_DIR --show-only templates/all-in-one/all-in-one-deployment.yaml \
--set allInOne.enabled=true --set allInOne.s3.enabled=true > /tmp/aio-s3-prestop-defaults.yaml
test "$(grep -c -- '-s3.preStopSeconds' /tmp/aio-s3-prestop-defaults.yaml)" -eq 0
grep -q "terminationGracePeriodSeconds: 60$" /tmp/aio-s3-prestop-defaults.yaml
helm template test $CHART_DIR --show-only templates/all-in-one/all-in-one-deployment.yaml \
--set allInOne.enabled=true --set allInOne.s3.enabled=true \
--set s3.preStopSeconds=20 > /tmp/aio-s3-prestop.yaml
grep -q -- "-s3.preStopSeconds=20 " /tmp/aio-s3-prestop.yaml
grep -q "terminationGracePeriodSeconds: 80$" /tmp/aio-s3-prestop.yaml
helm template test $CHART_DIR --show-only templates/all-in-one/all-in-one-deployment.yaml \
--set allInOne.enabled=true --set allInOne.s3.enabled=true \
--set s3.preStopSeconds=20 --set allInOne.s3.preStopSeconds=0 > /tmp/aio-s3-prestop-off.yaml
test "$(grep -c -- '-s3.preStopSeconds' /tmp/aio-s3-prestop-off.yaml)" -eq 0
grep -q "terminationGracePeriodSeconds: 60$" /tmp/aio-s3-prestop-off.yaml
helm template test $CHART_DIR --show-only templates/all-in-one/all-in-one-deployment.yaml \
--set allInOne.enabled=true --set allInOne.s3.enabled=true \
--set s3.preStopSeconds=20 --set allInOne.s3.preStopSeconds=25 > /tmp/aio-s3-prestop-own.yaml
grep -q -- "-s3.preStopSeconds=25 " /tmp/aio-s3-prestop-own.yaml
grep -q "terminationGracePeriodSeconds: 85$" /tmp/aio-s3-prestop-own.yaml
echo "S3 preStopSeconds renders correctly"
echo "=== Testing the S3 config checksum is stable across renders ==="
# s3-secret.yaml keeps its generated keys with lookup. Hashing the rendered
# Secret drew new keys for the checksum whenever lookup found nothing (on
# install), so the first upgrade changed the checksum and restarted the
# filer and S3 pods although nothing had changed.
s3_checksum() { # <template> <helm args...>: the checksum/s3config of one workload
local template=$1; shift
local sum
sum=$(helm template test $CHART_DIR --show-only "$template" "$@" \
| grep 'checksum/s3config:' | awk '{print $2}')
test -n "$sum" || { echo "no checksum/s3config in $template" >&2; return 1; }
echo "$sum"
}
s3_auth_args=(--set s3.enabled=true --set s3.enableAuth=true)
s3_sum=$(s3_checksum templates/s3/s3-deployment.yaml "${s3_auth_args[@]}")
filer_sum=$(s3_checksum templates/filer/filer-statefulset.yaml "${s3_auth_args[@]}")
test "$s3_sum" = "$filer_sum" # one Secret, one checksum
test "$s3_sum" = "$(s3_checksum templates/s3/s3-deployment.yaml "${s3_auth_args[@]}")"
s3_creds_sum=$(s3_checksum templates/s3/s3-deployment.yaml "${s3_auth_args[@]}" \
--set s3.credentials.admin.accessKey=a --set s3.credentials.admin.secretKey=b)
test "$s3_sum" != "$s3_creds_sum"
filer_audit_sum=$(s3_checksum templates/filer/filer-statefulset.yaml "${s3_auth_args[@]}" \
--set-json 'filer.s3.auditLogConfig={"fluent_port":24224}')
test "$filer_sum" != "$filer_audit_sum"
# Settings the Secret does not read must not restart the S3 and filer pods.
test "$filer_sum" = "$(s3_checksum templates/filer/filer-statefulset.yaml "${s3_auth_args[@]}" \
--set s3.replicas=3 --set s3.resources.limits.cpu=1)"
all_in_one_args=(--set allInOne.enabled=true --set allInOne.s3.enabled=true --set allInOne.s3.enableAuth=true)
aio_sum=$(s3_checksum templates/all-in-one/all-in-one-deployment.yaml "${all_in_one_args[@]}")
test "$aio_sum" = "$(s3_checksum templates/all-in-one/all-in-one-deployment.yaml "${all_in_one_args[@]}")"
aio_creds_sum=$(s3_checksum templates/all-in-one/all-in-one-deployment.yaml "${all_in_one_args[@]}" \
--set s3.credentials.admin.accessKey=a --set s3.credentials.admin.secretKey=b)
test "$aio_sum" != "$aio_creds_sum"
echo "S3 config checksum is stable and follows only what the S3 Secret reads"
echo "=== Testing S3 credentials from an existing secret ==="
credential_args=(
--set s3.credentials.admin.existingSecret=minio-root
@@ -170,6 +259,40 @@ jobs:
grep -q "security-config" /tmp/security.yaml
echo "Security configuration renders correctly"
echo "=== Testing volume server HTTPS ==="
awk '/^ *\[https.client\]$/{f=1;next} /^ *\[/{f=0} f' /tmp/security.yaml | grep -q 'enabled = false'
helm template test $CHART_DIR \
--set global.seaweedfs.enableSecurity=true \
--set global.seaweedfs.volumeHttps.enabled=true \
--set volumes.hot.replicas=1 \
> /tmp/volume-https.yaml
awk '/^ *\[https.client\]$/{f=1;next} /^ *\[/{f=0} f' /tmp/volume-https.yaml > /tmp/volume-https-client.txt
grep -q 'enabled = true' /tmp/volume-https-client.txt
grep -q 'cert = "/usr/local/share/ca-certificates/client/tls.crt"' /tmp/volume-https-client.txt
grep -q 'ca = "/usr/local/share/ca-certificates/ca/tls.crt"' /tmp/volume-https-client.txt
awk '/^ *\[https.volume\]$/{f=1;next} /^ *\[/{f=0} f' /tmp/volume-https.yaml \
| grep -q 'cert = "/usr/local/share/ca-certificates/volume/tls.crt"'
grep -q "'\*.test-seaweedfs-volume.default'" /tmp/volume-https.yaml
grep -q "'\*.test-seaweedfs-volume-hot.default'" /tmp/volume-https.yaml
test "$(grep -c 'scheme: HTTPS' /tmp/volume-https.yaml)" -eq 4
test "$(grep -c 'checksum/volume-https:' /tmp/volume-https.yaml)" -eq 4
if grep -q 'checksum/volume-https' /tmp/security.yaml; then
echo "FAIL: the volume HTTPS rollout annotation should only render while the option is on"
exit 1
fi
if helm template test $CHART_DIR --set global.seaweedfs.volumeHttps.enabled=true > /dev/null 2>/tmp/volume-https-nosec.err; then
echo "FAIL: volumeHttps without enableSecurity should fail to render"
exit 1
fi
grep -q "requires global.seaweedfs.enableSecurity" /tmp/volume-https-nosec.err
if helm template test $CHART_DIR --set global.seaweedfs.enableSecurity=true \
--set global.seaweedfs.volumeHttps.enabled=true --set allInOne.enabled=true > /dev/null 2>/tmp/volume-https-aio.err; then
echo "FAIL: volumeHttps with allInOne should fail to render"
exit 1
fi
grep -q "not supported with allInOne" /tmp/volume-https-aio.err
echo "Volume server HTTPS renders [https.volume], [https.client], certificate names, HTTPS probes and rollout annotations"
echo "=== Testing secure bucket-creation hook certificate mounts ==="
helm template test $CHART_DIR \
--show-only templates/shared/post-install-bucket-hook.yaml \
@@ -521,7 +644,7 @@ jobs:
print("cluster alias: hook waits on the renamed names and the chart's addresses")
# Keys kept in a Secret are referenced, not inlined, so the hook
# must render valueFrom for non-string extraEnvironmentVars.
# must render valueFrom for map-valued extraEnvironmentVars.
out = render(dict(jwt, **{
"global.seaweedfs.extraEnvironmentVars.WEED_JWT_FILER_SIGNING_KEY.secretKeyRef.name": "signing-keys",
"global.seaweedfs.extraEnvironmentVars.WEED_JWT_FILER_SIGNING_KEY.secretKeyRef.key": "filerWrite",
@@ -560,6 +683,22 @@ jobs:
--set master.data.storageClass=standard > /tmp/pvc.yaml
grep -q "PersistentVolumeClaim" /tmp/pvc.yaml
echo "PVC configuration renders correctly"
echo "=== Testing existingClaim volumes are mounted ==="
# The filer's data and logs and the master's logs declared an existingClaim
# volume but mounted only hostPath, persistentVolumeClaim and emptyDir, so
# the filer kept its metadata store on the container's own filesystem.
helm template test $CHART_DIR --show-only templates/filer/filer-statefulset.yaml \
--set filer.data.type=existingClaim --set filer.data.claimName=filer-data \
--set filer.logs.type=existingClaim --set filer.logs.claimName=filer-logs > /tmp/filer-existing-claim.yaml
grep -A 1 -- '- name: data-filer$' /tmp/filer-existing-claim.yaml | grep -q 'mountPath: /data$'
grep -A 1 -- '- name: seaweedfs-filer-log-volume$' /tmp/filer-existing-claim.yaml | grep -q 'mountPath: "/logs/"$'
grep -q -- '-logdir=/logs' /tmp/filer-existing-claim.yaml
helm template test $CHART_DIR --show-only templates/master/master-statefulset.yaml \
--set master.logs.type=existingClaim --set master.logs.claimName=master-logs > /tmp/master-existing-claim.yaml
grep -A 1 -- '- name: seaweedfs-master-log-volume$' /tmp/master-existing-claim.yaml | grep -q 'mountPath: "/logs/"$'
grep -q -- '-logdir=/logs' /tmp/master-existing-claim.yaml
echo "existingClaim volumes are mounted"
echo "=== Testing with custom replicas ==="
helm template test $CHART_DIR \
@@ -597,12 +736,47 @@ jobs:
--set filer.secretExtraEnvironmentVars.WEED_MYSQL_PASSWORD.secretKeyRef.name=db \
--set filer.secretExtraEnvironmentVars.WEED_MYSQL_PASSWORD.secretKeyRef.key=password > /tmp/filer-mysql-on-secret.yaml
grep -A 4 -- "- name: WEED_MYSQL_PASSWORD$" /tmp/filer-mysql-on-secret.yaml | grep -q "name: db"
# It replaces the db secret's entry rather than adding a second one, which
# Helm 4's server-side apply rejects as a duplicate.
test "$(grep -c -- "- name: WEED_MYSQL_PASSWORD$" /tmp/filer-mysql-on-secret.yaml)" -eq 1
test "$(grep -c -- "- name: WEED_MYSQL_USERNAME$" /tmp/filer-mysql-on-secret.yaml)" -eq 1
helm template test $CHART_DIR \
--set-string filer.extraEnvironmentVars.WEED_MYSQL_ENABLED=true \
--set-string filer.extraEnvironmentVars.WEED_MYSQL_USERNAME=app > /tmp/filer-mysql-on-plain.yaml
test "$(grep -c -- "- name: WEED_MYSQL_USERNAME$" /tmp/filer-mysql-on-plain.yaml)" -eq 1
grep -A 1 -- "- name: WEED_MYSQL_USERNAME$" /tmp/filer-mysql-on-plain.yaml | grep -q 'value: "app"'
# Credentials render ahead of the merged vars so a $(...) reference resolves.
helm template test $CHART_DIR \
--set-string filer.extraEnvironmentVars.WEED_MYSQL_ENABLED=true \
--set-string filer.extraEnvironmentVars.WEED_MYSQL_USERNAME=app \
--set-string 'filer.extraEnvironmentVars.WEED_MYSQL_DSN=$(WEED_MYSQL_USERNAME)@tcp(h)/db' > /tmp/filer-mysql-dsn.yaml
test "$(grep -n -- "- name: WEED_MYSQL_USERNAME$" /tmp/filer-mysql-dsn.yaml | cut -d: -f1)" -lt "$(grep -n -- "- name: WEED_MYSQL_DSN$" /tmp/filer-mysql-dsn.yaml | cut -d: -f1)"
# Vars a credential's own $(...) references point at render before it.
helm template test $CHART_DIR \
--set-string filer.extraEnvironmentVars.WEED_MYSQL_ENABLED=true \
--set-string filer.extraEnvironmentVars.DB_USER=app \
--set-string 'filer.extraEnvironmentVars.WEED_MYSQL_USERNAME=$(DB_USER)' > /tmp/filer-mysql-ref.yaml
test "$(grep -n -- "- name: DB_USER$" /tmp/filer-mysql-ref.yaml | cut -d: -f1)" -lt "$(grep -n -- "- name: WEED_MYSQL_USERNAME$" /tmp/filer-mysql-ref.yaml | cut -d: -f1)"
helm template test $CHART_DIR \
--set-string filer.extraEnvironmentVars.WEED_MYSQL_ENABLED=true \
--set filer.secretExtraEnvironmentVars.DB_USER.secretKeyRef.name=db \
--set filer.secretExtraEnvironmentVars.DB_USER.secretKeyRef.key=user \
--set-string 'filer.extraEnvironmentVars.WEED_MYSQL_USERNAME=$(DB_USER)' > /tmp/filer-mysql-ref-secret.yaml
test "$(grep -c -- "- name: DB_USER$" /tmp/filer-mysql-ref-secret.yaml)" -eq 1
test "$(grep -n -- "- name: DB_USER$" /tmp/filer-mysql-ref-secret.yaml | cut -d: -f1)" -lt "$(grep -n -- "- name: WEED_MYSQL_USERNAME$" /tmp/filer-mysql-ref-secret.yaml | cut -d: -f1)"
# A flag the chart cannot read counts as selected, not as off.
helm template test $CHART_DIR \
--set filer.secretExtraEnvironmentVars.WEED_MYSQL_ENABLED.secretKeyRef.name=store \
--set filer.secretExtraEnvironmentVars.WEED_MYSQL_ENABLED.secretKeyRef.key=enabled > /tmp/filer-mysql-secret.yaml
grep -q "name: test-seaweedfs-db-secret" /tmp/filer-mysql-secret.yaml
grep -q "name: WEED_MYSQL_HOSTNAME" /tmp/filer-mysql-secret.yaml
# A secret-backed key also replaces the matching plain entry, not adds one.
test "$(grep -c -- "- name: WEED_MYSQL_ENABLED$" /tmp/filer-mysql-secret.yaml)" -eq 1
helm template test $CHART_DIR \
--set-string filer.extraEnvironmentVars.WEED_MYSQL_ENABLED=true \
--set filer.secretExtraEnvironmentVars.WEED_MYSQL_HOSTNAME.secretKeyRef.name=db \
--set filer.secretExtraEnvironmentVars.WEED_MYSQL_HOSTNAME.secretKeyRef.key=host > /tmp/filer-mysql-hostname-secret.yaml
test "$(grep -c -- "- name: WEED_MYSQL_HOSTNAME$" /tmp/filer-mysql-hostname-secret.yaml)" -eq 1
helm template test $CHART_DIR \
--set-string filer.extraEnvironmentVars.WEED_MYSQL2_HOSTNAME=other > /tmp/filer-mysql2.yaml
grep -q "name: WEED_MYSQL2_HOSTNAME" /tmp/filer-mysql2.yaml
@@ -1185,6 +1359,43 @@ jobs:
print("all-in-one env: shared key renders once, component value wins")
PYEOF
echo "=== Testing extraEnvironmentVars: numbers and booleans render as values ==="
# Only a string rendered as value; anything else went to valueFrom, so an
# unquoted 3306 or true in a values file (or --set) became "valueFrom: 3306"
# and the API server rejected the workload. A map still renders valueFrom,
# and a null still renders an empty variable. --set-json decodes numbers as
# float64, as values files do, which quote would print as 1e+06.
helm template test $CHART_DIR \
--set-json global.seaweedfs.extraEnvironmentVars.WEED_GLOBAL_INT=1000000 \
--set master.extraEnvironmentVars.WEED_MASTER_VOLUME_GROWTH_COPY_1=7 \
--set master.extraEnvironmentVars.WEED_EMPTY=null \
--set filer.extraEnvironmentVars.WEED_MYSQL_ENABLED=true \
--set filer.extraEnvironmentVars.WEED_MYSQL_PORT=3306 \
--set filer.extraEnvironmentVars.WEED_REF.secretKeyRef.name=s \
--set filer.extraEnvironmentVars.WEED_REF.secretKeyRef.key=k > /tmp/env-scalars.yaml
python3 - /tmp/env-scalars.yaml <<'PYEOF'
import sys, yaml
docs = [d for d in yaml.safe_load_all(open(sys.argv[1])) if d]
def env(suffix):
sts = next(d for d in docs if d.get("kind") == "StatefulSet"
and d["metadata"]["name"].endswith(suffix))
return {e["name"]: e for e in sts["spec"]["template"]["spec"]["containers"][0]["env"]}
master, filer = env("-master"), env("-filer")
want = [(master, "WEED_MASTER_VOLUME_GROWTH_COPY_1", "7"), (master, "WEED_GLOBAL_INT", "1000000"),
(filer, "WEED_MYSQL_ENABLED", "true"), (filer, "WEED_MYSQL_PORT", "3306"),
(filer, "WEED_GLOBAL_INT", "1000000")]
failed = [f"{name}: {e.get(name)}" for e, name, value in want
if e.get(name) != {"name": name, "value": value}]
if master.get("WEED_EMPTY") != {"name": "WEED_EMPTY"}:
failed.append(f"WEED_EMPTY: {master.get('WEED_EMPTY')}")
if filer.get("WEED_REF", {}).get("valueFrom") != {"secretKeyRef": {"name": "s", "key": "k"}}:
failed.append(f"WEED_REF: {filer.get('WEED_REF')}")
if failed:
print("FAIL: extraEnvironmentVars rendered wrong:", *failed, sep="\n ", file=sys.stderr)
sys.exit(1)
print("extraEnvironmentVars: numbers and booleans render as values, a map as valueFrom, a null as empty")
PYEOF
echo "=== Testing bucket versioning: YAML bool false suspends like string \"false\" ==="
# bool false used to be a silent no-op while string "false" suspended.
BOOL_FALSE=$(helm template test $CHART_DIR \
@@ -1194,6 +1405,67 @@ jobs:
echo "$BOOL_FALSE" | grep -q -- '-status Suspended' || { echo "FAIL: bool false versioning did not Suspend the bucket"; exit 1; }
echo "Bucket versioning: YAML bool false suspends consistently with string \"false\""
echo "=== Testing bucket ttl, replication and disk in one fs.configure ==="
FS_CONFIGURE=$(helm template test $CHART_DIR \
--set s3.enabled=true \
--set s3.createBuckets[0].name=cfgbucket \
--set s3.createBuckets[0].ttl=7d \
--set-string s3.createBuckets[0].replication=001 \
--set s3.createBuckets[0].disk=ssd | grep 'fs.configure -locationPrefix=/buckets/cfgbucket/' || true)
echo "$FS_CONFIGURE" | grep -q -- '-ttl=7d -replication=001 -disk=ssd -apply' || { echo "FAIL: bucket fs.configure missing ttl/replication/disk: $FS_CONFIGURE"; exit 1; }
echo "Bucket fs.configure: ttl, replication and disk render together"
FS_CONFIGURE=$(helm template test $CHART_DIR \
--set s3.enabled=true \
--set s3.createBuckets[0].name=replbucket \
--set-string s3.createBuckets[0].replication=002 | grep 'fs.configure -locationPrefix=/buckets/replbucket/' || true)
echo "$FS_CONFIGURE" | grep -q -- '/buckets/replbucket/ -replication=002 -apply' || { echo "FAIL: replication-only bucket did not render fs.configure: $FS_CONFIGURE"; exit 1; }
echo "Bucket fs.configure: replication without ttl"
FS_CONFIGURE=$(helm template test $CHART_DIR \
--set s3.enabled=true \
--set s3.createBuckets[0].name=diskbucket \
--set s3.createBuckets[0].disk=ssd | grep 'fs.configure -locationPrefix=/buckets/diskbucket/' || true)
echo "$FS_CONFIGURE" | grep -q -- '/buckets/diskbucket/ -disk=ssd -apply' || { echo "FAIL: disk-only bucket did not render fs.configure: $FS_CONFIGURE"; exit 1; }
echo "Bucket fs.configure: disk without ttl"
if helm template test $CHART_DIR \
--set s3.enabled=true \
--set s3.createBuckets[0].name=numbucket \
--set s3.createBuckets[0].replication=0 >/dev/null 2>&1; then
echo "FAIL: a numeric replication should fail to render"; exit 1
fi
echo "Bucket fs.configure: numeric replication is rejected"
if helm template test $CHART_DIR \
--set s3.enabled=true \
--set s3.createBuckets[0].name=badbucket \
--set-string s3.createBuckets[0].replication=00O >/dev/null 2>&1; then
echo "FAIL: a non-digit replication should fail to render"; exit 1
fi
echo "Bucket fs.configure: non-digit replication is rejected"
if helm template test $CHART_DIR \
--set s3.enabled=true \
--set s3.createBuckets[0].name=bigbucket \
--set-string s3.createBuckets[0].replication=256 >/dev/null 2>&1; then
echo "FAIL: a replication above 255 should fail to render"; exit 1
fi
FS_CONFIGURE=$(helm template test $CHART_DIR \
--set s3.enabled=true \
--set s3.createBuckets[0].name=maxbucket \
--set-string s3.createBuckets[0].replication=255 | grep 'fs.configure -locationPrefix=/buckets/maxbucket/' || true)
echo "$FS_CONFIGURE" | grep -q -- '/buckets/maxbucket/ -replication=255 -apply' || { echo "FAIL: replication 255 should render: $FS_CONFIGURE"; exit 1; }
echo "Bucket fs.configure: replication is limited to 000-255"
FS_CONFIGURE=$(helm template test $CHART_DIR \
--set s3.enabled=true \
--set s3.createBuckets[0].name=nullbucket \
--set s3.createBuckets[0].ttl=7d \
--set s3.createBuckets[0].replication=null | grep 'fs.configure -locationPrefix=/buckets/nullbucket/' || true)
echo "$FS_CONFIGURE" | grep -q -- '/buckets/nullbucket/ -ttl=7d -apply' || { echo "FAIL: null replication should be treated as unset: $FS_CONFIGURE"; exit 1; }
echo "Bucket fs.configure: null replication is unset"
echo ""
echo "=== Testing hook Job labels ==="
# The hook Jobs were the only pods in the chart without the standard
@@ -1807,6 +2079,176 @@ jobs:
PYEOF
echo "License + master persistence tests passed"
echo ""
echo "=== Testing commonLabels ==="
python3 - "$CHART_DIR" <<'PYEOF'
import copy, subprocess, sys, yaml
chart = sys.argv[1]
class NoDuplicates(yaml.SafeLoader):
pass
def mapping(loader, node, deep=False):
keys = [loader.construct_object(k, deep=deep) for k, _ in node.value]
dups = {k for k in keys if keys.count(k) > 1}
if dups:
raise ValueError(f"duplicate keys {dups} at line {node.start_mark.line + 1}")
return yaml.SafeLoader.construct_mapping(loader, node, deep)
NoDuplicates.add_constructor(yaml.resolver.BaseResolver.DEFAULT_MAPPING_TAG, mapping)
BASE = [
"--set", "global.seaweedfs.enableSecurity=true",
"--set", "global.seaweedfs.monitoring.enabled=true",
"--set", "global.seaweedfs.monitoring.additionalLabels.release=prometheus",
"--set", "master.data.type=persistentVolumeClaim",
"--set", "master.ingress.enabled=true",
"--set", "master.ingress.labels.team=ingress",
"--set", "filer.data.type=persistentVolumeClaim",
"--set", "filer.ingresses.http.enabled=true",
"--set", "s3.enabled=true", "--set", "s3.enableAuth=true",
"--set", "s3.httpRoute.enabled=true",
"--set", "sftp.enabled=true", "--set", "worker.enabled=true",
"--set", "admin.enabled=true", "--set", "admin.secret.adminPassword=ci",
"--set", "cosi.enabled=true", "--set", "networkPolicy.enabled=true",
"--set", "volumes.dc2.replicas=1",
# render the security ConfigMap without its generated jwt key
"--set", "global.seaweedfs.enableSecurity=true",
"--set", "global.seaweedfs.securityConfig.jwtSigning.volumeWrite=false",
]
LABELS = "/tmp/common-labels-values.yaml"
with open(LABELS, "w") as f:
yaml.safe_dump({"commonLabels": {
"example.com/team": "storage",
"example.com/release": "{{ .Release.Name }}",
"team": "common", # also set by master.ingress.labels
"release": "common", # also set by monitoring.additionalLabels
"app.kubernetes.io/component": "hijack",
"app.kubernetes.io/instance": "hijack",
}}, f)
EXPECTED = {
"example.com/team": "storage",
"example.com/release": "test",
"team": "common",
"release": "common",
}
def render(extra):
out = subprocess.check_output(["helm", "template", "test", chart] + BASE + extra, text=True)
docs = {}
for d in yaml.load_all(out, Loader=NoDuplicates):
if not d:
continue
# generated credentials differ between renders
if d["kind"] == "Secret" and d["metadata"]["name"].endswith(
("-s3-secret", "-sftp-secret", "-sftp-ssh-secret")
):
d.pop("data", None)
d.pop("stringData", None)
docs[(d["kind"], d["metadata"]["name"])] = d
return docs
plain, labelled = render([]), render(["-f", LABELS])
failed = []
if set(plain) != set(labelled):
failed.append(f"commonLabels changed the set of objects: {set(plain) ^ set(labelled)}")
for key, d in labelled.items():
if key not in plain:
continue
before = plain[key]["metadata"].get("labels") or {}
labels = d["metadata"].get("labels") or {}
for k, v in EXPECTED.items():
if k in before:
continue
if labels.get(k) != v:
failed.append(f"{key}: {k}={labels.get(k)!r}, want {v!r}")
# labels the chart or an object-specific value sets are never overridden
for k, v in before.items():
if labels.get(k) != v:
failed.append(f"{key}: {k} changed from {v!r} to {labels.get(k)!r}")
# nothing else changes: selectors, pod templates, volumeClaimTemplates, checksums
stripped = copy.deepcopy(d)
rest = {k: v for k, v in labels.items() if k in before}
if rest or "labels" in plain[key]["metadata"]:
stripped["metadata"]["labels"] = rest
else:
stripped["metadata"].pop("labels", None)
if stripped != plain[key]:
failed.append(f"{key}: commonLabels changed more than metadata.labels")
if failed:
print("\nFAIL:", file=sys.stderr)
for f in failed:
print(f" - {f}", file=sys.stderr)
sys.exit(1)
print(f"{len(labelled)} objects carry commonLabels and are otherwise unchanged")
PYEOF
# an empty value renders exactly like the default
helm template test $CHART_DIR --set-string 'commonLabels=' > /tmp/common-labels-empty.yaml
cmp /tmp/default.yaml /tmp/common-labels-empty.yaml
# the whole value may be a template rendering to a map, e.g. a parent chart's labels
echo "commonLabels: 'example.com/owner: {{ .Release.Name }}'" > /tmp/common-labels-string-values.yaml
helm template test $CHART_DIR --show-only templates/master/master-service.yaml \
-f /tmp/common-labels-string-values.yaml > /tmp/common-labels-string.yaml
grep -q "^ example.com/owner: test$" /tmp/common-labels-string.yaml
if helm template test $CHART_DIR --set-json 'commonLabels=["a"]' > /dev/null 2>&1; then
echo "a commonLabels that is not a map must fail the render"; exit 1
fi
echo "commonLabels tests passed"
echo ""
echo "=== Testing namespaceOverride ==="
NS_ARGS=(
--set s3.enabled=true --set s3.enableAuth=true
--set 's3.createBuckets[0].name=data'
--set sftp.enabled=true --set sftp.enableAuth=true
--set admin.enabled=true --set admin.secret.adminPassword=ci
--set worker.enabled=true
--set cosi.enabled=true --set cosi.bucketClassName=seaweedfs
--set global.seaweedfs.enableSecurity=true
--set global.seaweedfs.monitoring.enabled=true
--set networkPolicy.enabled=true
--set volumes.extra.replicas=1
)
helm template test $CHART_DIR --namespace release-ns "${NS_ARGS[@]}" \
--set namespaceOverride=other-ns > /tmp/ns-override.yaml
helm template test $CHART_DIR --namespace release-ns \
--set allInOne.enabled=true --set allInOne.s3.enabled=true \
--set master.enabled=false --set volume.enabled=false --set filer.enabled=false \
--set 'allInOne.s3.createBuckets[0].name=data' \
--set global.seaweedfs.enableSecurity=true \
--set namespaceOverride=other-ns > /tmp/ns-override-aio.yaml
grep -q 'kind: Job' /tmp/ns-override-aio.yaml
python3 - /tmp/ns-override.yaml /tmp/ns-override-aio.yaml <<'PYEOF'
import sys, yaml
CLUSTER_SCOPED = {"ClusterRole", "ClusterRoleBinding", "BucketClass", "BucketAccessClass"}
failed = []
for path in sys.argv[1:]:
text = open(path).read()
if "release-ns" in text:
failed.append(f"{path}: release namespace still rendered")
for d in yaml.safe_load_all(text):
if not d:
continue
kind, md = d["kind"], d["metadata"]
if kind in CLUSTER_SCOPED:
for s in d.get("subjects") or []:
if s.get("namespace") != "other-ns":
failed.append(f"{path}: {kind}/{md['name']} subject in {s.get('namespace')!r}")
elif md.get("namespace") != "other-ns":
failed.append(f"{path}: {kind}/{md['name']} in {md.get('namespace')!r}")
if failed:
print("\n".join(failed), file=sys.stderr)
sys.exit(1)
PYEOF
grep -q -- '-filer=test-seaweedfs-filer-client.other-ns:' /tmp/ns-override.yaml
grep -q 'test-seaweedfs-master-0.test-seaweedfs-master.other-ns:' /tmp/ns-override.yaml
grep -qF "'*.other-ns.svc.cluster.local'" /tmp/ns-override.yaml
# Setting it to the release namespace changes nothing.
helm template test $CHART_DIR --namespace release-ns > /tmp/ns-default.yaml
helm template test $CHART_DIR --namespace release-ns \
--set namespaceOverride=release-ns > /tmp/ns-same.yaml
diff /tmp/ns-default.yaml /tmp/ns-same.yaml
echo "namespaceOverride moves every namespaced object and address"
echo "All template rendering tests passed!"
- name: Resolve an image tag that is published
@@ -1834,7 +2276,7 @@ jobs:
echo "IMAGE_TAG=$TAG" >> $GITHUB_ENV
- name: Create kind cluster
uses: helm/kind-action@v1.15.0
uses: helm/kind-action@v1.15.2
- name: Verify volume resize hook security contexts
run: |
@@ -1958,6 +2400,38 @@ jobs:
kubectl delete namespace "$NS"
echo "SFTP host key lifecycle tests passed"
- name: Verify the filer db secret survives uninstall and install
run: |
set -e
CHART_DIR="k8s/charts/seaweedfs"
NS="filer-db-secret"
SECRET="db-seaweedfs-db-secret"
DB_ARGS="--set image.tag=$IMAGE_TAG --set-string filer.extraEnvironmentVars.WEED_MYSQL_ENABLED=true --set master.enabled=false --set volume.enabled=false"
kubectl create namespace "$NS"
echo "=== install generates the db credentials ==="
helm install db $CHART_DIR -n "$NS" $DB_ARGS
[ "$(kubectl get secret "$SECRET" -n "$NS" -o jsonpath='{.data.user}' | base64 -d)" = YourSWUser ] \
|| { echo "FAIL: install did not create the default user"; exit 1; }
[ "$(kubectl get secret "$SECRET" -n "$NS" -o jsonpath='{.data.password}' | base64 -d | wc -c)" -eq 32 ] \
|| { echo "FAIL: install did not generate a 32-character password"; exit 1; }
echo "=== credentials in the kept Secret survive uninstall and install ==="
PASSWORD='p"a\ss: #x {{ .Values }}'
kubectl patch secret "$SECRET" -n "$NS" --type merge \
-p "$(jq -n --arg p "$PASSWORD" '{stringData: {user: "seaweed_app", password: $p}}')"
helm uninstall db -n "$NS"
kubectl get secret "$SECRET" -n "$NS" >/dev/null || { echo "FAIL: uninstall removed the db secret"; exit 1; }
helm install db $CHART_DIR -n "$NS" $DB_ARGS
[ "$(kubectl get secret "$SECRET" -n "$NS" -o jsonpath='{.data.user}' | base64 -d)" = seaweed_app ] \
|| { echo "FAIL: reinstall reset the db user"; exit 1; }
[ "$(kubectl get secret "$SECRET" -n "$NS" -o jsonpath='{.data.password}' | base64 -d)" = "$PASSWORD" ] \
|| { echo "FAIL: reinstall replaced the db password"; exit 1; }
helm uninstall db -n "$NS"
kubectl delete namespace "$NS"
echo "Filer db secret reinstall test passed"
- name: Verify JWT signing key persistence across upgrades
run: |
# chart-testing puts its pip-less venv first on PATH; use setup-python.
+7 -1
View File
@@ -14,6 +14,10 @@ on:
- 'weed/**'
- '.github/workflows/java_integration_tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
jobs:
test:
name: Java Integration Tests
@@ -31,6 +35,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Set up Java
@@ -38,7 +43,7 @@ jobs:
with:
java-version: ${{ matrix.java }}
distribution: 'temurin'
cache: 'maven'
cache: ${{ runner.environment == 'github-hosted' && 'maven' || '' }}
- name: Build SeaweedFS
run: |
@@ -55,6 +60,7 @@ jobs:
# Start SeaweedFS with optimized settings for CI
# Include S3 API for s3copier integration tests
weed server -dir="$WEED_DATA_DIR" \
-ip=127.0.0.1 -ip.bind=127.0.0.1 \
-master.raftHashicorp \
-master.electionTimeout=1s \
-master.volumeSizeLimitMB=100 \
+1 -1
View File
@@ -77,7 +77,7 @@ jobs:
with:
java-version: '17'
distribution: 'temurin'
cache: 'maven'
cache: ${{ runner.environment == 'github-hosted' && 'maven' || '' }}
server-id: central
server-username: MAVEN_CENTRAL_USERNAME
server-password: MAVEN_CENTRAL_PASSWORD
+5 -1
View File
@@ -12,6 +12,10 @@ on:
- 'other/java/**'
- '.github/workflows/java_unit_tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
jobs:
test:
name: Java Unit Tests
@@ -30,7 +34,7 @@ jobs:
with:
java-version: ${{ matrix.java }}
distribution: 'temurin'
cache: 'maven'
cache: ${{ runner.environment == 'github-hosted' && 'maven' || '' }}
- name: Build and Install SeaweedFS Client
working-directory: other/java/client
+3 -3
View File
@@ -24,8 +24,8 @@ on:
workflow_dispatch: # Allow manual trigger
concurrency:
group: ${{ github.head_ref }}/kafka-quicktest
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -43,7 +43,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version: ^1.26
cache: true
cache: ${{ runner.environment == 'github-hosted' }}
cache-dependency-path: |
**/go.sum
id: go
+9 -6
View File
@@ -23,8 +23,8 @@ on:
- '.github/workflows/kafka-tests.yml'
concurrency:
group: ${{ github.head_ref }}/kafka-tests
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
# Force different runners for better isolation
env:
@@ -53,6 +53,7 @@ jobs:
- name: Set up Go 1.x
uses: actions/setup-go@v7
with:
cache: ${{ runner.environment == 'github-hosted' }}
go-version: ^1.26
id: go
@@ -98,6 +99,7 @@ jobs:
- name: Set up Go 1.x
uses: actions/setup-go@v7
with:
cache: ${{ runner.environment == 'github-hosted' }}
go-version: ^1.26
id: go
@@ -149,7 +151,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version: ^1.26
cache: true
cache: ${{ runner.environment == 'github-hosted' }}
cache-dependency-path: |
**/go.sum
id: go
@@ -328,7 +330,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version: ^1.26
cache: true
cache: ${{ runner.environment == 'github-hosted' }}
cache-dependency-path: |
**/go.sum
id: go
@@ -490,7 +492,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version: ^1.26
cache: true
cache: ${{ runner.environment == 'github-hosted' }}
cache-dependency-path: |
**/go.sum
id: go
@@ -648,7 +650,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version: ^1.26
cache: true
cache: ${{ runner.environment == 'github-hosted' }}
cache-dependency-path: |
**/go.sum
id: go
@@ -805,6 +807,7 @@ jobs:
- name: Set up Go 1.x
uses: actions/setup-go@v7
with:
cache: ${{ runner.environment == 'github-hosted' }}
go-version: ^1.26
id: go
+6 -4
View File
@@ -19,8 +19,8 @@ on:
- 'test/kms/**'
concurrency:
group: ${{ github.head_ref || github.ref }}-kms-tests
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -32,7 +32,7 @@ defaults:
jobs:
kms-provider-tests:
name: KMS Provider Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
@@ -43,6 +43,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -77,7 +78,7 @@ jobs:
s3-kms-e2e-tests:
name: S3 KMS End-to-End Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 25
steps:
@@ -88,6 +89,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -21,8 +21,8 @@ on:
- '.github/workflows/master-cold-start-tests.yml'
concurrency:
group: ${{ github.head_ref || github.ref }}/master-cold-start-tests
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -30,7 +30,7 @@ permissions:
jobs:
master-cold-start-tests:
name: Master Cold Start Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 10
steps:
- name: Check out code
@@ -42,6 +42,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Build weed binary
run: |
@@ -25,8 +25,8 @@ on:
- '.github/workflows/metadata-subscribe-tests.yml'
concurrency:
group: ${{ github.head_ref }}/metadata-subscribe-tests
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -37,7 +37,7 @@ env:
jobs:
metadata-subscribe-integration:
name: Metadata Subscribe Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
@@ -48,6 +48,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Build SeaweedFS
run: |
+4 -2
View File
@@ -10,8 +10,8 @@ on:
branches: [ 'winfsp-bench**' ]
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -32,6 +32,7 @@ jobs:
- uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Install WinFsp and rclone
run: choco install winfsp rclone -y --no-progress
@@ -150,6 +151,7 @@ jobs:
- uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Repair the fusermount3 setuid bit
uses: ./.github/actions/fix-fusermount-setuid
@@ -24,8 +24,8 @@ on:
- '.github/workflows/mount-windows-conformance.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -49,6 +49,7 @@ jobs:
- uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
# cgofuse loads winfsp-x64.dll at run time, so WinFsp is needed here but
# not to build.
+3 -2
View File
@@ -24,8 +24,8 @@ on:
- '.github/workflows/mount-windows.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -49,6 +49,7 @@ jobs:
- uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
# cgofuse loads winfsp-x64.dll at run time, so WinFsp is needed here but
# not to build.
+4 -3
View File
@@ -21,8 +21,8 @@ on:
- '.github/workflows/multi-master-tests.yml'
concurrency:
group: ${{ github.head_ref || github.ref }}/multi-master-tests
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -30,7 +30,7 @@ permissions:
jobs:
multi-master-failover-tests:
name: Multi-Master Failover Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 10
steps:
- name: Check out code
@@ -40,6 +40,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Install SeaweedFS
run: |
+7 -4
View File
@@ -45,8 +45,8 @@ on:
type: string
concurrency:
group: ${{ github.head_ref || github.ref }}/performance
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -67,6 +67,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Build weed
run: go build -o weed_bin ./weed
@@ -175,13 +176,14 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Install Rust toolchain
if: matrix.impl == 'rust'
uses: dtolnay/rust-toolchain@stable
- name: Cache cargo registry and target
if: matrix.impl == 'rust'
if: matrix.impl == 'rust' && runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
@@ -327,13 +329,14 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Install Rust toolchain
if: matrix.impl == 'rust'
uses: dtolnay/rust-toolchain@stable
- name: Cache cargo registry and target
if: matrix.impl == 'rust'
if: matrix.impl == 'rust' && runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
+10 -2
View File
@@ -27,7 +27,7 @@ permissions:
jobs:
pjdfstest:
name: pjdfstest
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 60
steps:
@@ -38,15 +38,23 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Start local Docker registry
run: docker run -d --restart=always -p 5000:5000 --name registry registry:2
run: |
if ! docker start registry 2>/dev/null; then
docker rm -f registry >/dev/null 2>&1 || true
docker run -d --restart=always -p 5000:5000 --name registry registry:2
fi
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4
with:
driver-opts: network=host
- name: Set up Docker Compose
uses: docker/setup-compose-action@v2
- name: Build weed race binary
run: |
cd docker
+6 -1
View File
@@ -18,13 +18,17 @@ on:
- 'go.sum'
- '.github/workflows/plugin-workers.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
jobs:
plugin-worker:
name: "Plugin Worker: ${{ matrix.worker }}"
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 10
strategy:
fail-fast: false
@@ -41,6 +45,7 @@ jobs:
- name: Set up Go 1.x
uses: actions/setup-go@v7
with:
cache: ${{ runner.environment == 'github-hosted' }}
go-version: ^1.26
id: go
+791
View File
@@ -0,0 +1,791 @@
name: POSIX Compliance Tests
on:
push:
branches: [ main, master, develop ]
paths:
- 'weed/mount/**'
- 'weed/command/mount*'
- 'weed/command/fuse*'
- 'test/fuse_integration/**'
pull_request:
branches: [ main, master, develop ]
paths:
- 'weed/mount/**'
- 'weed/command/mount*'
- 'weed/command/fuse*'
- 'test/fuse_integration/**'
workflow_dispatch:
inputs:
test_type:
description: 'Type of POSIX tests to run (default: full for comprehensive testing)'
required: true
default: 'full'
type: choice
options:
- critical
- basic
- extended
- full
enable_external_tests:
description: 'Run external test suites (slower)'
required: false
default: false
type: boolean
permissions:
contents: read
pull-requests: write
issues: write
env:
GO_VERSION: '1.26'
TIMEOUT: '45m'
jobs:
posix-compliance-ubuntu:
runs-on: ubuntu-latest
timeout-minutes: 60
strategy:
matrix:
fuse-version: ['2.9', '3.0']
fail-fast: false
steps:
- name: Checkout code
uses: actions/checkout@v7
with:
fetch-depth: 1
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go ${{ env.GO_VERSION }}
uses: actions/setup-go@v7
with:
go-version: ${{ env.GO_VERSION }}
cache: ${{ runner.environment == 'github-hosted' }}
cache-dependency-path: |
go.sum
test/fuse_integration/go.sum
- name: Install system dependencies
run: |
sudo apt-get update
sudo apt-get install -y \
attr \
acl \
build-essential \
git \
python3-pip
# Install FUSE version specific packages
if [ "${{ matrix.fuse-version }}" = "2.9" ]; then
echo "Installing FUSE 2.9..."
sudo apt-get install -y fuse libfuse-dev
elif [ "${{ matrix.fuse-version }}" = "3.0" ]; then
echo "Installing FUSE 3.0..."
sudo apt-get install -y fuse3 libfuse3-dev
# Also install fuse2 for compatibility if needed
sudo apt-get install -y fuse libfuse-dev
fi
- name: Set up user permissions for FUSE
run: |
# Create fuse group if it doesn't exist
sudo groupadd -f fuse
# Add user to fuse group
sudo usermod -a -G fuse $USER
# Set permissions on FUSE device
sudo chmod 666 /dev/fuse
# Allow non-root mounts to pass -o allow_other (weed -allowOthers)
echo 'user_allow_other' | sudo tee -a /etc/fuse.conf
sudo chmod 644 /etc/fuse.conf
# Ensure fuse module is loaded
sudo modprobe fuse || true
# Verify setup
echo "FUSE setup verification:"
ls -la /dev/fuse
groups $USER | grep fuse || echo "User not in fuse group yet (will be effective after login)"
lsmod | grep fuse || echo "FUSE module not loaded"
- name: Install external test tools
if: ${{ github.event.inputs.enable_external_tests == 'true' }}
run: |
# Install nfstest for POSIX API verification
pip3 install --user nfstest
# Install FIO for performance testing
sudo apt-get install -y fio
# Install additional test utilities
sudo apt-get install -y stress-ng
- name: Build SeaweedFS
run: |
make
# Verify binary exists and is executable (it's installed to GOPATH/bin)
weed version
# Make weed binary available in PATH (it's already there from go install)
sudo cp $(which weed) /usr/local/bin/weed
which weed
weed version
- name: Set up SeaweedFS cluster
run: |
start_and_wait() {
local name=$1
local url=$2
local pidfile=$3
shift 3
local cmd="$@"
echo "Starting $name..."
$cmd > "/tmp/seaweedfs/$name.log" 2>&1 &
local pid=$!
echo $pid > "$pidfile"
echo "Waiting for $name to start..."
for i in {1..30}; do
if curl -sf "$url" > /dev/null 2>&1; then
echo "$name is ready"
return 0
fi
if [ $i -eq 30 ]; then
echo "$name failed to start"
cat "/tmp/seaweedfs/$name.log"
exit 1
fi
sleep 2
done
}
mkdir -p /tmp/seaweedfs/{master,volume,filer,mount}
start_and_wait "master" "http://127.0.0.1:9333/cluster/status" "/tmp/seaweedfs/master.pid" \
"weed master -ip=127.0.0.1 -port=9333 -mdir=/tmp/seaweedfs/master -raftBootstrap=true"
start_and_wait "volume" "http://127.0.0.1:8080/status" "/tmp/seaweedfs/volume.pid" \
"weed volume -mserver=127.0.0.1:9333 -ip=127.0.0.1 -port=8080 -dir=/tmp/seaweedfs/volume -max=100"
start_and_wait "filer" "http://127.0.0.1:8888/" "/tmp/seaweedfs/filer.pid" \
"weed filer -master=127.0.0.1:9333 -ip=127.0.0.1 -port=8888"
echo "SeaweedFS cluster status:"
curl -s http://127.0.0.1:9333/cluster/status || true
- name: Set up FUSE mount
run: |
# Create mount point
MOUNT_POINT="/tmp/seaweedfs/mount"
mkdir -p $MOUNT_POINT
echo "Mounting SeaweedFS FUSE filesystem..."
# Mount SeaweedFS FUSE filesystem in background
weed mount \
-filer=127.0.0.1:8888 \
-dir=$MOUNT_POINT \
-filer.path=/ \
-dirAutoCreate=true \
-allowOthers=true \
-nonempty=true \
> /tmp/seaweedfs/mount.log 2>&1 &
MOUNT_PID=$!
echo $MOUNT_PID > /tmp/seaweedfs/mount.pid
# Wait for mount to be ready. The mount point directory already
# exists, so only a real mount counts — a directory listing would
# pass on the local disk even if weed mount died.
echo "Waiting for FUSE mount to be ready..."
for i in {1..30}; do
if mountpoint -q $MOUNT_POINT 2>/dev/null; then
echo "FUSE mount is ready"
break
fi
if ! kill -0 $MOUNT_PID 2>/dev/null; then
echo "FUSE mount process exited"
cat /tmp/seaweedfs/mount.log
exit 1
fi
if [ $i -eq 30 ]; then
echo "FUSE mount failed to be ready"
cat /tmp/seaweedfs/mount.log
echo "Mount status:"
mount | grep fuse || echo "No FUSE mounts found"
ps aux | grep weed || echo "No weed processes found"
exit 1
fi
sleep 2
done
# Verify mount is working
echo "Testing FUSE mount functionality..."
echo "test" > $MOUNT_POINT/test_file.txt
if [ "$(cat $MOUNT_POINT/test_file.txt)" = "test" ]; then
echo "FUSE mount is working correctly"
rm $MOUNT_POINT/test_file.txt
else
echo "FUSE mount is not working properly"
exit 1
fi
# Export mount point for tests
echo "SEAWEEDFS_MOUNT_POINT=$MOUNT_POINT" >> $GITHUB_ENV
- name: Set up test environment
run: |
cd test/fuse_integration
# Initialize Go module for tests
if [ ! -f go.mod ]; then
go mod init seaweedfs-posix-tests
go mod tidy
fi
# Create reports directory
mkdir -p reports
# Set up external tools if requested
# Note: External tools setup is now optional and won't fail the build
if [ "${{ github.event.inputs.enable_external_tests }}" = "true" ]; then
echo "Setting up external POSIX testing tools..."
make -f posix_Makefile setup-external-tools || true
# setup-external-tools exits 0 even when a sub-target failed, so
# require the tools only when they are actually installed
if [ -d external_tools/pjdfstest ] && command -v nfstest_posix >/dev/null 2>&1; then
export POSIX_REQUIRE_EXTERNAL_TOOLS="true"
echo "POSIX_REQUIRE_EXTERNAL_TOOLS=true" >> $GITHUB_ENV
echo "✅ External tools setup completed successfully"
else
echo "⚠️ External tools setup incomplete - continuing with basic tests only"
echo "External tools will be skipped but tests will continue"
echo "This is expected behavior when build tools are not available"
fi
fi
# Verify SeaweedFS cluster is accessible
echo "Verifying SeaweedFS cluster accessibility..."
curl -sf http://127.0.0.1:9333/cluster/status
curl -sf http://127.0.0.1:8080/status
curl -sf http://127.0.0.1:8888/
# Verify mount point
echo "Mount point: $SEAWEEDFS_MOUNT_POINT"
ls -la $SEAWEEDFS_MOUNT_POINT
# Verify mount is actually working by testing file operations
echo "Verifying SeaweedFS mount functionality..."
TEST_FILE="$SEAWEEDFS_MOUNT_POINT/.mount_verification_test"
echo "SeaweedFS mount test" > "$TEST_FILE"
if [ "$(cat "$TEST_FILE")" = "SeaweedFS mount test" ]; then
echo "✅ Mount verification successful"
rm -f "$TEST_FILE"
else
echo "❌ Mount verification failed - mount may not be working properly"
exit 1
fi
# Check if this is actually a FUSE mount — the plain directory created
# above passes the file test, so without this the suite can run
# against the local disk and report a pass it never earned
if mount | grep -q "$SEAWEEDFS_MOUNT_POINT"; then
echo "✅ FUSE mount detected:"
mount | grep "$SEAWEEDFS_MOUNT_POINT"
else
echo "❌ Mount point not found in mount table - refusing to test a local directory"
exit 1
fi
- name: Run POSIX compliance tests
id: posix-tests
env:
SEAWEEDFS_MOUNT_POINT: ${{ env.SEAWEEDFS_MOUNT_POINT }}
POSIX_REQUIRE_EXTERNAL_TOOLS: ${{ env.POSIX_REQUIRE_EXTERNAL_TOOLS }}
run: |
cd test/fuse_integration
# Determine which tests to run
TEST_TYPE="${{ github.event.inputs.test_type }}"
if [ -z "$TEST_TYPE" ]; then
TEST_TYPE="full"
fi
echo "Running POSIX tests: $TEST_TYPE"
if [ "$TEST_TYPE" = "full" ]; then
echo "🚀 Running COMPREHENSIVE POSIX test suite (including external test suites)"
else
echo "Using mount point: $SEAWEEDFS_MOUNT_POINT"
fi
# Set test configuration to use our mounted filesystem
export TEST_MOUNT_POINT="$SEAWEEDFS_MOUNT_POINT"
export TEST_SKIP_CLUSTER_SETUP="true"
# Debug: Show environment variables
echo "🔍 Test Environment:"
echo " TEST_MOUNT_POINT=$TEST_MOUNT_POINT"
echo " TEST_SKIP_CLUSTER_SETUP=$TEST_SKIP_CLUSTER_SETUP"
echo " SEAWEEDFS_MOUNT_POINT=$SEAWEEDFS_MOUNT_POINT"
case "$TEST_TYPE" in
"critical")
echo "Running critical POSIX tests (basic compliance only)..."
make -f posix_Makefile test-posix-critical
;;
"basic")
echo "Running basic POSIX tests (core file operations)..."
make -f posix_Makefile test-posix-basic
;;
"extended")
echo "Running extended POSIX tests (including advanced features)..."
make -f posix_Makefile test-posix-extended
;;
"full")
echo "🚀 Running COMPREHENSIVE POSIX test suite (all tests including external)..."
echo " This includes: basic, extended, and external test suites"
echo " External suites: pjdfstest, nfstest_posix, and custom stress tests"
make -f posix_Makefile test-posix-full
;;
*)
echo "Unknown test type: $TEST_TYPE, defaulting to full suite"
echo "🚀 Running full POSIX test suite (all tests including external)..."
make -f posix_Makefile test-posix-full
;;
esac
- name: Run external test suites
if: ${{ github.event.inputs.enable_external_tests == 'true' }}
continue-on-error: true
run: |
cd test/fuse_integration
# Point the external suites at the same mount as the Go tests;
# without these the Makefile falls back to a local-disk path.
export TEST_MOUNT_POINT="$SEAWEEDFS_MOUNT_POINT"
export TEST_SKIP_CLUSTER_SETUP="true"
# Run external tests (may fail on some systems)
make -f posix_Makefile test-nfstest-posix || echo "nfstest failed or not available"
make -f posix_Makefile test-fio-posix || echo "FIO tests failed or not available"
- name: Generate compliance report
if: always()
run: |
cd test/fuse_integration
make -f posix_Makefile generate-report
- name: Collect system information for debugging
if: failure()
run: |
cd test/fuse_integration
mkdir -p reports/debug
echo "System Information" > reports/debug/system_info.txt
uname -a >> reports/debug/system_info.txt
echo "" >> reports/debug/system_info.txt
echo "Mount Information" >> reports/debug/system_info.txt
mount >> reports/debug/system_info.txt
echo "" >> reports/debug/system_info.txt
echo "Disk Space" >> reports/debug/system_info.txt
timeout 15 df -h >> reports/debug/system_info.txt || true
echo "" >> reports/debug/system_info.txt
echo "Memory Information" >> reports/debug/system_info.txt
free -h >> reports/debug/system_info.txt
echo "" >> reports/debug/system_info.txt
echo "FUSE Information" >> reports/debug/system_info.txt
ls -la /dev/fuse >> reports/debug/system_info.txt
lsmod | grep fuse >> reports/debug/system_info.txt
echo "" >> reports/debug/system_info.txt
echo "SeaweedFS Version" >> reports/debug/system_info.txt
weed version >> reports/debug/system_info.txt
echo "" >> reports/debug/system_info.txt
echo "Go Version" >> reports/debug/system_info.txt
go version >> reports/debug/system_info.txt
echo "" >> reports/debug/system_info.txt
echo "Running Processes" >> reports/debug/system_info.txt
ps aux | grep -E "(weed|fuse)" >> reports/debug/system_info.txt
echo "" >> reports/debug/system_info.txt
# Collect SeaweedFS service logs
echo "=== SeaweedFS Service Logs ===" >> reports/debug/system_info.txt
if [ -f /tmp/seaweedfs/master.log ]; then
echo "Master Log:" >> reports/debug/system_info.txt
tail -50 /tmp/seaweedfs/master.log >> reports/debug/system_info.txt
echo "" >> reports/debug/system_info.txt
fi
if [ -f /tmp/seaweedfs/volume.log ]; then
echo "Volume Log:" >> reports/debug/system_info.txt
tail -50 /tmp/seaweedfs/volume.log >> reports/debug/system_info.txt
echo "" >> reports/debug/system_info.txt
fi
if [ -f /tmp/seaweedfs/filer.log ]; then
echo "Filer Log:" >> reports/debug/system_info.txt
tail -50 /tmp/seaweedfs/filer.log >> reports/debug/system_info.txt
echo "" >> reports/debug/system_info.txt
fi
if [ -f /tmp/seaweedfs/mount.log ]; then
echo "Mount Log:" >> reports/debug/system_info.txt
tail -50 /tmp/seaweedfs/mount.log >> reports/debug/system_info.txt
echo "" >> reports/debug/system_info.txt
fi
# Copy full logs as separate files
if [ -f /tmp/seaweedfs/master.log ]; then
cp /tmp/seaweedfs/master.log reports/debug/master.log
fi
if [ -f /tmp/seaweedfs/volume.log ]; then
cp /tmp/seaweedfs/volume.log reports/debug/volume.log
fi
if [ -f /tmp/seaweedfs/filer.log ]; then
cp /tmp/seaweedfs/filer.log reports/debug/filer.log
fi
if [ -f /tmp/seaweedfs/mount.log ]; then
cp /tmp/seaweedfs/mount.log reports/debug/mount.log
fi
- name: Cleanup SeaweedFS cluster and FUSE mount
if: always()
run: |
stop_process() {
local name=$1
local pidfile=$2
if [ -f "$pidfile" ]; then
local pid=$(cat "$pidfile")
if kill -0 $pid 2>/dev/null; then
echo "Stopping $name process (PID: $pid)..."
kill -TERM $pid || true
sleep 2
kill -KILL $pid 2>/dev/null || true
fi
fi
}
echo "Cleaning up SeaweedFS cluster and FUSE mount..."
MOUNT_POINT="/tmp/seaweedfs/mount"
if mountpoint -q $MOUNT_POINT 2>/dev/null; then
echo "Unmounting FUSE filesystem..."
fusermount -u $MOUNT_POINT || umount $MOUNT_POINT || true
fi
stop_process "mount" "/tmp/seaweedfs/mount.pid"
stop_process "filer" "/tmp/seaweedfs/filer.pid"
stop_process "volume" "/tmp/seaweedfs/volume.pid"
stop_process "master" "/tmp/seaweedfs/master.pid"
pkill -f "weed " || true
fusermount -u $MOUNT_POINT 2>/dev/null || true
umount $MOUNT_POINT 2>/dev/null || true
rm -rf /tmp/seaweedfs || true
echo "Cleanup completed"
- name: Upload test results
uses: actions/upload-artifact@v7
if: always()
with:
name: posix-test-results-ubuntu-fuse${{ matrix.fuse-version }}
path: |
test/fuse_integration/reports/
test/fuse_integration/*.log
retention-days: 30
- name: Upload test coverage
uses: actions/upload-artifact@v7
if: always() && (github.event.inputs.test_type == 'full' || github.event.inputs.test_type == 'extended')
with:
name: posix-coverage-ubuntu
path: test/fuse_integration/reports/posix_coverage.html
retention-days: 14
- name: Comment PR with results
if: github.event_name == 'pull_request' && always()
uses: actions/github-script@v9
with:
script: |
const fs = require('fs');
const path = require('path');
const reportPath = 'test/fuse_integration/reports/posix_compliance_summary.txt';
if (fs.existsSync(reportPath)) {
const report = fs.readFileSync(reportPath, 'utf8');
const comment = `## POSIX Compliance Test Results
**Go Version:** ${{ env.GO_VERSION }}
**FUSE Version:** ${{ matrix.fuse-version }}
**Test Type:** ${{ github.event.inputs.test_type || 'full' }}
<details>
<summary>Test Summary</summary>
\`\`\`
${report}
\`\`\`
</details>
📊 Full results available in [test artifacts](https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }})
`;
try {
await github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: comment
});
} catch (err) {
console.log('Could not post PR comment: ' + err.message);
}
}
posix-compliance-macos:
runs-on: macos-latest
timeout-minutes: 60
steps:
- name: Checkout code
uses: actions/checkout@v7
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version: ${{ env.GO_VERSION }}
cache: ${{ runner.environment == 'github-hosted' }}
- name: Install macFUSE
run: |
# Install macFUSE
brew install macfuse
# Note: macFUSE may require system extension approval on macOS
# This step may need manual intervention in some cases
- name: Build SeaweedFS
run: |
make
# Verify binary exists and is executable (it's installed to GOPATH/bin)
weed version
- name: Run critical POSIX tests (macOS)
continue-on-error: true # macOS FUSE can be more restrictive
run: |
cd test/fuse_integration
if [ ! -f go.mod ]; then
go mod init seaweedfs-posix-tests
go mod tidy
fi
mkdir -p reports
# Run basic tests only on macOS due to FUSE limitations
make -f posix_Makefile test-posix-critical
- name: Upload macOS test results
uses: actions/upload-artifact@v7
if: always()
with:
name: posix-test-results-macos
path: |
test/fuse_integration/reports/
test/fuse_integration/*.log
retention-days: 30
security-analysis:
runs-on: ubuntu-latest
if: github.event_name == 'pull_request'
steps:
- name: Checkout code
uses: actions/checkout@v7
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version: ${{ env.GO_VERSION }}
cache: ${{ runner.environment == 'github-hosted' }}
- name: Install security analysis tools
run: |
echo "Installing staticcheck for security analysis..."
go install honnef.co/go/tools/cmd/staticcheck@latest
- name: Run security analysis on FUSE code
run: |
# Analyze mount and FUSE-related code for security and quality issues
echo "Running staticcheck security analysis..."
# Run staticcheck on FUSE-related code
echo "Analyzing FUSE mount code..."
staticcheck ./weed/mount/... > staticcheck-report.txt 2>&1 || true
staticcheck ./weed/command/mount* ./weed/command/fuse* >> staticcheck-report.txt 2>&1 || true
# Convert to JSON format for consistency
echo "{" > gosec-report.json
echo " \"tool\": \"staticcheck\"," >> gosec-report.json
echo " \"timestamp\": \"$(date -Iseconds)\"," >> gosec-report.json
echo " \"issues\": [" >> gosec-report.json
if [ -s staticcheck-report.txt ]; then
echo " \"$(cat staticcheck-report.txt | head -20 | sed 's/"/\\"/g' | tr '\n' ' ')\"" >> gosec-report.json
fi
echo " ]," >> gosec-report.json
echo " \"stats\": {" >> gosec-report.json
echo " \"files_analyzed\": $(find ./weed/mount ./weed/command -name '*.go' | wc -l)," >> gosec-report.json
echo " \"issues_found\": $(wc -l < staticcheck-report.txt 2>/dev/null || echo 0)" >> gosec-report.json
echo " }" >> gosec-report.json
echo "}" >> gosec-report.json
echo "Security analysis completed"
echo "Issues found: $(wc -l < staticcheck-report.txt 2>/dev/null || echo 0)"
- name: Upload security analysis results
uses: actions/upload-artifact@v7
if: always()
with:
name: security-analysis-results
path: gosec-report.json
retention-days: 30
performance-baseline:
runs-on: ubuntu-latest
if: github.event.inputs.test_type == 'full' || github.event_name == 'schedule'
steps:
- name: Checkout code
uses: actions/checkout@v7
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version: ${{ env.GO_VERSION }}
cache: ${{ runner.environment == 'github-hosted' }}
- name: Install dependencies
run: |
sudo apt-get update
sudo apt-get install -y fuse libfuse-dev fio
sudo usermod -a -G fuse $USER
sudo chmod 666 /dev/fuse
# The posix suites mount with -allowOthers; allow_other needs this
echo 'user_allow_other' | sudo tee -a /etc/fuse.conf
- name: Build SeaweedFS
run: make
- name: Run performance baseline tests
run: |
cd test/fuse_integration
if [ ! -f go.mod ]; then
go mod init seaweedfs-posix-tests
go mod tidy
fi
mkdir -p reports
# Run performance benchmarks
make -f posix_Makefile benchmark-posix
# This job has no FUSE mount, so the fio suite can only skip; keep the
# run green but honest about producing no mounted-FS results
make -f posix_Makefile test-fio-posix || echo "FIO skipped - no mounted filesystem in this job"
- name: Store performance baseline
uses: actions/upload-artifact@v7
with:
name: performance-baseline-results
path: |
test/fuse_integration/reports/posix_benchmark_results.log
test/fuse_integration/reports/fio_*_results.log
retention-days: 90
# Schedule regular compliance checks
scheduled-compliance-check:
runs-on: ubuntu-latest
if: github.event_name == 'schedule'
steps:
- name: Checkout code
uses: actions/checkout@v7
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version: ${{ env.GO_VERSION }}
cache: ${{ runner.environment == 'github-hosted' }}
- name: Install dependencies
run: |
sudo apt-get update
sudo apt-get install -y fuse libfuse-dev
pip3 install --user nfstest
# The posix suites mount with -allowOthers; allow_other needs this
echo 'user_allow_other' | sudo tee -a /etc/fuse.conf
- name: Build SeaweedFS
run: make
- name: Run comprehensive compliance check
run: |
cd test/fuse_integration
make -f posix_Makefile test-posix-full
make -f posix_Makefile generate-report
- name: Create compliance issue if tests fail
if: failure()
uses: actions/github-script@v9
with:
script: |
const issue = await github.rest.issues.create({
owner: context.repo.owner,
repo: context.repo.repo,
title: 'POSIX Compliance Check Failed - ' + new Date().toISOString().split('T')[0],
body: `## POSIX Compliance Check Failure
The scheduled POSIX compliance check has failed. This may indicate:
- Regression in FUSE mount functionality
- Changes affecting POSIX compatibility
- Infrastructure issues with the test environment
**Action Required:** Review the test results and investigate any failures.
**Test Run:** [Workflow Run](https://github.com/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId})
**Date:** ${new Date().toISOString()}
---
This issue was automatically created by the POSIX compliance workflow.
`,
labels: ['bug', 'posix-compliance', 'automated-issue']
});
console.log('Created issue:', issue.data.number);
# Schedule this workflow to run weekly
# Uncomment the following lines to enable scheduled runs:
# schedule:
# - cron: '0 2 * * 1' # Every Monday at 2 AM UTC
+4 -2
View File
@@ -23,8 +23,8 @@ on:
- '.github/workflows/postgres-tests.yml'
concurrency:
group: ${{ github.head_ref }}/postgres-tests
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -41,6 +41,7 @@ jobs:
- name: Set up Go 1.x
uses: actions/setup-go@v7
with:
cache: ${{ runner.environment == 'github-hosted' }}
go-version: ^1.26
id: go
@@ -56,6 +57,7 @@ jobs:
uses: docker/setup-buildx-action@v4
- name: Cache Docker layers
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: /tmp/.buildx-cache
+135
View File
@@ -0,0 +1,135 @@
name: "rust: vector desktop platforms"
on:
pull_request:
paths:
- 'seaweed-vector/**'
- 'seaweed-worker/**'
- 'seaweed-common/**'
- 'weed/pb/**'
- '.github/workflows/rust-vector-platforms.yml'
- 'install.sh'
- 'test/installer/**'
push:
tags: ['*']
workflow_dispatch:
permissions:
contents: read
concurrency:
group: vector-platforms-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
jobs:
vector:
permissions:
contents: read
strategy:
fail-fast: false
matrix:
include:
- runner: macos-15-intel
suffix: darwin_amd64
protoc: macos-x86_64
- runner: macos-latest
suffix: darwin_arm64
protoc: macos-aarch_64
- runner: windows-latest
suffix: windows_amd64
protoc: win32
runs-on: ${{ matrix.runner }}
timeout-minutes: 120
env:
CARGO_BUILD_JOBS: '2'
CARGO_PROFILE_TEST_DEBUG: '0'
SEAWEEDFS_COMMIT: ${{ github.sha }}
defaults:
run:
shell: bash
steps:
- uses: actions/checkout@v7
with:
persist-credentials: false
- uses: dtolnay/rust-toolchain@stable
- uses: actions/setup-python@v7
with:
python-version: '3.13'
- uses: actions/cache/restore@v6
id: cargo-cache
if: ${{ !startsWith(github.ref, 'refs/tags/') }}
with:
path: |
~/.cargo/registry
~/.cargo/git
seaweed-vector/target
key: vector-${{ matrix.suffix }}-${{ hashFiles('seaweed-vector/Cargo.lock') }}
restore-keys: vector-${{ matrix.suffix }}-
- name: Fetch dependencies
run: cargo fetch --locked --manifest-path seaweed-vector/Cargo.toml
- name: Select vendored protoc
env:
PROTOC_PACKAGE: protoc-bin-vendored-${{ matrix.protoc }}
run: |
python - <<'PY'
import os, pathlib, tomllib
lock=tomllib.loads(pathlib.Path('seaweed-vector/Cargo.lock').read_text())
name=os.environ['PROTOC_PACKAGE']
version=next(p['version'] for p in lock['package'] if p['name']==name)
cargo=pathlib.Path(os.environ.get('CARGO_HOME',pathlib.Path.home()/'.cargo'))
executable='protoc.exe' if os.name=='nt' else 'protoc'
path=next(cargo.glob(f'registry/src/*/{name}-{version}/bin/{executable}'))
with open(os.environ['GITHUB_ENV'],'a') as out: out.write(f'PROTOC={path}\n')
PY
- name: Build vector engine
id: cargo-build
run: cargo build --locked --release --manifest-path seaweed-vector/Cargo.toml -p weed-vector-server
- name: Test vector crates
run: cargo test --locked --release --manifest-path seaweed-vector/Cargo.toml --workspace
- name: Test installer
run: |
if [ "$RUNNER_OS" = "Windows" ]; then
BASH_BINARY="$(cygpath -w "$BASH")"
export BASH_BINARY
fi
python -m unittest discover -s test/installer
- name: Save compiled dependencies
if: ${{ always() && !startsWith(github.ref, 'refs/tags/') && steps.cargo-build.outcome == 'success' && steps.cargo-cache.outputs.cache-hit != 'true' }}
uses: actions/cache/save@v6
with:
path: |
~/.cargo/registry
~/.cargo/git
seaweed-vector/target
key: vector-${{ matrix.suffix }}-${{ hashFiles('seaweed-vector/Cargo.lock') }}
- name: Package engine
env:
ASSET_SUFFIX: ${{ matrix.suffix }}
run: |
python - <<'PY'
import hashlib, os, pathlib, tarfile, zipfile
suffix=os.environ['ASSET_SUFFIX']
binary='weed-vector.exe' if os.name=='nt' else 'weed-vector'
source=pathlib.Path('seaweed-vector/target/release')/binary
asset=f'weed-vector_{suffix}'+('.zip' if os.name=='nt' else '.tar.gz')
if os.name=='nt':
with zipfile.ZipFile(asset,'w',zipfile.ZIP_DEFLATED) as archive: archive.write(source,binary)
else:
with tarfile.open(asset,'w:gz') as archive: archive.add(source,arcname=binary)
digest=hashlib.md5(pathlib.Path(asset).read_bytes()).hexdigest()
pathlib.Path(asset+'.md5').write_text(f'{digest} {asset}\n')
PY
- uses: actions/upload-artifact@v7
with:
name: vector-${{ matrix.suffix }}
path: weed-vector_${{ matrix.suffix }}.*
release:
if: startsWith(github.ref, 'refs/tags/')
needs: vector
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- uses: actions/download-artifact@v8
with:
pattern: vector-*
merge-multiple: true
- uses: softprops/action-gh-release@v3
with:
files: weed-vector_*
+181
View File
@@ -0,0 +1,181 @@
name: "Rust Vector Tests"
on:
pull_request:
branches: [ master ]
paths:
- 'seaweed-vector/**'
- 'seaweed-volume/**'
- 'seaweed-worker/**'
- 'seaweed-common/**'
- 'weed/pb/plugin.proto'
- 'weed/pb/vector.proto'
- 'weed/pb/vector_pb/**'
- 'weed/s3api/**'
- 'weed/command/**'
- 'test/s3vectors/**'
- '.github/workflows/rust-vector-tests.yml'
push:
branches: [ master, main ]
paths:
- 'seaweed-vector/**'
- 'seaweed-volume/**'
- 'seaweed-worker/**'
- 'seaweed-common/**'
- 'weed/pb/plugin.proto'
- 'weed/pb/vector.proto'
- 'weed/pb/vector_pb/**'
- 'weed/s3api/**'
- 'weed/command/**'
- 'test/s3vectors/**'
- '.github/workflows/rust-vector-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
jobs:
rust-vector-build:
name: Rust Vector Build and Unit Tests
runs-on: ubuntu-24.04
timeout-minutes: 45
steps:
- name: Checkout code
uses: actions/checkout@v7
with:
persist-credentials: false
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
- name: Fingerprint build toolchain
id: toolchain
run: echo "fingerprint=$(getconf GNU_LIBC_VERSION | tr ' ' '-')-rustc-$(rustc -V | awk '{print $2}')" >> "$GITHUB_OUTPUT"
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
~/.cargo/registry
~/.cargo/git
seaweed-vector/target/release
key: rust-vector-${{ steps.toolchain.outputs.fingerprint }}-${{ hashFiles('seaweed-vector/Cargo.lock') }}
restore-keys: |
rust-vector-${{ steps.toolchain.outputs.fingerprint }}-
- name: Build the vector crates
run: cd seaweed-vector && cargo build --release
- name: Run unit tests
run: cd seaweed-vector && cargo test --release --workspace
vectors-e2e:
name: S3 Vectors Integration (${{ matrix.volume }} volume${{ matrix.hardened && ', hardened' || '' }}${{ matrix.worker && ', worker' || '' }})
runs-on: ubuntu-24.04
timeout-minutes: 45
strategy:
fail-fast: false
matrix:
volume: [go, rust]
# Hardened: volume servers behind mutual TLS, every gRPC hop (master,
# volume, filer, gateway, engine) under mutual TLS, read JWTs
# required, and every chunk encrypted by the gateway.
hardened: ["", "1"]
worker: ["", "1"]
# Worker: the admin server and the Lance maintenance worker join the
# cluster, for the tests that build vector indexes through a job.
env:
CARGO_PROFILE_DEV_DEBUG: "0"
CARGO_BUILD_JOBS: "2"
CARGO_TARGET_DIR: ${{ github.workspace }}/seaweed-vector/target
VOLUME_SERVER_IMPL: ${{ matrix.volume }}
VECTOR_QUERY_MODE: single
VECTOR_FILER_GROUP: vectors-ci
VECTOR_TLS: ${{ matrix.hardened }}
VECTOR_READ_JWT: ${{ matrix.hardened }}
VECTOR_ENCRYPT_VOLUMES: ${{ matrix.hardened }}
VECTOR_WORKER: ${{ matrix.worker }}
steps:
- uses: actions/checkout@v7
with:
persist-credentials: false
- uses: actions/setup-go@v7
with:
go-version-file: go.mod
cache: ${{ runner.environment == 'github-hosted' }}
- uses: dtolnay/rust-toolchain@stable
- name: Fingerprint build toolchain
id: toolchain
run: echo "fingerprint=$(getconf GNU_LIBC_VERSION | tr ' ' '-')-rustc-$(rustc -V | awk '{print $2}')" >> "$GITHUB_OUTPUT"
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
~/.cargo/registry
~/.cargo/git
seaweed-vector/target/debug
key: vectors-e2e-${{ runner.os }}-${{ runner.arch }}-${{ steps.toolchain.outputs.fingerprint }}-${{ hashFiles('seaweed-vector/Cargo.lock', 'seaweed-volume/Cargo.lock') }}
restore-keys: |
vectors-e2e-${{ runner.os }}-${{ runner.arch }}-${{ steps.toolchain.outputs.fingerprint }}-
- name: Test Go vector gateway
run: |
go test ./weed/s3api/s3vectors -count=1
go test ./weed/command -run 'Vector|Vectors' -count=1
# The worker workspace has no protoc of its own for lance's build
# scripts; use the vendored one its lock names, as its own workflow does.
- name: Use the vendored protoc for the worker
if: matrix.worker == '1'
run: |
cd seaweed-worker
cargo fetch --locked
version=$(awk '/^name = "protoc-bin-vendored-linux-x86_64"$/{found=1; next} found && /^version = /{gsub(/"/,"",$3); print $3; exit}' Cargo.lock)
test -n "$version" || { echo "protoc-bin-vendored-linux-x86_64 is not in Cargo.lock" >&2; exit 1; }
protoc=$(find ~/.cargo/registry/src -path "*protoc-bin-vendored-linux-x86_64-$version/bin/protoc" | head -1)
test -x "$protoc" || { echo "no vendored protoc $version in the registry" >&2; exit 1; }
echo "PROTOC=$protoc" >> "$GITHUB_ENV"
- name: Build integration binaries
working-directory: test/s3vectors
# dev profile: correctness suite, no release codegen needed. weed-volume
# (the heaviest crate) only builds on the rust leg.
run: |
set -e
make ./weed ./weed-vector CARGO_PROFILE=dev
if [ "${{ matrix.volume }}" = "rust" ]; then make ./weed-volume CARGO_PROFILE=dev; fi
if [ "${{ matrix.worker }}" = "1" ]; then make ./weed-vector-cli ./weed-worker CARGO_PROFILE=dev; fi
- uses: actions/setup-python@v7
with:
python-version: "3.12"
- name: Install test dependencies
run: python -m pip install -r test/s3vectors/requirements.txt
- name: Run the S3 Vectors integration matrix
working-directory: test/s3vectors
run: |
./run_test.sh --basetemp="$RUNNER_TEMP/vector-tests" --junitxml="$RUNNER_TEMP/vector-results.xml"
python - <<'PYTHON'
import os
from pathlib import Path
import xml.etree.ElementTree as ET
cases = ET.parse(Path(os.environ["RUNNER_TEMP"]) / "vector-results.xml").findall(".//testcase")
assert len(cases) >= 17, f"expected at least 17 cases, got {len(cases)}"
assert not any(case.find("skipped") is not None for case in cases), "integration cases were skipped"
volume = os.environ["VOLUME_SERVER_IMPL"]
assert all(case.get("name", "").endswith(f"[{volume}-single]") for case in cases), volume
PYTHON
- name: Upload integration diagnostics
if: always()
uses: actions/upload-artifact@v7
with:
name: vectors-e2e-diagnostics-${{ matrix.volume }}${{ matrix.hardened && '-hardened' || '' }}${{ matrix.worker && '-worker' || '' }}
path: |
${{ runner.temp }}/vector-tests
${{ runner.temp }}/vector-results.xml
if-no-files-found: ignore
retention-days: 7
+85 -63
View File
@@ -21,8 +21,8 @@ on:
- '.github/workflows/rust-volume-server-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.ref }}
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -52,9 +52,68 @@ jobs:
- 'seaweed-volume/**'
- '.github/workflows/rust-volume-server-tests.yml'
build-test-binaries:
name: Build Test Binaries
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
- name: Checkout code
uses: actions/checkout@v7
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
# cargo tracks its own inputs but not the runner's C toolchain, so a cached
# target/ can carry C objects built against a different glibc than we link against.
- name: Fingerprint build toolchain
id: toolchain
run: echo "fingerprint=$(getconf GNU_LIBC_VERSION | tr ' ' '-')-rustc-$(rustc -V | awk '{print $2}')" >> "$GITHUB_OUTPUT"
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
~/.cargo/registry
~/.cargo/git
seaweed-volume/target
key: rust-${{ steps.toolchain.outputs.fingerprint }}-${{ hashFiles('seaweed-volume/Cargo.lock', 'seaweed-common/Cargo.lock') }}
restore-keys: |
rust-${{ steps.toolchain.outputs.fingerprint }}-
- name: Build Go weed binary
run: |
cd weed
go build -tags 5BytesOffset -o weed .
chmod +x weed
./weed version
- name: Build Rust volume binary
run: cd seaweed-volume && cargo build --release
- name: Package binaries
run: tar cf rust-test-binaries.tar weed/weed seaweed-volume/target/release/weed-volume
- name: Upload test binaries
uses: actions/upload-artifact@v7
with:
# archive layout mirrors the workspace: weed/weed and
# seaweed-volume/target/release/weed-volume
name: rust-test-binaries
if-no-files-found: error
retention-days: 1
path: rust-test-binaries.tar
rust-unit-tests:
name: Rust Unit Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 15
steps:
@@ -71,6 +130,7 @@ jobs:
run: echo "fingerprint=$(getconf GNU_LIBC_VERSION | tr ' ' '-')-rustc-$(rustc -V | awk '{print $2}')" >> "$GITHUB_OUTPUT"
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
@@ -108,9 +168,6 @@ jobs:
- name: Run Rust unit tests
run: cd seaweed-volume && cargo test
- name: Run Rust unit tests (redb experimental cursor)
run: cd seaweed-volume && cargo test --features redb-experimental-cursor --lib storage::needle_map
rust-unit-tests-windows:
name: Rust Unit Tests (Windows)
runs-on: windows-latest
@@ -134,6 +191,7 @@ jobs:
run: echo "fingerprint=windows-rustc-$(rustc -V | awk '{print $2}')" >> "$GITHUB_OUTPUT"
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
@@ -147,13 +205,11 @@ jobs:
- name: Run Rust unit tests
run: cd seaweed-volume && cargo test
- name: Run Rust unit tests (redb experimental cursor)
run: cd seaweed-volume && cargo test --features redb-experimental-cursor --lib storage::needle_map
rust-integration-tests:
name: Rust Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
needs: [build-test-binaries]
steps:
- name: Checkout code
@@ -163,36 +219,18 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
# cargo tracks its own inputs but not the runner's C toolchain, so a cached
# target/ can carry C objects built against a different glibc than we link against.
- name: Fingerprint build toolchain
id: toolchain
run: echo "fingerprint=$(getconf GNU_LIBC_VERSION | tr ' ' '-')-rustc-$(rustc -V | awk '{print $2}')" >> "$GITHUB_OUTPUT"
- name: Cache cargo registry and target
uses: actions/cache@v6
- name: Download test binaries
uses: actions/download-artifact@v8
with:
path: |
~/.cargo/registry
~/.cargo/git
seaweed-volume/target
key: rust-${{ steps.toolchain.outputs.fingerprint }}-${{ hashFiles('seaweed-volume/Cargo.lock', 'seaweed-common/Cargo.lock') }}
restore-keys: |
rust-${{ steps.toolchain.outputs.fingerprint }}-
name: rust-test-binaries
- name: Build Go weed binary
- name: Restore binaries
run: |
cd weed
go build -tags 5BytesOffset -o weed .
chmod +x weed
./weed version
- name: Build Rust volume binary
run: cd seaweed-volume && cargo build --release
tar xf rust-test-binaries.tar
chmod +x weed/weed seaweed-volume/target/release/weed-volume
./weed/weed version
- name: Run integration tests
env:
@@ -226,8 +264,9 @@ jobs:
rust-volume-go-tests:
name: Go Tests with Rust Volume (${{ matrix.test-type }} - Shard ${{ matrix.shard }})
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 45
needs: [build-test-binaries]
env:
# Keep in step with the length of matrix.shard below.
SHARD_COUNT: 3
@@ -245,36 +284,19 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
# cargo tracks its own inputs but not the runner's C toolchain, so a cached
# target/ can carry C objects built against a different glibc than we link against.
- name: Fingerprint build toolchain
id: toolchain
run: echo "fingerprint=$(getconf GNU_LIBC_VERSION | tr ' ' '-')-rustc-$(rustc -V | awk '{print $2}')" >> "$GITHUB_OUTPUT"
- name: Cache cargo registry and target
uses: actions/cache@v6
- name: Download test binaries
uses: actions/download-artifact@v8
with:
path: |
~/.cargo/registry
~/.cargo/git
seaweed-volume/target
key: rust-${{ steps.toolchain.outputs.fingerprint }}-${{ hashFiles('seaweed-volume/Cargo.lock', 'seaweed-common/Cargo.lock') }}
restore-keys: |
rust-${{ steps.toolchain.outputs.fingerprint }}-
name: rust-test-binaries
- name: Build Go weed binary
- name: Restore binaries
run: |
cd weed
go build -tags 5BytesOffset -o weed .
chmod +x weed
./weed version
- name: Build Rust volume binary
run: cd seaweed-volume && cargo build --release
tar xf rust-test-binaries.tar
chmod +x weed/weed seaweed-volume/target/release/weed-volume
./weed/weed version
./seaweed-volume/target/release/weed-volume --help >/dev/null || true
# Dealing the listed tests out one by one keeps the shards even. Bucketing
# them by first letter did not: names cluster, so ^Test[I-S] drew 50 of
+4 -3
View File
@@ -17,8 +17,8 @@ on:
- '.github/workflows/rust-worker-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.ref }}
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -26,7 +26,7 @@ permissions:
jobs:
rust-worker-build:
name: Rust Plugin Worker Build and Unit Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 45
steps:
@@ -45,6 +45,7 @@ jobs:
run: echo "fingerprint=$(getconf GNU_LIBC_VERSION | tr ' ' '-')-rustc-$(rustc -V | awk '{print $2}')" >> "$GITHUB_OUTPUT"
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
+3 -1
View File
@@ -32,7 +32,7 @@ jobs:
permissions:
contents: write
needs: cleanup
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
strategy:
matrix:
include:
@@ -46,6 +46,7 @@ jobs:
uses: dtolnay/rust-toolchain@stable
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
@@ -114,6 +115,7 @@ jobs:
targets: ${{ matrix.target }}
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
+97 -39
View File
@@ -15,39 +15,27 @@ jobs:
build-rust-volume-linux:
permissions:
contents: write
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
strategy:
matrix:
include:
- target: x86_64-unknown-linux-gnu
# Static musl, as the container images ship: a glibc build only
# starts on distributions at least as new as the runner's.
- target: x86_64-unknown-linux-musl
asset_suffix: linux_amd64
- target: aarch64-unknown-linux-gnu
- target: aarch64-unknown-linux-musl
asset_suffix: linux_arm64
cross: true
steps:
- uses: actions/checkout@v7
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
- name: Set up the musl target
uses: ./.github/actions/rust-musl-target
with:
targets: ${{ matrix.target }}
- name: Install cross-compilation tools
if: matrix.cross
run: |
sudo dpkg --add-architecture arm64
sudo sed -i 's/^deb /deb [arch=amd64] /' /etc/apt/sources.list
echo "deb [arch=arm64] http://ports.ubuntu.com/ jammy main restricted universe multiverse" | sudo tee /etc/apt/sources.list.d/arm64.list
echo "deb [arch=arm64] http://ports.ubuntu.com/ jammy-updates main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list.d/arm64.list
sudo apt-get update
sudo apt-get install -y gcc-aarch64-linux-gnu libssl-dev:arm64
echo "CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER=aarch64-linux-gnu-gcc" >> "$GITHUB_ENV"
echo "OPENSSL_DIR=/usr" >> "$GITHUB_ENV"
echo "OPENSSL_INCLUDE_DIR=/usr/include" >> "$GITHUB_ENV"
echo "OPENSSL_LIB_DIR=/usr/lib/aarch64-linux-gnu" >> "$GITHUB_ENV"
target: ${{ matrix.target }}
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
@@ -119,15 +107,16 @@ jobs:
build-rust-worker-linux:
permissions:
contents: write
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
strategy:
matrix:
include:
- target: x86_64-unknown-linux-gnu
# Static musl, as the container images ship: a glibc build only
# starts on distributions at least as new as the runner's.
- target: x86_64-unknown-linux-musl
asset_suffix: linux_amd64
- target: aarch64-unknown-linux-gnu
- target: aarch64-unknown-linux-musl
asset_suffix: linux_arm64
cross: true
steps:
- uses: actions/checkout@v7
@@ -136,23 +125,13 @@ jobs:
# should not find another one sitting in the checkout's git config.
persist-credentials: false
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
- name: Set up the musl target
uses: ./.github/actions/rust-musl-target
with:
targets: ${{ matrix.target }}
- name: Install cross-compilation tools
if: matrix.cross
run: |
sudo dpkg --add-architecture arm64
sudo sed -i 's/^deb /deb [arch=amd64] /' /etc/apt/sources.list
echo "deb [arch=arm64] http://ports.ubuntu.com/ jammy main restricted universe multiverse" | sudo tee /etc/apt/sources.list.d/arm64.list
echo "deb [arch=arm64] http://ports.ubuntu.com/ jammy-updates main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list.d/arm64.list
sudo apt-get update
sudo apt-get install -y gcc-aarch64-linux-gnu
echo "CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER=aarch64-linux-gnu-gcc" >> "$GITHUB_ENV"
target: ${{ matrix.target }}
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
@@ -209,6 +188,83 @@ jobs:
weed-worker_${{ matrix.asset_suffix }}.tar.gz
weed-worker_${{ matrix.asset_suffix }}.tar.gz.md5
# The vector engine behind the S3 Vectors API: Linux only, like the worker,
# whose lance dependency tree it shares.
build-rust-vector-linux:
permissions:
contents: write
runs-on: ubuntu-24.04
strategy:
matrix:
include:
- target: x86_64-unknown-linux-musl
asset_suffix: linux_amd64
- target: aarch64-unknown-linux-musl
asset_suffix: linux_arm64
steps:
- uses: actions/checkout@v7
with:
persist-credentials: false
- name: Set up the musl target
uses: ./.github/actions/rust-musl-target
with:
target: ${{ matrix.target }}
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
~/.cargo/registry
~/.cargo/git
seaweed-vector/target/${{ matrix.target }}/release
key: rust-vector-release-${{ matrix.target }}-${{ hashFiles('seaweed-vector/Cargo.lock') }}
restore-keys: |
rust-vector-release-${{ matrix.target }}-
- name: Use the vendored protoc
run: |
cd seaweed-vector
cargo fetch
version=$(awk '/^name = "protoc-bin-vendored-linux-x86_64"$/{found=1; next} found && /^version = /{gsub(/"/,"",$3); print $3; exit}' Cargo.lock)
test -n "$version" || { echo "protoc-bin-vendored-linux-x86_64 is not in Cargo.lock" >&2; exit 1; }
protoc=$(find ~/.cargo/registry/src -path "*protoc-bin-vendored-linux-x86_64-$version/bin/protoc" | head -1)
test -x "$protoc" || { echo "no vendored protoc $version in the registry" >&2; exit 1; }
echo "PROTOC=$protoc" >> "$GITHUB_ENV"
- name: Build the vector engine
run: |
cd seaweed-vector
cargo build --release -p weed-vector-server --target ${{ matrix.target }}
- name: Package binary
run: |
cp seaweed-vector/target/${{ matrix.target }}/release/weed-vector weed-vector
tar czf weed-vector_${{ matrix.asset_suffix }}.tar.gz weed-vector
rm weed-vector
md5sum weed-vector_${{ matrix.asset_suffix }}.tar.gz > weed-vector_${{ matrix.asset_suffix }}.tar.gz.md5
- name: Upload release assets
if: startsWith(github.ref, 'refs/tags/')
uses: softprops/action-gh-release@v3
with:
files: |
weed-vector_${{ matrix.asset_suffix }}.tar.gz
weed-vector_${{ matrix.asset_suffix }}.tar.gz.md5
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Upload artifacts
if: ${{ !startsWith(github.ref, 'refs/tags/') }}
uses: actions/upload-artifact@v7
with:
name: rust-vector-${{ matrix.asset_suffix }}
path: |
weed-vector_${{ matrix.asset_suffix }}.tar.gz
weed-vector_${{ matrix.asset_suffix }}.tar.gz.md5
build-rust-volume-darwin:
permissions:
contents: write
@@ -230,6 +286,7 @@ jobs:
targets: ${{ matrix.target }}
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
@@ -305,6 +362,7 @@ jobs:
uses: dtolnay/rust-toolchain@stable
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
+4 -3
View File
@@ -21,8 +21,8 @@ on:
- '.github/workflows/s3-etag-acl-tests.yml'
concurrency:
group: ${{ github.head_ref || github.ref }}/s3-etag-acl-tests
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -30,7 +30,7 @@ permissions:
jobs:
s3-etag-acl-tests:
name: S3 ETag + ACL Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 15
steps:
- name: Check out code
@@ -40,6 +40,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Install SeaweedFS
run: |
@@ -13,8 +13,8 @@ on:
- '.github/workflows/s3-example-integration-tests.yml'
concurrency:
group: ${{ github.head_ref }}/s3-integration-tests
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -22,7 +22,7 @@ permissions:
jobs:
tests:
name: S3 Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
@@ -33,6 +33,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Build SeaweedFS
+8 -4
View File
@@ -12,8 +12,8 @@ on:
- '.github/workflows/s3-filer-group-tests.yml'
concurrency:
group: ${{ github.head_ref }}/s3-filer-group-tests
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -25,17 +25,21 @@ defaults:
jobs:
s3-filer-group-tests:
name: S3 Filer Group Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -50,7 +54,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
echo "=== Starting Filer Group Tests ==="
# Run tests with automatic server management
+42 -17
View File
@@ -4,6 +4,7 @@ on:
pull_request:
paths:
- 'weed/s3api/**'
- 'weed/remote_storage/**'
- 'weed/filer/**'
- 'weed/server/**'
- 'test/s3/**'
@@ -12,8 +13,8 @@ on:
- '.github/workflows/s3-go-tests.yml'
concurrency:
group: ${{ github.head_ref }}/s3-go-tests
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -25,7 +26,7 @@ defaults:
jobs:
s3-versioning-tests:
name: S3 Versioning Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
strategy:
matrix:
@@ -35,10 +36,14 @@ jobs:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -53,7 +58,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
echo "=== Starting Tests ==="
# Run tests with automatic server management
@@ -92,7 +97,7 @@ jobs:
s3-versioning-compatibility:
name: S3 Versioning Compatibility Test
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
@@ -103,6 +108,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -140,7 +146,7 @@ jobs:
s3-cors-compatibility:
name: S3 CORS Compatibility Test
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
@@ -151,6 +157,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -188,7 +195,7 @@ jobs:
s3-retention-tests:
name: S3 Retention Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
strategy:
matrix:
@@ -198,10 +205,14 @@ jobs:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -216,7 +227,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
echo "=== Starting Tests ==="
# Run tests with automatic server management
@@ -255,7 +266,7 @@ jobs:
s3-lifecycle-tests:
name: S3 Lifecycle Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 10
strategy:
fail-fast: false
@@ -283,6 +294,7 @@ jobs:
- TestLifecycleSuspendedVersioningExpiration
- TestLifecycleTagFilter
- TestLifecycleVersionedBucketCreatesDeleteMarker
- TestLifecycleVersionedBucketExpiresOnSecondRun
steps:
- name: Check out code
@@ -292,6 +304,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -326,17 +339,21 @@ jobs:
s3-checksum-tests:
name: S3 Checksum Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -351,7 +368,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
echo "=== Starting Tests ==="
make test-with-server
@@ -381,7 +398,7 @@ jobs:
s3-cors-tests:
name: S3 CORS Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
strategy:
matrix:
@@ -391,10 +408,14 @@ jobs:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -409,7 +430,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
echo "=== Starting Tests ==="
# Run tests with automatic server management
@@ -448,7 +469,7 @@ jobs:
s3-retention-worm:
name: S3 Retention WORM Integration Test
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
@@ -459,6 +480,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -497,7 +519,7 @@ jobs:
s3-versioning-stress:
name: S3 Versioning Stress Test
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 35
steps:
@@ -508,6 +530,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -565,7 +588,7 @@ jobs:
s3-tagging-tests:
# CI job for S3 object tagging tests
name: S3 Tagging Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
@@ -576,6 +599,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -620,7 +644,7 @@ jobs:
s3-remote-cache-tests:
name: S3 Remote Cache Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
@@ -631,6 +655,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
+14 -7
View File
@@ -20,8 +20,8 @@ on:
- '.github/workflows/s3-iam-tests.yml'
concurrency:
group: ${{ github.head_ref }}/s3-iam-tests
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -34,7 +34,7 @@ jobs:
# Unit tests for IAM components
iam-unit-tests:
name: IAM Unit Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 15
steps:
@@ -45,6 +45,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Get dependencies
@@ -80,7 +81,7 @@ jobs:
# S3 IAM integration tests with SeaweedFS services
s3-iam-integration-tests:
name: S3 IAM Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 25
strategy:
matrix:
@@ -90,10 +91,14 @@ jobs:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -109,7 +114,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
echo "=== Starting S3 IAM Integration Tests (${{ matrix.test-type }}) ==="
# Set WEED_BINARY to use the installed version
@@ -197,7 +202,7 @@ jobs:
# Distributed IAM tests
s3-iam-distributed-tests:
name: S3 IAM Distributed Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 25
steps:
@@ -208,6 +213,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -257,7 +263,7 @@ jobs:
# Performance and stress tests
s3-iam-performance-tests:
name: S3 IAM Performance Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
@@ -268,6 +274,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
+8 -4
View File
@@ -16,8 +16,8 @@ on:
- '.github/workflows/s3-keycloak-tests.yml'
concurrency:
group: ${{ github.head_ref }}/s3-keycloak-tests
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -30,17 +30,21 @@ jobs:
# Dedicated job for Keycloak integration tests
s3-keycloak-integration-tests:
name: S3 Keycloak Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -56,7 +60,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
echo "=== Starting S3 Keycloak Integration Tests ==="
# Set WEED_BINARY to use the installed version
@@ -0,0 +1,74 @@
name: "S3 listing study tests"
on:
push:
branches: [ master ]
paths:
- 'weed/s3api/**'
- 'weed/filer/**'
- 'weed/server/**'
- 'test/s3/listing_study/**'
- '.github/workflows/s3-listing-study-tests.yml'
pull_request:
branches: [ master ]
paths:
- 'weed/s3api/**'
- 'weed/filer/**'
- 'weed/server/**'
- 'test/s3/listing_study/**'
- '.github/workflows/s3-listing-study-tests.yml'
concurrency:
group: ${{ github.event.pull_request.number || github.ref }}/s3-listing-study-tests
cancel-in-progress: true
permissions:
contents: read
jobs:
listing-study:
name: s3-listing-study tools and strategies
runs-on: ubuntu-24.04
timeout-minutes: 40
defaults:
run:
working-directory: test/s3/listing_study
steps:
- name: Check out code
uses: actions/checkout@v7
with:
persist-credentials: false
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Cache pinned listing tools
uses: actions/cache@v6
with:
path: test/s3/listing_study/.work/tools
key: listing-study-tools-${{ runner.os }}-${{ hashFiles('test/s3/listing_study/Makefile', 'test/s3/listing_study/install_tools.py') }}
- name: Build weed and install the study's pinned tools
run: make build-weed tools
- name: Run the listing study against SeaweedFS
timeout-minutes: 15
run: make test
- name: Show logs on failure
if: failure()
run: |
tail -200 .work/weed.log || true
- name: Upload results
if: always()
uses: actions/upload-artifact@v7
with:
name: s3-listing-study-results
path: |
test/s3/listing_study/.work/run/results.json
test/s3/listing_study/.work/weed.log
retention-days: 14
@@ -6,14 +6,15 @@ on:
- 'weed/s3api/**'
- 'test/s3/delete/**'
- 'test/s3/distributed_lock/**'
- 'test/s3/sqlite_cbs/**'
- 'test/s3/versioning/**'
- 'test/volume_server/framework/**'
- 'docker/compose/s3.json'
- '.github/workflows/s3-mutation-regression-tests.yml'
concurrency:
group: ${{ github.head_ref }}/s3-mutation-regression-tests
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -21,7 +22,7 @@ permissions:
jobs:
s3-versioning-regressions:
name: S3 Versioning Regression Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 25
steps:
@@ -32,6 +33,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Run S3 versioning regression tests
timeout-minutes: 20
@@ -74,7 +76,7 @@ jobs:
s3-delete-regressions:
name: S3 Delete Regression Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
@@ -85,6 +87,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Run S3 delete regression tests
timeout-minutes: 15
@@ -112,7 +115,7 @@ jobs:
s3-distributed-lock-regressions:
name: S3 Distributed Lock Regression Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
@@ -123,6 +126,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Build SeaweedFS
run: |
@@ -146,3 +150,41 @@ jobs:
name: s3-distributed-lock-regression-logs
path: test/s3/distributed_lock/tmp/seaweedfs_s3_distributed_lock_*
retention-days: 3
s3-sqlite-cbs-regressions:
name: S3 SQLite CBS Regression Tests
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Build SeaweedFS
run: |
go build -o weed/weed -buildvcs=false ./weed
- name: Run sqlite cbs regressions
timeout-minutes: 15
env:
TMPDIR: ${{ github.workspace }}/test/s3/sqlite_cbs/tmp
S3_SQLITE_CBS_KEEP_LOGS: "1"
WEED_BINARY: ${{ github.workspace }}/weed/weed
run: |
set -x
mkdir -p "$TMPDIR"
go test -v -count=1 -timeout=12m ./test/s3/sqlite_cbs
- name: Upload sqlite cbs logs on failure
if: failure()
uses: actions/upload-artifact@v7
with:
name: s3-sqlite-cbs-regression-logs
path: test/s3/sqlite_cbs/tmp/seaweedfs_sqlite_cbs_*
retention-days: 3
+8 -4
View File
@@ -23,6 +23,10 @@ env:
S3_ENDPOINT_URL: http://localhost:8333
BUCKET_NAME: test-parquet-bucket
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
jobs:
parquet-integration-tests:
name: PyArrow Parquet Tests (Python ${{ matrix.python-version }})
@@ -32,7 +36,7 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: ['3.9', '3.11', '3.12']
python-version: ['3.10', '3.11', '3.12']
steps:
- name: Checkout code
@@ -42,13 +46,13 @@ jobs:
uses: actions/setup-go@v7
with:
go-version: ^1.26
cache: true
cache: ${{ runner.environment == 'github-hosted' }}
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v7
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache: ${{ runner.environment == 'github-hosted' && 'pip' || '' }}
cache-dependency-path: 'test/s3/parquet/requirements.txt'
- name: Install system dependencies
@@ -150,7 +154,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version: ^1.26
cache: true
cache: ${{ runner.environment == 'github-hosted' }}
- name: Run Go unit tests
run: |
+20 -9
View File
@@ -24,8 +24,8 @@ on:
- '.github/workflows/s3-policy-tests.yml'
concurrency:
group: ${{ github.head_ref }}/s3-policy-tests
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -38,7 +38,7 @@ jobs:
# Unit tests for policy components
policy-unit-tests:
name: S3 Policy Unit Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 15
steps:
@@ -49,6 +49,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Get dependencies
@@ -81,17 +82,21 @@ jobs:
# S3 Policy Variables Integration Tests
s3-policy-variables-tests:
name: S3 Policy Variables Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 25
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -106,7 +111,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
echo "=== Starting S3 Policy Variables Integration Tests ==="
@@ -187,7 +192,7 @@ jobs:
# S3 Policy Enforcement Integration Tests
s3-policy-enforcement-tests:
name: S3 Policy Enforcement Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
strategy:
matrix:
@@ -197,10 +202,14 @@ jobs:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -215,7 +224,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
echo "=== Starting S3 Policy Enforcement Tests (${{ matrix.test-case }}) ==="
@@ -308,7 +317,7 @@ jobs:
# Trusted Proxy Detection Tests
trusted-proxy-tests:
name: Trusted Proxy Detection Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
@@ -319,6 +328,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -395,7 +405,7 @@ jobs:
# S3 Policy Shell Integration Tests
s3-policy-shell-tests:
name: S3 Policy Shell Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 15
steps:
@@ -406,6 +416,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -21,8 +21,8 @@ on:
- '.github/workflows/s3-proxy-signature-tests.yml'
concurrency:
group: ${{ github.head_ref || github.ref }}/s3-proxy-signature-tests
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -30,7 +30,7 @@ permissions:
jobs:
proxy-signature-tests:
name: S3 Proxy Signature Verification Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 15
steps:
- name: Check out code into the Go module directory
@@ -40,6 +40,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -15,8 +15,8 @@ on:
- '.github/workflows/s3-sdk-v2-routing-tests.yml'
concurrency:
group: ${{ github.head_ref || github.ref }}/s3-sdk-v2-routing-tests
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -24,7 +24,7 @@ permissions:
jobs:
s3-sdk-v2-routing-tests:
name: S3 SDK V2 Routing Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 10
steps:
- name: Check out code
@@ -34,6 +34,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Install SeaweedFS
run: |
+13 -2
View File
@@ -46,7 +46,7 @@ permissions:
jobs:
snowflake-s3compat-tests:
name: Snowflake S3Compat API tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
env:
WORK_DIR: /tmp/seaweedfs-snowflake-tests
@@ -60,6 +60,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Set up Java
@@ -67,7 +68,15 @@ jobs:
with:
java-version: '17'
distribution: 'temurin'
cache: 'maven'
- name: Cache suite Maven dependencies
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: ~/.m2/repository
key: snowflake-s3compat-m2-${{ hashFiles('test/s3/snowflake/run.sh') }}
restore-keys: |
snowflake-s3compat-m2-
- name: Install SeaweedFS
run: |
@@ -77,6 +86,8 @@ jobs:
- name: Run Snowflake S3Compat API tests
timeout-minutes: 20
env:
MAVEN_ARGS: -ntp -s ${{ github.workspace }}/test/s3/snowflake/maven-settings.xml
run: |
# Starts weed server, creates the buckets/objects the suite needs,
# clones the upstream suite, and runs mvn -Dtest=S3CompatApiTest.
+8 -4
View File
@@ -11,8 +11,8 @@ on:
workflow_dispatch:
concurrency:
group: ${{ github.head_ref }}/s3-spark-tests
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -20,17 +20,21 @@ permissions:
jobs:
s3-spark-issue-repro-tests:
name: S3 Spark Issue Reproduction Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 45
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -56,7 +60,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
echo "=== Starting S3 Spark Integration Tests ==="
go test -v -timeout 30m . 2>&1 | tee test-output.log || {
+20 -10
View File
@@ -23,8 +23,8 @@ on:
- 'test/s3/sse/**'
concurrency:
group: ${{ github.head_ref }}/s3-sse-tests
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -36,7 +36,7 @@ defaults:
jobs:
s3-sse-integration-tests:
name: S3 SSE Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
strategy:
matrix:
@@ -46,10 +46,14 @@ jobs:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -64,7 +68,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
echo "=== Starting SSE Tests ==="
# Run tests with automatic server management
@@ -107,7 +111,7 @@ jobs:
s3-sse-compatibility:
name: S3 SSE Compatibility Test
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
@@ -118,6 +122,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -155,7 +160,7 @@ jobs:
s3-sse-metadata-persistence:
name: S3 SSE Metadata Persistence Test
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
@@ -166,6 +171,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -204,7 +210,7 @@ jobs:
s3-sse-copy-operations:
name: S3 SSE Copy Operations Test
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 25
steps:
@@ -215,6 +221,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -253,7 +260,7 @@ jobs:
s3-sse-multipart:
name: S3 SSE Multipart Upload Test
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 25
steps:
@@ -264,6 +271,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -302,7 +310,7 @@ jobs:
s3-sse-performance:
name: S3 SSE Performance Test
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 35
# Only run performance tests on master branch pushes to avoid overloading PR testing
if: github.event_name == 'push' && (github.ref == 'refs/heads/master' || github.ref == 'refs/heads/main')
@@ -315,6 +323,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -352,7 +361,7 @@ jobs:
s3-volume-encryption:
name: S3 Volume Encryption Test
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
@@ -363,6 +372,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
+116 -36
View File
@@ -12,6 +12,10 @@ on:
- 'go.sum'
- '.github/workflows/s3-tables-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -19,17 +23,21 @@ permissions:
jobs:
s3-tables-tests:
name: S3 Tables Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Run go mod tidy
@@ -48,7 +56,8 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
# A stale FUSE mount can hang or fail df; keep diagnostics best effort.
timeout 15 df -h || true
echo "=== Starting S3 Tables Tests ==="
# Run S3 Tables integration tests
@@ -79,17 +88,21 @@ jobs:
iceberg-catalog-tests:
name: Iceberg Catalog Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -109,7 +122,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
echo "=== Starting Iceberg Catalog Tests ==="
# Run Iceberg catalog integration tests using Makefile (handles build)
@@ -140,17 +153,21 @@ jobs:
trino-iceberg-catalog-tests:
name: Trino Iceberg Catalog Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -179,7 +196,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
echo "=== Starting Trino Iceberg Catalog Tests ==="
# Run Trino + Iceberg catalog integration tests
@@ -210,17 +227,21 @@ jobs:
dremio-iceberg-catalog-tests:
name: Dremio Iceberg Catalog Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -250,7 +271,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
docker info
echo "=== Starting Dremio Iceberg Catalog Tests ==="
@@ -283,17 +304,21 @@ jobs:
doris-iceberg-catalog-tests:
name: Doris Iceberg Catalog Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 35
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -323,7 +348,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
docker info
echo "=== Starting Doris Iceberg Catalog Tests ==="
@@ -356,7 +381,7 @@ jobs:
clickhouse-iceberg-catalog-tests:
name: ClickHouse Iceberg Catalog Integration Tests (${{ matrix.tag }})
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
strategy:
fail-fast: false
@@ -373,10 +398,14 @@ jobs:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -408,7 +437,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
docker info
echo "=== Starting ClickHouse Iceberg Catalog Tests ==="
@@ -441,7 +470,7 @@ jobs:
olake-iceberg-catalog-tests:
name: OLake Iceberg Catalog Integration Tests (${{ matrix.tag }})
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
strategy:
fail-fast: false
@@ -460,10 +489,14 @@ jobs:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -496,7 +529,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
docker info
echo "=== Starting OLake Iceberg Catalog Tests ==="
@@ -552,17 +585,21 @@ jobs:
polaris-integration-tests:
name: Polaris Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Run go mod tidy
@@ -590,7 +627,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
echo "=== Starting Polaris Tests ==="
go test -v -timeout 20m ./test/s3tables/polaris 2>&1 | tee test/s3tables/polaris/test-output.log || {
@@ -620,17 +657,21 @@ jobs:
spark-iceberg-catalog-tests:
name: Spark Iceberg Catalog Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -643,6 +684,12 @@ jobs:
pull() { for i in 1 2 3; do docker pull "$1" && return 0; sleep 15; done; return 1; }
pull apache/spark:3.5.1
- name: Cache Iceberg jars
uses: actions/cache@v6
with:
path: ~/.cache/seaweedfs-test-jars
key: iceberg-jars-${{ hashFiles('test/s3tables/catalog_spark/setup_test.go') }}
- name: Run go mod tidy
run: go mod tidy
@@ -659,7 +706,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
echo "=== Starting Spark Iceberg Catalog Tests ==="
# Run Spark + Iceberg catalog integration tests
@@ -690,17 +737,21 @@ jobs:
risingwave-catalog-tests:
name: RisingWave Catalog Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -730,7 +781,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
echo "=== Starting RisingWave Catalog Tests ==="
# Run RisingWave catalog integration tests
@@ -761,17 +812,21 @@ jobs:
sts-integration-tests:
name: STS Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -785,6 +840,7 @@ jobs:
- name: Restore python:3 image cache
id: python-image
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: /tmp/python3-image.tar
@@ -824,7 +880,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
echo "=== Starting STS Integration Tests ==="
# Run STS integration tests
@@ -855,17 +911,21 @@ jobs:
lakekeeper-integration-tests:
name: Lakekeeper Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Run go mod tidy
@@ -884,7 +944,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
echo "=== Starting Lakekeeper Integration Tests ==="
# Run Lakekeeper integration tests
@@ -915,17 +975,21 @@ jobs:
unity-catalog-integration-tests:
name: Unity Catalog Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -955,7 +1019,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
docker info
echo "=== Starting Unity Catalog Tests ==="
@@ -988,17 +1052,21 @@ jobs:
lancedb-namespace-tests:
name: LanceDB Namespace Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -1027,7 +1095,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
docker info
echo "=== Starting LanceDB Namespace Tests ==="
@@ -1058,7 +1126,7 @@ jobs:
table-lifecycle-tests:
name: Table Lifecycle Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 40
steps:
@@ -1069,6 +1137,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -1126,7 +1195,7 @@ jobs:
duckdb-lance-tests:
name: DuckDB Lance Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
@@ -1137,10 +1206,14 @@ jobs:
# so do not leave a token in the checkout for it to pick up.
persist-credentials: false
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -1170,7 +1243,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
docker info
echo "=== Starting DuckDB Lance Tests ==="
@@ -1201,7 +1274,7 @@ jobs:
spark-lance-namespace-tests:
name: Spark Lance Namespace Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 40
steps:
@@ -1212,10 +1285,14 @@ jobs:
# so do not leave a token in the checkout for it to pick up.
persist-credentials: false
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -1244,7 +1321,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
docker info
echo "=== Starting Spark Lance Namespace Tests ==="
@@ -1275,7 +1352,7 @@ jobs:
s3-tables-build-verification:
name: S3 Tables Build Verification
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 15
steps:
@@ -1286,6 +1363,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Verify S3 Tables Package Builds
@@ -1340,7 +1418,7 @@ jobs:
s3-tables-fmt-check:
name: S3 Tables Format Check
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 10
steps:
@@ -1351,6 +1429,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Check Go Format
@@ -1391,7 +1470,7 @@ jobs:
s3-tables-vet:
name: S3 Tables Go Vet Check
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 10
steps:
@@ -1402,6 +1481,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Run Go Vet
@@ -0,0 +1,57 @@
name: "S3 Tigris Compatibility Tests"
on:
push:
branches: [ master ]
paths:
- 'weed/s3api/**'
- 'weed/server/filer_server_handlers_*.go'
- 'test/s3/tigris_compat/**'
- '.github/workflows/s3-tigris-compat-tests.yml'
pull_request:
branches: [ master ]
paths:
- 'weed/s3api/**'
- 'weed/server/filer_server_handlers_*.go'
- 'test/s3/tigris_compat/**'
- '.github/workflows/s3-tigris-compat-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
jobs:
s3-tigris-compat-tests:
name: s3-api-compat-tests (Tigris)
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Run s3-api-compat-tests
working-directory: test/s3/tigris_compat
run: make test
- name: Show server log on failure
if: failure()
working-directory: test/s3/tigris_compat
run: tail -200 weed-test.log || true
- name: Upload report
if: always()
uses: actions/upload-artifact@v7
with:
name: s3-tigris-compat-results
path: test/s3/tigris_compat/results.json
if-no-files-found: ignore
retention-days: 7
+48 -29
View File
@@ -25,8 +25,8 @@ on:
- '.github/workflows/s3tests.yml'
concurrency:
group: ${{ github.head_ref }}/s3tests
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -34,7 +34,7 @@ permissions:
jobs:
basic-s3-tests:
name: Basic S3 tests (KV store)
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 15
steps:
- name: Check out code into the Go module directory
@@ -44,12 +44,13 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Set up Python
uses: actions/setup-python@v7
with:
python-version: '3.9'
python-version: '3.10'
- name: Clone s3-tests
run: |
@@ -57,9 +58,9 @@ jobs:
cd s3-tests
sudo apt-get update -qq
sudo apt-get install -y -qq libxml2-dev libxslt1-dev zlib1g-dev
pip install -r requirements.txt
pip install tox
pip install -e .
python3 -m pip install -r requirements.txt
python3 -m pip install tox
python3 -m pip install -e .
- name: Fix S3 tests bucket creation conflicts
run: |
@@ -379,14 +380,21 @@ jobs:
s3tests/functional/test_s3.py::test_copy_object_ifnonematch_good \
s3tests/functional/test_s3.py::test_lifecycle_set \
s3tests/functional/test_s3.py::test_lifecycle_get \
s3tests/functional/test_s3.py::test_lifecycle_set_filter \
s3tests/functional/test_s3.py::test_lifecycle_expiration \
s3tests/functional/test_s3.py::test_lifecyclev2_expiration
s3tests/functional/test_s3.py::test_lifecycle_set_filter
# The expiration tests race the lifecycle worker's meta-log replay;
# under the suite's write flood a shard can trail an object's PUT
# event past the test's fixed sleep window. One retry absorbs it.
lifecycle_expiration() {
tox -- s3tests/functional/test_s3.py::test_lifecycle_expiration \
s3tests/functional/test_s3.py::test_lifecyclev2_expiration
}
lifecycle_expiration || { echo "lifecycle expiration tests flaked; retry 1"; lifecycle_expiration; } || { echo "lifecycle expiration tests flaked; retry 2"; lifecycle_expiration; }
# cleanup() trap handles worker/server kill + data dir wipe.
versioning-tests:
name: S3 Versioning & Object Lock tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 15
steps:
- name: Check out code into the Go module directory
@@ -396,12 +404,13 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Set up Python
uses: actions/setup-python@v7
with:
python-version: '3.9'
python-version: '3.10'
- name: Clone s3-tests
run: |
@@ -409,9 +418,9 @@ jobs:
cd s3-tests
sudo apt-get update -qq
sudo apt-get install -y -qq libxml2-dev libxslt1-dev zlib1g-dev
pip install -r requirements.txt
pip install tox
pip install -e .
python3 -m pip install -r requirements.txt
python3 -m pip install tox
python3 -m pip install -e .
- name: Fix S3 tests bucket creation conflicts
run: |
@@ -556,7 +565,7 @@ jobs:
cors-tests:
name: S3 CORS tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 10
steps:
- name: Check out code into the Go module directory
@@ -566,12 +575,13 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Set up Python
uses: actions/setup-python@v7
with:
python-version: '3.9'
python-version: '3.10'
- name: Clone s3-tests
run: |
@@ -579,9 +589,9 @@ jobs:
cd s3-tests
sudo apt-get update -qq
sudo apt-get install -y -qq libxml2-dev libxslt1-dev zlib1g-dev
pip install -r requirements.txt
pip install tox
pip install -e .
python3 -m pip install -r requirements.txt
python3 -m pip install tox
python3 -m pip install -e .
- name: Run S3 CORS tests
timeout-minutes: 10
@@ -681,7 +691,7 @@ jobs:
copy-tests:
name: SeaweedFS Custom S3 Copy tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 10
steps:
- name: Check out code into the Go module directory
@@ -691,6 +701,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Run SeaweedFS Custom S3 Copy tests
@@ -783,7 +794,7 @@ jobs:
sql-store-tests:
name: Basic S3 tests (SQL store)
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 15
steps:
- name: Check out code into the Go module directory
@@ -793,12 +804,13 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Set up Python
uses: actions/setup-python@v7
with:
python-version: '3.9'
python-version: '3.10'
- name: Clone s3-tests
run: |
@@ -806,9 +818,9 @@ jobs:
cd s3-tests
sudo apt-get update -qq
sudo apt-get install -y -qq libxml2-dev libxslt1-dev zlib1g-dev
pip install -r requirements.txt
pip install tox
pip install -e .
python3 -m pip install -r requirements.txt
python3 -m pip install tox
python3 -m pip install -e .
- name: Run Ceph S3 tests with SQL store
timeout-minutes: 15
@@ -1245,9 +1257,16 @@ jobs:
s3tests/functional/test_s3.py::test_copy_object_ifnonematch_good \
s3tests/functional/test_s3.py::test_lifecycle_set \
s3tests/functional/test_s3.py::test_lifecycle_get \
s3tests/functional/test_s3.py::test_lifecycle_set_filter \
s3tests/functional/test_s3.py::test_lifecycle_expiration \
s3tests/functional/test_s3.py::test_lifecyclev2_expiration
s3tests/functional/test_s3.py::test_lifecycle_set_filter
# The expiration tests race the lifecycle worker's meta-log replay;
# under the suite's write flood a shard can trail an object's PUT
# event past the test's fixed sleep window. One retry absorbs it.
lifecycle_expiration() {
tox -- s3tests/functional/test_s3.py::test_lifecycle_expiration \
s3tests/functional/test_s3.py::test_lifecyclev2_expiration
}
lifecycle_expiration || { echo "lifecycle expiration tests flaked; retry 1"; lifecycle_expiration; } || { echo "lifecycle expiration tests flaked; retry 2"; lifecycle_expiration; }
# cleanup() trap handles worker/server kill + data dir wipe.
+10 -2
View File
@@ -29,7 +29,7 @@ permissions:
jobs:
samba-integration:
name: samba-integration
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 45
steps:
@@ -40,15 +40,23 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Start local Docker registry
run: docker run -d --restart=always -p 5000:5000 --name registry registry:2
run: |
if ! docker start registry 2>/dev/null; then
docker rm -f registry >/dev/null 2>&1 || true
docker run -d --restart=always -p 5000:5000 --name registry registry:2
fi
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4
with:
driver-opts: network=host
- name: Set up Docker Compose
uses: docker/setup-compose-action@v2
- name: Build weed race binary
run: |
cd docker
+4 -3
View File
@@ -17,8 +17,8 @@ on:
- '.github/workflows/sftp-tests.yml'
concurrency:
group: ${{ github.head_ref }}/sftp-tests
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -29,7 +29,7 @@ env:
jobs:
sftp-integration:
name: SFTP Integration Testing
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
@@ -40,6 +40,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Install dependencies
run: |
@@ -15,6 +15,10 @@ on:
- '.github/workflows/spark-integration-tests.yml'
workflow_dispatch:
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
checks: write
@@ -38,12 +42,13 @@ jobs:
with:
java-version: '11'
distribution: 'temurin'
cache: maven
cache: ${{ runner.environment == 'github-hosted' && 'maven' || '' }}
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Build SeaweedFS binary
run: |
+5 -6
View File
@@ -10,12 +10,11 @@ permissions:
contents: write
concurrency:
# Only one chart regeneration per branch at a time; a newer run on the same
# branch cancels an in-flight one so overlapping runs never conflict on
# note/star_history.svg during rebase. Scoped by ref so a manual run on
# another branch can't cancel the daily master update.
# Only one chart regeneration per branch at a time so overlapping runs never
# conflict on note/star_history.svg during rebase. Scoped by ref so a manual
# run on another branch can't cancel the daily master update.
group: star-history-${{ github.ref }}
cancel-in-progress: true
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
jobs:
render:
@@ -33,7 +32,7 @@ jobs:
uses: actions/setup-python@v7
with:
python-version: "3.x"
cache: pip
cache: ${{ runner.environment == 'github-hosted' && 'pip' || '' }}
cache-dependency-path: .github/scripts/star_history.py
- name: Install matplotlib
@@ -6,14 +6,22 @@ on:
paths:
- 'telemetry/**'
- 'weed/telemetry/**'
- 'go.mod'
- 'go.sum'
- '.github/workflows/telemetry-integration.yml'
pull_request:
branches: [ master ]
paths:
- 'telemetry/**'
- 'weed/telemetry/**'
- 'go.mod'
- 'go.sum'
- '.github/workflows/telemetry-integration.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -30,6 +38,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Build telemetry server
run: cd telemetry/server && go build -o telemetry-server .
+5
View File
@@ -8,6 +8,10 @@ on:
branches: [ master ]
paths: ['terraform/**', '.github/workflows/terraform_ci.yml']
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -62,6 +66,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: go.mod
cache: ${{ runner.environment == 'github-hosted' }}
- name: Build weed
run: go build -o "$RUNNER_TEMP/weed" ./weed
@@ -27,6 +27,10 @@ defaults:
run:
working-directory: weed
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
jobs:
awscli-tests:
runs-on: ubuntu-latest
@@ -37,6 +41,7 @@ jobs:
- uses: actions/setup-go@v7
with:
go-version: ^1.26
cache: ${{ runner.environment == 'github-hosted' }}
- name: Build SeaweedFS
run: |
@@ -86,7 +91,11 @@ jobs:
set -e
dd if=/dev/urandom of=generated bs=1M count=32
aws --no-verify-ssl s3 cp --no-progress generated s3://bucket/test-multipart
aws --no-verify-ssl s3 cp --no-progress s3://bucket/test-multipart downloaded
for i in 1 2 3 4 5; do
aws --no-verify-ssl s3 cp --no-progress s3://bucket/test-multipart downloaded && break
if [ "$i" -eq 5 ]; then exit 1; fi
sleep 2
done
diff -q generated downloaded
rm -f generated downloaded
+6 -1
View File
@@ -18,18 +18,23 @@ on:
- 'go.sum'
- '.github/workflows/tls-rotation-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
jobs:
tls-rotation-tests:
name: TLS Rotation Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 10
steps:
- name: Set up Go 1.x
uses: actions/setup-go@v7
with:
cache: ${{ runner.environment == 'github-hosted' }}
go-version: ^1.26
id: go
+8 -4
View File
@@ -11,8 +11,8 @@ on:
- '.github/workflows/tus-tests.yml'
concurrency:
group: ${{ github.head_ref || github.ref }}/tus-tests
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -24,17 +24,21 @@ defaults:
jobs:
tus-integration-tests:
name: TUS Protocol Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -49,7 +53,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
echo "=== Starting TUS Tests ==="
# Run tests with automatic server management
@@ -18,18 +18,23 @@ on:
- 'go.sum'
- '.github/workflows/vacuum-integration-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
jobs:
vacuum-integration-tests:
name: Vacuum Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 15
steps:
- name: Set up Go 1.x
uses: actions/setup-go@v7
with:
cache: ${{ runner.environment == 'github-hosted' }}
go-version: ^1.26
id: go
@@ -21,8 +21,8 @@ on:
- '.github/workflows/volume-server-integration-tests.yml'
concurrency:
group: ${{ github.head_ref || github.ref }}/volume-server-integration-tests
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -33,10 +33,44 @@ env:
SHARD_COUNT: 3
jobs:
build-weed:
name: Build SeaweedFS Binary
runs-on: ubuntu-24.04
timeout-minutes: 15
steps:
- name: Checkout code
uses: actions/checkout@v7
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Build SeaweedFS binary
run: |
cd weed
go build -o weed .
chmod +x weed
./weed version
- name: Package binary
run: tar cf weed-binary.tar weed/weed
- name: Upload binary
uses: actions/upload-artifact@v7
with:
name: weed-binary
if-no-files-found: error
retention-days: 1
path: weed-binary.tar
volume-server-integration-tests:
name: Volume Server Integration Tests (${{ matrix.test-type }} - Shard ${{ matrix.shard }})
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 45
needs: [build-weed]
strategy:
fail-fast: false
matrix:
@@ -51,13 +85,18 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Build SeaweedFS binary
- name: Download binary
uses: actions/download-artifact@v8
with:
name: weed-binary
- name: Restore binary
run: |
cd weed
go build -o weed .
chmod +x weed
./weed version
tar xf weed-binary.tar
chmod +x weed/weed
./weed/weed version
# Dealing the listed tests out one by one keeps the shards even. Bucketing
# them by first letter did not: names cluster, so ^Test[I-S] drew 50 of
+7
View File
@@ -144,3 +144,10 @@ weed_bin
telemetry/server/telemetry-server
.aider*
/seaweed-volume/docs
__pycache__/
/test/s3vectors/weed
/test/s3vectors/weed-vector
/test/s3vectors/weed-vector-cli
/test/s3vectors/weed-worker
/test/s3vectors/weed-volume
.pytest_cache/
+45 -4
View File
@@ -17,7 +17,9 @@ SeaweedFS is a simple and highly scalable distributed file system. There are two
1. to store billions of files!
2. to serve the files fast!
One `weed` binary serves an S3 object store, a POSIX file system, and a lakehouse with S3 Tables, all over the same data. Each blob is one disk read away, capacity grows by starting another volume server, and cloud storage can be cached or tiered transparently. Both read and write operations have O(1) complexity and can run at the full speed supported by the underlying hardware.
One SeaweedFS cluster serves an S3 object store, a POSIX file system, a lakehouse with S3 Tables and a built-in Iceberg REST Catalog, and [vector search with S3 Vector Buckets](#vector-search-with-s3-vector-buckets), all on shared storage. Each blob is one disk read away, capacity grows by starting another volume server, and cloud storage can be cached or tiered transparently. Blob reads and writes have O(1) complexity and can run at the full speed supported by the underlying hardware.
![SeaweedFS feature overview: S3 objects, POSIX files, table buckets with an Iceberg REST Catalog and Lance namespace, and S3 Vector Buckets share one cluster of masters, filers, and volume servers, with cloud caching, tiering, and cross-cluster replication.](note/seaweedfs-features.svg)
- [Download Binaries for different platforms](https://github.com/seaweedfs/seaweedfs/releases/latest)
- [Wiki Documentation](https://github.com/seaweedfs/seaweedfs/wiki)
@@ -40,6 +42,7 @@ Table of Contents
* [Scalable](#scalable)
* [The most complete S3 API](#the-most-complete-s3-api)
* [A data warehouse with S3 Tables](#a-data-warehouse-with-s3-tables)
* [Vector search with S3 Vector Buckets](#vector-search-with-s3-vector-buckets)
* [A fast cache for cloud storage](#a-fast-cache-for-cloud-storage)
* [Active-active replication and more](#active-active-replication-and-more)
* [Architecture](#architecture)
@@ -64,6 +67,8 @@ Download the latest binary from the [releases](https://github.com/seaweedfs/seaw
curl -fsSL https://raw.githubusercontent.com/seaweedfs/seaweedfs/master/install.sh | bash
```
For SeaweedFS Enterprise binaries, pass `--enterprise` to the script: `... | bash -s -- --enterprise`.
Then start a ready-to-use S3 object store:
```bash
@@ -204,12 +209,13 @@ On one laptop, [`weed benchmark`][Benchmarks] writes 1KB files at 15,700 per sec
## The most complete S3 API ##
The S3 gateway implements the object, bucket, S3 Tables, IAM, and STS APIs on one endpoint, so the AWS SDKs and CLI, rclone, restic, Spark, and Trino work unchanged.
The S3 gateway implements the object, bucket, S3 Tables, S3 Vectors, IAM, and STS APIs on one endpoint, so the AWS SDKs and CLI, rclone, restic, Spark, and Trino work unchanged. S3 Vectors uses the `weed-vector` service described [below](#vector-search-with-s3-vector-buckets).
| API | Operations |
| --- | --- |
| S3 bucket and object | 73 |
| S3 Tables | 36 |
| S3 Vectors | 19 |
| IAM | 39 |
| STS | 5 |
@@ -223,15 +229,49 @@ The full operation list is in [Amazon S3 API][AmazonS3API], and [Supported APIs
## A data warehouse with S3 Tables ##
SeaweedFS is a lakehouse in one system. [S3 Table Buckets][S3TableBucket] hold Apache Iceberg tables by default, or [Lance][LanceCatalog] tables for vectors and multimodal data, and the built-in [Iceberg REST Catalog][IcebergCatalog] and Lance namespace serve them directly. There is no Hive Metastore, Glue, or separate catalog service to deploy, secure, and back up.
SeaweedFS is a lakehouse in one system. [S3 Table Buckets][S3TableBucket] hold Apache Iceberg tables by default, or [Lance][LanceCatalog] tables for vectors and multimodal data, and the built-in [Iceberg REST Catalog][IcebergCatalog] and Lance namespace serve them directly. The S3 Tables API and Iceberg REST Catalog share the same namespaces, table metadata, and permissions. There is no Hive Metastore, Glue, or separate catalog service to deploy, secure, and back up.
* Query engines operate on the same tables at the same time: [Spark][SparkIceberg], [Trino][TrinoIceberg], [Dremio][DremioIceberg], [DuckDB][DuckDBIceberg], [Apache Doris][DorisIceberg], [RisingWave][RisingWaveIceberg], ClickHouse, and [LanceDB][LanceDB]. Catalog commits are atomic compare-and-swap, so concurrent writers are safe. [Lakekeeper][Lakekeeper] can front the same storage with STS-vended credentials.
* Query engines operate on the same tables at the same time: [Spark][SparkIceberg], [Trino][TrinoIceberg], [Dremio][DremioIceberg], [DuckDB][DuckDBIceberg], [Apache Doris][DorisIceberg], [RisingWave][RisingWaveIceberg], ClickHouse, and [LanceDB][LanceDB]. Catalog commits are atomic compare-and-swap, so concurrent writers are safe. [Lakekeeper][Lakekeeper] can front the same storage with STS-vended credentials. [OLake Go][OLake] is a high-performance, open-source EL (Extract–Load) platform that bridges operational databases and open lakehouse storage - writing Apache Iceberg tables or Parquet files on Amazon S3.
* [Automated table maintenance][IcebergMaintenance]: compaction, snapshot expiration, orphan file removal, and manifest rewriting, configured per bucket or table through the S3 Tables maintenance APIs, and the same for [Lance][LanceMaintenance].
* IAM at the bucket, namespace, and table level with standard bucket policies, see [S3 Tables Security][S3TablesSecurity].
* A [Hadoop compatible file system][Hadoop] for Spark, Flink, and HBase.
`S3_TABLE_BUCKET=warehouse ./weed mini -dir=./data` brings the whole stack up on a laptop.
## Vector search with S3 Vector Buckets ##
Keep embeddings alongside your objects and tables in the same SeaweedFS cluster. S3 Vector Buckets expose the AWS S3 Vectors API for semantic search, retrieval-augmented generation (RAG), and recommendations, using Lance tables on SeaweedFS storage.
* Create vector buckets and indexes; insert, retrieve, list, and delete vectors with the AWS SDKs or CLI pointed at your S3 endpoint.
* Find the nearest vectors with cosine or Euclidean distance, filter by metadata, and return distances and metadata with the results.
* Control access with `s3vectors:*` IAM actions and vector bucket policies; tag buckets and indexes for organization and access control.
Install the separate vector engine; the default installer installs only `weed`:
```bash
curl -fsSL https://raw.githubusercontent.com/seaweedfs/seaweedfs/master/install.sh | bash -s -- --component vector
```
Release 4.53 provides vector binaries for Linux amd64 and arm64. If your release has no binary for your platform, build from [source](seaweed-vector). From a directory where you want a new checkout:
```bash
git clone https://github.com/seaweedfs/seaweedfs.git
cd seaweedfs
(cd seaweed-vector && cargo build --release -p weed-vector-server)
```
If you already have a checkout, start at its root and skip the clone and `cd seaweedfs` steps.
For Windows source builds, set `PROTOC` to a native `protoc.exe` before running Cargo; the [desktop build workflow](.github/workflows/rust-vector-platforms.yml) shows how to select the vendored executable.
Run the installed `weed-vector` binary alongside the local cluster started with `weed mini` above (or use `seaweed-vector/target/release/weed-vector` after building from source):
```bash
weed-vector --masters=localhost:9333
```
It registers with the master and the S3 gateway discovers it automatically. An AWS S3 Vectors client uses the same S3 endpoint at http://localhost:8333. The vector engine stores its data through the cluster's filers and volume servers, so embeddings share the storage layer's replication and erasure coding. See the [AWS SDK examples](test/s3vectors/s3vectors_test.py) for bucket, index, and query operations.
## A fast cache for cloud storage ##
[Cloud Drive][CloudDrive] mounts a bucket from S3, Google Cloud Storage, Azure, Backblaze B2, Wasabi, Storj, or any S3-compatible store into SeaweedFS and serves it at local speed:
@@ -487,6 +527,7 @@ Your support will be really appreciated by me and other supporters!
[RisingWaveIceberg]: https://github.com/seaweedfs/seaweedfs/wiki/RisingWave-Iceberg-Integration
[LanceDB]: https://github.com/seaweedfs/seaweedfs/wiki/LanceDB-Integration
[Lakekeeper]: https://github.com/seaweedfs/seaweedfs/wiki/Lakekeeper-Iceberg-Integration
[OLake]: https://github.com/seaweedfs/seaweedfs/wiki/OLake-Go-Iceberg-Integration
[IcebergMaintenance]: https://github.com/seaweedfs/seaweedfs/wiki/Iceberg-Table-Maintenance
[LanceMaintenance]: https://github.com/seaweedfs/seaweedfs/wiki/Lance-Maintenance-Worker
[S3TablesSecurity]: https://github.com/seaweedfs/seaweedfs/wiki/S3-Tables-Security
+4
View File
@@ -271,6 +271,10 @@ curl "http://localhost:9333/vol/vacuum?garbageThreshold=0.4"
Vacuuming makes a volume read-only, copies live needles to a new volume, and
swaps it in.
The master runs one vacuum at a time. A request that arrives while the
automatic vacuum or another request is running does nothing and gets
`409 Conflict` with `{"error":"another vacuum is running, try again later"}`.
### Delete a collection
```bash
+1
View File
@@ -169,6 +169,7 @@ seaweed-volume/
- `--memprofile`
- `--compactionMBps` (default 0)
- `--maintenanceMBps` (default 0)
- `--compactionMaxParallelCommits` (default 0)
- `--fileSizeLimitMB` (default 256)
- `--concurrentUploadLimitMB` (default 0)
- `--concurrentDownloadLimitMB` (default 0)
+1 -1
View File
@@ -543,7 +543,7 @@ a language-agnostic gRPC stream that external maintenance workers connect on: th
opens `WorkerStream`, sends `WorkerHello` with the job types it can `detect` and `execute`,
answers `RequestConfigSchema` with a `JobTypeDescriptor`, replies to `RunDetectionRequest`
with `JobProposal`s and to `ExecuteJobRequest` with `JobProgressUpdate`s and `JobCompleted`.
`weed worker -admin=host:23646` is the Go reference implementation of exactly that contract,
`weed worker -admin=host:12535` is the Go reference implementation of exactly that contract,
from outside the admin process.
Nothing in it is Go-specific, and the Rust toolchain is already in the tree.
+12 -1
View File
@@ -33,6 +33,7 @@ ARG TARGETARCH
ARG TAGS
COPY weed-volume-prebuilt/ /prebuilt/
COPY weed-worker-prebuilt/ /prebuilt-worker/
COPY weed-vector-prebuilt/ /prebuilt-vector/
COPY --from=builder /go/src/github.com/seaweedfs/seaweedfs/seaweed-volume /build/seaweed-volume
# seaweed-common is a path dependency of seaweed-volume that lives beside it,
# so the source build below needs it in the same relative position.
@@ -65,11 +66,19 @@ RUN if [ -f "/prebuilt-worker/weed-worker-${TARGETARCH}" ]; then \
echo "No pre-built Rust worker for ${TARGETARCH}" && \
touch /weed-worker; \
fi
# The vector engine carries the same lance tree, so it is pre-built or absent too.
RUN if [ -f "/prebuilt-vector/weed-vector-${TARGETARCH}" ]; then \
echo "Using pre-built vector engine for ${TARGETARCH}" && \
cp "/prebuilt-vector/weed-vector-${TARGETARCH}" /weed-vector; \
else \
echo "No pre-built vector engine for ${TARGETARCH}" && \
touch /weed-vector; \
fi
# Pre-built binaries arrive via GitHub Actions artifacts, which drop the
# executable bit, so the copied file is 0644 and exec fails with "Permission
# denied". Restore it (no-op for the empty placeholders, which stay size 0).
RUN chmod 0755 /weed-volume /weed-worker
RUN chmod 0755 /weed-volume /weed-worker /weed-vector
FROM alpine AS final
LABEL author="Chris Lu"
@@ -78,6 +87,8 @@ COPY --from=builder /go/bin/weed /usr/bin/
COPY --from=rust_builder /weed-volume /usr/bin/weed-volume
# Same for the Rust maintenance worker, which serves Lance table buckets
COPY --from=rust_builder /weed-worker /usr/bin/weed-worker
# And the vector engine behind the S3 Vectors API
COPY --from=rust_builder /weed-vector /usr/bin/weed-vector
RUN mkdir -p /etc/seaweedfs
COPY --from=builder /go/src/github.com/seaweedfs/seaweedfs/docker/filer.toml /etc/seaweedfs/filer.toml
COPY --from=builder /go/src/github.com/seaweedfs/seaweedfs/docker/entrypoint.sh /entrypoint.sh
+1 -1
View File
@@ -3,7 +3,7 @@ FROM golang:1.26 AS builder
RUN apt-get update
RUN apt-get install -y build-essential libsnappy-dev zlib1g-dev libbz2-dev libgflags-dev liblz4-dev libzstd-dev
ARG ROCKSDB_VERSION=v10.10.1
ARG ROCKSDB_VERSION=v11.1.2
ENV ROCKSDB_VERSION=${ROCKSDB_VERSION}
# build RocksDB
+1 -1
View File
@@ -3,7 +3,7 @@ FROM golang:1.26 AS builder
RUN apt-get update
RUN apt-get install -y build-essential libsnappy-dev zlib1g-dev libbz2-dev libgflags-dev liblz4-dev libzstd-dev
ARG ROCKSDB_VERSION=v10.10.1
ARG ROCKSDB_VERSION=v11.1.2
ENV ROCKSDB_VERSION=${ROCKSDB_VERSION}
# build RocksDB
@@ -21,7 +21,7 @@ The testing environment uses **official SeaweedFS commands** and includes:
- **1 Master Server** (port 9333) - Coordinates the cluster with 50MB volume size limit
- **6 Volume Servers** (ports 8080-8085) - Distributed across 2 data centers and 3 racks for diversity
- **1 Filer** (port 8888) - Provides file system interface
- **1 Admin Server** (port 23646) - Detects volumes needing EC and manages workers using official `admin` command
- **1 Admin Server** (port 12535) - Detects volumes needing EC and manages workers using official `admin` command
- **3 EC Workers** - Execute erasure coding tasks using official `worker` command with task-specific working directories
- **1 Load Generator** - Continuously writes and deletes files using SeaweedFS shell commands
- **1 Monitor** - Tracks cluster health and EC progress using shell scripts
@@ -47,7 +47,7 @@ This provides:
- Docker and Docker Compose installed
- GNU Make installed
- At least 4GB RAM available for containers
- Ports 8080-8085, 8888, 9333, 23646 available
- Ports 8080-8085, 8888, 9333, 12535 available
### Start the Environment
@@ -127,7 +127,7 @@ Run `make help` to see all available targets:
|---------|-----|-------------|
| Master UI | http://localhost:9333 | Cluster status and topology |
| Filer | http://localhost:8888 | File operations |
| Admin Server | http://localhost:23646/ | Task management |
| Admin Server | http://localhost:12535/ | Task management |
| Monitor | http://localhost:9999/status | Complete cluster monitoring |
| Volume Servers | http://localhost:8080-8085/status | Individual volume server stats |
+15 -15
View File
@@ -26,7 +26,7 @@ build-and-restart: build ## Build with latest changes and restart services
@echo "2️⃣ Recreating workers to reconnect..."
@docker-compose -f $(COMPOSE_FILE) up -d worker1 worker2 worker3
@echo "✅ All services recreated with latest changes!"
@echo "🌐 Admin UI: http://localhost:23646/"
@echo "🌐 Admin UI: http://localhost:12535/"
@echo "💡 Workers will reconnect to the new admin server"
restart-workers: ## Restart all workers to reconnect to admin server
@@ -64,7 +64,7 @@ start: ## Start the complete SeaweedFS cluster with admin and workers
@echo "✅ Cluster started!"
@echo ""
@echo "📊 Access points:"
@echo " • Admin UI: http://localhost:23646/"
@echo " • Admin UI: http://localhost:12535/"
@echo " • Master UI: http://localhost:9333/"
@echo " • Filer: http://localhost:8888/"
@echo ""
@@ -107,7 +107,7 @@ start-staged: ## Start services in proper order with delays
@echo "✅ All services started!"
@echo ""
@echo "📊 Access points:"
@echo " • Admin UI: http://localhost:23646/"
@echo " • Admin UI: http://localhost:12535/"
@echo " • Master UI: http://localhost:9333/"
@echo " • Filer: http://localhost:8888/"
@echo ""
@@ -136,7 +136,7 @@ status: ## Check the status of all services
@echo "Master:"
@curl -s http://localhost:9333/cluster/status | jq '.IsLeader' 2>/dev/null || echo " ❌ Master not ready"
@echo "Admin:"
@curl -s http://localhost:23646/ | grep -q "Admin" && echo " ✅ Admin ready" || echo " ❌ Admin not ready"
@curl -s http://localhost:12535/ | grep -q "Admin" && echo " ✅ Admin ready" || echo " ❌ Admin not ready"
logs: ## Show logs from all services
@echo "📜 Following logs from all services..."
@@ -157,7 +157,7 @@ master-logs: ## Show logs from master server
admin-ui: ## Open admin UI in browser (macOS)
@echo "🌐 Opening admin UI in browser..."
@open http://localhost:23646/ || echo "💡 Manually open: http://localhost:23646/"
@open http://localhost:12535/ || echo "💡 Manually open: http://localhost:12535/"
test: ## Run integration test to verify task assignment and completion
@echo "🧪 Running Admin-Worker Integration Test"
@@ -166,11 +166,11 @@ test: ## Run integration test to verify task assignment and completion
@echo "1️⃣ Checking cluster health..."
@sleep 5
@curl -s http://localhost:9333/cluster/status | jq '.IsLeader' > /dev/null && echo "✅ Master healthy" || echo "❌ Master not ready"
@curl -s http://localhost:23646/ | grep -q "Admin" && echo "✅ Admin healthy" || echo "❌ Admin not ready"
@curl -s http://localhost:12535/ | grep -q "Admin" && echo "✅ Admin healthy" || echo "❌ Admin not ready"
@echo ""
@echo "2️⃣ Checking worker registration..."
@sleep 10
@echo "💡 Check admin UI for connected workers: http://localhost:23646/"
@echo "💡 Check admin UI for connected workers: http://localhost:12535/"
@echo ""
@echo "3️⃣ Generating load to trigger EC tasks..."
@echo "📝 Creating test files to fill volumes..."
@@ -193,13 +193,13 @@ test: ## Run integration test to verify task assignment and completion
@echo " • EC shards being distributed"
@echo ""
@echo "✅ Integration test setup complete!"
@echo "📊 Monitor progress at: http://localhost:23646/"
@echo "📊 Monitor progress at: http://localhost:12535/"
quick-test: ## Quick verification that core services are running
@echo "⚡ Quick Health Check"
@echo "===================="
@echo "Master: $$(curl -s http://localhost:9333/cluster/status | jq -r '.IsLeader // "not ready"')"
@echo "Admin: $$(curl -s http://localhost:23646/ | grep -q "Admin" && echo "ready" || echo "not ready")"
@echo "Admin: $$(curl -s http://localhost:12535/ | grep -q "Admin" && echo "ready" || echo "not ready")"
@echo "Workers: $$(docker-compose -f $(COMPOSE_FILE) ps worker1 worker2 worker3 | grep -c Up) running"
validate: ## Validate integration test configuration
@@ -223,7 +223,7 @@ demo: start ## Start cluster and run demonstration
@echo " • Load generator (creates files continuously)"
@echo ""
@echo "📊 Watch the process:"
@echo " 1. Visit: http://localhost:23646/"
@echo " 1. Visit: http://localhost:12535/"
@echo " 2. Observe workers connecting"
@echo " 3. Watch tasks being created and assigned"
@echo " 4. See tasks progress from pending → completed"
@@ -244,13 +244,13 @@ vacuum-test: ## Create test data with garbage and verify vacuum detection
@echo ""
@echo "1️⃣ Checking cluster health..."
@curl -s http://localhost:9333/cluster/status | jq '.IsLeader' > /dev/null && echo "✅ Master ready" || (echo "❌ Master not ready. Run 'make start' first." && exit 1)
@curl -s http://localhost:23646/ | grep -q "Admin" && echo "✅ Admin ready" || (echo "❌ Admin not ready. Run 'make start' first." && exit 1)
@curl -s http://localhost:12535/ | grep -q "Admin" && echo "✅ Admin ready" || (echo "❌ Admin not ready. Run 'make start' first." && exit 1)
@echo ""
@echo "2️⃣ Creating test data with garbage..."
@docker-compose -f $(COMPOSE_FILE) exec vacuum-tester go run create_vacuum_test_data.go -files=25 -delete=0.5 -size=200
@echo ""
@echo "3️⃣ Configuration Instructions:"
@echo " Visit: http://localhost:23646/maintenance/config/vacuum"
@echo " Visit: http://localhost:12535/maintenance/config/vacuum"
@echo " Set for testing:"
@echo " • Enable Vacuum Tasks: ✅ Checked"
@echo " • Garbage Threshold: 0.20 (20%)"
@@ -258,7 +258,7 @@ vacuum-test: ## Create test data with garbage and verify vacuum detection
@echo " • Min Volume Age: [0] [Minutes]"
@echo " • Max Concurrent: 2"
@echo ""
@echo "4️⃣ Monitor vacuum tasks at: http://localhost:23646/maintenance"
@echo "4️⃣ Monitor vacuum tasks at: http://localhost:12535/maintenance"
@echo ""
@echo "💡 Use 'make vacuum-status' to check volume garbage ratios"
@@ -335,8 +335,8 @@ vacuum-help: ## Show vacuum testing help and examples
@echo " make vacuum-demo # Full guided demonstration"
@echo ""
@echo "Configuration:"
@echo " Visit: http://localhost:23646/maintenance/config/vacuum"
@echo " Monitor: http://localhost:23646/maintenance"
@echo " Visit: http://localhost:12535/maintenance/config/vacuum"
@echo " Monitor: http://localhost:12535/maintenance"
@echo ""
@echo "Custom Parameters:"
@echo " make vacuum-data FILES=50 DELETE=0.8 SIZE=200"
@@ -243,7 +243,7 @@ func printTestingInstructions() {
fmt.Println("🧪 Testing Instructions:")
fmt.Println()
fmt.Println("1. Configure Vacuum for Testing:")
fmt.Println(" Visit: http://localhost:23646/maintenance/config/vacuum")
fmt.Println(" Visit: http://localhost:12535/maintenance/config/vacuum")
fmt.Println(" Set:")
fmt.Printf(" - Garbage Percentage Threshold: 20 (20%% - lower than default 30)\n")
fmt.Printf(" - Scan Interval: [30] [Seconds] (faster than default)\n")
@@ -253,7 +253,7 @@ func printTestingInstructions() {
fmt.Println()
fmt.Println("2. Monitor Vacuum Tasks:")
fmt.Println(" Visit: http://localhost:23646/maintenance")
fmt.Println(" Visit: http://localhost:12535/maintenance")
fmt.Println(" Watch for vacuum tasks to appear in the queue")
fmt.Println()
@@ -7,7 +7,7 @@ echo ""
# Check if SeaweedFS is running
echo "📋 Checking SeaweedFS status..."
MASTER_URL="${MASTER_HOST:-master:9333}"
ADMIN_URL="${ADMIN_HOST:-admin:23646}"
ADMIN_URL="${ADMIN_HOST:-admin:12535}"
if ! curl -s http://$MASTER_URL/cluster/status > /dev/null; then
echo "❌ SeaweedFS master not running at $MASTER_URL"
@@ -42,7 +42,7 @@ echo ""
# Phase 3: Configure vacuum
echo "⚙️ Phase 3: Vacuum configuration instructions..."
echo " 1. Visit: http://localhost:23646/maintenance/config/vacuum"
echo " 1. Visit: http://localhost:12535/maintenance/config/vacuum"
echo " 2. Set these values for testing:"
echo " - Enable Vacuum Tasks: ✅ Checked"
echo " - Garbage Threshold: 0.30"
@@ -57,7 +57,7 @@ echo ""
# Phase 4: Monitor tasks
echo "🎯 Phase 4: Monitoring vacuum tasks..."
echo " Visit: http://localhost:23646/maintenance"
echo " Visit: http://localhost:12535/maintenance"
echo " You should see vacuum tasks appear within 30 seconds"
echo ""
@@ -93,9 +93,9 @@ echo ""
echo "🎉 Demo Complete!"
echo ""
echo "🔍 Things to check:"
echo " 1. Maintenance Queue: http://localhost:23646/maintenance"
echo " 1. Maintenance Queue: http://localhost:12535/maintenance"
echo " 2. Volume Status: http://localhost:9333/vol/status"
echo " 3. Admin Dashboard: http://localhost:23646"
echo " 3. Admin Dashboard: http://localhost:12535"
echo ""
echo "💡 Next Steps:"
echo " - Try different garbage thresholds (0.10, 0.50, 0.80)"
@@ -114,9 +114,9 @@ services:
admin:
image: chrislusf/seaweedfs:local
ports:
- "23646:23646" # HTTP admin interface (default port)
- "33646:33646" # gRPC worker communication (23646 + 10000)
command: "-v=2 admin -port=23646 -masters=master:9333 -dataDir=/data"
- "12535:12535" # HTTP admin interface (default port)
- "22535:22535" # gRPC worker communication (12535 + 10000)
command: "-v=2 admin -port=12535 -masters=master:9333 -dataDir=/data"
depends_on:
- master
- filer
@@ -127,7 +127,7 @@ services:
worker1:
image: chrislusf/seaweedfs:local
command: "-v=2 worker -admin=admin:23646 -capabilities=erasure_coding,vacuum -maxConcurrent=2"
command: "-v=2 worker -admin=admin:12535 -capabilities=erasure_coding,vacuum -maxConcurrent=2"
depends_on:
- admin
volumes:
@@ -139,7 +139,7 @@ services:
worker2:
image: chrislusf/seaweedfs:local
command: "-v=2 worker -admin=admin:23646 -capabilities=erasure_coding,vacuum -maxConcurrent=2"
command: "-v=2 worker -admin=admin:12535 -capabilities=erasure_coding,vacuum -maxConcurrent=2"
depends_on:
- admin
volumes:
@@ -151,7 +151,7 @@ services:
worker3:
image: chrislusf/seaweedfs:local
command: "-v=2 worker -admin=admin:23646 -capabilities=erasure_coding,vacuum -maxConcurrent=2"
command: "-v=2 worker -admin=admin:12535 -capabilities=erasure_coding,vacuum -maxConcurrent=2"
depends_on:
- admin
volumes:
@@ -199,7 +199,7 @@ services:
curl -s http://master:9333/cluster/status | jq '.IsLeader, .Peers' || echo 'Master not ready';
echo;
echo 'Admin status:';
curl -s http://admin:23646/ | grep -o 'Admin.*Interface' || echo 'Admin not ready';
curl -s http://admin:12535/ | grep -o 'Admin.*Interface' || echo 'Admin not ready';
echo;
echo 'Volume count by server:';
curl -s http://master:9333/vol/status | jq '.Volumes | length' || echo 'Volumes not ready';
@@ -237,4 +237,4 @@ services:
- seaweed_net
environment:
- MASTER_HOST=master:9333
- ADMIN_HOST=admin:23646
- ADMIN_HOST=admin:12535
+2 -2
View File
@@ -23,7 +23,7 @@ else
fi
echo -e "${BLUE}2. Checking if required ports are available...${NC}"
for port in 9333 8080 8081 8082 8083 8084 8085 8888 23646; do
for port in 9333 8080 8081 8082 8083 8084 8085 8888 12535; do
if lsof -i :$port > /dev/null 2>&1; then
echo -e "${YELLOW}⚠️ Port $port is in use${NC}"
else
@@ -41,7 +41,7 @@ else
fi
echo -e "${BLUE}4. Verifying admin server configuration...${NC}"
if docker-compose -f docker-compose-ec-test.yml config | grep -q "admin:23646"; then
if docker-compose -f docker-compose-ec-test.yml config | grep -q "admin:12535"; then
echo -e "${GREEN}✅ Admin server port configuration is correct${NC}"
else
echo -e "${RED}❌ Admin server port configuration is incorrect${NC}"
+8 -10
View File
@@ -3,17 +3,15 @@
# unreachable. See docker/Dockerfile.e2e.
set -e
# Every rewrite starts from the pristine list, so a mirror that just failed does
# not become the pattern the next rewrite has to match.
[ -f /etc/apt/sources.list.orig ] || cp /etc/apt/sources.list /etc/apt/sources.list.orig
for mirror in azure.archive.ubuntu.com archive.ubuntu.com; do
# Any archive host, so this works whether the pristine list came from the
# base image (archive.ubuntu.com) or a CI runner (azure.archive.ubuntu.com).
sed "s|http://[a-z0-9.]*archive\.ubuntu\.com/ubuntu|http://$mirror/ubuntu|g; s|http://security\.ubuntu\.com/ubuntu|http://$mirror/ubuntu|g" \
/etc/apt/sources.list.orig > /etc/apt/sources.list
if apt-get -o Acquire::Retries=3 -o Acquire::http::Timeout=15 update && \
DEBIAN_FRONTEND=noninteractive apt-get -o Acquire::Retries=3 -o Acquire::http::Timeout=15 install -y \
for apt_source in /etc/apt/sources.list /etc/apt/sources.list.d/*.list /etc/apt/sources.list.d/*.sources; do
[ -f "$apt_source" ] || continue
[ -f "$apt_source.orig" ] || cp "$apt_source" "$apt_source.orig"
sed -E "s|(https?://)[a-z0-9.]*archive\.ubuntu\.com/ubuntu|\1$mirror/ubuntu|g; s|(https?://)security\.ubuntu\.com/ubuntu|\1$mirror/ubuntu|g" \
"$apt_source.orig" > "$apt_source"
done
if timeout --kill-after=5s 120s apt-get -o Acquire::Retries=1 -o Acquire::http::Timeout=15 -o Acquire::https::Timeout=15 -o APT::Update::Error-Mode=any update && \
DEBIAN_FRONTEND=noninteractive timeout --kill-after=5s 180s apt-get -o Acquire::Retries=1 -o Acquire::http::Timeout=15 -o Acquire::https::Timeout=15 install -y \
--no-install-recommends --no-install-suggests "$@"; then
apt-get clean
rm -rf /var/lib/apt/lists/*
+21 -8
View File
@@ -64,7 +64,7 @@ case "$1" in
'master')
ARGS="-mdir=/data -volumeSizeLimitMB=1024"
shift
exec /usr/bin/weed -logtostderr=true master $ARGS $@
exec /usr/bin/weed -logtostderr=true master $ARGS "$@"
;;
'volume')
@@ -73,7 +73,7 @@ case "$1" in
ARGS="-dir=/data"
fi
shift
exec /usr/bin/weed -logtostderr=true volume $ARGS $@
exec /usr/bin/weed -logtostderr=true volume $ARGS "$@"
;;
'volume-rust')
@@ -87,7 +87,7 @@ case "$1" in
echo "Use 'volume' for the Go volume server instead." >&2
exit 1
fi
exec /usr/bin/weed-volume $ARGS $@
exec /usr/bin/weed-volume $ARGS "$@"
;;
'worker-rust')
@@ -100,13 +100,26 @@ case "$1" in
exec /usr/bin/weed-worker "$@"
;;
'vector')
shift
if [ ! -s /usr/bin/weed-vector ]; then
echo "Error: the vector engine is not available on this platform ($(uname -m))." >&2
exit 1
fi
# The gateway dials the engine from another container, so not loopback.
if [ -z "$WEED_VECTOR_LISTEN" ] && ! isArgPassed "-listen" "$@"; then
set -- --listen 0.0.0.0:8090 "$@"
fi
exec /usr/bin/weed-vector "$@"
;;
'server')
ARGS="-dir=/data -volume.max=0 -master.volumeSizeLimitMB=1024"
if isArgPassed "-volume.max" "$@"; then
ARGS="-dir=/data -master.volumeSizeLimitMB=1024"
fi
shift
exec /usr/bin/weed -logtostderr=true server $ARGS $@
exec /usr/bin/weed -logtostderr=true server $ARGS "$@"
;;
'mini')
@@ -115,19 +128,19 @@ case "$1" in
ARGS=""
fi
shift
exec /usr/bin/weed -logtostderr=true mini $ARGS $@
exec /usr/bin/weed -logtostderr=true mini $ARGS "$@"
;;
'filer')
ARGS=""
shift
exec /usr/bin/weed -logtostderr=true filer $ARGS $@
exec /usr/bin/weed -logtostderr=true filer $ARGS "$@"
;;
's3')
ARGS="-domainName=$S3_DOMAIN_NAME -key.file=$S3_KEY_FILE -cert.file=$S3_CERT_FILE"
shift
exec /usr/bin/weed -logtostderr=true s3 $ARGS $@
exec /usr/bin/weed -logtostderr=true s3 $ARGS "$@"
;;
'shell')
@@ -137,6 +150,6 @@ case "$1" in
;;
*)
exec /usr/bin/weed $@
exec /usr/bin/weed "$@"
;;
esac
+39
View File
@@ -0,0 +1,39 @@
# Vector buckets locally
Run on Linux containers (amd64 or arm64):
```sh
docker compose up -d
export AWS_ACCESS_KEY_ID=vector-demo
export AWS_SECRET_ACCESS_KEY=vector-demo-secret
export AWS_DEFAULT_REGION=us-east-1
aws --endpoint-url http://localhost:8333 s3vectors create-index \
--vector-bucket-name embeddings --index-name images \
--data-type float32 --dimension 3 --distance-metric cosine
aws --endpoint-url http://localhost:8333 s3vectors put-vectors \
--vector-bucket-name embeddings --index-name images \
--vectors '[{"key":"first","data":{"float32":[1,0,0]}}]'
aws --endpoint-url http://localhost:8333 s3vectors query-vectors \
--vector-bucket-name embeddings --index-name images \
--query-vector '{"float32":[1,0,0]}' --top-k 1
```
The engine registers with the master. Wait for that registration before creating
an index. The Lance worker builds and refreshes ANN indexes as data grows;
queries can scan data before an ANN index is built. The demo's admin UI is at
http://localhost:12535 (user `admin`, password `vector-demo`). Demo credentials
and plaintext cluster connections are for this local example. Production
settings are in the Helm/operator TLS and authentication documentation.
`docker compose down` preserves the named data volume. Add `-v` to discard it.
Override `SEAWEEDFS_IMAGE` to test a newer image. With metrics-enabled engines,
append `--metrics-port=9329 --metrics-ip=0.0.0.0` to the vector command and scrape
`vector:9329/metrics` inside the Compose network.
A native development setup can use `weed mini -vector -vectorBucket=embeddings`
with `weed-vector` beside `weed` or on PATH. `-vector.binary` selects an explicit
binary, and `-vector.metricsPort=9329` enables HTTP readiness and metrics on a
compatible engine. The child inherits the cluster's configuration directory and
stops with mini. Run `weed-worker` separately for ANN maintenance. `weed shell`
provides `s3vectors -operation <Operation> -file request.json` for the same API
operations using trusted cluster credentials.
+34
View File
@@ -0,0 +1,34 @@
services:
seaweedfs:
image: ${SEAWEEDFS_IMAGE:-chrislusf/seaweedfs:4.53}
command: mini -dir=/data -ip=seaweedfs -ip.bind=0.0.0.0 -admin.password=vector-demo -admin.worker.ip=0.0.0.0 -tableBucket=embeddings:LANCE
environment: &environment
WEED_JWT_FILER_SIGNING_KEY: vector-demo-cluster-key
AWS_ACCESS_KEY_ID: vector-demo
AWS_SECRET_ACCESS_KEY: vector-demo-secret
ports:
- "127.0.0.1:8333:8333"
- "127.0.0.1:12535:12535"
volumes:
- data:/data
healthcheck:
test: [CMD-SHELL, "wget -q -O /dev/null http://127.0.0.1:9333/cluster/status"]
interval: 5s
timeout: 3s
retries: 30
vector:
image: ${SEAWEEDFS_IMAGE:-chrislusf/seaweedfs:4.53}
command: vector --listen=vector:8090 --masters=seaweedfs:9333
environment: *environment
depends_on:
seaweedfs:
condition: service_healthy
worker:
image: ${SEAWEEDFS_IMAGE:-chrislusf/seaweedfs:4.53}
command: worker-rust --admin=seaweedfs:12535 --masters=seaweedfs:9333 --namespace=http://seaweedfs:9101
environment: *environment
depends_on:
seaweedfs:
condition: service_healthy
volumes:
data:
+1 -1
View File
@@ -45,7 +45,7 @@ services:
admin:
image: chrislusf/seaweedfs:dev # use a remote dev image
ports:
- 23646:23646
- 12535:12535
command: 'admin -master=master:9333'
depends_on:
- master
Loaded 100 of 1961 files, more files were not shown because too many files have changed in this diff. Show more