Compare commits

...
Author SHA1 Message Date
user.emailandDevin 7ab73b914e dlm: advance the mutation seq on a lost-renewal replay
The replayed lease extension kept the old seq, so InsertBackupLock
dropped it as stale and a promoted backup would have expired the
recovered lock early.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 19:49:33 -07:00
user.emailandDevin f7684c4478 dlm: replay the current token on a lost renewal's retry
A renewal rotates the token server-side before the response is sent. When
that response is lost the client only knows the superseded token: retries
mismatched, the token was cleared, and the holder conflicted with its own
lease until it expired. The lock now remembers the superseded token, so a
retry from the same owner replays the current token, and a superseded
token can still release its own lock.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 19:30:07 -07:00
user.emailandDevin db02eea349 cluster: bound connection acquisition inside the lock attempt
A blocking dial ran under context.Background(), so a caller supplying
grpc.WithBlock() could wait past the attempt window before the RPC
deadline even started. The attempt context now covers both
getOrCreateConnection and the DistributedLock call, and WithGrpcClient
threads it into GrpcDial on both the cached and streaming paths.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 18:42:49 -07:00
user.email dbf2c3d30f cluster: keep the renew token across a failed attempt
A renewal can rotate the token server-side before its response is lost,
and clearing the token on any RPC failure then abandons a live lease the
client can neither renew nor unlock. Keep the token so the next attempt
retries with it: it still validates when the request never landed, and a
definitive rejection clears it through the response's empty RenewToken.
2026-10-26 17:49:33 -07:00
user.email ad73b66200 cluster: bound the whole lock-acquisition window
A single bounded attempt can take most of noLockServerRetryPeriod before
the retry clock even starts, so an unreachable owner stalls acquisition
roughly two attempts past the intended window. Start the clock before
the first attempt and cap each attempt by what remains; contention still
waits it out.
2026-10-26 17:49:18 -07:00
user.email fcc7bd04c6 cluster: bound each DistributedLock attempt
A lock-ring member reachable from the masters but not from this filer
makes one DistributedLock attempt wait out the TCP connect timeout, so a
write whose key hashes to it stalls far past noLockServerRetryPeriod
before failing. Give the attempt a 10s context deadline so the retry
window can take effect; a timed-out attempt keeps no renew token, so the
lock server still grants the key to at most one owner.
2026-10-26 17:38:19 -07:00
Chris LuandDevin a9d6bfc51f test: boto3 s3vectors conformance harness (P1) (#12105)
* vector: boto3 s3vectors conformance harness

test/vectors spins a mini cluster, an s3 gateway with -port.vectors and
-vectors.grpc, and a weed-vector-server, then exercises the AWS client
end to end: bucket and index lifecycle, put/get/list/delete vectors,
and a filtered query asserting distances and metadata.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: run the s3vectors conformance subset

An e2e job in rust-vector-tests: builds weed and weed-vector-server,
then runs test/vectors against a mini cluster plus the vectors gateway.
The workflow now also watches vector.proto, its stubs, and test/vectors.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: fix e2e protoc and codespell findings

The e2e job needed the same vendored-protoc step as the unit-test job —
lance's build scripts locate protoc themselves. Codespell flags the AWS
`$nin` operator literal and the `lits` variable: exempt `nin` in
.codespellrc like the existing word exemptions, rename the variable,
and fold the negation into the IN/NOT IN symbol directly.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: classify catalog errors by code, not message text

* s3: drop the implicit vector namespace on vector-bucket delete

* test: point the vector engine at the gateway lance port

* test: bind the vectors gateway listeners to loopback

* test: make the filter and distance assertions load-bearing

* test: rebuild the harness binaries on every run

* ci: fix the vectors end-to-end job

* ci: run the vector tests on gateway and command changes

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 15:53:50 -07:00
Chris Lu c55b7c5a29 s3: fix CreateIndex metadata panic and ListVectors segmented paging (#12106)
* s3: nil-check metadataConfiguration before reading its keys

* s3: honor maxResults/nextToken in segmented ListVectors

* s3: exclude non-Lance buckets from ListVectorBuckets

* s3: test segmented ListVectors paging token math

* ci: silence codespell on the "$nin" operator and rename lits
2026-10-26 15:40:25 -07:00
Chris LuandDevin dcc94f818d vector: S3 Vectors API gateway (P1) (#12103)
* vector: S3 Vectors API gateway over VectorService

weed/s3api/s3vectors serves the AWS S3 Vectors JSON API: bucket CRUD
maps to S3 Tables table buckets, index CRUD to Lance table records plus
the VectorService CreateIndex/DropIndex RPCs, and the data plane
(Put/Get/List/Delete/QueryVectors) calls the shared service. The AWS
filter document translates to a Lance SQL predicate over the metadata
JSON column; unsupported operators fail loudly rather than overmatch.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: -port.vectors and -vectors.grpc on weed s3

Mirror the Lance server startup: the S3 Vectors API binds its own port
and dials the weed-vector-server gRPC address; both disabled by default.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: resolve index ARNs, authorize index ops, fix catalog fields

- vector buckets declare FormatLance so a new bucket accepts index
  tables instead of defaulting to ICEBERG.
- ListIndexes decodes the `tables` response field; `tableSummaries`
  hid every index.
- resolveIndexARNs honors indexArn alone; handlers no longer treat
  indexName as the only carrier.
- data-plane RPCs fetch the index record first, so callers authorize
  against the index resource and a missing index 404s before touching
  the engine; QueryVectors reports the index's configured metric.
- IndexArn added to the index-level request types.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: serve the vectors port over TLS when configured

When -key.file/-cert.file are set, the vectors listener mirrors the
main API and serves TLS rather than a second plaintext endpoint.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: harden the s3vectors gateway handlers

- $ne no longer shadows the operator: exclusion filters emit <>
  instead of equality.
- a null element in a PutVectors list fails the request instead of
  panicking.
- requests are capped at 32 MiB; distances return via *float32 so an
  exact match still reports 0 when returnDistance is set.
- CreateIndex declares the `vector` namespace first and validates
  dimension (1..=4096) before the catalog write; a failed
  VectorService.CreateIndex rolls the table record back.
- non-filterable metadata keys persist as JSON rather than a
  comma-joined string, and index records carry their createdAt.
- ListVectors honors segmentIndex/segmentCount by slicing the full
  listing in the gateway.
- catalog errors map to AccessDeniedException/ConflictException instead
  of a blanket ValidationException, and malformed ARNs 400 instead of
  slicing.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: /{Operation} routes and vectors-port TLS, conn cleanup

Path-dispatched clients (POST /QueryVectors) reach the same handlers as
X-Amz-Target calls. A failed listener bind now closes the vector gRPC
connection instead of leaving it open with the endpoint disabled.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 14:57:44 -07:00
Chris Lu 74abcb630e ci: retry the timing-sensitive s3 lifecycle expiration tests once (#12073)
* ci: retry the timing-sensitive s3 lifecycle expiration tests once

test_lifecycle_expiration and test_lifecyclev2_expiration assert exact
object counts after fixed sleeps. The lifecycle worker expires objects
via meta-log replay, and under the suite's write flood a shard's replay
can trail an object's PUT event past the test's window (observed on
master and PR runs, SQL-store job only). Split them out of the main
list and allow one retry; each attempt is independent so real
regressions still fail.

* ci: allow a second retry for s3 lifecycle expiration tests
2026-10-26 13:39:40 -07:00
Jay Hemnanianduser.email 1e7afbacf4 s3: list every bucket when auth is off (#12077)
* s3: list every bucket when auth is off

With no identities configured, every request may create, read and
write any bucket, but ListBuckets treated the caller as an
unauthenticated user and returned an empty list, so `aws s3 ls` showed
nothing. List all buckets in that mode, with prefix, max-buckets and
continuation tokens as before. Listing with auth on is unchanged.

* s3: cover the auth-on anonymous ListBuckets case

The no-auth listing change also moves the nil-identity gate; pin the
security-relevant direction that a request without an identity still
sees no buckets once auth enforcement is on.

* s3: verify ListBuckets continuation token reaches the next page

---------

Co-authored-by: user.email <chrislusf@users.noreply.github.com>
2026-10-26 13:38:36 -07:00
Chris LuandDevin 87f15e4d6c vector: weed-vector-server, the VectorService gRPC (P1) (#12102)
* vector: VectorService protobuf contract

The shared RPC boundary between the Go S3 gateway and the Rust vector
engine from the vector design doc: CreateIndex/DropIndex, Put/Get/
List/Delete/Query vectors. A Table reference carries the Lance table id
and an optional pinned version, matching the snapshot discipline queries
are required to keep.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: generated Go stubs for vector.proto

protoc output via weed/pb/Makefile.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: carry return_data through QueryVectors

S3 Vectors QueryVectors honours returnData, but the contract had no
way to ask for or carry vector data in a query response, so the
gateway would need a second GetVectors round-trip per query. Add
return_data to the request and data to QueryResult now, while the
contract is still new.

* vector: pin metadata numbers to double precision in the contract

Metadata arrives as JSON, where numbers carry double precision; say
so on the field so neither side promises integer fidelity past 2^53.

* vector: declare_table on the lance namespace client

CreateIndex needs a way to ask the catalog for a new table's location;
expose POST /v1/table/{id}/declare.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: cargo fmt coverage.rs

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: weed-vector-server, the VectorService gRPC

A tonic server over weed-vector-core: CreateIndex declares the table
through the namespace and writes an empty {key, vector, metadata}
dataset; PutVectors upserts by key (delete + append until merge_insert
is exercised); Get/List/Delete/Query route through the engine's
resolved table, honoring pinned versions and the prefilter default.
Metadata travels as a Utf8 JSON column, proto Struct on the wire.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: stamp the metric in the schema, tighten put validation

CreateIndex now bounds dimension at the S3 Vectors 4096 maximum and
records the distance metric in the vector field's Arrow metadata, so a
later reader recovers the index's configured metric from the table
itself. batch_of checks every vector's data length individually before
flattening — equal totals could still let a short vector borrow floats
from its neighbor — and writes against the dataset's own schema.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: upsert vectors in a single merge commit

PutVectors ran delete-then-append: a failed append left the old keys
deleted, and two concurrent puts could both append a replacement. Use
lance's merge_insert on the key column — one commit, no delete window,
conflicts fail the request instead of duplicating keys.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: read the index's metric; fetch metadata in one take

QueryVectors hardcoded L2 — it now reads the metric the index was
created with from the vector field's metadata. return_metadata fetched
each hit's row one at a time; collect the row ids into a single
take_rows call and join metadata back on the hit key.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: honor return_data in query results

QueryResult grew a data field but query_vectors never populated it.
Fetch the vector column in the same batched take_rows as metadata and
return each hit's stored embedding.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: reject a malformed query as invalid_argument

A zero top_k silently became one result and a dim-mismatched query
vector surfaced as an internal scan error; reject both before querying.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: reject a malformed list next_token

A token that failed to parse silently restarted the listing from
offset zero; surface it as invalid_argument instead.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: answer an empty put without a commit and cover ragged input

An empty vectors list would still run a merge commit; return the
current version instead. Pin the ragged-vector check with a test.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: keep non_filterable_metadata_keys on the table schema

create_index dropped the declared filter policy. Persist it as JSON on
the metadata field so the index carries which keys a filter may touch;
enforcing the list belongs to the gateway that translates the request
filter into Lance SQL.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: reject duplicate keys in one put

A repeated key for a new row committed both copies, and for an existing
key the merge failure surfaced as internal. Reject duplicates up front
as invalid_argument.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: report open failures as not_found or unavailable

A catalog 404 or a missing dataset/version is not a client mistake, and a
catalog outage is retryable: mapping every open error to invalid_argument
made both indistinguishable from a bad request.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: page list_vectors when max_results is unset

A proto3 client that omits max_results sends 0, which used to limit the
scan to zero rows and return no token.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: drop stray empty service.rs.tmp

* vector: rename datas map to data_map

codespell flags datas as a misspelling of data; use a distinct map name.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 13:32:34 -07:00
Chris LuandDevin d771a91571 vector: VectorService gRPC contract (P1) (#12101)
* vector: VectorService protobuf contract

The shared RPC boundary between the Go S3 gateway and the Rust vector
engine from the vector design doc: CreateIndex/DropIndex, Put/Get/
List/Delete/Query vectors. A Table reference carries the Lance table id
and an optional pinned version, matching the snapshot discipline queries
are required to keep.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: generated Go stubs for vector.proto

protoc output via weed/pb/Makefile.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: carry return_data through QueryVectors

S3 Vectors QueryVectors honours returnData, but the contract had no
way to ask for or carry vector data in a query response, so the
gateway would need a second GetVectors round-trip per query. Add
return_data to the request and data to QueryResult now, while the
contract is still new.

* vector: pin metadata numbers to double precision in the contract

Metadata arrives as JSON, where numbers carry double precision; say
so on the field so neither side promises integer fidelity past 2^53.

* vector: clarify the query filter is raw Lance SQL over one metadata column

The doc said 'metadata columns' but the table stores all metadata in a
single JSON column named metadata; the gateway translates S3 Vectors
filter expressions onto it.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 13:26:11 -07:00
Chris Lu 6b379badbb filer: shut down test filers' meta-log buffers, fixing the 32-bit OOM (#12104)
* filer: shut down test filers' meta-log buffers, fixing the 32-bit OOM

Each log_buffer.NewLogBuffer preallocates five 8 MiB windows and spawns
loopFlush/loopInterval goroutines that keep them reachable. The
weed/filer tests built ~70 filers and aggregators that almost never
called ShutdownLogBuffer, pinning ~2.8 GB plus live churn, and the
linux/386 job ran out of its ~4 GiB address space inside
newSealedBuffers.

- MetaAggregator.Shutdown now retires MetaLogBuffer too: the "aggr"
  buffer's loops previously outlived the aggregator.
- newTestFiler and newTestFilerWithStubStore register t.Cleanup
  shutdowns; the one-off filer/aggregator literals defer theirs.

* filer: wait for the aggregated meta-log buffer to stop in Shutdown

ShutdownLogBuffer only signals the stop; loopFlush and its pooled windows
exit asynchronously. Match Filer.Shutdown's local-buffer sequence so the
aggregator returns with the buffer's memory already released.

* filer: cancel test filers' flush context before buffer shutdown

The stub-store helpers build real NewFiler instances, so the queued
shutdown flush retries appendToFile every ~737ms under flushCtx, which
nothing ever cancelled — the buffer and its pool stayed alive past each
test. Cancelling first lets the flush drop the window and WaitForShutdown
confirms loopFlush exited.

* filer: wait for test log buffers to drain after shutdown

The stub-flush buffers cannot hang, but without WaitForShutdown their
loopFlush goroutines release the pooled windows asynchronously — the
memory this PR means to free is still counted as live while the next
test allocates.
2026-10-26 13:24:48 -07:00
Chris Lu aff18b2336 vector: restore spike CLI flags lost in the #12098 rebase (#12099)
* vector: --vector takes hyphenated floats again

* vector: --vector-column/--key-column on the get subcommand

* vector: --vector-column/--key-column on describe and coverage

* vector: surface describe resolve errors when column flags are set

* vector: cover the restored CLI flags with parse tests
2026-10-26 12:24:01 -07:00
Chris LuandDevin 72ac424198 filer: deliver meta-log events under their stream position, not their stamp (#12084)
* filer: deliver meta-log events under their stream position, not their stamp

A mutation's event timestamp is stamped before the filer store write so
the entry and its event share a commit time, while the log-buffer append
happens after the write. A write delayed behind later-stamped mutations
keeps its early stamp in the payload while AddDataToBuffer bumps
LogEntry.TsNs to keep the stream monotone - so subscribers could
receive events whose payload TsNs arrives out of order.

Consumers treat the delivered TsNs as a stream position (cursors,
watermarks, replay windows). The s3 lifecycle daily-run drains events in
order and pins its per-shard cursor behind the first not-yet-due match;
an event delivered with a stale stamp after the cursor already passed it
is filtered out forever and its pending match silently never fires -
the flaky test_lifecyclev2_expiration signature of exactly one object of
a same-rule pair surviving, with errors=0 on every pass.

Normalize event.TsNs to logEntry.TsNs at delivery (and for the
skeleton-filtered heartbeat) so delivered timestamps are monotone by
construction. The commit-time stamp remains available on the entries
themselves via CommitTsNs.

* pb,filer: stamp metadata events with a mutation timestamp

TsNs on the wire now carries the delivery stream position, so the
mutation's commit stamp needs its own field. Add mutation_ts_ns to
SubscribeMetadataResponse and set it when the event is created.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: carry the mutation stamp through event delivery

Delivery rewrites TsNs to the stream position; the payload stamp survives
in MutationTsNs, backfilled from TsNs for payloads written before the
field existed. The notification callback forwards it to the wire message.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* pb: normalize chunk-replay timestamps like eachLogEntryFn

ReadLogFileRefs bypasses the server decode path, so a delayed append
still arrived under its stale payload stamp and could hide behind a
subscriber cursor. Deliver the stream position and preserve the stamp in
MutationTsNs.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: order replay and tombstones by the mutation stamp

Replay treated the delivered TsNs as the commit time, so a peer write
delayed in its own log could overwrite a newer local entry. Compare and
stamp with MutationTsNs, falling back to TsNs for older peers.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* remote.sync: carry the mutation stamp through reshaped renames

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: test replay ordering on the mutation stamp

* filer: tombstone remote deletes by delivery position, order by commit stamp

* filer: keep the peer commit stamp when restamping aggregated events

* pb: keep the log cursor on the entry position past callback restamps

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 12:18:40 -07:00
Chris Lu 68565fa883 admin: bump a-h/templ to v0.3.1100 to match @latest codegen (#12100)
CI installs the templ CLI with `@latest`, which now generates code
calling templruntime.NewAttributeExpression — a symbol added after the
v0.3.1070 runtime pinned in go.mod, breaking `make install` builds on
clean runners. Bump the runtime to v0.3.1100 and regenerate.
2026-10-26 12:03:41 -07:00
Chris LuandDevin 3abf8b2572 filer,pb,mount: keep per-chunk cipher keys out of logs (#12085)
* filer: format ChunkView without its cipher key

readChunkSliceAt logs a failed chunk fetch with %+v, which printed the
per-chunk AES-256-GCM key into error-level logs on every failed read.
Give ChunkView a String() that omits CipherKey so %v and %+v stay safe.

* pb: scrub cipher keys when an entry must be logged whole

ScrubCipherKeys clones an Entry with chunk CipherKey fields cleared, so
a warning that needs the full entry still cannot leak the keys.

* log entry identifiers instead of whole metadata messages

Several log lines formatted whole protobuf messages — SubscribeMetadataResponse,
EventNotification, Entry-carrying requests — whose chunks include cipher_key
under -encryptVolumeData, so the keys escaped into logs (some at default
verbosity). Log the directory, entry name and event timestamp instead; where a
chunk list was printed, print file ids via FileIdsString.

* log queue: unwrap SubscribeMetadataResponse and never print raw protos

The fallback printed any message with %+v, so a SubscribeMetadataResponse
carrying entries would still put cipher keys in the log. Unwrap its event
notifications (batched events included) and reduce unknown messages to
their type name.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* remote sync logs: name the event's real path

Rename targets logged under resp.Directory while the entry lives under
NewParentPath; route them through MetadataEventTargetDirectory. Sync
failure logs printed only the directory, so same-dir failures shared an
identifier — log the source and destination paths instead.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* log queue: log the entry's destination path on moves

A cross-directory rename printed "file" -> "file" under the source
path, hiding where the entry landed. Use NewParentPath so the log
shows the destination directory next to the new name.

* admin: install the go.mod-pinned templ, not @latest

install-templ fetched templ@latest, which now emits
templruntime.NewAttributeExpression - a symbol absent from the
runtime pinned in go.mod - so every fresh 'make' regenerated
uncompilable _templ.go files. Install the go.mod version and
reinstall when the binary on PATH reports a different one.

* admin: drop stray Makefile change belonging to the templ pin PR

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* notification: join the move destination path in log queue output

A NewParentPath of "/" printed //name for moves into the root;
path.Join keeps a single leading slash.

* log queue: skip a nil event notification instead of logging an empty move

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* pb: tolerate nil chunk entries while scrubbing cipher keys for logs

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* subscription and backup failure logs name the event's file

The process/marshal failure logs printed only the event directory and
timestamp, so a failed sync left operators guessing which file failed.
Use MetadataEventSourceFullPath/MetadataEventTargetFullPath so each
failure names the source and target paths without printing the entries.

* replication: identify entry-less events without printing the proto

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: scrub cipher keys from the bucket-metadata debug dump

buildBucketMetadata marshals the whole filer entry to JSON for a V(3)
log line; a chunked entry would emit each chunk's cipher_key base64.
Marshal the scrubbed copy instead.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 11:52:56 -07:00
Chris Lu c27d7fe5a4 s3: keep transient object-entry errors out of ErrObjectNotFound (#12083)
* s3: keep transient object-entry errors out of ErrObjectNotFound

getObjectEntry wrapped every fetch error as ErrObjectNotFound, so a
transient filer failure (busy filer, ctx deadline, transport blip) was
indistinguishable from a real miss. LifecycleDelete resolves
ErrObjectNotFound permanently as NOOP_RESOLVED, so one flaky lookup at
dispatch time silently skipped the object's expiration - errors=0, no
retry, no log. This matches the flaky s3tests
test_lifecyclev2_expiration signature: exactly one object of a
same-rule pair surviving past its window.

Only collapse genuine misses (filer not-found, version/latest-version
sentinels); everything else propagates so the worker retries it as
RETRY_LATER instead of resolving it.

Also seed bucketConfigCache in the retention test server and teach the
fake filer to inject per-path lookup errors.

* s3: keep invalid version ids on the not-found path in getObjectEntry

A malformed versionId used to collapse into ErrObjectNotFound like every
other fetch error, so the handlers answered NoSuchKey. With transient
errors now propagated, an invalid id would surface as InternalError;
fold it back into the miss classification.

* filer_pb: keep Unavailable out of LookupEntry's not-found normalization

* s3: reject Unavailable in isObjectEntryMiss before message matching

* s3: keep transient fallback errors out of latest-version resolution

* filer_pb: limit LookupEntry's not-found text match to miss statuses

* s3: limit isFilerNotFound text matching to miss statuses
2026-10-26 11:52:15 -07:00
Chris LuandDevin 8498ad716b vector: spike CLI, reference-comparison suite, CI (P0 phase 3) (#12098)
* vector: project _distance and _rowid explicitly

Lance warns that a knn projection without _distance will stop
auto-including it; spell both scoring columns out.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: reference-comparison tests

The spike's exit gate: assert the engine's answers equal a bare
scan().nearest() on the same pinned version — over a fresh index, over
unindexed fragments, over a two-segment index after optimize_indices,
and after a delete. A version-pinned handle querying through
maintenance sees only its own version.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: weed-vector spike CLI

describe / coverage / query / get against a namespace-registered table
or a plain dataset URI, so the spike runs against a real SeaweedFS Lance
table bucket and against a local fixture alike. Flag and env names
mirror weed-worker.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: rust-vector-tests workflow

Build and test seaweed-vector in release on pushes and PRs touching it
or its path-dependency trees, mirroring rust-worker-tests.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: derive Serialize for the spike CLI's JSON output

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: --postfilter on the spike CLI

Track the Query field rename; the flag stays an explicit opt-out of
prefiltering for comparison runs.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: assert the pinned query's answers

The maintenance test passed even with zero pinned hits: an empty result
satisfies all() trivially. Assert the hit count and compare the pinned
answers against the reference on the pinned handle.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: vector tests also watch plugin.proto

weed-vector-core path-depends on weed-lance-worker, which compiles
weed/pb/plugin.proto through seaweed-worker-core; a proto-only change
would skip these checks and could break this workspace.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: describe prints layout and coverage

The describe doc comment promised the vector layout and index coverage;
print both, and propagate count_rows errors instead of printing 0.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 11:48:46 -07:00
eda4b9efdd vector: ANN coverage introspection + prefilter (P0 phase 2) (#12097)
* vector: share the test dataset builder

The coverage tests need the same tempdir Lance datasets the query tests
build; move them into a cfg(test) helper module.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: prefilter the metadata filter

prefilter=false post-filters the top-K rows, which can return fewer than
k results and never sees unindexed rows beyond the nearest cut. Expose
the choice so the S3 Vectors layer can honour AWS filter semantics.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: index coverage introspection

coverage() unions each index's fragment bitmaps against the manifest's
fragment list to report which fragments a vector index covers, which it
has not reached, and how many rows the flat pass still has to scan.
Tests pin the behaviour the spike is for: rows appended after an index
build stay in the top-K, and a version-pinned dataset never sees the
next commit.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: default filtered queries to prefilter

postfilter=false is now the normal path: the metadata filter cuts the
eligible rows before ranking, so a query returns k hits whenever k
eligible rows exist. Lance's post-filter remains reachable through
Query::postfilter for comparison runs.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: intersect index bitmaps with the live manifest

A bitmap records fragments the index covered when built; a fragment the
manifest dropped afterwards stayed in the reported coverage.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: keep bitmap-less indices in coverage

An index whose metadata has no fragment bitmap was dropped from the
report entirely, indistinguishable from no index at all. Count its
segments and mark its coverage unknown instead.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: count live rows in unindexed coverage

physical_rows counts deleted rows and can be absent in the manifest;
sum FileFragment::count_rows for the unindexed fragments instead.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: error on a null key in a selected key column

* vector: compare filtered ANN and flat hits as sets

* vector: coverage reports the checked-out version

* vector: project _distance and _rowid explicitly

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: chrislusf <chris@seaweedfs.com>
2026-10-26 11:45:07 -07:00
Chris LuandDevin 7388d0d4fc vector: seaweed-vector workspace + flat/indexed top-K engine (P0 phase 1) (#12096)
* vector: scaffold the seaweed-vector workspace

Standalone cargo workspace for the vector query engine, mirroring the
seaweed-worker layout: own lockfile, same lint policy, path dependencies
across the two trees since there is no common root manifest.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: resolve a table's vector layout

VectorTable::resolve finds the vector column (explicit name, else the
column named vector, else the sole fixed-size float list), the optional
key column, and the metadata columns a query can project.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: flat and indexed top-k, key lookup

query::flat runs an exact scan (use_index(false)); query::ann lets Lance
split the query between index segments and a flat pass over fragments no
index covers yet, so rows written after the last index build stay
eligible. Both honour a metadata filter and return the raw _distance in
the query's metric.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: reject a non-utf8 or non-identifier key column

An explicit --key-column naming a non-UTF8 field made hits() silently
report row ids as keys, and get() interpolated the name into a SQL
filter verbatim. Require Utf8 at resolve and in hits(), and require a
plain identifier before building the IN predicate.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: fall back to row id on a null key

keys.value() on a null slot fabricates an empty-string key; a row with
no key reports its row id instead, matching the no-key-column case.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: quote the key column in the IN filter

A Utf8 key column named e.g. document-key made the predicate parse as
subtraction, and an identifier interpolated unquoted is injectable.
Backtick-quote it and double its inner backticks; double quotes are
string literals in the filter dialect, not identifiers.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* vector: project only the columns a hit needs

The knn projection pulled every metadata column — including other
vector columns — although Hit returns none of them. Filters still
evaluate off the scan path, so restrict the output to the key and the
scoring columns.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 10:13:05 -07:00
Chris LuandDevin 6f8f8e7079 admin: install the go.mod-pinned templ, not @latest (#12090)
* admin: install the go.mod-pinned templ, not @latest

install-templ fetched templ@latest, which now emits
templruntime.NewAttributeExpression - a symbol absent from the
runtime pinned in go.mod - so every fresh 'make' regenerated
uncompilable _templ.go files. Install the go.mod version and
reinstall when the binary on PATH reports a different one.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* admin: invoke templ from Go's install dir, not PATH

A templ binary earlier on PATH than GOBIN still shadowed the pinned
compiler after install-templ ran, so generate kept emitting code for
the wrong runtime. Resolve GOBIN (or GOPATH/bin) and call the pinned
binary by absolute path for install checks, generate, watch, and fmt.

* admin: regenerate templ output with the pinned v0.3.1070

The checked-in _templ.go files were stamped v0.3.1020, older than the
go.mod pin. Regenerate so the committed code matches the compiler the
runtime links against.

* admin: resolve templ under the first GOPATH entry

go install drops binaries in the first GOPATH element's bin, so a
multi-element GOPATH left TEMPL pointing at a path that does not exist.
Split on the OS list separator so Windows GOPATHs resolve too.

* admin: install pinned templ before fmt and pattern-rule generates

fmt and a direct %_templ.go rebuild invoked TEMPL without ensuring the
pinned binary exists; an order-only prerequisite keeps the check cheap
without forcing regeneration.

* admin: keep spaced GOPATH/templ paths intact in the Makefile

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 09:30:33 -07:00
6ebc055a94 master, volume, filer: start on a read-only mounted directory and serve existing data read-only (#12081)
* master, volume, filer: start on a read-only mounted directory and serve existing data read-only

A read-only mount was treated as writable (permission bits only), so every
service failed on its first write: the master panicked writing raft conf.tmp,
the filer exited opening the leveldb LOCK, and the volume server skipped every
volume while still offering the directory as free capacity.

- util.IsReadOnlyMount: access(W_OK) == EROFS.
- volume: on a read-only -dir take the existing O_RDONLY/noWriteOrDelete load,
  index .idx in memory when no .sdx exists, offer no free slots (same branches
  as isDiskSpaceLow), skip the vid stamp there.
- master: on a read-only -mdir run raft from a private copy and refuse new
  volume ids.
- filer: leveldb2/leveldb3 set their existing ReadOnly option.

* volume: skip read-only mounts when picking EC shard copy targets

A shard copy landing on a read-only mount fails at write time; offer
only writable locations like the other placement paths do.

* volume: mirror the read-only mount handling in the Rust volume server

The .dat open only fell back to read-only on EACCES, so a real read-only
mount (EROFS) still failed the load. The index side already degrades
once no_write_or_delete is set. Add is_read_only_mount via access(W_OK)
== EROFS, mark DiskLocation.read_only_mount, and mirror the Go checks:
no free slots, heartbeat max = used, no vid-stamp writes, no EC shard
copy targets, and a read-only index dir forces the volume read-only.

* volume: write vid stamps while only the index dir is read-only

* filer: skip the write-bit check when a leveldb store opens read-only

* filer: serve reads of buckets with no leveldb3 dir on a read-only store

* volume: derive a stable dir uuid when a read-only mount cannot store one

* master: keep the read-only raft copy across restarts so votes survive

* volume: index a missing .idx in memory on a read-only mount

* volume: release the .idx handle after an in-memory read-only load

* volume: mount EC shards read-only when their index dir is on a read-only mount

* volume: mark the index dir read-only before the tail repair opens .idx

* volume: write vid stamps while only the index dir is read-only (Rust)

* volume: derive a stable dir uuid and report no free slots on a read-only mount (Rust)

* volume: index a missing .idx in memory on a read-only mount (Rust)

* volume: refuse an EC copy whose shard owner is on a read-only disk

* master: key the read-only raft copy by the absolute -mdir

Two masters sharing the same relative -mdir text from different working
directories would have collided on one temp copy and shared raft state.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: mix device and host into the read-only dir uuid

Deriving the uuid from the path alone let independent servers at the
same mount point report one uuid, which RegisterUuids rejects. Hash in
the device id and hostname so restarts stay stable but disks differ.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: vic6996-web <320442181+vic6996-web@users.noreply.github.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 09:28:49 -07:00
dependabot[bot] 0214f9b34e build(deps): bump github.com/pkg/sftp from 1.13.11 to 1.13.12 (#12086)
Bumps [github.com/pkg/sftp](https://github.com/pkg/sftp) from 1.13.11 to 1.13.12.
- [Release notes](https://github.com/pkg/sftp/releases)
- [Commits](https://github.com/pkg/sftp/compare/v1.13.11...v1.13.12)

---
updated-dependencies:
- dependency-name: github.com/pkg/sftp
  dependency-version: 1.13.12
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-26 09:24:50 -07:00
dependabot[bot] b7d3c8a5a6 build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azcore from 1.23.2 to 1.23.3 (#12087)
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azcore

Bumps [github.com/Azure/azure-sdk-for-go/sdk/azcore](https://github.com/Azure/azure-sdk-for-go) from 1.23.2 to 1.23.3.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.23.2...sdk/azcore/v1.23.3)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/azcore
  dependency-version: 1.23.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-26 09:24:36 -07:00
dependabot[bot] 74161edb7a build(deps): bump github.com/getsentry/sentry-go from 0.49.0 to 0.50.0 (#12089)
Bumps [github.com/getsentry/sentry-go](https://github.com/getsentry/sentry-go) from 0.49.0 to 0.50.0.
- [Release notes](https://github.com/getsentry/sentry-go/releases)
- [Changelog](https://github.com/getsentry/sentry-go/blob/master/CHANGELOG.md)
- [Commits](https://github.com/getsentry/sentry-go/compare/v0.49.0...v0.50.0)

---
updated-dependencies:
- dependency-name: github.com/getsentry/sentry-go
  dependency-version: 0.50.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-26 09:24:25 -07:00
dependabot[bot] 07cfe2d5e2 build(deps): bump github.com/redis/go-redis/v9 from 9.22.0 to 9.23.0 (#12088)
Bumps [github.com/redis/go-redis/v9](https://github.com/redis/go-redis) from 9.22.0 to 9.23.0.
- [Release notes](https://github.com/redis/go-redis/releases)
- [Changelog](https://github.com/redis/go-redis/blob/master/RELEASE-NOTES.md)
- [Commits](https://github.com/redis/go-redis/compare/v9.22.0...v9.23.0)

---
updated-dependencies:
- dependency-name: github.com/redis/go-redis/v9
  dependency-version: 9.23.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-26 09:24:09 -07:00
dependabot[bot] 692423d641 build(deps): bump github.com/ThreeDotsLabs/watermill from 1.5.2 to 1.5.3 (#12091)
Bumps [github.com/ThreeDotsLabs/watermill](https://github.com/ThreeDotsLabs/watermill) from 1.5.2 to 1.5.3.
- [Release notes](https://github.com/ThreeDotsLabs/watermill/releases)
- [Changelog](https://github.com/ThreeDotsLabs/watermill/blob/master/RELEASE-PROCEDURE.md)
- [Commits](https://github.com/ThreeDotsLabs/watermill/compare/v1.5.2...v1.5.3)

---
updated-dependencies:
- dependency-name: github.com/ThreeDotsLabs/watermill
  dependency-version: 1.5.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-26 09:23:51 -07:00
dependabot[bot] 2ffe61189f build(deps): bump actions/github-script from 7 to 9 (#12095)
Bumps [actions/github-script](https://github.com/actions/github-script) from 7 to 9.
- [Release notes](https://github.com/actions/github-script/releases)
- [Commits](https://github.com/actions/github-script/compare/v7...v9)

---
updated-dependencies:
- dependency-name: actions/github-script
  dependency-version: '9'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-26 09:23:32 -07:00
dependabot[bot] 41c33bdb21 build(deps): bump github/codeql-action from 4.39.0 to 4.39.1 (#12092)
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.39.0 to 4.39.1.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/v4.39.0...v4.39.1)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-version: 4.39.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-26 09:20:41 -07:00
d010664634 build(deps): bump docker/login-action from 4.6.0 to 4.7.0 (#12094)
* build(deps): bump docker/login-action from 4.6.0 to 4.7.0

Bumps [docker/login-action](https://github.com/docker/login-action) from 4.6.0 to 4.7.0.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](https://github.com/docker/login-action/compare/v4.6.0...v4.7.0)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-version: 4.7.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* ci: correct stale version comments on SHA-pinned docker actions

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 09:20:15 -07:00
dependabot[bot] 66e8211821 build(deps): bump actions/download-artifact from 7 to 8 (#12093)
Bumps [actions/download-artifact](https://github.com/actions/download-artifact) from 7 to 8.
- [Release notes](https://github.com/actions/download-artifact/releases)
- [Commits](https://github.com/actions/download-artifact/compare/v7...v8)

---
updated-dependencies:
- dependency-name: actions/download-artifact
  dependency-version: '8'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-26 09:19:02 -07:00
a235473105 [Volume] Preserve request semantics in replica proxy reads (#12075)
* fix 12074

* volume: shuffle a copy of cached replica locations for proxy reads

LookupVolumeId serves cached location slices shared between lookups; shuffling in place races with concurrent proxied reads of the same volume.

* seaweed-volume: forward inbound method and surface lookup failure in proxy reads

Mirror the Go handler: proxy_request used to always send GET, turning a client HEAD into a replica download, and a failed master lookup returned 404 like missing data. Forward the inbound method and answer 500 for lookup failures in proxy mode.

* volume: keep master lookup error detail out of the proxy error response

* volume: bound the proxied-request wait in the method-preservation test

* seaweed-volume: answer 404 for missing volumes in proxy-mode lookups

Mirror Go's LookupVolumeId: a per-volume error in the master's lookup
reply (e.g. a missing volume) means "no locations" and answers 404
downstream, not a lookup failure that proxy mode answers 500.

* volume: guard nil lookup result in proxy path

A master response that omits the requested vid leaves
LookupVolumeId returning a nil *LookupResult with a nil error;
dereferencing it panics the read handler.

* seaweed-volume: answered lookup errors are missing data, not failures

A master reply carrying a per-volume error with HTTP 404 (volume not
found) now yields empty locations like Go's LookupResult, so proxy mode
answers 404. Other failure statuses such as 503 during master warmup
remain lookup failures that proxy mode answers 500.

* seaweed-volume: drop hop-by-hop headers from replica proxy reads

Go's inbound request drops Host, Content-Length and Transfer-Encoding
into dedicated fields, so its forwarded header set never carries them.
hyper keeps them; forwarding a client's Content-Length with no body
stalls the replica's read, and forwarding its Host misnames the target.

* volume: bound the proxy master lookup to the request context

* volume: drop hop-by-hop and Connection-nominated headers from proxy reads

* seaweed-volume: drop Connection-nominated headers from proxy reads

* volume: skip the proxy lookup error when the inbound request is gone

* volume: drop proxy-specific headers from replica proxy reads

* seaweed-volume: drop proxy-specific headers from replica proxy reads

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: drop hop-by-hop headers from replica proxy responses

* seaweed-volume: drop hop-by-hop headers from replica proxy responses

* seaweed-volume: a failed lookup status without an error body is still a failure

* volume: do not re-proxy an already-proxied read for a missing volume

A proxied request that reaches a server which still lacks the volume was
looked up and forwarded again, so a stale location answer could bounce
the read between two servers until the client gave up. The Rust volume
server's proxy_missing_volume already returns 404 for proxied=true;
mirror it here.

* seaweed-volume: replace stale proxied params in the forwarded query

Go builds the proxied URL with query.Set(reqIsProxied), which replaces
every proxied param the client sent. Appending after a client-supplied
proxied=false left the replica's first-value lookup reading the stale
marker, defeating the loop guard.

---------

Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 08:13:34 -07:00
Chris LuandDevin dfd9ecab05 mount: resume metadata-log replay where a dropped stream stopped (#12079)
* pb: checkpoint metadata-log replay progress per settled entry

A chunk-mode subscriber's resume cursor only advanced after the whole
LogFileRefs batch drained, so a stream dropped mid-replay resubscribed
from the original stale position and re-read the same log files while
live traffic kept appending - under a flapping filer the replay window
never shrank, and each reconnect re-listed and re-fetched the same
backlog.

ReadLogFileRefs now reports every entry the merge has settled -
delivered or filter-skipped - and FollowMetadata checkpoints StartTsNs
on it, so a reconnect resumes where the previous stream stopped and the
backlog drains across restarts. The peer aggregator wires the same
progress into its resubscribe/offset position.

* pb: emit a replay checkpoint only after the merge passes its timestamp

Checkpointing an entry's timestamp the moment it settles could strand a
same-timestamp entry still queued on another filer's stream: resuming is
exclusive, so a reconnect would skip the unsettled half of the tie. The
merge now emits a timestamp only once a later entry proves every tied
entry was delivered (and once at clean EOF).

* pb+filer: never regress the replay cursor below the settled checkpoint

The drain's final assignment took the last settled entry's own timestamp,
which can lag the log position progressFn just checkpointed - a skewed or
legacy entry rewinds option.StartTsNs (and the peer's lastTsNs), so the next
drain or a resubscribe re-reads the settled prefix. Guard both assignments
so the cursor only moves forward.

* pb: cover the resubscribe path of replay checkpoints

A drain that aborts mid-replay leaves StartTsNs at the last checkpoint
the merge passed, and the next subscribe resumes there instead of
re-reading the whole ref set. The test drives the real
makeSubscribeMetadataFunc path: refs, a mid-batch apply failure, then a
resubscribe whose SinceNs must carry the checkpoint.

* pb: emit the tail replay checkpoint only for a settled timestamp

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 08:06:15 -07:00
0d721fccee fix(volume): keep preallocated growth above -volume.minFreeSpace (#12067)
* fix(volume): keep preallocated growth above -volume.minFreeSpace

When free space fell below -volume.minFreeSpace, every volume went
read-only and the master kept growing new volumes. With preallocation on,
each new volume reserved its full size, driving the disk further below the
floor (15 Oct incident: 66 empty volumes reserved about 1980 GiB).

Two causes, both in weed/storage/store.go:

- MaybeAdjustVolumeMax sized a disk's volume slots from raw free space, so
  space inside the floor was counted as available. It now counts only free
  space above the floor.
- addVolume trusted the once-a-minute low-disk flag. A preallocated volume
  is now refused when the disk is low now, or when its reservation would
  not fit above the floor. Volumes without preallocation, including merges
  that copy existing data, are not affected.

addVolume runs under a store-level lock, so two concurrent creates cannot
both pass the room check against the same free space.

* volume: mirror headroom guard in the rust store

find_free_location only trusted the once-a-probe is_disk_space_low flag,
so a burst of preallocated creates could still run the disk under
-volume.minFreeSpace between probes, and maybe_adjust_volume_max sized
slots from raw free space. Adds MinFreeSpace::is_low/available_space to
match Go and applies both checks the same way. store.write() already
serializes add_volume, so no extra lock is needed.

* volume: read rust headroom tests from a fake disk

The floor arithmetic used a live free-space reading, so unrelated disk
activity could flip a test between pass and fail. Store.disk_stats_of is
now the same seam Go uses (diskStatusOf): the growth-loop and floor tests
run against a controlled disk, mirroring the Go suite one for one.

* volume: hold back create metadata from the headroom check

An exact-fit preallocation left free space exactly at the floor, so the
.vif and .idx writes that follow a create could dip under it and get the
location flagged read-only at the next probe. Reserve room for .vif on
the data filesystem, and for .idx growth only when dir_idx shares it.

* volume: mirror the metadata allowance in the rust store

Same rule as the Go side: charge .vif against the checked filesystem and
.idx only when dir_idx shares it, and grow the exact-fit test's fake
disk by the per-create cost.

* volume: skip the reservation check where create cannot preallocate

* volume: mirror the platform preallocation gate in the rust store

* volume.copy: count the preallocation reservation in the destination space check

The copy path picked a location with room for the copied .dat, then reserved
the master's full volume size limit on it. A volume smaller than the limit
could land on a disk whose free space above -volume.minFreeSpace fit the
copy but not the reservation, driving the disk under the floor the same way
the growth loop did.

Fetch the preallocation setting before choosing the location and require
room for the reservation itself, only where CreateVolumeFile actually
reserves blocks.

* seaweed-volume: store min-free-space-adjusted available space like Go

Go's DiskLocation.AvailableSpace holds free space above the
-volume.minFreeSpace floor; the Rust probe cached raw free instead, so the
VolumeCopy location check could accept a disk already inside the floor.
Store the same adjusted figure and cover it with a test.

* seaweed-volume: reserve the copied .dat like Go's CreateVolumeFile

VolumeCopy called set_len for the reservation, which only grows a sparse
file — no blocks are held, so a disk with room for the copied bytes but
not the full reservation only fails mid-copy. preallocate_file fallocates
KEEP_SIZE on Linux like Go, reserving the blocks up front; off Linux it
is a no-op, matching PREALLOCATE_SUPPORTED.

* seaweed-volume: charge a copy's destination pick for the .dat reservation

Go's VolumeCopy now sizes its free-location check against the master's
volume size limit when the platform preallocates, not just the copied
bytes. The Rust pick only measured the copied bytes, so a destination
could accept a copy whose reservation then dips under the floor. The
config fetch moves ahead of the pick (same order as Go) and feeds both
the space check and the reservation itself.

* volume.copy: charge the picked location's free space under volumeCreateLock

The copy path picked a destination off the once-a-minute AvailableSpace
probe and released the decision long before the .dat reservation landed.
Two copies could pass the same cached headroom and together drive the disk
under -volume.minFreeSpace.

ReserveFreeLocation serializes the pick under volumeCreateLock, verifies
the disk still has live room for the reservation, and charges it against
the location's cached free space until the next probe refreshes it.

* seaweed-volume: mirror the copy pick's live-check and headroom charge

Go's VolumeCopy now picks through Store.ReserveFreeLocation: live disk
stats under the store write lock, then the reservation charged against the
location's cached free space. The Rust plan_copy_destination gains the
same store.reserve_free_location so two copies cannot spend the same
headroom before the first .dat reservation lands.

* volume.copy: saturate the free-space charge against a live-space drop

The two statfs reads bracketing a pick can race a large external write; a
plain subtraction would underflow and leave the location looking roomy
until the next probe.

* volume.copy: release the headroom reservation when a copy does not land

ReserveFreeLocation charges the picked location's cached free space, but
a failed copy never returned it, so the headroom stayed booked until the
next disk-space probe and blocked retries. ReleaseFreeLocation refunds
the charge; the copy's cleanup defer now registers with the reservation
so every failure path (existing-replica delete, .note write, stream send)
releases it, including the post-transfer mount/validate failures covered
by the outer defer.

* seaweed-volume: mirror the copy headroom release on failed copies

release_free_location refunds reserve_free_location's charge. The copy
returns it when the plan's existing-replica delete fails, when the .note
write fails, and in cleanup_failed_copy, so a failed copy stops holding
the headroom until the next disk-space probe.

* volume.copy: keep pending copy headroom out of probe readings

ReleaseFreeLocation refunded the reservation into AvailableSpace even when
a CheckDiskSpace probe had already rebased the cache on live stats — the
probe's reading still shows the in-flight copy's bytes as free, so the
refund double-counted headroom and two copies could spend the same space,
driving the disk under -volume.minFreeSpace.

Track the outstanding charge in DiskLocation.ReservedSpace instead: probes
subtract it so they cannot re-spend it, a failed copy removes only its own
charge, and a landed copy commits it — the charge and the new usage cancel
out. hasRoomForNewVolume respects pending headroom too, so a preallocating
create cannot overrun an in-flight copy.

* seaweed-volume: mirror pending copy headroom across disk probes

reserved_space tracks the in-flight copy's charge separately from the
cached free space: check_disk_space subtracts it, release removes only the
failed copy's charge, a landed copy commits it, and has_room_for_new_volume
respects it — matching the Go side.

* volume.copy: settle a copy's headroom as its bytes land on disk

The pending charge held the copy's full reservation until commit, but
written and fallocated bytes already show up in statfs, so the live-space
check counted them twice. A slow copy could then block other copies and
volume growth despite free space above -volume.minFreeSpace.

Settle the charge instead: commit the bytes CreateVolumeFile actually
reserved (a failed fallocate leaves nothing committed), then settle each
file's write progress as it lands. The failure paths now release only the
unsettled remainder; allocatedSize is exported for the .dat's st_blocks.

* seaweed-volume: mirror copy headroom settling as bytes land

reserved_space keeps only the promised bytes statfs cannot see yet:
preallocate_dat settles whatever blocks the filesystem actually reserved
(0 when fallocate cannot), each copied file's progress settles the rest as
it lands, and a failed copy releases only the unsettled remainder —
matching the Go side.

* volume.copy: release a failed copy's headroom charge only once

Both defers fired on the same in-closure error: the inner released the
unsettled charge and the outer released it again, driving ReservedSpace
negative (an atomic.Uint64 underflow) and inflating AvailableSpace —
the location would then fail every later headroom check, or worse,
pass them with fabricated space. Zero the charge once it is released;
the Rust mirror already releases via swap(0) for the same reason.

* volume.copy: write into the preallocated .dat instead of staging over it

The .dat stream staged into .copying and renamed over the file
CreateVolumeFile had just fallocated, so the copy's transient footprint
was preallocate+dat — more than the reservation the pick just checked —
and the reservation itself was discarded at rename. Writing in place
reuses the fallocated extent, keeps the peak at the reserved maximum,
and leaves the copied volume preallocated like a created one.

* seaweed-volume: mirror the in-place write into the preallocated .dat

The .dat copy opened the just-fallocated file with truncate, freeing its
blocks at open and spending the headroom again as the stream re-grew it.
Append mode preserves them, matching the Go side.

---------

Co-authored-by: chrislusf <chrislusf@gmail.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
2026-10-26 08:03:25 -07:00
b5af82409f fix: volume.fsck -volumeId purges live chunks referenced from outside the volume's bucket (#12076)
* fix: volume.fsck -volumeId purges live chunks referenced from outside the volume's bucket

-volumeId narrowed the filer read to <bucketsPath>/<collection> when
the selected volumes shared one bucketed collection (#9347). A filer
started with -collection, or an fs.configure rule, stores files from
any path in that collection, so the narrowed read misses their
references and reports live chunks as orphans; with
-reallyDeleteFromVolume they are deleted. #11794 already made
-collection read from the root for this reason. Do the same for
-volumeId: the flag keeps selecting which volumes are checked and
purged, and the filer is always read from "/".

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

* test: volume.fsck keeps chunks referenced outside the volume's bucket

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
2026-10-26 07:58:43 -07:00
Nguyễn Lộc Phúcanduser.email 937633a77d iceberg: fix CREATE TABLE rejecting a partition on the first column (#12072)
* iceberg: fix CREATE TABLE rejecting a partition on the first column

Apache Spark number the columns of a new
table from 0 and leave assigning fresh IDs to the catalog, so a table
partitioned or sorted by its first column arrives with source-id 0. The
create-table request decoded its partition spec and write order into
iceberg-go's bound PartitionSpec and SortOrder, which require positive
source IDs, so every such CREATE TABLE, and CTAS through stage-create,
was refused with "Malformed request: Invalid request body".

Decode both as iceberg.UnboundPartitionSpec and table.UnboundSortOrder,
as iceberg-go documents for catalog create-table requests;
newTableMetadata already resolves the placeholders through the request
schema by name and assigns fresh IDs. A spec or order that does not
resolve against the schema is now answered with 400 instead of 500, and
the decode error message carries the underlying reason.

Tests check that Spark's request bodies resolve to the same metadata as
1-based placeholders, that the handler accepts them, that unresolvable
source IDs get 400, and add a Spark integration test that partitions by
the first column.

* iceberg: gofmt the create-table source-id tests

---------

Co-authored-by: user.email <chrislusf@users.noreply.github.com>
2026-10-26 07:54:10 -07:00
a6be3c89c2 filer: support nanosecond timestamps for conditional delete (#11339) (#12071)
* filer: support nanosecond timestamps for conditional delete (#11339)

* shell: fs.verify prune carries the entry mtime in nanoseconds

IfNotModifiedAfter now resolves to sub-second precision on the filer,
so pass the looked-up mtime_ns along with mtime; the re-lookup check
catches same-second rewrites that share the second, and the test mock
mirrors the filer's seconds/nanoseconds cutoff.

* shell: volume.fsck guards empty-directory deletes with mtime nanoseconds

The looked-up mtime_ns joins the seconds in IfNotModifiedAfter so a
directory rewritten within the same second is not dropped; the stub
filer mirrors the filer's scale heuristic and the race fixtures use a
realistic mtime so the nanosecond branch engages.

* filer: cover the seconds guard and equality boundary

Test second-resolution cutoffs on both sides of the entry's second and
the exact-equality nanosecond guard that still deletes.

* filer: compare the nanosecond guard as a time, not UnixNano

UnixNano overflows for mtimes outside 1678-2262, letting a far-future
entry slip past the guard; time.After compares exactly at every epoch.
The seconds branch keeps Unix() so a same-second cutoff still deletes.

* filer_pb: add explicit if_not_modified_after_ns to DeleteEntryRequest

* filer: prefer explicit nanoseconds guard, keep if_not_modified_after as seconds

* shell: send seconds and nanoseconds delete cutoffs, skip unexpressible mtimes

* filer: merge conditional-delete guards into one nanosecond cutoff

DeleteEntryMetaAndData carried both the seconds and nanoseconds guards
even though the precedence between them is fixed, so every caller passed
a dead parameter. The gRPC handler now folds if_not_modified_after into
nanoseconds at the boundary, where (sec+1)*1e9-1 keeps the whole-second
cutoff exact, and the filer takes a single ifNotModifiedAfterNs.

* filer: hold the conditional-delete cutoff as a time, not int64 nanoseconds

A seconds-only guard at or past year 2262 saturated to MaxInt64
nanoseconds (~2262), so a delete whose cutoff matched a far-future mtime
was skipped while the request still reported success, and any entry
between 2262 and the real cutoff survived pruning. The merged cutoff is
now a time.Time, where a whole-second guard maps to the last nanosecond
of its second for any int64 second.

---------

Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-26 07:51:33 -07:00
b85281f0da s3: group ListMultipartUploads by delimiter into CommonPrefixes (#12069)
* s3: group ListMultipartUploads by delimiter into CommonPrefixes

The delimiter was echoed but ignored, so every upload was listed. Keys
that continue past the delimiter after the prefix are now listed once as
a common prefix, counted toward max-uploads like an upload.

* s3: bound tracked multipart common prefixes to the page

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: omit upload id marker when a page ends on a common prefix

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 06:10:09 -07:00
b7b571d469 webdav: copy a file onto a collection URL without losing the collection (#12068)
* webdav: copy a file onto a collection URL without losing the collection

An overwriting COPY of a file to dir/ removed dir, then OpenFile refused
the name ending in / and the request failed with 403, leaving neither.
Drop the trailing slash from the destination when the source is a file.

* webdav: cover file-over-collection COPY through the real handler

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* webdav: keep a root destination out of the slash trim

A file COPY or MOVE onto / would rewrite the Destination to an empty
path, turning the existing refusal into x/net's bad-gateway error; leave
the root untouched so RemoveAll refuses it as before.

---------

Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 06:08:24 -07:00
github-actions[bot] 96f310dec8 docs: regenerate star history chart 2026-10-26 08:02:43 +00:00
Chris Lu ffa87c25fd ci: fix aarch64/amd64 musl rust builds on ubuntu-24.04 (#12070)
The 24.04 bump broke build-rust-binaries: aws-lc-sys C objects compile against noble's glibc 2.39 headers, which redirect sscanf/strtol to the C23 __isoc23_* variants musl does not provide, failing the link.

Same fix as the enterprise build on seaweedfs/artifactory:
- arm64: keep aarch64-linux-gnu-gcc, link a tiny forwarding shim that
  resolves __isoc23_sscanf/__isoc23_strtol to musl's vsscanf/strtol.
- amd64: point CC at musl-gcc so C deps compile against musl headers.
2026-10-25 21:09:37 -07:00
github-actions[bot] 153ab668c9 4.51 2026-10-25 19:39:07 +00:00
Chris Lu 6dd6ad512c iam/oidc: drop single-use web-identity tokens and the jti replay cache (#12065)
* iam/oidc: drop single-use web-identity tokens and the jti replay cache

Reverts the single-use semantics added in #11936 (and the #12017
refactor): AssumeRoleWithWebIdentity consumed each web-identity
token's jti into a bounded replay cache, so a second exchange of the
same token was rejected until exp + skew.

AWS STS accepts a web-identity token repeatedly until it expires,
and Kubernetes workload identity depends on that: the kubelet keeps
a projected ServiceAccount token until ~80% of its lifetime, so pod
restarts, co-located containers sharing the token file, repeated CLI
invocations, and pre-rotation SDK refreshes all re-present the same
jti and were refused on 4.50.

The replay machinery also carried a real maintenance cost — a
bounded concurrent cache, fingerprint-keyed entries for jti-less
tokens, adopt/merge handoffs so provider rebuilds could not reopen
consumed tokens, and provider-reuse bookkeeping in the refresh path
that existed only to preserve that state. Removing it restores AWS
semantics and deletes the whole apparatus.

Refs #12063. Alternative to the per-provider opt-out in #12064.

* iam: reuse unchanged OIDC providers on refresh

refreshOIDCProvidersFrom rebuilt every managed provider on any store
change, so an unrelated write discarded each provider's JWKS cache and
an unreachable issuer could then fail exchanges the cached keys would
have allowed. Keep a per-record provider cache keyed on the record
content fingerprint and only rebuild when the record changes.

* iam/oidc: require exp in ValidateToken

Dropping the single-use check also removed the implicit bound on
expiry-less tokens: a signed web-identity token without exp could
mint STS sessions indefinitely. Require the exp claim, matching AWS
AssumeRoleWithWebIdentity which rejects tokens without one.

* iam/oidc: bound idle connections on the provider transport

A provider replaced on refresh drops its http.Transport without a
close; give it an IdleConnTimeout so pooled connections and their
goroutines cannot linger.

* s3api: cover web-identity token reuse over HTTP

Sign one token and exchange that exact string twice, directly covering
the Kubernetes projected-token reuse this change restores.

* test(iam): refresh reuses an unchanged record's provider

Covers the provider-reuse cache: an unrelated record's refresh keeps a
provider's JWKS state, a changed record rebuilds it.
2026-10-25 12:04:59 -07:00
515380c349 s3: answer GET and HEAD of a delete marker as S3 does (#12051)
* s3: answer GET and HEAD of a delete marker as S3 does

A delete marker as the current version now returns 404 NoSuchKey with
x-amz-delete-marker: true and the marker's x-amz-version-id. Asking for the
marker by versionId returns 405 MethodNotAllowed with the same headers and
Last-Modified, instead of 404 NoSuchKey.

* s3: check the tag-conditioned read grant before answering a delete marker

The marker branch returned before the entry-based policy recheck the rest
of the read path performs, so an s3:ExistingObjectTag-conditioned Deny did
not apply to the delete-marker response and its version headers.

* s3: send Allow: DELETE with the delete-marker 405

RFC 9110 15.5.6 requires a 405 answer to list the methods the resource
supports; S3 sends allow: DELETE for a delete marker named by versionId.

* s3: keep delete-marker headers on conditional-read misses

If-Match or If-Unmodified-Since against a current delete marker short-
circuited to a bare NoSuchKey. Carry the resolved marker through the
conditional result so the 404 keeps x-amz-delete-marker and
x-amz-version-id, matching the unconditional read.

* s3: exercise delete-marker responses through GET and HEAD handlers

* s3: read marker version id from the entry before the caller's guess

A delete marker written outside the normal PUT path can miss the
version-id attribute while the v_<id> file name still names it; the
new x-amz-version-id header would then report "null". Reuse
versionIdFromEntry so both the handler and conditional-miss paths
emit the real id.

* s3: answer 405 for versionId=null on a suspended-versioning marker

The null marker lives in .versions while getSpecificObjectVersion only
checks the plain path, so GET/HEAD ?versionId=null answered
NoSuchVersion. On that miss, look up the null marker and answer it
like any other marker named by versionId.

* s3: propagate null-marker lookup failures instead of answering 404

A transient filer error reading .versions/v_null was swallowed as
"no marker", so GET/HEAD answered NoSuchVersion for a live read
failure. NotFound and absent entries still mean no marker; anything
else is InternalError now.

---------

Co-authored-by: Chris Lu <chris.lu@gmail.com>
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
2026-10-25 12:02:19 -07:00
Chris LuandDevin 9620e7ebfe filer: auto-enable per-path commit records when a peer has its own store (#12062)
* filer: enable per-path commit records only when a peer has its own store

Since seaweedfs#11698 every metadata change writes a per-path commit
record into the filer store (an MpTs key, plus an MpDl key on deletes),
and the records stay after the path is deleted. They order the events a
filer replays from peer filers that keep their own store, since replay
only runs when peerSignature != f.Signature.

Filers that all share one store (one Redis, MySQL, TiKV, ...) never
replay each other's events, so the records are never read, yet each
change still costs one or two extra store writes and the key count grows
with every path ever changed — unbounded growth for in-memory stores.

This turns the records off by default and latches them on the first time
a peer reports a different store signature — no option needed. The latch
is persisted in the store (MpRep) so it survives restarts and converges
filers sharing a store even before they discover the peer themselves.
While off, a filer keeps an MpOff marker so enabling later bumps the
records' generation (MpGen, folded into each record key) past the
unrecorded window: a record older than a change made while off cannot
let an older replayed peer event win. Ordering for unrecorded changes
falls back to the entry's mtime, the same as entries written before the
records existed.

Supersedes the manual option approach in seaweedfs#12060.

* filer: keep delete tombstones while records off, harden enable path

Review on the first commit:

- A delete committed while the records were off left no entry and no
  usable tombstone: after enabling, an older replayed create could
  resurrect the path. Deletes and rename sources are now always recorded
  under the generation-0 MpDl key — they are never fenced by a
  generation bump, and they cannot go stale because every delete is
  recorded. Creates/updates still write nothing while off.

- bumpPathCommitGen could regress MpGen under concurrent bumpers on a
  shared store. It now loops: each pass writes only a generation higher
  than any it has seen, so concurrent bumps converge upward.

- doSubscribeToOneFiler proceeded into replay when
  enablePathCommitRecords failed, running the whole session without
  ordering. It now aborts the subscribe so the reconnect retries with
  the records on.

* filer: bind pending commit-record repairs to their generation

A failed MpTs write kept in metaTsPending could land after a shared-store
generation bump, writing a pre-bump timestamp as a current-generation
record and overriding newer unrecorded state. Each pending change record
now carries the generation it failed under: repairs write it only while
that generation is current, and ordering reads ignore it otherwise.
Pending delete records need no binding — MpDl keys are never fenced.

* filer: never move the local records' generation backward

The converge loop in bumpPathCommitGen seeded from zero and both it and
syncPathCommitGen stored unconditionally: a stale writer could leave the
stored generation below what this filer already adopted, and the next
sync or bump then walked the local value back — orphaning records it
wrote under the higher generation. The bump now seeds from the local
generation and every store only moves it forward.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: name a records generation after the off marker it consumes

Two filers consuming the same off marker each counted one past the
stored generation, so the second could store N+2 after the first
stored N+1 and write records under a key the other could not read
until the next check. The bump now takes the marker's own timestamp —
filers racing on one marker land on one value — a marker older than
the stored generation is cleaned as a leftover, and a stored value
that already passed the bump is adopted instead of counted past. A
newly enabling filer whose marker was already consumed names its own
off window with the current time, so its unrecorded period is still
fenced.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: tell a fresh off marker from one the generation already consumed

A marker whose timestamp reads at or below the stored generation was
dropped as leftover, but a new off period can also read that old — the
off filer's clock lagging, or a host clock moving back — and dropping
it kept records from before that period live, so stale replay could
win.

The stored generation now names the marker it was bumped for: a marker
the store still shows is consumed unless it is that one, however old
it reads, and only the consumed marker itself is cleared without a
bump. The generation trails in the value, so a whole-slice read still
yields it on stores written before the pair was stored. Enabling
fences the filer's own off window with the current time instead of a
marker a sharer may already have consumed.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: cover racing off-marker consumers settling on one generation

* filer: adopt the stored generation, not the consumed marker it names

A filer syncing with no off marker present read the value's first field —
the consumed marker's timestamp — so a restart after a lagging-clock off
period under-counted the generation and kept records the bump dropped.

* filer: re-check the off marker before consuming it

A sync whose marker read ran behind a later off period's consume cycle
still bumped for its stale read: the stored generation already covered a
different marker, so the bump overwrote the consumed name, counted the
generation past records a newer consumption meant to keep, and cleared
whatever marker was live by then. Each pass now re-reads the marker and
only bumps or clears while it is still the one the pass saw.

* filer: keep a failed generation read from resetting the records' generation

The final generation read ignored errors and then stored whatever gen
held — possibly zero — so a transient store failure could hide records
the current generation wrote. Unexpected read errors now return before
the local generation changes.

Also cover a sync whose marker read runs behind another consumer's
delete: it must adopt the stored generation, not bump for the stale read.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: fail a bump that cannot verify the stored generation

A generation read that failed mid-bump was treated as absent: the
write could land below a generation a racing bumper had stored, or
re-store a marker that bumper had already consumed. Read failures
now abort the pass like every other store error on the path.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: cover generation reads that fail instead of returning a value

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: carry the commit timestamp on the entry itself

The per-path MpTs record only exists because a commit timestamp had
nowhere else to live. Stamping commit_ts_ns on the entry before its
store write - and reusing that stamp as the emitted event's TsNs -
makes the entry its own commit record: replay and bootstrap traversal
order on it with zero extra writes for creates and updates.

Deletes still need a durable key since a removed entry cannot carry a
timestamp, so MpDl tombstones stay (one write per delete, growth
bounded by delete history). Entries predating the field fall back to
mtime, matching what peers and old records already assume.

This replaces the MpTs/MpOff/MpGen/MpRep machinery entirely: shared-
store filers write nothing extra, and dedicated-store replication
keeps last-writer-wins ordering plus tombstone coverage for deletes
and ancestor drops.

* filer: keep a directory update replay from tombstoning its subtree

A same-path update wrote a delete record before the delete-and-reinsert,
so a directory metadata change read as a subtree delete: a later peer
event for a child older than the update was then skipped even though
nobody deleted the children, leaving files missing on this filer. Only
file updates take the tombstone now — the re-inserted directory entry's
own commit stamp orders the path itself.

* filer: order unstamped entries by their recorded commit before mtime

Entries written before stamping existed have no CommitTsNs, so the
ordering check fell back to the client-settable mtime and dropped the
MpTs record those versions wrote: an entry committed at event time 300
with mtime 100 lost to a delayed peer event at 200 after upgrade. Read
the legacy record for unstamped entries in localCommitTsNs, CommitTsNs,
and the traverse stamp (which also stopped falling back to mtime,
regressing entry_ts_ns to 0).

* filer: release queued rename event stamps when the rename fails

Each moved entry carried an in-flight stamp its queued event would
adopt at emit time. When the move succeeded but a later child move or
the commit failed, the handler rolled back without emitting the queued
events or releasing their stamps, pinning the delivered and flushed
watermarks below the aborted rename forever. Release every queued
stamp on both failure exits in both rename handlers.

* filer: apply a directory update replay with an in-place upsert

Deleting the stored entry before inserting its replacement left a
window where the directory had no entry and no ordering record — a
tombstone cannot stand in for it without reading as a subtree delete
that suppresses the children's own records. A stale peer event accepted
in that gap could replay an older update or delete and sweep the
directory's children. Upsert the replacement instead: the entry keeps
its recorded commit order across the whole update, and a directory has
no chunks or hard links for a delete to release.

* filer: update an existing directory entry on replay instead of upserting

InsertEntry is not an upsert on SQL stores configured without upsert
support — PostgreSQL with enableUpsert=false issues a plain INSERT and
its duplicate-key error does not match the MySQL-only fallback, so a
replay updating a present directory failed on every retry. Pick
UpdateEntry when the entry exists and InsertEntry when it is absent;
UpdateEntry touches no row that is not there.

* filer: detect a shared-store delete racing a replayed directory update

UpdateEntry reports success even when the row was deleted between the
find and the update — the UPDATE changes zero rows and the replayed
change is dropped while the subscription offset advances. Re-read the
entry after the update and retry the write, re-checking the recorded
commit order each pass so a newer peer change wins instead of being
resurrected over.

* filer: declare commit_ts_ns in field-number order

Field 6 sits between extended and hard_link_id; the declaration now
reads in the same order. Wire format unchanged. Also syncs the Java
client's checked-in copy of filer.proto, which the pb Makefile copies
on gen.

* filer: option to skip delete tombstones entirely

filer.options.deleteTombstones=false stops writing MpDl records on
deletes - a single filer or shared-store cluster never replays peer
events, so the tombstones are dead weight. The tradeoff is explicit:
if a dedicated-store peer does replicate, a stale event can resurrect
a deleted path.

* filer: drop delete tombstones entirely

Deletes no longer write MpDl records, and the pending-repair machinery
they needed is gone with them: per-mutation metadata overhead is now
zero for every store topology. Ordering information lives only on the
entry itself via commit_ts_ns.

The accepted tradeoff: a dedicated-store peer replaying an event older
than a delete can resurrect the path, and a kept directory no longer
fences stale child creates. Recorded tombstones and MpTs records left
by older builds are still read, so upgrades keep the protections they
already have.

* filer: stop reading legacy per-path records too

Drops the MpTs/MpDl read path along with the writes: meta_path_ts.go
keeps only the entry-field readers (CommitTsNs, EntryCommitTsNs,
localCommitTsNs). Entries written before stamping existed fall back to
mtime, and tombstones or commit records older builds left behind no
longer fence replayed events.

* filer: trim commit timestamp comments

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-25 11:37:06 -07:00
Chris Lu 53834117a6 mini: lift the post-mint allocation hold on first inventory report (#12061)
* mini: lift the post-mint allocation hold on first inventory report

DeferAllocation armed a fixed warmup window whenever the leader minted a
fresh topology id, so the first write on a brand-new weed mini blocked
for the whole window even though no prior cluster state could exist.
The window still has to cover a rebuilt master's veteran servers, which
report on their first post-reconnect heartbeat.

Mini has exactly one possible reporter — the embedded volume server,
whose first heartbeat carries the full inventory — so hold allocation
only until that report is applied. The bound raise and topology-id
freeze checks run while the heartbeat is processed, so a wiped master
state still raises the bound or trips the freeze before the hold lifts.

Standalone weed master and weed server keep the fixed window: volume
servers outside the process can still be on their way.

* master: apply queued topology-id reports before opening the allocation gate

SetTopologyInitPending(false) cleared the pending flag before replaying
topology ids stashed while the local id was still being minted. A
NextVolumeId landing in the drain window saw no gate and no freeze yet.
The post-mint allocation hold used to cover the gap by delaying the
winner past the freeze; once the hold can release on first inventory,
the gate must stay closed until the queued checks have run.

* mini: gate the first-report release on single-master mode

releaseAllocOnFirstReport assumes the embedded volume server is the only
possible reporter of prior cluster state. A mini started with
-master.peers joins other masters, whose own volume servers can still be
on their way when the first report lands, so keep the regular
quiet-pulse hold there.

* master: check reported topology id on empty full reports

A HasNoVolumes heartbeat carries the same LastSeenTopologyId a populated
list does, but the mismatch freeze only ran for non-empty reports. An
EC-only veteran could therefore release the post-mint allocation hold
through NoteInventoryApplied while its stale topology id never reached
CheckReportedTopologyId.

* master: drain the topology-id stash inside the init-gate lock

CheckReportedTopologyId only held reportedTopologyIdsLock while stashing,
so a report in flight during SetTopologyInitPending(false) could land its
freeze after topologyInitPending cleared, letting a NextVolumeId slip
through the gap. The whole check now runs under the lock, and the gate is
released inside the same critical section as the drain.

* mini, master: pin and enforce the single volume-server reporter

isSingleMasterMode only checks master peers; SendHeartbeat still accepted
any volume server, so a late foreign report could carry prior cluster
state after the first-report release dropped the hold. Mini now pins the
embedded volume server's address and the master refuses registration from
any other reporter while ReleaseAllocOnFirstReport is set.

* master: admit the volume-server reporter before applying heartbeat state

* test: cover SendHeartbeat admission ordering

A refused reporter must not freeze growth or raise the id bound, and the
pinned reporter's stale topology id still must.

* mini: re-derive the single-reporter release on every run

The package-level master options survive in-process reruns, so a
multi-master mini following a single-master one must not inherit the
first-report release or a stale reporter pin.

* master: admit addressless first heartbeats under the reporter pin

An unregistered heartbeat with no Ip skipped the single-reporter
admission check but still reached the seen-bound raise and the
topology-id freeze before the delta-heartbeat skip. Admission now runs
for every unregistered heartbeat; an empty address can never equal the
pin.
2026-10-25 09:25:39 -07:00
Chris LuandDevin 88b91cccba command: no extra localhost listener when -ip.bind is specified (#12055)
* util: add listenOnLocalhost parameter to NewIpAndLocalListeners

Nil keeps the current behavior of also binding localhost when the host
is a specific address; all callers pass nil for now.

* command: add -ip.bind.localhost flag to opt out of the extra localhost listener

Binding a command to a specific -ip.bind address also opened a second
listener on localhost for every port, with no way to turn it off. Each
command that calls NewIpAndLocalListeners (master, master.follower,
filer, s3, iam, volume, sftp, mq.broker, mq.agent) now accepts
-ip.bind.localhost=false to bind only the requested address. Embedded
services get prefixed flags (s3.ip.bind.localhost, sftp.ip.bind.localhost,
iam.ip.bind.localhost), and the command-level flag also disables them so
a single switch turns off every extra listener the command opens.

* mini: only require all-interface port availability while the loopback listener is on

With -ip.bind.localhost=false each service binds just -ip.bind, so a port
held on another address no longer collides; the all-interface precheck
would reject or relocate usable ports. WebDAV, the Admin UI, and the
Admin worker gRPC (which binds -admin.worker.ip, not -ip.bind) never open
a loopback companion listener, so their checks now use the address they
actually bind.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* mq.broker: bind the gRPC listener to -ip.bind instead of all interfaces

The listener used an empty host so -ip was ignored for binding and the
localhost companion was never created, leaving -ip.bind.localhost with
nothing to control. Give the broker the same -ip/-ip.bind pair the other
servers use; embedded mode binds to -ip.bind like its peers.

* server: add -sftp.ip.bind.localhost and -iam.ip.bind.localhost switches

weed filer already exposes the prefixed opt-outs for its embedded sftp
and iam; give weed server the same pair so each extra listener can be
disabled individually while -ip.bind.localhost=false still turns them
all off.

* mq.broker: keep the default all-interface bind unless localhost is off

An unset -ip.bind used to mean the standalone broker listened on every
interface; defaulting it to -ip silently cut off multi-homed clients. Only
fall back to -ip when -ip.bind.localhost=false, where a specific bind is
needed for the opt-out to take effect.

* mini: validate the Admin HTTP port on the address it will bind

The port check always used -ip.bind while startMiniAdminWithWorker falls
back to loopback when Admin has no credential, so a loopback-held port
passed validation and killed the admin at startup.

* mq.broker: keep the all-interface default bind when -ip.bind is unset

-ip only becomes the fallback once -ip.bind.localhost=false needs a
specific address for the listener.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* mini: validate the Admin HTTP port on its effective bind IP

When the Admin listener falls back to loopback (no credential or mTLS
configured), the pre-flight port check now probes that same address
instead of -ip.bind.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* util: keep NewIpAndLocalListeners source-compatible for 3-arg callers

* command: drop the -ip.bind.localhost flag; an explicit -ip.bind drops the extra listener

Instead of an opt-out flag, the loopback companion listener now simply
follows whether the bind address was chosen: it is created only while
-ip.bind is unset and the bind address falls back to -ip. Passing
-ip.bind=<addr> binds each port to exactly that address.

Each service options struct carries a listenOnLocalhost *bool that
embedders (weed server, weed mini, embedded s3/sftp/iam in weed filer)
set from the parent's -ip.bind state, captured before the bind flag is
repointed or filled in by the -ip fallback; standalone commands derive
it from their own flag before the fallback runs.

iam and mq.agent gain a real -ip.bind flag so they have a way to pin a
single address. mq.broker keeps its new -ip.bind flag but drops the
-ip.bind.localhost fallback hack: empty still listens on all interfaces.

* server: default embedded mq.agent brokers to the broker's bind address

With an explicit -ip.bind the embedded broker no longer listens on
localhost, so an unset -mq.agent.brokers defaulting to localhost:17777
could not reach it. Default to <serverBindIp>:<mq.broker.port> (the
address the embedded broker binds), falling back to localhost when the
bind address is a wildcard. An explicit -mq.agent.brokers still wins.

Also fix the NewIpAndLocalListeners skip message to say localhost.

* server: bind embedded mq.agent to serverBindIp

The embedded agent's ipBind was never set, so startQueueAgent fell back
to the advertised -ip address and an explicit -ip.bind did not pin the
agent listener.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-25 06:52:16 -07:00
fefac028ec feat: send EC shards in parallel per destination disk (#12050)
* feat: send EC shards in parallel per destination disk

One stream per destination disk. max_shard_transfers unset means no global cap, 0 sends shards sequentially, and N is a global cap that still keeps one stream per disk. The value is persisted through TaskPolicy. The maintenance form is a checkbox plus Max transfers because unset and 0 differ. The key is intentionally omitted from GetConfigSpec.

* ec shard distribution: treat negative max_shard_transfers as sequential

The config layer (normalizedMaxShardTransfers) clamps negative values to 0,
which selects sequential distribution. shardTransferCap diverged: a negative
cap that reached ShardDistributionOptions directly would have selected
uncapped parallel mode, the most aggressive setting. Align the options layer
with the same normalization so invalid input degrades to the conservative
mode.

* ec: serialize shard transfers to a server when a target leaves disk choice to the receiver

disk_id 0 is auto-placement on the receiving volume server, so an
auto-placed stream can land on a disk another queued stream is pinned
to. Grouping by (destNode, disk_id) alone let both run at once; collapse
all of that server's transfers under the destNode key instead.

* worker: clamp max_shard_transfers to the int32 wire range

ReadIntConfig returned the 0 fallback for values above MaxInt32, turning
a large positive cap into sequential mode. Read int64 and clamp into
range instead.

* worker: drop unused shard-transfer form helper

The maintenance form renders and collects max_shard_transfers in
plugin.templ JavaScript; the Go copy had no callers.

* worker: bound max_shard_transfers conversion under an in-range guard

---------

Co-authored-by: Chepurnov Simon <195125934+CHEPURNOV-S-S@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-10-25 06:37:26 -07:00
Jay HemnaniandChris Lu 0c9872f842 filer, volume: ignore Range when If-Range no longer matches (#12052)
* filer, volume: ignore Range when If-Range no longer matches

A GET with Range and a stale If-Range validator got 206 with the requested
bytes of the current file. RFC 9110 13.1.5: send the whole content instead,
so a resumed download restarts rather than mixing two versions.

* volume(rs): drop Range when If-Range no longer matches

Mirror the Go fix for the Rust volume server: evaluate If-Range against the
response ETag (strong comparison) or Last-Modified once the needle's
validators are known, then drop the range on a stale validator so the whole
current representation is sent. A dropped range on the RangeFromSource
strategy falls back to a full stream of the data-file region; buffered and
chunk-manifest paths already take Option<&str>.

* volume(rs): fail If-Range when the header value is not readable

* volume(rs): accept all HTTP-date formats in If-Range like Go

* server: cover alternate HTTP-date formats in If-Range test

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-10-25 06:36:40 -07:00
Chris LuandDevin da1d7444b8 master: shorten the first-write hold after a fresh topology id (#12059)
* master: release the post-mint allocation hold after a quiet pulse

DeferAllocation armed a fixed warmup window whenever the leader minted a
fresh topology id, so the first write on a brand-new cluster blocked for
the whole window even though no prior state could exist. The window
still has to cover a rebuilt master's veteran servers, which report on
their first post-reconnect heartbeat.

Track the last heartbeat that could carry prior cluster state — a server
joining, held volumes or shards, a nonzero id bound, or a foreign
topology id — and lift the hold one pulse after it goes quiet, capped by
the original window.

* master: make the post-mint allocation hold configurable

master.volume_growth.allocation_deferral overrides the warmup window a
leader holds volume id allocation for after minting a fresh topology
id. Unset keeps the existing window; "0s" disables the hold for setups
that know the cluster is fresh (test harnesses, single-node labs); any
other duration replaces the cap while the quiet-pulse early release
still applies.

* master: re-check the allocation deadline while it is held

NextVolumeId slept for the whole computed delay, so a heartbeat landing
mid-sleep that pushed the quiet deadline out was ignored — the pending
allocation could still run ahead of a straggler's report. Wait in short
slices, re-evaluating the deadline and the growth freeze each round.

* master: count only ids above the issued bound as prior-state evidence

Heartbeat echoes of volumes and shards this master allocated during the
hold refreshed the quiet window, so each growth allocation re-extended
the deferral until the hard cap. Evidence now requires an id beyond the
bound this leader has issued, a higher reported bound, or a foreign
topology id; a joining server (dn == nil) still counts once.

* master: reject an unparseable allocation_deferral instead of disabling the hold

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* master: do not count a skipped heartbeat as a joining server

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test: assert the allocation hold extends past the original quiet deadline

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* master: release the post-mint hold after two quiet pulses

One pulse proved short for a veteran that reconnects on a staggered
restart: with only five seconds of quiet, allocation could resume and
reuse ids before a slower server landed its first heartbeat. Two pulses
still halves the fresh-cluster first-write stall versus the full warmup
cap, and known-fresh setups keep the "0s" override.

* master: pick once more when assign times out as growth concludes

The pick loop's final attempt can predate the growth it triggered
concluding: growth registers its volumes in the layout before clearing
the pending flag, so a timeout landing on that boundary used to shed a
hard 406 over volumes that had just arrived. Pick once more after the
loop in both the http and grpc assign paths before deciding how to
shed.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-25 06:17:42 -07:00
Chris LuandDevin d15396fc4d s3tables: scope the anonymous identity's grant on the table catalogs (#12056)
* s3tables: scope the resolved anonymous identity instead of defaulting to allow

Catalog authorization fell back to the open default whenever the request
resolved to the configured anonymous identity, ignoring its grant. The
fallback now applies only when no principal was resolved (zero-config),
so anonymous requests are checked against the identity's configured
actions like any other principal. The S3 Tables API and the Iceberg and
Lance catalogs share this handler, so all three are covered.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test: authenticate the catalog clients in the s3tables suites

Anonymous calls into the Lance and Iceberg catalogs are now scoped by
the anonymous identity's grant, so the suites that used to rely on the
anonymous pass authenticate like their real deployments do: x-api-key
for the Lance probes and REST drivers, client-credentials OAuth for
pyiceberg.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* worker(lance): authenticate namespace calls with the storage credential

Anonymous catalog calls are now scoped by the anonymous grant, so a
worker that holds an S3 credential sends it as x-api-key on the
namespace requests instead of relying on the anonymous pass.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test: authenticate the pushdown catalog connection in catalog_lancedb

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* lance: follow catalog redirects only within the same host

reqwest forwards custom headers on a follow, so a redirect to another
host would carry the x-api-key credential to a server it was never
meant for. Keep same-host hops for proxies that normalize the path or
upgrade the scheme.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* lance: warn when the catalog api key travels in cleartext

The x-api-key pair is a real storage credential: over plain http to a
non-loopback host the network can read it. Warn at client build so the
operator sees it; https and loopback stay quiet, and the key still
sends since an http catalog is a valid deployment.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* lance: require the same origin on catalog redirects

A same-host hop can still downgrade https to http and put the api key on
the wire; matching scheme, host, and port keeps the credential on the
listener it was issued for.

* lance: bound catalog redirect hops so loops fail fast

The custom redirect policy drops reqwest's default 10-hop limit, so a
catalog that redirects back to itself would hold each call open until
the request timeout. Keep the bound alongside the origin check.

* lance: gofmt the redirect helpers

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-25 06:13:22 -07:00
Chris LuandDevin 39bdbc18db admin: keep read-only sessions out of the filer /etc tree (#12057)
* admin: keep read-only sessions out of the filer /etc tree

The file browser read routes are open to any authenticated session, so a
read-only login could download or view arbitrary filer paths. The filer
stores the credential and cluster configuration under /etc (identity JSON
with plaintext S3 secret keys, SSE-S3 keys, remote storage config), which
let a view-only session recover live secrets that the user API redacts.

Gate the read paths on the session role: read-only sessions get 403 for
/api/files/download, /view, /properties, /metadata on /etc paths, the
browser page redirects away, and recursive metadata exports skip the /etc
subtree for read-only sessions.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* admin: only reject real .. path components in file browser paths

path.Clean already resolves interior parent references, so any '..'
surviving in the cleaned path can only be a leading escape from a
relative input or part of a valid name. Matching on the substring
rejected legitimate entries like /buckets/reports/report..csv.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* admin: cover the read-only /etc export exclusion with a test

Pull the excluded-prefix selection into exportExcludedPrefixes so the
role-dependent /etc skip is exercised directly — the cleanReadPath
tests alone would still pass if the exclusion were dropped.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-25 06:06:21 -07:00
Chris Lu 403d51631b s3api: stop echoing stored tags and internal attributes on GET/HEAD (#12053)
* s3api: stop echoing stored tags and internal attributes on GET/HEAD

setResponseHeaders copied every entry.Extended key into the response,
only filtering xattr- and x-seaweedfs-. Stored object tags
(X-Amz-Tagging-*) and the gateway's own attributes (Seaweed-X-Amz-Acl,
-Owner, -Version-Id, -Object-Lock-Mode, ...) therefore leaked to any
caller holding s3:GetObject, and to anonymous readers on public-read
objects, without s3:GetObjectTagging or READ_ACP.

Tag count, user metadata, storage class and the real S3 headers
(x-amz-version-id, x-amz-object-lock-*) are unaffected.

* s3api: match stored tag and internal attribute keys case-insensitively
2026-10-25 06:05:06 -07:00
Chris Lu 52d91008ac ci: bump hosted runners to ubuntu-24.04
GitHub is retiring ubuntu-22.04; move all 94 runs-on pins across 40 workflows to ubuntu-24.04. rust_binaries_release: the arm64 cross step was jammy-era — noble keeps apt sources in deb822 ubuntu.sources (the sources.list edit was a silent no-op that left amd64 repos open to arm64 and 404'd) and ports suites must say noble. Pin deb822 stanzas to amd64 and add noble ports entries.
2026-10-24 23:07:28 -07:00
github-actions[bot] ec8f979350 4.50 2026-10-25 03:09:42 +00:00
Chris LuandDevin 1a0485a43d test: bound trino container memory + dump logs when it exits before ready (#12044)
* test: dump trino container state and logs when it exits before ready

When the Trino container dies during startup, waitForTrino only
reported the docker exec error ("container is not running"), which
gives no hint whether the JVM was OOM-killed, the image broke, or the
config is bad. Capture docker inspect exit state and the container
logs in the failure message so runner-side flakes are diagnosable
from the test output alone.

* test: cap trino container memory at 4g in catalog tests

Trino's jvm.config sizes the heap at 80% of host RAM. On a loaded
self-hosted runner the JVM attempts to claim most of system memory
at boot and the container is killed ~15s after docker run — the
flake seen in CI today. Bounding the container to 4g gives the JVM
a fixed ~3.2g ceiling (observed usage is ~750mb), immune to host
memory pressure.

* test: keep docker errors in trino container diagnostics

Report docker inspect/logs failures instead of dropping them, and
include the last docker exec output in the container-dead fatal so a
missing container still explains what happened.

* test: detect trino container exit in readiness fallback

When the polling loop times out while the container still appears
alive, the trino --version fallback may itself fail because the
container has since exited. Re-check its output for the
exited-container signature and route through the diagnostic dump.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test: dump trino container diagnostics on readiness timeout too

The alive-but-never-ready timeout path reported only the last docker
exec error, losing the container state and logs that the dead-container
path already captures.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 20:00:06 -07:00
Chris LuandDevin 8c1a0a0cc2 filer: fix prefixed listing when the resume point sorts below the prefix (#12049)
* test: prefixed directory listing with start below the prefix

A listing constrained to prefix "item-b" and resumed from "item" must
return item-b.versions. The seek-based stores return an empty page
because the iterator lands before the prefix range and breaks on the
first non-matching name.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: clamp prefixed-listing seek to the prefix range

Seek-based stores (leveldb, leveldb2, leveldb3, rocksdb, tikv, etcd)
started the scan at startFileName and stopped at the first key outside
the prefix. A startFileName below the prefix landed the iterator on a
non-matching sibling first, so the listing returned an empty page even
though matching entries followed. etcd had the mirror problem: a range
beginning below the prefix end leaked non-matching entries. Begin the
seek at max(startFileName, prefix), which is what foundationdb and the
sql stores already do.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer(hbase): honor startFileName lower bound in prefixed listings

The scan starts at the prefix but only skipped the exact startFileName,
so entries sorting before it were returned again on resume.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: keep paging a prefixed listing while matches can still follow

The fallback for stores without native prefix support stopped paging as
soon as the last scanned name reached the prefix, dropping prefix
matches that continue on the next page.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: return the last delivered name from the prefixed-listing fallback

The fallback for stores without native prefix support pages raw entries
and filters client-side. Keep the scan cursor and the returned cursor
separate: resuming must continue after the last delivered match, not the
last scanned name, or matches still inside the current raw page are
skipped on the next page. Also keep paging while the scan cursor is
inside or below the prefix range instead of stopping at the first name
that reaches the prefix.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer(hbase): start prefixed-listing scans at the resume key

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 19:59:19 -07:00
github-actions[bot] 0c32eab48e docs: regenerate star history chart 2026-10-25 00:48:29 +00:00
Chris Lu 85470416cb volume: overlap the local needle write with the replica fan-out (#10074)
* volume: overlap the local needle write with the replica fan-out

ReplicatedWrite wrote the needle to local disk and then fanned it out to
the replicas serially, so a replicated write cost local + replicas. The
needle is already fully buffered, so the two can run at once. Snapshot
the fields the fan-out reads first (the local write stamps the volume
TTL and updates needle flags, so the replica goroutine must not touch
the shared needle), then run the local write and the replica fan-out
concurrently and ack only after both. Where local disk and replica
network are separate resources the write latency drops to max(local,
replicas) instead of their sum; co-located on one disk the two writes
contend, so this helps distributed placements, not single-host tests.

* volume: return the local isUnchanged from the write goroutine

* volume: trim overlap-write comments

* volume-rs: overlap the replica fan-out with the local needle write

* volume: test the overlapped local write and replica fan-out

* volume: decode replica pairs only when replicas exist

A single-copy or incoming replicate write never uses the decoded pair map, so a malformed Pairs blob only burned an error counter and a log line. Gate the decode on replicaCount.

* volume: release the stalled replica on test failure paths

An early t.Fatal left the httptest handler blocked on releaseReplica, so the deferred Close hung instead of reporting the failure. Release idempotently ahead of the server close.

* test: mismatched If-None-Match serves the object over If-Modified-Since

The precedence case asserted the old If-Modified-Since-first ordering and
expected 304. The volume servers now evaluate If-None-Match first, so a
non-matching tag must return 200 with the body even when If-Modified-Since
would allow 304 (RFC 9110 13.1.3).
2026-10-24 15:16:16 -07:00
Chris LuandDevin e442eda88e s3: authorize POST policy uploads through the shared object check (#12047)
* s3: authorize POST policy uploads through the shared object check

doesPolicySignatureMatch gated a verified POST Object signature on the
identity's legacy Write action, which cannot represent attached IAM
policies, STS session policies, or key-scoped grants. Users authorized
only by an attached policy (e.g. s3:PutObject on bucket/*) were denied
with AccessDenied even though the equivalent PUT succeeds.

The handler already runs AuthorizeObjectWrite against the real object
key after the form's x-amz-* fields are promoted to headers, so the
coarse check only shadowed the full evaluation. Signature verification
now just returns the identity; authorization stays in one place.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test: POST Object authorizes attached-policy identities, still denies others

Cover the signature check and AuthorizeObjectWrite for an identity whose
only write grant comes from an attached IAM policy, plus the deny leg
for an identity whose attached policy omits s3:PutObject.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 15:09:34 -07:00
Chris Lu 216c1032e6 test: mismatched If-None-Match serves the object over If-Modified-Since (#12045)
The precedence case asserted the old If-Modified-Since-first ordering and
expected 304. The volume servers now evaluate If-None-Match first, so a
non-matching tag must return 200 with the body even when If-Modified-Since
would allow 304 (RFC 9110 13.1.3).
2026-10-24 15:06:19 -07:00
03f773f087 redis2: pipeline entry reads when listing a directory (#12042)
ListDirectoryEntries fetched every member with its own GET, so a page of
1024 entries cost 1024 round trips. Fetch the values in pipelined batches
of 128 as the loop reaches them and walk the replies in order. Each reply
keeps its own error, so missing and expired members are cleaned up and
skipped exactly as before, and an unreadable member still ends the
listing at that name. Batching keeps a caller that lists with a huge
limit and stops after a few entries from reading the whole directory.

Co-authored-by: Patrick Schmidt <1310072+Woellchen@users.noreply.github.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
2026-10-24 15:04:37 -07:00
Junker der Provinz f9dac64374 helm: remove filer.redirectOnRead, which weed filer does not accept (#12040)
weed filer lost -redirectOnRead in 9b3109a5d (1.59), which also removed
the chart value, but 2bed15f1d added it back. Setting it makes the filer
exit with "flag provided but not defined".
2026-10-24 10:45:57 -07:00
7b43dec665 shell: run volume.configure.replication in parallel per volume server (#12041)
* shell: run volume.configure.replication in parallel per volume server

Volume servers were configured one after another, and each server's
result overwrote the previous error, so a failure on one server was
lost when a later server succeeded. Run one job per volume server,
bounded by -maxParallelization like the other shell commands, and
return the errors from all servers.

VolumeConfigure unmounts and remounts the volume, so replicas of the
same volume on different servers take turns instead of all going
offline at once.

Co-authored-by: NyaMisty <5344431+NyaMisty@users.noreply.github.com>

* shell: test that volume.configure.replication overlaps servers up to the limit

* shell: assert every server ran in the parallel configure-replication test

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: NyaMisty <5344431+NyaMisty@users.noreply.github.com>
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 10:29:38 -07:00
Chris LuandDevin 95d8a709e8 ci: run s3-go-tests on remote_storage changes (#12043)
The S3 Remote Cache tests exercise the remote-storage S3 client, but the
path filter did not include weed/remote_storage, so a tagging change
there can regress the suite without running it on the PR.

Generated with [Devin](https://devin.ai)

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 09:42:17 -07:00
Chris LuandDevin e432b75822 remote_storage/s3: send only real object tags to the remote (#12039)
* remote_storage/s3: send only real object tags to the remote

WriteFile and toTagging copied every entry.Extended attribute into the
remote object's tag set, including Seaweed-internal keys such as
Seaweed-X-Amz-Acl whose values are not legal S3 tag values. Remotes
that validate x-amz-tagging now reject these uploads with InvalidTag.

Only X-Amz-Tagging-* attributes are S3 tags: filter to that prefix,
strip it, and encode through s3TaggingHeader so keys and values are
properly escaped.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* remote_storage/s3: skip remote tagging call when the tag set is unchanged

UpdateFileMetadata fired PutObjectTagging on any Extended diff, but most
attribute changes are not tag changes. Compare the effective tag sets and
return early when they match.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* remote_storage/s3: assert WriteFile errors in tag-header tests

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 09:41:01 -07:00
81c90cb446 filer.backup: retry a failed checkpoint read instead of replaying from the epoch (#12036)
* filer.backup: retry a failed checkpoint read instead of replaying from the epoch

Fixes #12035

* test: reset sink.local.directory in backup offset test cleanup

The new test sets sink.local.directory on the shared viper but only
reset the enabled flag, leaking a key that points at a deleted
t.TempDir() to later tests in the package.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test: cover the retried attempt re-reading the checkpoint

The fake filer now fails only its first KvGet, and the test runs a
second doFilerBackup attempt asserting the checkpoint is re-read and
the attempt proceeds past the offset stage. LookupDirectoryEntry is
stubbed so the SSE-S3 KEK probe returns a miss instead of retrying
10x2s.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 08:22:29 -07:00
Chris LuandDevin 4c2c0ff21c s3: fix wildcard matcher skipping '*' targets in policy patterns (#12034)
* s3: check pattern '*' before literal equality in matchWildcardString

When the target string carried a '*' at the offset of the pattern's '*',
the literal-equality case consumed it as a one-character match and no
backtrack point was recorded, so the comparison returned false. A Deny
Resource carrying a policy variable, or an Allow carrying NotResource,
could then be sidestepped by an object key containing '*' at the aligned
offset.

* test: cover literal asterisks in wildcard match targets

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 08:20:22 -07:00
Chris Lu 1d2d9dfed2 test(fuse): raise mini/filer and mount readiness waits to 90s (#12037)
Same startup-timing flake class as the Makefile waits: volume/filer
readiness at 30s is marginal on loaded runners — observed failing on a
github-hosted runner too (TestPOSIXCompliance +
TestExternalPOSIXSuites). The process-death early return still reports
real startup failures quickly.
2026-10-24 08:13:36 -07:00
c4d743a69f s3: make ListObjectsV2 continuation tokens round-trip XML-unsafe keys (#11996)
* s3: keep ListObjectsV2 continuation tokens round-tripping XML-unsafe keys

A page ending on a key XML 1.0 cannot carry (for example one containing
\x01) produced a NextContinuationToken from that raw key. The XML
encoder replaced the control characters with U+FFFD, so a client resuming
at the token restarted after the mangled name and skipped every key
sorting between it and the real one. encoding-type=url encoded the keys
but not the token, so it hit the same skip.

Encode the token as base64url behind a prefix when the key is not valid
XML text; keys that survive XML unchanged keep the bare key as the token,
so their tokens are byte-identical to before. The handler decodes the
prefix back to the raw key before listing.

Fixes #11990

* s3: keep undecodable base64: list tokens resuming at the raw key

* s3: round-trip ListObjectsV1 markers over XML-unsafe keys

* s3: round-trip ListObjectVersions key markers over XML-unsafe keys

* s3: round-trip ListMultipartUploads key markers over XML-unsafe keys

* s3: decode only tokens whose key could not have stayed bare

* s3: document and pin the encoded-wins rule for ambiguous list tokens

A token like base64:AQ reads as both a legacy bare key and the encoded form of an XML-unsafe key. The encoded reading already won in code; state why and cover it so the choice is not reverted as a false simplification.

* s3: keep request-marker echoes in the response's wire encoding

The decoded marker/key-marker is a raw key; echoing it writes XML-unsafe
bytes straight into the response, where the encoder turns them into
U+FFFD. Re-encode each echo through encodeListContinuationToken so the
client reads back the token it sent, and apply urlPathEscape for V1 when
encoding-type=url is active, matching the listed keys.

* s3: wire-encode the V1 NextMarker echo like the request marker

* s3: pin resuming an XML-unsafe key past a sibling directory

* s3: wire-encode ListMultipartUploads markers and keys under encoding-type=url

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: unescape url-encoded marker echoes on list resume

Under encoding-type=url the V1 and ListMultipartUploads responses emit
marker echoes through urlPathEscape, and clients pass them back verbatim.
The incoming marker was used as the resume position without undoing that
escaping, so resuming after a url-unsafe-but-XML-safe key like "a b"
(marker "a%20b") restarted the listing at the literal escaped text.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: assert the full decoded V1 sequence in the marker round-trip test

The test only checked the page count, so a regression that skipped one
key while repeating another still passed. Requesting encoding-type=url
keeps the XML-unsafe keys legible for the decoded comparison and
exercises resuming with the escaped marker echo.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: only unescape markers in the canonical urlPathEscape form

A url-encoded request marker like a%41 cannot be encoder output, since
urlPathEscape never emits %41 for 'A'. Unescaping it anyway resumed at
aA and skipped every key sorting between a%41 and aA, including a%42.
Require the unescape to re-escape to the received marker so only
verbatim echoes decode; anything else stays the raw key the client sent.

* s3: pin a raw-key marker that looks like an escape sequence

* s3: cover the marker-echo decode guard with unit cases

* s3: wire-encode the ListObjectsV2 StartAfter echo under encoding-type=url

AWS encodes Delimiter, Prefix, Key, and StartAfter in V2 responses when
url encoding is requested, but StartAfter was echoed raw. The echo now
takes urlPathEscape and the incoming start-after param runs through
decodeMarkerEcho, so a client resuming from the emitted form lands on
the raw key.

* Merge master and reconcile ListMultipartUploads marker encoding

Resolve the overlap with the merged encoding-type=url pass: keys and
markers now go through the single post-pass so nothing is url-escaped
twice. Markers are still token-encoded when the response is not
url-encoded, and the key-marker decode checks the query-escaped wire
form the endpoint actually emits.

* s3: echo the decoded start-after under encoding-type=url

The escaped echo of the raw input is not a fixpoint: a client resubmitting
the echoed StartAfter decoded it to the literal escape text and resumed at
the wrong position.

* s3: keep url-mode multipart key-marker decodes off the token path

Under encoding-type=url the emitted KeyMarker/NextKeyMarker are the
url-escaped raw key, never a base64-prefixed token, so the request decode
must unescape only. Running the token decode afterward misread a literal
"base64:..." key as an encoded token and resumed at the decoded bytes,
skipping the keys in between.

---------

Co-authored-by: Chris Lu <chris.lu@gmail.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 07:18:10 -07:00
fcd28031e7 mq/kafka: wait for a filer to register instead of panicking on an empty list (#12019)
* mq/kafka: wait for a filer to register instead of panicking on an empty list

NewSeaweedMQBrokerHandler queried the masters for filers once, ~2 s after
start, and handed an empty list to NewFilerClientAccessor, which panics. A
fresh filer registers only after its empty-membership confirm window
(10 s), so a gateway started with a fresh cluster crashed (Kafka Quick
Test, kafka-gateway exit 2). Retry while the list is empty for up to 60 s,
then return an error.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* mq/kafka: wait out a transient-empty broker listing too

A fresh SMQ broker registers with the masters on the same announce
delay as a filer, so the one-shot broker lookup could fail startup the
same way. Generalize waitForFilers into waitForClusterNodes and bound
both discoveries by the same window; the empty-broker check becomes
unreachable and drops away.

* mq/kafka: wait out the broker health check like node discovery

A broker announces to the masters before it has acquired its balancer
lock, so the gateway's three-attempt probe could still fail fast on a
fresh cluster. Retry FindBrokerLeader for the same window the caller
waits out empty listings.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 07:09:43 -07:00
Chris LuandDevin 34c7bd03ad ci: clean up stale FUSE mounts before running tests (#12020)
* ci(s3tables): tolerate stale FUSE mounts in disk diagnostics

* ci(s3tables): bound df diagnostics with timeout, explain once

A stale FUSE mount can also hang statfs, not just fail it, so wrap df in
timeout 15; one comment at the first site covers all sixteen.

* ci: tolerate stale FUSE mounts in df diagnostics elsewhere

The same self-hosted runners serve the other workflow files; a bare
df -h there can stall or fail a job the same way.

* ci: unmount disconnected FUSE test mounts before jobs start

* ci: skip FUSE cleanup when the variant checkout is skipped

* ci: reclaim the crashed run's test dir once its stale FUSE mount is detached

* ci: stop tracking python bytecode for the FUSE cleanup action

The helper's __pycache__ slipped into the tree; ignore it so local
test runs cannot re-add it.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: run the FUSE cleanup action's unit tests on action changes

test_cleanup.py was never wired into CI, so a regression in cleanup.py
would ship silently. A path-filtered workflow runs it on every change
under .github/actions/.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: match every FUSE test mount layout in the cleanup action

The candidates list only matched seaweedfs_fuse_test_*/mount, so stale
mounts left by fuse_failover and fuse_p2p runs (mountN leaves under
seaweedfs_fuse_* dirs) and the posix-compliance workflow's fixed
/tmp/seaweedfs/mount were never reclaimed and could still wedge df.

* ci: run the FUSE mount cleanup in the fuse_failover and fuse_p2p workflows

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 07:07:55 -07:00
18f9d9a0b6 feat: Free Volume Deadband (M) for EC shard placement (#12029)
* feat: Free Volume Deadband (M) for EC Place nodes and racks

|ΔfreeSlots| ≤ M is a tie; remainder is liveShards. M=0 stays strict
freeSlots. M is free volume slots (× dataShards). pickNodeInRackEligible
and pickRackEligible share the rule. Worker plugin config
free_slots_deadband, default 0. Not on the write path.

* fix: anchor EC freeSlots deadband on class maxFree

Pairwise |Δfree|≤M vs the current winner is not transitive: 120→110→100
at M=10 slots picked 100. After evenness, rank by maxFree, keep
|free-maxFree|≤M, then min liveShards. Shared by node and rack. M=0 is
still only maxFree then live.

* worker: add free_slots_deadband to ErasureCodingTaskConfig

* erasure_coding: persist free_slots_deadband through TaskPolicy

ApplyMaintenanceConfigFromToml accepted the key but the admin save path
round-trips config through ToTaskPolicy, so the value was dropped on
reload; serialize it like the other EC settings.

* erasure_coding: bound free_slots_deadband at its declared maximum

The TOML loader and the plugin worker config only clamped negatives, so
a value above the spec's 100000 could reach the int32 TaskPolicy field
and wrap; cap at 100000 on both entry paths and declare MaxValue on the
plugin field.

* erasure_coding: document free_slots_deadband in scaffold admin.toml

---------

Co-authored-by: Chepurnov Simon <195125934+CHEPURNOV-S-S@users.noreply.github.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
2026-10-24 07:06:46 -07:00
52e05ccda4 s3: accept a self-copy that changes the storage class (#12031)
* s3: accept a self-copy that changes the storage class or encryption

AWS changes an object's storage class or encryption by copying the
object onto itself with the new x-amz-storage-class or SSE headers, and
only rejects a self-copy that changes none of metadata, storage class,
website redirect location or encryption. CopyObject rejected every
self-copy without REPLACE metadata or tagging directives as
InvalidRequest, so these requests failed.

* s3: reject an invalid x-amz-storage-class on CopyObject

A self-copy with a bad storage class used to fail as InvalidRequest;
once such copies are admitted the value must be validated like PutObject
does, or it would be stored as opaque metadata.

* test: cover the invalid storage class rejection on self-copy

* s3: only a storage class change admits a self-copy

Admitting encryption changes on a self-copy is unsafe: the metadata-only
path compares only KMS key IDs, so a new encryption context or bucket-key
setting is advertised over the old data key, and SSE-KMS key rotation
onto the same entry keeps the old data key.

Count only a requested storage class that differs from the object's
(STANDARD when none is stored) as a change, so a self-copy to the
current class or with only encryption headers is rejected as before.
Such a self-copy updates the object in place and has to keep its
encryption; one that would re-encrypt it or uses SSE-KMS is rejected
too.

---------

Co-authored-by: Wleeaf <179125857+wleeaf@users.noreply.github.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
2026-10-24 07:06:20 -07:00
Chris LuandDevin 514c0c1e99 topology: stop a stale duplicate-vid report from re-quarantining the survivor (#12032)
* topology: drop a duplicate-vid report whose own copy is already gone

flagDuplicateVolumeId runs on a goroutine after registration, so by the
time it executes the reporter's copy may have been unregistered (a
finished move drops the losing copy). The surviving copies are then
legitimate, yet the stale report still quarantined them and stored a
phantom (node, vid) key for the departed reporter. Skip the marks when
the reporter no longer serves the volume under its own layout key.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(topology): cover a stale duplicate report against a lone survivor

The existing tests only flag the conflict before the losing copy is
unregistered. Run the delayed duplicate check after the reporter's copy
is already gone and assert nothing is quarantined and the survivor stays
writable, so removing the stale-report guard fails the test outright
instead of needing a lucky schedule.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(topology): cover the enforceQuarantine-driven release retry

A failed writable push re-arms the record, and the next volume report
drains that retry through enforceQuarantine itself - a different entry
point than the pulse-driven pass the other tests exercise.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 07:05:46 -07:00
8841eb6c70 volume: check If-None-Match before If-Modified-Since and match ETag lists (#12027)
* filer: match If-Match and If-None-Match against "*" and ETag lists

checkPreconditions compared the whole header as one ETag, so "*", a
list of ETags and a weak W/ tag in If-None-Match never matched
(RFC 9110 13.1.1, 13.1.2).

* volume: check If-None-Match before If-Modified-Since and match ETag lists

A volume GET answered 304 from If-Modified-Since even when If-None-Match
named a different ETag, and compared If-None-Match as one exact string,
so "*", lists and weak tags never matched (RFC 9110 13.1.2, 13.2.2).

* filer: split etag lists only outside quoted tags

A comma inside a quoted opaque tag is a legal etagc, not a list
separator; strings.Split cut such tags into fragments that
CanonicalizeETag could turn into a false match.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: match repeated If-Match and If-None-Match header lines

Header.Get reads only the first field line, so a match on a later
repeated header was missed; join all field values like any other
comma-separated list.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: match repeated If-None-Match header lines

Header.Get reads only the first field line; join all field values so a
match on a later repeated header is seen. Also covers a comma inside a
quoted opaque tag.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: send the etag on a 304 like the filer does

RFC 9110 15.4.5 lists ETag among the fields a 304 must carry when a 200
would send it; SetEtag now runs before the notModified check.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: mirror the conditional-GET fixes in the rust server

not_modified_response now checks If-None-Match before If-Modified-Since,
matches "*", weak tags, quoted commas and repeated field lines through
the same list rules, and sends the etag on a 304.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: treat an empty If-None-Match like the Go server does

* filer: ignore empty elements in etag precondition lists

* volume: ignore empty elements in etag lists like the filer

* filer: let a present-but-empty precondition line suppress the date check

* volume: gate If-Modified-Since on a present If-None-Match line

* volume: gate If-Modified-Since on header presence like the Go server

---------

Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: chrislusf <chris.lu@gmail.com>
2026-10-24 07:05:29 -07:00
Chris LuandDevin 59cccc788c master: read the writable-volumes cap from master.toml (#12009)
* master: read the writable-volumes cap from master.toml

-maxWritableVolumes becomes the master.max_writable_volumes config key,
so the new master limit is set the same way as other master options
instead of a flag wired into every command that embeds a master.

* master: name the writable cap master.max_writable_volumes_per_layout

The cap applies per volume layout (collection + replica placement + ttl +
disk type), so the key names its scope instead of reading as a global cap.

* master: keep an embedder's MaxWritableVolumes when the key is unset

NewMasterServer overwrote option.MaxWritableVolumes with the viper
value, which reads as 0 whenever master.toml omits the key, silently
dropping a cap a Go caller supplied through MasterOption. Seed the
default from the option instead so the field survives an absent key;
an explicit config entry still wins.

* master: drop stray nested-checkout gitlinks

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 07:04:45 -07:00
9e7fd2eb90 filer: match If-Match and If-None-Match against "*" and ETag lists (#12026)
* filer: match If-Match and If-None-Match against "*" and ETag lists

checkPreconditions compared the whole header as one ETag, so "*", a
list of ETags and a weak W/ tag in If-None-Match never matched
(RFC 9110 13.1.1, 13.1.2).

* filer: split etag lists only outside quoted tags

A comma inside a quoted opaque tag is a legal etagc, not a list
separator; strings.Split cut such tags into fragments that
CanonicalizeETag could turn into a false match.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: match repeated If-Match and If-None-Match header lines

Header.Get reads only the first field line, so a match on a later
repeated header was missed; join all field values like any other
comma-separated list.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: ignore empty elements in etag precondition lists

* filer: let a present-but-empty precondition line suppress the date check

---------

Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: chrislusf <chris.lu@gmail.com>
2026-10-24 07:04:41 -07:00
Chris LuandDevin d6a68468ad volume: keep a below-minimum fsync batch open for stragglers (#12016)
* storage: hold a below-minimum fsync batch open for stragglers

When the durable-write queue drained, the worker flushed whatever it
held, so low-concurrency fsync writes collapsed to one sync per
request -- a seek plus a rotation on rotational media no matter the
payload. Once the queue empties with the batch still below the
configured minimum bytes and requests, the batch now stays open up to
SyncBatchOption.Wait, armed once per batch so the added latency stays
bounded. DiskLocation carries the option and defaults it; Store exposes
a setter the server calls once at startup.

* volume server: expose the fsync batch window as -fsync* flags

-fsyncMinBatchBytes, -fsyncMinBatchRequests and -fsyncBatchWait carry
the operator's batching floor into every disk's SyncBatchOption.
-fsyncBatchWait=0 restores the previous flush-on-drain behavior.

* server/mini: register the fsync batch window flags

weed server and weed mini build their own VolumeServerOptions and left
the three new fsync flag pointers nil, so startVolumeServer panicked
dereferencing them at startup.

* storage: bound the fsync batch window by Wait while the queue refills

Once the straggler timer was armed, a steady trickle of queued writes
kept taking the len(requests)>0 continue path, so the first write could
wait past Wait until the size or count cap flushed the batch. Track the
armed deadline and flush once it passes even with requests buffered.

* storage: sync the straggler test on the worker taking the first write

The fixed sleep let both writes queue together (passing under the old
flush-on-drain behavior) or the straggler arrive after the window closed
(flaky failure). Poll the drained queue instead so the second write can
only land inside the armed window or a later batch.

* volume server: default the fsync batch window when its options are unset

weed server and weed mini build VolumeServerOptions without the fsync
flag pointers, and outside embedders can too. Resolve the batch window
from the defaults so startVolumeServer never dereferences nil options.

* storage: drive the batch-window test through an injectable timer

The window is armed by the worker now, so the straggler provably joins
an open batch and expiry is fired by hand instead of racing a sleep.

* seaweed-volume: accept -fsyncMinBatch/-fsyncBatchWait flags

Same flags and defaults as weed volume so the Rust server's durable
write batching can be tuned identically.

* seaweed-volume: hold a below-minimum fsync batch open for stragglers

Mirrors the Go worker's batchWait: once the queue drains under the
minimum batch, the window stays open for stragglers and an expired
window still bounds the batch while the queue refills. Paused-time
coverage replaces any sleep-based assertion.

* seaweed-volume: sync the straggler test on the armed batch window

The test fired both writes together, so the straggler could already be
queued when the worker drained — a regression that drops mid-window
writes from the open batch would still pass. Wait until the worker
arms the window before sending the last write (Go's test polls the
drained channel for the same guarantee).

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* seaweed-volume: reject fsyncMinBatchBytes that does not fit usize

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 07:04:05 -07:00
b0e6f7a096 s3: allow up to 50 tags in a bucket tag set (#12030)
* s3: allow up to 50 tags in a bucket tag set

PutBucketTagging validated the tag set with the object tag rules, which
allow 10 tags. AWS allows a bucket up to 50 tags, so tag sets of 11 to 50
tags were rejected with InvalidTag. Validate bucket tags against 50 and
keep the limit of 10 for objects and S3 Tables.

A tag set that repeats a key is rejected with InvalidTag, as AWS does,
instead of keeping one of the values, so 51 tags with a repeated key no
longer pass as 50.

* s3: reject duplicate tag keys in PutObjectTagging

The tag set collapses to a map, so a repeated key was silently kept once.\nMatch the bucket tagging check and answer InvalidTag instead.

* s3: reject empty tag keys

AWS requires a tag key to be at least one character; the shared\nvalidator only checked the 128-char upper bound.

* s3: reject duplicate tag keys in the tagging header parser

parseTagsHeader (the x-amz-tagging path used by CopyObject with
TaggingDirective=REPLACE) silently kept the last value when a key
repeated, while the PutObject and multipart header paths and the new
XML checks return InvalidTag for the same input. A repeated key after
URL decoding now fails the same way.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: validate x-amz-tagging tags on PutObject before storing the entry

The header path only rejected duplicate keys, so an upload could store
more than ten tags or keys and values that fail S3 tag rules. Run the
parsed set through ValidateTags like the XML tagging handlers do.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: validate x-amz-tagging tags on multipart initiate

ParseS3Metadata stored header tags with no tag-set validation, the same
gap putToFiler had. Multipart uploads now reject oversize or malformed
tag sets with InvalidTag.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: count tag length in characters, not bytes

* s3: validate object tags before uploading chunks

* s3: reuse parseTagsHeader and drop raw header from warnings

---------

Co-authored-by: Wleeaf <179125857+wleeaf@users.noreply.github.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 07:04:01 -07:00
de02158212 filer: let NewChunkStreamReaderFromFiler fall back to the master on a cache miss (#11964)
refreshUrls invalidates every cached location of a volume after a failed
read and re-looks it up through the reader's lookup function. With the
cache-only masterClient.LookupFileId, one failed read to one replica left
the volume missing from the filer's cache until restart, so metadata-log
replay skipped it. Use GetLookupFileIdFunction like the other read paths.

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-10-24 07:03:47 -07:00
Chris LuandDevin 8844d9823f iam/oidc: wrap the jti replay cache in its own type (#12017)
* iam/oidc: wrap the jti replay cache in its own type

The replay-protection state lived as ten fields and nine methods spread
across OIDCProvider. jtiReplayCache now owns the store, count, bound,
sweep scheduling, and cleanup worker; the provider keeps a single jti
field and delegates configure, consume, stop, adopt, and merge.

* iam/oidc: lazily allocate the jti replay cache

A zero-value OIDCProvider left jti nil, so Initialize, single-use
validation, Shutdown, and the replay-state handoffs panicked on a nil
cache. Allocate the cache through a sync.Once-guarded getter instead.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 07:03:42 -07:00
Chris LuandDevin 3e5aa8c3b3 topology: mark replica quarantined before recording the pair (#11983)
* topology: serialize quarantine records with their layout marks

A quarantine publishes a (node, vid) record and marks the replica read-only at the layout level; release and heartbeat flag updates must observe that pair as one state. Store-then-mark let a concurrent release delete the record before the mark landed, stranding the surviving replica read-only; mark-then-store let a heartbeat clear the mark before the record was visible, returning a conflicting replica to the writable list.

quarantineLock now covers every record<->mark pair: flagDuplicateVolumeId's quarantine, releaseResolvedQuarantine's undo, the isQuarantined-gated marks on registration, and the quarantine-folded read-only flag writes on heartbeat paths. Remote server pushes stay outside the lock; a push that lands after the record was released is undone, and a failed release push re-adds the record for a later retry.

The release test now waits for in-flight async flags to settle instead of asserting on a single read.

* topology: fold quarantine into RegisterVolume's flag rebuild

RegisterVolume rewrites every replica's location read-only flag from what
its node last reported, and reported flags deliberately exclude quarantine.
Any registration of a shared vid -- a reconnected replica, a new copy, the
self-heal and delta-heartbeat re-register paths -- therefore cleared the
quarantine mark on every replica in the location, while the gate afterwards
only re-marked the registering node. Two call sites had no gate at all.

RegisterVolume now takes an isQuarantined predicate and evaluates it under
accessLock for every replica it touches, so the mark is never dropped in
the first place: a release unmarks after the rebuild, and a quarantine
mark re-applies on the next rebuild, with no window between.

* topology: keep quarantine when a vacuum commit re-adds a replica

SetVolumeAvailable re-adds the replica into the location list; a re-added
slot starts with a cleared read-only flag, so a quarantined replica came
back flag-free and the next ensureCorrectWritables could return a
conflicting volume to the writable list. Pass the quarantine state in and
set the flag back under the same accessLock, with the caller holding
quarantineLock so release cannot interleave between the check and the mark.

* topology: merge quarantine records instead of overwriting them

enforceQuarantine wrote back the state copy it loaded before its remote
push, so a concurrent writer's enforced=true was lost: the server stayed
read-only while the record said unenforced, and a later release skipped
the writable push it owed. The write-back now reloads the record under
quarantineLock and merges -- enforced wins from either side, reported is
taken from the newer pre-enforcement observation only while no push has
landed yet.

All three compensating writable pushes had the same hole in reverse: a
failed un-quarantine left the server read-only with no record to retry
from, and release's restore blindly overwrote any record a fresh flag
pass had created in between. They now go through restoreQuarantineRecord,
which adds the record only when absent and otherwise sets enforced on
the current one, preserving its reported flag.

* topology: hold quarantineLock over registration's quarantine check-and-mark

RegisterVolume reads the quarantine record and marks the replica
read-only under accessLock, but the pair ran without quarantineLock.
A release could push writable, watch the record get re-read, and then
delete it: the replica stayed read-only with no record left to clear,
and unchanged writable heartbeats skip SetReplicaReadOnlyFlag so the
mark never repaired. Holding quarantineLock around the check-and-mark
in all three production callers serializes it against the release
clear-and-delete.

* topology: serialize remote quarantine pushes behind a record recheck

releaseResolvedQuarantine deletes the record under quarantineLock and
sends the writable request after unlock. A conflict flagged while that
request is in flight creates a fresh record and marks the replica
read-only again, and the older writable request can still land last,
leaving the server writable under an active quarantine. The reverse
flap strands a stale read-only mark the same way.

Route every QuarantineVolumeFn call through pushQuarantine, which
holds quarantinePushLock across the remote call and re-checks the
record under quarantineLock right before sending: a read-only push
requires the pair to exist, a writable push requires it to be gone.
A skipped push means the decision already moved on, so the final
server state always follows the newest record.

* topology: serialize quarantine pushes per replica, not globally

A single quarantinePushLock held across the remote mark call let one slow
volume server (up to the 10s client timeout) delay pushes for unrelated
replicas. Keying the lock by (node, vid) keeps pushes to one replica
ordered in record-decision order while pushes to others run in parallel.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* topology: restore the layout read-only mark with a released record

A failed writable push re-armed the quarantine record but left the
replica marked writable in the layout, so PickForWrite could route to a
replica the server still holds read-only.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* topology: wait out async duplicate checks in the release-retry test

TestFailedReleasePushRestoresQuarantineMark asserts layout marks around a
release pass, but the two RegisterVolumeLayout calls each start a
background flagDuplicateVolumeId that can still be mid-flight. A late
mark could land after the final release and fail the writable assertion
even though the code behaved correctly. Track the async passes in a
WaitGroup and drain them before the explicit flag call so the rest of
the test is deterministic.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* topology: keep quarantine when a replica's layout key is unreadable

* topology: keep a failed writable push's restore ordered under the push lock

* topology: retry the quarantine release on every duplicate check

* ci: isolate Java and S3 test servers on persistent runners

* ci: drop the duplicated -ip flags in the lifecycle test server

The bind flags were already on the command; the isolation pass added a
second pair.

* topology: retry a failed quarantine writable push on the next report

* topology: retry pending quarantine releases off the leader pulse

A failed writable push re-arms the quarantine record with a retry that only
enforceQuarantine fires, and that runs only for volumes a heartbeat carries.
Once the server has reported the replica read-only it drops out of the delta
reports, and restoreQuarantineRecord keeps the stored flag matching the
server's so the digests agree and no full list is requested - the pending
release then never retries and the idle volume stays read-only forever. Run
the same release pass from the leader's pulse loop so a pending retry no
longer waits on a report.

* topology: exercise the pulse-driven quarantine release retry

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 07:00:45 -07:00
5700aa5e91 s3: evaluate bucket CORS for path-style OPTIONS on the bucket root (#12028)
* s3: evaluate bucket CORS for path-style OPTIONS on the bucket root

A path-style preflight for /bucket (no trailing slash) missed the bucket
OPTIONS route, whose template is /{bucket}/, and was answered by the global
OPTIONS handler from the global AllowedOrigins instead of the bucket CORS
configuration. Register the bucket OPTIONS route without a path matcher, and
keep OPTIONS for the health probe paths on the service-level handler ahead of
each path-style router.

Fixes #12025

* test: drop the issue reference from the bucket-root preflight test comment

---------

Co-authored-by: Wleeaf <179125857+wleeaf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 06:58:59 -07:00
Chris LuandDevin 9281ed8007 master: never auto-admit raft peers on a single-node configuration (#12021)
* master: never auto-admit raft peers on a single-node configuration

A master started without cluster peers (peers=none, or no peers so the
list holds only itself) declares a one-node cluster, yet any foreign
master that registered over KeepConnected was admitted as a second
voter. The resulting two-voter cluster cannot reach quorum when the
foreign peer is unreachable, wedging a previously healthy single
master. Gate raft admission on having been configured with cluster
peers; masters started with a peer list keep the announce-to-join
flow used to grow the cluster.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* master: source SingleMaster from -peers=none, not the peer count

A master started with an empty peer list ends up with a one-peer map
containing itself, and NewMasterServer inferred declared single-master
mode from that count. The master still mints a raft cluster through the
bootstrap path, but AdmitRaftPeer then ignores every later joiner, so a
deployment that grows from one master to several can never form a
quorum. isSingleMasterMode already spells out the declared case
(-peers=none); populate MasterOption.SingleMaster from it instead.

* master: cover clustered raft admission with a real in-memory leader

The previous clustered test had no raft leader and asserted nothing, so
an AdmitRaftPeer that always returned early still passed. Bootstrap a
single-node hashicorp raft over the in-memory transport, wait for
leadership, and assert the leader reports and admits a missing peer.

* master: return the raft admission result and assert committed join

AdmitRaftPeer ran AddVoter on a detached goroutine, so the only observable
state was an in-flight marker deleted on success and failure alike, and the
clustered test could never tell whether the join worked. Admission now
hands back a result channel; the test runs a connected second in-memory
node and asserts the peer sees itself a committed voter.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 06:58:16 -07:00
Chris LuandDevin fd3b437e04 s3api: write the null version's object-lock metadata to the plain entry (#12033)
* s3api: write the null version's lock metadata to the plain entry

PutObjectRetention and PutObjectLegalHold with versionId=null loaded the
pre-versioning entry but wrote the update under <key>.versions/v_null, so
the acknowledged lock never bound the object it was read from. Resolve the
entry and its write path in one helper that applies the same null-version
rule as getSpecificObjectVersion.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3api: cover the null-version lock write path

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3api: persist object-lock metadata with an in-place entry update

mkFile rebuilds the target entry from chunks alone, so a lock write on a
plain (null-version) entry discarded attributes, inline content, and the
remote reference. UpdateEntry keeps the whole fetched entry, matching how
the tagging and ACL writers persist metadata.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 06:57:02 -07:00
Chris LuandDevin 689ea6ea3e release: digest-bound provenance and SBOM attestations for container images (#12024)
* release: emit digest-bound provenance and SBOM attestations

Per-platform builds now record SLSA provenance at mode=max and a
BuildKit SBOM, so the builder identity and resolved dependencies are
captured instead of the empty predicate the 4.48 build record carried.
After the multi-arch index is assembled and signed, the merge job
publishes GitHub artifact attestations bound to the merged digest and a
syft SPDX SBOM of the image, giving digest-pinning consumers a
verifiable record on the exact index they pull.

* release: keep unfixed findings in the trivy SARIF upload

ignore-unfixed dropped every finding without an available patch, so the
uploaded report understated the image's exposure. The scan stays
report-only; only the completeness of what is uploaded changes.

* release: generate and attest SBOMs per platform digest

An index-wide syft scan only describes its default platform, so the SBOM
attested to the merged multi-arch digest misrepresented every other
manifest. Scan and attest each platform digest instead, which also drops
the sbom-action upload defaults that raced release-asset permissions and
artifact names across the merge matrix.

* release: resolve per-platform index to its image manifest before syft/cosign

push-by-digest records point at OCI indexes (image + attestation
manifests), so scanning or attesting the index digest picks the
runner's platform and fails outright for the other arches under
set -e. Resolve the non-attestation manifest inside each index and
bind the SBOM attestation to the image digest a same-platform pull
of the merged index actually returns.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 06:47:24 -07:00
Chris LuandDevin be081b49b7 notification: named webhook targets for per-path event routing (#11925)
* notification: honor the config prefix in the multi-queue guard

validateOneEnabledQueue checked "<name>.enabled" without the
notification prefix, so the fatal-on-two-queues guard never saw the
real keys and could never fire.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* notification: support named webhook targets under [notification.webhook.<name>]

The webhook queue only accepted one endpoint, so events could not be
routed to different services by path prefix. Each enabled
[notification.webhook.<name>] subsection now spins up its own target —
own filter, http client, retry middleware and worker budget — published
on a per-target topic so every matching endpoint gets each event exactly
once. The flat [notification.webhook] section keeps working as the
unnamed target, and named subsections alone now activate the queue.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* notification: require enabled for the flat webhook target

buildTargets included the flat section whenever an endpoint was set, so a
disabled [notification.webhook] block — which the scaffold ships with a
placeholder endpoint — would start receiving events as soon as any named
target activated the queue.

* notification: resolve named target enabled flags through config

namedTargetPrefixes read the raw map value, which never sees environment
variable overrides; GetBool resolves WEED_* env bindings, so a named
target disabled in the file can now be switched on by
WEED_NOTIFICATION_WEBHOOK_<NAME>_ENABLED.

* notification: cancel the webhook queue only after all workers stop

Each target's router goroutine deferred the shared cancel, so the first
worker to exit tore down the queue context for every other target and
the dead letter logger.

* notification: isolate webhook targets behind per-target queues

SendMessage published to each target's topic inline, so a subscriber
channel that filled up blocked every later target and the filer's
notification path with it. Each target now drains its own inbox through
a forwarder goroutine; SendMessage fails fast on a full inbox, matching
the "throw message" contract the filer already applies to errors.

* notification: name the webhook target in failure and dead letter logs

Every target dead-letters to the same topic, so without the name there
is no way to tell which endpoint failed or needs a replay.

* notification: assert delivery in the named targets test

The test only exercised SendMessage; it now records requests on both
endpoints and checks matching, non-matching, and unrouted keys reach
the right servers exactly once.

* notification: wait for the webhook router before accepting events

* notification: apply backpressure instead of dropping a full webhook inbox

* notification: bound queued and in-flight webhook deliveries per target

* notification: release the pending token when shutdown races the enqueue

A SendMessage that wins the pending select as the queue is canceling
could block forever on a full, undrained inbox.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* notification: stop waiting for a router that failed to start

A router that exits before reaching Running() would leave initialize
blocked on a context nothing had canceled yet.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* notification: route webhook events past saturated targets

A full pending reservation made SendMessage wait on the slow target
before it ever reached the healthy ones after it, coupling every
endpoint's latency to the most congested one. Targets that can take a
reservation now get the event immediately; saturated targets defer to a
second pass that still blocks for a slot, so backpressure is kept and no
event is dropped.

* notification: retry webhook deliveries before dead-lettering

PoisonQueue must wrap the retry middleware: added after it, the poison
middleware intercepts the first handler error, republishes to the dead
letter topic, and returns nil — the retry chain never sees a failure, so
max_retries/backoff settings were dead code and every delivery failed
straight to dead letter on the first attempt.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 02:12:17 -07:00
fb3fb9dcb8 s3api: support s3:RequestObjectTagKeys in bucket policies (#11993)
* s3api: support s3:RequestObjectTagKeys in bucket policies

Condition operators accept the ForAllValues: and ForAnyValue: qualifiers,
and s3:RequestObjectTagKeys holds the keys of the tags taken from
X-Amz-Tagging wherever s3:RequestObjectTag/<key> is set.

CopyObject no longer stores an empty-key tag for an empty segment in
X-Amz-Tagging, so it keeps the key set a policy has checked.

Fixes #7679

* s3: parse tagging header like the tag handlers, not url.ParseQuery

url.ParseQuery stops at 10,000 parameters, so an X-Amz-Tagging header
padded with empty & segments made requestObjectTags return nil: the copy
handler still stored the denied tag while every s3:RequestObjectTag*
condition saw nothing. Scan non-empty segments the way parseTagsHeader
does so the policy sees the same tags the object gets.

* test: gofmt s3tables test files after #11997

The format check fails on these files on master as merged, so every
PR's S3 Tables Format Check goes red until they are re-formatted.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: user.email <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 02:04:39 -07:00
Chris LuandDevin 7a9a9f10af ci: invoke pip via python -m in s3tests clone steps (#12010)
* ci: invoke pip via python -m in s3tests clone steps

The self-hosted runner reaches the Clone s3-tests step with the
setup-python toolcache interpreter on PATH but no pip shim, so the
bare 'pip install' exits 127. python -m pip resolves through the
interpreter that is already on PATH, matching the helm_ci workflow.

* ci: invoke pip via python3 -m in s3tests clone steps

The adjacent fix_s3_tests_bucket_conflicts step already runs python3, so
python3 is the interpreter guaranteed on the self-hosted PATH; a missing
python alias would fail the same way the missing pip shim did.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 23:05:56 -07:00
ecd778a4a0 iceberg maintenance: a merged file with null row ids names no first_row_id (#11973)
* iceberg maintenance: a merged file with null row ids names no first_row_id

Compaction stamps a merged v3 data file with the _row_id of its first row as
first_row_id (mergedFileFirstRowID). When the bin mixed rows that carry a
materialized _row_id with rows that carry none (a data file with no
first_row_id, as a v2 table's files are after an upgrade to v3, gives null
ids), the null rows then inherit first_row_id + their position in the file:
ids that other rows already hold, so two rows answer to the same row id.

A merged file now names a first_row_id only when its footer shows no null
_row_id in any row group. Otherwise the commit assigns the file a fresh range
from the table's next-row-id: the null rows get new, unused ids and the
materialized ones keep theirs (a materialized value overrides inheritance).

Test: TestMergedFileFirstRowIDOnlyWithoutNullIDs ("materialized first, a null
later" fails without the fix).

* iceberg maintenance: name no first_row_id when the merged file has a null _row_id

mergedFileFirstRowID stamped the merged file with its first row's id
without checking later rows. A null _row_id in a merged file inherits
first_row_id+position, an id another row already holds, so the commit
produced two rows answering to the same row id.

Check the footer's null counts for the _row_id column across row groups
first: any null means the file is committed without a first_row_id, so its
nulls take fresh ids from the table's next-row-id while materialized
values still override inheritance.

* shell: retry filer discovery until master knows a filer

A shell started against a freshly booted cluster (weed mini, the
s3tables/lance test harnesses) can list cluster nodes before the filer's
first heartbeat lands on the master. The empty list left FilerAddress
unset and the next command died with "dial tcp: missing address". Poll
ListClusterNodes until a filer appears or the connect timeout expires.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* shell: discover the filer on first use instead of at startup

The startup retry loop made every weed shell invocation wait out the
connect timeout on clusters that have no filer, even for commands that
only talk to the master. Discovery now runs once, on the first command
that actually needs a filer, keeping the retry-until-registered fix for
the fresh-cluster race without stalling master-only commands.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* shell: stop filer discovery retries on non-transient errors

A master that keeps answering ListClusterNodes with a permanent error
(permission denied, unimplemented) cannot gain a filer by retrying, so
discovery returns instead of polling until the deadline.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-23 22:53:43 -07:00
Chris LuandDevin 6a34d82f05 shell: retry filer discovery until master knows a filer (#11989)
* shell: discover the filer on first use instead of at startup

The startup retry loop made every weed shell invocation wait out the
connect timeout on clusters that have no filer, even for commands that
only talk to the master. Discovery now runs once, on the first command
that actually needs a filer, keeping the retry-until-registered fix for
the fresh-cluster race without stalling master-only commands.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* shell: stop filer discovery retries on non-transient errors

A master that keeps answering ListClusterNodes with a permanent error
(permission denied, unimplemented) cannot gain a filer by retrying, so
discovery returns instead of polling until the deadline.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* shell: explain an empty filer list when discovery times out

* shell: retry filer discovery while the address is still empty

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 22:52:51 -07:00
dependabot[bot]anduser.email b86b2f3cee build(deps): bump github.com/go-chi/chi/v5 from 5.3.1 to 5.4.0 (#12004)
Bumps [github.com/go-chi/chi/v5](https://github.com/go-chi/chi) from 5.3.1 to 5.4.0.
- [Release notes](https://github.com/go-chi/chi/releases)
- [Changelog](https://github.com/go-chi/chi/blob/master/CHANGELOG.md)
- [Commits](https://github.com/go-chi/chi/compare/v5.3.1...v5.4.0)

---
updated-dependencies:
- dependency-name: github.com/go-chi/chi/v5
  dependency-version: 5.4.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
2026-10-23 22:47:23 -07:00
57a122fbcf filer.remote.sync: dispatch an update event by the side of the mount it touches instead of slicing the mount prefix off a path outside it (#11958)
* filer.remote.sync: dispatch an update event by the side of the mount it touches instead of slicing the mount prefix off a path outside it

The daemon subscribes to its mount prefix and to /etc/remote. An update event
whose path lies outside the mount still reaches processUpdateEvent: an update
of the /etc/remote directory entry itself (which fs.meta.load of a mapping dump
re-creates) through the /etc/remote prefix, or a rename across the mount
boundary through the path on the inside. processUpdateEvent then maps both
paths to remote keys by slicing the mount prefix off, which panics with
"slice bounds out of range [20:11]" on a path shorter than the mount and names
a key in the wrong tree on a longer one. The event sits in the replay window,
so every restart dies on it and the persisted offset never moves.

Before dispatch, an update event is now mapped onto the side the mount covers:
a rename into the mount is the creation of the new entry, a rename out of it
the deletion of the old entry's object (kept when the entry is remote-only and
the object holds its only copy), an event with neither path inside is not the
mount's and is dropped, and an update with both paths inside is unchanged.

* filer.remote.sync: extract processCreateEvent

Moves the create branch of eachEntryFunc into a function, matching
processDeleteEvent and processUpdateEvent, so the create path can be
exercised directly in tests. No behavior change.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer.remote.sync: upload a create reshaped from a rename into the mount

A file synced under the mount keeps its RemoteEntry across renames. The
rename out deletes the remote object, and the rename back in reshapes to a
create whose inherited stamp still matches the entry, so shouldSendToRemote
skipped the upload and the new key stayed empty. The reshaped create now
sends unconditionally: the stamp describes the old key, not this one.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer.remote.sync: a create reshaped from a rename into the mount writes an empty entry as a zero-byte object

* filer.remote.sync: a create reshaped from a rename sends a stamped entry truncated since its sync

A stamped entry still reports remote-only once truncated to zero (the
stamp's RemoteSize stays positive), so the remote-only exemption skipped
it and the new key got no zero-byte object. Gate the exemption on
isDroppedLocalCopy: only an entry whose size still matches its stamp has
its content already at the new key via the rename's remote copy.

* filer.remote.sync: recover a superseded create reshaped from a rename-in

A create reshaped from a rename into the mount whose write is superseded
was skipped like a plain create, but no later event covers this
destination: remote.uncache accepts the rename's inherited RemoteEntry
stamp and drops the local chunks, and the update it logs skips the
upload through isDroppedLocalCopy, while the source mount's sync still
deletes the source object. The destination object is then missing and
the content exists nowhere.

recoverSupersededCreate finishes the event from the entry the filer
holds now: a rewritten entry is left to its own event's upload or
uploaded here on the stamp edge, and a remote-only entry is completed by
copying the object its stamp describes -- resolved through a
pending-rename marker under this mount, or through the filer's mount
mappings to the source mount's remote location.

---------

Co-authored-by: james <james@medable.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: chrislusf <chris.lu@gmail.com>
2026-10-23 22:43:39 -07:00
Chris LuandDevin 5e01caa522 master: cap parallel vacuums per rack via master.vacuum.max_parallel_per_rack (#11935)
* master: cap parallel vacuums per rack via -maxParallelVacuumPerRack

Hosts that run one volume server per disk under a single rack label could
not bound host-level vacuum I/O: maxParallelVacuumPerServer limits each
process, so every disk on the machine compacted at once. A second quota
map keyed on rack id debits alongside the per-server one, so at most N
volume copies compact in parallel within a rack. Defaults to 0 (no
limit); the flag is wired on master, server, mini and master.follower.

* master: resolve a data node's rack key once per vacuum sweep

A volume server that unregisters mid-sweep loses its parent node, so the
quota credit path calling dn.GetRack() panicked and crashed the master.
Rack names are also local to their data center, so same-named racks in
different centers shared one quota bucket. Each node's rack key is now
captured when the limiters are built and carries the data center id, so
checks, debits and credits never touch the live parent pointer.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* master: reserve vacuum quota per needed copy count, not per copy

Each copy of a volume checked the shared server/rack quota independently,
so several same-rack replicas all passed against one remaining slot and
the debits drove the bucket negative. Needs are now counted per server
and per rack first; a volume whose own copies exceed a cap reserves the
whole bucket once it is free instead of pending forever.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* master.follower: register -maxParallelVacuumPerRack as a flag

It was only a fixed default, so passing the flag on the command line
failed flag parsing.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* master: key the rack vacuum quota by rack node identity

Joining data center and rack names with "/" still let two different racks
share one bucket when the names themselves contained slashes. Key the
limiter by the rack node instead, which is unique per rack regardless of
naming.

* master: keep the tiered-replica vacuum test through the rebase

* master: document 0 as no limit for -maxParallelVacuumPerServer

The clamped per-server need makes a non-positive cap mean "no limit",
matching the new rack flag; say so in the flag help.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* master: key the rack-key cache by data node identity

Node ids are only unique within a rack, so two servers that share an id
in different racks would have reused the first server's cached rack and
escaped the per-rack vacuum cap.

* master: read the per-rack vacuum cap from master.toml

-maxParallelVacuumPerRack becomes the master.max_parallel_vacuum_per_rack
config key, so the new master limit is set the same way as other master
options instead of a flag wired into every command that embeds a master.

* master: let master.max_parallel_vacuum_per_server override the flag

The released -maxParallelVacuumPerServer flag keeps working, but a value
in master.toml now wins when set, matching how master.mdir already
overrides -mdir.

* master: group the vacuum caps under master.vacuum in master.toml

Rename master.max_parallel_vacuum_per_rack to master.vacuum.max_parallel_per_rack
and master.max_parallel_vacuum_per_server to master.vacuum.max_parallel_per_server,
so the keys sit in one subsystem table like master.volume_growth and
master.maintenance.

* master: deprecate -maxParallelVacuumPerServer for master.toml

The flag still works - and is overridden by
master.vacuum.max_parallel_per_server when set - but the help text now
points operators at the config key, matching the codebase style of
help-text-only deprecation.

* topology: read node parent atomically

UnlinkChildNode clears a data node's parent under the rack's lock while
readers like the vacuum sweep's rack capture reach it with no shared
lock, so a torn interface read could crash the master. Store the parent
in an atomic.Pointer and nil-guard the detached walk in
DataNode.GetTopology.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 22:41:36 -07:00
Chris Lu a1b3131bf6 test: handle unsupported catalog clients in S3 Tables CI (#12023)
test: distinguish unsupported catalog clients from S3 Tables failures
2026-10-23 22:40:54 -07:00
Chris Lu aba0cb7b2a ci: isolate S3 lifecycle mini from LAN clusters (#12022) 2026-10-23 22:31:15 -07:00
Chris Lu 81e0859c12 test(s3 remote cache): wait for the master to list the filer before shell commands (#12018)
A fresh single-filer mini registers its filer only after the ~10s
empty-membership confirm window, and weed shell locates the filer via
the master's listing, so remote.configure sent right after the S3
health check races it. Ports the wait from the enterprise test to the
OSS copy.
2026-10-23 20:54:52 -07:00
Chris LuandDevin 0d00e92178 iceberg maintenance: match percent-decoded keys when checking file references (#11978)
* iceberg maintenance: match percent-decoded keys when checking file references

Writers may percent-escape partition directory names in the file paths they
record in manifests (e.g. "part%3D1"), while the S3 layer decodes the key
before storing the object. remove_orphans compared the escaped reference to
the decoded filer path, counted the live file as an orphan, and deleted it
once it passed the age window — readers then got NoSuchKey on the next scan.

Match both spellings of every referenced path in remove_orphans, apply the
same coverage to the kept-file set in expire_snapshots, and delete expired
files at their decoded key so the object the S3 layer stored is the one
removed.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* iceberg maintenance: retry filer lookups at the percent-decoded key

A manifest reference carrying an escaped segment resolves to a filer name
that does not exist, because the S3 layer decoded the key on write. When a
lookup misses and the reference contains escapes, retry once at the decoded
spelling so maintenance can still read the file.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* iceberg maintenance: resolve referenced files verbatim-first, decoded fallback

filer_pb.LookupEntry reports misses as ErrNotFound rather than a gRPC
NotFound, so the escaped-reference retry never fired against a real filer.
Both maintenance reads and expired-snapshot deletes now resolve through one
verbatim-first/decoded-fallback lookup, and expiry refuses to delete a
resolved path outside the table's data directory.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* iceberg maintenance: test expiry of a file stored at a literal percent key

* iceberg maintenance: resolve referenced files in one filer lookup

lookupFileByIcebergPath re-fetched the entry resolveFilerPath had just
looked up, doubling the RPC per referenced file during compaction and
expiration. The resolver now returns the entry alongside the path so a
hit costs one lookup and a decoded fallback two.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* iceberg maintenance: keep every decoded spelling of a referenced path

The kept-reference sets recorded a path plus a single decode of it, but
lookupResolvedEntry decodes recursively until the spelling is stable. A
doubly-escaped reference could resolve to a spelling absent from the set
and be deleted while still referenced. addFilerPathSpellings now walks
the same decode chain the resolver does.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* iceberg maintenance: keep every decoded spelling of a referenced path

* iceberg maintenance: test expiry keeps a doubly-escaped kept reference

* iceberg maintenance: clean decoded spellings in the referenced-path set

* iceberg maintenance: test expiry keeps a kept reference resolved via decoded parent segment

* iceberg maintenance: decode only the reference in the spelling set

Decoding the composed filer path also decoded the table directory, so a
table at a literal percent-escaped path produced a kept set missing the
spelling the resolver returns, and expiry could delete a live file.

* test: gofmt the s3tables cluster helpers

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 20:49:49 -07:00
Chris LuandDevin 8541f959a9 filer.conf: merge fsync on presence like worm (#11953)
* filer.conf: merge fsync on presence like worm

fsync is a pure preference, so OR-merging it meant a nested rule could
never turn a parent's fsync off. Make PathConf.fsync optional and merge
it most-specific-wins, matching worm: an unset rule inherits, an explicit
value overrides in either direction. readOnly and disableChunkDeletion
stay OR'ed — they are fail-closed locks quota enforcement depends on.

Pre-version-2 configs were written with EmitUnpopulated, so every rule
carries an explicit fsync=false that meant nothing; it is dropped to
unset on load so it does not read as an opt-out.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer.conf: test fsync override survives a save/reload round trip

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 20:45:46 -07:00
Chris LuandDevin 673d1a02e0 operation: euthanize pooled upload conns that stall far below a fast host (#12003)
* operation: euthanize pooled upload conns that stall far below a fast host

A pooled keep-alive conn can wedge and drain the request body at a trickle
while fresh conns to the same volume run at full rate; every later upload
routed onto it stalls for tens of seconds, indefinitely. Track per-host
healthy-rate uploads (>=1MiB/s on a >=256KiB body) and, on a host that was
fast within the last minute, cancel an upload still in flight past
max(10s, size/256KiB/s): the transport discards the wedged conn and the
existing retry dials a fresh one. A watchdog kill also clears the fast mark
so the retried attempt runs unbounded, and hosts that never prove fast keep
today's unbounded behavior - a chronically slow link is never penalized.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* operation: keep the stall watchdog armed through the response read

uploadDone was set as soon as Do() returned, so a conn that accepted the
request body then wedged on the reply escaped the watchdog and could
park the upload for hours — the same failure this watchdog exists to
kill. Deferring the done-store keeps the timer live until the function
returns, so a stalled io.ReadAll is cancelled and retried like a
stalled write. The redundant post_err recheck goes with it.

* operation: pin watchdog coverage of a stalled response read

* operation: disarm the stall watchdog for an upload's remaining attempts

Once the watchdog has cancelled an attempt, a concurrent healthy upload
to the same host can restore the fast mark during the retry backoff.
The re-armed watchdog would then cancel the retry too, and a large
upload that merely exceeds the floor could burn every attempt. Track
the watchdog kill on uploadStatusError and carry a
stallWatchdogDisarmed flag through retriedUploadData so the rest of
that upload's attempts run unbounded; other uploads to the host keep
their watchdog.

* operation: pin watchdog disarm on retry, fresh-dial, and slow-upload cases

* operation: drop the send-stall watchdog once the request body drains

A wedged pooled conn is only blameworthy while it is still trickling the
request body; once the body is on the wire the wait is server-side commit,
which the transport's response header timeout already bounds. Cancelling
then would fail a healthy write whose replication or fsync is simply slow.
Track body EOF through a wrapping reader, gate the send-side timer on it,
and re-arm the watchdog only around the response-body read, which the
header timeout does not cover.

* operation: prove a host fast only on a verified upload

A quick rejection moves bytes no faster than the wire did, so mark the
host from the body-send duration and only after the response checks out.

* operation: count an abandoned request body as sent

The transport closes a request body it never finished draining when the
server answers early, so EOF alone under-reports the send boundary and
the stall watchdog could still blame the conn during the reply wait.

* operation: scope the body-sent mark to the current attempt

A lagging Close or EOF from the first attempt's body could stamp the
shared mark after the inner retry reset it, disarming the watchdog for a
send that never happened; each attempt now points the watchdog at its
own sent timestamp.

* operation: drop only the fast mark the stall timer observed

A concurrent upload can re-prove the host while a stalled attempt is in
flight; deleting the key outright would erase that newer mark, so the
watchdog now removes only the value it armed against.

* operation: keep the stall disarm on a per-call upload option

retriedUploadData marked the caller's UploadOption, so a caller reusing it
across UploadData calls would start the next upload with the watchdog off
and lose the cancellation this adds.

* test: stall watchdog retry must fail if the conn gets cancelled

The fake retry slept through any cancellation, so a re-armed watchdog
firing mid-retry still let the upload succeed — the guard was untested.
Park on the request context like a real wedged conn so a re-armed
watchdog cancels the attempt and fails the test.

* operation: prove the fast-host mark with elapsed-to-response

Body-drain time only measures when the transport consumed the reader:
kernel socket buffers can swallow a small body instantly on a slow link,
inflating the rate and marking the host fast. The next upload then gets
its conn cancelled mid-send despite making progress. Elapsed-to-response
bounds remote speed from below — the server cannot answer before the
body lands — so the mark can only under-mark, never over-mark.

* operation: scale the stall budget off the host's measured rate

Fixed bps limits fit no link well: 256KiB/s never fires on a slow link
and waits far too long on a fast one. Record the wire rate of each
sizeable upload and let a later upload run for slowFactor times the
body's span at that rate — a wedged conn shows as an order-of-magnitude
drop on any link speed. The recorded rate uses the wire body size so a
pre-compressed payload no longer inflates the mark.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 20:34:31 -07:00
Chris LuandDevin 6e8a2cdb14 filer.backup: process metadata events through the concurrent processor (#11951)
* filer.backup: process metadata events through the concurrent processor

Each event was handled inline on the subscription loop, so one
fetch+write at a time capped the stream at a few entries per second and
a busy source kept building lag. Route events through the same
MetadataProcessor filer.sync uses: it fans jobs out behind -concurrency
while serializing same-path and subtree-barrier conflicts, and the
checkpoint now advances on the processor's contiguous watermark via
GetResumeTsNs, with Resubscribe replaying anything a failure pins.
-disableErrorRetry keeps its log-and-skip meaning by returning nil after
logging, since the processor would otherwise pin the checkpoint forever.

* filer.backup: drain in-flight jobs before the stream retry loop reconnects

FollowMetadata returns as soon as the stream ends while admitted jobs keep writing; the next doFilerBackup builds a fresh processor that cannot order its replayed writes against them, so an older upload could finish last and clobber a newer file.

* filer.backup: checkpoint the completed watermark on a ticker and at stream end

The offset callback only fires while events arrive, so work draining on a quiet stream stayed uncheckpointed until the next event and a restart recopied it. Persist the processed watermark every 3s when it advances, and once more after the post-stream drain.

* filer.backup: bound the checkpoint wait before reconnecting

setOffset has no context, so a stalled KvPut would hold persistDone and block the retry loop forever. The persist goroutine now performs the final write itself on stopPersist, and the caller waits at most 30s.

* filer.backup: carry the processed position across retry attempts

A follow attempt that ended with a pinned failure recomputed startFrom on retry, so a -timeAgo window could move past the failed event and skip it. doFilerBackup now returns the failure pin (or processed watermark) and the retry loop clamps startFrom to it.

* filer.backup: resume retries at the previous attempt's pinned position

The retry loop threaded resumeTsNs through but never consulted it, so a
-timeAgo run recomputed startFrom as now-minus-the-window on every attempt
— a failed event older than the window was skipped on reconnect. Use the
pinned position when one exists; -timeAgo and the persisted checkpoint
only choose the first starting point.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer.backup: resume below the failure pin so the failed event replays

The metadata reader treats the resume position as exclusive — events at
or below it are skipped — so resuming at the failed event's own timestamp
would skip it. Back one tick off the pin.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer.backup: label the progress line as the received-event rate

The counter increments when a job is admitted, not when its copy
finishes, so the old wording showed backup throughput during a burst of
slow copies while the processed watermark had not moved.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* replication: bound the offset checkpoint RPC

setOffset called KvPut on the background context, so a stalled write
could pin a caller's drain or persistence loop indefinitely.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer.backup: keep the resume pin when filer-source setup fails

A retried attempt carries resumeTsNs so the failed event replays from the
same position, but filer-source and SSE initialization failures returned 0,
so the next attempt recomputed -timeAgo and could move the window past the
pinned event.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* replication: serialize checkpoint writes per key

A checkpoint write can outlive its caller's wait and land beside the next
subscription attempt's writes, moving saved progress backward. Serialize
writes per key inside setOffset and drop any value older than one already
written from this process.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer.backup: gofmt

* replication: serialize checkpoint writes per filer and key

A single process-wide lock made a stalled checkpoint RPC block every
stream's checkpoint, including the opposite direction of a bidirectional
filer.sync. Keeping the last-written watermark per persisted key also let
one stream's offset shadow another stream checkpointing the same key to
a different target filer. Lock and track progress per (filer, key).

* replication: keep the resume position behind a cleared failure pin

A redelivery that clears failedTs lets the processed watermark advance
past the earliest failure, but events dropped while the processor was
stopped — possibly sharing the failure's timestamp — have to replay.
Latched earliestFailedTsNs floors the resume position one tick below
that failure so reconnects still cover them.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* replication: persist floored checkpoints through the write ordering

setOffset drops a value older than the last written, which also swallows
a deliberate regression: the resume floor, an operator-seeded offset, or
a snapshot high-water mark. allowRegress marks those callers so the
checkpoint can move backward when a failure forces replay.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* replication: order checkpoint writes by issuance, not value

The value-based stale guard could never fire — every caller passed
allowRegress because floors and seeds are deliberate regressions — and
ordering by value cannot tell a stale high watermark apart from real
progress: a write that outlives its attempt could still land after a
floored resume and skip events owed a replay. Order by issuance instead:
a write issued before the last one to land under the key is dropped, so
late writes from a superseded attempt can no longer move the checkpoint
in either direction while deliberate regressions still apply.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* replication: keep the resume position one tick below the frontier

* remote.sync: resume and persist through the floored watermark

* test: concurrent metadata jobs fan out up to the limit

Distinct-path events must all be in flight before any finishes — a
serial loop could only ever have one running — while the limit still
bounds the pool. Pins the property behind the backup-throughput fix.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 20:02:55 -07:00
github-actions[bot] cd562b8514 docs: regenerate star history chart 2026-10-24 02:56:42 +00:00
Chris LuandDevin 617599859b docker: build RocksDB v11.1.2 for grocksdb v1.11.1 (#12002)
The rocksdb release image fails to compile: grocksdb v1.11.1 calls
C.rocksdb_block_based_options_set_block_align,
C.rocksdb_block_based_options_set_index_block_search_type, and
C.rocksdb_block_based_options_set_separate_key_value_in_data_block, all
of which only exist in RocksDB v11.x headers. grocksdb v1.11.1's own
build script pins 11.1.2, so move every default from v10.10.1 to
v11.1.2 to match.

Generated with [Devin](https://devin.ai)

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 18:50:27 -07:00
Chris Lu ce4c1fe31e ci: install fuse3 when runners lack fusermount3 (#12013)
Self-hosted runners do not ship fuse3, so the setuid repair step
failed with 'no fusermount3 in PATH' before the FUSE jobs could run.
Install the package instead of failing outright.
2026-10-23 18:49:04 -07:00
Chris Lu 462e4445e2 test: skip catalog client images that cannot execute on the host CPU (#12014)
New upstream releases build for x86-64-v3 (AVX2) - ClickHouse since
26.5, plus the DuckDB iceberg extension and pylance wheels - so on a
runner whose CPU lacks AVX2 the container dies on SIGILL before the
test reaches any SeaweedFS code. Probe each third-party image first
and skip on the illegal-instruction exit instead of failing; other
errors still surface normally.
2026-10-23 18:46:26 -07:00
Chris Lu 0d4e11a6e5 filer: bootstrap a timestamp-less peer entry onto a missing local path (#11975)
A zero local commit time conflated 'no local state' with 'state at the
epoch', so a missing entry was skipped whenever the sender reported no
commit time and the entry mtime was at or before the epoch. Only hold
the copy off when a local entry actually exists.
2026-10-23 18:45:09 -07:00
Chris Lu 85f2f9d2d4 ci: make docker registry and compose steps idempotent on self-hosted runners (#12012)
The registry container is started with --restart=always but never
removed during teardown, so once a machine runs the job, every later
run on that machine fails with a container name conflict. Reuse the
running container when it exists and recreate it only when missing or
broken.

Some self-hosted runners (e.g. seaweedfs-server03) also lack the
Docker Compose plugin, so install it via docker/setup-compose-action,
which is a no-op where compose is already available.
2026-10-23 18:37:36 -07:00
Chris LuandDevin 679835e4fc volume.fsck: run all filer-entry purges through the shared worker pool (#11960)
* volume.fsck: queue per-volume orphan deletes through the shared purge pool

The worker pool added for -reallyDeleteFilerEntries only covered entries
whose volume was missing entirely; the per-volume check that finds filer
entries pointing at absent needles still issued one synchronous HTTP
DELETE per orphan, which is the bulk of the work the report measured.
One pool now serves both purge phases and is drained before the empty-
directory sweep.

* volume.fsck: keep hard link ids in the filer fid file

Orphans queued by the per-volume check must hash into the pool by their
hard link id just like the missing-volume path, so two paths sharing an
id still serialize on one worker. The fid file now stores the id beside
each path and readFilerFileIdFile hands it to the caller.

* volume.fsck: cover orphan purge queuing and hard link id round-trip

* volume.fsck: fail on a truncated hard link id in the fid file

A .fid record cut short inside the hard link id length was emitted with a
nil id, so its purge could hash to a different worker than the entries it
shares the link with. Read the length strictly like the header and path.

* volume.fsck: drain the save consumer before returning a root genFn error

The early return left the save goroutine blocked on outputChan while the
caller ran finishPurge, so a late queuePurge could send on a closed
channel and panic.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* storage: skip EC shard-size encoding tests when disk space is low

TestCalculateExpectedShardSizeWithRealEncoding writes a .dat file plus
14 EC shards per subtest (~1.3GB peak for the 512MB case) into the temp
dir. Shared CI runners can run low on space while other test packages
write to /tmp in parallel, and the encode then dies with ENOSPC
mid-write, surfacing as a test failure. Check free space up front and
skip the subtest instead.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 18:33:51 -07:00
3810fb1123 s3: url-encode ListMultipartUploads results for encoding-type=url (#11998)
* s3: url-encode ListMultipartUploads results for encoding-type=url

The response echoed EncodingType=url but returned raw upload keys, and
XML 1.0 cannot carry control characters, so the encoder replaced them
with U+FFFD. The listed key then matched no upload: AbortMultipartUpload
with it answered 404 NoSuchUpload, leaving such uploads uncleanable.

Encode the key-named elements AWS encodes for this API (Key, Prefix,
Delimiter, KeyMarker, NextKeyMarker) the way ListObjects encodes its
keys when encoding-type=url is requested. Upload ids stay opaque, and
responses without encoding-type=url are unchanged.

Fixes #11991

* s3: echo EncodingType only when encoding-type=url is honored

A request with an unrecognized encoding-type echoed the value back in
<EncodingType> while the keys stayed raw, matching neither AWS (which
omits the element unless url encoding applies) nor the ListObjects
handlers, which gate the echo on the parsed url flag.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 18:21:26 -07:00
Chris LuandDevin 12d0537215 test: gofmt s3tables test files (#12011)
Unformatted test files leave the S3 Tables Format Check red on every
branch until they are re-formatted.

Generated with [Devin](https://devin.ai)

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 18:18:32 -07:00
Chris LuandDevin b2817e853d test: raise mini-server startup wait from 30s to 90s in Makefiles (#12005)
* test: raise mini-server startup wait from 30s to 90s in Makefiles

Self-hosted/persistent runners under load regularly need >30s for
weed mini (master+volume+filer+s3+iceberg+lance+webdav+worker) to
become healthy; hosted runners start in <10s. Siblings already wait
90s (s3/checksum, s3/cors). Fixes the S3 Filer Group first-run flake
on busy self-hosted runners.

* test: align Makefile startup guards and progress totals with 90s bound

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 18:15:01 -07:00
Chris Lu 1e0f571e75 ci: set up Go for CodeQL autobuild (#12007)
The workflow relied on a Go toolchain preinstalled on the runner
image; self-hosted runners without setup-go fail at autobuild with
'go: executable file not found'.
2026-10-23 17:52:50 -07:00
Chris Lu baf0dd680e ec.encode: require shard placement to meet -shardReplicaPlacement (#11947)
* ec: measure shard placement against replica-placement caps

ecShardPlacementViolations reports where a volume's newest-generation
shard placement exceeds the per-node and per-rack caps a
-shardReplicaPlacement requests, using the same digit mapping the
balance planner enforces (SameRackCount = shards/node, DiffRackCount =
shards/rack, dc-qualified rack keys).

* ec.encode: -strictPlacement keeps originals when placement breaches caps

-shardReplicaPlacement tells the balancer the requested per-node and
per-rack shard limits, but nothing verifies the outcome. With
-strictPlacement the pre-delete shard verification also measures the
final placement against those caps: a volume still over the node or
rack limit aborts before its source is deleted, so the encode only
completes when the requested failure-domain spread is actually met.

* ec.encode: cover the strict-placement delete gate with a test

The gate in verifyEcShardsBeforeDelete now runs against a stubbed
topology: a compliant spread passes while an over-cap placement is
refused before source deletion. The retry knobs become package vars so
the test does not wait out the heartbeat-propagation backoff.

* ec.encode: enforce -shardReplicaPlacement without a flag

Drop the -strictPlacement option: the requested per-node and per-rack
caps are always requirements at the pre-delete verification, so the
encode only completes when the placement actually honors them.
2026-10-23 17:29:36 -07:00
f5e319ae8e iam: GetUserPolicy returns NoSuchEntity for a policy name the user does not have (#11944)
* iam: GetUserPolicy returns NoSuchEntity for a policy name the user does not have

When the user has stored inline policies, an unknown name fell through to the
policy rebuilt from the user's actions, so a deleted policy still read back.
The standalone IAM server already returns NoSuchEntity here.

* iam: return service failure when listing inline policies fails

A ListUserInlinePolicies error in GetUserPolicy fell through to
action-based reconstruction, masking a store failure as a policy
document. Propagate it as ServiceFailureException instead, matching
the standalone iam server and the other callers in this file.

* iam: return service failure when the inline policy store read fails

---------

Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 17:08:49 -07:00
Chris Lu f8579bf728 s3: gate POST Object acl fields behind PutObjectAcl (#11939)
* s3: gate POST Object acl fields behind PutObjectAcl

The POST Object handler promotes acl and x-amz-grant-* form fields onto
the upload's ACL headers, and resolveFileMode honors the promoted canned
ACL. The PUT path only applies such headers after an s3:PutObjectAcl
grant; mirror that on POST by running the shared per-key authorization
with the WriteAcp action whenever the request carries ACL headers.

* s3: cover the POST Object acl gate

Signed POST requests carrying an acl form field are denied for a writer
without the WriteAcp action and for a writer a bucket policy denies
s3:PutObjectAcl on the target key; the authorization boundary is
exercised directly as well.

* s3: cover x-amz-grant-* POST fields and the no-ACL case

* s3: encode POST policy conditions as JSON in the acl-gate test
2026-10-23 17:05:44 -07:00
Chris LuandDevin 168789d5f3 master: cap writable volumes per collection via -maxWritableVolumes (#11923)
* master: cap writable volumes per collection via -maxWritableVolumes

Some deployments serialize disk writes (e.g. an Unraid array where
concurrent writes to multiple disks collapse throughput). With more
than one writable volume in a collection, the master's weighted pick
spreads assignments over all of them, so several disks get written at
once.

-maxWritableVolumes=N narrows the pick candidates to the N fullest
writable volumes, so a collection's writes concentrate on one (or few)
volumes at a time. 0 keeps the current spread-everywhere behavior.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* master: give master.follower a maxWritableVolumes default

* master: cap writable volumes inside the requested location

* master: pick the capped volumes without sorting the writable list

* master: assert the capped candidate set instead of random pick outcomes

* master: describe -maxWritableVolumes as a per-layout cap

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 17:04:36 -07:00
Chris Lu 8b26da8c53 ci: gate go/java/python tool caches to github-hosted runners (#12001)
Follow-up to #11985/#11986: setup-go/setup-java/setup-python steps
added or restructured since (e.g. by #11981's shared-artifact layout)
were missed. On self-hosted runners setup-go restores/saves a ~5GB
module cache archive per job — pure waste when the module cache is
already on local disk — and the restore blocks the job for minutes.
2026-10-23 16:25:23 -07:00
Chris Lu cd97c93deb command: keep filer, s3, and sftp up when the local unix socket cannot bind (#11999)
* pb: share the stale-socket tolerant unix listener

ServeGrpcOnLocalSocket already treats the local socket as optional; pull the remove-and-listen block out so the filer, s3, and sftp http sockets can use it too.

* command: keep filer, s3, and sftp up when the local socket cannot bind

A stale socket file owned by another uid (e.g. left in /tmp by a containerized job) made os.Remove fail with EPERM and the Fatalf took the whole server down. The gRPC local socket already treats the unix listener as optional; do the same for the http/sftp sockets.

* filer: clear shared localSocket flag when the unix socket cannot bind

s3.localFilerSocket and (in weed server) the sftp local socket share the
filer's -localSocket flag pointer. If the filer cannot bind the socket it
now keeps running without it, so consumers still holding the configured
path would dial a dead or foreign-owned socket.

* command: keep sftp on its own localSocket flag under weed server

sftpOptions.localSocket was overwritten with the filer's -localSocket
flag, so -sftp.localSocket was ignored and an explicit filer socket path
made the sftp server try to bind the same socket the filer already held.

* s3: fall back to TCP when the filer local socket cannot be dialed

The embedded S3 server copies the filer's -localSocket flag before the filer tries to bind it, so a socket that never comes up left every filer HTTP request dialing a dead path. Try the unix socket first and fall back to the request's TCP address on dial failure.

* filer: keep the shared localSocket flag read-only after startup

Clearing *fo.localSocket raced with the embedded s3 goroutine reading the same flag pointer and could not help a reader that had already copied it. The s3 filer client's TCP fallback covers the unbound-socket case without mutating shared flag state.
2026-10-23 16:07:32 -07:00
Chris Lu 6c87fb70f9 server: skip local unix sockets that cannot be bound (#12000)
server: skip default local unix sockets that cannot be bound

weed server fatally exited during S3, filer, or SFTP startup when the default /tmp/seaweedfs-<svc>-<port>.sock path could not be removed or bound. On shared hosts (e.g. CI runners where another user's weed left a socket in the sticky /tmp directory) a stale socket file turns into 'operation not permitted' and kills the whole server, even though the TCP listeners are unaffected. Warn and skip the unix socket listener instead, matching how the gRPC local socket already degrades. An explicitly configured -localSocket path still fails fast, since a co-located S3 server may be configured to dial the filer through it.
2026-10-23 16:02:33 -07:00
Chris Lu 29b81a827d volume: also listen on localhost like other components (#11995)
* volume: also listen on localhost like other components

Master, filer and s3 bind both the detected host ip and localhost via
NewIpAndLocalListeners, but the volume server used a plain NewListener on
bindIp, so a volume bound to the host ip never answers on loopback. Any
tooling that polls localhost (e.g. the java test workflow's readiness
curl on localhost:8080/status) then waits out its full timeout even
though the volume is up and registered with the master.

Use NewIpAndLocalListeners for the volume grpc, public and cluster http
listeners, and add a small multiHttpdownServer composite so graceful
shutdown still covers both listeners.

* volume: stop and wait dual listeners concurrently

A serial multiHttpdownServer.Stop let the localhost listener keep
accepting requests while the bound-ip listener drained, and a serial
Wait hid a localhost serving failure until the bound-ip listener ended.
Run both waits and both stops in parallel: Wait reports the first error
promptly, Stop closes every listener when shutdown begins.
2026-10-23 15:44:18 -07:00
Chris Lu aeb5db1b13 tests: pin weed mini admin ports and disable unused aux services (#11997)
* tests: pin weed mini admin ports and disable unused aux services

Every weed mini that does not configure them still binds the fixed
default ports for the admin server (12535 + derived 22535 gRPC), its
legacy compat listener (23646 + 33646), WebDAV (7333), Lance (9101) and
the Iceberg REST catalog (8181). Sequential or concurrent minis on one
host then collide on them: the port check warns and shifts, and the
admin's late bind can land on a stolen ephemeral port, adding noise and
flakiness to test logs (e.g. run 40017181958, job 120659267479).

Add testutil.MiniAdminArgs to pin the admin server to an allocated
mini-style port (HTTP + port+10000 gRPC, both below the Linux ephemeral
port floor since admin binds late) and disable the legacy compat
listener. Every test that spawns weed mini now also disables the aux
services it does not use: WebDAV, Lance and the Iceberg REST endpoint
get port=0 unless the test actually exercises them, in which case their
ports stay pinned as before.

* tests: take mini admin ports from the below-ephemeral range

* tests: reuse MiniAdminArgs for the remote-s3 fetch mini

* tests: fully disable webdav and co-allocate mini admin ports

- -webdav.port=0 only makes the service bind a random port; add
  -webdav=false so minis that do not test WebDAV skip it entirely.
- Spark, RisingWave and s3 Spark launchers allocated the admin port in
  a second MustFreeMiniPort call, which could pick a port already
  handed out by the first batch. Fold 'Admin' into the same
  MustFreeMiniPorts call.
- test/fuse_integration is a separate Go module and cannot import
  testutil; allocate the admin ports with its local freePort helper
  instead.

* test/fuse: derive the mini admin gRPC port from its HTTP port

Two independent freePort calls could return the same port, making admin.port equal admin.port.grpc (or collide with the filer port). Allocate once and derive the gRPC port with the +10000 convention.

* test/fuse: pin the mini admin port in the fuse_test framework copy

The fuse_test package framework spawns weed mini too; without a pinned admin port it still binds the fixed defaults 12535/23646 and collides with other minis.

* tests: drop the redundant -webdav.port=0 flag

-webdav=false alone disables the service; the port flag is dead config.
2026-10-23 15:38:09 -07:00
Chris Lu c925869dcc shell: retry filer discovery until a filer registers (#11994)
* shell: retry filer discovery until a filer registers

When -filer is not given, the shell discovers a filer with a single
ListClusterNodes call right after connecting to the master. A filer can
take several seconds to register after its port reports ready, so the
list routinely comes back empty and the shell proceeds with an empty
FilerAddress. Commands then fail with "dial tcp: missing address", which
is how the s3tables Dremio integration test flaked in CI.

Treat an empty list as retryable and keep polling ListClusterNodes until
a filer appears or -connectTimeout expires. On timeout, print a clear
"no filer registered" notice instead of failing later inside a command;
master-only shell commands keep working either way.

* shell: report the last filer discovery attempt failure

A successful empty ListClusterNodes followed by a master outage left the
diagnostic claiming no filer registered, hiding the real RPC failure.
Track the outcome of the latest attempt instead: if the final list call
failed, report that error; only when the last response was genuinely
empty report that no filer registered within the timeout.

* shell: test filer discovery retries against a fake master

Extract the discovery loop into listFilers and cover the retry behavior
with a fake master_pb.SeaweedServer: an empty cluster turning populated,
an empty cluster exhausting the window without error, the last attempt's
RPC failure being reported, and master-only commands still running after
the discovery timeout.
2026-10-23 15:21:41 -07:00
34f06ee92f test/fuse: fix cluster startup in the fuse test framework (#11977)
* test/fuse: fix cluster startup in the fuse test framework

weed volume Fatalfs when -dir does not exist, so every cluster setup
failed its volume wait after 30s. The mount also exited on the unknown
-cacheSizeMB flag (it is -cacheCapacityMB now).

* test/fuse: manage cluster processes and allocate free service ports

Each service now runs through a managed process that captures its log and
reports an early exit from the port wait, instead of a bare 30s timeout.
Ports are allocated free pairs (http + grpc sibling) rather than fixed
19333/18080/18888 so the framework no longer collides with weed mini's
fixed ports or another framework instance.

* test/fuse: bail out of the mount wait when the mount process exits early

The port waits already report an early exit with the captured log tail; the
mount wait still polled the device id for the full timeout, so a mount that
died at startup (bad flag, missing filer) surfaced as a bare 30s timeout
with no diagnosis. Check the managed process the same way and include its
log tail.

* test/fuse: deduplicate service ports handed out by freeServicePort

A probed port is released before the service binds it, so the three
sequential allocations for master, volume and filer could all return the
same still-unbound port and leave later services dying at startup. Track
handed-out ports so each allocation is unique.

* test/fuse: allocate service ports in Setup after the external-mount branch

* test/fuse: verify service readiness through its own /healthz endpoint

* test/fuse: bound the readiness probe so a silent listener cannot hang setup

* test/fuse: only accept a ready response while the service process still runs

* test/fuse: include the service log tail on readiness timeout

A startup that stalls instead of exiting still failed with a bare
"not ready within timeout" — the same opaque message that made the
original flake hard to diagnose. The process-exit path already tails
the captured service log; the timeout path now does too.

* test/fuse: report pjdfstest category results without failing the suite

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: allow allow_other mounts in posix-compliance

The job mounts with -allowOthers but never sets user_allow_other in
/etc/fuse.conf, so fusermount rejects the option and the mount exits
before the suite runs. The other FUSE workflows set it; do the same.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test/fuse: keep the standalone-framework service logs on failure

Cleanup removed the temp dir holding master/volume/filer/mount logs, so
a CI failure left nothing to diagnose. Dump the tails on failure and
copy them under /tmp/seaweedfs-fuse-logs/standalone/ for the artifact
upload, next to the mini framework's logs.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test/fuse: gate setup on a real write through the mount

Mount readiness only proves the daemon is serving; a fresh cluster still
cannot assign a file id until a volume heartbeats in and a grow round
finishes, so the first ~10-30s of writes surface as EIO on close. Probe
with a real write/read/remove before handing the mount to the suite.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test/fuse: gate the mini framework setup on a real write too

Same cold-start window as the standalone framework: mini's volume has to
register and grow before a write can be assigned. Keep the two
frameworks consistent so later suites do not flake on the same stall.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test/fuse: save each framework's CI logs under its test name

Fixed-name mini.log/mount.log in /tmp/seaweedfs-fuse-logs let a later
framework instance overwrite an earlier test's logs, so an uploaded
artifact could describe a passing run instead of the failure under
investigation. Apply to both framework copies since they share the
directory.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test/fuse: bound each waitForWritable probe attempt

A wedged mount can hold a file op open past the 90s deadline, so the
loop could not check the deadline or a mount exit and Setup hung until
the overall test timeout. Each attempt now runs off the loop and is
abandoned when the remaining time runs out. Same fix in both framework
copies.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test/fuse: keep one write probe in flight until it finishes or the deadline

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: chrislusf <chris.lu@seaweedfs.com>
2026-10-23 08:10:31 -07:00
Chris Lu 2c8079625c test: stop EC integration tests from pinning default master port 9333 (#11988)
The EC integration tests failed in CI with "raft.Server: Not current
leader" on every assign for tens of seconds — longer than any goraft
election takes. A stale weed process on the self-hosted runner was
still holding 9333, so each test's fresh master died at bind time and
the tests kept talking to a raft non-leader. Only the 9333 cluster
(startSeaweedFSCluster) failed; tests on 9334-9338 all passed.

Allocate mini-safe ports (http + grpc+10000 both verified free) for the
master and volume servers instead of hardcoding 9333/8080-8085, wait
for the master to actually report raft leadership via /cluster/status
before issuing raft-gated RPCs, and make waitForServer a real TCP dial —
grpc.NewClient never connects, so the old check always passed
instantly.
2026-10-23 08:08:49 -07:00
bati-valensasandChris Lu c4002fafc7 helm: fix bucket hook treating an existing bucket as missing (#11982)
* helm: fix bucket hook treating an existing bucket as missing

The skip check pipes awk into `grep -q`, which exits on the first match and
can leave awk writing into a closed pipe. awk then dies of SIGPIPE and
`set -o pipefail` makes the pipeline report that, so a bucket that exists
reads as missing: the hook runs `s3.bucket.create`, it errors with "already
exists", and `sh -ec` aborts the hook - failing the release.

Match in a single awk instead, which reads to EOF and cannot be killed by a
closing reader.

* helm: force string compare in bucket hook existence check

---------

Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-23 06:43:35 -07:00
Chris LuandDevin 384523761b ci: skip actions/cache on self-hosted runners (#11986)
cargo registry/target and docker-image caches are already local on a
persistent self-hosted runner; the save/restore pass is pure overhead
(the setup-go cache-save stall on awscli-tests was the same pattern).
Gate every actions/cache step on runner.environment == 'github-hosted'.

Generated with [Devin](https://devin.ai)

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 06:19:56 -07:00
Chris LuandDevin 277c9809c8 ci: skip setup-go cache on self-hosted runners (#11985)
* ci: skip setup-go cache on self-hosted runners

The module cache is already local on a persistent self-hosted runner;
archiving ~/go/pkg/mod into cache.tzst at every job's end is pure cost
and can stall (a ~4-minute zstdmt hang was observed on awscli-tests).
runner.environment gates the cache to GitHub-hosted runners only.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: gate the remaining setup-go caches too

The first pass only covered go-version-file steps; go-version steps in
nine workflows (including awscli-tests' own file) kept uploading the
cache archive on self-hosted runners.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 06:05:33 -07:00
Chris LuandDevin 728fc69976 ci: cache docker build layers for container_dev (#11980)
* ci: cache docker build layers for container_dev

The Go-build image rebuilds its layers on every run; type=gha backend
caching reuses them across runs and platforms.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: pin container build to the pushed sha under layer caching

Dockerfile.go_build clones master inside the image; a cache-from layer
would otherwise keep serving the previous commit's clone and publish
stale code under the dev tag. Passing BRANCH=github.sha makes the clone
layer's key commit-specific while toolchain layers still hit.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: pass BRANCH via build-args so the per-commit cache key reaches buildx

docker/build-push-action has no `args` input — the BRANCH override was
silently dropped, leaving the build arg at its `master` default and the
clone/checkout layers serving a stale commit under the gha cache.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 06:02:08 -07:00
Chris LuandDevin 9b59bc6b0d ci: build test binaries once, share them across shard jobs (#11981)
* ci: build test binaries once, share them across shard jobs

Each of the 6 rust-volume-go-tests shards, the rust-integration-tests
job, and the 6 volume-server-integration-tests shards rebuilt the same
weed and weed-volume binaries — 13 duplicate builds per run. A
build-test-binaries / build-weed job now compiles them once under the
existing cargo cache and publishes them as artifacts; the consumers
download and chmod them instead of building.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: preserve binary paths through the artifact round-trip

upload-artifact stores single files flat, so weed/weed would come back
at the workspace root and the consumers' expected paths break. Pack the
binaries in a tar before upload and extract on download.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: extract the shared test binaries in the rust go-test shards

The download step leaves rust-test-binaries.tar at the workspace root;
without untarring it the chmod fails on missing files and every shard
fails before running a test.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 04:50:29 -07:00
Chris LuandDevin f7af816ab4 ci: cancel superseded workflow runs on PR pushes (#11979)
* ci: cancel superseded workflow runs on PR pushes

PR-triggered workflows without a concurrency group keep running (and
holding runner slots) after a force-push or quick follow-up commit
supersedes them. Add the same group/cancel-in-progress block the go
workflow already uses, keyed on head_ref, to the remaining PR workflows.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: key concurrency groups on PR number or ref, not head_ref

github.head_ref is empty on tag pushes and manual runs (every run then
shares one group and cancels the last) and collides across forks when
two PRs share a branch name. pull_request.number is unique per PR;
github.ref covers push/tag/dispatch. Applied to all workflows using the
pattern, including pre-existing ones.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: restrict cancel-in-progress to PR and branch-push runs

* ci: serialize container_latest manual runs by destination tag

Two workflow_dispatch runs sharing an image_tag could otherwise build
concurrently: their per-arch images publish to the same platform tags,
so one run's create-manifest job can combine platforms from different
source refs. Group dispatch runs by the destination tag (with
cancel-in-progress already disabled for non-PR/non-branch events) so
same-tag publishes queue instead of interleaving.

* ci: serialize manual publish runs by destination tag in remaining container workflows

* ci: group foundationdb release dispatch runs by release tag

A manual run now shares the tag-push concurrency group for the same
release, so a re-publish of a tag queues behind (or ahead of) the push
run instead of racing it on the same image tag.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: serialize foundationdb_version dispatch runs by destination tag

Two manual runs resolving to the same docker tag now share a concurrency
group (explicit image_tag, else the computed
foundationdb_<fdb>_seaweedfs_<ref> default), so they queue instead of
racing pushes to the same tag.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: drop unsupported toLower() from foundationdb_version concurrency group

* ci: key rocksdb_version dispatch group on its published tag

* ci: queue publish runs via turnstyle instead of cancel-free shared groups

* Revert "ci: queue publish runs via turnstyle instead of cancel-free shared groups"

This reverts commit 039148f8d5.

* ci: queue same-tag publish runs instead of replacing the pending one

Dispatch/tag-only workflows can never cancel in progress, so opt their
concurrency groups into queue: max — up to 100 same-tag publishes wait
in line rather than each new dispatch cancelling the pending one.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 04:49:52 -07:00
Chris Lu 7c478fd89a fuse integration: run posix suites on weed mini and allow allow_other mounts (#11984)
The separate master/volume/filer processes in framework.go bound fixed
ports, discarded child output, and never noticed an early process exit,
so a slow or dead volume server burned the whole 30s timeout with no
logs. Switch to the same weed-mini approach the fuse_test package
framework already uses: dynamic filer ports, child output captured to
log files, and readiness waits that bail out the moment the process
dies. The external-mount path (TEST_MOUNT_POINT) is unchanged.

The posix-compliance workflow mounts with -allowOthers, but fusermount
refuses allow_other unless user_allow_other is set in /etc/fuse.conf,
mirroring the fuse-integration workflow FUSE setup.
2026-10-23 04:33:14 -07:00
ad89f003a5 s3: end a catalog-vended session with the access key it was vended for (#11945)
* s3: end a catalog-vended session with the access key it was vended for

A table credential the Iceberg catalog vends is a self-contained STS session:
it stays valid until it expires (an hour by default) even after the access key
that loaded the table is deleted or deactivated, or its identity is removed
or disabled. An operator revoking a leaked key cannot cut off the table
credentials that key already obtained.

Sessions are stateless JWTs, so there is no revocation list to consult. This
change follows the same pattern as role sessions, which are already refused
once their role's ID no longer matches (checkSessionRoleBinding): a vended
session carries the access key it was vended for (src_ak, recorded by the
previous change), and validateSTSSessionToken checks it against the identity
table held in memory. A session of provider iceberg-catalog that carries a
source key is refused (InvalidAccessKeyId, as for an unknown key) once its
identity is gone or disabled, or the key is no longer one of that identity's
active, unexpired credentials.

Scope:
- Only catalog-vended sessions are checked; web-identity, AssumeRole and
  every other STS session are validated as before.
- The vendor binds a session to the caller's key only when the caller is a
  stored identity holding that key. A caller that signed with another STS
  session's key, an OIDC user, or a request that recorded no key gets an
  unbound session, as today; binding it would refuse every request made with
  it. A session minted before this change carries no key and is unbound.
- The check is one map lookup under the read lock and a scan of one
  identity's keys (about 11 ns, 0 allocations, with 1,000 identities).

Tests (s3api_iceberg_vended_session_test.go) mint real sessions through the
catalog vendor and validate them the way a data-path request is validated:
deactivating, expiring or deleting key A ends A's session while the session
vended through key B on the same identity keeps working; disabling or
deleting the identity ends it; callers that are not a stored identity holding
the key get unbound sessions that keep working; other STS sessions are not
checked. The first two tests and the third fail without this change.

* s3: keep the vended-session key binding when the key dies before vend

vendedSessionKeyLive gated the binding, so a key revoked between the
request's authentication and the vend skipped it and minted an unbound
session that stayed usable until it expired. Every catalog auth path
pairs the identity name in context with the key that authenticated, so
a stored-identity principal always means the request signed with that
identity's own key. Bind whenever the principal is a stored identity: a
session bound to a now-dead key is refused on its first request, which
is what the binding is for.

* s3: gate the vended-session key binding on the authenticated caller type

lookupByIdentityName(principal) named only that some identity exists, not
that the request signed with a stored key: an STS session or OIDC principal
that collides with an identity name got a session bound to its temporary
key, which vendedSessionKeyLive then refused on every request. Verifiers
now record the verified key as a stored credential, and the vendor binds
only when that marker is present — keeping the binding even when the key
was revoked between authentication and the vend.

* s3: enforce the vended key binding under a dedicated claim

Sessions minted before the binding existed carry the caller's key in the
audit-only SourceAccessKeyId — including temporary STS keys that can never
resolve in the identity table. Enforcing on that field would end live table
credentials at upgrade; the binding now travels under BoundAccessKeyId,
leaving legacy sessions unbound as before.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: bind vended sessions to a credential fingerprint

The key id alone cannot identify a credential: an operator can delete a
key and re-create it under the same id with a different secret, reviving
sessions minted for the original. Verifiers now record a fingerprint of
the secret at authentication; the session dies unless it still matches.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: keep recording the caller's key on every catalog session

SourceAccessKeyId was unconditional audit metadata before the binding
existed; gating it on the stored-credential marker dropped the record
for callers that authenticated another way. With the binding under its
own claim there is no ambiguity left, so the field records the caller's
key again regardless of how it authenticated.

* s3: key the vended-session credential fingerprint with the signing key

The bound-credential fingerprint travels inside the vended session
token. As a plain SHA-256 of the secret it gave any token holder an
offline guessing oracle for weak secrets, and CodeQL flags it twice.

Store the verified secret in the request marker instead and HMAC it
with the STS signing key at vend and at validation. The claim can no
longer be mined, and the per-request auth path stops hashing secrets.

---------

Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 03:58:06 -07:00
Chris LuandDevin bf3081e23a s3 lifecycle: route version-file writes to current-version expiration (#11956)
* s3lifecycle: schedule ExpirationDays from version-file writes

On a versioned bucket a PUT lands in the meta-log as the version file
plus the .versions/ directory update — there is no bare-key event, so
the version-folder skip dropped every version-file event for all kinds
but ExpiredDeleteMarker. ExpirationDays therefore only fired via the
recovery walker's cold-start / rule-change pass: once a shard had a
saved cursor, no daily run ever scheduled another delete marker.

A version-file write is the latest for its logical key at write time,
so route it through the same event-driven evaluation the regular path
runs. The match carries the file's identity with an empty VersionID —
the dispatcher resolves the current latest at due time, and a fresher
PUT meanwhile resolves to STALE_IDENTITY there.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* lifecycle test: versioned Expiration fires on the second daily run

Establishes the shard cursor with a first pass, then PUTs and backdates
a new versioned object and runs again. The delete marker must appear
without a rule change forcing a recovery walk.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: run TestLifecycleVersionedBucketExpiresOnSecondRun in the lifecycle matrix

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 03:19:41 -07:00
Chris LuandDevin 2e7e691bed volume: drop stale index entries that resolve to a different needle (#11948)
* volume: drop index entries that resolve to a different needle

An index row whose offset lands on a needle with a different id and a
different size cannot have been produced by any write path: every appender
indexes a key at the offset and size of the record it wrote, and
WriteNeedleBlob keeps its indexed size consistent with the blob header.
Such rows are corruption (for example an offset replayed with stale high
bytes) and until now made the volume reject every write for the key with
"mismatching cookie" while reads resolved to the wrong needle.

When the duplicate check finds the indexed record belongs to a different
key and size, drop the stale entry and let the append re-index the key.
Same-key conflicts still fail the cookie check, and entries staying
consistent with a re-keyed blob keep their existing semantics.

* volume: verify the indexed needle header before streaming reads

The buffered read path re-parses the record and rejects a needle whose
stored size disagrees with the index, but the streamed path serves data
bytes without ever looking at the header, so a stale index row would
stream another needle's payload. Probe the header once up front and
report a miss when the record is neither the requested key nor the
indexed size, the same staleness test the write path applies.

* rust volume: drop index entries that resolve to a different needle

Mirror of the Go volume change: when the duplicate check finds the
indexed record belongs to a different key and size, delete the stale
entry and let the append re-index the key.

* volume: overwrite a stale index entry on commit, not up front

Deleting the stale row during the duplicate check published a tombstone
before the replacement append and, for durable writes, before its fsync.
A failed append or sync then rolled back the data but left the key
deleted. The publish step already overwrites any older offset, so leave
the entry in place and let it be replaced once the record commits.

* volume: report streaming header-probe failures to the io error tracker

A failed header read returned before the tracked body-read loop, so
repeated EIO on the probe never reached checkReadWriteError. Skip io.EOF
the way the loop does: it is a truncated record, not a disk fault.

* rust volume: overwrite a stale index entry on commit, not up front

Mirror of the Go volume change: the staged delete published a tombstone
before the replacement append and its .dat flush, so a failed append or
sync left the key deleted. publish_write already overwrites any older
offset once the record commits.

* rust volume: report needle meta read failures to the io error tracker

read_meta is the header probe ahead of a streamed body: an EIO there
escaped check_read_write_error entirely, so a failing disk could not
trip the tolerance. Report the outcome the way read_full already does.

* volume: retry the wrapped offset before judging an index entry stale

A 4-byte index value can index past the 32 GiB wrap: the low offset may
hold another needle while the real record sits at +MaxPossibleVolumeSize,
which is where reads already retry on a size mismatch. Skipping the
cookie check on the low-offset record alone would accept a foreign-cookie
write over a live key. Retry the wrapped header read first, the same
condition the read path uses; a wrapped read that fails for reasons other
than EOF stops the write like the low-offset read does.

* volume: probe the streamed needle header under the access lock

With HasSlowRead the access lock is dropped after the index lookup, so a
compaction commit could swap the backend between the lookup and the
header probe and fail a valid download on the old offset. Take the read
lock for the probe and resolve the compaction revision first, the way
the streaming loop re-resolves it on every page.

* rust volume: retry the wrapped offset before judging an index entry stale

Mirror of the Go volume change: probe offset + MAX_POSSIBLE_VOLUME_SIZE
under the 4-byte-offset build before treating a mismatched record as
stale, so a live wrapped entry still gets its cookie checked.

* rust volume: do not clear the io error streak on a successful meta read

read_meta runs before the payload read in read_needle_for_get, so recording success there reset the streak before every failing download and the volume could never reach quarantine. Report failures only; read_full still records success after a complete read.

* volume: record a meta-only read's success against the io error streak

A meta read no longer clears the streak because a payload read still
follows on GETs, but requests the meta read ends outright — HEAD answers
and cookie mismatches — never reached the body path, so their success
left earlier failures counted. Note the success on those terminal paths;
mirrored in the Rust volume server.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: report streamed reads to the io error streak

Streamed GETs never touched the tracker: a failing chunk read went
uncounted and a completed download never cleared the streak, unlike
readNeedleDataInto which does both.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: reset the io error streak only at terminal meta reads

A streamed GET's metadata read runs before its payload read, so clearing
the streak there erased the count before every failing download and the
volume could never reach quarantine. Record the meta-read success only
where the request actually ends: must-meta-only reads, HEAD responses,
and cookie mismatches — matching the Rust volume server's placement.

* rust volume: record the streamed read's success at the final frame

Hyper can complete a Content-Length response without polling the body
again, so a success recorded only when the stream is next polled could
be skipped, leaving an earlier io error streak in place after a
successful download.

* volume: record a meta-only read's success on conditional 304 replies

* rust volume: record a meta-only read's success on conditional 304 replies

* volume: reset the io error streak only after the checksum passes

A streamed read that ends in a CRC mismatch still cleared the EIO
streak, so corruption could mask progress toward quarantine. The Rust
side records success only after the final frame's checksum verifies;
move the reset below the CRC check to match.

* volume: satisfy codespell in the meta-read comment

* rust volume: report range reads to the io error streak

* rust volume: cover the range read's io error reporting

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 02:30:53 -07:00
Chris LuandDevin 27fa285d48 filer: do not stall startup waiting out the leader-settle window (#11972)
* filer: do not stall startup waiting out the leader-settle window

ListExistingPeerUpdates re-polled the leader for up to 10s when it
reported zero filers. A fresh single-filer cluster can never gain peers,
so every filer start paid the full settle timeout before listening —
past the 6s health window in weed mini and short probe budgets.

Take the leader's first answer instead; the fallback fullest-answer
scan below still covers a leader that is mid-convergence.

* filer: bound leader-settle re-poll below the startup health window

* filer: update ListExistingPeerUpdates doc for empty-leader fallback

* filer: re-check empty membership at the announce gate before registering

A fresh filer that scans empty membership during a master restart or
re-election skipped bootstrap and announced immediately, serving missing
files until a peer happened to reconnect — and a late peer only got an
incremental subscription, so pre-existing metadata was never backfilled.

The announce gate now re-polls ListExistingPeerUpdates while the scan can
still be disproved (10s), backfills from any late peers before their
streams start, and then waits for their catch-up like startup peers. A
genuinely empty cluster pays a delayed master listing rather than a
stalled startup, since this runs after the ports already serve.

* filer: overlap the late-peer copy cutoff and stream start

A peer event stamped just behind this filer's clock could land after
the traversal read but carry a timestamp below the stream start —
the startup scan margins both cutoffs by LogFlushInterval for exactly
this skew; mirror it for peers discovered at the announce gate.

* filer: bound the late-peer bootstrap copy by the shared announce deadline

MaybeBootstrapFromOnePeer now takes the caller's context so a stalled
peer metadata copy cannot out-wait the announce cap, and the bootstrap
re-check shares one announceCatchUpCap deadline with the catch-up wait
instead of each spending a full minute.

* filer: merge bootstrap entries by commit time instead of blind insert

The late-peer copy runs while gRPC already serves, so a blind InsertEntry
could replace a write the local filer just accepted. Routing each entry
through applyTraversedEntry orders it against the local commit time under
the replay locks — a fresher local write survives the backfill.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 02:04:27 -07:00
Chris LuandDevin b65832bdc9 iam/oidc: reject reused jti claims with a bounded replay cache (#11936)
* iam/oidc: reject reused jti claims with a bounded replay cache

* iam/sts: pass jti replay settings through convertToOIDCConfig

* iam/oidc: keep consumed jtis and stop the cleanup worker across re-init

Re-initializing a provider reset the JTI store, so an already-used token
could be presented again after a config refresh; the store is now
allocated once and survives Initialize. The cleanup worker is stopped on
every re-init (previously only when protection stayed enabled), is
started only after the rest of Initialize has succeeded, and Shutdown no
longer consumes a permanent sync.Once, so a re-initialized worker can be
stopped again.

* iam/oidc: enforce the jti replay cache bound for every token

Tokens without a usable jti claim skipped the cache entirely and stayed
reusable; they are now keyed by the token's own fingerprint so a repeat
presentation is still a replay. Tokens without exp previously failed as
expired on first use; their entry now gets a bounded retention. The
replay test-and-set now runs before any capacity work so replayed tokens
cannot each trigger a full-cache scan, the entry count is incremented as
part of the insert so concurrent validations cannot overshoot the bound,
and a guaranteed serialized sweep runs before a capacity rejection while
opportunistic sweeps are rate limited and confined to one at a time. Raw
token identifiers are no longer written to logs or errors, a stable
fingerprint is logged instead.

* iam/sts: shut down providers dropped by the managed-provider swap

Each managed OIDC provider runs a background JTI cleanup worker, but
SetIAMManagedOIDCProviders replaced the provider map without stopping
the old providers, so every refresh leaked goroutines. Providers absent
from the new map are now shut down; providers that stay keep running.

* iam/integration: reuse providers across refreshes for unchanged records

Every refresh rebuilt providers from their records, so a refreshed
provider forgot the JTIs it had already consumed and an already-used
token could be presented again. Records are now fingerprinted and an
unchanged record reuses its provider, keeping the consumed-token cache;
changed records are rebuilt, removed records are dropped, and the swap
in STS releases the workers of whatever is no longer referenced.

* iam/oidc: consume replay entries only on the single-use exchange path

Bearer authentication validates the same token on every request, so
consuming it in shared ValidateToken/Authenticate made every token
single-use and rejected all but the first request. The replay check now
runs only for the STS exchange via AuthenticateSingleUse; bearer paths
validate without consuming. A token without exp also no longer gets a
bounded replay entry that ages out while the token stays valid: its
entry is kept until the capacity bound evicts it, so it can never become
usable again.

* iam/oidc: keep the mock provider on the mock path for single-use calls

* iam/oidc: require exp on single-use tokens

A cache slot frees only by expiry, so an expiry-less entry would hold it
forever; a burst of distinct tokens could exhaust the bound permanently
and block every later exchange.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: give each web-identity test token a unique jti

Subtests minted tokens with identical claims inside one second, so the
exchange saw the same token fingerprint twice and the second call read
as a replay. Each exchange now carries its own jti.

* iam/oidc: carry the consumed-token replay store across provider rebuilds

A record change rebuilds the managed provider with a fresh JTI store, so a token consumed before the rebuild could be presented again until it expired. The new provider adopts the replaced provider's replay store before it is published, and the replaced provider is shut down so only one cleanup worker sweeps the shared store.

* iam/oidc: stop the cleanup worker of providers dropped from the store

A refresh removed the provider object from the managed cache but left its JTI cleanup goroutine running for the life of the process.

* iam/oidc: merge a provider's consumed replay entries into another's store

MergeReplayStateFrom folds one provider's consumed-token entries into
another's, skipping already-expired ones. Unlike AdoptReplayStateFrom it
swaps no pointers, so it is safe on a provider still serving validators.

* iam/integration: share one provider and replay store across identical issuer records

Records for the same issuer URL with identical validation settings now
reuse a single provider, so a token consumed through one account's
record is rejected through the other's — previously each record owned a
separate jti store and a spent token could be replayed cross-account.

A superseded provider's consumed entries merge into the shared store so
no rejection is lost, and a removed or rebuilt provider is only shut
down once no remaining record references it.

* iam/oidc: hand the superseded provider's live replay store to the shared one

* iam/oidc: carry merged replay entries without charging the capacity bound

* iam/oidc: key the jti-less replay fingerprint on the signed input

Hashing the whole token lets ECDSA signature malleability (s -> N-s)
mint a second cache key for identical claims, so a consumed token could
be replayed for another exchange. Key the fallback on header.payload
instead.

* iam/oidc: publish the jti cache bound before the enabled flag

Initialize stored jtiEnabled ahead of jtiMaxSize, so a provider flipped
from disabled to enabled exposed enabled=true with a zero bound for a
window; a single-use validation landing there would compute
newCount > maxSize and reject with "JTI cache capacity exceeded". Store
the bound first so the flag is the last thing published.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 00:46:14 -07:00
Chris LuandDevin 186bc5cf7f s3: bound ListMultipartUploads memory to the page size (#11971)
* s3: bound ListMultipartUploads memory to the page size

listMultipartUploads loaded every open upload in the bucket into a slice
and selectMultipartUploads filtered and sorted a second full copy, so one
request cost O(total open uploads) heap regardless of prefix, key-marker,
or max-uploads; concurrent calls multiplied it and could OOM the gateway.

The response is ordered by the object key kept in extended attributes, not
the upload id the filer sorts by, so every entry still has to be seen.
Streaming the listing into a bounded top-K collector keeps at most
maxUploads (capped at maxUploadsList) candidates, bounding heap by the
page instead of the folder. A caller-supplied max-uploads above the
server cap now also paginates rather than accumulating unboundedly.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: cover bounded ListMultipartUploads accumulation

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: report the effective max-uploads cap in ListMultipartUploads

* s3: cover ListMultipartUploads retry through the collector

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 00:46:10 -07:00
Chris LuandDevin 7154732b84 worker/ec: retry the pinned volume copy after a mid-copy compaction (#11929)
* worker/ec: retry the pinned volume copy after a mid-copy compaction

The EC task pins .idx/.dat copies to the CompactionRevision read from
ReadVolumeFileStatus so the pair comes from one volume generation. When
vacuum compaction lands between that read and either CopyFile, the
server rejects the copy with "volume N is compacted" and the whole task
fails — on clusters with queued maintenance this makes most EC tasks
die to a race they cannot avoid. On a compaction mismatch the copy now
restarts under a freshly read revision, up to a bounded number of
attempts; other errors still fail immediately.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* worker/ec: read the retried copy's expiry from its own file status

A mid-copy compaction makes the source reload the volume, and the fresh
ReadVolumeFileStatus can carry a different expireAtSec than the
pre-compaction read. Keeping the attempt-0 expiry could let the EC
volume outlive its source TTL.

* worker/ec: cover the compaction retry with a stub volume server

* worker/ec: also observe revised .idx bytes in the compaction retry test

* worker/ec: read the fake source under its mutex in the retry test

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-22 21:32:57 -07:00
Chris LuandDevin f2d5c27f65 operation: renew the write JWT when a fan-out rollback delete is rejected (#11970)
* operation: renew the cleanup JWT when a fan-out rollback delete is rejected

A stalled holder can out-wait the assignment JWT's 10s expiry, so the
type=replicate delete that rolls back landed copies comes back 401 and the
needle leaks. Retry the delete once with a fresh token minted through the new
LookupJwtFunc hook when a holder answers unauthorized.

* s3api: wire a cleanup-JWT renewer into chunked uploads

Keep the master client on the server and use it to mint fresh volume JWTs for
rollback deletes that run past the assignment token's expiry.

* admin: wire a cleanup-JWT renewer into file-browser uploads

* operation: bound the cleanup-JWT renewal with a per-attempt deadline

A rollback delete that hits 401 renewed the token through a background
context, so a master outage could park the wait for a connection and the
lookup RPC inside the failed upload's cleanup path. Thread a context
through LookupJwtFunc and LookupJwt, and bound each renewal with a 10s
deadline covering both the master wait and the lookup call.

* operation: let every holder's 401 retry with a renewed cleanup JWT

A single renewed flag meant only the first rejecting holder refreshed the
token; a holder that stalled past the renewed JWT's lifetime left later
copies stranded with a 401 they could not resolve. The retry limit is now
per holder — each delete is attempted once with the current token and once
more after a bounded renewal — so a slow middle holder cannot strand the
rest.

* operation: log the volume server's error body on a failed cleanup delete

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* operation: count test upload attempts atomically

The chunk upload loop fans AssignFunc out to parallel goroutines, so the
plain int counter in the partial-results test raced under -race.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-22 21:32:53 -07:00
github-actions[bot] b97ed0f101 4.49 2026-10-23 03:49:00 +00:00
github-actions[bot] 13cbe78537 docs: regenerate star history chart 2026-10-23 02:28:25 +00:00
Mohd Quamar Tyagi 94dd96a322 weed filer.remote.sync: ignore .system directory under buckets (#11966) (#11969) 2026-10-22 18:40:30 -07:00
Chris Lu 00e64441c6 fix: clone protobuf snapshots without copying internal locks (#11968)
* fix: clone protobuf snapshots without copying internal locks

* fix: reuse cloned snapshot fields when filtering
2026-10-22 18:38:33 -07:00
Chris Lu f08f075d23 deps: patch two open dependabot alerts (klauspost/compress, aws-smithy-json) (#11959)
* test/kafka: bump klauspost/compress to v1.20.3 in simple-publisher

Patch an s2 integer-overflow / out-of-bounds write affecting
klauspost/compress <= 1.18.6; matches the version the main module
already uses. The go directive rises to 1.25, still under the ^1.26
toolchain the kafka test workflows run with.

* seaweed-volume: bump aws-smithy-json to 0.62.7

Patch an uncontrolled-recursion denial of service in the unknown-key
skip path of aws-smithy-json <= 0.62.6 (reachable over the S3 SDK's
response parsing). The smithy runtime family moves together:
aws-smithy-runtime-api 1.12.3 and aws-smithy-types 1.5.0, all selected
within the workspace's Rust 1.91.1 compatibility bound.
2026-10-22 14:38:16 -07:00
Chris LuandDevin 7c980774e7 FUSE mount: add posix compliance reports (#7183)
* fuse integration: add POSIX compliance test suite

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: add posix-compliance workflow

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fuse integration: skip non-executable weed path candidates

* fuse integration: size readv buffer to the written segment

* fuse integration: reference posix_Makefile explicitly in recursive make calls

GNU make does not propagate -f through MAKEFLAGS, so bare $(MAKE) re-read the default Makefile in this directory.

* fuse integration: fail posix test targets when go test fails

Every suite pipes go test through tee under /bin/sh, so the recipe status was tee's and test failures stayed green. Run recipes under bash -o pipefail and emit the color escapes with printf since bash echo does not expand them.

* fuse integration: run only benchmarks in benchmark-posix

go test -bench=. without -run also executes the whole suite, mixing compliance output into the benchmark log.

* fuse integration: emit single-number test counts in reports

grep -c prints 0 and exits nonzero when nothing matches, so '|| echo 0' produced two lines that broke the expr arithmetic and JSON fields. awk prints exactly one count either way.

* fuse integration: skip the sendfile test on macOS

macOS sendfile(2) only accepts a socket destination; the test copies into a regular file and always gets ENOTSOCK.

* fuse integration: honor TestConfig and own only self-started resources

NewFuseTestFramework dropped its config argument and Cleanup read a fresh
DefaultTestConfig, so SkipCleanup never did anything. Cleanup also skipped
teardown whenever TEST_SKIP_CLUSTER_SETUP was set, even when TEST_MOUNT_POINT
was empty and Setup had started the cluster itself. The framework now keeps
the config it was built with, runs external-mount tests inside a per-run work
dir under the mount, and only skips teardown when an external mount was
actually used.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fuse integration: pass the debug flag before the subcommand

-v is a weed global flag; appended after master/volume/filer/mount it lands
on the subcommand flagset, which rejects it and the process dies at flag
parsing whenever EnableDebug is on.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fuse integration: require a real mount before running tests

Setup creates the mount point before weed mount starts, so statting and
listing it succeeded on the plain directory and tests ran against local
disk whenever the mount was slow or died. waitForMount now waits until the
mount point's device id differs from its parent's.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fuse integration: resolve the weed binary to an absolute path

The started processes run with cmd.Dir at the framework's temp dir, so a
relative candidate like ../weed no longer resolves. Prefer PATH, then probe
the usual build outputs and return absolute paths only.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fuse integration: build the external test work dir with MkdirTemp

The suite staged cloned tools and generated scripts under a predictable
/tmp path another user could pre-create and swap mid-run.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fuse integration: use the pjdfstest tree setup-pjdfstest installs

The test gated on a pjdfstest binary in PATH but then proved a suite it
cloned itself, so a tree installed by make setup-pjdfstest was never found.
Look for the built suite under external_tools first, then clone into the
work dir as before.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: require a real mount before posix tests

The wait loop fell back to listing the mount point, which passes on the
directory created before weed mount runs — a dead or slow mount meant the
suite exercised local disk. Only mountpoint counts now, and a dead mount
process fails fast instead of timing out.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: collect failure logs before cleanup deletes them

The always-on cleanup step removed /tmp/seaweedfs before the failure-only
debug step could copy the service logs, so failed runs uploaded nothing.
Run cleanup after log collection.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fuse integration: point MOUNT_POINT at TEST_MOUNT_POINT

Targets that work against MOUNT_POINT (pjdfstest, nfstest, fio configs)
ignored TEST_MOUNT_POINT and ran against the local-disk default while the
go tests used the external mount.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fuse integration: skip the permission-denied check as root

Root bypasses mode-bit checks, so writing the read-only file succeeded and
the assertion failed in root-run environments like containers.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fuse integration: align the posix docs with the suite

The doc listed a docker-test-posix target and make variables that do not
exist, and ran go test against single files that cannot compile alone. The
roadmap also still claimed the platform-specific tests this suite now
implements were all skipped.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fuse integration: honor SkipCleanup in the mini-based framework too

Same gap as the fuse-package framework: Cleanup read a fresh
DefaultTestConfig, so a test's SkipCleanup setting was ignored.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fuse integration: normalize config in Setup and clean the work dir too

Setup(nil) panicked on config.EnableDebug and external-mount Cleanup left
the temp dir behind; use the stored config for the start* calls and remove
tempDir alongside the external work dir.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: point external posix suites at the workflow FUSE mount

* fuse integration: create the fio target directory before running

* fuse integration: use unix.O_DIRECT instead of a hard-coded x86 value

0x4000 is O_DIRECT only on x86 and amd64; on arm64 it is O_DIRECTORY.
The test module already depends on golang.org/x/sys.

* fuse integration: run autoreconf and configure before building pjdfstest

A fresh pjdfstest clone is autotools-based and has no Makefile, so the
Go-side setup fallback always failed and the suite silently skipped.

* ci: require external posix tools only when setup installed them

setup-external-tools always exits 0 because each sub-target tolerates
failure, so POSIX_REQUIRE_EXTERNAL_TOOLS was set even with no tools
installed and the suite then hard-failed on the missing suites.

* ci: fail the posix workflow when the mount point is not a FUSE mount

The file write/read check passes on the plain directory created before
weed mount, so a dead mount let the whole suite grade the local disk.

* fuse integration: skip when the host has no FUSE device node

Setup failed the test outright on machines without /dev/fuse instead of
skipping; cover every suite entry point by skipping inside Setup.

* fuse integration: drop the chmod 666 /dev/fuse troubleshooting tip

World-writable device access is a security risk; the fuse group
membership on the line below is the safe fix.

* fuse integration: require go 1.21+ in check-go, not 1.20+

* fuse integration: refuse external suites on an unmounted path

* fuse integration: verify TEST_MOUNT_POINT is a real mount

* fuse integration: clean TEST_MOUNT_POINT before the parent-device check

filepath.Dir of a trailing-slash path returns the mount itself, so a mounted TEST_MOUNT_POINT ending in / was refused.

* fuse integration: detect WinFsp mounts on Windows in mountedAt

A directory mount is a reparse point and a drive mount is a volume root; a plain directory is now refused instead of accepted.

* fuse integration: strip a trailing slash off MOUNT_POINT in require-mounted

The mount-table grep fallback on hosts without mountpoint(8) would not match a slash-suffixed entry.

* fuse integration: require an external mount to be a FUSE filesystem

A device-ID boundary alone does not prove FUSE: a tmpfs mount would pass mountedAt and report its results as FUSE compliance. Check the filesystem type (statfs magic on linux, fstypename on the BSDs); platforms without a check keep the mount and write verification.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-22 14:37:03 -07:00
Chris Lu 8f6121694a test: fix HEAD bucket assertions and FUSE failover startup timing (#11967)
test: fix HEAD bucket assertions and warm up failover clusters
2026-10-22 14:34:52 -07:00
Chris Lu 137d29633b s3: fix prefix filtering and delimiter pagination (#11962) 2026-10-22 09:31:20 -07:00
github-actions[bot] ca0314ea4f docs: regenerate star history chart 2026-10-22 12:49:43 +00:00
Chris LuandJavier de la Torre 82f0a82564 filer+volume: make acknowledged writes durable under the fsync rule (#11943)
* volume: never compact a volume tiered to remote storage

A tiered volume's data backend is the remote object, but compaction reads
the local .dat path. When tiering removed the local .dat, every vacuum of a
tiered volume with garbage fails with "no such file or directory", and the
master's sweep and the worker's vacuum task keep selecting it. When the
local .dat was kept, the compaction succeeds against the stale copy and the
commit reloads the volume to read the remote object (old layout) through the
compacted index, so live needles stop reading.

Mirror the Rust volume server (#11545):
- CompactByIndex and CompactByVolumeData refuse a volume with a remote file
  before creating .cpd/.cpx ("volume N is tiered to remote storage, cannot
  compact").
- CommitCompact refuses under the data-file locks when the volume became
  tiered during the copy, and removes .cpd/.cpx.
- Store.CheckCompactVolume reports no garbage for a tiered volume, so the
  master's vacuum and volume.vacuum do not select it.
- The worker's vacuum detection skips a volume with a remote copy, as balance
  and EC detection already do.

Tests (each fails on master): TestCompactRefusesTieredVolume,
TestCommitCompactRefusesVolumeTieredDuringCopy (on master the commit
succeeds and a surviving needle no longer reads),
TestPickPrimaryReplicaSkipsTieredVolumes.

* volume: skip tiered replicas in vacuum selection, keep their real ratio

Reporting a zero garbage ratio for a tiered volume hid it from
volume.tier.compact, whose preflight reads that ratio while the volume is
still remote, and did not exclude it from a vacuum with a zero threshold.
Report the real ratio again, as the Rust volume server does, and have the
master's vacuum check skip a replica whose volume info names remote
storage, whatever the threshold or request.

When CommitCompact refuses a volume tiered during the copy, also close the
temporary needle map and remove .cpldb: nothing else would, since no later
vacuum selects the volume.

The tiered-compaction test no longer depends on random needle sizes
reaching a 0.3 garbage ratio.

* volume tier: do not attach a copy of a .dat a compaction replaced

The upload copies the .dat, then records the remote file in the .vif and
switches the backend. A compaction commit in between swaps in a compacted
.dat and index; the upload then served the old-layout remote object through
the new index, so live needles stopped reading. CommitCompact's tiered
check cannot see this: the volume is not tiered until the switch.

AttachRemoteDatFile records the remote file and switches the backend under
the data-file locks, and refuses when the backend is no longer the file
that was copied. The upload then removes the unused remote copy.

* volume tier: keep a committed .vif entry when only the dir fsync fails

AttachRemoteDatFile rolled back the in-memory remote entry on any
SaveVolumeInfo error. A NotCrashDurableError comes after the rename, so
the .vif on disk already names the copy; with the entry gone, the upload
handler deleted that object, and a reload would serve from a missing
remote file. Keep the entry and switch to the copy as the .vif says,
and still return the error so the caller keeps the local .dat.

* vacuum worker: filter tiered replicas instead of skipping the volume

A partially tiered volume is the leftover of an interrupted
volume.tier.upload; skipping the whole volume left garbage on its local
replicas unreclaimed. Per-replica filtering matches the master's
batchVacuumVolumeCheck and is safe: a tiered replica refuses both
compaction and commit.

* filer: sync leveldb entry writes when the path rule asks for fsync

The fsync storage rule only reached the volume upload; the metadata
writes behind an acknowledged create, update or delete still went to
the leveldb journal with Sync:false, so a power loss could roll back
an acknowledged bucket or object entry.

A write-side flag on the entry context carries the request: the filer
sets it from the path's storage rule in CreateEntry, UpdateEntry,
EnsureDirectoryEntry and DeleteEntryMetaAndData, and the HTTP write
path sets it for an explicit ?fsync=true. The leveldb stores write
with Sync:true when the flag is present.

* volume: flush the .idx before acknowledging a durable write

A write with fsync synced only the .dat: the index row written on
publish stayed in the page cache, so a power loss could leave the
needle's bytes durable with nothing indexing them.

The durable paths now run one .idx sync after publish, before the
requests complete — one per syncWrite and one per batch run, paired
with the .dat sync, the same split the Rust volume server makes
between flush_dat and flush_idx. A failed .idx sync cannot be rolled
back, so it stops the volume taking writes and fails the requests
instead of acknowledging an index entry that may not survive.

---------

Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
2026-10-21 23:57:09 -07:00
7603528aff s3api: support s3:RequestObjectTag/<key> in bucket policies (#11910)
* s3api: support s3:RequestObjectTag/<key> in bucket policies

The tags come from X-Amz-Tagging only where the handler stores them from
there: not on PutObjectTagging, which reads the body, and not on a copy
that keeps the source's tags. For a repeated key the last value counts,
since that is what CopyObject stores.

* s3api: extract RequestObjectTag only where the handler stores it

The header was honored on any request carrying it, so a tag attached to
a GET, DELETE, part upload, multipart completion, directory marker or a
PUT subresource satisfied a tag-conditioned policy the object never
receives. Extraction now follows the handlers: PutObject, a copy under
X-Amz-Tagging-Directive: REPLACE, and CreateMultipartUpload store the
header; a plain object PUT is identified by carrying only X-Amz-*
signing parameters.

* s3api: exclude multipart part PUTs before the copy-source check

A PUT carrying X-Amz-Copy-Source plus a numeric partNumber and uploadId
routes to CopyObjectPartHandler, which stores a part and never reads
X-Amz-Tagging. Checking the copy source first let such a request satisfy
an s3:RequestObjectTag condition under REPLACE with tags that are never
stored. Mirror the router: the part handlers match only when both
partNumber (digits only) and uploadId are present, so a lone or
non-numeric partNumber still falls through to the storing handlers.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3api: gate tag extraction on the routed subresource, not query key shape

Rejecting every PUT whose query held a key outside X-Amz- dropped
X-Amz-Tagging from ordinary object PUTs that carry arbitrary parameters —
Signature V2 presigns (AWSAccessKeyId/Signature/Expires) among them — so a
tag-conditioned Deny could be bypassed by appending an unrelated query
parameter while putToFiler still stored the tags. Exclude the object PUT
subresources the router actually dispatches (tagging, renameObject, acl,
retention, legal-hold) ahead of the copy-source check instead.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3api: cover the routed-subresource and part-copy tag extraction cases

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Chris Lu <chris.lu@gmail.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 23:55:01 -07:00
Chris LuandDevin cec96c02ea s3: reject requests carrying more than one routed subresource (#11942)
* s3: reject requests carrying more than one routed subresource

Routing picks the first matching subresource route while authorization
resolves a single action from the query, and the two can disagree. A
bucket-level PUT ?tagging&policy is authorized as s3:PutBucketTagging but
served by PutBucketPolicyHandler, letting a tagger rewrite the bucket
policy; the same shape lets a list grant read the policy via
?list-type=2&policy or ?versions&policy, and lets an object ACL writer
retag via PUT ?acl&tagging. Declining ambiguous requests keeps the
authorized action and the served handler identical.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: reuse hasAmbiguousSubresource in the routed-subresource guard

The standalone key list dropped versioning, so ?versioning&policy still
reached route matching and auth with two operations. The bucket-level
check already keys off operationSubresources plus bucketQueryActions;
running the same predicate keeps the lists from drifting apart.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 22:45:53 -07:00
Junker der Provinzandzouyixiong ee2718b14c filer: reduce forwarded lock log noise (#11933)
Forwarding a lock to its owner filer is routine in a multi-filer setup,
so the forward and acquire messages move from V(0) to V(3). The acquire
message is only logged when the forwarded response holds a renew token
and no application error. Transport failures still log at V(0).

Co-authored-by: zouyixiong <3390062+zouyixiong@users.noreply.github.com>
2026-10-21 20:46:23 -07:00
Chris LuandDevin 7ff77a4f3b s3api: skip error response body when http forbids it (#11931)
WriteResponse wrote the payload even for HEAD requests and statuses
that disallow a body (1xx, 204, 304), which made net/http return
ErrBodyNotAllowed and logged 'write err: http: request method or
response status code does not allow body' for ordinary requests.

Generated with [Devin](https://devin.ai)

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 20:43:32 -07:00
Chris LuandDevin 0b197867a1 filer leveldb2: tunable cache, buffer, block and table sizes (#11927)
* filer leveldb2: read cache and buffer sizes from filer.toml

The leveldb2 store hardcoded its block cache (32MiB) and write buffer
(16MiB) and offered no way to tune the table block size or compaction
table size. Optional block_cache_size_mb, write_buffer_size_mb,
block_size_kb and compaction_table_size_mb keys under [leveldb2] now
override them; zero or negative values keep the previous defaults.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer leveldb2: document tuning options in scaffold filer.toml

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer leveldb2: saturate size options instead of overflowing int

The option fields take int, so a MiB config value past math.MaxInt/MiB
(e.g. 2048 on 32-bit) wrapped negative and goleveldb would panic sizing
the memtable. Clamp the product instead of the input.

* filer leveldb2: reject size options too large for int bytes

Clamping a MiB setting at math.MaxInt silently hands goleveldb a
different size than configured; fail Initialize on values that cannot
fit instead.

* filer leveldb3: same tunable cache, buffer, block and table sizes

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer leveldb2/3: exercise oversized-option rejection on 64-bit too

The tests skipped whenever int was 64-bit, but write_buffer_size_mb
values above MaxInt/1MiB still overflow there, so the rejection path
had no coverage on the primary platform. Use MaxInt/1MiB + 1 so the
rejection triggers on every platform.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 20:39:06 -07:00
Chris LuandDevin 1451096c17 volume.fsck: delete missing filer entries through a worker pool (#11924)
* volume.fsck: delete missing filer entries through a worker pool

-reallyDeleteFilerEntries issued one synchronous HTTP DELETE per orphan
path inside the BFS consumer, so a million-entry purge ran one request
at a time. Queue the confirmed paths onto a bounded pool (default 16,
-tunable via -purgeConcurrency) so deletes run in parallel while the
traversal continues.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume.fsck: serialize command output across purge workers

* volume.fsck: serialize purges that share a hard link id

* volume.fsck: cover the purge worker pool

* volume.fsck: take the purge shard remainder in unsigned arithmetic

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 20:37:57 -07:00
Junker der Provinz 4854c6a0f4 s3: skip the local filer socket on Windows (#11921)
The filer only listens on its unix socket when GOOS is not windows, so
with a filer socket path configured on Windows the S3 server dialed a
socket that never exists.

Fixes #7278
2026-10-21 20:35:19 -07:00
74da88fc29 s3, iceberg: record the access key a request was signed with (#11901)
* s3, iceberg: record the access key a request was signed with

Several access keys can belong to one identity, but nothing downstream of
authentication says which one signed a request: the audit log carries the
identity name only, and every table credential the Iceberg catalog vends for
that identity gets the same session name. When an identity is shared (one key
per person or per service), the log cannot tell the holders apart.

The verifier that matched a credential now records that credential's access
key id in the request's identity holder (verifyV4Signature for header,
presigned, streaming and STS requests; both SigV2 verifiers). Nothing re-reads
the request for a key, so a second, unverified key elsewhere in the request
(a junk X-Amz-Credential on a SigV2 request, a junk SigV4 header on a SigV2
presigned URL) is never recorded.

- s3err.AccessLog gains access_key_id (omitempty): a static key's id, or an
  STS session's temporary key id. Every other field is unchanged.
- The Iceberg and Lance catalogs install the holder before authenticating a
  SigV4 request; a Bearer token records the key it was minted from (the token
  is verified with a key derived from it); a Lance x-api-key records its key.
- VendTableCredentials ends the session name with "-k" and the first 12 hex
  digits of SHA-256 of the key id (the table part gives way within the
  64-character limit; the tag is never cut), and carries the key id in the
  session token (src_ak, SessionInfo.SourceAccessKeyId).

Tests: auth_access_key_attribution_test.go and auth_access_key_shapes_test.go
(two keys on one identity, the deny path, SigV2 and presigned SigV2 with a
junk key elsewhere, streaming seed, STS session, failed signatures record
nothing), s3api_iceberg_session_name_test.go, iceberg and lance Auth tests,
sts source_access_key_test.go.

* s3: record the access key of a verified POST-policy form and of a retargeted signature

Two verified signatures left access_key_id empty in the audit log:

- A browser POST upload is verified in PostPolicyBucketHandler from its form
  fields by doesPolicySignatureV2Match / doesPolicySignatureV4Match, which did
  not record the key. Both now take the request context and record the
  credential's key once its signature verifies; a form that fails
  verification records nothing.
- verifyV4Signature refuses a signature that verified for one virtual-hosted
  bucket but was routed to another. The key was recorded only after that
  check, so the refusal carried none. It is now recorded as soon as a host
  candidate verifies.

Test: TestPostPolicyUploadRecordsTheSigningKey (SigV4 and SigV2 forms, and a
form with a wrong signature) and TestSignedVirtualHostRetargetRecordsTheSigningKey;
both fail before this change.

* s3: test VendTableCredentials end to end for the caller's access key

TestVendTableCredentialsRecordsTheCallersKey mints sessions through
VendTableCredentials with a configured STS service, the key recorded the way
the catalog's authentication records it, then validates each session token:
SourceAccessKeyId and the session name's key tag match the key, and a caller
without a key gets an untagged session. It fails if the vendor stops passing
either field.

---------

Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 20:33:08 -07:00
Jaehoon KimandChris Lu 744ab586c5 fix: volume.fsck purges live chunks of files renamed during its filer walk (SQL stores) (#11883)
* shell: volume.fsck reads the filer from one store snapshot when it can

volume.fsck collects the filer's chunk references by walking directory
by directory, so each directory is seen at a different time. A file
renamed from a directory not yet listed into one already listed is never
seen, its chunks look like orphans, and -reallyDeleteFromVolume deletes
live data. Checkpoint rotation and write-to-tmp-then-rename hit this.

Add an optional filer store capability, EntryScanner, that visits every
entry below a directory from one consistent snapshot. The SQL stores
implement it with one REPEATABLE READ read-only transaction read in
primary-key pages, so a concurrent rename is seen under exactly one of
its names. Key-value rows are skipped by their dirhash, a bucket
directory whose table is missing fails the scan (from the snapshot it
cannot be told from one dropped a moment ago), and hard links are
resolved in the same transaction. SQLite offers it only in WAL mode, on
its own connection.

TraverseBfsMetadata serves the snapshot when asked via request metadata
sw-traverse-mode=snapshot and reports the mode in a response header;
without the metadata it walks as before, so peer bootstrap and admin
export are unchanged. Snapshot responses carry no entry_ts_ns, whose
per-entry lookup would cost more than the scan.

volume.fsck requests the snapshot and falls back to the walk on an older
filer or a store that cannot scan.

* filer, shell: fail a scan that cannot start, not fall back to the walk

A transient BeginTx failure and a preliminary GetEntry error were treated
like a missing capability or missing root, so fsck would take the
directory walk - the very path that misses renamed-away entries. Only a
genuinely unsupported scan or a confirmed-absent root may fall back.

* filer: only snapshot-listed bucket names keep a missing table fatal

A cached-only name, whose bucket may have been deleted between copying
the cache and opening the snapshot, is skipped when its table is gone.

* filer: roll back to a savepoint before skipping a cached-only table

On PostgreSQL a failed statement aborts the snapshot transaction, so
ignoring a missing-table error without the savepoint still left every
later cached-only table unreadable.

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-10-21 20:31:42 -07:00
364e19a493 remote_storage/s3: use the AWS default credential chain when a source is configured (#11881)
* remote_storage/s3: use the AWS default credential chain when a source is configured

Type s3 forced anonymous credentials whenever no access/secret key was
set, so IRSA, container credentials, profiles and env keys were never
consulted. Keep anonymous (public bucket) access only when the environment
names no AWS credential source.

* remote_storage/s3: also recognize shared-config and default-profile env sources

* remote_storage/s3: resolve env credentials on their own transport

A guarded HTTP client is for the S3 endpoint; the remote credential
providers it would also serve reach link-local addresses that guard must
not dial. Profiles load shared config, and a test resolves env keys.

* remote_storage/s3: clear every detected credential source in the test

* remote_storage/s3: resolve the full default chain on the guarded client

The eager credential session used NewSession, whose chain covers only
env, shared-credentials and ECS/EC2 providers. Web identity and
assume-role profiles resolve through shared config, so IRSA never
worked when a custom HTTP client was pinned. Enable shared config and
carry the configured region so STS can sign assume-role requests.

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 20:30:28 -07:00
df33e81915 volume: report the needle read error with a failed read (#11934)
* volume: report the needle read error with a failed read

A read that fails on the volume server, for example on a CRC mismatch,
returned a bare 500, so mount and filer logs only showed the status.
The volume server now writes the error into the response body, and the
read helpers add it to the error they return.

Co-authored-by: Bruce <82158894+half-life666@users.noreply.github.com>

* volume: keep the 500 bare on HEAD requests

writeJsonError's body write fails with ErrBodyNotAllowed under net/http
when the request method is HEAD, adding an error-writing log line on top
of the failed read. Send only the status, matching the no-error path.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* util/http: decode gzip-encoded error bodies in statusError

Full-chunk reads set Accept-Encoding: gzip themselves, so the
transport does not transparently decompress; a gzip-encoded error
body reached statusError as compressed bytes and produced an
unreadable message. statusError now takes the response and decodes
Content-Encoding before extracting the bounded message.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: log the read error when the 500 stays bare

On HEAD requests the body write is skipped, which also skipped the
V(0) response log inside writeJson — a failed HEAD read left no
cause in normal volume-server logs. Log the error in the same
format before the bare 500.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* util/http: check the status before decoding gzip in GetAuthenticated

The gzip reader eagerly consumes the header when it is created, so
statusError saw a drained or corrupt body on a gzipped error response
and the server message was lost. Match the other read helpers and check
the status first.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* util/http: export StatusError

Other volume read paths outside this file need the same error body.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: redact jwt query param when logging failed-read URLs

The new read-failure logs print the request URL, and volume reads accept
?jwt= as a credential, so a failed authenticated read would write the
bearer token to the volume server log.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* util/http: surface the error body on the remaining chunk-read paths

* volume: redact duplicate jwt params in failed-read logs

* operation: pass the lookup jwt on the chunk needle read

---------

Co-authored-by: Bruce <82158894+half-life666@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-10-21 20:24:48 -07:00
Chris LuandDevin 867a537802 volume.configure: keep the superblock in sync with the .vif (#11930)
* volume.configure: keep the superblock in sync with the .vif

ConfigureVolume only rewrote the .vif's replication field, leaving the
.dat superblock stale. A mounted volume also kept reporting the old
placement in heartbeats, and its next compaction would rewrite the stale
placement into a fresh .dat.

Parse the placement once, then update the live volume when mounted: the
in-memory .vif copy and superblock, and the .dat header through the
volume's own backend under the data-file locks. For unmounted volumes
rewrite the header directly; remote-only volumes skip it since the .vif
carries the placement.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume.configure: update the .dat superblock in the Rust volume server

Mirror the Go ConfigureVolume change: after rewriting the .vif, rewrite
the replica placement in the .dat header too, so a remounted volume's
superblock, heartbeats, and later compactions agree with the .vif. A
missing .dat (remote-only volume) is skipped since the .vif carries the
placement.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 20:18:56 -07:00
Chris LuandDevin 85c1b990b1 shell: bound the initial master connection wait behind -connectTimeout (#11922)
* shell: bound the initial master connection wait behind -connectTimeout

weed shell waited on WaitUntilConnected with a background context, so
an unreachable master hung the command forever. The wait now runs on
a 30s (configurable) timeout and reports the failure instead.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* shell: stop the master reconnect loop when RunShell returns

KeepConnectedToMaster ran on context.Background(), so once RunShell
returned — including the new connect-timeout early return — the
reconnect goroutine kept dialing forever, and repeated RunShell calls
accumulated one each. Give it its own cancellable context, separate
from connectCtx whose deadline must not cut reconnects under a live
shell.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* shell: bound filer discovery by -connectTimeout too

The discovery WithClient and ListClusterNodes ran on the unbounded
background context, so a master disconnect after the initial connect —
or a persistently failing ListClusterNodes — could block startup past
the deadline. Reusing connectCtx bounds the whole startup phase.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* shell: report filer discovery failures instead of silently losing the filer target

* shell: give filer discovery its own budget after master connects

Discovery shared connectCtx, so a connection that landed near the
-connectTimeout deadline left ListClusterNodes almost no time and no room
for WithClient to retry. A fresh timeout keeps discovery bounded without
starving it.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* shell: cover the -connectTimeout master wait with a test

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* shell: note filer discovery in the -connectTimeout help text

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 20:17:51 -07:00
ac319fe318 volume: never compact a volume tiered to remote storage (#11895)
* volume: never compact a volume tiered to remote storage

A tiered volume's data backend is the remote object, but compaction reads
the local .dat path. When tiering removed the local .dat, every vacuum of a
tiered volume with garbage fails with "no such file or directory", and the
master's sweep and the worker's vacuum task keep selecting it. When the
local .dat was kept, the compaction succeeds against the stale copy and the
commit reloads the volume to read the remote object (old layout) through the
compacted index, so live needles stop reading.

Mirror the Rust volume server (#11545):
- CompactByIndex and CompactByVolumeData refuse a volume with a remote file
  before creating .cpd/.cpx ("volume N is tiered to remote storage, cannot
  compact").
- CommitCompact refuses under the data-file locks when the volume became
  tiered during the copy, and removes .cpd/.cpx.
- Store.CheckCompactVolume reports no garbage for a tiered volume, so the
  master's vacuum and volume.vacuum do not select it.
- The worker's vacuum detection skips a volume with a remote copy, as balance
  and EC detection already do.

Tests (each fails on master): TestCompactRefusesTieredVolume,
TestCommitCompactRefusesVolumeTieredDuringCopy (on master the commit
succeeds and a surviving needle no longer reads),
TestPickPrimaryReplicaSkipsTieredVolumes.

* volume: skip tiered replicas in vacuum selection, keep their real ratio

Reporting a zero garbage ratio for a tiered volume hid it from
volume.tier.compact, whose preflight reads that ratio while the volume is
still remote, and did not exclude it from a vacuum with a zero threshold.
Report the real ratio again, as the Rust volume server does, and have the
master's vacuum check skip a replica whose volume info names remote
storage, whatever the threshold or request.

When CommitCompact refuses a volume tiered during the copy, also close the
temporary needle map and remove .cpldb: nothing else would, since no later
vacuum selects the volume.

The tiered-compaction test no longer depends on random needle sizes
reaching a 0.3 garbage ratio.

* volume tier: do not attach a copy of a .dat a compaction replaced

The upload copies the .dat, then records the remote file in the .vif and
switches the backend. A compaction commit in between swaps in a compacted
.dat and index; the upload then served the old-layout remote object through
the new index, so live needles stopped reading. CommitCompact's tiered
check cannot see this: the volume is not tiered until the switch.

AttachRemoteDatFile records the remote file and switches the backend under
the data-file locks, and refuses when the backend is no longer the file
that was copied. The upload then removes the unused remote copy.

* volume tier: keep a committed .vif entry when only the dir fsync fails

AttachRemoteDatFile rolled back the in-memory remote entry on any
SaveVolumeInfo error. A NotCrashDurableError comes after the rename, so
the .vif on disk already names the copy; with the entry gone, the upload
handler deleted that object, and a reload would serve from a missing
remote file. Keep the entry and switch to the copy as the .vif says,
and still return the error so the caller keeps the local .dat.

* vacuum worker: filter tiered replicas instead of skipping the volume

A partially tiered volume is the leftover of an interrupted
volume.tier.upload; skipping the whole volume left garbage on its local
replicas unreclaimed. Per-replica filtering matches the master's
batchVacuumVolumeCheck and is safe: a tiered replica refuses both
compaction and commit.

---------

Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-10-21 20:16:02 -07:00
Chris LuandDevin 164a7d8b56 upload: upload -dir files concurrently behind a new -c flag (#11918)
* upload: upload -dir files concurrently behind a new -c flag

Each file under -dir was submitted serially inside filepath.Walk, so
every small file paid a sequential master-assign plus upload round
trip. A bounded pool now runs those submissions concurrently while
keeping results in walk order; failures are reported per file and the
command exits non-zero when any upload failed.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* submit: close opened readers when a batch assign fails

SubmitFiles returns early when the assign RPC fails, before
FilePart.Upload gets a chance to close each opened file. Close the
readers on that path so repeated assign failures do not leak handles.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* upload: stream -dir results as each file completes

Batching the output until every upload finished meant an interrupted
run lost the file ids of completed uploads. A per-index ready channel
lets a printer emit each result in walk order as soon as it lands.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* submit: close the opened file when Stat fails

newFilePart returned the still-open handle with the error, and the
concurrent upload path records the error without ever reaching
SubmitFiles, leaking one descriptor per failing file.

* upload: still upload files collected before a walk error

A walk that aborts partway used to leave the already-uploaded files
behind; collecting paths first must not drop them, so upload what was
gathered and still exit non-zero.

* upload: submit batched files concurrently

SubmitFiles fetched all file ids in one assign RPC but then uploaded
the files one after another, so a multi-file list still spent its time
in serial per-file round trips even with -dir concurrency. Uploads now
run through a small worker pool while results keep their input order.

Verified end to end: a 5-file list uploads with consecutive file keys
and each stored needle matches its source bytes.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* upload: print per-file results before exiting on a partial batch failure

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 20:08:57 -07:00
chuanying chenandChris Lu d35f4c1c6b Missing data read in super block extra parsing (#8392)
* storage: fail on short super block extra read even without an error

Some backends (e.g. s3_backend) clear io.EOF after a short read, returning
n < len(p) with a nil error. Checking n only when the error is non-nil
lets truncated extra data reach proto.Unmarshal with a zero-filled tail.

* storage: apply the same short-read check to the super block header

A backend that returns n < SuperBlockSize with a nil error previously
slipped through and parsed a partially zero-filled header.

* storage: test super block extra reads

Covers a round trip through ReadSuperBlock, a truncated extra region, a
backend returning io.EOF alongside a full read, and a backend returning
a short read with a nil error.

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-10-21 20:07:52 -07:00
1aad4cfe70 iam: optionally require an iam: grant for self-service access-key actions (#11937)
* iam: optionally require an iam: grant for self-service access-key actions

AuthorizeIamAction lets every authenticated identity run CreateAccessKey,
DeleteAccessKey, UpdateAccessKey, ListAccessKeys and GetUser on itself with
no grant. On AWS these actions need an IAM policy (the usual one grants them
on arn:aws:iam::*:user/${aws:username}); nothing is allowed implicitly.

Where identities hold data-plane rights only, the implicit carve-out lets a
leaked key mint further keys for its own identity, with no cap, that keep
working after the operator deletes the key that leaked; it can also
deactivate or delete the keys the operator issued.

A new option, off by default, makes these actions need the iam:<Action> grant
like every other IAM action: -s3.iam.selfServiceRequiresGrant on weed
server, filer and mini, -iam.selfServiceRequiresGrant on weed s3. With it
set, the grant is authorized against the target user's ARN, the caller's own
when UserName is empty, so a policy can confine it to the caller's own keys
(Resource arn:aws:iam:::user/${aws:username}), the same resource scoping
CreateServiceAccount already uses. The admin and iam:* grants are unchanged;
with the option off nothing changes.

Tests (s3api_embedded_iam_authz_test.go): with the option set, an identity
holding only s3:* and a bare identity are refused all five actions on an
empty, own and other target (fails without the change); a grant on the
caller's own user ARN, literal or ${aws:username}, allows them on itself and
not on another user; the admin and an iam:* grant keep every action; with the
option off self-service is allowed as before.

* iam: wire selfServiceRequiresGrant on the standalone iam server

Every other command that runs the embedded IAM API accepts the flag; the
deprecated weed iam could not enable the enforcement at all.

* iam: wire selfServiceRequiresGrant on weed server and filer -iam

The IamOptions used by 'weed server -iam' and 'weed filer -iam' never
received the flag, so startIamServer dereferenced a nil pointer at
startup. Share the -s3.iam.selfServiceRequiresGrant pointer with them.

---------

Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-10-21 20:06:30 -07:00
Chris LuandDevin ea859a0844 sftp: reject empty passwords in PasswordAuthenticator (#11938)
* sftp: reject empty passwords in PasswordAuthenticator

FileStore.ValidatePassword already refuses a zero-length stored or
supplied password, but the guard belongs at the authenticator too: an
empty password is never valid, and checking before the store call keeps
any user.Store implementation without its own empty-secret check from
matching on zero-length input.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* sftp: keep the brute-force delay on empty-password failures

The early rejection returned before the shared failure path, so an empty
password failed instantly while other failures slept 100-199 ms. Folding
the length check into the store call keeps the store untouched and every
failed attempt equally slow.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 20:03:14 -07:00
Chris LuandDevin c0152e6be1 s3: keep '\' a literal object-key character behind -s3.allowBackslashInKey (#11911)
* s3: keep '\' a literal object-key character when AllowBackslashInKey is set

NormalizeObjectKey rewrites every S3 object key: '\' becomes '/' and
duplicate slashes collapse. On AWS S3 both are ordinary key characters,
so keys that are distinct there alias one object here — writing
probe/back/slash.txt silently overwrites probe/back\slash.txt.

The filer only treats '/' as a path separator, so a verbatim '\' stays
inside one entry name. Gate the backslash rewrite and the matching '\'
handling in IsValidObjectKey behind a process-wide mode the S3 server
sets from the new S3ApiServerOption. Duplicate-slash collapsing stays:
an entry name cannot contain an empty segment.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* command: add -s3.allowBackslashInKey flag for AWS-compatible key handling

Registers the flag on all four S3 entry points (s3, server -s3,
filer -s3, mini) and passes it through S3Options into
S3ApiServerOption. Default off, so existing normalization is
unchanged.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: correct IsValidObjectKey/NormalizeObjectKey docs under allowBackslashInKey

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: normalize DeleteObjects body keys the same as path keys

DeleteObjects keys arrive in the XML body, so they bypassed the
normalization applied to route-var keys. The lock, the authorization
checks, the table-bucket layout check and the filer delete itself all
addressed the raw key while the conditional check resolved the
normalized one, so a key spelled with '\' or '//' could be reported
deleted while the stored entry survived. Echo the client-sent key in
the result while operating on the normalized form.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: report a normalized-empty DeleteObjects key as a delete no-op

A multi-delete key of only slashes normalizes to the empty string. Left
to reach the filer delete, the empty name resolves to the bucket entry
itself, so an unauthorized caller could remove a bucket through the
body-key path. Such a key can never name a stored entry, so like a
missing key the delete is an idempotent no-op reported before any
permission check or filer call.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: drop unused AllowBackslashInObjectKey getter

* s3: answer conditional delete of a normalized-empty key with NoSuchKey

* s3: normalize the probed key in authorizeAnonymousBucketList

The masked-miss probe read the raw mux var, so a spelling like folder//
or folder\ was authorized against a key that can never exist while the
request itself targets the normalized folder/. Probe the normalized key
so the answer matches what the read path authorizes.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: cover anonymous authz of a literal-backslash key under allowBackslashInKey

A public grant on the slash key must not authorize an anonymous read of
the distinct backslash object when the flag keeps '\' literal; grant the
backslash key's own ACL and it reads back. Default mode still folds the
spelling onto the covered slash key.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: publish the key mode only after the S3 server is constructed

A failed NewS3ApiServerWithStore could previously flip the process-wide
key mode out from under an already-running gateway.

* s3: normalize the copy-source key before validation and authorization

pathToBucketObjectAndVersion returned the raw header spelling, so an
equivalent spelling (a//b, a\b) was authorized and compared against the
normalized destination while resolveCopySourceEntry looked up the
normalized key — a deny on the canonical key could be bypassed and
self-copy detection missed. Normalize once at parse so the CopyObject and
UploadPartCopy paths agree.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 20:02:04 -07:00
e057244b0a storage: read the super block extra before unmarshaling it (#11932)
* storage: read the super block extra before unmarshaling it

ReadSuperBlock allocated a buffer of ExtraSize bytes but never filled it,
so any volume with a super block extra failed to load with an invalid
wire-format error.

Co-authored-by: chuanying chen <18412672+ccy00808@users.noreply.github.com>

* storage: test super block extra read error paths

---------

Co-authored-by: chuanying chen <18412672+ccy00808@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-10-21 19:55:27 -07:00
Javier de la TorreandJavier de la Torre 763e2a2090 iceberg: record geometry bounds on the data files compaction writes (#11898)
* iceberg: record geometry bounds on the data files compaction writes

Compaction collected column statistics for every primitive column except
geometry: the bound converter has no geometry case, so a compacted file's
manifest entry carried value and null counts for a geometry column but no
lower or upper bound. Readers that skip files by these bounds for a spatial
filter (for example DuckDB for `geom && <box>`) had to open every compacted
file, where the files before compaction could be skipped.

The bound follows the spec's single-value serialization for geometry: the
lower bound is the point (xmin, ymin), the upper bound (xmax, ymax), each an
8-byte little-endian double; z and m are not written. It is the union of the
written file's row-group geospatial statistics, which parquet-go computes
from every non-null value.

A bound that is too small makes a reader skip rows, so any doubt gives none:
a row group with non-null values but no geospatial statistics (parquet-go
omits them when a value does not parse as WKB), a box that is not finite, an
inverted box, or a chunk whose statistics cannot be read. A row group of
nulls only constrains nothing. Geography gets no bound, as iceberg-go writes
none for it (geodesic edges, boxes across the antimeridian).

Tests: points and polygons far apart through a bin-packed compaction (the
bound equals the union of every geometry), a sorted compaction (any bound
covers every geometry), a column of nulls only, an unparseable value in a
second row group (no bound; without the rule the file got the bound of the
first row group only), geography, and the aggregation rules.

* iceberg: do not order compaction bins by a geometry column

When a table declares no usable sort order, compaction orders a bin's files
by the first column every file has bounds for. A v3 table whose first column
is a geometry qualifies as soon as its files carry geometry bounds (DuckDB
writes them, and compacted files now do too). Those bounds decode into geo
literals that compareLiterals cannot compare, so the order sorted nothing,
yet an oversized bin was still split into "ordered" runs by splitOrderedBin.

Geometry and geography columns are now skipped in both the declared sort
order and the inferred one, so such a table is ordered by its next column
with bounds.

Test: TestResolveCompactionOrderSkipsGeometry, a geometry column first and
an id column second; on master the order is the geometry column.

---------

Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
2026-10-21 19:28:26 -07:00
917b6f4afd iceberg: authorize a commit before staging its metadata file (#11891)
* iceberg: authorize a commit before staging its metadata file

Both commit paths (CommitTable and CommitTransaction) write the new
vN.metadata.json into the table's metadata/ directory and only then reach
UpdateTable, whose authorization refuses a caller that may read the table but
not commit to it. The refused commit then deletes the staged file on the way
out, so such a caller could make the server write and delete a file under the
table's prefix on every attempt. The same shape was closed for creates in
#10991.

Both paths now ask Manager.AuthorizeUpdateTable right after loading the table
and answer 403 ForbiddenException before anything is staged. UpdateTable still
decides under the policies in force when the pointer flips, so a policy changed
mid-commit is caught as before.

Test: TestCommitDeniedBeforeStagingMetadata records every file the handlers
create; on master both the single-table and the transaction commit by a
GetTable-only identity write v2.metadata.json before answering 403, with the
change nothing is written. TestCommitAllowedStagesMetadata is the positive
control (the owner, and an identity whose own actions include UpdateTable).

* iceberg: authorize a view commit before staging its metadata

handleUpdateView has the same write-before-authorize gap the table commit
paths had: it stages the new metadata file and only reaches the UpdateView
authorization at the pointer flip, so a caller who may only load the view
could still make the server write and delete files under it. Gate it on
AuthorizeUpdateView, mirroring authorizeCommit.

* iceberg: seed committed view metadata in the denied-commit test

Without stored view metadata the request fails on the missing-metadata
check before it could stage a file, so the zero-write assertion could
not catch a write-before-denial regression. Seed the same metadata the
allowed-path test uses so the denied request would otherwise stage.

---------

Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-10-21 19:13:02 -07:00
Chris LuandDevin 106fd31af1 s3: resolve versionId ACL and attributes requests to the version-scoped actions (#11916)
* s3: resolve versionId ACL requests to the version ACL actions

A request for an object's ACL carrying versionId acted on the historical version but authorized against the current-object action (s3:GetObjectAcl/s3:PutObjectAcl), because the acl block in resolveFromQueryParameters returned before versionId was considered - the same gap closed for tagging. An identity granted only current-object ACL access - or explicitly denied s3:*ObjectVersionAcl - could still read and modify ACLs on noncurrent versions.

?acl&versionId=<id> on an object now resolves to s3:GetObjectVersionAcl/s3:PutObjectVersionAcl; ?acl alone keeps the current-object actions and bucket ACL is unaffected. s3:PutObjectVersionAcl is mapped onto the WriteAcp identity action so policies can name it.

* s3: resolve versionId attributes requests to the version attributes action

GET /bucket/key?attributes&versionId=<id> fetched the historical version's attributes but authorized against s3:GetObjectAttributes, so a grant limited to the current-object attributes action could reach noncurrent versions. The request now resolves to s3:GetObjectVersionAttributes, and both attributes actions map onto the Read identity action so policies can name them (s3:GetObjectAttributes was previously rejected as an unmapped action).

* s3api: recheck the version action on versioned ACL/attributes requests

The phase-2 policy recheck inside GetObjectAclHandler,
PutObjectAclHandler, and GetObjectAttributesHandler passed the
current-object action even when the request carried versionId and phase 1
had resolved the version action. A Deny scoped to s3:*VersionAcl or
s3:GetObjectVersionAttributes could never fire in the recheck, and an
Allow limited to the version action went unevaluated. Select the action
the same way the resolver does — versioned whenever versionId is present.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3api: recheck the version action on versioned tagging requests

The same phase-2 gap exists in the tagging handlers left by the earlier
version-tagging resolution change: versioned GetObjectTagging /
PutObjectTagging / DeleteObjectTagging rechecks evaluated the
current-object tagging action, so a Deny on s3:*VersionTagging could not
fire.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3api: resolve HEAD ?acl as the head-object action, not an ACL action

No HEAD acl route exists, so HEAD /bucket/key?acl lands on
HeadObjectHandler — which serves object headers, not the ACL. Resolving
it to an ACL action denied readers who hold only GetObject(Version) and
let ACL grants reach a head-object read. Dropping HEAD from the acl
branch lets the request fall through to GetObject/GetObjectVersion,
matching how the tagging branch already treats HEAD.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3api: require the version read grant for versioned attributes

AWS asks for s3:GetObjectVersion alongside s3:GetObjectVersionAttributes
when a GetObjectAttributes request names a versionId; the resolved
version-scoped action on its own left a lone s3:GetObjectVersion Deny
unable to cover the attributes read. Check the version read grant too:
the identity side through VerifyActionPermission, the bucket-policy side
through the same recheck that evaluates the scoped action.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3api: check the version read grant on a synthesized versioned GET

The versioned attributes check passed the literal s3:GetObjectVersion
action, but both policy paths re-resolve the action from the request, so
the attributes query parameter mapped it back to
s3:GetObjectVersionAttributes and a lone version-read Deny never fired.
The literal action also reached the legacy CanDo path, where a plain
Read grant no longer matched. Evaluate the grant on a request shaped
like a plain versioned GET instead.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3api: reject a named versionId on buckets without versioning

Tagging, ACL, and attributes handlers answered versionId requests on
never-versioned buckets as the current object while evaluating the
version-scoped action, so a grant limited to the version action could
authorize changes to the live entry. Reject the request up front through
the existing checkVersionIdConfigured gate.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3api: carry presign params onto synthesized policy requests

objectKeyPolicyRequest rebuilt the query with only versionId and the STS
token, so a presigned request's s3:authType condition value silently
changed from REST-QUERY-STRING to unset during phase-2 rechecks (version
read grant, copy source, delete keys, post-policy writes). A policy
conditioned on s3:authType would then deny a request it already allowed.
Carry the SigV2/SigV4 presign keys through; none of them steer
ResolveS3Action.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 16:50:34 -07:00
Chris LuandDevin 137f23a229 ec distribution: implement the documented shard distribution API (#11915)
* ec distribution: parse EC and replication configs into a shard plan

NewECConfig/DefaultECConfig and NewReplicationConfig(FromString) turn an
EC ratio and a replica placement into ECConfig/ReplicationConfig values,
and CalculateDistribution derives the target and maximum shard counts per
data center, rack, and node. ECDistribution.Summary and
FaultToleranceAnalysis describe, per topology level, whether losing one
failure domain leaves enough shards to reconstruct the volume, so the EC
durability guarantees of a given setup can be answered directly instead of
only documented informally.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ec distribution: analyze topology and plan shard moves

TopologyAnalysis records nodes and shard locations per DC/rack/node, and
Rebalancer.PlanRebalance emits the moves that level the layout: data
centers first, then racks within each DC, then nodes within each rack,
moving parity shards before data shards and honoring per-node free slots.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ec distribution: cover the plan math and rebalancing with tests

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ec distribution: keep AddShardLocation and node ShardIDs in sync

AddShardLocation updated the count maps but never the node's ShardIDs —
yet every rebalancer count reads len(node.ShardIDs), so the documented
AddNode-then-AddShardLocation flow produced an empty rebalance plan. It
now appends to the node's shard list and skips shards the node already
has, making the two registration styles equivalent and idempotent.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ec distribution: key rack-scoped maps by dc/rack

Rack names are only unique within a data center — two DCs can both hold a
"rack1" — so ShardsByRack, RackToShards, and RackToNodes keyed by the bare
rack name merged distinct domains and corrupted both the DC-level shard
counts and rack-level balancing. Rack lookups now use the dc/rack pair;
DCToRacks still lists bare rack names.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ec distribution: never move a shard onto a node already holding it

The move picker selected the destination node before the shard, so a node
already holding the chosen shard could be targeted. Pick the shard first
(parity-first) and choose among destination nodes that lack it.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ec distribution: tolerate zero-value replication configs

A ReplicationConfig{} with zeroed minimums made maxShare divide by zero;
clamp each level's unit count to at least one. Covers the new behaviors
with tests.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ec distribution: keep rack keys distinct when names contain "/"

rackKey joined data center and rack with "/", so dc "a"/rack "b/c" and
dc "a/b"/rack "c" shared one key and were counted as the same failure
domain. Separate the parts with a NUL byte, which cannot appear in a
name.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ec distribution: plan rebalances on a copy of the topology

recordMove mutated the caller's TopologyAnalysis while planning, so an
interrupted or partial execution left the input describing moves that
never happened, and replanning on it returned nothing. PlanRebalance
now simulates on a cloned analysis; plan.Analysis reports the layout
the moves would produce and the caller's input is left untouched.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ec distribution: balance nodes against the rack's current shard count

The node-level target still read the original analysis, so a rack that
had just sent or received shards during DC/rack balancing aimed at a
stale total and left its nodes unbalanced.

* ec distribution: register each node id once

Adding the same node twice appended it to RackToNodes twice, so every
count that walks rack nodes double-counted its shards.

* ec distribution: normalize replication counts in the stored plan

A zero-value ReplicationConfig computed targets against one domain per
level while Summary still reported zero DCs, racks, and nodes.

* ec distribution: cap a survivable level's max at the loss budget

* ec distribution: skip shards already inside the destination domain

* ec distribution: register nodes implied by shard locations

* ec distribution: merge repeat node registrations

A shard location now auto-registers an unknown node with no free slots;
a later AddNode carrying the real capacity was then dropped entirely.
Repeat registration now updates the node's capacity and adopts any new
shards, while the first registration keeps the topology placement.

* ec distribution: try other overloaded sources before stopping

When the most overloaded domain could not shed any shard, balance
returned early and skipped every other source that still had movable
shards. A source that cannot move now gets set aside for the rest of
the pass.

* ec distribution: collapse chained shard moves to the final destination

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ec distribution: schedule collapsed moves behind the departures that free slots

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 16:50:01 -07:00
Junker der ProvinzandDan Foster e645449872 fix(storage): count preallocated space when auto-sizing max volumes (#11919)
The max volume count subtracted each volume's remaining room from the
free disk space, except when the master had preallocation turned on,
in which case it subtracted nothing. That is wrong for a disk that
mixes preallocated and plain volumes, and on platforms where
preallocation is not supported.

Use the larger of the .dat file's size and its allocated blocks, in the
Go and the Rust volume server, so a preallocated volume counts as full
and a plain one keeps its remaining room reserved.

Fixes #5778

Co-authored-by: Dan Foster <4570306+danfoster@users.noreply.github.com>
2026-10-21 15:11:06 -07:00
664 changed files with 50626 additions and 6584 deletions

No files matched your search

+2 -1
View File
@@ -20,4 +20,5 @@ ignore-regex = \b[a-z]+[A-Z]\w*\b|\b[A-Z][a-z]+[A-Z]\w*\b
# unknwon: GitHub username / Go module path (`github.com/unknwon/goconfig`)
# atleast: CLI mode literal string in test/benchmark/fuse_db/bin/sqlite_verify.py
# sme: local variable for a *streamMutateError in mount tests
ignore-words-list = visibles,fo,te,ser,bject,unparseable,keep-alives,tread,anc,ue,auther,thirdparty,unknwon,atleast,sme
# nin: AWS S3 Vectors "$nin" filter operator literal (API-defined, cannot rename)
ignore-words-list = visibles,fo,te,ser,bject,unparseable,keep-alives,tread,anc,ue,auther,thirdparty,unknwon,atleast,sme,nin
@@ -0,0 +1,11 @@
name: Clean up stale SeaweedFS test mounts
description: Detach disconnected FUSE test mounts left by earlier jobs on a shared runner.
runs:
using: composite
steps:
- name: Unmount disconnected SeaweedFS test mounts
if: runner.os == 'Linux' && runner.environment == 'self-hosted'
shell: bash
env:
CLEANUP_SCRIPT: ${{ github.action_path }}/cleanup.py
run: python3 "$CLEANUP_SCRIPT"
@@ -0,0 +1,89 @@
#!/usr/bin/env python3
"""Detach disconnected SeaweedFS integration-test mounts without touching live ones."""
import errno
import os
from pathlib import Path
import re
import shutil
import subprocess
import sys
import tempfile
def test_mounts(mountinfo):
"""Read the kernel table directly: traversing a stale mount may block."""
roots = {tempfile.gettempdir(), "/tmp", "/var/tmp"}
for line in mountinfo.splitlines():
fields, filesystem = line.split(" - ", 1)
if filesystem.split()[0].split(".")[0] != "fuse":
continue
fields = fields.split()
# mountinfo escapes whitespace and backslashes with octal sequences.
target = re.sub(r"\\([0-7]{3})", lambda m: chr(int(m[1], 8)), fields[4])
path = Path(target)
# Test mounts are a mount* leaf under a seaweedfs_fuse_* scratch dir
# (seaweedfs_fuse_test_*/mount, seaweedfs_fuse_failover_*/mount0, ...)
# or the posix-compliance workflow's fixed /tmp/seaweedfs/mount.
if (not path.name.startswith("mount")
or str(path.parent.parent) not in roots
or not (path.parent.name.startswith("seaweedfs_fuse")
or path.parent.name == "seaweedfs")):
continue
yield fields[0], target
def mounted(mount_id, target):
return (mount_id, target) in test_mounts(Path("/proc/self/mountinfo").read_text())
def cleanup():
failed = False
for mount_id, target in test_mounts(Path("/proc/self/mountinfo").read_text()):
try:
# A separate, bounded probe prevents a wedged FUSE daemon from
# hanging cleanup. Only ENOTCONN proves the daemon disconnected;
# a timeout or another error is not enough to detach a live mount.
probe = subprocess.run(
[sys.executable, __file__, "--probe", target], timeout=5,
capture_output=True, check=False,
)
except subprocess.TimeoutExpired:
print(f"::warning::FUSE probe timed out; leaving {target} mounted")
continue
if probe.returncode != errno.ENOTCONN or not mounted(mount_id, target):
continue
print(f"Unmounting disconnected FUSE test mount: {target}", flush=True)
# Lazy unmount detaches a dead mount even when an old process still
# holds it open. Prefer the unprivileged FUSE helper; sudo is fallback.
for command in (
["fusermount3", "-uz", "--", target],
["fusermount", "-uz", "--", target],
["sudo", "-n", "umount", "-l", "--", target],
):
try:
subprocess.run(command, timeout=10, check=False)
except (OSError, subprocess.TimeoutExpired) as error:
print(f"{command[0]}: {error}", file=sys.stderr)
if not mounted(mount_id, target):
break
else:
print(f"::error::Could not unmount disconnected FUSE test mount: {target}")
failed = True
continue
# The detached mount's parent still holds the crashed run's data and
# logs; test_mounts only yields targets under a SeaweedFS test dir in
# a temp root, so the parent is safe to reclaim here.
shutil.rmtree(Path(target).parent, ignore_errors=True)
return int(failed)
if __name__ == "__main__":
if len(sys.argv) == 3 and sys.argv[1] == "--probe":
try:
os.stat(sys.argv[2])
except OSError as error:
sys.exit(error.errno)
sys.exit(0)
sys.exit(cleanup())
@@ -0,0 +1,137 @@
import errno
import importlib.util
from pathlib import Path
import subprocess
import unittest
from unittest.mock import patch
spec = importlib.util.spec_from_file_location("cleanup", Path(__file__).with_name("cleanup.py"))
cleanup = importlib.util.module_from_spec(spec)
spec.loader.exec_module(cleanup)
TARGET = "/tmp/seaweedfs_fuse_test_123/mount"
MOUNTINFO = f"42 1 0:9 / {TARGET} rw - fuse.seaweedfs seaweedfs rw\n"
class CleanupTests(unittest.TestCase):
def run_cleanup(self, results, mounted=True):
with patch.object(cleanup.Path, "read_text", return_value=MOUNTINFO), \
patch.object(cleanup, "mounted", side_effect=mounted if isinstance(mounted, list) else None,
return_value=mounted) as check, \
patch.object(cleanup.shutil, "rmtree") as rmtree, \
patch.object(cleanup.subprocess, "run", side_effect=results) as run:
status = cleanup.cleanup()
return status, run, check, rmtree
def result(self, status):
return subprocess.CompletedProcess([], status)
def test_only_fuse_integration_test_mounts_are_candidates(self):
rows = MOUNTINFO
rows += MOUNTINFO.replace("fuse.seaweedfs", "ext4")
rows += MOUNTINFO.replace(TARGET, "/mnt/production")
rows += MOUNTINFO.replace(TARGET, "/tmp/seaweedfs_fuse_test_123/data")
rows += MOUNTINFO.replace(TARGET, "/home/user/seaweedfs_fuse_test_123/mount")
rows += MOUNTINFO.replace(TARGET, "/tmp/seaweedfs/data")
self.assertEqual(list(cleanup.test_mounts(rows)), [("42", TARGET)])
def test_all_test_mount_layouts_are_candidates(self):
targets = (
"/tmp/seaweedfs_fuse_test_123/mount",
"/tmp/seaweedfs_fuse_failover_456/mount0",
"/tmp/seaweedfs_fuse_failover_456/mount1",
"/var/tmp/seaweedfs_fuse_p2p_test_789/mount2",
"/tmp/seaweedfs/mount",
)
rows = "".join(
MOUNTINFO.replace("42", str(40 + i)).replace(TARGET, target)
for i, target in enumerate(targets)
)
self.assertEqual(
list(cleanup.test_mounts(rows)),
[(str(40 + i), target) for i, target in enumerate(targets)],
)
def test_mountinfo_paths_are_unescaped_once(self):
target = "/tmp/seaweedfs_fuse_test_space and\\slash/mount"
encoded = target.replace("\\", r"\134").replace(" ", r"\040")
rows = MOUNTINFO.replace(TARGET, encoded)
self.assertEqual(list(cleanup.test_mounts(rows)), [("42", target)])
def test_no_mounts_is_a_noop(self):
with patch.object(cleanup.Path, "read_text", return_value=""), \
patch.object(cleanup.subprocess, "run") as run:
self.assertEqual(cleanup.cleanup(), 0)
run.assert_not_called()
def test_healthy_mount_is_untouched(self):
status, run, _, _ = self.run_cleanup([self.result(0)])
self.assertEqual(status, 0)
self.assertEqual(run.call_count, 1)
def test_other_probe_errors_are_untouched(self):
for error in (errno.EACCES, errno.ENOENT, errno.EIO):
with self.subTest(error=error):
status, run, _, _ = self.run_cleanup([self.result(error)])
self.assertEqual(status, 0)
self.assertEqual(run.call_count, 1)
def test_probe_timeout_does_not_detach_a_possibly_live_mount(self):
status, run, _, _ = self.run_cleanup([subprocess.TimeoutExpired("probe", 5)])
self.assertEqual(status, 0)
self.assertEqual(run.call_count, 1)
def test_disconnected_mount_is_lazily_unmounted(self):
status, run, _, _ = self.run_cleanup([self.result(errno.ENOTCONN), self.result(0)], [True, False])
self.assertEqual(status, 0)
self.assertEqual(run.call_args_list[1].args[0], ["fusermount3", "-uz", "--", TARGET])
self.assertEqual(run.call_args_list[0].kwargs["timeout"], 5)
self.assertEqual(run.call_args_list[1].kwargs["timeout"], 10)
def test_replaced_or_removed_mount_is_untouched(self):
status, run, _, _ = self.run_cleanup([self.result(errno.ENOTCONN)], False)
self.assertEqual(status, 0)
self.assertEqual(run.call_count, 1)
def test_mount_identity_is_checked_without_stat(self):
with patch.object(cleanup.Path, "read_text", return_value=MOUNTINFO):
self.assertTrue(cleanup.mounted("42", TARGET))
self.assertFalse(cleanup.mounted("43", TARGET))
def test_missing_or_timed_out_helper_uses_fallback(self):
for error in (FileNotFoundError(), subprocess.TimeoutExpired("fusermount3", 10)):
with self.subTest(error=error):
status, run, _, _ = self.run_cleanup(
[self.result(errno.ENOTCONN), error, self.result(0)], [True, True, False])
self.assertEqual(status, 0)
self.assertEqual(run.call_args.args[0], ["fusermount", "-uz", "--", TARGET])
def test_failed_helpers_fall_back_to_sudo(self):
status, run, _, _ = self.run_cleanup(
[self.result(errno.ENOTCONN), self.result(1), self.result(1), self.result(0)],
[True, True, True, False])
self.assertEqual(status, 0)
self.assertEqual(run.call_args.args[0], ["sudo", "-n", "umount", "-l", "--", TARGET])
def test_cleanup_fails_when_stale_mount_remains(self):
status, run, _, rmtree = self.run_cleanup(
[self.result(errno.ENOTCONN), self.result(1), self.result(1), self.result(1)])
self.assertEqual(status, 1)
self.assertEqual(run.call_count, 4)
rmtree.assert_not_called()
def test_unmounted_test_dir_is_reclaimed(self):
status, _, _, rmtree = self.run_cleanup(
[self.result(errno.ENOTCONN), self.result(0)], [True, False])
self.assertEqual(status, 0)
rmtree.assert_called_once_with(Path(TARGET).parent, ignore_errors=True)
def test_probe_process_reports_errno(self):
for target, expected in ((str(Path(__file__)), 0), (str(Path(__file__) / "absent"), errno.ENOTDIR)):
with self.subTest(target=target):
result = subprocess.run([cleanup.sys.executable, cleanup.__file__, "--probe", target], check=False)
self.assertEqual(result.returncode, expected)
if __name__ == "__main__":
unittest.main()
@@ -23,8 +23,14 @@ runs:
bin=$(command -v fusermount3 || true)
if [ -z "$bin" ]; then
echo "no fusermount3 in PATH" >&2
exit 1
# Self-hosted runners are not guaranteed to carry fuse3.
sudo apt-get update
sudo apt-get install -y fuse3
bin=$(command -v fusermount3 || true)
if [ -z "$bin" ]; then
echo "no fusermount3 in PATH after installing fuse3" >&2
exit 1
fi
fi
if can_mount "$bin"; then
ls -l "$bin"
+30
View File
@@ -0,0 +1,30 @@
name: "actions: unit tests"
on:
push:
branches: [ master ]
paths:
- '.github/actions/**'
- '.github/workflows/action-tests.yml'
pull_request:
branches: [ master ]
paths:
- '.github/actions/**'
- '.github/workflows/action-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
jobs:
cleanup-stale-fuse-mounts:
runs-on: ubuntu-latest
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Run cleanup action unit tests
run: python3 .github/actions/cleanup-stale-fuse-mounts/test_cleanup.py
+11 -5
View File
@@ -9,8 +9,8 @@ on:
- '.github/workflows/codeql.yml'
concurrency:
group: ${{ github.head_ref }}/codeql
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
jobs:
CodeQL-Build:
@@ -25,9 +25,15 @@ jobs:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@v4.39.0
uses: github/codeql-action/init@v4.39.1
# Override language selection by uncommenting this and choosing your languages
with:
languages: go
@@ -35,7 +41,7 @@ jobs:
# Autobuild attempts to build any compiled languages (C/C++, C#, or Java).
# If this step fails, then you should remove it and run the build manually (see below).
- name: Autobuild
uses: github/codeql-action/autobuild@v4.39.0
uses: github/codeql-action/autobuild@v4.39.1
# ℹ️ Command-line programs to run using the OS shell.
# 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun
@@ -49,4 +55,4 @@ jobs:
# make release
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v4.39.0
uses: github/codeql-action/analyze@v4.39.1
+4
View File
@@ -8,6 +8,10 @@ on:
pull_request:
branches: [master]
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
+45 -4
View File
@@ -13,6 +13,10 @@ on:
- 'go.sum'
- '.github/workflows/container_dev.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -20,7 +24,7 @@ jobs:
# ── Pre-build the Rust binaries natively ────────────────────────────
build-rust-binaries:
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
strategy:
matrix:
include:
@@ -40,7 +44,12 @@ jobs:
- name: Install musl tools (amd64)
if: ${{ !matrix.cross }}
run: sudo apt-get install -y musl-tools
run: |
sudo apt-get install -y musl-tools
# C deps (aws-lc-sys) must compile against musl headers, not the host
# gcc's glibc headers — glibc's C23 redirects (__isoc23_sscanf,
# __isoc23_strtol) don't exist in musl and fail the link.
echo "CC_x86_64_unknown_linux_musl=musl-gcc" >> "$GITHUB_ENV"
- name: Install cross-compilation tools (arm64)
if: matrix.cross
@@ -49,8 +58,34 @@ jobs:
echo "CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_LINKER=aarch64-linux-gnu-gcc" >> "$GITHUB_ENV"
# Disable glibc fortify source — its __memcpy_chk etc. symbols don't exist in musl
echo "CFLAGS_aarch64_unknown_linux_musl=-U_FORTIFY_SOURCE" >> "$GITHUB_ENV"
# aws-lc-sys compiles against glibc headers, which redirect
# sscanf/strtol to C23 __isoc23_* symbols musl does not provide.
# Link a tiny forwarding shim so those references resolve to the
# musl implementations. Prototypes are declared manually: including
# glibc headers here would re-redirect the calls back to __isoc23_*.
cat > /tmp/isoc23-shim.c <<'SHIM'
#include <stdarg.h>
extern int vsscanf(const char *s, const char *fmt, va_list ap);
extern long strtol(const char *nptr, char **endptr, int base);
int __isoc23_sscanf(const char *s, const char *fmt, ...) {
va_list ap;
va_start(ap, fmt);
int r = vsscanf(s, fmt, ap);
va_end(ap);
return r;
}
long __isoc23_strtol(const char *nptr, char **endptr, int base) {
return strtol(nptr, endptr, base);
}
SHIM
aarch64-linux-gnu-gcc -O2 -c /tmp/isoc23-shim.c -o /tmp/isoc23-shim.o
# link-args land after libc on the link line; rescan musl libc so
# the shim's own vsscanf/strtol references resolve, and libgcc for
# the fp128 builtins musl's vfscanf/floatscan pull in on aarch64.
echo "CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_RUSTFLAGS=-Clink-arg=/tmp/isoc23-shim.o -Clink-arg=-lc -Clink-arg=-lgcc" >> "$GITHUB_ENV"
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
@@ -170,14 +205,14 @@ jobs:
- name: Login to Docker Hub
if: github.event_name != 'pull_request'
uses: docker/login-action@v4.6.0
uses: docker/login-action@v4.7.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Login to GHCR
if: github.event_name != 'pull_request'
uses: docker/login-action@v4.6.0
uses: docker/login-action@v4.7.0
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
@@ -193,6 +228,12 @@ jobs:
platforms: linux/amd64, linux/arm64
tags: ${{ steps.docker_meta.outputs.tags }}
labels: ${{ steps.docker_meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max
# Key the checkout layer on the pushed sha so cached layers can't
# serve stale code; the Dockerfile fetch fallback picks it up.
build-args: |
BRANCH=${{ github.sha }}
- name: Sign
if: github.event_name != 'pull_request'
@@ -24,6 +24,10 @@ on:
required: false
default: ''
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.event.inputs.image_tag || format('foundationdb_{0}_seaweedfs_{1}', github.event.inputs.fdb_version || '7.4.5', github.event.inputs.seaweedfs_ref || github.ref_name) }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -64,6 +68,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: go.mod
cache: ${{ runner.environment == 'github-hosted' }}
- name: Run FoundationDB tagged tests
env:
@@ -136,7 +141,7 @@ jobs:
- name: Login to Docker Hub
if: github.event_name != 'pull_request'
uses: docker/login-action@v4.6.0
uses: docker/login-action@v4.7.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
+49 -10
View File
@@ -30,6 +30,11 @@ on:
type: boolean
default: false
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.event.inputs.image_tag || github.run_id }}
cancel-in-progress: false
queue: max
permissions:
contents: read
security-events: write
@@ -61,7 +66,7 @@ jobs:
# ── Pre-build the Rust binaries natively ────────────────────────────
build-rust-binaries:
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
strategy:
matrix:
include:
@@ -83,7 +88,12 @@ jobs:
- name: Install musl tools (amd64)
if: ${{ !matrix.cross }}
run: sudo apt-get install -y musl-tools
run: |
sudo apt-get install -y musl-tools
# C deps (aws-lc-sys) must compile against musl headers, not the host
# gcc's glibc headers — glibc's C23 redirects (__isoc23_sscanf,
# __isoc23_strtol) don't exist in musl and fail the link.
echo "CC_x86_64_unknown_linux_musl=musl-gcc" >> "$GITHUB_ENV"
- name: Install cross-compilation tools (arm64)
if: matrix.cross
@@ -92,8 +102,34 @@ jobs:
echo "CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_LINKER=aarch64-linux-gnu-gcc" >> "$GITHUB_ENV"
# Disable glibc fortify source — its __memcpy_chk etc. symbols don't exist in musl
echo "CFLAGS_aarch64_unknown_linux_musl=-U_FORTIFY_SOURCE" >> "$GITHUB_ENV"
# aws-lc-sys compiles against glibc headers, which redirect
# sscanf/strtol to C23 __isoc23_* symbols musl does not provide.
# Link a tiny forwarding shim so those references resolve to the
# musl implementations. Prototypes are declared manually: including
# glibc headers here would re-redirect the calls back to __isoc23_*.
cat > /tmp/isoc23-shim.c <<'SHIM'
#include <stdarg.h>
extern int vsscanf(const char *s, const char *fmt, va_list ap);
extern long strtol(const char *nptr, char **endptr, int base);
int __isoc23_sscanf(const char *s, const char *fmt, ...) {
va_list ap;
va_start(ap, fmt);
int r = vsscanf(s, fmt, ap);
va_end(ap);
return r;
}
long __isoc23_strtol(const char *nptr, char **endptr, int base) {
return strtol(nptr, endptr, base);
}
SHIM
aarch64-linux-gnu-gcc -O2 -c /tmp/isoc23-shim.c -o /tmp/isoc23-shim.o
# link-args land after libc on the link line; rescan musl libc so
# the shim's own vsscanf/strtol references resolve, and libgcc for
# the fp128 builtins musl's vfscanf/floatscan pull in on aarch64.
echo "CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_RUSTFLAGS=-Clink-arg=/tmp/isoc23-shim.o -Clink-arg=-lc -Clink-arg=-lgcc" >> "$GITHUB_ENV"
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
@@ -164,10 +200,13 @@ jobs:
uses: actions/checkout@v7
with:
ref: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.source_ref || github.ref }}
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Free Disk Space
run: |
echo "Available disk space before cleanup:"
df -h
timeout 15 df -h || true
# Remove pre-installed tools
sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /opt/hostedtoolcache/CodeQL
# Clean package managers
@@ -179,7 +218,7 @@ jobs:
[ -d ~/.cache/go-build ] && rm -rf ~/.cache/go-build || true
[ -d /go/pkg ] && rm -rf /go/pkg || true
echo "Available disk space after cleanup:"
df -h
timeout 15 df -h || true
- name: Configure variant
id: config
@@ -250,13 +289,13 @@ jobs:
buildkitd-config: /tmp/buildkitd.toml
- name: Login to Docker Hub
if: needs.setup.outputs.publish == 'true'
uses: docker/login-action@v4.6.0
uses: docker/login-action@v4.7.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Login to GHCR
if: needs.setup.outputs.publish == 'true'
uses: docker/login-action@v4.6.0
uses: docker/login-action@v4.7.0
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
@@ -304,7 +343,7 @@ jobs:
fi
- name: Login to GHCR
if: needs.setup.outputs.publish == 'true'
uses: docker/login-action@v4.6.0
uses: docker/login-action@v4.7.0
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
@@ -405,7 +444,7 @@ jobs:
output: trivy-results.sarif
exit-code: '0'
- name: Upload Trivy scan results to GitHub Security
uses: github/codeql-action/upload-sarif@v4.39.0
uses: github/codeql-action/upload-sarif@v4.39.1
if: always()
with:
sarif_file: trivy-results.sarif
@@ -472,12 +511,12 @@ jobs:
ghcr.io/chrislusf/seaweedfs
tags: type=raw,value=${{ github.event_name == 'workflow_dispatch' && github.event.inputs.image_tag || 'latest' }},suffix=${{ steps.config.outputs.tag_suffix }}
- name: Login to Docker Hub
uses: docker/login-action@v4.6.0
uses: docker/login-action@v4.7.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Login to GHCR
uses: docker/login-action@v4.6.0
uses: docker/login-action@v4.7.0
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
@@ -11,6 +11,11 @@ on:
required: true
default: ''
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || format('refs/tags/{0}', github.event.inputs.release_tag) || github.run_id }}
cancel-in-progress: false
queue: max
permissions:
contents: read
@@ -53,7 +58,7 @@ jobs:
-
name: Login to Docker Hub
if: github.event_name != 'pull_request'
uses: docker/login-action@v4.6.0
uses: docker/login-action@v4.7.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
+96 -15
View File
@@ -25,7 +25,7 @@ on:
rocksdb_version:
description: 'RocksDB git tag to use when variant=rocksdb'
required: false
default: 'v10.10.1'
default: 'v11.1.2'
permissions:
contents: read
@@ -47,7 +47,7 @@ jobs:
# amd64 and arm64 without QEMU, turning a 5-hour emulated cargo build into
# ~15 minutes of native compilation.
build-rust-binaries:
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
strategy:
matrix:
include:
@@ -67,7 +67,12 @@ jobs:
- name: Install musl tools (amd64)
if: ${{ !matrix.cross }}
run: sudo apt-get install -y musl-tools
run: |
sudo apt-get install -y musl-tools
# C deps (aws-lc-sys) must compile against musl headers, not the host
# gcc's glibc headers — glibc's C23 redirects (__isoc23_sscanf,
# __isoc23_strtol) don't exist in musl and fail the link.
echo "CC_x86_64_unknown_linux_musl=musl-gcc" >> "$GITHUB_ENV"
- name: Install cross-compilation tools (arm64)
if: matrix.cross
@@ -76,8 +81,34 @@ jobs:
echo "CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_LINKER=aarch64-linux-gnu-gcc" >> "$GITHUB_ENV"
# Disable glibc fortify source — its __memcpy_chk etc. symbols don't exist in musl
echo "CFLAGS_aarch64_unknown_linux_musl=-U_FORTIFY_SOURCE" >> "$GITHUB_ENV"
# aws-lc-sys compiles against glibc headers, which redirect
# sscanf/strtol to C23 __isoc23_* symbols musl does not provide.
# Link a tiny forwarding shim so those references resolve to the
# musl implementations. Prototypes are declared manually: including
# glibc headers here would re-redirect the calls back to __isoc23_*.
cat > /tmp/isoc23-shim.c <<'SHIM'
#include <stdarg.h>
extern int vsscanf(const char *s, const char *fmt, va_list ap);
extern long strtol(const char *nptr, char **endptr, int base);
int __isoc23_sscanf(const char *s, const char *fmt, ...) {
va_list ap;
va_start(ap, fmt);
int r = vsscanf(s, fmt, ap);
va_end(ap);
return r;
}
long __isoc23_strtol(const char *nptr, char **endptr, int base) {
return strtol(nptr, endptr, base);
}
SHIM
aarch64-linux-gnu-gcc -O2 -c /tmp/isoc23-shim.c -o /tmp/isoc23-shim.o
# link-args land after libc on the link line; rescan musl libc so
# the shim's own vsscanf/strtol references resolve, and libgcc for
# the fp128 builtins musl's vfscanf/floatscan pull in on aarch64.
echo "CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_RUSTFLAGS=-Clink-arg=/tmp/isoc23-shim.o -Clink-arg=-lc -Clink-arg=-lgcc" >> "$GITHUB_ENV"
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
@@ -179,6 +210,10 @@ jobs:
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Download pre-built Rust binaries
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
uses: actions/download-artifact@v8
@@ -218,7 +253,7 @@ jobs:
sudo docker system prune -af --volumes
[ -d ~/.cache/go-build ] && rm -rf ~/.cache/go-build || true
[ -d /go/pkg ] && rm -rf /go/pkg || true
df -h
timeout 15 df -h || true
- name: Docker meta
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
@@ -253,7 +288,7 @@ jobs:
- name: Login to GHCR
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
uses: docker/login-action@v4.6.0
uses: docker/login-action@v4.7.0
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
@@ -270,8 +305,11 @@ jobs:
file: ${{ matrix.dockerfile }}
platforms: ${{ matrix.platform }}
labels: ${{ steps.docker_meta.outputs.labels }}
# Flat single-platform manifest so imagetools create assembles cleanly.
provenance: false
# mode=max keeps resolvedDependencies complete and records the
# builder identity; attestations ride inside each pushed index and
# survive the imagetools merge into the multi-arch manifest.
provenance: mode=max
sbom: true
outputs: type=image,name=${{ env.IMAGE }},push-by-digest=true,name-canonical=true,push=true
cache-from: type=gha,scope=${{ matrix.variant }}-${{ matrix.arch }}
# max only for rocksdb: its RocksDB compile is sha-independent and worth
@@ -281,7 +319,7 @@ jobs:
${{ matrix.build_args }}
BUILDKIT_INLINE_CACHE=1
BRANCH=${{ github.sha }}
${{ matrix.variant == 'rocksdb' && format('ROCKSDB_VERSION={0}', github.event.inputs.rocksdb_version || 'v10.10.1') || '' }}
${{ matrix.variant == 'rocksdb' && format('ROCKSDB_VERSION={0}', github.event.inputs.rocksdb_version || 'v11.1.2') || '' }}
- name: Export digest
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
@@ -307,6 +345,8 @@ jobs:
permissions:
contents: read
id-token: write
attestations: write
packages: write
strategy:
fail-fast: false
matrix:
@@ -338,7 +378,7 @@ jobs:
- name: Login to GHCR
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
uses: docker/login-action@v4.6.0
uses: docker/login-action@v4.7.0
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
@@ -360,7 +400,7 @@ jobs:
- name: Login to Docker Hub
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
uses: docker/login-action@v4.6.0
uses: docker/login-action@v4.7.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
@@ -414,6 +454,45 @@ jobs:
${{ env.IMAGE }}@${{ steps.manifest.outputs.digest }}
chrislusf/seaweedfs@${{ steps.manifest.outputs.digest }}
# Attest the merged multi-arch digest itself, so digest-pinning consumers
# can verify provenance against the exact index they pull. Both steps
# reuse the GHCR login above and push as OCI referrers.
- name: Attest build provenance for ${{ matrix.variant }}
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
uses: actions/attest-build-provenance@v4
with:
subject-name: ${{ env.IMAGE }}
subject-digest: ${{ steps.manifest.outputs.digest }}
push-to-registry: true
# SBOMs are generated and attested per platform manifest: an index-wide
# scan only describes syft's default platform, while the manifests it
# merges differ (some ship the real binary, others placeholders), so the
# SBOM has to be bound to the platform digest a consumer actually pulls.
- name: Install syft
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
uses: anchore/sbom-action/download-syft@v0
- name: Attest per-platform SBOMs for ${{ matrix.variant }}
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
working-directory: /tmp/digests
run: |
set -euo pipefail
for f in *; do
# Each saved digest is an OCI index (image + attestation manifests);
# scan and attest the image manifest a same-platform pull resolves.
ref="${{ env.IMAGE }}@sha256:${f}"
image=$(crane manifest "$ref" | jq -r '
[.manifests[]? | select((.annotations["vnd.docker.reference.type"] // "") != "attestation-manifest"
and (.platform.os // "unknown") != "unknown") | .digest][0] // empty')
ref="${{ env.IMAGE }}@${image:-sha256:${f}}"
syft "registry:${ref}" -o "spdx-json=/tmp/sbom-${f}.spdx.json"
cosign attest --yes \
--predicate "/tmp/sbom-${f}.spdx.json" \
--type spdxjson \
"$ref"
done
# Report-only trivy scan: uploads fixable HIGH/CRITICAL findings to GitHub
# Security for visibility, but never blocks the release. Releases (including
# `latest`) ship regardless — vulnerabilities are tracked, not gated, since
@@ -434,7 +513,7 @@ jobs:
variant: large_disk
steps:
- name: Login to GHCR
uses: docker/login-action@v4.6.0
uses: docker/login-action@v4.7.0
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
@@ -452,7 +531,9 @@ jobs:
scanners: vuln
vuln-type: os,library
severity: HIGH,CRITICAL
ignore-unfixed: true
# Keep unfixed findings in the report so digest-pinning consumers see
# the complete picture, not just what a patch could resolve.
ignore-unfixed: false
limit-severities-for-sarif: true
format: sarif
output: trivy-results.sarif
@@ -460,7 +541,7 @@ jobs:
- name: Upload Trivy scan results to GitHub Security
if: always()
uses: github/codeql-action/upload-sarif@v4.39.0
uses: github/codeql-action/upload-sarif@v4.39.1
with:
sarif_file: trivy-results.sarif
category: trivy-${{ matrix.variant }}
@@ -484,13 +565,13 @@ jobs:
latest_tag: latest_large_disk
steps:
- name: Login to Docker Hub
uses: docker/login-action@v4.6.0
uses: docker/login-action@v4.7.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Login to GHCR
uses: docker/login-action@v4.6.0
uses: docker/login-action@v4.7.0
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
@@ -4,9 +4,9 @@ on:
workflow_dispatch:
inputs:
rocksdb_version:
description: 'RocksDB git tag or branch to build (e.g. v10.10.1)'
description: 'RocksDB git tag or branch to build (e.g. v11.1.2)'
required: true
default: 'v10.10.1'
default: 'v11.1.2'
seaweedfs_ref:
description: 'SeaweedFS git tag, branch, or commit to build'
required: true
@@ -16,6 +16,11 @@ on:
required: false
default: ''
concurrency:
group: ${{ github.workflow }}-${{ github.event.inputs.image_tag || format('rocksdb_{0}_seaweedfs_{1}', github.event.inputs.rocksdb_version, github.event.inputs.seaweedfs_ref) }}
cancel-in-progress: false
queue: max
permissions:
contents: read
@@ -85,13 +90,13 @@ jobs:
echo "seaweedfs_ref=$seaweed" >> "$GITHUB_OUTPUT"
- name: Set up QEMU
uses: docker/setup-qemu-action@99012661954931238ded8c8b007157a8430204e1 # v1
uses: docker/setup-qemu-action@99012661954931238ded8c8b007157a8430204e1 # v4.4.0
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v1
uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0
- name: Login to Docker Hub
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v1
uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4.7.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
+1
View File
@@ -27,6 +27,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'telemetry/server/go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Build Telemetry Server
if: github.event_name == 'workflow_dispatch' && inputs.deploy
+4
View File
@@ -1,6 +1,10 @@
name: 'Dependency Review'
on: [pull_request]
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
+5 -3
View File
@@ -19,8 +19,8 @@ on:
- '.github/workflows/e2e.yml'
concurrency:
group: ${{ github.head_ref }}/e2e
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -32,7 +32,7 @@ defaults:
jobs:
e2e:
name: FUSE Mount
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
- name: Check out code into the Go module directory
@@ -42,6 +42,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Configure Docker Hub mirror
run: |
@@ -52,6 +53,7 @@ jobs:
uses: docker/setup-buildx-action@v4
- name: Cache Docker layers
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: /tmp/.buildx-cache
+6 -1
View File
@@ -18,18 +18,23 @@ on:
- 'go.sum'
- '.github/workflows/ec-integration-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
jobs:
ec-integration-tests:
name: EC Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
- name: Set up Go 1.x
uses: actions/setup-go@v7
with:
cache: ${{ runner.environment == 'github-hosted' }}
go-version: ^1.26
id: go
+4 -3
View File
@@ -19,8 +19,8 @@ on:
- '.github/actions/fix-fusermount-setuid/**'
concurrency:
group: ${{ github.head_ref || github.ref }}/fuse-dlm-integration
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -28,7 +28,7 @@ permissions:
jobs:
fuse-dlm-integration:
name: FUSE DLM Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
@@ -39,6 +39,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Configure FUSE
run: |
+7 -3
View File
@@ -24,8 +24,8 @@ on:
- '.github/actions/fix-fusermount-setuid/**'
concurrency:
group: ${{ github.head_ref || github.ref }}/fuse-failover
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -33,7 +33,7 @@ permissions:
jobs:
fuse-failover:
name: FUSE Volume Server Failover
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 40
steps:
@@ -42,10 +42,14 @@ jobs:
with:
persist-credentials: false
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Configure FUSE
run: |
+4 -3
View File
@@ -17,8 +17,8 @@ on:
- '.github/actions/fix-fusermount-setuid/**'
concurrency:
group: ${{ github.head_ref }}/fuse-integration
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -26,7 +26,7 @@ permissions:
jobs:
fuse-integration:
name: FUSE Integration Testing
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 50
steps:
@@ -37,6 +37,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Configure FUSE
run: |
+7 -3
View File
@@ -25,8 +25,8 @@ on:
- '.github/actions/fix-fusermount-setuid/**'
concurrency:
group: ${{ github.head_ref || github.ref }}/fuse-p2p-integration
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -34,17 +34,21 @@ permissions:
jobs:
fuse-p2p-integration:
name: FUSE P2P Peer Chunk Sharing
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Configure FUSE
run: |
+9 -2
View File
@@ -17,8 +17,8 @@ on:
- '.github/workflows/go.yml'
concurrency:
group: ${{ github.head_ref }}/go
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -35,6 +35,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Get dependencies
run: |
cd weed; go get -v -t -d ./...
@@ -57,6 +58,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Go Vet linux/386 (type-checks code and tests for 32-bit int overflows)
run: |
GOOS=linux GOARCH=386 go vet ./... 2>&1 | grep -v "MessageState contains sync.Mutex" | grep -v "IdentityAccessManagement contains sync.RWMutex" | tee vet-32bit-output.txt
@@ -72,6 +74,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Build
run: cd weed; go build -tags "elastic gocdk sqlite ydb tarantool tikv rclone" -v .
@@ -94,6 +97,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
# Nothing else on a PR compiles these, so per-OS syscall constants creep
# back into shared files unnoticed and only a release build catches it.
- name: Cross-compile
@@ -128,6 +132,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Test
env:
RUN_REDIS_TESTS: "1"
@@ -143,6 +148,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
# 386 test binaries run natively on the amd64 runner. This catches what vet
# can't: unaligned 64-bit atomics and arithmetic that wraps at runtime.
# -short skips the e2e suites already covered on amd64.
@@ -159,6 +165,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
# weed/kms/azure is excluded from every default build, so without this
# nothing compiles it and tag-gated code silently rots.
- name: Build and test with azurekms
+4
View File
@@ -8,6 +8,10 @@ on:
branches: [ master ]
paths: ['k8s/**', '.github/workflows/helm_ci.yml', '.github/scripts/helm_jwt_keys.py']
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
+7 -1
View File
@@ -14,6 +14,10 @@ on:
- 'weed/**'
- '.github/workflows/java_integration_tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
jobs:
test:
name: Java Integration Tests
@@ -31,6 +35,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Set up Java
@@ -38,7 +43,7 @@ jobs:
with:
java-version: ${{ matrix.java }}
distribution: 'temurin'
cache: 'maven'
cache: ${{ runner.environment == 'github-hosted' && 'maven' || '' }}
- name: Build SeaweedFS
run: |
@@ -55,6 +60,7 @@ jobs:
# Start SeaweedFS with optimized settings for CI
# Include S3 API for s3copier integration tests
weed server -dir="$WEED_DATA_DIR" \
-ip=127.0.0.1 -ip.bind=127.0.0.1 \
-master.raftHashicorp \
-master.electionTimeout=1s \
-master.volumeSizeLimitMB=100 \
+1 -1
View File
@@ -77,7 +77,7 @@ jobs:
with:
java-version: '17'
distribution: 'temurin'
cache: 'maven'
cache: ${{ runner.environment == 'github-hosted' && 'maven' || '' }}
server-id: central
server-username: MAVEN_CENTRAL_USERNAME
server-password: MAVEN_CENTRAL_PASSWORD
+5 -1
View File
@@ -12,6 +12,10 @@ on:
- 'other/java/**'
- '.github/workflows/java_unit_tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
jobs:
test:
name: Java Unit Tests
@@ -30,7 +34,7 @@ jobs:
with:
java-version: ${{ matrix.java }}
distribution: 'temurin'
cache: 'maven'
cache: ${{ runner.environment == 'github-hosted' && 'maven' || '' }}
- name: Build and Install SeaweedFS Client
working-directory: other/java/client
+3 -3
View File
@@ -24,8 +24,8 @@ on:
workflow_dispatch: # Allow manual trigger
concurrency:
group: ${{ github.head_ref }}/kafka-quicktest
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -43,7 +43,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version: ^1.26
cache: true
cache: ${{ runner.environment == 'github-hosted' }}
cache-dependency-path: |
**/go.sum
id: go
+9 -6
View File
@@ -23,8 +23,8 @@ on:
- '.github/workflows/kafka-tests.yml'
concurrency:
group: ${{ github.head_ref }}/kafka-tests
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
# Force different runners for better isolation
env:
@@ -53,6 +53,7 @@ jobs:
- name: Set up Go 1.x
uses: actions/setup-go@v7
with:
cache: ${{ runner.environment == 'github-hosted' }}
go-version: ^1.26
id: go
@@ -98,6 +99,7 @@ jobs:
- name: Set up Go 1.x
uses: actions/setup-go@v7
with:
cache: ${{ runner.environment == 'github-hosted' }}
go-version: ^1.26
id: go
@@ -149,7 +151,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version: ^1.26
cache: true
cache: ${{ runner.environment == 'github-hosted' }}
cache-dependency-path: |
**/go.sum
id: go
@@ -328,7 +330,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version: ^1.26
cache: true
cache: ${{ runner.environment == 'github-hosted' }}
cache-dependency-path: |
**/go.sum
id: go
@@ -490,7 +492,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version: ^1.26
cache: true
cache: ${{ runner.environment == 'github-hosted' }}
cache-dependency-path: |
**/go.sum
id: go
@@ -648,7 +650,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version: ^1.26
cache: true
cache: ${{ runner.environment == 'github-hosted' }}
cache-dependency-path: |
**/go.sum
id: go
@@ -805,6 +807,7 @@ jobs:
- name: Set up Go 1.x
uses: actions/setup-go@v7
with:
cache: ${{ runner.environment == 'github-hosted' }}
go-version: ^1.26
id: go
+6 -4
View File
@@ -19,8 +19,8 @@ on:
- 'test/kms/**'
concurrency:
group: ${{ github.head_ref || github.ref }}-kms-tests
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -32,7 +32,7 @@ defaults:
jobs:
kms-provider-tests:
name: KMS Provider Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
@@ -43,6 +43,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -77,7 +78,7 @@ jobs:
s3-kms-e2e-tests:
name: S3 KMS End-to-End Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 25
steps:
@@ -88,6 +89,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -21,8 +21,8 @@ on:
- '.github/workflows/master-cold-start-tests.yml'
concurrency:
group: ${{ github.head_ref || github.ref }}/master-cold-start-tests
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -30,7 +30,7 @@ permissions:
jobs:
master-cold-start-tests:
name: Master Cold Start Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 10
steps:
- name: Check out code
@@ -42,6 +42,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Build weed binary
run: |
@@ -25,8 +25,8 @@ on:
- '.github/workflows/metadata-subscribe-tests.yml'
concurrency:
group: ${{ github.head_ref }}/metadata-subscribe-tests
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -37,7 +37,7 @@ env:
jobs:
metadata-subscribe-integration:
name: Metadata Subscribe Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
@@ -48,6 +48,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Build SeaweedFS
run: |
+4 -2
View File
@@ -10,8 +10,8 @@ on:
branches: [ 'winfsp-bench**' ]
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -32,6 +32,7 @@ jobs:
- uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Install WinFsp and rclone
run: choco install winfsp rclone -y --no-progress
@@ -150,6 +151,7 @@ jobs:
- uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Repair the fusermount3 setuid bit
uses: ./.github/actions/fix-fusermount-setuid
@@ -24,8 +24,8 @@ on:
- '.github/workflows/mount-windows-conformance.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -49,6 +49,7 @@ jobs:
- uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
# cgofuse loads winfsp-x64.dll at run time, so WinFsp is needed here but
# not to build.
+3 -2
View File
@@ -24,8 +24,8 @@ on:
- '.github/workflows/mount-windows.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -49,6 +49,7 @@ jobs:
- uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
# cgofuse loads winfsp-x64.dll at run time, so WinFsp is needed here but
# not to build.
+4 -3
View File
@@ -21,8 +21,8 @@ on:
- '.github/workflows/multi-master-tests.yml'
concurrency:
group: ${{ github.head_ref || github.ref }}/multi-master-tests
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -30,7 +30,7 @@ permissions:
jobs:
multi-master-failover-tests:
name: Multi-Master Failover Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 10
steps:
- name: Check out code
@@ -40,6 +40,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Install SeaweedFS
run: |
+7 -4
View File
@@ -45,8 +45,8 @@ on:
type: string
concurrency:
group: ${{ github.head_ref || github.ref }}/performance
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -67,6 +67,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Build weed
run: go build -o weed_bin ./weed
@@ -175,13 +176,14 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Install Rust toolchain
if: matrix.impl == 'rust'
uses: dtolnay/rust-toolchain@stable
- name: Cache cargo registry and target
if: matrix.impl == 'rust'
if: matrix.impl == 'rust' && runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
@@ -327,13 +329,14 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Install Rust toolchain
if: matrix.impl == 'rust'
uses: dtolnay/rust-toolchain@stable
- name: Cache cargo registry and target
if: matrix.impl == 'rust'
if: matrix.impl == 'rust' && runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
+10 -2
View File
@@ -27,7 +27,7 @@ permissions:
jobs:
pjdfstest:
name: pjdfstest
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 60
steps:
@@ -38,15 +38,23 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Start local Docker registry
run: docker run -d --restart=always -p 5000:5000 --name registry registry:2
run: |
if ! docker start registry 2>/dev/null; then
docker rm -f registry >/dev/null 2>&1 || true
docker run -d --restart=always -p 5000:5000 --name registry registry:2
fi
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4
with:
driver-opts: network=host
- name: Set up Docker Compose
uses: docker/setup-compose-action@v2
- name: Build weed race binary
run: |
cd docker
+6 -1
View File
@@ -18,13 +18,17 @@ on:
- 'go.sum'
- '.github/workflows/plugin-workers.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
jobs:
plugin-worker:
name: "Plugin Worker: ${{ matrix.worker }}"
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 10
strategy:
fail-fast: false
@@ -41,6 +45,7 @@ jobs:
- name: Set up Go 1.x
uses: actions/setup-go@v7
with:
cache: ${{ runner.environment == 'github-hosted' }}
go-version: ^1.26
id: go
+791
View File
@@ -0,0 +1,791 @@
name: POSIX Compliance Tests
on:
push:
branches: [ main, master, develop ]
paths:
- 'weed/mount/**'
- 'weed/command/mount*'
- 'weed/command/fuse*'
- 'test/fuse_integration/**'
pull_request:
branches: [ main, master, develop ]
paths:
- 'weed/mount/**'
- 'weed/command/mount*'
- 'weed/command/fuse*'
- 'test/fuse_integration/**'
workflow_dispatch:
inputs:
test_type:
description: 'Type of POSIX tests to run (default: full for comprehensive testing)'
required: true
default: 'full'
type: choice
options:
- critical
- basic
- extended
- full
enable_external_tests:
description: 'Run external test suites (slower)'
required: false
default: false
type: boolean
permissions:
contents: read
pull-requests: write
issues: write
env:
GO_VERSION: '1.26'
TIMEOUT: '45m'
jobs:
posix-compliance-ubuntu:
runs-on: ubuntu-latest
timeout-minutes: 60
strategy:
matrix:
fuse-version: ['2.9', '3.0']
fail-fast: false
steps:
- name: Checkout code
uses: actions/checkout@v7
with:
fetch-depth: 1
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go ${{ env.GO_VERSION }}
uses: actions/setup-go@v7
with:
go-version: ${{ env.GO_VERSION }}
cache: ${{ runner.environment == 'github-hosted' }}
cache-dependency-path: |
go.sum
test/fuse_integration/go.sum
- name: Install system dependencies
run: |
sudo apt-get update
sudo apt-get install -y \
attr \
acl \
build-essential \
git \
python3-pip
# Install FUSE version specific packages
if [ "${{ matrix.fuse-version }}" = "2.9" ]; then
echo "Installing FUSE 2.9..."
sudo apt-get install -y fuse libfuse-dev
elif [ "${{ matrix.fuse-version }}" = "3.0" ]; then
echo "Installing FUSE 3.0..."
sudo apt-get install -y fuse3 libfuse3-dev
# Also install fuse2 for compatibility if needed
sudo apt-get install -y fuse libfuse-dev
fi
- name: Set up user permissions for FUSE
run: |
# Create fuse group if it doesn't exist
sudo groupadd -f fuse
# Add user to fuse group
sudo usermod -a -G fuse $USER
# Set permissions on FUSE device
sudo chmod 666 /dev/fuse
# Allow non-root mounts to pass -o allow_other (weed -allowOthers)
echo 'user_allow_other' | sudo tee -a /etc/fuse.conf
sudo chmod 644 /etc/fuse.conf
# Ensure fuse module is loaded
sudo modprobe fuse || true
# Verify setup
echo "FUSE setup verification:"
ls -la /dev/fuse
groups $USER | grep fuse || echo "User not in fuse group yet (will be effective after login)"
lsmod | grep fuse || echo "FUSE module not loaded"
- name: Install external test tools
if: ${{ github.event.inputs.enable_external_tests == 'true' }}
run: |
# Install nfstest for POSIX API verification
pip3 install --user nfstest
# Install FIO for performance testing
sudo apt-get install -y fio
# Install additional test utilities
sudo apt-get install -y stress-ng
- name: Build SeaweedFS
run: |
make
# Verify binary exists and is executable (it's installed to GOPATH/bin)
weed version
# Make weed binary available in PATH (it's already there from go install)
sudo cp $(which weed) /usr/local/bin/weed
which weed
weed version
- name: Set up SeaweedFS cluster
run: |
start_and_wait() {
local name=$1
local url=$2
local pidfile=$3
shift 3
local cmd="$@"
echo "Starting $name..."
$cmd > "/tmp/seaweedfs/$name.log" 2>&1 &
local pid=$!
echo $pid > "$pidfile"
echo "Waiting for $name to start..."
for i in {1..30}; do
if curl -sf "$url" > /dev/null 2>&1; then
echo "$name is ready"
return 0
fi
if [ $i -eq 30 ]; then
echo "$name failed to start"
cat "/tmp/seaweedfs/$name.log"
exit 1
fi
sleep 2
done
}
mkdir -p /tmp/seaweedfs/{master,volume,filer,mount}
start_and_wait "master" "http://127.0.0.1:9333/cluster/status" "/tmp/seaweedfs/master.pid" \
"weed master -ip=127.0.0.1 -port=9333 -mdir=/tmp/seaweedfs/master -raftBootstrap=true"
start_and_wait "volume" "http://127.0.0.1:8080/status" "/tmp/seaweedfs/volume.pid" \
"weed volume -mserver=127.0.0.1:9333 -ip=127.0.0.1 -port=8080 -dir=/tmp/seaweedfs/volume -max=100"
start_and_wait "filer" "http://127.0.0.1:8888/" "/tmp/seaweedfs/filer.pid" \
"weed filer -master=127.0.0.1:9333 -ip=127.0.0.1 -port=8888"
echo "SeaweedFS cluster status:"
curl -s http://127.0.0.1:9333/cluster/status || true
- name: Set up FUSE mount
run: |
# Create mount point
MOUNT_POINT="/tmp/seaweedfs/mount"
mkdir -p $MOUNT_POINT
echo "Mounting SeaweedFS FUSE filesystem..."
# Mount SeaweedFS FUSE filesystem in background
weed mount \
-filer=127.0.0.1:8888 \
-dir=$MOUNT_POINT \
-filer.path=/ \
-dirAutoCreate=true \
-allowOthers=true \
-nonempty=true \
> /tmp/seaweedfs/mount.log 2>&1 &
MOUNT_PID=$!
echo $MOUNT_PID > /tmp/seaweedfs/mount.pid
# Wait for mount to be ready. The mount point directory already
# exists, so only a real mount counts — a directory listing would
# pass on the local disk even if weed mount died.
echo "Waiting for FUSE mount to be ready..."
for i in {1..30}; do
if mountpoint -q $MOUNT_POINT 2>/dev/null; then
echo "FUSE mount is ready"
break
fi
if ! kill -0 $MOUNT_PID 2>/dev/null; then
echo "FUSE mount process exited"
cat /tmp/seaweedfs/mount.log
exit 1
fi
if [ $i -eq 30 ]; then
echo "FUSE mount failed to be ready"
cat /tmp/seaweedfs/mount.log
echo "Mount status:"
mount | grep fuse || echo "No FUSE mounts found"
ps aux | grep weed || echo "No weed processes found"
exit 1
fi
sleep 2
done
# Verify mount is working
echo "Testing FUSE mount functionality..."
echo "test" > $MOUNT_POINT/test_file.txt
if [ "$(cat $MOUNT_POINT/test_file.txt)" = "test" ]; then
echo "FUSE mount is working correctly"
rm $MOUNT_POINT/test_file.txt
else
echo "FUSE mount is not working properly"
exit 1
fi
# Export mount point for tests
echo "SEAWEEDFS_MOUNT_POINT=$MOUNT_POINT" >> $GITHUB_ENV
- name: Set up test environment
run: |
cd test/fuse_integration
# Initialize Go module for tests
if [ ! -f go.mod ]; then
go mod init seaweedfs-posix-tests
go mod tidy
fi
# Create reports directory
mkdir -p reports
# Set up external tools if requested
# Note: External tools setup is now optional and won't fail the build
if [ "${{ github.event.inputs.enable_external_tests }}" = "true" ]; then
echo "Setting up external POSIX testing tools..."
make -f posix_Makefile setup-external-tools || true
# setup-external-tools exits 0 even when a sub-target failed, so
# require the tools only when they are actually installed
if [ -d external_tools/pjdfstest ] && command -v nfstest_posix >/dev/null 2>&1; then
export POSIX_REQUIRE_EXTERNAL_TOOLS="true"
echo "POSIX_REQUIRE_EXTERNAL_TOOLS=true" >> $GITHUB_ENV
echo "✅ External tools setup completed successfully"
else
echo "⚠️ External tools setup incomplete - continuing with basic tests only"
echo "External tools will be skipped but tests will continue"
echo "This is expected behavior when build tools are not available"
fi
fi
# Verify SeaweedFS cluster is accessible
echo "Verifying SeaweedFS cluster accessibility..."
curl -sf http://127.0.0.1:9333/cluster/status
curl -sf http://127.0.0.1:8080/status
curl -sf http://127.0.0.1:8888/
# Verify mount point
echo "Mount point: $SEAWEEDFS_MOUNT_POINT"
ls -la $SEAWEEDFS_MOUNT_POINT
# Verify mount is actually working by testing file operations
echo "Verifying SeaweedFS mount functionality..."
TEST_FILE="$SEAWEEDFS_MOUNT_POINT/.mount_verification_test"
echo "SeaweedFS mount test" > "$TEST_FILE"
if [ "$(cat "$TEST_FILE")" = "SeaweedFS mount test" ]; then
echo "✅ Mount verification successful"
rm -f "$TEST_FILE"
else
echo "❌ Mount verification failed - mount may not be working properly"
exit 1
fi
# Check if this is actually a FUSE mount — the plain directory created
# above passes the file test, so without this the suite can run
# against the local disk and report a pass it never earned
if mount | grep -q "$SEAWEEDFS_MOUNT_POINT"; then
echo "✅ FUSE mount detected:"
mount | grep "$SEAWEEDFS_MOUNT_POINT"
else
echo "❌ Mount point not found in mount table - refusing to test a local directory"
exit 1
fi
- name: Run POSIX compliance tests
id: posix-tests
env:
SEAWEEDFS_MOUNT_POINT: ${{ env.SEAWEEDFS_MOUNT_POINT }}
POSIX_REQUIRE_EXTERNAL_TOOLS: ${{ env.POSIX_REQUIRE_EXTERNAL_TOOLS }}
run: |
cd test/fuse_integration
# Determine which tests to run
TEST_TYPE="${{ github.event.inputs.test_type }}"
if [ -z "$TEST_TYPE" ]; then
TEST_TYPE="full"
fi
echo "Running POSIX tests: $TEST_TYPE"
if [ "$TEST_TYPE" = "full" ]; then
echo "🚀 Running COMPREHENSIVE POSIX test suite (including external test suites)"
else
echo "Using mount point: $SEAWEEDFS_MOUNT_POINT"
fi
# Set test configuration to use our mounted filesystem
export TEST_MOUNT_POINT="$SEAWEEDFS_MOUNT_POINT"
export TEST_SKIP_CLUSTER_SETUP="true"
# Debug: Show environment variables
echo "🔍 Test Environment:"
echo " TEST_MOUNT_POINT=$TEST_MOUNT_POINT"
echo " TEST_SKIP_CLUSTER_SETUP=$TEST_SKIP_CLUSTER_SETUP"
echo " SEAWEEDFS_MOUNT_POINT=$SEAWEEDFS_MOUNT_POINT"
case "$TEST_TYPE" in
"critical")
echo "Running critical POSIX tests (basic compliance only)..."
make -f posix_Makefile test-posix-critical
;;
"basic")
echo "Running basic POSIX tests (core file operations)..."
make -f posix_Makefile test-posix-basic
;;
"extended")
echo "Running extended POSIX tests (including advanced features)..."
make -f posix_Makefile test-posix-extended
;;
"full")
echo "🚀 Running COMPREHENSIVE POSIX test suite (all tests including external)..."
echo " This includes: basic, extended, and external test suites"
echo " External suites: pjdfstest, nfstest_posix, and custom stress tests"
make -f posix_Makefile test-posix-full
;;
*)
echo "Unknown test type: $TEST_TYPE, defaulting to full suite"
echo "🚀 Running full POSIX test suite (all tests including external)..."
make -f posix_Makefile test-posix-full
;;
esac
- name: Run external test suites
if: ${{ github.event.inputs.enable_external_tests == 'true' }}
continue-on-error: true
run: |
cd test/fuse_integration
# Point the external suites at the same mount as the Go tests;
# without these the Makefile falls back to a local-disk path.
export TEST_MOUNT_POINT="$SEAWEEDFS_MOUNT_POINT"
export TEST_SKIP_CLUSTER_SETUP="true"
# Run external tests (may fail on some systems)
make -f posix_Makefile test-nfstest-posix || echo "nfstest failed or not available"
make -f posix_Makefile test-fio-posix || echo "FIO tests failed or not available"
- name: Generate compliance report
if: always()
run: |
cd test/fuse_integration
make -f posix_Makefile generate-report
- name: Collect system information for debugging
if: failure()
run: |
cd test/fuse_integration
mkdir -p reports/debug
echo "System Information" > reports/debug/system_info.txt
uname -a >> reports/debug/system_info.txt
echo "" >> reports/debug/system_info.txt
echo "Mount Information" >> reports/debug/system_info.txt
mount >> reports/debug/system_info.txt
echo "" >> reports/debug/system_info.txt
echo "Disk Space" >> reports/debug/system_info.txt
timeout 15 df -h >> reports/debug/system_info.txt || true
echo "" >> reports/debug/system_info.txt
echo "Memory Information" >> reports/debug/system_info.txt
free -h >> reports/debug/system_info.txt
echo "" >> reports/debug/system_info.txt
echo "FUSE Information" >> reports/debug/system_info.txt
ls -la /dev/fuse >> reports/debug/system_info.txt
lsmod | grep fuse >> reports/debug/system_info.txt
echo "" >> reports/debug/system_info.txt
echo "SeaweedFS Version" >> reports/debug/system_info.txt
weed version >> reports/debug/system_info.txt
echo "" >> reports/debug/system_info.txt
echo "Go Version" >> reports/debug/system_info.txt
go version >> reports/debug/system_info.txt
echo "" >> reports/debug/system_info.txt
echo "Running Processes" >> reports/debug/system_info.txt
ps aux | grep -E "(weed|fuse)" >> reports/debug/system_info.txt
echo "" >> reports/debug/system_info.txt
# Collect SeaweedFS service logs
echo "=== SeaweedFS Service Logs ===" >> reports/debug/system_info.txt
if [ -f /tmp/seaweedfs/master.log ]; then
echo "Master Log:" >> reports/debug/system_info.txt
tail -50 /tmp/seaweedfs/master.log >> reports/debug/system_info.txt
echo "" >> reports/debug/system_info.txt
fi
if [ -f /tmp/seaweedfs/volume.log ]; then
echo "Volume Log:" >> reports/debug/system_info.txt
tail -50 /tmp/seaweedfs/volume.log >> reports/debug/system_info.txt
echo "" >> reports/debug/system_info.txt
fi
if [ -f /tmp/seaweedfs/filer.log ]; then
echo "Filer Log:" >> reports/debug/system_info.txt
tail -50 /tmp/seaweedfs/filer.log >> reports/debug/system_info.txt
echo "" >> reports/debug/system_info.txt
fi
if [ -f /tmp/seaweedfs/mount.log ]; then
echo "Mount Log:" >> reports/debug/system_info.txt
tail -50 /tmp/seaweedfs/mount.log >> reports/debug/system_info.txt
echo "" >> reports/debug/system_info.txt
fi
# Copy full logs as separate files
if [ -f /tmp/seaweedfs/master.log ]; then
cp /tmp/seaweedfs/master.log reports/debug/master.log
fi
if [ -f /tmp/seaweedfs/volume.log ]; then
cp /tmp/seaweedfs/volume.log reports/debug/volume.log
fi
if [ -f /tmp/seaweedfs/filer.log ]; then
cp /tmp/seaweedfs/filer.log reports/debug/filer.log
fi
if [ -f /tmp/seaweedfs/mount.log ]; then
cp /tmp/seaweedfs/mount.log reports/debug/mount.log
fi
- name: Cleanup SeaweedFS cluster and FUSE mount
if: always()
run: |
stop_process() {
local name=$1
local pidfile=$2
if [ -f "$pidfile" ]; then
local pid=$(cat "$pidfile")
if kill -0 $pid 2>/dev/null; then
echo "Stopping $name process (PID: $pid)..."
kill -TERM $pid || true
sleep 2
kill -KILL $pid 2>/dev/null || true
fi
fi
}
echo "Cleaning up SeaweedFS cluster and FUSE mount..."
MOUNT_POINT="/tmp/seaweedfs/mount"
if mountpoint -q $MOUNT_POINT 2>/dev/null; then
echo "Unmounting FUSE filesystem..."
fusermount -u $MOUNT_POINT || umount $MOUNT_POINT || true
fi
stop_process "mount" "/tmp/seaweedfs/mount.pid"
stop_process "filer" "/tmp/seaweedfs/filer.pid"
stop_process "volume" "/tmp/seaweedfs/volume.pid"
stop_process "master" "/tmp/seaweedfs/master.pid"
pkill -f "weed " || true
fusermount -u $MOUNT_POINT 2>/dev/null || true
umount $MOUNT_POINT 2>/dev/null || true
rm -rf /tmp/seaweedfs || true
echo "Cleanup completed"
- name: Upload test results
uses: actions/upload-artifact@v7
if: always()
with:
name: posix-test-results-ubuntu-fuse${{ matrix.fuse-version }}
path: |
test/fuse_integration/reports/
test/fuse_integration/*.log
retention-days: 30
- name: Upload test coverage
uses: actions/upload-artifact@v7
if: always() && (github.event.inputs.test_type == 'full' || github.event.inputs.test_type == 'extended')
with:
name: posix-coverage-ubuntu
path: test/fuse_integration/reports/posix_coverage.html
retention-days: 14
- name: Comment PR with results
if: github.event_name == 'pull_request' && always()
uses: actions/github-script@v9
with:
script: |
const fs = require('fs');
const path = require('path');
const reportPath = 'test/fuse_integration/reports/posix_compliance_summary.txt';
if (fs.existsSync(reportPath)) {
const report = fs.readFileSync(reportPath, 'utf8');
const comment = `## POSIX Compliance Test Results
**Go Version:** ${{ env.GO_VERSION }}
**FUSE Version:** ${{ matrix.fuse-version }}
**Test Type:** ${{ github.event.inputs.test_type || 'full' }}
<details>
<summary>Test Summary</summary>
\`\`\`
${report}
\`\`\`
</details>
📊 Full results available in [test artifacts](https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }})
`;
try {
await github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: comment
});
} catch (err) {
console.log('Could not post PR comment: ' + err.message);
}
}
posix-compliance-macos:
runs-on: macos-latest
timeout-minutes: 60
steps:
- name: Checkout code
uses: actions/checkout@v7
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version: ${{ env.GO_VERSION }}
cache: ${{ runner.environment == 'github-hosted' }}
- name: Install macFUSE
run: |
# Install macFUSE
brew install macfuse
# Note: macFUSE may require system extension approval on macOS
# This step may need manual intervention in some cases
- name: Build SeaweedFS
run: |
make
# Verify binary exists and is executable (it's installed to GOPATH/bin)
weed version
- name: Run critical POSIX tests (macOS)
continue-on-error: true # macOS FUSE can be more restrictive
run: |
cd test/fuse_integration
if [ ! -f go.mod ]; then
go mod init seaweedfs-posix-tests
go mod tidy
fi
mkdir -p reports
# Run basic tests only on macOS due to FUSE limitations
make -f posix_Makefile test-posix-critical
- name: Upload macOS test results
uses: actions/upload-artifact@v7
if: always()
with:
name: posix-test-results-macos
path: |
test/fuse_integration/reports/
test/fuse_integration/*.log
retention-days: 30
security-analysis:
runs-on: ubuntu-latest
if: github.event_name == 'pull_request'
steps:
- name: Checkout code
uses: actions/checkout@v7
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version: ${{ env.GO_VERSION }}
cache: ${{ runner.environment == 'github-hosted' }}
- name: Install security analysis tools
run: |
echo "Installing staticcheck for security analysis..."
go install honnef.co/go/tools/cmd/staticcheck@latest
- name: Run security analysis on FUSE code
run: |
# Analyze mount and FUSE-related code for security and quality issues
echo "Running staticcheck security analysis..."
# Run staticcheck on FUSE-related code
echo "Analyzing FUSE mount code..."
staticcheck ./weed/mount/... > staticcheck-report.txt 2>&1 || true
staticcheck ./weed/command/mount* ./weed/command/fuse* >> staticcheck-report.txt 2>&1 || true
# Convert to JSON format for consistency
echo "{" > gosec-report.json
echo " \"tool\": \"staticcheck\"," >> gosec-report.json
echo " \"timestamp\": \"$(date -Iseconds)\"," >> gosec-report.json
echo " \"issues\": [" >> gosec-report.json
if [ -s staticcheck-report.txt ]; then
echo " \"$(cat staticcheck-report.txt | head -20 | sed 's/"/\\"/g' | tr '\n' ' ')\"" >> gosec-report.json
fi
echo " ]," >> gosec-report.json
echo " \"stats\": {" >> gosec-report.json
echo " \"files_analyzed\": $(find ./weed/mount ./weed/command -name '*.go' | wc -l)," >> gosec-report.json
echo " \"issues_found\": $(wc -l < staticcheck-report.txt 2>/dev/null || echo 0)" >> gosec-report.json
echo " }" >> gosec-report.json
echo "}" >> gosec-report.json
echo "Security analysis completed"
echo "Issues found: $(wc -l < staticcheck-report.txt 2>/dev/null || echo 0)"
- name: Upload security analysis results
uses: actions/upload-artifact@v7
if: always()
with:
name: security-analysis-results
path: gosec-report.json
retention-days: 30
performance-baseline:
runs-on: ubuntu-latest
if: github.event.inputs.test_type == 'full' || github.event_name == 'schedule'
steps:
- name: Checkout code
uses: actions/checkout@v7
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version: ${{ env.GO_VERSION }}
cache: ${{ runner.environment == 'github-hosted' }}
- name: Install dependencies
run: |
sudo apt-get update
sudo apt-get install -y fuse libfuse-dev fio
sudo usermod -a -G fuse $USER
sudo chmod 666 /dev/fuse
# The posix suites mount with -allowOthers; allow_other needs this
echo 'user_allow_other' | sudo tee -a /etc/fuse.conf
- name: Build SeaweedFS
run: make
- name: Run performance baseline tests
run: |
cd test/fuse_integration
if [ ! -f go.mod ]; then
go mod init seaweedfs-posix-tests
go mod tidy
fi
mkdir -p reports
# Run performance benchmarks
make -f posix_Makefile benchmark-posix
# This job has no FUSE mount, so the fio suite can only skip; keep the
# run green but honest about producing no mounted-FS results
make -f posix_Makefile test-fio-posix || echo "FIO skipped - no mounted filesystem in this job"
- name: Store performance baseline
uses: actions/upload-artifact@v7
with:
name: performance-baseline-results
path: |
test/fuse_integration/reports/posix_benchmark_results.log
test/fuse_integration/reports/fio_*_results.log
retention-days: 90
# Schedule regular compliance checks
scheduled-compliance-check:
runs-on: ubuntu-latest
if: github.event_name == 'schedule'
steps:
- name: Checkout code
uses: actions/checkout@v7
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version: ${{ env.GO_VERSION }}
cache: ${{ runner.environment == 'github-hosted' }}
- name: Install dependencies
run: |
sudo apt-get update
sudo apt-get install -y fuse libfuse-dev
pip3 install --user nfstest
# The posix suites mount with -allowOthers; allow_other needs this
echo 'user_allow_other' | sudo tee -a /etc/fuse.conf
- name: Build SeaweedFS
run: make
- name: Run comprehensive compliance check
run: |
cd test/fuse_integration
make -f posix_Makefile test-posix-full
make -f posix_Makefile generate-report
- name: Create compliance issue if tests fail
if: failure()
uses: actions/github-script@v9
with:
script: |
const issue = await github.rest.issues.create({
owner: context.repo.owner,
repo: context.repo.repo,
title: 'POSIX Compliance Check Failed - ' + new Date().toISOString().split('T')[0],
body: `## POSIX Compliance Check Failure
The scheduled POSIX compliance check has failed. This may indicate:
- Regression in FUSE mount functionality
- Changes affecting POSIX compatibility
- Infrastructure issues with the test environment
**Action Required:** Review the test results and investigate any failures.
**Test Run:** [Workflow Run](https://github.com/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId})
**Date:** ${new Date().toISOString()}
---
This issue was automatically created by the POSIX compliance workflow.
`,
labels: ['bug', 'posix-compliance', 'automated-issue']
});
console.log('Created issue:', issue.data.number);
# Schedule this workflow to run weekly
# Uncomment the following lines to enable scheduled runs:
# schedule:
# - cron: '0 2 * * 1' # Every Monday at 2 AM UTC
+4 -2
View File
@@ -23,8 +23,8 @@ on:
- '.github/workflows/postgres-tests.yml'
concurrency:
group: ${{ github.head_ref }}/postgres-tests
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -41,6 +41,7 @@ jobs:
- name: Set up Go 1.x
uses: actions/setup-go@v7
with:
cache: ${{ runner.environment == 'github-hosted' }}
go-version: ^1.26
id: go
@@ -56,6 +57,7 @@ jobs:
uses: docker/setup-buildx-action@v4
- name: Cache Docker layers
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: /tmp/.buildx-cache
+124
View File
@@ -0,0 +1,124 @@
name: "Rust Vector Tests"
on:
pull_request:
branches: [ master ]
paths:
- 'seaweed-vector/**'
- 'seaweed-worker/**'
- 'seaweed-common/**'
- 'weed/pb/plugin.proto'
- 'weed/pb/vector.proto'
- 'weed/pb/vector_pb/**'
- 'weed/s3api/**'
- 'weed/command/**'
- 'test/vectors/**'
- '.github/workflows/rust-vector-tests.yml'
push:
branches: [ master, main ]
paths:
- 'seaweed-vector/**'
- 'seaweed-worker/**'
- 'seaweed-common/**'
- 'weed/pb/plugin.proto'
- 'weed/pb/vector.proto'
- 'weed/pb/vector_pb/**'
- 'weed/s3api/**'
- 'weed/command/**'
- 'test/vectors/**'
- '.github/workflows/rust-vector-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
jobs:
rust-vector-build:
name: Rust Vector Build and Unit Tests
runs-on: ubuntu-24.04
timeout-minutes: 45
steps:
- name: Checkout code
uses: actions/checkout@v7
with:
persist-credentials: false
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
- name: Fingerprint build toolchain
id: toolchain
run: echo "fingerprint=$(getconf GNU_LIBC_VERSION | tr ' ' '-')-rustc-$(rustc -V | awk '{print $2}')" >> "$GITHUB_OUTPUT"
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
~/.cargo/registry
~/.cargo/git
seaweed-vector/target/release
key: rust-vector-${{ steps.toolchain.outputs.fingerprint }}-${{ hashFiles('seaweed-vector/Cargo.lock') }}
restore-keys: |
rust-vector-${{ steps.toolchain.outputs.fingerprint }}-
# lance's build scripts compile their own protos and look for a protoc.
# Point them at the one protoc-bin-vendored ships, so no job depends on a
# system package and every build sees the same version.
- name: Use the vendored protoc
run: |
cd seaweed-vector
cargo fetch
version=$(awk '/^name = "protoc-bin-vendored-linux-x86_64"$/{found=1; next} found && /^version = /{gsub(/"/,"",$3); print $3; exit}' Cargo.lock)
test -n "$version" || { echo "protoc-bin-vendored-linux-x86_64 is not in Cargo.lock" >&2; exit 1; }
protoc=$(find ~/.cargo/registry/src -path "*protoc-bin-vendored-linux-x86_64-$version/bin/protoc" | head -1)
test -x "$protoc" || { echo "no vendored protoc $version in the registry" >&2; exit 1; }
echo "PROTOC=$protoc" >> "$GITHUB_ENV"
- name: Build the vector crates
run: cd seaweed-vector && cargo build --release
- name: Run unit tests
run: cd seaweed-vector && cargo test --release --workspace
vectors-e2e:
name: S3 Vectors End-to-End
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
with:
persist-credentials: false
- uses: actions/setup-go@v5
with:
go-version-file: go.mod
- name: Build weed
run: go build -o test/vectors/weed ./weed
- uses: dtolnay/rust-toolchain@stable
# Same as the unit-test job: lance's build scripts need a protoc; use
# the vendored one so no system package is required.
- name: Use the vendored protoc
run: |
cd seaweed-vector
cargo fetch
version=$(awk '/^name = "protoc-bin-vendored-linux-x86_64"$/{found=1; next} found && /^version = /{gsub(/"/,"",$3); print $3; exit}' Cargo.lock)
test -n "$version" || { echo "protoc-bin-vendored-linux-x86_64 is not in Cargo.lock" >&2; exit 1; }
protoc=$(find ~/.cargo/registry/src -path "*protoc-bin-vendored-linux-x86_64-$version/bin/protoc" | head -1)
test -x "$protoc" || { echo "no vendored protoc $version in the registry" >&2; exit 1; }
echo "PROTOC=$protoc" >> "$GITHUB_ENV"
- name: Build weed-vector-server
run: cargo build --release --manifest-path seaweed-vector/Cargo.toml -p weed-vector-server
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- name: Install test deps
run: pip install boto3 pytest
- name: Run the s3vectors conformance subset
working-directory: test/vectors
run: |
cp ../../seaweed-vector/target/release/weed-vector-server weed-vector-server
./run_test.sh
+85 -57
View File
@@ -21,8 +21,8 @@ on:
- '.github/workflows/rust-volume-server-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.ref }}
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -52,9 +52,68 @@ jobs:
- 'seaweed-volume/**'
- '.github/workflows/rust-volume-server-tests.yml'
build-test-binaries:
name: Build Test Binaries
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
- name: Checkout code
uses: actions/checkout@v7
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
# cargo tracks its own inputs but not the runner's C toolchain, so a cached
# target/ can carry C objects built against a different glibc than we link against.
- name: Fingerprint build toolchain
id: toolchain
run: echo "fingerprint=$(getconf GNU_LIBC_VERSION | tr ' ' '-')-rustc-$(rustc -V | awk '{print $2}')" >> "$GITHUB_OUTPUT"
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
~/.cargo/registry
~/.cargo/git
seaweed-volume/target
key: rust-${{ steps.toolchain.outputs.fingerprint }}-${{ hashFiles('seaweed-volume/Cargo.lock', 'seaweed-common/Cargo.lock') }}
restore-keys: |
rust-${{ steps.toolchain.outputs.fingerprint }}-
- name: Build Go weed binary
run: |
cd weed
go build -tags 5BytesOffset -o weed .
chmod +x weed
./weed version
- name: Build Rust volume binary
run: cd seaweed-volume && cargo build --release
- name: Package binaries
run: tar cf rust-test-binaries.tar weed/weed seaweed-volume/target/release/weed-volume
- name: Upload test binaries
uses: actions/upload-artifact@v7
with:
# archive layout mirrors the workspace: weed/weed and
# seaweed-volume/target/release/weed-volume
name: rust-test-binaries
if-no-files-found: error
retention-days: 1
path: rust-test-binaries.tar
rust-unit-tests:
name: Rust Unit Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 15
steps:
@@ -71,6 +130,7 @@ jobs:
run: echo "fingerprint=$(getconf GNU_LIBC_VERSION | tr ' ' '-')-rustc-$(rustc -V | awk '{print $2}')" >> "$GITHUB_OUTPUT"
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
@@ -134,6 +194,7 @@ jobs:
run: echo "fingerprint=windows-rustc-$(rustc -V | awk '{print $2}')" >> "$GITHUB_OUTPUT"
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
@@ -152,8 +213,9 @@ jobs:
rust-integration-tests:
name: Rust Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
needs: [build-test-binaries]
steps:
- name: Checkout code
@@ -163,36 +225,18 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
# cargo tracks its own inputs but not the runner's C toolchain, so a cached
# target/ can carry C objects built against a different glibc than we link against.
- name: Fingerprint build toolchain
id: toolchain
run: echo "fingerprint=$(getconf GNU_LIBC_VERSION | tr ' ' '-')-rustc-$(rustc -V | awk '{print $2}')" >> "$GITHUB_OUTPUT"
- name: Cache cargo registry and target
uses: actions/cache@v6
- name: Download test binaries
uses: actions/download-artifact@v8
with:
path: |
~/.cargo/registry
~/.cargo/git
seaweed-volume/target
key: rust-${{ steps.toolchain.outputs.fingerprint }}-${{ hashFiles('seaweed-volume/Cargo.lock', 'seaweed-common/Cargo.lock') }}
restore-keys: |
rust-${{ steps.toolchain.outputs.fingerprint }}-
name: rust-test-binaries
- name: Build Go weed binary
- name: Restore binaries
run: |
cd weed
go build -tags 5BytesOffset -o weed .
chmod +x weed
./weed version
- name: Build Rust volume binary
run: cd seaweed-volume && cargo build --release
tar xf rust-test-binaries.tar
chmod +x weed/weed seaweed-volume/target/release/weed-volume
./weed/weed version
- name: Run integration tests
env:
@@ -226,8 +270,9 @@ jobs:
rust-volume-go-tests:
name: Go Tests with Rust Volume (${{ matrix.test-type }} - Shard ${{ matrix.shard }})
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 45
needs: [build-test-binaries]
env:
# Keep in step with the length of matrix.shard below.
SHARD_COUNT: 3
@@ -245,36 +290,19 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
# cargo tracks its own inputs but not the runner's C toolchain, so a cached
# target/ can carry C objects built against a different glibc than we link against.
- name: Fingerprint build toolchain
id: toolchain
run: echo "fingerprint=$(getconf GNU_LIBC_VERSION | tr ' ' '-')-rustc-$(rustc -V | awk '{print $2}')" >> "$GITHUB_OUTPUT"
- name: Cache cargo registry and target
uses: actions/cache@v6
- name: Download test binaries
uses: actions/download-artifact@v8
with:
path: |
~/.cargo/registry
~/.cargo/git
seaweed-volume/target
key: rust-${{ steps.toolchain.outputs.fingerprint }}-${{ hashFiles('seaweed-volume/Cargo.lock', 'seaweed-common/Cargo.lock') }}
restore-keys: |
rust-${{ steps.toolchain.outputs.fingerprint }}-
name: rust-test-binaries
- name: Build Go weed binary
- name: Restore binaries
run: |
cd weed
go build -tags 5BytesOffset -o weed .
chmod +x weed
./weed version
- name: Build Rust volume binary
run: cd seaweed-volume && cargo build --release
tar xf rust-test-binaries.tar
chmod +x weed/weed seaweed-volume/target/release/weed-volume
./weed/weed version
./seaweed-volume/target/release/weed-volume --help >/dev/null || true
# Dealing the listed tests out one by one keeps the shards even. Bucketing
# them by first letter did not: names cluster, so ^Test[I-S] drew 50 of
+4 -3
View File
@@ -17,8 +17,8 @@ on:
- '.github/workflows/rust-worker-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.ref }}
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -26,7 +26,7 @@ permissions:
jobs:
rust-worker-build:
name: Rust Plugin Worker Build and Unit Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 45
steps:
@@ -45,6 +45,7 @@ jobs:
run: echo "fingerprint=$(getconf GNU_LIBC_VERSION | tr ' ' '-')-rustc-$(rustc -V | awk '{print $2}')" >> "$GITHUB_OUTPUT"
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
+3 -1
View File
@@ -32,7 +32,7 @@ jobs:
permissions:
contents: write
needs: cleanup
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
strategy:
matrix:
include:
@@ -46,6 +46,7 @@ jobs:
uses: dtolnay/rust-toolchain@stable
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
@@ -114,6 +115,7 @@ jobs:
targets: ${{ matrix.target }}
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
+20 -8
View File
@@ -15,7 +15,7 @@ jobs:
build-rust-volume-linux:
permissions:
contents: write
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
strategy:
matrix:
include:
@@ -37,9 +37,13 @@ jobs:
if: matrix.cross
run: |
sudo dpkg --add-architecture arm64
sudo sed -i 's/^deb /deb [arch=amd64] /' /etc/apt/sources.list
echo "deb [arch=arm64] http://ports.ubuntu.com/ jammy main restricted universe multiverse" | sudo tee /etc/apt/sources.list.d/arm64.list
echo "deb [arch=arm64] http://ports.ubuntu.com/ jammy-updates main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list.d/arm64.list
# noble keeps apt sources in deb822 ubuntu.sources, not sources.list —
# pin them to amd64 so apt does not probe arm64 indexes on
# archive/security.ubuntu.com (they 404).
sudo sed -i '/^Types: deb$/a Architectures: amd64' /etc/apt/sources.list.d/ubuntu.sources
for suite in noble noble-updates noble-security; do
echo "deb [arch=arm64] http://ports.ubuntu.com/ubuntu-ports $suite main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list.d/arm64.list
done
sudo apt-get update
sudo apt-get install -y gcc-aarch64-linux-gnu libssl-dev:arm64
echo "CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER=aarch64-linux-gnu-gcc" >> "$GITHUB_ENV"
@@ -48,6 +52,7 @@ jobs:
echo "OPENSSL_LIB_DIR=/usr/lib/aarch64-linux-gnu" >> "$GITHUB_ENV"
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
@@ -119,7 +124,7 @@ jobs:
build-rust-worker-linux:
permissions:
contents: write
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
strategy:
matrix:
include:
@@ -145,14 +150,19 @@ jobs:
if: matrix.cross
run: |
sudo dpkg --add-architecture arm64
sudo sed -i 's/^deb /deb [arch=amd64] /' /etc/apt/sources.list
echo "deb [arch=arm64] http://ports.ubuntu.com/ jammy main restricted universe multiverse" | sudo tee /etc/apt/sources.list.d/arm64.list
echo "deb [arch=arm64] http://ports.ubuntu.com/ jammy-updates main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list.d/arm64.list
# noble keeps apt sources in deb822 ubuntu.sources, not sources.list —
# pin them to amd64 so apt does not probe arm64 indexes on
# archive/security.ubuntu.com (they 404).
sudo sed -i '/^Types: deb$/a Architectures: amd64' /etc/apt/sources.list.d/ubuntu.sources
for suite in noble noble-updates noble-security; do
echo "deb [arch=arm64] http://ports.ubuntu.com/ubuntu-ports $suite main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list.d/arm64.list
done
sudo apt-get update
sudo apt-get install -y gcc-aarch64-linux-gnu
echo "CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER=aarch64-linux-gnu-gcc" >> "$GITHUB_ENV"
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
@@ -230,6 +240,7 @@ jobs:
targets: ${{ matrix.target }}
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
@@ -305,6 +316,7 @@ jobs:
uses: dtolnay/rust-toolchain@stable
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
+4 -3
View File
@@ -21,8 +21,8 @@ on:
- '.github/workflows/s3-etag-acl-tests.yml'
concurrency:
group: ${{ github.head_ref || github.ref }}/s3-etag-acl-tests
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -30,7 +30,7 @@ permissions:
jobs:
s3-etag-acl-tests:
name: S3 ETag + ACL Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 15
steps:
- name: Check out code
@@ -40,6 +40,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Install SeaweedFS
run: |
@@ -13,8 +13,8 @@ on:
- '.github/workflows/s3-example-integration-tests.yml'
concurrency:
group: ${{ github.head_ref }}/s3-integration-tests
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -22,7 +22,7 @@ permissions:
jobs:
tests:
name: S3 Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
@@ -33,6 +33,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Build SeaweedFS
+8 -4
View File
@@ -12,8 +12,8 @@ on:
- '.github/workflows/s3-filer-group-tests.yml'
concurrency:
group: ${{ github.head_ref }}/s3-filer-group-tests
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -25,17 +25,21 @@ defaults:
jobs:
s3-filer-group-tests:
name: S3 Filer Group Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -50,7 +54,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
echo "=== Starting Filer Group Tests ==="
# Run tests with automatic server management
+42 -17
View File
@@ -4,6 +4,7 @@ on:
pull_request:
paths:
- 'weed/s3api/**'
- 'weed/remote_storage/**'
- 'weed/filer/**'
- 'weed/server/**'
- 'test/s3/**'
@@ -12,8 +13,8 @@ on:
- '.github/workflows/s3-go-tests.yml'
concurrency:
group: ${{ github.head_ref }}/s3-go-tests
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -25,7 +26,7 @@ defaults:
jobs:
s3-versioning-tests:
name: S3 Versioning Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
strategy:
matrix:
@@ -35,10 +36,14 @@ jobs:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -53,7 +58,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
echo "=== Starting Tests ==="
# Run tests with automatic server management
@@ -92,7 +97,7 @@ jobs:
s3-versioning-compatibility:
name: S3 Versioning Compatibility Test
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
@@ -103,6 +108,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -140,7 +146,7 @@ jobs:
s3-cors-compatibility:
name: S3 CORS Compatibility Test
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
@@ -151,6 +157,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -188,7 +195,7 @@ jobs:
s3-retention-tests:
name: S3 Retention Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
strategy:
matrix:
@@ -198,10 +205,14 @@ jobs:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -216,7 +227,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
echo "=== Starting Tests ==="
# Run tests with automatic server management
@@ -255,7 +266,7 @@ jobs:
s3-lifecycle-tests:
name: S3 Lifecycle Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 10
strategy:
fail-fast: false
@@ -283,6 +294,7 @@ jobs:
- TestLifecycleSuspendedVersioningExpiration
- TestLifecycleTagFilter
- TestLifecycleVersionedBucketCreatesDeleteMarker
- TestLifecycleVersionedBucketExpiresOnSecondRun
steps:
- name: Check out code
@@ -292,6 +304,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -326,17 +339,21 @@ jobs:
s3-checksum-tests:
name: S3 Checksum Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -351,7 +368,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
echo "=== Starting Tests ==="
make test-with-server
@@ -381,7 +398,7 @@ jobs:
s3-cors-tests:
name: S3 CORS Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
strategy:
matrix:
@@ -391,10 +408,14 @@ jobs:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -409,7 +430,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
echo "=== Starting Tests ==="
# Run tests with automatic server management
@@ -448,7 +469,7 @@ jobs:
s3-retention-worm:
name: S3 Retention WORM Integration Test
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
@@ -459,6 +480,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -497,7 +519,7 @@ jobs:
s3-versioning-stress:
name: S3 Versioning Stress Test
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 35
steps:
@@ -508,6 +530,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -565,7 +588,7 @@ jobs:
s3-tagging-tests:
# CI job for S3 object tagging tests
name: S3 Tagging Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
@@ -576,6 +599,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -620,7 +644,7 @@ jobs:
s3-remote-cache-tests:
name: S3 Remote Cache Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
@@ -631,6 +655,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
+14 -7
View File
@@ -20,8 +20,8 @@ on:
- '.github/workflows/s3-iam-tests.yml'
concurrency:
group: ${{ github.head_ref }}/s3-iam-tests
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -34,7 +34,7 @@ jobs:
# Unit tests for IAM components
iam-unit-tests:
name: IAM Unit Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 15
steps:
@@ -45,6 +45,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Get dependencies
@@ -80,7 +81,7 @@ jobs:
# S3 IAM integration tests with SeaweedFS services
s3-iam-integration-tests:
name: S3 IAM Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 25
strategy:
matrix:
@@ -90,10 +91,14 @@ jobs:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -109,7 +114,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
echo "=== Starting S3 IAM Integration Tests (${{ matrix.test-type }}) ==="
# Set WEED_BINARY to use the installed version
@@ -197,7 +202,7 @@ jobs:
# Distributed IAM tests
s3-iam-distributed-tests:
name: S3 IAM Distributed Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 25
steps:
@@ -208,6 +213,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -257,7 +263,7 @@ jobs:
# Performance and stress tests
s3-iam-performance-tests:
name: S3 IAM Performance Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
@@ -268,6 +274,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
+8 -4
View File
@@ -16,8 +16,8 @@ on:
- '.github/workflows/s3-keycloak-tests.yml'
concurrency:
group: ${{ github.head_ref }}/s3-keycloak-tests
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -30,17 +30,21 @@ jobs:
# Dedicated job for Keycloak integration tests
s3-keycloak-integration-tests:
name: S3 Keycloak Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -56,7 +60,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
echo "=== Starting S3 Keycloak Integration Tests ==="
# Set WEED_BINARY to use the installed version
@@ -12,8 +12,8 @@ on:
- '.github/workflows/s3-mutation-regression-tests.yml'
concurrency:
group: ${{ github.head_ref }}/s3-mutation-regression-tests
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -21,7 +21,7 @@ permissions:
jobs:
s3-versioning-regressions:
name: S3 Versioning Regression Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 25
steps:
@@ -32,6 +32,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Run S3 versioning regression tests
timeout-minutes: 20
@@ -74,7 +75,7 @@ jobs:
s3-delete-regressions:
name: S3 Delete Regression Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
@@ -85,6 +86,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Run S3 delete regression tests
timeout-minutes: 15
@@ -112,7 +114,7 @@ jobs:
s3-distributed-lock-regressions:
name: S3 Distributed Lock Regression Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
@@ -123,6 +125,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Build SeaweedFS
run: |
+7 -3
View File
@@ -23,6 +23,10 @@ env:
S3_ENDPOINT_URL: http://localhost:8333
BUCKET_NAME: test-parquet-bucket
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
jobs:
parquet-integration-tests:
name: PyArrow Parquet Tests (Python ${{ matrix.python-version }})
@@ -42,13 +46,13 @@ jobs:
uses: actions/setup-go@v7
with:
go-version: ^1.26
cache: true
cache: ${{ runner.environment == 'github-hosted' }}
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v7
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache: ${{ runner.environment == 'github-hosted' && 'pip' || '' }}
cache-dependency-path: 'test/s3/parquet/requirements.txt'
- name: Install system dependencies
@@ -150,7 +154,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version: ^1.26
cache: true
cache: ${{ runner.environment == 'github-hosted' }}
- name: Run Go unit tests
run: |
+20 -9
View File
@@ -24,8 +24,8 @@ on:
- '.github/workflows/s3-policy-tests.yml'
concurrency:
group: ${{ github.head_ref }}/s3-policy-tests
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -38,7 +38,7 @@ jobs:
# Unit tests for policy components
policy-unit-tests:
name: S3 Policy Unit Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 15
steps:
@@ -49,6 +49,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Get dependencies
@@ -81,17 +82,21 @@ jobs:
# S3 Policy Variables Integration Tests
s3-policy-variables-tests:
name: S3 Policy Variables Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 25
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -106,7 +111,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
echo "=== Starting S3 Policy Variables Integration Tests ==="
@@ -187,7 +192,7 @@ jobs:
# S3 Policy Enforcement Integration Tests
s3-policy-enforcement-tests:
name: S3 Policy Enforcement Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
strategy:
matrix:
@@ -197,10 +202,14 @@ jobs:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -215,7 +224,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
echo "=== Starting S3 Policy Enforcement Tests (${{ matrix.test-case }}) ==="
@@ -308,7 +317,7 @@ jobs:
# Trusted Proxy Detection Tests
trusted-proxy-tests:
name: Trusted Proxy Detection Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
@@ -319,6 +328,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -395,7 +405,7 @@ jobs:
# S3 Policy Shell Integration Tests
s3-policy-shell-tests:
name: S3 Policy Shell Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 15
steps:
@@ -406,6 +416,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -21,8 +21,8 @@ on:
- '.github/workflows/s3-proxy-signature-tests.yml'
concurrency:
group: ${{ github.head_ref || github.ref }}/s3-proxy-signature-tests
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -30,7 +30,7 @@ permissions:
jobs:
proxy-signature-tests:
name: S3 Proxy Signature Verification Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 15
steps:
- name: Check out code into the Go module directory
@@ -40,6 +40,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -15,8 +15,8 @@ on:
- '.github/workflows/s3-sdk-v2-routing-tests.yml'
concurrency:
group: ${{ github.head_ref || github.ref }}/s3-sdk-v2-routing-tests
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -24,7 +24,7 @@ permissions:
jobs:
s3-sdk-v2-routing-tests:
name: S3 SDK V2 Routing Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 10
steps:
- name: Check out code
@@ -34,6 +34,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Install SeaweedFS
run: |
+3 -2
View File
@@ -46,7 +46,7 @@ permissions:
jobs:
snowflake-s3compat-tests:
name: Snowflake S3Compat API tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
env:
WORK_DIR: /tmp/seaweedfs-snowflake-tests
@@ -60,6 +60,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Set up Java
@@ -67,7 +68,7 @@ jobs:
with:
java-version: '17'
distribution: 'temurin'
cache: 'maven'
cache: ${{ runner.environment == 'github-hosted' && 'maven' || '' }}
- name: Install SeaweedFS
run: |
+8 -4
View File
@@ -11,8 +11,8 @@ on:
workflow_dispatch:
concurrency:
group: ${{ github.head_ref }}/s3-spark-tests
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -20,17 +20,21 @@ permissions:
jobs:
s3-spark-issue-repro-tests:
name: S3 Spark Issue Reproduction Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 45
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -56,7 +60,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
echo "=== Starting S3 Spark Integration Tests ==="
go test -v -timeout 30m . 2>&1 | tee test-output.log || {
+20 -10
View File
@@ -23,8 +23,8 @@ on:
- 'test/s3/sse/**'
concurrency:
group: ${{ github.head_ref }}/s3-sse-tests
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -36,7 +36,7 @@ defaults:
jobs:
s3-sse-integration-tests:
name: S3 SSE Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
strategy:
matrix:
@@ -46,10 +46,14 @@ jobs:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -64,7 +68,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
echo "=== Starting SSE Tests ==="
# Run tests with automatic server management
@@ -107,7 +111,7 @@ jobs:
s3-sse-compatibility:
name: S3 SSE Compatibility Test
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
@@ -118,6 +122,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -155,7 +160,7 @@ jobs:
s3-sse-metadata-persistence:
name: S3 SSE Metadata Persistence Test
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
@@ -166,6 +171,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -204,7 +210,7 @@ jobs:
s3-sse-copy-operations:
name: S3 SSE Copy Operations Test
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 25
steps:
@@ -215,6 +221,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -253,7 +260,7 @@ jobs:
s3-sse-multipart:
name: S3 SSE Multipart Upload Test
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 25
steps:
@@ -264,6 +271,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -302,7 +310,7 @@ jobs:
s3-sse-performance:
name: S3 SSE Performance Test
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 35
# Only run performance tests on master branch pushes to avoid overloading PR testing
if: github.event_name == 'push' && (github.ref == 'refs/heads/master' || github.ref == 'refs/heads/main')
@@ -315,6 +323,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -352,7 +361,7 @@ jobs:
s3-volume-encryption:
name: S3 Volume Encryption Test
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
@@ -363,6 +372,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
+110 -36
View File
@@ -12,6 +12,10 @@ on:
- 'go.sum'
- '.github/workflows/s3-tables-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -19,17 +23,21 @@ permissions:
jobs:
s3-tables-tests:
name: S3 Tables Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Run go mod tidy
@@ -48,7 +56,8 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
# A stale FUSE mount can hang or fail df; keep diagnostics best effort.
timeout 15 df -h || true
echo "=== Starting S3 Tables Tests ==="
# Run S3 Tables integration tests
@@ -79,17 +88,21 @@ jobs:
iceberg-catalog-tests:
name: Iceberg Catalog Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -109,7 +122,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
echo "=== Starting Iceberg Catalog Tests ==="
# Run Iceberg catalog integration tests using Makefile (handles build)
@@ -140,17 +153,21 @@ jobs:
trino-iceberg-catalog-tests:
name: Trino Iceberg Catalog Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -179,7 +196,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
echo "=== Starting Trino Iceberg Catalog Tests ==="
# Run Trino + Iceberg catalog integration tests
@@ -210,17 +227,21 @@ jobs:
dremio-iceberg-catalog-tests:
name: Dremio Iceberg Catalog Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -250,7 +271,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
docker info
echo "=== Starting Dremio Iceberg Catalog Tests ==="
@@ -283,17 +304,21 @@ jobs:
doris-iceberg-catalog-tests:
name: Doris Iceberg Catalog Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 35
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -323,7 +348,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
docker info
echo "=== Starting Doris Iceberg Catalog Tests ==="
@@ -356,7 +381,7 @@ jobs:
clickhouse-iceberg-catalog-tests:
name: ClickHouse Iceberg Catalog Integration Tests (${{ matrix.tag }})
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
strategy:
fail-fast: false
@@ -373,10 +398,14 @@ jobs:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -408,7 +437,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
docker info
echo "=== Starting ClickHouse Iceberg Catalog Tests ==="
@@ -441,7 +470,7 @@ jobs:
olake-iceberg-catalog-tests:
name: OLake Iceberg Catalog Integration Tests (${{ matrix.tag }})
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
strategy:
fail-fast: false
@@ -460,10 +489,14 @@ jobs:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -496,7 +529,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
docker info
echo "=== Starting OLake Iceberg Catalog Tests ==="
@@ -552,17 +585,21 @@ jobs:
polaris-integration-tests:
name: Polaris Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Run go mod tidy
@@ -590,7 +627,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
echo "=== Starting Polaris Tests ==="
go test -v -timeout 20m ./test/s3tables/polaris 2>&1 | tee test/s3tables/polaris/test-output.log || {
@@ -620,17 +657,21 @@ jobs:
spark-iceberg-catalog-tests:
name: Spark Iceberg Catalog Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -659,7 +700,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
echo "=== Starting Spark Iceberg Catalog Tests ==="
# Run Spark + Iceberg catalog integration tests
@@ -690,17 +731,21 @@ jobs:
risingwave-catalog-tests:
name: RisingWave Catalog Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -730,7 +775,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
echo "=== Starting RisingWave Catalog Tests ==="
# Run RisingWave catalog integration tests
@@ -761,17 +806,21 @@ jobs:
sts-integration-tests:
name: STS Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -785,6 +834,7 @@ jobs:
- name: Restore python:3 image cache
id: python-image
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: /tmp/python3-image.tar
@@ -824,7 +874,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
echo "=== Starting STS Integration Tests ==="
# Run STS integration tests
@@ -855,17 +905,21 @@ jobs:
lakekeeper-integration-tests:
name: Lakekeeper Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Run go mod tidy
@@ -884,7 +938,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
echo "=== Starting Lakekeeper Integration Tests ==="
# Run Lakekeeper integration tests
@@ -915,17 +969,21 @@ jobs:
unity-catalog-integration-tests:
name: Unity Catalog Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -955,7 +1013,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
docker info
echo "=== Starting Unity Catalog Tests ==="
@@ -988,17 +1046,21 @@ jobs:
lancedb-namespace-tests:
name: LanceDB Namespace Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -1027,7 +1089,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
docker info
echo "=== Starting LanceDB Namespace Tests ==="
@@ -1058,7 +1120,7 @@ jobs:
table-lifecycle-tests:
name: Table Lifecycle Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 40
steps:
@@ -1069,6 +1131,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -1126,7 +1189,7 @@ jobs:
duckdb-lance-tests:
name: DuckDB Lance Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
@@ -1137,10 +1200,14 @@ jobs:
# so do not leave a token in the checkout for it to pick up.
persist-credentials: false
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -1170,7 +1237,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
docker info
echo "=== Starting DuckDB Lance Tests ==="
@@ -1201,7 +1268,7 @@ jobs:
spark-lance-namespace-tests:
name: Spark Lance Namespace Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 40
steps:
@@ -1212,10 +1279,14 @@ jobs:
# so do not leave a token in the checkout for it to pick up.
persist-credentials: false
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -1244,7 +1315,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
docker info
echo "=== Starting Spark Lance Namespace Tests ==="
@@ -1275,7 +1346,7 @@ jobs:
s3-tables-build-verification:
name: S3 Tables Build Verification
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 15
steps:
@@ -1286,6 +1357,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Verify S3 Tables Package Builds
@@ -1340,7 +1412,7 @@ jobs:
s3-tables-fmt-check:
name: S3 Tables Format Check
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 10
steps:
@@ -1351,6 +1423,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Check Go Format
@@ -1391,7 +1464,7 @@ jobs:
s3-tables-vet:
name: S3 Tables Go Vet Check
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 10
steps:
@@ -1402,6 +1475,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Run Go Vet
+44 -25
View File
@@ -25,8 +25,8 @@ on:
- '.github/workflows/s3tests.yml'
concurrency:
group: ${{ github.head_ref }}/s3tests
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -34,7 +34,7 @@ permissions:
jobs:
basic-s3-tests:
name: Basic S3 tests (KV store)
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 15
steps:
- name: Check out code into the Go module directory
@@ -44,6 +44,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Set up Python
@@ -57,9 +58,9 @@ jobs:
cd s3-tests
sudo apt-get update -qq
sudo apt-get install -y -qq libxml2-dev libxslt1-dev zlib1g-dev
pip install -r requirements.txt
pip install tox
pip install -e .
python3 -m pip install -r requirements.txt
python3 -m pip install tox
python3 -m pip install -e .
- name: Fix S3 tests bucket creation conflicts
run: |
@@ -379,14 +380,21 @@ jobs:
s3tests/functional/test_s3.py::test_copy_object_ifnonematch_good \
s3tests/functional/test_s3.py::test_lifecycle_set \
s3tests/functional/test_s3.py::test_lifecycle_get \
s3tests/functional/test_s3.py::test_lifecycle_set_filter \
s3tests/functional/test_s3.py::test_lifecycle_expiration \
s3tests/functional/test_s3.py::test_lifecyclev2_expiration
s3tests/functional/test_s3.py::test_lifecycle_set_filter
# The expiration tests race the lifecycle worker's meta-log replay;
# under the suite's write flood a shard can trail an object's PUT
# event past the test's fixed sleep window. One retry absorbs it.
lifecycle_expiration() {
tox -- s3tests/functional/test_s3.py::test_lifecycle_expiration \
s3tests/functional/test_s3.py::test_lifecyclev2_expiration
}
lifecycle_expiration || { echo "lifecycle expiration tests flaked; retry 1"; lifecycle_expiration; } || { echo "lifecycle expiration tests flaked; retry 2"; lifecycle_expiration; }
# cleanup() trap handles worker/server kill + data dir wipe.
versioning-tests:
name: S3 Versioning & Object Lock tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 15
steps:
- name: Check out code into the Go module directory
@@ -396,6 +404,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Set up Python
@@ -409,9 +418,9 @@ jobs:
cd s3-tests
sudo apt-get update -qq
sudo apt-get install -y -qq libxml2-dev libxslt1-dev zlib1g-dev
pip install -r requirements.txt
pip install tox
pip install -e .
python3 -m pip install -r requirements.txt
python3 -m pip install tox
python3 -m pip install -e .
- name: Fix S3 tests bucket creation conflicts
run: |
@@ -556,7 +565,7 @@ jobs:
cors-tests:
name: S3 CORS tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 10
steps:
- name: Check out code into the Go module directory
@@ -566,6 +575,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Set up Python
@@ -579,9 +589,9 @@ jobs:
cd s3-tests
sudo apt-get update -qq
sudo apt-get install -y -qq libxml2-dev libxslt1-dev zlib1g-dev
pip install -r requirements.txt
pip install tox
pip install -e .
python3 -m pip install -r requirements.txt
python3 -m pip install tox
python3 -m pip install -e .
- name: Run S3 CORS tests
timeout-minutes: 10
@@ -681,7 +691,7 @@ jobs:
copy-tests:
name: SeaweedFS Custom S3 Copy tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 10
steps:
- name: Check out code into the Go module directory
@@ -691,6 +701,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Run SeaweedFS Custom S3 Copy tests
@@ -783,7 +794,7 @@ jobs:
sql-store-tests:
name: Basic S3 tests (SQL store)
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 15
steps:
- name: Check out code into the Go module directory
@@ -793,6 +804,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Set up Python
@@ -806,9 +818,9 @@ jobs:
cd s3-tests
sudo apt-get update -qq
sudo apt-get install -y -qq libxml2-dev libxslt1-dev zlib1g-dev
pip install -r requirements.txt
pip install tox
pip install -e .
python3 -m pip install -r requirements.txt
python3 -m pip install tox
python3 -m pip install -e .
- name: Run Ceph S3 tests with SQL store
timeout-minutes: 15
@@ -1245,9 +1257,16 @@ jobs:
s3tests/functional/test_s3.py::test_copy_object_ifnonematch_good \
s3tests/functional/test_s3.py::test_lifecycle_set \
s3tests/functional/test_s3.py::test_lifecycle_get \
s3tests/functional/test_s3.py::test_lifecycle_set_filter \
s3tests/functional/test_s3.py::test_lifecycle_expiration \
s3tests/functional/test_s3.py::test_lifecyclev2_expiration
s3tests/functional/test_s3.py::test_lifecycle_set_filter
# The expiration tests race the lifecycle worker's meta-log replay;
# under the suite's write flood a shard can trail an object's PUT
# event past the test's fixed sleep window. One retry absorbs it.
lifecycle_expiration() {
tox -- s3tests/functional/test_s3.py::test_lifecycle_expiration \
s3tests/functional/test_s3.py::test_lifecyclev2_expiration
}
lifecycle_expiration || { echo "lifecycle expiration tests flaked; retry 1"; lifecycle_expiration; } || { echo "lifecycle expiration tests flaked; retry 2"; lifecycle_expiration; }
# cleanup() trap handles worker/server kill + data dir wipe.
+10 -2
View File
@@ -29,7 +29,7 @@ permissions:
jobs:
samba-integration:
name: samba-integration
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 45
steps:
@@ -40,15 +40,23 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Start local Docker registry
run: docker run -d --restart=always -p 5000:5000 --name registry registry:2
run: |
if ! docker start registry 2>/dev/null; then
docker rm -f registry >/dev/null 2>&1 || true
docker run -d --restart=always -p 5000:5000 --name registry registry:2
fi
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4
with:
driver-opts: network=host
- name: Set up Docker Compose
uses: docker/setup-compose-action@v2
- name: Build weed race binary
run: |
cd docker
+4 -3
View File
@@ -17,8 +17,8 @@ on:
- '.github/workflows/sftp-tests.yml'
concurrency:
group: ${{ github.head_ref }}/sftp-tests
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -29,7 +29,7 @@ env:
jobs:
sftp-integration:
name: SFTP Integration Testing
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
@@ -40,6 +40,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Install dependencies
run: |
@@ -15,6 +15,10 @@ on:
- '.github/workflows/spark-integration-tests.yml'
workflow_dispatch:
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
checks: write
@@ -38,12 +42,13 @@ jobs:
with:
java-version: '11'
distribution: 'temurin'
cache: maven
cache: ${{ runner.environment == 'github-hosted' && 'maven' || '' }}
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Build SeaweedFS binary
run: |
+5 -6
View File
@@ -10,12 +10,11 @@ permissions:
contents: write
concurrency:
# Only one chart regeneration per branch at a time; a newer run on the same
# branch cancels an in-flight one so overlapping runs never conflict on
# note/star_history.svg during rebase. Scoped by ref so a manual run on
# another branch can't cancel the daily master update.
# Only one chart regeneration per branch at a time so overlapping runs never
# conflict on note/star_history.svg during rebase. Scoped by ref so a manual
# run on another branch can't cancel the daily master update.
group: star-history-${{ github.ref }}
cancel-in-progress: true
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
jobs:
render:
@@ -33,7 +32,7 @@ jobs:
uses: actions/setup-python@v7
with:
python-version: "3.x"
cache: pip
cache: ${{ runner.environment == 'github-hosted' && 'pip' || '' }}
cache-dependency-path: .github/scripts/star_history.py
- name: Install matplotlib
@@ -14,6 +14,10 @@ on:
- 'weed/telemetry/**'
- '.github/workflows/telemetry-integration.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -30,6 +34,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Build telemetry server
run: cd telemetry/server && go build -o telemetry-server .
+5
View File
@@ -8,6 +8,10 @@ on:
branches: [ master ]
paths: ['terraform/**', '.github/workflows/terraform_ci.yml']
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -62,6 +66,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: go.mod
cache: ${{ runner.environment == 'github-hosted' }}
- name: Build weed
run: go build -o "$RUNNER_TEMP/weed" ./weed
@@ -27,6 +27,10 @@ defaults:
run:
working-directory: weed
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
jobs:
awscli-tests:
runs-on: ubuntu-latest
@@ -37,6 +41,7 @@ jobs:
- uses: actions/setup-go@v7
with:
go-version: ^1.26
cache: ${{ runner.environment == 'github-hosted' }}
- name: Build SeaweedFS
run: |
+6 -1
View File
@@ -18,18 +18,23 @@ on:
- 'go.sum'
- '.github/workflows/tls-rotation-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
jobs:
tls-rotation-tests:
name: TLS Rotation Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 10
steps:
- name: Set up Go 1.x
uses: actions/setup-go@v7
with:
cache: ${{ runner.environment == 'github-hosted' }}
go-version: ^1.26
id: go
+8 -4
View File
@@ -11,8 +11,8 @@ on:
- '.github/workflows/tus-tests.yml'
concurrency:
group: ${{ github.head_ref || github.ref }}/tus-tests
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -24,17 +24,21 @@ defaults:
jobs:
tus-integration-tests:
name: TUS Protocol Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -49,7 +53,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
df -h
timeout 15 df -h || true
echo "=== Starting TUS Tests ==="
# Run tests with automatic server management
@@ -18,18 +18,23 @@ on:
- 'go.sum'
- '.github/workflows/vacuum-integration-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
jobs:
vacuum-integration-tests:
name: Vacuum Integration Tests
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 15
steps:
- name: Set up Go 1.x
uses: actions/setup-go@v7
with:
cache: ${{ runner.environment == 'github-hosted' }}
go-version: ^1.26
id: go
@@ -21,8 +21,8 @@ on:
- '.github/workflows/volume-server-integration-tests.yml'
concurrency:
group: ${{ github.head_ref || github.ref }}/volume-server-integration-tests
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -33,10 +33,44 @@ env:
SHARD_COUNT: 3
jobs:
build-weed:
name: Build SeaweedFS Binary
runs-on: ubuntu-24.04
timeout-minutes: 15
steps:
- name: Checkout code
uses: actions/checkout@v7
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Build SeaweedFS binary
run: |
cd weed
go build -o weed .
chmod +x weed
./weed version
- name: Package binary
run: tar cf weed-binary.tar weed/weed
- name: Upload binary
uses: actions/upload-artifact@v7
with:
name: weed-binary
if-no-files-found: error
retention-days: 1
path: weed-binary.tar
volume-server-integration-tests:
name: Volume Server Integration Tests (${{ matrix.test-type }} - Shard ${{ matrix.shard }})
runs-on: ubuntu-22.04
runs-on: ubuntu-24.04
timeout-minutes: 45
needs: [build-weed]
strategy:
fail-fast: false
matrix:
@@ -51,13 +85,18 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Build SeaweedFS binary
- name: Download binary
uses: actions/download-artifact@v8
with:
name: weed-binary
- name: Restore binary
run: |
cd weed
go build -o weed .
chmod +x weed
./weed version
tar xf weed-binary.tar
chmod +x weed/weed
./weed/weed version
# Dealing the listed tests out one by one keeps the shards even. Bucketing
# them by first letter did not: names cluster, so ^Test[I-S] drew 50 of
+1
View File
@@ -144,3 +144,4 @@ weed_bin
telemetry/server/telemetry-server
.aider*
/seaweed-volume/docs
__pycache__/
+1 -1
View File
@@ -3,7 +3,7 @@ FROM golang:1.26 AS builder
RUN apt-get update
RUN apt-get install -y build-essential libsnappy-dev zlib1g-dev libbz2-dev libgflags-dev liblz4-dev libzstd-dev
ARG ROCKSDB_VERSION=v10.10.1
ARG ROCKSDB_VERSION=v11.1.2
ENV ROCKSDB_VERSION=${ROCKSDB_VERSION}
# build RocksDB
+1 -1
View File
@@ -3,7 +3,7 @@ FROM golang:1.26 AS builder
RUN apt-get update
RUN apt-get install -y build-essential libsnappy-dev zlib1g-dev libbz2-dev libgflags-dev liblz4-dev libzstd-dev
ARG ROCKSDB_VERSION=v10.10.1
ARG ROCKSDB_VERSION=v11.1.2
ENV ROCKSDB_VERSION=${ROCKSDB_VERSION}
# build RocksDB
+9 -9
View File
@@ -115,8 +115,8 @@ require (
github.com/Azure/azure-sdk-for-go/sdk/keyvault/azkeys v0.10.0
github.com/DATA-DOG/go-sqlmock v1.5.2
github.com/Jille/raft-grpc-transport v1.6.1
github.com/ThreeDotsLabs/watermill v1.5.2
github.com/a-h/templ v0.3.1070
github.com/ThreeDotsLabs/watermill v1.5.3
github.com/a-h/templ v0.3.1100
github.com/apache/cassandra-gocql-driver/v2 v2.1.2
github.com/apache/iceberg-go v0.7.0
github.com/apple/foundationdb/bindings/go v0.0.0-20250911184653-27f7192f47c3
@@ -129,7 +129,7 @@ require (
github.com/cespare/xxhash/v2 v2.3.0
github.com/cognusion/imaging v1.0.4
github.com/fluent/fluent-logger-golang v1.10.1
github.com/getsentry/sentry-go v0.49.0
github.com/getsentry/sentry-go v0.50.0
github.com/go-ldap/ldap/v3 v3.4.14
github.com/golang-jwt/jwt/v5 v5.3.1
github.com/google/flatbuffers/go v0.0.0-20230108230133-3b8644d32c50
@@ -142,11 +142,11 @@ require (
github.com/minio/crc64nvme v1.1.1
github.com/orcaman/concurrent-map/v2 v2.0.1
github.com/parquet-go/parquet-go v0.32.0
github.com/pkg/sftp v1.13.11
github.com/pkg/sftp v1.13.12
github.com/rabbitmq/amqp091-go v1.15.0
github.com/rclone/rclone v1.75.1
github.com/rdleal/intervalst v1.5.0
github.com/redis/go-redis/v9 v9.22.0
github.com/redis/go-redis/v9 v9.23.0
github.com/schollz/progressbar/v3 v3.19.1
github.com/seaweedfs/go-fuse/v2 v2.9.4
github.com/shirou/gopsutil/v4 v4.26.7
@@ -294,9 +294,9 @@ require (
cloud.google.com/go/iam v1.13.0 // indirect
cloud.google.com/go/monitoring v1.30.0 // indirect
filippo.io/edwards25519 v1.2.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.23.2
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.23.3
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.14.1
github.com/Azure/azure-sdk-for-go/sdk/internal v1.12.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/internal v1.13.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.8.2
github.com/Azure/azure-sdk-for-go/sdk/storage/azfile v1.7.0 // indirect
github.com/Azure/go-ntlmssp v0.1.1 // indirect
@@ -314,7 +314,7 @@ require (
github.com/ProtonMail/go-srp v0.0.7 // indirect
github.com/PuerkitoBio/goquery v1.12.0 // indirect
github.com/abbot/go-http-auth v0.4.0 // indirect
github.com/andybalholm/brotli v1.2.6 // indirect
github.com/andybalholm/brotli v1.2.7 // indirect
github.com/andybalholm/cascadia v1.3.4 // indirect
github.com/appscode/go-querystring v0.0.0-20170504095604-0126cfb3f1dc // indirect
github.com/arangodb/go-velocypack v0.0.0-20200318135517-5af53c29c67e // indirect
@@ -362,7 +362,7 @@ require (
github.com/flynn/noise v1.1.0 // indirect
github.com/gabriel-vasile/mimetype v1.4.13 // indirect
github.com/geoffgarside/ber v1.2.0 // indirect
github.com/go-chi/chi/v5 v5.3.1 // indirect
github.com/go-chi/chi/v5 v5.4.0 // indirect
github.com/go-darwin/apfs v0.0.0-20211011131704-f84b94dbf348 // indirect
github.com/go-jose/go-jose/v4 v4.1.5 // indirect
github.com/go-logr/logr v1.4.4 // indirect
+18 -18
View File
@@ -34,14 +34,14 @@ filippo.io/edwards25519 v1.2.0 h1:crnVqOiS4jqYleHd9vaKZ+HKtHfllngJIiOpNpoJsjo=
filippo.io/edwards25519 v1.2.0/go.mod h1:xzAOLCNug/yB62zG1bQ8uziwrIqIuxhctzJT18Q77mc=
github.com/AdaLogics/go-fuzz-headers v0.0.0-20240806141605-e8a1dd7889d6 h1:He8afgbRMd7mFxO99hRNu+6tazq8nFF9lIwo9JFroBk=
github.com/AdaLogics/go-fuzz-headers v0.0.0-20240806141605-e8a1dd7889d6/go.mod h1:8o94RPi1/7XTJvwPpRSzSUedZrtlirdB3r9Z20bi2f8=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.23.2 h1:utpeoEeZjd+A8J41zvoLsOOrqXHhX1Kx/X/tCW9dEYQ=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.23.2/go.mod h1:iptorS+VYKFL2N6PnebpS91dubG35eAOEERnT4PJbQU=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.23.3 h1:qgnWTiWs1G7AqcKZegFvkJD1bmXJcaDU3uC0ELSgZ5U=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.23.3/go.mod h1:XtxD48QPInqoj+GMlcRlXHEYsoOxFnP6w2JgIZ9rDS4=
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.14.1 h1:u93s+zU2JD62im61Bm5CZIc1ZrOJaIAWEg0WOrMVkEo=
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.14.1/go.mod h1:oXtinPO4OLj9d1DOTrqrL1oRwGhcqadvAmrl6wTeGlk=
github.com/Azure/azure-sdk-for-go/sdk/azidentity/cache v0.4.0 h1:xFaZZ+IubdftrDHnGGwZ6QvQ3KHTtWl2MCK+GMt2vxs=
github.com/Azure/azure-sdk-for-go/sdk/azidentity/cache v0.4.0/go.mod h1:mCBhUhlMjLLJKr5aqw2TNS/VqJOie8MzWq3DAMJeKso=
github.com/Azure/azure-sdk-for-go/sdk/internal v1.12.0 h1:fhqpLE3UEXi9lPaBRpQ6XuRW0nU7hgg4zlmZZa+a9q4=
github.com/Azure/azure-sdk-for-go/sdk/internal v1.12.0/go.mod h1:7dCRMLwisfRH3dBupKeNCioWYUZ4SS09Z14H+7i8ZoY=
github.com/Azure/azure-sdk-for-go/sdk/internal v1.13.0 h1:Terj1CMW4+NY5CpWjSRkqLoxh2DkYDdv/FH2z0897Uc=
github.com/Azure/azure-sdk-for-go/sdk/internal v1.13.0/go.mod h1:7Ut2Qw9vcwpLW5Q4MBy43uSxXS7tK2yjs94OJ6j5thE=
github.com/Azure/azure-sdk-for-go/sdk/keyvault/azkeys v0.10.0 h1:m/sWOGCREuSBqg2htVQTBY8nOZpyajYztF0vUvSZTuM=
github.com/Azure/azure-sdk-for-go/sdk/keyvault/azkeys v0.10.0/go.mod h1:Pu5Zksi2KrU7LPbZbNINx6fuVrUp/ffvpxdDj+i8LeE=
github.com/Azure/azure-sdk-for-go/sdk/keyvault/internal v0.7.1 h1:FbH3BbSb4bvGluTesZZ+ttN/MDsnMmQP36OSnDuSXqw=
@@ -116,12 +116,12 @@ github.com/Shopify/sarama v1.38.1 h1:lqqPUPQZ7zPqYlWpTh+LQ9bhYNu2xJL6k1SJN4WVe2A
github.com/Shopify/sarama v1.38.1/go.mod h1:iwv9a67Ha8VNa+TifujYoWGxWnu2kNVAQdSdZ4X2o5g=
github.com/Shopify/toxiproxy/v2 v2.5.0 h1:i4LPT+qrSlKNtQf5QliVjdP08GyAH8+BUIc9gT0eahc=
github.com/Shopify/toxiproxy/v2 v2.5.0/go.mod h1:yhM2epWtAmel9CB8r2+L+PCmhH6yH2pITaPAo7jxJl0=
github.com/ThreeDotsLabs/watermill v1.5.2 h1:0ES33Eq1jEsP/pWvtE4n8bE0bs+9Jq7boT7wGBCVY6Q=
github.com/ThreeDotsLabs/watermill v1.5.2/go.mod h1:i9/968UriGphWfEbfMuYSD1qFbYRjb0mE0r+rV0FPp4=
github.com/ThreeDotsLabs/watermill v1.5.3 h1:GoTR7fW1ZT+itzUv/w3VB6Um1B7oTBrOWjJ8sOA9XqU=
github.com/ThreeDotsLabs/watermill v1.5.3/go.mod h1:i9/968UriGphWfEbfMuYSD1qFbYRjb0mE0r+rV0FPp4=
github.com/TomiHiltunen/geohash-golang v0.0.0-20150112065804-b3e4e625abfb h1:wumPkzt4zaxO4rHPBrjDK8iZMR41C1qs7njNqlacwQg=
github.com/TomiHiltunen/geohash-golang v0.0.0-20150112065804-b3e4e625abfb/go.mod h1:QiYsIBRQEO+Z4Rz7GoI+dsHVneZNONvhczuA+llOZNM=
github.com/a-h/templ v0.3.1070 h1:3iCOXHXqFXa99OQzgkUOLJbNzv/QjFGLwKZkBAg+ADA=
github.com/a-h/templ v0.3.1070/go.mod h1:iIqB3g2RHBe/WlnPwa+Bvhh1TKXHWLRnBD9NWHOLm78=
github.com/a-h/templ v0.3.1100 h1:lQhjVs1IL+M6QWUvojaFBd21+kzwATppSyWkWB9oHoM=
github.com/a-h/templ v0.3.1100/go.mod h1:j1WAOR7749mlo1AxFLQ7qbDFW5VwyoqszRTlKqEHN9M=
github.com/a1ex3/zstd-seekable-format-go/pkg v0.10.0 h1:iLDOF0rdGTrol/q8OfPIIs5kLD8XvA2q75o6Uq/tgak=
github.com/a1ex3/zstd-seekable-format-go/pkg v0.10.0/go.mod h1:DrEWcQJjz7t5iF2duaiyhg4jyoF0kxOD6LtECNGkZ/Q=
github.com/aalpar/deheap v1.1.2 h1:MABHLcnjqsffb8GLkUFDigqpBBxOMz0DoKM9QfELeTw=
@@ -144,8 +144,8 @@ github.com/alexbrainman/sspi v0.0.0-20250919150558-7d374ff0d59e h1:4dAU9FXIyQktp
github.com/alexbrainman/sspi v0.0.0-20250919150558-7d374ff0d59e/go.mod h1:cEWa1LVoE5KvSD9ONXsZrj0z6KqySlCCNKHlLzbqAt4=
github.com/anchore/go-lzo v0.1.1 h1:IwL/fvkdtlIrYIXck6WxZ3nb8WjjHziYYmGxlooyOnM=
github.com/anchore/go-lzo v0.1.1/go.mod h1:3kLx0bve2oN1iDwgM1U5zGku1Tfbdb0No5qp1eL1fIk=
github.com/andybalholm/brotli v1.2.6 h1:ftYnfj6usCp+UGV5kSJ3+chpMQgU+gJf/AxsUQ52REI=
github.com/andybalholm/brotli v1.2.6/go.mod h1:rzTDkvFWvIrjDXZHkuS16NPggd91W3kUSvPlQ1pLaKY=
github.com/andybalholm/brotli v1.2.7 h1:UgafRPGMam5ySHfRkhthMqOFA6IFoeF3D7mWIDLjG5A=
github.com/andybalholm/brotli v1.2.7/go.mod h1:rzTDkvFWvIrjDXZHkuS16NPggd91W3kUSvPlQ1pLaKY=
github.com/andybalholm/cascadia v1.3.4 h1:vM2lgh0Vru9Vwyfm4cQqWP2HHMW0u0+2PAW7Q38Qufg=
github.com/andybalholm/cascadia v1.3.4/go.mod h1:BLRmbRjpEtNKieZOCCvYj4RqN+KRA41GBe/5O+G93kM=
github.com/antihax/optional v1.0.0/go.mod h1:uupD/76wgC+ih3iEmQUL+0Ugr19nfwCT1kdvxnR2qWY=
@@ -474,8 +474,8 @@ github.com/geoarrow/geoarrow-go v0.0.0-20260403143023-f54751c3e3a1 h1:8VcjuP5pKZ
github.com/geoarrow/geoarrow-go v0.0.0-20260403143023-f54751c3e3a1/go.mod h1:XMOIOA5J96jEzAOpBROPyz0n14sHsooeF+y2FtGdrCY=
github.com/geoffgarside/ber v1.2.0 h1:/loowoRcs/MWLYmGX9QtIAbA+V/FrnVLsMMPhwiRm64=
github.com/geoffgarside/ber v1.2.0/go.mod h1:jVPKeCbj6MvQZhwLYsGwaGI52oUorHoHKNecGT85ZCc=
github.com/getsentry/sentry-go v0.49.0 h1:Ehejknu1l023Ub7QoRBVLAI7g3Jnhqku4oWx4B4Sh5s=
github.com/getsentry/sentry-go v0.49.0/go.mod h1:nuMJAoCfe1u0Bts2ocyNI+TW8HT84vRMqwA5Qq/SKUI=
github.com/getsentry/sentry-go v0.50.0 h1:gkO59rn/i0RCAXUUTflC5QVbTQGPXbaFabs2olVzc9g=
github.com/getsentry/sentry-go v0.50.0/go.mod h1:+xsDuN+SoRN/xfs1hdq78wnrE3omIZf+FCuh9FOW1C8=
github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
github.com/gin-contrib/sse v1.1.0 h1:n0w2GMuUpWDVp7qSpvze6fAu9iRxJY4Hmj6AmBOU05w=
github.com/gin-contrib/sse v1.1.0/go.mod h1:hxRZ5gVpWMT7Z0B0gSNYqqsSCNIJMjzvm6fqCz9vjwM=
@@ -483,8 +483,8 @@ github.com/gin-gonic/gin v1.11.0 h1:OW/6PLjyusp2PPXtyxKHU0RbX6I/l28FTdDlae5ueWk=
github.com/gin-gonic/gin v1.11.0/go.mod h1:+iq/FyxlGzII0KHiBGjuNn4UNENUlKbGlNmc+W50Dls=
github.com/go-asn1-ber/asn1-ber v1.5.8 h1:H9AZkK22UOmfX8J84ubyaZxKJZ3FMHVwn8swoMML7iQ=
github.com/go-asn1-ber/asn1-ber v1.5.8/go.mod h1:hEBeB/ic+5LoWskz+yKT7vGhhPYkProFKoKdwZRWMe0=
github.com/go-chi/chi/v5 v5.3.1 h1:3j4HZLGZQ3JpMCrPJF/Jl3mYJfWLKBfNJ6quurUGCf8=
github.com/go-chi/chi/v5 v5.3.1/go.mod h1:R+tYY2hNuVUUjxoPtqUdgBqevM9s9njzkTLutVsOCto=
github.com/go-chi/chi/v5 v5.4.0 h1:+ufzvFTGKyin26Or5DLrlIDq4ygIDBVcj7oVNeeaYFE=
github.com/go-chi/chi/v5 v5.4.0/go.mod h1:d1IG7GtKPSjPyBP365PH8sAGVHk3qfB/IhjFNRH1wWM=
github.com/go-darwin/apfs v0.0.0-20211011131704-f84b94dbf348 h1:JnrjqG5iR07/8k7NqrLNilRsl3s1EPRQEGvbPyOce68=
github.com/go-darwin/apfs v0.0.0-20211011131704-f84b94dbf348/go.mod h1:Czxo/d1g948LtrALAZdL04TL/HnkopquAjxYUuI02bo=
github.com/go-errors/errors v1.5.1 h1:ZwEMSLRCapFLflTpT7NKaAc7ukJ8ZPEjzlxt8rPN8bk=
@@ -1005,8 +1005,8 @@ github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINE
github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pkg/sftp v1.13.11 h1:0N92SLTB8JqASJB14ZLHHzFnBV8mG9zw4K7jghEFWuE=
github.com/pkg/sftp v1.13.11/go.mod h1:uNkH9roSXglNJqM+glJJi+TQXQUm0fXFWqCFmT8hsN0=
github.com/pkg/sftp v1.13.12 h1:Myhzy/ZzLPnCtE2F2WtDHsgs8RfgMnH2gfbHuCJhkMU=
github.com/pkg/sftp v1.13.12/go.mod h1:uNkH9roSXglNJqM+glJJi+TQXQUm0fXFWqCFmT8hsN0=
github.com/pkg/xattr v0.4.12 h1:rRTkSyFNTRElv6pkA3zpjHpQ90p/OdHQC1GmGh1aTjM=
github.com/pkg/xattr v0.4.12/go.mod h1:di8WF84zAKk8jzR1UBTEWh9AUlIZZ7M/JNt8e9B6ktU=
github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10 h1:GFCKgmp0tecUJ0sJuv4pzYCqS9+RGSn52M3FUwPs+uo=
@@ -1063,8 +1063,8 @@ github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 h1:N/ElC8H3+5X
github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475/go.mod h1:bCqnVzQkZxMG4s8nGwiZ5l3QUCyqpo9Y+/ZMZ9VjZe4=
github.com/rdleal/intervalst v1.5.0 h1:SEB9bCFz5IqD1yhfH1Wv8IBnY/JQxDplwkxHjT6hamU=
github.com/rdleal/intervalst v1.5.0/go.mod h1:xO89Z6BC+LQDH+IPQQw/OESt5UADgFD41tYMUINGpxQ=
github.com/redis/go-redis/v9 v9.22.0 h1:laDvpYXTJtZLloinw1fA5Kqd6HAEH2XKxOkG/PDq2F0=
github.com/redis/go-redis/v9 v9.22.0/go.mod h1:y2g0Wj8rQvuK0ELM+oxSudcLtC09JScs98I/X9gRWY4=
github.com/redis/go-redis/v9 v9.23.0 h1:/TTVdJa8BRNsybwwGGnmiYktHNlUXtbOQxlWpHBPL3s=
github.com/redis/go-redis/v9 v9.23.0/go.mod h1:EWP2UUk+XDaI1s5nQB5mablZN6vccXlmBQTyfnF87D0=
github.com/redis/rueidis v1.0.77 h1:ZR41bgJcm7oRFb3aSDPrRhC0eonDSrPzjvvZvHIlNjM=
github.com/redis/rueidis v1.0.77/go.mod h1:L8mnCQJJaSNL6I4pIR6Rz732HTGS9vmuXm0yT9dRvjo=
github.com/redis/rueidis/rueidiscompat v1.0.77 h1:S1xiLQsFv8XQ46e0LelC8ihGqbDzVRocXXZLzVAeh64=
+2 -2
View File
@@ -1,6 +1,6 @@
apiVersion: v1
description: SeaweedFS
name: seaweedfs
appVersion: "4.48"
appVersion: "4.51"
# Dev note: Trigger a helm chart release by `git tag -a helm-<version>`
version: 4.48.2
version: 4.51.0
@@ -170,9 +170,6 @@ spec:
{{- if .Values.filer.metricsIp }}
-metricsIp={{ .Values.filer.metricsIp }} \
{{- end }}
{{- if .Values.filer.redirectOnRead }}
-redirectOnRead \
{{- end }}
{{- if .Values.filer.disableHttp }}
-disableHttp \
{{- end }}
@@ -172,7 +172,8 @@ spec:
{{- $bucketName := .name }}
{{- $bucketLock := or .lock .objectLock .withLock }}
bucket_list=$(/bin/echo 's3.bucket.list' | /usr/bin/weed shell) || { echo "Error listing s3 buckets"; exit 1; }
if echo "$bucket_list" | awk '{print $1}' | grep -Fxq "{{ $bucketName }}"; then
{{- /* One awk: grep -q exits early, awk takes SIGPIPE, pipefail fails the test. */}}
if printf '%s\n' "$bucket_list" | awk -v bucket="{{ $bucketName }}" '$1 "" == bucket { found = 1 } END { exit !found }'; then
echo "Bucket '{{ $bucketName }}' already exists, skipping creation."
else
echo "Creating bucket '{{ $bucketName }}'..."
-3
View File
@@ -740,9 +740,6 @@ filer:
# encrypt data on volume servers
encryptVolumeData: false
# Whether proxy or redirect to volume server during file GET request
redirectOnRead: false
# Limit sub dir listing size (default 100000)
dirListLimit: 100000
+1037 -1099
View File
File diff suppressed because it is too large. Load diff

Before

Width:  |  Height:  |  Size: 55 KiB

After

Width:  |  Height:  |  Size: 53 KiB

+23 -2
View File
@@ -166,6 +166,10 @@ message Entry {
repeated FileChunk chunks = 3;
FuseAttributes attributes = 4;
map<string, bytes> extended = 5;
// filer-local commit timestamp of the version stored here: the meta log
// event's ts_ns for the mutation that produced this entry, or 0 for
// entries written before the field existed
int64 commit_ts_ns = 6;
bytes hard_link_id = 7;
int32 hard_link_counter = 8; // only exists in hard link meta data
bytes content = 9; // if not empty, the file content
@@ -281,6 +285,7 @@ message WriteCondition {
IF_EXTENDED_NOT_EQUAL = 7; // fail if present and extended[ext_key] == ext_value
IF_EXTENDED_TIME_ELAPSED = 8; // fail if present and extended[ext_key] (unix seconds) is in the future
IF_CHUNKS_EQUAL = 9; // fail unless the stored chunk fid multiset equals fids (absent entry = no chunks)
IF_ENTRY_EQUAL = 10; // fail unless the stored entry equals expected_entry
}
// Clause is one primitive comparison. IF_ETAG_MATCH holds when the current
// entry's ETag equals any value in etags; IF_ETAG_NOT_MATCH holds when it
@@ -309,6 +314,7 @@ message WriteCondition {
string gate_key = 7; // IF_EXTENDED_TIME_ELAPSED: only enforce when extended[gate_key] == gate_value
string gate_value = 8; // gate value (e.g. retention mode COMPLIANCE for governance bypass)
repeated string fids = 9; // chunk fid strings for IF_CHUNKS_EQUAL
Entry expected_entry = 10; // whole-entry comparison for IF_ENTRY_EQUAL
}
repeated Clause clauses = 1; // all must hold (logical AND)
}
@@ -323,6 +329,7 @@ enum FilerError {
EXISTING_IS_FILE = 4; // cannot overwrite file with directory
ENTRY_ALREADY_EXISTS = 5; // O_EXCL and entry already exists
PRECONDITION_FAILED = 6; // WriteCondition not satisfied
READ_ONLY = 7; // path matches a read-only storage rule (e.g. bucket over quota)
}
// ObjectMutation is one entry-level change applied by ObjectTransaction. All
@@ -351,6 +358,7 @@ message ObjectMutation {
bytes content = 11; // PATCH_EXTENDED: new Entry.content when set_content
bool touch_mtime = 12; // PATCH_EXTENDED: set the entry's Mtime to now (e.g. a metadata-replace copy)
bool remove_empty_parent = 13; // DELETE: also remove the parent directory when the delete leaves it empty (best-effort)
bool keep_remote_object = 14; // DELETE: leave the object on the mounted remote storage
}
// Recompute re-derives a pointer entry (directory/name on the mutation) from the
@@ -388,6 +396,8 @@ message ObjectTransactionRequest {
string condition_key = 6; // if set, evaluate the condition against this entry instead of lock_key (still locking lock_key)
string route_key = 7; // ring key identifying the owner filer; a non-owner forwards the whole transaction to it
bool is_moved = 8; // set on a forwarded transaction so the receiver applies it locally instead of forwarding again
string request_id = 9; // idempotency token: the owner records the outcome and replays it on a retried request instead of re-applying
string condition_alt_key = 10; // if the condition's entry is absent, evaluate it against this entry instead (e.g. null slot, else latest version)
}
message ObjectTransactionResponse {
@@ -477,6 +487,7 @@ message UpdateEntryRequest {
// the write, under the filer's per-path lock. The caller must route the
// key's writes to this entry's owner filer for the check to be authoritative.
WriteCondition condition = 6;
bool is_moved = 7; // set on a forwarded request so the owner applies it locally
}
message UpdateEntryResponse {
SubscribeMetadataResponse metadata_event = 1;
@@ -513,7 +524,9 @@ message DeleteEntryRequest {
bool ignore_recursive_error = 6;
bool is_from_other_cluster = 7;
repeated int32 signatures = 8;
int64 if_not_modified_after = 9;
int64 if_not_modified_after = 9; // unix time in seconds
bool keep_remote_object = 10;
int64 if_not_modified_after_ns = 11; // unix time in nanoseconds; overrides if_not_modified_after when set
}
message DeleteEntryResponse {
@@ -573,6 +586,9 @@ message AssignVolumeResponse {
// fsync is the storage rule's fsync decision for the assigned path, so the
// client can carry it onto the volume server upload request.
bool fsync = 11;
// error_code classifies error for callers that must branch on it; error
// keeps the human-readable text for clients that predate the field.
FilerError error_code = 12;
}
message LookupVolumeRequest {
@@ -709,6 +725,10 @@ message TraverseBfsMetadataRequest {
message TraverseBfsMetadataResponse {
string directory = 1;
Entry entry = 2;
// The sender's per-path commit timestamp (see meta_path_ts.go): the newest
// event time this filer committed for the path, 0 when unrecorded so
// subscribers fall back to the entry's mtime.
int64 entry_ts_ns = 3;
}
message LogEntry {
@@ -771,7 +791,8 @@ message FilerConf {
string replication = 3;
string ttl = 4;
string disk_type = 5;
bool fsync = 6;
// unset inherits from the enclosing path rule, set overrides it
optional bool fsync = 6;
uint32 volume_growth_count = 7;
bool read_only = 8;
string data_center = 9;
+6832
View File
File diff suppressed because it is too large. Load diff
+28
View File
@@ -0,0 +1,28 @@
# Vector query engine for SeaweedFS Lance table buckets.
#
# `core` is the engine: open a table through the Lance namespace and run
# top-K over it. The crate layout follows vector_buckets.md; a `server`
# crate with the VectorService gRPC endpoint lands with the S3 Vectors
# milestone.
[workspace]
resolver = "2"
members = ["crates/core", "crates/server"]
[workspace.package]
version = "0.1.0"
edition = "2024"
# The edition needs 1.85; the lance dependency tree needs more. Same floor as
# seaweed-worker, which this workspace path-depends on.
rust-version = "1.94.1"
# No common root manifest links this tree to seaweed-worker or
# seaweed-common; keep the lint policy identical to theirs.
[workspace.lints.clippy]
needless_update = "allow"
undocumented_unsafe_blocks = "warn"
[workspace.dependencies]
anyhow = "1"
tokio = { version = "1", features = ["full"] }
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
+41
View File
@@ -0,0 +1,41 @@
[package]
name = "weed-vector-core"
version.workspace = true
edition.workspace = true
rust-version.workspace = true
description = "Vector query engine over SeaweedFS Lance table buckets"
[lib]
name = "weed_vector_core"
[[bin]]
name = "weed-vector"
path = "src/main.rs"
[dependencies]
# The catalog client and dataset-opening helper the maintenance worker uses.
# A path dependency because the two trees are separate cargo workspaces with
# no common root manifest.
weed-lance-worker = { path = "../../../seaweed-worker/crates/lance" }
lance = { version = "10", default-features = false, features = ["aws"] }
lance-index = "10"
lance-linalg = "10"
lance-table = "10"
arrow-array = "58"
arrow-cast = "58"
arrow-schema = "58"
roaring = "0.11"
anyhow.workspace = true
clap = { version = "4", features = ["derive", "env"] }
futures = "0.3"
serde = { version = "1", features = ["derive"] }
serde_json = "1"
tokio.workspace = true
tracing.workspace = true
tracing-subscriber.workspace = true
[dev-dependencies]
tempfile = "3"
[lints]
workspace = true
+201
View File
@@ -0,0 +1,201 @@
//! Which fragments the vector index covers, and which it has not reached yet.
use std::collections::BTreeMap;
use anyhow::{Context, Result};
use lance::Dataset;
use lance::index::DatasetIndexExt;
use roaring::RoaringBitmap;
/// How much of the dataset one named index covers at this version.
#[derive(Debug, serde::Serialize)]
pub struct IndexCoverage {
pub index_name: String,
/// Physical index segments: the IndexMetadata deltas sharing the name.
pub segments: usize,
/// False when any delta of this index lacks a fragment bitmap: the index
/// exists but which fragments it covers is unknown.
pub coverage_known: bool,
pub indexed_fragments: Vec<u64>,
pub unindexed_fragments: Vec<u64>,
pub unindexed_rows: usize,
}
/// Coverage for every index built on `vector_column` at this version.
pub async fn coverage(dataset: &Dataset, vector_column: &str) -> Result<Vec<IndexCoverage>> {
let field = dataset
.schema()
.field(vector_column)
.with_context(|| format!("no such column {vector_column}"))?;
let mut by_name: BTreeMap<String, Vec<Option<RoaringBitmap>>> = BTreeMap::new();
for index in dataset.load_indices().await?.iter() {
if index.fields != [field.id] {
continue;
}
by_name
.entry(index.name.clone())
.or_default()
.push(index.fragment_bitmap.clone());
}
let all: RoaringBitmap = dataset.fragments().iter().map(|f| f.id as u32).collect();
let mut out = Vec::new();
for (name, bitmaps) in by_name {
let coverage_known = bitmaps.iter().all(|b| b.is_some());
let mut indexed = RoaringBitmap::new();
for bitmap in bitmaps.iter().flatten() {
indexed |= bitmap;
}
// The bitmap records what the index covered when built; fragments
// the manifest has since dropped are not part of this version.
let indexed = indexed & &all;
let unindexed = &all - &indexed;
let mut unindexed_rows = 0;
for fragment in dataset.get_fragments() {
// physical_rows counts deleted rows too, and may be absent in the
// manifest; count live rows instead.
if unindexed.contains(fragment.id() as u32) {
unindexed_rows += fragment.count_rows(None).await?;
}
}
out.push(IndexCoverage {
index_name: name,
segments: bitmaps.len(),
coverage_known,
indexed_fragments: indexed.iter().map(|id| id as u64).collect(),
unindexed_rows,
unindexed_fragments: unindexed.iter().map(|id| id as u64).collect(),
});
}
Ok(out)
}
#[cfg(test)]
mod tests {
use super::*;
use crate::query::{Query, ann, flat};
use crate::table::VectorTable;
use crate::testing::{DIM, append_dataset, vectors, write_dataset};
use lance::index::vector::VectorIndexParams;
use lance_index::IndexType;
use lance_linalg::distance::DistanceType;
use tempfile::TempDir;
async fn indexed_dataset(dir: &TempDir, tags: &[&str]) -> Dataset {
let mut dataset = write_dataset(dir, &vectors(tags.len(), DIM), tags).await;
let params = VectorIndexParams::ivf_flat(4, DistanceType::L2);
dataset
.create_index(
&["vector"],
IndexType::Vector,
Some("vidx".to_string()),
&params,
true,
)
.await
.unwrap();
dataset.checkout_latest().await.unwrap();
dataset
}
#[tokio::test]
async fn coverage_reports_the_unindexed_fragment() {
let dir = TempDir::new().unwrap();
let mut dataset = indexed_dataset(&dir, &["a"; 64]).await;
append_dataset(&mut dataset, &vectors(4, DIM), &["b"; 4]).await;
let coverages = coverage(&dataset, "vector").await.unwrap();
assert_eq!(coverages.len(), 1);
let cov = &coverages[0];
assert_eq!(cov.index_name, "vidx");
assert_eq!(cov.segments, 1);
assert_eq!(cov.indexed_fragments, vec![0]);
assert_eq!(cov.unindexed_fragments.len(), 1);
assert_eq!(cov.unindexed_rows, 4);
}
#[tokio::test]
async fn ann_searches_unindexed_fragments() {
let dir = TempDir::new().unwrap();
let mut dataset = indexed_dataset(&dir, &["a"; 64]).await;
append_dataset(&mut dataset, &[vec![9.0; DIM]], &["b"]).await;
let table = VectorTable::resolve(&dataset, None, None).unwrap();
let hits = ann(
&dataset,
&table,
&Query {
vector: vec![9.0; DIM],
k: 1,
filter: None,
postfilter: false,
distance: DistanceType::L2,
nprobes: Some(4),
},
)
.await
.unwrap();
assert_eq!(hits.len(), 1);
assert_eq!(hits[0].key, "k64");
assert!(hits[0].distance < 1e-6);
}
#[tokio::test]
async fn prefiltered_ann_matches_filtered_flat() {
let dir = TempDir::new().unwrap();
let tags: Vec<&str> = (0..64)
.map(|i| if i % 2 == 0 { "a" } else { "b" })
.collect();
let mut dataset = indexed_dataset(&dir, &tags).await;
append_dataset(&mut dataset, &vectors(4, DIM), &["a"; 4]).await;
let table = VectorTable::resolve(&dataset, None, None).unwrap();
let query = Query {
vector: vectors(1, DIM).remove(0),
k: 10,
filter: Some("tag = 'a'".to_string()),
postfilter: false,
distance: DistanceType::L2,
nprobes: Some(4),
};
let filtered = ann(&dataset, &table, &query).await.unwrap();
let exact = flat(&dataset, &table, &query).await.unwrap();
// equal-distance hits have no specified order between the two scans
let mut filtered_keys = filtered.iter().map(|h| &h.key).collect::<Vec<_>>();
let mut exact_keys = exact.iter().map(|h| &h.key).collect::<Vec<_>>();
filtered_keys.sort();
exact_keys.sort();
assert_eq!(filtered_keys, exact_keys);
assert!(filtered.len() <= 10);
assert!(filtered.iter().all(|h| h.key != "k1"));
}
#[tokio::test]
async fn pinned_version_ignores_newer_commits() {
let dir = TempDir::new().unwrap();
let mut dataset = write_dataset(&dir, &vectors(4, DIM), &["a"; 4]).await;
let pinned = dataset
.checkout_version(dataset.version().version)
.await
.unwrap();
append_dataset(&mut dataset, &vectors(4, DIM), &["b"; 4]).await;
assert_eq!(pinned.count_rows(None).await.unwrap(), 4);
let table = VectorTable::resolve(&pinned, None, None).unwrap();
let hits = flat(
&pinned,
&table,
&Query {
vector: vectors(1, DIM).remove(0),
k: 10,
filter: None,
postfilter: false,
distance: DistanceType::L2,
nprobes: None,
},
)
.await
.unwrap();
assert_eq!(hits.len(), 4);
assert!(hits.iter().all(|h| h.row_id < 4));
}
}
+18
View File
@@ -0,0 +1,18 @@
//! Vector queries over the Lance tables the SeaweedFS namespace registers.
//!
//! A table bucket's rows carry a key column, a fixed-size float vector column,
//! and metadata columns. This crate resolves that layout and runs top-K over
//! the whole dataset: index-accelerated where an ANN index exists, exact where
//! it does not, always covering the fragments no index has reached yet.
pub mod coverage;
pub mod query;
pub mod table;
#[cfg(test)]
mod testing;
#[cfg(test)]
mod verify;
pub use coverage::IndexCoverage;
pub use query::{Hit, Query};
pub use table::VectorTable;
+305
View File
@@ -0,0 +1,305 @@
use anyhow::{Context, Result, bail};
use clap::{Parser, Subcommand};
use weed_lance_worker::catalog::{NamespaceClient, parse_id};
use weed_lance_worker::dataset::{self, FallbackOptions};
use weed_vector_core::coverage;
use weed_vector_core::query::{self, Query};
use weed_vector_core::table::VectorTable;
/// Mirrors `weed-worker`'s flags where the two touch the same thing: the
/// namespace is WEED_LANCE_NAMESPACE and the fallback storage credentials are
/// the same env vars.
#[derive(Parser, Debug)]
#[command(name = "weed-vector", about = "SeaweedFS vector query spike")]
struct Args {
/// Lance namespace the tables live in.
#[arg(
long,
default_value = "http://localhost:9101",
env = "WEED_LANCE_NAMESPACE"
)]
namespace: String,
/// Storage credentials to use where the namespace vends none.
#[arg(long, env = "WEED_S3_ACCESS_KEY")]
access_key: Option<String>,
#[arg(long, env = "WEED_S3_SECRET_KEY")]
secret_key: Option<String>,
#[command(subcommand)]
command: Command,
}
#[derive(Subcommand, Debug)]
enum Command {
/// A table's location, version, vector layout and index coverage.
Describe {
/// Delimiter-joined table id, e.g. bucket$table.
id: String,
#[arg(long)]
vector_column: Option<String>,
#[arg(long)]
key_column: Option<String>,
},
/// Which fragments each vector index covers.
Coverage {
id: String,
#[arg(long)]
vector_column: Option<String>,
#[arg(long)]
key_column: Option<String>,
},
/// A top-K query over the whole dataset.
Query {
id: String,
/// Comma-separated floats, e.g. --vector 0.1,0.2,0.3,0.4
#[arg(long, allow_hyphen_values = true)]
vector: String,
#[arg(long, default_value = "10")]
k: usize,
/// Lance SQL filter over the metadata columns.
#[arg(long)]
filter: Option<String>,
/// l2, cosine or dot.
#[arg(long, default_value = "l2")]
metric: String,
/// Exact scan, no index.
#[arg(long)]
flat: bool,
/// Filter the top-K rows after ranking (Lance's default); off keeps
/// the filter before the ANN search.
#[arg(long)]
postfilter: bool,
/// Pin the query to this committed version.
#[arg(long)]
version: Option<u64>,
#[arg(long)]
vector_column: Option<String>,
#[arg(long)]
key_column: Option<String>,
#[arg(long)]
nprobes: Option<usize>,
},
/// Fetch rows by key.
Get {
id: String,
keys: Vec<String>,
#[arg(long)]
vector_column: Option<String>,
#[arg(long)]
key_column: Option<String>,
},
}
/// `id` is a delimiter-joined table id; when it looks like a URI (`s3://...`,
/// `/path/...`, `file://...`) the dataset is opened directly, skipping the
/// namespace — useful for a spike run against a local dataset.
async fn open_dataset(
client: &NamespaceClient,
id: &str,
fallback: &FallbackOptions,
) -> Result<lance::Dataset> {
if id.contains("://") || id.starts_with('/') {
let mut builder = lance::dataset::builder::DatasetBuilder::from_uri(id);
if !fallback.is_empty() {
builder = builder.with_storage_options(fallback.clone());
}
return builder.load().await.with_context(|| format!("open {id}"));
}
let table = dataset::open(client, &parse_id(id), fallback).await?;
Ok(table.dataset)
}
#[tokio::main]
async fn main() -> Result<()> {
weed_lance_worker::tls::install_default_crypto_provider();
tracing_subscriber::fmt()
.with_env_filter(
tracing_subscriber::EnvFilter::try_from_default_env()
.unwrap_or_else(|_| tracing_subscriber::EnvFilter::new("info")),
)
.init();
let args = Args::parse();
let mut fallback = FallbackOptions::new();
if let (Some(access), Some(secret)) = (args.access_key, args.secret_key) {
fallback.insert("aws_access_key_id".to_string(), access);
fallback.insert("aws_secret_access_key".to_string(), secret);
}
let client = NamespaceClient::new(args.namespace).with_fallback(&fallback);
match args.command {
Command::Describe {
id,
vector_column,
key_column,
} => {
let dataset = open_dataset(&client, &id, &fallback).await?;
let schema: arrow_schema::Schema = dataset.schema().into();
let resolved =
VectorTable::resolve(&dataset, vector_column.as_deref(), key_column.as_deref());
let vector = match resolved {
Err(e) if vector_column.is_some() || key_column.is_some() => return Err(e),
r => r.ok(),
};
let coverages = match &vector {
Some(v) => coverage::coverage(&dataset, &v.vector_column).await?,
None => Vec::new(),
};
println!(
"{}",
serde_json::json!({
"id": id,
"uri": dataset.uri(),
"version": dataset.version().version,
"fragments": dataset.get_fragments().len(),
"rows": dataset.count_rows(None).await?,
"schema": schema.fields().iter().map(|f| serde_json::json!({
"name": f.name(),
"type": f.data_type().to_string(),
"nullable": f.is_nullable(),
})).collect::<Vec<_>>(),
"vector": vector.map(|v| serde_json::json!({
"vector_column": v.vector_column,
"dimension": v.dimension,
"key_column": v.key_column,
"metadata_columns": v.metadata_columns,
})),
"coverage": coverages,
})
);
}
Command::Coverage {
id,
vector_column,
key_column,
} => {
let dataset = open_dataset(&client, &id, &fallback).await?;
let vector =
VectorTable::resolve(&dataset, vector_column.as_deref(), key_column.as_deref())?;
let coverages = coverage::coverage(&dataset, &vector.vector_column).await?;
println!("{}", serde_json::to_string_pretty(&coverages)?);
}
Command::Get {
id,
keys,
vector_column,
key_column,
} => {
let dataset = open_dataset(&client, &id, &fallback).await?;
let vector =
VectorTable::resolve(&dataset, vector_column.as_deref(), key_column.as_deref())?;
for batch in query::get(&dataset, &vector, &keys).await? {
println!("{}", arrow_cast::pretty::pretty_format_batches(&[batch])?);
}
}
Command::Query {
id,
vector,
k,
filter,
metric,
flat,
postfilter,
version,
vector_column,
key_column,
nprobes,
} => {
let opened = open_dataset(&client, &id, &fallback).await?;
let dataset = match version {
Some(v) => opened.checkout_version(v).await?,
None => opened,
};
let vector_table =
VectorTable::resolve(&dataset, vector_column.as_deref(), key_column.as_deref())?;
let vector: Vec<f32> = vector
.split(',')
.map(|part| part.trim().parse::<f32>())
.collect::<std::result::Result<_, _>>()
.context("--vector takes comma-separated floats")?;
let distance = match metric.as_str() {
"l2" | "euclidean" => lance_linalg::distance::DistanceType::L2,
"cosine" => lance_linalg::distance::DistanceType::Cosine,
"dot" | "dotproduct" => lance_linalg::distance::DistanceType::Dot,
other => bail!("unknown metric {other}: l2, cosine or dot"),
};
let query = Query {
vector,
k,
filter,
postfilter,
distance,
nprobes,
};
let hits = if flat {
query::flat(&dataset, &vector_table, &query).await?
} else {
query::ann(&dataset, &vector_table, &query).await?
};
println!("{}", serde_json::to_string_pretty(&hits)?);
}
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn vector_flag_takes_hyphenated_floats() {
let args = Args::try_parse_from([
"weed-vector",
"query",
"t",
"--vector",
"-1,-1,-1,-1",
"--k",
"3",
])
.unwrap();
match args.command {
Command::Query { vector, k, .. } => {
assert_eq!(vector, "-1,-1,-1,-1");
assert_eq!(k, 3);
}
_ => panic!("expected query"),
}
}
#[test]
fn column_flags_on_get_describe_coverage() {
for argv in [
vec!["weed-vector", "get", "t", "k1"],
vec!["weed-vector", "describe", "t"],
vec!["weed-vector", "coverage", "t"],
] {
let mut argv = argv;
argv.extend(["--vector-column", "v", "--key-column", "kk"]);
let args = Args::try_parse_from(argv).unwrap();
let (v, k) = match args.command {
Command::Get {
vector_column,
key_column,
..
}
| Command::Describe {
vector_column,
key_column,
..
}
| Command::Coverage {
vector_column,
key_column,
..
} => (vector_column, key_column),
_ => panic!("unexpected subcommand"),
};
assert_eq!(v.as_deref(), Some("v"));
assert_eq!(k.as_deref(), Some("kk"));
}
}
}
+299
View File
@@ -0,0 +1,299 @@
//! Top-K queries over a [`VectorTable`]: exact flat scan, or the ANN index
//! with a flat pass over the fragments it does not cover yet.
use anyhow::{Context, Result, bail};
use arrow_array::{Array, Float32Array, RecordBatch, StringArray, UInt64Array};
use futures::TryStreamExt;
use lance::Dataset;
use lance::dataset::scanner::Scanner;
use lance_linalg::distance::DistanceType;
use crate::table::VectorTable;
const DISTANCE_COLUMN: &str = "_distance";
const ROW_ID_COLUMN: &str = "_rowid";
/// One vector search request.
#[derive(Debug, Clone)]
pub struct Query {
pub vector: Vec<f32>,
pub k: usize,
/// A Lance SQL filter over the metadata columns.
pub filter: Option<String>,
/// Filter the top-K rows after ranking instead of the eligible rows
/// before it. Lance's default post-filters and can return fewer than k
/// results while eligible neighbours go unreported; keep it off for
/// S3 Vectors filter semantics.
pub postfilter: bool,
pub distance: DistanceType,
pub nprobes: Option<usize>,
}
/// One row of a top-K answer.
#[derive(Debug, Clone, serde::Serialize)]
pub struct Hit {
/// The key column value, or the row id when the table has no key column.
pub key: String,
/// Raw Lance `_distance` in the query's metric.
pub distance: f32,
pub row_id: u64,
}
/// Exact top-K over every fragment, ignoring any index.
pub async fn flat(dataset: &Dataset, table: &VectorTable, query: &Query) -> Result<Vec<Hit>> {
run(scanner(dataset, table, query, false)?, table).await
}
/// Top-K with the ANN index where it covers the fragment and a flat pass over
/// the rest. Whether the index is used is Lance's decision per fragment, so
/// rows written after the last index build are always candidates.
pub async fn ann(dataset: &Dataset, table: &VectorTable, query: &Query) -> Result<Vec<Hit>> {
run(scanner(dataset, table, query, true)?, table).await
}
/// Fetch rows by key.
pub async fn get(
dataset: &Dataset,
table: &VectorTable,
keys: &[String],
) -> Result<Vec<RecordBatch>> {
let Some(key_column) = &table.key_column else {
bail!("the table has no key column to look up");
};
if keys.is_empty() {
return Ok(Vec::new());
}
let list = keys
.iter()
.map(|k| format!("'{}'", k.replace('\'', "''")))
.collect::<Vec<_>>()
.join(", ");
let mut scanner = dataset.scan();
scanner
.filter(&format!("`{}` in ({list})", key_column.replace('`', "``")))?
.with_row_id();
scanner
.try_into_stream()
.await?
.try_collect()
.await
.context("scan by keys")
}
fn scanner(
dataset: &Dataset,
table: &VectorTable,
query: &Query,
use_index: bool,
) -> Result<Scanner> {
if query.vector.len() != table.dimension {
bail!(
"query dim({}) does not match the table's dim({})",
query.vector.len(),
table.dimension
);
}
let vector = Float32Array::from(query.vector.clone());
let mut scanner = dataset.scan();
scanner
.nearest(&table.vector_column, &vector, query.k)?
.distance_metric(query.distance)
.use_index(use_index)
.with_row_id();
if let Some(filter) = &query.filter {
scanner.filter(filter)?;
scanner.prefilter(!query.postfilter);
}
if let Some(nprobes) = query.nprobes {
scanner.nprobes(nprobes);
}
let mut columns = Vec::new();
if let Some(key) = &table.key_column {
columns.push(key.clone());
}
columns.push(DISTANCE_COLUMN.to_string());
columns.push(ROW_ID_COLUMN.to_string());
scanner.project(&columns)?;
Ok(scanner)
}
async fn run(scanner: Scanner, table: &VectorTable) -> Result<Vec<Hit>> {
let mut stream = scanner.try_into_stream().await?;
let mut out = Vec::new();
while let Some(batch) = stream.try_next().await? {
out.extend(hits(&batch, table)?);
}
Ok(out)
}
fn hits(batch: &RecordBatch, table: &VectorTable) -> Result<Vec<Hit>> {
let distances = batch
.column_by_name(DISTANCE_COLUMN)
.and_then(|c| c.as_any().downcast_ref::<Float32Array>())
.context("the scan did not return a _distance column")?;
let row_ids = batch
.column_by_name(ROW_ID_COLUMN)
.and_then(|c| c.as_any().downcast_ref::<UInt64Array>())
.context("the scan did not return a _rowid column")?;
let keys = table.key_column.as_ref().and_then(|name| {
batch
.column_by_name(name)
.and_then(|c| c.as_any().downcast_ref::<StringArray>())
});
if table.key_column.is_some() && keys.is_none() {
bail!("key column did not come back as Utf8");
}
(0..batch.num_rows())
.map(|i| {
let key = match keys {
Some(keys) if !keys.is_null(i) => keys.value(i).to_string(),
Some(_) => bail!(
"key column {} has a null at row {i}",
table.key_column.as_deref().unwrap_or_default()
),
None => row_ids.value(i).to_string(),
};
Ok(Hit {
key,
distance: distances.value(i),
row_id: row_ids.value(i),
})
})
.collect()
}
#[cfg(test)]
mod tests {
use super::*;
use crate::table::VectorTable;
use crate::testing::{DIM, write_dataset};
use tempfile::TempDir;
fn l2(a: &[f32], b: &[f32]) -> f32 {
a.iter().zip(b).map(|(x, y)| (x - y) * (x - y)).sum()
}
#[tokio::test]
async fn flat_returns_exact_top_k() {
let dir = TempDir::new().unwrap();
let vectors = vec![
vec![1.0, 0.0, 0.0, 0.0],
vec![0.9, 0.1, 0.0, 0.0],
vec![0.0, 1.0, 0.0, 0.0],
vec![0.0, 0.0, 1.0, 0.0],
];
let dataset = write_dataset(&dir, &vectors, &["a", "b", "a", "b"]).await;
let table = VectorTable::resolve(&dataset, None, None).unwrap();
let query = Query {
vector: vec![1.0, 0.0, 0.0, 0.0],
k: 2,
filter: None,
postfilter: false,
distance: DistanceType::L2,
nprobes: None,
};
let hits = flat(&dataset, &table, &query).await.unwrap();
assert_eq!(hits.len(), 2);
assert_eq!(hits[0].key, "k0");
assert!(hits[0].distance <= hits[1].distance);
let mut brute: Vec<(String, f32)> = vectors
.iter()
.enumerate()
.map(|(i, v)| (format!("k{i}"), l2(&query.vector, v)))
.collect();
brute.sort_by(|a, b| a.1.total_cmp(&b.1));
for (hit, (key, distance)) in hits.iter().zip(brute.iter()) {
assert_eq!(&hit.key, key);
assert!((hit.distance - distance).abs() < 1e-5);
}
}
#[tokio::test]
async fn flat_applies_the_filter() {
let dir = TempDir::new().unwrap();
let vectors = vec![
vec![1.0, 0.0, 0.0, 0.0],
vec![0.9, 0.1, 0.0, 0.0],
vec![0.0, 1.0, 0.0, 0.0],
];
let dataset = write_dataset(&dir, &vectors, &["a", "b", "a"]).await;
let table = VectorTable::resolve(&dataset, None, None).unwrap();
let hits = flat(
&dataset,
&table,
&Query {
vector: vec![1.0, 0.0, 0.0, 0.0],
k: 3,
filter: Some("tag = 'a'".to_string()),
postfilter: false,
distance: DistanceType::L2,
nprobes: None,
},
)
.await
.unwrap();
assert_eq!(hits.len(), 2);
assert_eq!(hits[0].key, "k0");
assert_eq!(hits[1].key, "k2");
}
#[tokio::test]
async fn get_fetches_rows_by_key() {
let dir = TempDir::new().unwrap();
let vectors = vec![vec![1.0; DIM], vec![2.0; DIM], vec![3.0; DIM]];
let dataset = write_dataset(&dir, &vectors, &["a", "b", "c"]).await;
let table = VectorTable::resolve(&dataset, None, None).unwrap();
let batches = get(&dataset, &table, &["k0".to_string(), "k2".to_string()])
.await
.unwrap();
let rows: usize = batches.iter().map(|b| b.num_rows()).sum();
assert_eq!(rows, 2);
}
#[tokio::test]
async fn query_rejects_a_wrong_dimension() {
let dir = TempDir::new().unwrap();
let vectors = vec![vec![1.0; DIM]];
let dataset = write_dataset(&dir, &vectors, &["a"]).await;
let table = VectorTable::resolve(&dataset, None, None).unwrap();
let query = Query {
vector: vec![1.0; DIM + 1],
k: 1,
filter: None,
postfilter: false,
distance: DistanceType::L2,
nprobes: None,
};
assert!(flat(&dataset, &table, &query).await.is_err());
}
#[tokio::test]
async fn ann_matches_flat_without_an_index() {
let dir = TempDir::new().unwrap();
let vectors = vec![
vec![1.0, 0.0, 0.0, 0.0],
vec![0.0, 1.0, 0.0, 0.0],
vec![0.0, 0.0, 1.0, 0.0],
];
let dataset = write_dataset(&dir, &vectors, &["a", "b", "c"]).await;
let table = VectorTable::resolve(&dataset, None, None).unwrap();
let query = Query {
vector: vec![1.0, 0.0, 0.0, 0.0],
k: 2,
filter: None,
postfilter: false,
distance: DistanceType::L2,
nprobes: None,
};
let indexed = ann(&dataset, &table, &query).await.unwrap();
let exact = flat(&dataset, &table, &query).await.unwrap();
assert_eq!(
indexed.iter().map(|h| &h.key).collect::<Vec<_>>(),
exact.iter().map(|h| &h.key).collect::<Vec<_>>()
);
}
}
+178
View File
@@ -0,0 +1,178 @@
//! The columns a vector query needs, resolved from a table's Arrow schema.
use anyhow::{Context, Result, bail};
use arrow_schema::DataType;
use lance::Dataset;
/// The default name for the vector column, matching what the S3 Vectors API
/// writes.
const DEFAULT_VECTOR_COLUMN: &str = "vector";
/// The default name for the key column.
const DEFAULT_KEY_COLUMN: &str = "key";
/// Layout of a Lance table used as a vector index.
#[derive(Debug, Clone)]
pub struct VectorTable {
/// The row key column; `None` falls back to `_rowid` in results.
pub key_column: Option<String>,
pub vector_column: String,
pub dimension: usize,
pub metadata_columns: Vec<String>,
}
impl VectorTable {
/// Resolve the vector layout of `dataset`. `vector_column` and
/// `key_column` override detection: the vector column defaults to the one
/// named `vector`, or the sole fixed-size float list when unambiguous.
pub fn resolve(
dataset: &Dataset,
vector_column: Option<&str>,
key_column: Option<&str>,
) -> Result<Self> {
let schema: arrow_schema::Schema = dataset.schema().into();
resolve_schema(&schema, vector_column, key_column)
}
}
pub(crate) fn resolve_schema(
schema: &arrow_schema::Schema,
vector_column: Option<&str>,
key_column: Option<&str>,
) -> Result<VectorTable> {
let vector_column = match vector_column {
Some(name) => name.to_string(),
None => detect_vector_column(schema)?,
};
let field = schema
.field_with_name(&vector_column)
.with_context(|| format!("no such column {vector_column}"))?;
let dimension = vector_dimension(field.data_type())
.with_context(|| format!("column {vector_column} is not a fixed-size float vector"))?;
let key_column = match key_column {
Some(name) => {
let field = schema
.field_with_name(name)
.with_context(|| format!("no such column {name}"))?;
if field.data_type() != &DataType::Utf8 {
bail!("key column {name} is not Utf8");
}
Some(name.to_string())
}
None => detect_key_column(schema),
};
let metadata_columns = schema
.fields()
.iter()
.map(|f| f.name().clone())
.filter(|name| name != &vector_column && Some(name) != key_column.as_ref())
.collect();
Ok(VectorTable {
key_column,
vector_column,
dimension,
metadata_columns,
})
}
fn detect_vector_column(schema: &arrow_schema::Schema) -> Result<String> {
let mut named = None;
let mut candidates = Vec::new();
for field in schema.fields() {
if vector_dimension(field.data_type()).is_none() {
continue;
}
if field.name() == DEFAULT_VECTOR_COLUMN {
named = Some(field.name().clone());
}
candidates.push(field.name().clone());
}
if let Some(name) = named {
return Ok(name);
}
match candidates.as_slice() {
[only] => Ok(only.clone()),
[] => bail!("no fixed-size float vector column found"),
many => bail!("several vector columns, pick one of: {}", many.join(", ")),
}
}
fn detect_key_column(schema: &arrow_schema::Schema) -> Option<String> {
schema
.field_with_name(DEFAULT_KEY_COLUMN)
.ok()
.filter(|f| f.data_type() == &DataType::Utf8)
.map(|f| f.name().clone())
}
fn vector_dimension(data_type: &DataType) -> Option<usize> {
let DataType::FixedSizeList(child, dimension) = data_type else {
return None;
};
match child.data_type() {
DataType::Float16 | DataType::Float32 | DataType::Float64 => Some(*dimension as usize),
_ => None,
}
}
#[cfg(test)]
mod tests {
use super::*;
use arrow_schema::{Field, Schema};
use std::sync::Arc;
fn vector_field(name: &str, dim: i32) -> Field {
Field::new(
name,
DataType::FixedSizeList(Arc::new(Field::new("item", DataType::Float32, true)), dim),
false,
)
}
#[test]
fn resolve_prefers_the_column_named_vector() {
let schema = Schema::new(vec![
Field::new("key", DataType::Utf8, false),
vector_field("embedding", 4),
vector_field("vector", 8),
Field::new("tag", DataType::Utf8, true),
]);
let table = resolve_schema(&schema, None, None).unwrap();
assert_eq!(table.vector_column, "vector");
assert_eq!(table.dimension, 8);
assert_eq!(table.key_column.as_deref(), Some("key"));
assert_eq!(table.metadata_columns, vec!["embedding", "tag"]);
}
#[test]
fn resolve_uses_the_sole_vector_column() {
let schema = Schema::new(vec![
vector_field("embedding", 4),
Field::new("tag", DataType::Utf8, true),
]);
let table = resolve_schema(&schema, None, None).unwrap();
assert_eq!(table.vector_column, "embedding");
assert_eq!(table.key_column, None);
}
#[test]
fn resolve_rejects_ambiguous_vector_columns() {
let schema = Schema::new(vec![vector_field("a", 4), vector_field("b", 4)]);
assert!(resolve_schema(&schema, None, None).is_err());
}
#[test]
fn resolve_rejects_a_non_vector_column() {
let schema = Schema::new(vec![
Field::new("vector", DataType::Utf8, true),
vector_field("embedding", 4),
]);
assert!(resolve_schema(&schema, Some("vector"), None).is_err());
let table = resolve_schema(&schema, None, Some("vector")).unwrap();
assert_eq!(table.vector_column, "embedding");
assert_eq!(table.key_column.as_deref(), Some("vector"));
}
}
+93
View File
@@ -0,0 +1,93 @@
//! Small Lance datasets for the query and coverage tests.
use std::sync::Arc;
use arrow_array::{
FixedSizeListArray, RecordBatch, RecordBatchIterator, StringArray, types::Float32Type,
};
use arrow_schema::{DataType, Field, Schema};
use lance::Dataset;
use lance::dataset::WriteParams;
use tempfile::TempDir;
pub const DIM: usize = 4;
fn vector_column(vectors: &[Vec<f32>], dim: usize) -> FixedSizeListArray {
FixedSizeListArray::from_iter_primitive::<Float32Type, _, _>(
vectors
.iter()
.map(|v| Some(v.iter().copied().map(Some).collect::<Vec<_>>())),
dim as i32,
)
}
pub fn schema(dim: usize) -> Arc<Schema> {
Arc::new(Schema::new(vec![
Field::new("key", DataType::Utf8, false),
Field::new(
"vector",
DataType::FixedSizeList(
Arc::new(Field::new("item", DataType::Float32, true)),
dim as i32,
),
false,
),
Field::new("tag", DataType::Utf8, true),
]))
}
pub fn batch(offset: usize, vectors: &[Vec<f32>], tags: &[&str], dim: usize) -> RecordBatch {
let keys: Vec<String> = (0..vectors.len())
.map(|i| format!("k{}", offset + i))
.collect();
RecordBatch::try_new(
schema(dim),
vec![
Arc::new(StringArray::from(keys)),
Arc::new(vector_column(vectors, dim)),
Arc::new(StringArray::from(
tags.iter().map(|t| Some(t.to_string())).collect::<Vec<_>>(),
)),
],
)
.unwrap()
}
pub async fn write_dataset(dir: &TempDir, vectors: &[Vec<f32>], tags: &[&str]) -> Dataset {
write_dataset_at(dir.path().to_str().unwrap(), vectors, tags).await
}
pub async fn write_dataset_at(uri: &str, vectors: &[Vec<f32>], tags: &[&str]) -> Dataset {
let dim = vectors[0].len();
let reader = RecordBatchIterator::new(
vec![Ok(batch(0, vectors, tags, dim))].into_iter(),
schema(dim),
);
Dataset::write(reader, uri, None).await.unwrap()
}
/// Deterministic non-trivial vectors for index coverage tests.
pub fn vectors(rows: usize, dim: usize) -> Vec<Vec<f32>> {
(0..rows)
.map(|i| {
(0..dim)
.map(|j| ((i * 31 + j * 17) % 97) as f32 / 97.0)
.collect()
})
.collect()
}
/// Commit a new version whose fragment is not yet covered by the index the
/// dataset had before.
pub async fn append_dataset(dataset: &mut Dataset, vectors: &[Vec<f32>], tags: &[&str]) {
let offset = dataset.count_rows(None).await.unwrap();
let dim = vectors[0].len();
let reader = RecordBatchIterator::new(
vec![Ok(batch(offset, vectors, tags, dim))].into_iter(),
schema(dim),
);
dataset
.append(reader, Some(WriteParams::default()))
.await
.unwrap();
}
+201
View File
@@ -0,0 +1,201 @@
//! The spike's exit gate: the engine's answers must equal what a bare Lance
//! `scan().nearest()` produces on the same pinned version, and no eligible
//! row — indexed or not — may be missing.
use arrow_array::{Float32Array, UInt64Array};
use futures::TryStreamExt;
use lance::Dataset;
use lance::index::DatasetIndexExt;
use lance::index::vector::VectorIndexParams;
use lance_index::IndexType;
use lance_index::optimize::OptimizeOptions;
use lance_linalg::distance::DistanceType;
use tempfile::TempDir;
use crate::coverage::coverage;
use crate::query::{self, Query};
use crate::table::VectorTable;
use crate::testing::{self, DIM};
async fn indexed_dataset(dir: &TempDir) -> Dataset {
let mut dataset = testing::write_dataset(dir, &testing::vectors(64, DIM), &["a"; 64]).await;
let params = VectorIndexParams::ivf_flat(4, DistanceType::L2);
dataset
.create_index(
&["vector"],
IndexType::Vector,
Some("vidx".to_string()),
&params,
true,
)
.await
.unwrap();
dataset.checkout_latest().await.unwrap();
dataset
}
fn query_for(vector: Vec<f32>, k: usize) -> Query {
Query {
vector,
k,
filter: None,
postfilter: false,
distance: DistanceType::L2,
nprobes: Some(4),
}
}
/// The reference path: a bare scanner over the same dataset handle.
async fn reference(dataset: &Dataset, vector: &[f32], k: usize) -> Vec<(u64, f32)> {
let mut scanner = dataset.scan();
scanner
.nearest("vector", &Float32Array::from(vector.to_vec()), k)
.unwrap()
.distance_metric(DistanceType::L2)
.nprobes(4)
.with_row_id();
let mut stream = scanner.try_into_stream().await.unwrap();
let mut rows = Vec::new();
while let Some(batch) = stream.try_next().await.unwrap() {
let ids = batch
.column_by_name("_rowid")
.unwrap()
.as_any()
.downcast_ref::<UInt64Array>()
.unwrap();
let distances = batch
.column_by_name("_distance")
.unwrap()
.as_any()
.downcast_ref::<Float32Array>()
.unwrap();
for i in 0..batch.num_rows() {
rows.push((ids.value(i), distances.value(i)));
}
}
rows
}
async fn assert_matches_reference(dataset: &Dataset, vector: &[f32], k: usize) {
let table = VectorTable::resolve(dataset, None, None).unwrap();
let hits = query::ann(dataset, &table, &query_for(vector.to_vec(), k))
.await
.unwrap();
let expected = reference(dataset, vector, k).await;
assert_eq!(hits.len(), expected.len());
for (hit, (row_id, distance)) in hits.iter().zip(expected.iter()) {
assert_eq!(hit.row_id, *row_id);
assert!((hit.distance - distance).abs() < 1e-6, "{}", hit.key);
}
}
#[tokio::test]
async fn ann_matches_lance_reference() {
let dir = TempDir::new().unwrap();
let dataset = indexed_dataset(&dir).await;
assert_matches_reference(&dataset, &testing::vectors(1, DIM)[0], 10).await;
}
#[tokio::test]
async fn ann_matches_reference_over_unindexed_fragments() {
let dir = TempDir::new().unwrap();
let mut dataset = indexed_dataset(&dir).await;
testing::append_dataset(&mut dataset, &testing::vectors(8, DIM), &["a"; 8]).await;
let cov = &coverage(&dataset, "vector").await.unwrap()[0];
assert_eq!(cov.unindexed_fragments.len(), 1);
assert_eq!(cov.unindexed_rows, 8);
for vector in testing::vectors(4, DIM) {
assert_matches_reference(&dataset, &vector, 10).await;
}
}
#[tokio::test]
async fn ann_matches_reference_across_segments() {
let dir = TempDir::new().unwrap();
let mut dataset = indexed_dataset(&dir).await;
testing::append_dataset(&mut dataset, &testing::vectors(32, DIM), &["a"; 32]).await;
dataset
.optimize_indices(&OptimizeOptions::append())
.await
.unwrap();
let cov = &coverage(&dataset, "vector").await.unwrap()[0];
assert_eq!(cov.segments, 2);
assert!(cov.unindexed_fragments.is_empty());
for vector in testing::vectors(4, DIM) {
assert_matches_reference(&dataset, &vector, 10).await;
}
}
#[tokio::test]
async fn deletes_are_not_returned() {
let dir = TempDir::new().unwrap();
let mut dataset = indexed_dataset(&dir).await;
dataset.delete("key = 'k0'").await.unwrap();
assert_matches_reference(&dataset, &testing::vectors(1, DIM)[0], 10).await;
}
/// Writes a small indexed dataset with unindexed rows to
/// $WEED_VECTOR_SPIKE_DIR for exercising the `weed-vector` CLI:
///
/// WEED_VECTOR_SPIKE_DIR=/tmp/vds cargo test make_spike_dataset -- --ignored
/// weed-vector describe /tmp/vds
/// weed-vector coverage /tmp/vds
/// weed-vector query /tmp/vds --vector 0,0,0,0 --k 5
#[tokio::test]
#[ignore]
async fn make_spike_dataset() {
let dir = std::env::var("WEED_VECTOR_SPIKE_DIR").expect("set WEED_VECTOR_SPIKE_DIR");
let mut dataset = testing::write_dataset_at(&dir, &testing::vectors(64, DIM), &["a"; 64]).await;
let params = VectorIndexParams::ivf_flat(4, DistanceType::L2);
dataset
.create_index(
&["vector"],
IndexType::Vector,
Some("vidx".to_string()),
&params,
true,
)
.await
.unwrap();
dataset.checkout_latest().await.unwrap();
testing::append_dataset(&mut dataset, &testing::vectors(8, DIM), &["b"; 8]).await;
}
#[tokio::test]
async fn query_at_pinned_version_survives_maintenance() {
let dir = TempDir::new().unwrap();
let mut dataset = indexed_dataset(&dir).await;
let pinned = dataset
.checkout_version(dataset.version().version)
.await
.unwrap();
testing::append_dataset(&mut dataset, &testing::vectors(8, DIM), &["b"; 8]).await;
dataset
.optimize_indices(&OptimizeOptions::default())
.await
.unwrap();
assert_eq!(pinned.count_rows(None).await.unwrap(), 64);
let vector = testing::vectors(1, DIM)[0].clone();
let hits = query::ann(
&pinned,
&VectorTable::resolve(&pinned, None, None).unwrap(),
&query_for(vector.clone(), 10),
)
.await
.unwrap();
let expected = reference(&pinned, &vector, 10).await;
assert_eq!(hits.len(), expected.len());
assert!(hits.iter().all(|h| h.row_id < 64));
for (hit, (row_id, distance)) in hits.iter().zip(expected.iter()) {
assert_eq!(hit.row_id, *row_id);
assert!((hit.distance - distance).abs() < 1e-6);
}
assert_matches_reference(&dataset, &vector, 10).await;
}
+42
View File
@@ -0,0 +1,42 @@
[package]
name = "weed-vector-server"
version.workspace = true
edition.workspace = true
rust-version.workspace = true
description = "VectorService gRPC server over SeaweedFS Lance table buckets"
[lib]
name = "weed_vector_server"
[[bin]]
name = "weed-vector-server"
path = "src/main.rs"
[dependencies]
weed-vector-core = { path = "../core" }
weed-lance-worker = { path = "../../../seaweed-worker/crates/lance" }
lance = { version = "10", default-features = false, features = ["aws"] }
lance-linalg = "10"
arrow-array = "58"
arrow-cast = "58"
arrow-schema = "58"
prost = "0.14"
prost-types = "0.14"
reqwest = { version = "0.12", default-features = false }
tonic = { version = "0.14", features = ["tls-aws-lc"] }
tonic-prost = "0.14"
anyhow.workspace = true
clap = { version = "4", features = ["derive", "env"] }
futures = "0.3"
serde = { version = "1", features = ["derive"] }
serde_json = "1"
tokio.workspace = true
tracing.workspace = true
tracing-subscriber.workspace = true
[build-dependencies]
tonic-prost-build = "0.14"
protoc-bin-vendored = "3"
[lints]
workspace = true
+16
View File
@@ -0,0 +1,16 @@
fn main() -> Result<(), Box<dyn std::error::Error>> {
if std::env::var_os("PROTOC").is_none() {
// SAFETY: a build script's main runs single-threaded before anything
// else in this process, so no other thread can be reading the
// environment concurrently.
unsafe {
std::env::set_var("PROTOC", protoc_bin_vendored::protoc_bin_path()?);
}
}
tonic_prost_build::configure()
.build_server(true)
.build_client(true)
.compile_protos(&["../../../weed/pb/vector.proto"], &["../../../weed/pb/"])?;
println!("cargo:rerun-if-changed=../../../weed/pb/vector.proto");
Ok(())
}
+9
View File
@@ -0,0 +1,9 @@
//! The VectorService gRPC server the S3 gateway talks to.
pub mod service;
pub mod pb {
pub mod vector {
tonic::include_proto!("vector_pb");
}
}
+47
View File
@@ -0,0 +1,47 @@
use anyhow::Result;
use clap::Parser;
use tonic::transport::Server;
use weed_lance_worker::catalog::NamespaceClient;
use weed_lance_worker::dataset::FallbackOptions;
use weed_vector_server::pb::vector::vector_service_server::VectorServiceServer;
use weed_vector_server::service::VectorServer;
/// The VectorService the S3 gateway fans vector calls out to.
#[derive(Parser)]
struct Args {
/// Lance Namespace base URL, e.g. http://localhost:8888/lance.
#[arg(
long,
env = "WEED_VECTOR_NAMESPACE",
default_value = "http://localhost:8888/lance"
)]
namespace: String,
/// gRPC listen address.
#[arg(long, env = "WEED_VECTOR_LISTEN", default_value = "127.0.0.1:8080")]
listen: String,
/// object_store option fallback for deployments that vend no credentials:
/// key=value, repeatable.
#[arg(long = "storage-option", value_parser = parse_option)]
storage_options: Vec<(String, String)>,
}
fn parse_option(s: &str) -> Result<(String, String), String> {
s.split_once('=')
.map(|(k, v)| (k.to_string(), v.to_string()))
.ok_or_else(|| "expected key=value".to_string())
}
#[tokio::main]
async fn main() -> Result<()> {
tracing_subscriber::fmt::init();
let args = Args::parse();
let fallback: FallbackOptions = args.storage_options.into_iter().collect();
let client = NamespaceClient::new(args.namespace.clone()).with_fallback(&fallback);
let service = VectorServer::new(client, fallback);
tracing::info!(namespace = %args.namespace, listen = %args.listen, "weed-vector-server");
Server::builder()
.add_service(VectorServiceServer::new(service))
.serve(args.listen.parse()?)
.await?;
Ok(())
}
+615
View File
@@ -0,0 +1,615 @@
use anyhow::{Result, bail};
use arrow_array::{Array, ArrayRef, FixedSizeListArray, Float32Array, RecordBatch, StringArray};
use arrow_schema::{DataType, Field, Schema};
use futures::TryStreamExt;
use lance::Dataset;
use lance::dataset::ProjectionRequest;
use lance::dataset::WriteParams;
use lance::dataset::{MergeInsertBuilder, WhenMatched, WhenNotMatched};
use lance::io::ObjectStoreParams;
use lance::io::StorageOptionsAccessor;
use lance_linalg::distance::DistanceType;
use prost_types::{ListValue, Struct, Value, value::Kind};
use std::collections::BTreeMap;
use std::sync::Arc;
use tonic::{Request, Response, Status};
use weed_lance_worker::catalog::NamespaceClient;
use weed_lance_worker::dataset::{self, FallbackOptions};
use weed_vector_core::query::{self, Query};
use weed_vector_core::table::VectorTable;
use crate::pb::vector::{self as pb, vector_service_server::VectorService};
fn internal(err: impl Into<anyhow::Error>) -> Status {
Status::internal(format!("{:#}", err.into()))
}
fn invalid(err: impl Into<anyhow::Error>) -> Status {
Status::invalid_argument(format!("{:#}", err.into()))
}
fn open_status(err: anyhow::Error) -> Status {
let missing = err.chain().any(|cause| {
cause
.downcast_ref::<reqwest::Error>()
.is_some_and(|e| e.status() == Some(reqwest::StatusCode::NOT_FOUND))
|| cause.downcast_ref::<lance::Error>().is_some_and(|e| {
matches!(
e,
lance::Error::DatasetNotFound { .. }
| lance::Error::NotFound { .. }
| lance::Error::VersionNotFound { .. }
)
})
});
if missing {
Status::not_found(format!("{:#}", err))
} else {
Status::unavailable(format!("{:#}", err))
}
}
const KEY_COLUMN: &str = "key";
const METADATA_COLUMN: &str = "metadata";
const VECTOR_COLUMN: &str = "vector";
fn schema_for(
dimension: usize,
metric: pb::DistanceMetric,
non_filterable: &[String],
) -> Arc<Schema> {
let vector_field = Field::new(
VECTOR_COLUMN,
DataType::FixedSizeList(
Arc::new(Field::new("item", DataType::Float32, false)),
dimension as i32,
),
false,
)
.with_metadata(
[(
"distance_metric".to_string(),
metric.as_str_name().to_string(),
)]
.into_iter()
.collect(),
);
let mut metadata_field = Field::new(METADATA_COLUMN, DataType::Utf8, true);
if !non_filterable.is_empty() {
metadata_field = metadata_field.with_metadata(
[(
"non_filterable_metadata_keys".to_string(),
serde_json::to_string(non_filterable).unwrap_or_default(),
)]
.into_iter()
.collect(),
);
}
Arc::new(Schema::new(vec![
Field::new(KEY_COLUMN, DataType::Utf8, false),
vector_field,
metadata_field,
]))
}
fn batch_of(vectors: &[pb::Vector], dimension: usize, schema: Arc<Schema>) -> Result<RecordBatch> {
let mut seen = std::collections::HashSet::new();
for vector in vectors {
if !seen.insert(vector.key.as_str()) {
bail!("duplicate key in request: {}", vector.key);
}
}
let keys = StringArray::from(vectors.iter().map(|v| v.key.as_str()).collect::<Vec<_>>());
// Check every vector's length before flattening: equal totals would
// still let a short vector borrow from the next one's values.
if vectors.iter().any(|v| v.data.len() != dimension) {
bail!("a vector's data length does not match the index dimension");
}
let flat = vectors
.iter()
.flat_map(|v| v.data.iter().copied())
.collect::<Vec<_>>();
let values: ArrayRef = Arc::new(Float32Array::from(flat));
let vectors_array = FixedSizeListArray::try_new(
Field::new("item", DataType::Float32, false).into(),
dimension as i32,
values,
None,
)?;
let metas = vectors
.iter()
.map(|v| {
v.metadata
.as_ref()
.map(|m| serde_json::to_string(&struct_to_json(m)))
.transpose()
})
.collect::<Result<Vec<_>, _>>()?;
RecordBatch::try_new(
schema,
vec![
Arc::new(keys),
Arc::new(vectors_array),
Arc::new(StringArray::from(metas)),
],
)
.map_err(Into::into)
}
fn struct_to_json(s: &Struct) -> serde_json::Value {
serde_json::Value::Object(
s.fields
.iter()
.map(|(k, v)| (k.clone(), value_to_json(v)))
.collect(),
)
}
fn value_to_json(v: &Value) -> serde_json::Value {
match &v.kind {
None | Some(Kind::NullValue(_)) => serde_json::Value::Null,
Some(Kind::NumberValue(n)) => serde_json::json!(n),
Some(Kind::StringValue(s)) => serde_json::json!(s),
Some(Kind::BoolValue(b)) => serde_json::json!(b),
Some(Kind::StructValue(s)) => struct_to_json(s),
Some(Kind::ListValue(l)) => {
serde_json::Value::Array(l.values.iter().map(value_to_json).collect())
}
}
}
fn json_to_value(v: serde_json::Value) -> Value {
let kind = match v {
serde_json::Value::Null => Kind::NullValue(0),
serde_json::Value::Bool(b) => Kind::BoolValue(b),
serde_json::Value::Number(n) => Kind::NumberValue(n.as_f64().unwrap_or(0.0)),
serde_json::Value::String(s) => Kind::StringValue(s),
serde_json::Value::Array(items) => Kind::ListValue(ListValue {
values: items.into_iter().map(json_to_value).collect(),
}),
serde_json::Value::Object(map) => Kind::StructValue(json_to_struct(map)),
};
Value { kind: Some(kind) }
}
fn json_to_struct(map: serde_json::Map<String, serde_json::Value>) -> Struct {
Struct {
fields: map
.into_iter()
.map(|(k, v)| (k, json_to_value(v)))
.collect::<BTreeMap<_, _>>(),
}
}
fn parse_struct(s: &str) -> Result<Struct> {
let v: serde_json::Value = serde_json::from_str(s)?;
match v {
serde_json::Value::Object(map) => Ok(json_to_struct(map)),
_ => bail!("metadata is not a JSON object"),
}
}
fn key_list(keys: &[String]) -> String {
format!(
"`{KEY_COLUMN}` in ({})",
keys.iter()
.map(|k| format!("'{}'", k.replace('\'', "''")))
.collect::<Vec<_>>()
.join(", ")
)
}
fn rows_as_vectors(
batches: &[RecordBatch],
return_data: bool,
return_metadata: bool,
) -> Vec<pb::Vector> {
let mut out = Vec::new();
for batch in batches {
let Some(keys) = batch
.column_by_name(KEY_COLUMN)
.and_then(|c| c.as_any().downcast_ref::<StringArray>())
else {
continue;
};
let vectors = batch
.column_by_name(VECTOR_COLUMN)
.and_then(|c| c.as_any().downcast_ref::<FixedSizeListArray>());
let metas = batch
.column_by_name(METADATA_COLUMN)
.and_then(|c| c.as_any().downcast_ref::<StringArray>());
for i in 0..batch.num_rows() {
let data = match (return_data, vectors) {
(true, Some(list)) => list
.value(i)
.as_any()
.downcast_ref::<Float32Array>()
.map(|a| a.values().to_vec())
.unwrap_or_default(),
_ => Vec::new(),
};
let metadata = match (return_metadata, metas) {
(true, Some(m)) if !m.is_null(i) => parse_struct(m.value(i)).ok(),
_ => None,
};
out.push(pb::Vector {
key: keys.value(i).to_string(),
data,
metadata,
});
}
}
out
}
pub struct VectorServer {
client: NamespaceClient,
fallback: FallbackOptions,
}
impl VectorServer {
pub fn new(client: NamespaceClient, fallback: FallbackOptions) -> Self {
Self { client, fallback }
}
async fn open(&self, table: &pb::Table) -> Result<Dataset> {
let opened = dataset::open(&self.client, &table.id, &self.fallback).await?;
let mut dataset = opened.dataset;
if table.version != 0 {
dataset = dataset.checkout_version(table.version).await?;
}
Ok(dataset)
}
}
#[tonic::async_trait]
impl VectorService for VectorServer {
async fn create_index(
&self,
request: Request<pb::CreateIndexRequest>,
) -> Result<Response<pb::CreateIndexResponse>, Status> {
let req = request.into_inner();
let metric = req.distance_metric();
let table = req
.table
.ok_or_else(|| Status::invalid_argument("missing table"))?;
let dimension = req.dimension as usize;
// S3 Vectors caps at 4096; reject beyond it instead of letting an
// oversized request build a huge Arrow schema.
if dimension == 0 || dimension > 4096 {
return Err(Status::invalid_argument("dimension must be 1..=4096"));
}
let description = self
.client
.declare_table(&table.id, None)
.await
.map_err(internal)?;
let mut options = self.fallback.clone();
options.extend(description.storage_options.clone());
let schema = schema_for(dimension, metric, &req.non_filterable_metadata_keys);
let empty = RecordBatch::new_empty(schema.clone());
let reader = arrow_array::RecordBatchIterator::new(vec![Ok(empty)].into_iter(), schema);
let dataset = Dataset::write(
reader,
&description.location,
Some(WriteParams {
store_params: Some(ObjectStoreParams {
storage_options_accessor: Some(Arc::new(
StorageOptionsAccessor::with_static_options(options),
)),
..Default::default()
}),
..Default::default()
}),
)
.await
.map_err(internal)?;
Ok(Response::new(pb::CreateIndexResponse {
version: dataset.version().version,
}))
}
async fn drop_index(
&self,
_request: Request<pb::DropIndexRequest>,
) -> Result<Response<pb::DropIndexResponse>, Status> {
// Dropping is a catalog decision; the gateway deletes the table record.
Err(Status::unimplemented("drop through the catalog"))
}
async fn put_vectors(
&self,
request: Request<pb::PutVectorsRequest>,
) -> Result<Response<pb::PutVectorsResponse>, Status> {
let req = request.into_inner();
let table = req
.table
.ok_or_else(|| Status::invalid_argument("missing table"))?;
let dataset = self.open(&table).await.map_err(open_status)?;
if req.vectors.is_empty() {
return Ok(Response::new(pb::PutVectorsResponse {
version: dataset.version().version,
}));
}
let dimension = VectorTable::resolve(&dataset, None, Some(KEY_COLUMN))
.map_err(invalid)?
.dimension;
let batch = batch_of(
&req.vectors,
dimension,
Arc::new(Into::<arrow_schema::Schema>::into(dataset.schema())),
)
.map_err(invalid)?;
// Upsert by key in one merge commit: no window where the old rows are
// already deleted, and a losing writer fails the request instead of
// leaving duplicate keys.
let mut builder =
MergeInsertBuilder::try_new(Arc::new(dataset), vec![KEY_COLUMN.to_string()])
.map_err(internal)?;
builder.when_matched(WhenMatched::UpdateAll);
builder.when_not_matched(WhenNotMatched::InsertAll);
let (dataset, _stats) = builder
.try_build()
.map_err(internal)?
.execute_batches(vec![batch])
.await
.map_err(internal)?;
Ok(Response::new(pb::PutVectorsResponse {
version: dataset.version().version,
}))
}
async fn get_vectors(
&self,
request: Request<pb::GetVectorsRequest>,
) -> Result<Response<pb::GetVectorsResponse>, Status> {
let req = request.into_inner();
let table = req
.table
.ok_or_else(|| Status::invalid_argument("missing table"))?;
let dataset = self.open(&table).await.map_err(open_status)?;
let resolved = VectorTable::resolve(&dataset, None, Some(KEY_COLUMN)).map_err(invalid)?;
let batches = query::get(&dataset, &resolved, &req.keys)
.await
.map_err(internal)?;
Ok(Response::new(pb::GetVectorsResponse {
vectors: rows_as_vectors(&batches, req.return_data, req.return_metadata),
}))
}
async fn delete_vectors(
&self,
request: Request<pb::DeleteVectorsRequest>,
) -> Result<Response<pb::DeleteVectorsResponse>, Status> {
let req = request.into_inner();
let table = req
.table
.ok_or_else(|| Status::invalid_argument("missing table"))?;
let mut dataset = self.open(&table).await.map_err(open_status)?;
if !req.keys.is_empty() {
dataset
.delete(&key_list(&req.keys))
.await
.map_err(internal)?;
}
Ok(Response::new(pb::DeleteVectorsResponse {
version: dataset.version().version,
}))
}
async fn list_vectors(
&self,
request: Request<pb::ListVectorsRequest>,
) -> Result<Response<pb::ListVectorsResponse>, Status> {
let req = request.into_inner();
let table = req
.table
.ok_or_else(|| Status::invalid_argument("missing table"))?;
let dataset = self.open(&table).await.map_err(open_status)?;
let offset: u64 = if req.next_token.is_empty() {
0
} else {
req.next_token
.parse()
.map_err(|_| Status::invalid_argument("invalid next_token"))?
};
let mut columns = vec![KEY_COLUMN.to_string()];
if req.return_data {
columns.push(VECTOR_COLUMN.to_string());
}
if req.return_metadata {
columns.push(METADATA_COLUMN.to_string());
}
// An unset max_results arrives as 0; page it like S3 Vectors does.
const DEFAULT_PAGE: u32 = 500;
const MAX_PAGE: u32 = 1000;
let page = match req.max_results {
0 => DEFAULT_PAGE,
n => n.min(MAX_PAGE),
};
let mut scanner = dataset.scan();
scanner
.project(&columns)
.map_err(internal)?
.limit(Some(page as i64), Some(offset as i64))
.map_err(internal)?;
let batches: Vec<RecordBatch> = scanner
.try_into_stream()
.await
.map_err(internal)?
.try_collect()
.await
.map_err(internal)?;
let rows = batches.iter().map(|b| b.num_rows()).sum::<usize>();
let next = offset + rows as u64;
let next_token = if rows == page as usize && rows > 0 {
next.to_string()
} else {
String::new()
};
Ok(Response::new(pb::ListVectorsResponse {
vectors: rows_as_vectors(&batches, req.return_data, req.return_metadata),
next_token,
}))
}
async fn query_vectors(
&self,
request: Request<pb::QueryVectorsRequest>,
) -> Result<Response<pb::QueryVectorsResponse>, Status> {
let req = request.into_inner();
let table = req
.table
.ok_or_else(|| Status::invalid_argument("missing table"))?;
let dataset = self.open(&table).await.map_err(open_status)?;
let resolved = VectorTable::resolve(&dataset, None, Some(KEY_COLUMN)).map_err(invalid)?;
if req.query_vector.len() != resolved.dimension {
return Err(Status::invalid_argument(format!(
"query vector carries {} floats, the index dimension is {}",
req.query_vector.len(),
resolved.dimension
)));
}
if req.top_k == 0 {
return Err(Status::invalid_argument("top_k must be positive"));
}
// The index's configured metric lives in the vector field's metadata.
let distance = match dataset
.schema()
.field(VECTOR_COLUMN)
.and_then(|f| f.metadata.get("distance_metric").map(String::as_str))
{
Some("COSINE") => DistanceType::Cosine,
Some("DOT_PRODUCT") => DistanceType::Dot,
_ => DistanceType::L2,
};
let query = Query {
vector: req.query_vector,
k: req.top_k as usize,
filter: if req.filter.is_empty() {
None
} else {
Some(req.filter)
},
postfilter: req.postfilter,
distance,
nprobes: None,
};
let hits = query::ann(&dataset, &resolved, &query)
.await
.map_err(internal)?;
// One batched read for every hit's columns; keyed by the hit key so
// row order does not matter.
let mut metas: std::collections::HashMap<String, Option<Struct>> =
std::collections::HashMap::new();
let mut data_map: std::collections::HashMap<String, Vec<f32>> =
std::collections::HashMap::new();
if (req.return_metadata || req.return_data) && !hits.is_empty() {
let row_ids: Vec<u64> = hits.iter().map(|h| h.row_id).collect();
let mut columns = vec![KEY_COLUMN];
if req.return_metadata {
columns.push(METADATA_COLUMN);
}
if req.return_data {
columns.push(VECTOR_COLUMN);
}
let batch = dataset
.take_rows(
&row_ids,
ProjectionRequest::from_columns(columns, dataset.schema()),
)
.await
.map_err(internal)?;
let keys = batch
.column_by_name(KEY_COLUMN)
.and_then(|c| c.as_any().downcast_ref::<StringArray>());
let meta_col = batch
.column_by_name(METADATA_COLUMN)
.and_then(|c| c.as_any().downcast_ref::<StringArray>());
let data_col = batch
.column_by_name(VECTOR_COLUMN)
.and_then(|c| c.as_any().downcast_ref::<FixedSizeListArray>());
if let Some(keys) = keys {
for i in 0..batch.num_rows() {
let key = keys.value(i).to_string();
if let Some(meta_col) = meta_col {
metas.insert(
key.clone(),
if meta_col.is_null(i) {
None
} else {
parse_struct(meta_col.value(i)).ok()
},
);
}
if let Some(data_col) = data_col {
data_map.insert(
key,
data_col
.value(i)
.as_any()
.downcast_ref::<Float32Array>()
.map(|a| a.values().to_vec())
.unwrap_or_default(),
);
}
}
}
}
let mut vectors = Vec::with_capacity(hits.len());
for hit in &hits {
vectors.push(pb::QueryResult {
key: hit.key.clone(),
distance: if req.return_distance {
hit.distance
} else {
0.0
},
metadata: metas.remove(&hit.key).flatten(),
data: data_map.remove(&hit.key).unwrap_or_default(),
});
}
Ok(Response::new(pb::QueryVectorsResponse { vectors }))
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn batch_of_rejects_a_ragged_vector() {
let vectors = vec![
pb::Vector {
key: "a".to_string(),
data: vec![1.0, 2.0],
metadata: None,
},
pb::Vector {
key: "b".to_string(),
data: vec![1.0; 4],
metadata: None,
},
];
let schema = schema_for(4, pb::DistanceMetric::L2, &[]);
assert!(batch_of(&vectors, 4, schema).is_err());
}
#[test]
fn batch_of_rejects_a_repeated_key() {
let schema = schema_for(4, pb::DistanceMetric::L2, &[]);
let vectors = vec![
pb::Vector {
key: "a".to_string(),
data: vec![1.0; 4],
metadata: None,
},
pb::Vector {
key: "a".to_string(),
data: vec![2.0; 4],
metadata: None,
},
];
assert!(batch_of(&vectors, 4, schema).is_err());
}
}
+41 -15
View File
@@ -86,7 +86,7 @@ version = "1.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc"
dependencies = [
"windows-sys 0.61.2",
"windows-sys 0.60.2",
]
[[package]]
@@ -97,7 +97,7 @@ checksum = "291e6a250ff86cd4a820112fb8898808a366d8f9f58ce16d1f538353ad55747d"
dependencies = [
"anstyle",
"once_cell_polyfill",
"windows-sys 0.61.2",
"windows-sys 0.60.2",
]
[[package]]
@@ -510,10 +510,12 @@ dependencies = [
[[package]]
name = "aws-smithy-json"
version = "0.62.5"
version = "0.62.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9648b0bb82a2eedd844052c6ad2a1a822d1f8e3adee5fbf668366717e428856a"
checksum = "701a947f4797e52a911e114a898667c746c39feea467bbd1abd7b3721f702ffa"
dependencies = [
"aws-smithy-runtime-api",
"aws-smithy-schema",
"aws-smithy-types",
]
@@ -563,11 +565,12 @@ dependencies = [
[[package]]
name = "aws-smithy-runtime-api"
version = "1.11.6"
version = "1.12.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "876ab3c9c29791ba4ba02b780a3049e21ec63dabda09268b175272c3733a79e6"
checksum = "9db177daa6ba8afb9ee1aefcf548c907abcf52065e394ee11a92780057fe0e8c"
dependencies = [
"aws-smithy-async",
"aws-smithy-runtime-api-macros",
"aws-smithy-types",
"bytes",
"http 0.2.12",
@@ -579,10 +582,32 @@ dependencies = [
]
[[package]]
name = "aws-smithy-types"
version = "1.4.7"
name = "aws-smithy-runtime-api-macros"
version = "1.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9d73dbfbaa8e4bc57b9045137680b958d274823509a360abfd8e1d514d40c95c"
checksum = "8d7396fd9500589e62e460e987ecb671bad374934e55ec3b5f498cc7a8a8a7b7"
dependencies = [
"proc-macro2",
"quote",
"syn",
]
[[package]]
name = "aws-smithy-schema"
version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7442cb268338f0eb8278140a107c046756aa01093d8ef5e99628d34ae09c94f5"
dependencies = [
"aws-smithy-runtime-api",
"aws-smithy-types",
"http 1.4.0",
]
[[package]]
name = "aws-smithy-types"
version = "1.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "32b42fcf341259d85ca10fac9a2f6448a8ec691c6955a18e45bc3b71a85fab85"
dependencies = [
"base64-simd",
"bytes",
@@ -1325,7 +1350,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb"
dependencies = [
"libc",
"windows-sys 0.61.2",
"windows-sys 0.52.0",
]
[[package]]
@@ -2379,7 +2404,7 @@ version = "0.50.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5"
dependencies = [
"windows-sys 0.61.2",
"windows-sys 0.59.0",
]
[[package]]
@@ -3391,7 +3416,7 @@ dependencies = [
"errno 0.3.14",
"libc",
"linux-raw-sys 0.12.1",
"windows-sys 0.61.2",
"windows-sys 0.52.0",
]
[[package]]
@@ -3676,7 +3701,7 @@ version = "1.4.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c4db69cba1110affc0e9f7bcd48bbf87b3f4fc7c61fc9155afd4c469eb3d6c1b"
dependencies = [
"errno 0.3.14",
"errno 0.2.8",
"libc",
]
@@ -3743,7 +3768,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3a766e1110788c36f4fa1c2b71b387a7815aa65f88ce0229841826633d93723e"
dependencies = [
"libc",
"windows-sys 0.61.2",
"windows-sys 0.60.2",
]
[[package]]
@@ -3877,7 +3902,7 @@ dependencies = [
"getrandom 0.4.2",
"once_cell",
"rustix 1.1.4",
"windows-sys 0.61.2",
"windows-sys 0.52.0",
]
[[package]]
@@ -4576,6 +4601,7 @@ dependencies = [
"futures",
"hex",
"http-body 1.0.1",
"httpdate",
"hyper",
"hyper-util",
"image",
Loaded 100 of 664 files, more files were not shown because too many files have changed in this diff. Show more