Compare commits

...
Author SHA1 Message Date
Devin 9d84ecca2f upload -dir: finish every file before reporting a failed upload
A failed batch stopped all remaining batches, and the file-list path
returned before printing results, hiding ids of files that did upload.
Now every file is still attempted, results print for each batch, and
the command fails at the end if any upload failed.
2026-10-21 15:20:40 -07:00
Devin a1d2ebbb9b operation: warm lazy log fds in the file-part leak test 2026-10-21 15:15:01 -07:00
Devin 371d320854 operation: close open file parts when a batch fails to build
Batched -dir uploads now open up to 32 files at once; a mid-batch error
left every earlier handle open until process exit. Close the parts
already built, and close the handle when Stat fails after Open.
2026-10-21 15:14:12 -07:00
Chris LuandDevin 84cc179b80 upload -dir: fetch file ids in batches instead of per file
Directory upload submitted every file through its own SubmitFiles
call, so each file paid a separate serial assign RPC. Files are now
collected during the walk and submitted in batches, sharing one batch
assign and the parallel uploads.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 15:10:30 -07:00
Chris LuandDevin c8002d63bb upload: submit batched files concurrently
SubmitFiles fetched all file ids in one assign RPC but then uploaded
the files one after another, so a multi-file upload spent its time in
serial per-file round trips. Uploads now run through a small worker
pool while results keep their input order.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 15:10:30 -07:00
Junker der Provinz 9b642d4636 volume: use FALLOC_FL_KEEP_SIZE and log preallocation failures (#11920) 2026-10-21 15:07:55 -07:00
Javier de la TorreandJavier de la Torre a882565cd7 iceberg: keep a default-CRS geometry column without a CRS after compaction (#11899)
* iceberg: keep a default-CRS geometry column without a CRS after compaction

Writers such as DuckDB write a GEOMETRY column whose Iceberg type has the
default CRS (OGC:CRS84) with no CRS in the Parquet logical type. parquet-go
spells the default out when it writes such a column (geometryType's
LogicalType), even when the column it read declared none, so every
compacted file declared GEOMETRY(crs=OGC:CRS84) where its inputs declared
GEOMETRY. Both mean the same to the Parquet spec, but a reader may treat the
file column as another type than the table column: DuckDB casts it to the
table's GEOMETRY, and a cast column is no longer pruned by its row-group
statistics, so a spatial filter read every row group of a compacted file.

After a bin is merged, the footer's schema elements of every GEOMETRY
column whose Iceberg type has the default CRS (top-level or nested) are
rewritten to declare no CRS. The data, the page indexes and every offset
stay byte for byte; only the footer is re-encoded. An explicit CRS is left
as it is. Retyping the writer's column instead does not work: parquet-go
refuses to convert between two GEOMETRY types that both mean the default,
and its sorting writer converts.

Tests: geometry_crs_test.go, a bin-packed and a sorted compaction of
default-CRS inputs (one of them spelling the default out) write no CRS,
with a control showing parquet-go spells it out on its own; an explicit CRS
is kept; a nested geometry. On master the binpack and sort cases fail with
CRS "OGC:CRS84".

* iceberg: compress the row lineage columns of compacted v3 files

A v3 compaction appends _row_id and _last_updated_sequence_number to the
merged file. The two leaves were built without a codec, so they were
written uncompressed, 16 bytes a row, whatever codec the data columns use;
on a table of points the compacted files grew by about a third.

The lineage leaves now take the codec of the reference file's first leaf
that declares one, ZSTD when none does.

Test: TestCompactCompressesLineageColumns; on master both columns are
written UNCOMPRESSED next to ZSTD data columns.

* iceberg: match list and map geometry by Iceberg name in files without field ids

A Parquet footer path runs through the repeated group of a LIST or MAP
(shapes.list.element, attrs.key_value.value), which Iceberg leaves out of
its field names (shapes.element, attrs.value). The name lookup for a file
without field ids therefore missed every geometry inside a list or a map
and left the default CRS spelled out. Name the paths as Iceberg does,
including the legacy two-level list forms.

* iceberg: rewrite the geometry CRS footer inside the merged file's buffer

Removing a CRS shortens the footer, so the new one fits where the old one
was. Copying the whole merged file to replace a few hundred bytes of
footer doubled the memory of a compaction near the target file size.

* iceberg: take the lineage codec from the data columns, not an input's _row_id

An input that already carries _row_id may list it before its data
columns, and write it uncompressed; the new lineage columns then copied
that codec instead of the data columns'. Choose from the columns the
layout keeps.

---------

Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
2026-10-21 14:50:45 -07:00
Javier de la TorreandJavier de la Torre de2fc03c7d s3tables: accept the external-volume verification object at a table bucket's root (#11900)
* s3tables: accept the external-volume verification object at a table bucket's root

Snowflake verifies an S3-compatible external volume before it creates or
writes Iceberg tables in it (SYSTEM$VERIFY_EXTERNAL_VOLUME, also run
implicitly): it writes, reads, lists and deletes one small object,
<base>/verify_<epoch milliseconds>_<number>, a 21-byte body. A table bucket
refuses every key outside the Iceberg layout, so the check failed and
Snowflake could not use a table bucket as an external volume.

PUT, GET, HEAD, DELETE and DeleteObjects now accept exactly that key in a
table bucket, narrowly enough that it cannot be used to store data:
- at the bucket root only, so nothing is encoded in a prefix, written into
  a hidden tree or placed on a namespace or table name;
- one shape: verify_<13 digits of epoch ms within one hour of the server's
  clock>_<1-10 digits>;
- a PUT must declare a Content-Length of at most 1 KiB (no chunked or
  undeclared body) and carry no user metadata, tagging, Content-Disposition,
  Cache-Control, Expires or website redirect.
Multipart, copy, rename, ACL, tagging and POST policy writes keep the plain
layout rule. Buckets that are not table buckets are unaffected.

Tests: table_bucket_verify_object_test.go, the key accepted (with the
Iceberg layout as the positive control and the plain rule still refusing
it), 15 look-alikes refused (prefixed, hidden tree, wrong shape, outside the
window, ordinary keys), and seven PUT refusals through PutObjectHandler.

* s3tables: keep the verification key off namespaces and narrow its PUT

A namespace may be named like a verification key (verify_<ms>_<n>), and
the key then names the namespace's directory. A PUT replaced that
directory with a prefix object, dropping the namespace's catalog
metadata; a DELETE removed an empty namespace. The key now gets the plain
layout rule when it names a directory or its entry cannot be read, and a
PUT of it never falls back to a prefix object, so a namespace created
after the check is not replaced either.

The time window applies to PUT only: GET, HEAD and DELETE take a key of
any date, so an object a check left behind for more than an hour can
still be read and removed. A verification PUT also refuses an extraction
request and the stored Content-Language, Response-Content-Disposition and
Content-Encoding headers (aws-chunked framing alone is still accepted).

The integration test runs the check's PUT, HEAD, GET, list and DELETE
against a table bucket, and the PUT and DELETE over a namespace of the
key's shape.

---------

Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
2026-10-21 14:49:36 -07:00
Javier de la TorreandJavier de la Torre 21add33b6e s3 backend: optional HTTP client settings for the cloud tier (#11897)
* s3 backend: optional HTTP client settings for the cloud tier

The S3 tier backend builds its client from a bare aws.Config, so it always
uses Go's default transport: HTTP/2 wherever the endpoint offers it over TLS,
no cap on connections per host, and a 90s idle timeout. Against some
S3-compatible services that is a poor fit. Over HTTP/2 every upload and
range read of a tiered volume shares one connection and its flow-control
window; with a small window, tier uploads and concurrent reads of tiered
volumes become throughput-bound on that one connection, while HTTP/1.1 opens
a connection per concurrent request. Some services also limit parallel
connections per client address, or drop idle connections before 90s.

Add three optional settings to [storage.backend.s3.<id>], all off by
default so existing deployments keep the SDK's default client:
- disable_http2: keep the client on HTTP/1.1 (ForceAttemptHTTP2=false and an
  empty TLSNextProto, the documented opt-out);
- max_connections: MaxConnsPerHost, 0 = no cap;
- idle_connection_timeout_seconds: IdleConnTimeout, 0 = Go's default.

The session cache key includes the options when any is set, so two backends
on the same endpoint with different settings do not share a client.
ToProperties reports the settings; the master.toml scaffold documents them.

Tests: TestS3BackendHTTPOptionsReachTheClient fails without the change (the
settings never reach the client); TestS3BackendDefaultHTTPClientUnchanged,
TestS3BackendSessionsDifferByHTTPOptions, TestParseNonNegative.

* s3 backend: reject an idle timeout too large for time.Duration

An idle_connection_timeout_seconds above math.MaxInt64/1e9 overflowed when
multiplied by time.Second. The negative result made the transport keep its
default timeout and ToProperties export a negative value. Warn and use the
default instead.

* s3 backend: parse the idle timeout as 64-bit seconds

parseIdleConnTimeout went through strconv.Atoi, so on 32-bit platforms
any value above 2147483647 seconds was rejected while the same value was
accepted on 64-bit, and the overflow test failed under GOARCH=386. Parse
with ParseInt(…, 64) and export the effective value with FormatInt so the
accepted range is the same on every platform.

---------

Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
2026-10-21 14:44:39 -07:00
Chris LuandDevin 9a03c09396 s3api: cover the inline-content UploadPartCopy route at handler level (#11914)
s3api: exercise the inline-content route through CopyObjectPartHandler

Generated with [Devin](https://devin.ai)

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 14:15:04 -07:00
Javier de la TorreandJavier de la Torre f40267db44 s3: answer InvalidBucketName to a denied request on a malformed bucket name (#11893)
* s3: answer InvalidBucketName to a denied request on a malformed bucket name

The store authorizes a request before it looks at the bucket, so a caller
denied on a bucket whose name breaks the S3 naming rules (upper case, an
underscore, too short, two adjacent periods, a non-ASCII letter) gets 403
AccessDenied, where AWS S3 answers 400 InvalidBucketName. Clients and
compatibility suites that check the error code read the 403 as a permission
problem.

Auth and AuthPostPolicy now turn an AccessDenied for a resolved identity into
InvalidBucketName when the bucket name fails VerifyS3BucketName or uses a
character outside a-z, 0-9, '.' and '-' (VerifyS3BucketName checks with
unicode.IsLower/IsNumber, which accept letters such as 'ü'). The answer depends
on the name alone, so it reveals nothing about which buckets exist. An
authentication failure, an allowed request and a denial on a well-formed name
answer as before.

Test: TestAuthDeniedInvalidBucketName drives iam.Auth with signed and anonymous
requests; on master every malformed name answers 403, with the change 400,
while the identity's own bucket stays allowed and a well-formed foreign name
stays 403. TestRefineDenialForBucketName covers the rule table.

* s3: refine POST-policy and deferred anonymous-read denials on a malformed bucket name

Two denial paths bypass the refinement in Auth and AuthPostPolicy:

- A browser POST upload carries its credentials in the form, so
  AuthPostPolicy delegates it to PostPolicyBucketHandler, which verifies the
  form and writes AccessDenied itself when the signer may not write the bucket
  or the per-key authorization denies the object. Both denials now go through
  refineDenialForBucketName with the verified identity.

  The coarse Write check used to run inside doesPolicySignatureV2Match and
  doesPolicySignatureV4Match before the signature was compared, so a form with
  a wrong signature was told whether the key may write the bucket. It now runs
  in doesPolicySignatureMatch after the signature verifies, as on the
  header-signed path; a form that fails verification answers
  SignatureDoesNotMatch (or MissingFields) and is never refined.

- Unsigned GET/HEAD object reads are deferred by AuthWithPublicRead to the
  object handlers, where writeAnonymousAwareNotFound masks a miss as
  AccessDenied. That denial is refined with the anonymous identity, so a
  missing malformed bucket answers InvalidBucketName when an anonymous
  identity is configured, as Auth does. A denial on an entry that was found
  is left alone: the bucket exists.

Test: TestPostPolicyDeniedInvalidBucketName drives AuthPostPolicy with signed
forms (Write denied, per-key policy deny, bad signature, well-formed name);
TestAnonymousReadDeniedInvalidBucketName drives AuthWithPublicRead with GET and
HEAD, with and without an anonymous identity. Both fail before this change.

* s3: refine a denied anonymous read of an existing key on a malformed bucket name

A deferred anonymous GET/HEAD refined only its masked miss to
InvalidBucketName, so in a bucket whose name breaks the S3 rules (one
VerifyS3BucketName still accepts, such as a non-ASCII name) a missing key
answered 400 and an existing private key 403, telling the caller which
keys exist. The entry check now refines its denial the same way, so both
answer alike. Without an anonymous identity no identity resolves and both
stay 403, as in Auth.

The deny-precedence case in TestAnonymousObjectACLConfiguredIdentity runs
on the fixture bucket "b", shorter than S3's three-character minimum, so
its denial now answers InvalidBucketName.

---------

Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
2026-10-21 14:08:33 -07:00
Chris LuandDevin 6ee7c4c5c7 s3: no empty-directory marker in a "<dir>/" self-prefix listing (#11906)
An out-of-band empty directory listed under its own trailing-slash
prefix answered the probe with a fake entry of itself: a Contents key on
4.48 and, since recently, a CommonPrefix. That listing enumerates
children, where S3 returns nothing for an empty prefix, and clients that
read every returned key (Pulumi lock listing) choked on the marker.

The prefix object and directory key object paths are unchanged: a real
"dir/" object still lists as its own key.

Generated with [Devin](https://devin.ai)

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 14:03:02 -07:00
9c427b265f volume tier: accept deletes on a volume as soon as it is tiered (#11894)
* volume tier: accept deletes on a volume as soon as it is tiered

volume.tier.upload marks the volume read-only without deletes before the
upload (markVolumeReplicasWritable(..., false, false), i.e. noWriteOrDelete),
and nothing clears that until the volume server restarts, when load() gives
every tiered volume noWriteCanDelete. Every delete in between is refused with
"volume N is read only": the needle stays live in the index, and its bytes in
the remote object, until someone notices.

After the remote file is loaded, the upload handler now gives the volume the
flags a restart would (MarkTieredDeletable: writes refused, deletes accepted),
with two exceptions that keep the current behaviour:
- a replicated volume: the shell copies the uploaded replica's .idx and .vif
  to the other replicas after the upload, and a tombstone written before that
  copy would reach one replica only;
- a volume in I/O quarantine: that flag is not the upload's to lift.

Tests: TestTierUploadAcceptsDeletesOnASingleCopyVolume fails without the
change ("volume 1 is read only") and checks writes stay refused;
TestTierUploadKeepsRefusingDeletesOnAReplicatedVolume; TestMarkTieredDeletable
(quarantine kept).

* volume tier: check the I/O quarantine under noWriteLock

MarkTieredDeletable read the quarantine state before taking noWriteLock.
CollectHeartbeat could quarantine the volume and set noWriteOrDelete in
between, and the upload then cleared that refusal, so a quarantined volume
accepted deletes until the next heartbeat. Take noWriteLock first: the
heartbeat marks the quarantine before it takes noWriteLock, so either the
check sees it or the heartbeat's refusal lands afterwards and stays.

Also leave alone a volume past IoErrorTolerance that the next heartbeat
will quarantine, as the heartbeat itself does.

* volume: read noWriteOrDelete under noWriteLock in DeleteVolumeNeedle

DeleteVolumeNeedle checked v.noWriteOrDelete without the lock that
MarkVolumeReadonly, CollectHeartbeat and now MarkTieredDeletable hold when
they write it, so a delete racing a tier upload was a data race. Copy the
flag under the read lock; the delete path is otherwise unchanged.

* volume tier: mirror the delete acceptance rules in the Rust volume server

The Rust tier upload already promoted a freshly tiered volume to
no-write-can-delete via refresh_remote_write_mode, and did so for every
volume -- including the replicated and I/O-quarantined ones the Go
MarkTieredDeletable deliberately refuses. A live tier-up now goes
through mark_tiered_deletable (replicated volumes and quarantines keep
their upload-time mark until the next load), while load() keeps
promoting every tiered volume unconditionally, as Go does.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: keep a tiered volume delete-only when marked writable

MarkVolumeWritable preserved noWriteCanDelete on a remote-tiered volume
only when it was already set. A tiered volume still holding the
upload-time mark (a replicated volume, or one whose tier upload ran
before MarkTieredDeletable existed) had it clear, so marking writable
published the volume as fully writable while its .dat lives remotely --
writes then failed against a backend that cannot append, after the
master was told the volume is writable again.

load() gives every remote-tiered volume noWriteCanDelete; deriving the
same flag here keeps mark-writable consistent with a restart: "writable"
on a remote volume resolves to can-delete.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: keep a tiered volume delete-only when marked writable (rust)

set_writable preserved the remote delete-only mode only when the flag was
already set, so a tiered volume holding the upload-time read-only mark
(replicated, or quarantined under mark_tiered_deletable) was published
fully writable with its .dat remote. Derive no_write_can_delete from
has_remote_file() instead, matching load() and the Go MarkVolumeWritable.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume tier: serialize the quarantine check with the flag swap (rust)

A heartbeat refusal landing between should_quarantine() and the store
could be clobbered until the next heartbeat re-asserted it — the same
race the Go side closed by checking under noWriteLock. A
compare-exchange gives the same ordering: the swap fails, and the retry
sees the quarantine mark the heartbeat sets first.

* volume tier: MarkTieredDeletable refuses a non-tiered volume

A stray call on a local single-copy volume would pin it
read-only-can-delete. The only caller promotes right after
LoadRemoteFile, but the guard makes the method safe to reuse; the Rust
mark_tiered_deletable gets the same check, though its caller already
only reaches it under has_remote.

---------

Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
Co-authored-by: chrislusf <chris.lu@gmail.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 14:01:10 -07:00
Javier de la TorreandJavier de la Torre 97f8ca13ff s3tables, iceberg: log client refusals at V(1), not ERROR (#11896)
* s3tables, iceberg: log client refusals at V(1), not ERROR

Requests the S3 Tables and Iceberg REST APIs correctly refuse with a 4xx
(access denied, not authorized, no such namespace, already exists, not
empty, a commit conflict) were logged with glog.Errorf, so a client probing
for what it may do, or retrying an optimistic commit, filled the ERROR log
and hid real faults from anyone alerting on it.

- s3tables: HandleRequest wraps the ResponseWriter in a statusRecorder (with
  Unwrap for http.ResponseController) and logs a handler error by the status
  it answered: 4xx at V(1) as "<op> refused (<status>)", 5xx or no status at
  ERROR as before.
- s3tables: the "handleCreateNamespace called" trace moves from ERROR to V(3).
- iceberg: writeManagerError's mapping is split into classifyManagerError,
  shared with a new logManagerError that logs 4xx-mapped errors at V(1) and
  everything else at ERROR; used by CreateNamespace, CommitTable and
  CommitTransaction.
- storage: loading a remote-tier volume logged
  "readSuperBlock remote volume N: <nil>" at ERROR even when the read
  succeeded; it is now logged only when there is an error.

Tests: TestRequestRefusedWith4xxIsNotLoggedAsError, TestLogRequestErrorByStatus,
TestStatusRecorderKeepsTheFirstStatus, TestStatusRecorderUnwrapsForResponseController,
TestNoTraceLoggedAtErrorLevel (fails on master on the CreateNamespace trace),
TestLogManagerErrorRefusalsAreNotErrors, TestRemoteSuperBlockLogOnlyOnError.

* s3tables: do not log a malformed CreateNamespace body at ERROR

handleCreateNamespace logged a body it could not decode at ERROR before
answering 400; the dispatcher already logs that refusal once at V(1).

Also drop statusRecorder.Flush: it made the recorder an http.Flusher even
over a writer that cannot flush, so http.NewResponseController reported
success instead of ErrNotSupported. Unwrap already lets the controller
reach the wrapped writer.

* iceberg: require exactly one log line per classified manager error

The test accepted a refusal that produced no log, or both logs. Require
one V(1) refusal and no ERROR for a 4xx, and the reverse for a fault.

---------

Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
2026-10-21 14:00:33 -07:00
Chris LuandDevin 780760fccc s3: report the copied bytes' MD5 in UploadPartCopy ETags (#11884)
* s3api: regression test for UploadPartCopy ETag on ranged copies

Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3api: hash the copied bytes for a partial-chunk UploadPartCopy ETag

A ranged UploadPartCopy that lands inside a source chunk stored the source chunk's ETag on the destination chunk, so the part reported the MD5 of the wrong bytes. streamCopyChunkRange now tees the forwarded plaintext into an optional digest writer; copySingleChunkForRange feeds it an md5 only when the range is not the whole source chunk, since a whole-chunk copy's stored bytes (and ETag) are unchanged.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3api: stamp the copied range's MD5 on UploadPartCopy part entries

The part entry got no Attributes.Md5, so copyEntryETag fell back to filer.ETagChunks: a single-chunk part inherited the source chunk's MD5 and a multi-chunk part reported a <md5>-N composite of source chunks. copyChunksForRange now returns the MD5 of the copied bytes — a single chunk's own digest, or for multi-chunk parts a second ordered stream of the source range hashed beside the concurrent chunk copies (md5 cannot combine per-chunk digests). The handler stores it on Attributes.Md5 and mirrors putToFiler's ExtETagKey, so CopyPartResult, ListParts and the completed object's composite ETag use each part's own MD5 (#11855).\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3api: drop the source chunk's gzip flag on partial-range copies

A ranged UploadPartCopy fetches raw (decompressed) bytes from the source volume and stores them uncompressed, but the destination chunk kept the source's IsCompressed flag, so reads of a partial copy of a gzip-stored chunk would try to decompress raw bytes.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3api: hash the chunk range, not the entry, for the part MD5

getEncryptedStreamFromVolumes prefers inline Content over chunks, but copyChunksForRange copies the chunks — on an entry carrying both, the hash would describe different bytes than the copy.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3api: cover sparse ranges in the part MD5 and fail on hash errors

A single chunk's digest is the part MD5 only when it supplies the whole range; a sparse source zero-fills the rest, so gaps are hashed via the ordered range stream, and a fully chunkless range hashes zeros locally. A failed hash pass now fails the part copy instead of storing a chunk-composite ETag that describes the wrong bytes.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3api: rank copied-range destination chunks by source timestamps

The concurrent chunk copies stamped each destination chunk with an arbitrary finish time, but reads resolve overlapping chunks by ModifiedTsNs — a copy of overlapping source chunks could expose different bytes than the source (and than the part MD5 just hashed). Destination chunks now get timestamps ranked by the source's, preserving its overlap precedence.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3api: cancel the part-MD5 stream when a chunk copy fails

The hashing goroutine ran on context.Background(), so an early chunk-copy error return left it streaming and hashing the whole source range for a discarded result. A cancellable context now aborts it on every exit path.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3api: hash the copied range when the covering chunk has no usable ETag

TUS and mount writes leave chunks without an ETag, so a whole-range
UploadPartCopy would otherwise save the part with an empty Md5 and no
ExtETagKey, and multipart completion rejects the part.

* s3api: build the md5 source entry without copying the proto message

* s3api: drop the issue reference from the part-copy etag test

* s3api: route inline-content sources through the plaintext part copy

The raw-byte path walks entry.GetChunks() only, so a ranged
UploadPartCopy of an inline-Content source (filer HTTP or mount writes)
produced a zero-filled part. Send those through
copyObjectPartViaReencryption, which streams the real bytes and stamps
their MD5 on the part.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3api: store the empty-part MD5 on writeEmptyCopyPart entries

The slow path's empty part returned the right CopyPartResult ETag but
saved no Attributes.Md5 or ExtETagKey, so ListParts and the completed
object's composite ETag still fell back to a chunk-derived value.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 13:46:11 -07:00
Jay HemnaniandChris Lu a3e97ec709 s3: apply Range to HeadObject (#11856)
* s3: apply Range to HeadObject

* s3: apply Range to a HEAD of a stored directory key

Directory-key entries answer through serveDirectoryContent before
HeadObjectHandler's range branch, so a ranged HEAD returned 200 and the
full length.

* s3: keep the missing-directory check for plain directories only

A stored directory-key marker is an object, so a range on its empty body
answers 416 like on any other zero-byte object.

* s3: let partNumber keep its priority over Range on a directory key HEAD

* s3: serve ranged GETs of a stored directory key

The same branch now applies the validated range to the inline content on
GET, so a directory-key object answers 206 with the selected bytes like
any other object.

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-10-21 13:37:00 -07:00
23d8c62844 volume: route lookups to the leader master only (#11888)
* volume: route lookups to the leader master only (#11887)

After a leader change, the volume server kept sending volume and EC
shard lookups to masters that were not the leader:

* currentMaster was set as soon as a heartbeat stream opened, before
  the peer said whether it was leader, and kept a follower through the
  redirect and its 3s sleep. A follower answers LookupVolume from its
  own vid cache, which right after an election can be empty, so reads
  went 404 rather than failing over.
* EC lookups used Store.MasterAddress, a second, unlocked copy that the
  heartbeat loop set before each dial and cleared on errors, so EC and
  plain reads could disagree on the master or have none.
* the receive goroutine called stream.Send itself when the volume
  options changed, racing the send loop, and exited silently on
  failure, after which leader redirects were no longer read.

Publish currentMaster only once a master is known to be leader: the
peer answered a new stream without naming a leader, which the leader
always does, or a follower named the leader. Compare redirects against
the connected peer. Replace Store.MasterAddress with MasterFn, backed
by getCurrentMaster. Have the receiver ask the send loop for the extra
heartbeat instead of sending it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: ask the masters for the leader after losing the heartbeat

When the heartbeat stream to the leader failed, currentMaster kept the
lost master until a new stream answered, so lookups on a cache miss
kept dialing it through the retry delay even with a new leader already
elected.

On a regular heartbeat error, ask the configured masters for the raft
leader and publish it, within the retry delay rather than on top of it.
Without an answer the last known leader stays: clearing it would only
trade a dead address for an empty one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: stop the heartbeat on a failed assertion

The leader test closed stopChan only once every assertion had passed,
so a failure left the stream open and the fake master's GracefulStop
waited for the whole test run's timeout.

Start the heartbeat through a helper whose cleanup stops it, safe to
call twice, and waits for it to exit. It runs before the cleanups
registered earlier, so the stream is closed before the store and the
fake master go away. Both heartbeat tests use it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: clear the leader when no master names one

When the heartbeat lost its master and no configured master named a
leader, refreshLeaderFromPeers kept the lost one. That master is dead
or no longer leader, and a follower answers LookupVolume from its own
cache instead of refusing, so lookups could return locations no leader
vouched for.

Clear currentMaster instead. Lookups then fail fast ("missing address")
until a leader is known again, rather than reading a follower's cache
or waiting on a dial to a dead master.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* volume: keep volumeSizeLimit 64-bit aligned on 32-bit builds

MasterFn (a func, 4 bytes on 386) replaced MasterAddress (a string, 8
bytes) ahead of volumeSizeLimit and moved it to a 4-byte-aligned
offset, where atomic.LoadUint64 panics with an unaligned 64-bit
operation. The atomic.Uint64 type carries the alignment itself.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: publish the answering leader before a duplicate-uuid retry

Read the leader field first on each heartbeat reply. A follower reply
carries nothing else, so it no longer runs through the option updates;
and a leader that rejects the registration over duplicate uuids still
answers lookups, so it stays the lookup target through the retry wait
instead of leaving the previous master in place.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: publish the master only once it is known leader (rust)

* volume: ask the masters for the leader after losing the heartbeat (rust)

* volume: read the leader field first on each heartbeat reply (rust)

* volume: drop the seed fallback from ec shard lookups (rust)

cached_lookup_ec_shard_locations fell back to the static first seed when
no leader was published, so a mid-election EC lookup could land on a
follower and read its stale shard map. Error out like Go's
masterAddress() returning "" instead, via a current_master() getter
matching getCurrentMaster.

* volume: send volume lookups to the confirmed leader, not the seed (rust)

Every HTTP lookup path — proxy/redirect targets, replicated writes and
deletes, batch chunk deletes, remote chunk reads — dialed the static
first seed, so after a failover they kept asking a follower whose vid
map can be stale. Go routes all of these through GetMaster; switch the
lookup calls and their leader-known gates to current_master().

* volume: mark volumes readonly on the leader, retrying the named one (rust)

notify_master_volume_readonly fell back to the seed when no leader was
published, which the peer answers with "Not current leader". It now
uses only the confirmed leader, skips the call in masterless mode, and
on NotLeader asks the seed masters who leads and retries there — the
same path Go's notifyMasterVolumeReadonly takes. The peer leader query
is shared with refresh_leader_from_peers as lookup_leader_from_peers.

* volume: ask the confirmed leader for copy prealloc options (rust)

VolumeCopy's preallocate query went to the static seed; Go issues it
through GetMaster. Route it through current_master() too — when no
leader is known the lookup fails fast and the copy proceeds without
preallocating, as before.

* volume: keep the announced master port forms, not just http (rust)

publish_current_master stripped to http form, so a leader configured
with an explicit gRPC port (host:9333.19999) was redialed on the
default offset. Store the announced host:port[.grpcPort] form like Go's
currentMaster; callers pick to_http_address / to_grpc_address /
parse_grpc_address per use, and Ping admission compares on the http
form.

* volume: fail replicated writes and deletes when no leader is known (rust)

Gating replication on a known leader meant a write could be
acknowledged on fewer copies than required, and a delete could leave
the data readable on the other replicas. Go fails the request when the
replica-location lookup cannot run; drop the gate so the empty-leader
lookup errors the same way.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-10-21 13:36:05 -07:00
Chris LuandDevin 4bdbfbaaa8 s3tables: judge an absent tag target by its canonical ARN (#11912)
* s3tables: judge an absent tag target by its canonical ARN

authorizeAbsentTarget matched the bucket policy's Deny statements
against the request's raw resource ARN. The ARN parsers accept any
account and region but resolve the target by name, so a caller with an
identity grant could reformat the ARN, miss the Deny match, and learn
whether a denied name exists — the oracle the absent-target check was
meant to close. Rebuild the ARN from the resolved names and the real
bucket owner and evaluate the Deny against that, matching what the
existing-target path checks.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3tables: cover a foreign-account ARN on an absent denied table

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3tables: keep the absent-target deny check behind a supplied ARN

Callers that pass no resource ARN (ListTables, ListViews,
CreateNamespace) must reach CheckTableOrBucketPermission's admin
shortcut unchanged; running the bucket-policy Deny match first would
deny the admin lists those callers take.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 13:31:04 -07:00
Mohd Quamar TyagiandChris Lu e73c23a4ee fix(s3): set Access-Control-Allow-Credentials on CORS responses (#11886)
* fix(s3): set Access-Control-Allow-Credentials on CORS responses (fixes #11885)

* fix(s3): expand wildcard exposed headers for credentialed CORS responses

* fix(s3): keep wildcard in exposed headers for non-credentialed CORS requests

* fix(s3): expose x-amz-tagging-count on credentialed CORS responses

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-10-21 13:10:49 -07:00
Chris LuandDevin 8c3d8fcea0 s3tables: keep absent-target answers identical to denied ones (#11905)
* s3tables: authorize before revealing a list or tag target exists

ListTables, ListViews and the tag operations read the target's catalog
metadata before any permission check, so a denied caller could tell an
existing name (403 or the resource's own error) from an absent one (an
empty list, or 404). Evaluate the operation's permission with whatever
bucket context could be read and answer a denied caller the same way for
absent, damaged and denied targets.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3tables: answer bucket reads uniformly when the entry is not a table bucket

A plain S3 bucket directory answers GetTableBucket and ListNamespaces with
500 attribute-not-found while a missing name gets 404, and a denied
CreateNamespace is 403 while a missing bucket is 404. Run the permission
check before the absence answer and read not-a-table-bucket as
NoSuchTableBucket.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3tables: cover absent-target answers matching denied ones

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 13:07:47 -07:00
Chris Lu bb1ba8b247 iam: take the implicit user name from the verified identity (#11903)
* iam: take the implicit user name from the verified identity

For self-service actions sent without UserName, the implicit user was
resolved by re-parsing the raw Authorization header for a SigV4
credential and looking that access key up. The header is not what
authentication verified: a request can be authenticated by one
mechanism while carrying a header that names another user's access
key, and the action then ran against that other user.

The auth middleware already records the verified identity in the
request context, so use it, on both the standalone iam server and the
embedded IAM surface. When no verified identity is present the name is
left unset rather than guessed.

* iam: test implicit user name resolution

Cover a request authenticated by presigned credentials that also
carries a junk SigV4 header naming another user: the self-service
actions must run as the verified signer. Update the unit tests for the
context-based lookup.

* iam: resolve the implicit user for standalone GetUser too

The embedded surface fills UserName for GetUser; the standalone server
skipped it, so an implicit GetUser on `weed iam` answered NoSuchEntity.
Same verified-identity source as the other self-service actions.

* iam: cover all implicit-username actions in the regression test
2026-10-21 12:21:04 -07:00
Javier de la TorreandJavier de la Torre 593cfee85b s3tables: accept Puffin statistics files in a table's metadata/ (#11892)
Table and partition statistics are Puffin files that the table metadata lists
in its statistics entries; orphan cleanup keeps them (#11651), and its test
places one at metadata/stats-1.puffin. The table bucket layout rule accepted
only *.stats among them, so an engine that names a statistics file *.puffin
had the upload refused with 403 and could not write statistics at all.

The metadata patterns now admit a *.puffin file directly under metadata/,
with the same safe-character rule as the other catch-all patterns. Nothing
else widens: a Puffin file in a subdirectory of metadata/, with a trailing
extension, or outside <namespace>/<table>/ is still refused.

Test: the new metadata/*.puffin cases in
TestIcebergLayoutValidator_AcceptsRealWorldManifestNames and
TestTableBucketFileValidator_PuffinStatisticsOnlyInTableMetadata fail on
master ("invalid metadata file format") and pass with the change; the
rejected cases pass on both.

Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
2026-10-21 12:19:53 -07:00
Daniel García 03065155db filer.remote.sync: do not upload an entry whose only content is the remote object (#11890)
* filer.remote.sync: do not upload an entry whose local copy was dropped

remote.uncache and the filer's eviction under storage pressure drop a
synced entry's chunks and clear LastLocalSyncTsNs; remote.meta.sync does
the same when it pulls a newer remote version. When the remote mtime
falls in the entry's mtime second, which is the common case for a small
file synced promptly, shouldSendToRemote falls back to the cleared stamp
and reads the entry as dirty. The dropped chunks make the update a
content change, so the chunkless entry was uploaded and a zero-byte
object replaced the remote one, the entry's only content.

Skip an update that leaves the entry at the same key with no local data
and a sized RemoteEntry: the object at that key already is the content.
The stamp stays cleared, so remote.cache and eviction still see the
entry as uncached. Renames keep going through completeRemoteOnlyRename,
metadata-only edits of a remote-only entry still reach the remote, and a
rewrite in the remote mtime's second holds local data and is uploaded.

* filer.remote.sync: upload a truncate that leaves the entry remote-only

A FUSE truncate of a cached, remote-backed file to zero (truncateEntry,
or SetAttr with size 0) drops the chunks and keeps the sized
RemoteEntry, so the entry reads as remote-only and the dropped-copy skip
left the remote object with its old content.

Skip only while the entry's size is still the remote object's. Uncache
and eviction keep the size and remote.meta.sync sets it from the
object, so those stay skipped; a truncate changes the size and is
uploaded, in the remote mtime's second too.
2026-10-21 11:32:20 -07:00
jsasandjames f8cfea0dcf gcs: size the upload buffer from the entry instead of reserving the default 16 MiB chunk per writer (#11854)
Co-authored-by: james <james@medable.com>
2026-10-21 00:03:38 -07:00
Jay Hemnani c90da999f2 s3: answer a conditional write on a missing object with 404, not 412 (#11882) 2026-10-20 23:43:30 -07:00
Chris LuandDevin 379abbe278 volume: repair torn tails on canDelete volumes; reject damaged 1-3 byte records (#11876)
- Go/Rust: run the unindexed .dat tail repair on noWriteCanDelete volumes
  too — deletes still append tombstone records, so the same crash window
  applies and they previously degraded to fully read-only.
- Go readTailNeedle: reject a positive size of 1-3 for v2+ as corrupted
  before ReadBytes slices the body's 4-byte data-size field and panics;
  a torn last record is now cut instead of crashing the load.
- Tests: TestVolumeLoadRepairsTailOnCanDeleteVolume and
  TestVolumeLoadCutsDamagedSmallSizeTailOnCanDeleteVolume (Go), and the
  Rust mirrors volume_load_repairs_tail_on_can_delete_volume /
  volume_load_cuts_damaged_small_size_tail.

Generated with [Devin](https://devin.ai)

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-20 23:40:51 -07:00
GAURIandGAURI c601712cc4 README: describe OLake Go and fix its wiki link (#11880)
README: describe OLake Go and point its link at the renamed wiki page

Co-authored-by: GAURI <174253719+gauriiiiiiiiiiii@users.noreply.github.com>
2026-10-20 23:31:18 -07:00
Chris LuandDevin ab738bfd33 rust volume: repair an unindexed .dat tail at load (#11878)
* rust volume: repair an unindexed .dat tail at load

Mirror Go's repairUnindexedDatTail (#11870): a crash between the .dat\nappend and the .idx append leaves a .dat tail no index entry covers,\nwhich sent the volume read-only on the next load and left acknowledged\nneedles unreadable. Before the index loads, re-index the complete\nneedles in the tail (a size 0 record as the tombstone or empty write\nresolveEmptyTailRecords reads it as) and cut only an incomplete last\nrecord. A tail an interrupted append does not explain is left as it is\nand the integrity check quarantines the volume read-only as before.

Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: tests for the unindexed .dat tail repair

Reload volumes through the real load path after dropping .idx entries\nand cutting or damaging the .dat, mirroring Go's\nvolume_checking_tail_repair_test.go: complete unindexed needles are\nre-indexed (deletes as tombstones, empty writes as empty needles), an\nincomplete last record is cut, and a tail an interrupted append does\nnot explain is left untouched with the volume read-only.

Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: re-index a torn size 0 record as a live empty needle

A delete and a write with no data append the identical size 0 record, so a
recovered one cannot be told apart from the prior index state alone: the
write path accepts an empty overwrite of a live key and indexes it with
size 0, while a delete of the same key publishes a tombstone. Inferring a
tombstone from prior live state turned a lost index row after an empty
overwrite into a deletion -- the object read back not-found although the
write was answered.

Every re-indexed record is now indexed at its on-disk size, the entry
publishWriteRequest writes for an empty write. The other misread is
bounded: a recovered delete reads its key back as an empty object rather
than not-found, and the size 0 record carries no data to resurrect.

Review feedback on the Rust mirror (#11878): the same bug exists here;
fixing both sides identically.

Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: re-index a torn size 0 record as a live empty needle

Mirrors the Go fix in this pull request. A delete and a write with no data
append the identical size 0 record, so resolve_empty_tail_records could
not tell them apart from the prior index state: it treated a zero-size
record of a live id as a tombstone, although the write path indexes the
same bytes as a live size 0 entry. A crash after an empty overwrite of a
live key therefore recovered the object as deleted.

Every re-indexed record is now indexed at its on-disk size, the entry
publish_write writes for an empty write. A recovered delete reads its key
back as an empty object rather than not-found; the size 0 record carries
no data to resurrect.

Review: greptile-apps[bot] and devin-ai-integration[bot] on #11878.

Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: repair a .dat tail left past an empty .idx

The writable index loader creates the .idx before the first needle append,
so a crash after that append leaves a volume whose data nothing indexes.
CheckVolumeDataIntegrity skipped an empty index outright, so the volume
loaded writable and the surviving first needle stayed invisible while
later writes appended past it.

A zero-size index now runs the same repair with tailEntryPos -1: the walk
starts at the first record past the superblock with no anchor needle and
no append timestamp to compare against, gets the same complete-record
validation and incomplete-tail policy, and re-verifies through the normal
tail check once it wrote entries. A tail the repair cannot explain fails
the check, so the volume loads read-only instead of writable on an
unaccounted-for .dat.

Review feedback on the Rust mirror (#11878): the same bug exists here;
fixing both sides identically.

Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: repair a .dat tail left past an empty .idx

Mirrors the Go fix in this pull request. The writable index loader creates
the .idx before the first needle append, so a crash after that append
leaves a volume whose data nothing indexes. Both the repair and the
integrity check skipped an empty index, so the volume loaded writable and
the surviving first needle stayed invisible while later writes appended
past it.

An empty .idx now runs the same repair with no anchor: the walk starts at
the first record past the superblock, and a tail it cannot explain fails
the load like any other repair failure, since every later check skips an
empty index.

Review: devin-ai-integration[bot] on #11878.

Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: sync the .dat before publishing re-indexed .idx rows

A power loss after the repaired .idx rows become durable but before the
.dat tail records they name are flushed leaves the index pointing past
the end of the data file; the next load then keeps the volume read-only
until an operator steps in. Sync the .dat first so the rows never
outlive the records they reference.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: sync the .dat before publishing re-indexed .idx rows

Mirrors the Go fix: a power loss after the repaired .idx rows become
durable but before the .dat tail records they name are flushed leaves
the index pointing past the end of the data file. Sync the .dat first.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-20 23:13:51 -07:00
Chris LuandDevin 222e96d6ab iceberg maintenance: merge bin files across parquet schema spellings (#11875)
* iceberg maintenance: compare parquet schemas by field id in bin-pack merge

parquet.EqualNodes ignores Iceberg field ids, so a column dropped and\nre-added under the same name and type compared equal and was merged,\nfiling the new column's values under the dropped field id. Require the\nfield ids to agree wherever both sides carry one; the v3 layout's\nper-field check uses the same test.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* iceberg maintenance: merge bin files whose schemas differ by field id

A bin used to take the first file's Parquet schema as the writer's and\nrefuse every file that was not structurally identical, so a table never\ncompacted once files had a different column set or REQUIRED-vs-OPTIONAL\nrepetition. A file whose flat columns the writer's schema covers by\nfield id is converted into it (a column the file lacks reads as null);\na file that covers the writer's schema restarts the bin with it, so no\ncolumn is dropped. Equality-delete columns resolve on each file's own\nschema. Renames, type changes, nested columns and files without field\nids stay refused. The sorted and spatial merges are unchanged.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* iceberg maintenance: test bin-pack merge across parquet schema spellings

Regression coverage for the field-id merge: an added column in either\norder, REQUIRED vs OPTIONAL, equality deletes resolved per file, the\nre-added column refused on v1/v2 and v3, and the refusals that stay\n(rename, type change, one-sided columns, an added required column, no\nfield ids).\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* iceberg maintenance: resolve missing delete columns as null in bin-pack merge

An equality delete on an optional column added after a data file was\nwritten no longer refuses the bin: the file reads the column as null,\nso a real delete key cannot match its rows and a null key deletes them.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* iceberg maintenance: write bin-pack merges under the files' union schema

The writer schema is now the union of the bin's file schemas by field\nid, computed in a schema-only pass before any row is written: a column\nstays REQUIRED only where every file has it and spells it so, and the\nmerge refuses only true name/type/id conflicts. Files that need a union\nneither side covers (an added column plus a REQUIRED/OPTIONAL\ndifference, disjoint added columns) now merge in a single pass instead\nof refusing, and progressively wider bins no longer reread and rewrite\nevery earlier file per restart.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* iceberg maintenance: spell the re-added column's test file for codespell

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* iceberg maintenance: resolve equality-delete columns by field id

A column renamed after its data and delete files were written still carries
its field id there; resolving it by id instead of the current name keeps a
rename from turning every equality key null. The missing-column null-key
fallback is now for data files only: an equality-delete file that lacks its
equality column is refused, as the spec requires it to carry them.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* iceberg maintenance: read only the parquet footer choosing a bin's writer schema

The union-schema pass downloaded every input in full, then the merge
downloaded each again. The footer suffices for the schema, and a chunked
file's ReadAt pulls only the chunks the footer spans.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-20 23:13:43 -07:00
Chris Lu a724f4296c s3, s3tables: create buckets and namespaces exclusively (#11872)
* filer_pb: add MkdirExclusive

A taken name leaves the callback as a result, not an error, so filer
failover does not count a lost race as a filer failure.

* s3tables: create the entry with its attributes in one exclusive create

CreateTableBucket and CreateNamespace checked that the name was free and
then wrote the entry in several calls, so a second create in the window
replaced the first caller's entry including its owner. The create now
carries the marker, metadata and tags in one O_EXCL CreateEntry that
answers ErrEntryAlreadyExists with the same 409.

* s3: PutBucket uses an exclusive mkdir

The same window let a plain PutBucket replace a table bucket created
concurrently. The already-exists branch now answers from the entry itself,
since the registry just cached the name as absent, and the admin
auto-create path is exclusive too.

* s3: read the winning entry when a create loses the race

The concurrent-create branch called existingBucketError on a name the
registry had just cached as absent, answering NoSuchBucket, and the
auto-create fallback could assign an S3 owner to a table bucket it met
there. The negative cache is cleared before the existing-bucket error is
built, and a table bucket keeps its owner-less shape.

* s3: fail an auto-create that met a table bucket

A table bucket carries no S3 owner and must not gain one, and the upload
that auto-create ran for must not continue into it either.

* s3: fail an auto-create when the winning entry cannot be read

An unreadable existing entry fell through to success, so the write
continued into an unvalidated entry, possibly a table bucket.
2026-10-20 19:01:08 -07:00
Chris Lu f05d743459 s3: resolve versionId tagging requests to the version-tagging actions (#11874)
* s3: resolve versionId tagging requests to the version-tagging actions

A request for an object's tags with versionId acted on a historical
version but authorized against the current-object tagging action, so an
explicit Deny on s3:*ObjectVersionTagging never applied.

* iam: map PutObjectVersionTagging and DeleteObjectVersionTagging to the tagging action

* s3: only resolve tagging when the query value is empty

The tagging routes match ?tagging with an empty value, so a non-empty one
is not a tagging request and would resolve a tagging grant onto the plain
read or version read that actually serves it.

* s3: keep resolving tagging for any query value

The routes match ?tagging whenever the key is present, so tagging=x is a
tagging request served by the tagging handlers and must resolve to the
tagging actions.
2026-10-20 18:59:19 -07:00
Chris Lu 03a75cbc52 s3tables: only the admin repairs a table bucket that has no owner metadata (#11873)
* s3tables: only the admin repairs a table bucket that has no owner metadata

CreateNamespace wrote bucket metadata naming the caller as owner before any
permission check ran, so an ordinary caller that found the marker without
metadata was written in as the bucket's owner and passed every check after.
A bucket in that state is answered with the operation's own denial; the
admin or a trusted internal handler may still repair it.

* s3tables: repair the marker-only bucket under the entry lock

A creator finishing its own create between the missing-metadata check and
the repair write would have its owner overwritten. The mutation now adopts
whatever metadata is there and writes only while it is still absent.

* s3tables: gate the repair on the Admin action, not an operation grant

hasAdminAction also answers s3tables:*, so a non-admin identity holding
the wildcard operation grant could pass the repair check and write itself
in as the bucket's owner.
2026-10-20 18:58:54 -07:00
dependabot[bot] efa9d304d2 build(deps): bump github.com/klauspost/compress from 1.18.0 to 1.18.7 in /test/kafka/kafka-client-loadtest (#11879)
build(deps): bump github.com/klauspost/compress

Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.18.0 to 1.18.7.
- [Release notes](https://github.com/klauspost/compress/releases)
- [Commits](https://github.com/klauspost/compress/compare/v1.18.0...v1.18.7)

---
updated-dependencies:
- dependency-name: github.com/klauspost/compress
  dependency-version: 1.18.7
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-20 17:49:47 -07:00
github-actions[bot] b7f5d04428 docs: regenerate star history chart 2026-10-21 00:43:44 +00:00
Chris LuandDevin ece2f7a7c2 iam: targeted writes for single-user IAM actions (#11877)
* s3api: persist single-identity IAM changes through the credential manager

CreateAccessKey, DeleteAccessKey, UpdateAccessKey, PutUserPolicy, DeleteUserPolicy, SetUserStatus, UpdateUser and DeleteUser used to load the whole configuration, change one identity and then SaveConfiguration, rewriting and re-propagating every identity. They now write only the identity (and, for rename/delete, the groups and service accounts that referenced it) through the credential manager, which propagates that identity alone. Static identities and skipPersist keep the previous full-save/no-write contract.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* iamapi: persist single-identity IAM changes through the credential manager

Same targeted-write change as the embedded IAM API for the standalone weed iam dispatcher: UpdateUser, DeleteUser, CreateAccessKey, DeleteAccessKey, UpdateAccessKey, PutUserPolicy and DeleteUserPolicy write one identity through the credential manager instead of saving the whole configuration. CreateAccessKey keeps the full save when the handler created a user the store does not know.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* credential/memory: re-key identity on UpdateUser rename

UpdateUser stored the identity under the old username key even when identity.Name changed. Re-key it the way filer_etc does (delete old, save new) so the targeted rename path sees the moved record.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test: cover targeted single-identity IAM writes

Recording-store tests assert each rerouted action uses the single-identity credential store methods and never SaveConfiguration, in both the embedded S3 IAM dispatcher and the standalone iamapi dispatcher, plus the skipPersist and static-identity contracts and a memory-store rename case.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* credential: keep not-found and rename-collision semantics in stores

IamGrpcStore.DeleteAccessKey translates gRPC NotFound to ErrAccessKeyNotFound so a repeat delete stays the no-op the IAM handlers advertise. MemoryStore.UpdateUser rejects a rename onto an existing user instead of overwriting it, matching RenameUser.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-20 17:41:37 -07:00
Chris LuandDevin 610346ae8c iceberg: keep version-hint.text out of remove_orphans (#11867) (#11871)
Generated with [Devin](https://devin.ai)

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-20 13:05:38 -07:00
Javier de la TorreandJavier de la Torre df89afbef4 volume: repair an unindexed .dat tail at load instead of loading read-only (#11870)
* volume: repair an unindexed .dat tail at load instead of loading read-only

A needle write appends the needle to the .dat, then its entry to the .idx,
then answers the client. A crash between the two appends leaves a .dat that
ends past the last indexed needle; on the next start CheckVolumeDataIntegrity
reports "actual N bytes expected M bytes" and the volume loads read-only, and
a complete needle in that tail cannot be read because nothing indexes it.
After a process crash those bytes belong to a write that was never answered;
after a power loss a complete needle can survive in the .dat while its .idx
entry did not, although the client was answered. So complete needles are
re-indexed rather than cut.

CheckVolumeDataIntegrity now repairs the tail before the existing check,
which then runs on the repaired files as before. The repair:

- acts only on a local, version 3 volume within MaxPossibleVolumeSize, with a
  whole number of .idx entries, that is about to load writable (neither
  noWriteOrDelete nor noWriteCanDelete is set);
- requires the needle at the highest indexed offset to match its entry (id,
  size, whole record, CRC);
- walks from the end of that needle to the real end of the file and gives
  every complete needle (header, body, checksum, timestamp and padding inside
  the file, CRC valid, append timestamp later than the needle before it) the
  .idx entry the volume server would have written, in .dat order; a size 0
  record, which is how a delete is appended, is indexed as a tombstone;
- cuts only an incomplete last record: fewer bytes than a header, a record
  running past the end of the file, or a last record whose CRC fails;
- changes nothing otherwise (a damaged record with more data after it, a
  header with id 0 or a negative size, append timestamps going backwards), so
  the volume loads read-only exactly as before.

The .idx entries are written and synced before the .dat is truncated and
synced, so a crash during the repair leaves the needles indexed and the
remainder still past them; the next load cuts it. One warning per repaired
volume names the needles re-indexed and the bytes cut.

Tests: volume_checking_tail_repair_test.go reloads volumes through the normal
load path. Repaired: one and two complete unindexed needles, an unindexed
delete, a partial header, a partial body, a complete needle followed by a
partial one, a last needle with a bad CRC, a partial needle after an indexed
tombstone; each reloads clean afterwards. Left read-only and unchanged: the
last indexed needle with a wrong id or a bad CRC, a bad CRC followed by more
data, zero bytes past the last needle, an index past the data, a volume
persisted read-only, a partial .idx entry, and a clean volume.
TestCheckVolumeDataIntegrityVerifiesDeletionTail now also checks that on a
writable volume its 64 zero bytes are reported and kept.

Fixes #11864

* volume: fail the integrity check when a tail repair cannot complete

A repair that wrote the .idx or cut the .dat and then failed to sync only
logged the error; the check that followed saw the repaired files and the
volume loaded writable on a repair that may not be on disk. Any repair
error other than an unexplained tail now fails the check, so the volume
loads read-only, and the next load repairs again.

A failed .idx append could leave part of an entry behind, which made every
later load reject the .idx as a partial entry. The appended bytes are
truncated away again before the error is returned.

* volume: do not cut a tail record that could not be read

The walk cut a last record whose read failed for any reason, so an I/O
error (EIO) on a complete record erased it. Only a record whose bytes are
damaged (needle.ErrorCorrupted: a CRC mismatch or a body that does not
parse) is cut now; any other read error is returned with both files left
as they are, and the volume loads read-only.

* volume: treat a sized tail record with no data as damaged

A needle with no data is written with size 0. A tail record whose header
gives a body but whose data size field reads 0 parsed as an empty needle,
and ReadBytes checks the CRC only when there is data, so the repair indexed
it with its positive size and reads returned an empty body. Such a record
is now damaged like one with a bad CRC: cut when it is the last record,
otherwise the tail is left alone and the volume loads read-only.

* volume: keep an unindexed empty write as an empty needle

A delete and a write with no data both append a size 0 record, and the
repair indexed every size 0 record in the tail as a tombstone, so an
acknowledged empty write whose .idx entry was lost read back as deleted.
A delete is only appended for a needle that is live, so a size 0 record
of a needle that is absent or deleted (in the .idx and in the tail before
it) is an empty write and is re-indexed with size 0. One of a live needle
stays a tombstone, the reading weed fix and the backup's index
regeneration already give it. The .idx is scanned only when the tail holds
a size 0 record.

* volume: cut torn-tail test records inside their body, not at +40

The partial-record cases cut the .dat 40 bytes into a record. A needle
with up to 7 bytes of data is a 40-byte record, so newRandomNeedle's
random length (1..1023) left it whole about once in 150 runs per case:
the repair then correctly re-indexed it and cut nothing, and the test
failed on the .dat size. The cut now lands halfway through the record's
body, which is always past its header and short of its end.

---------

Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
2026-10-20 13:03:00 -07:00
dependabot[bot] 1253b97303 build(deps): bump github.com/klauspost/compress from 1.15.9 to 1.18.7 in /test/kafka/kafka-client-loadtest/tools (#11866)
build(deps): bump github.com/klauspost/compress

Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.15.9 to 1.18.7.
- [Release notes](https://github.com/klauspost/compress/releases)
- [Commits](https://github.com/klauspost/compress/compare/v1.15.9...v1.18.7)

---
updated-dependencies:
- dependency-name: github.com/klauspost/compress
  dependency-version: 1.18.7
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-20 13:02:26 -07:00
dependabot[bot] a1d8739464 build(deps): bump github.com/klauspost/compress from 1.17.0 to 1.18.7 in /test/kafka/simple-consumer (#11869)
build(deps): bump github.com/klauspost/compress

Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.17.0 to 1.18.7.
- [Release notes](https://github.com/klauspost/compress/releases)
- [Commits](https://github.com/klauspost/compress/compare/v1.17.0...v1.18.7)

---
updated-dependencies:
- dependency-name: github.com/klauspost/compress
  dependency-version: 1.18.7
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-20 13:01:44 -07:00
Chris LuandDevin 1cfd9bbb4c volume: run the durable sync outside the reader-visible lock (#11863)
* volume: run the durable sync outside the reader-visible lock

Reads of a volume waited on that volume's fsync because the write lock
was held across Sync()/sync_all. A new per-volume lock (Go
dataFileSyncLock, Rust data_file_sync_lock) now serializes every .dat
mutation and lifecycle op across append+publish+sync+rollback, while the
fsync itself runs with dataFileAccessLock/the io write guard dropped —
so readers proceed during a sync, and a failed sync's rollback still
cannot clobber another write or a compaction swap.

Same change in Go (volume_write.go syncWrite/processBatch, SyncToDisk)
and Rust (write_needles_grouped/write_synced_run).

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: publish after sync; fix Close/CommitCompact lock order

Address review on #11863:

- Close took dataFileSyncLock before waiting on isCompactionInProgress
  while CommitCompact claims the flag then takes the lock — a deadlock.
  Close now claims the flag before taking the locks (flag -> sync -> access).

- Go now publishes needle-map entries only after the .dat sync, matching
  Rust's ordering: appendWriteRequest/appendDeleteRequest stage the bytes,
  publishWriteRequest/publishDeleteRequest index them after Sync. A failed
  sync rolls back bytes no reader ever resolved — the streamed-read
  offset-invalidation window is gone. processBatch splits runs at repeated
  needle ids (Rust parity) so dedup/cookie checks stay sequential; the old
  snapshot/restore rollback machinery is removed.

- Rust commit_compact (store-lock caller) takes data_file_sync_lock via
  try_lock -> WouldBlock before the write lease; commit_compact_held and
  the tier-down _held paths document/require the caller-held sync lock,
  and the two vol.instance().write_lock() grpc sites take it first. The
  swap can no longer land between a durable write's append and publish.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: restore write watermarks when an inline fsync fails

The syncWrite rollback truncated the .dat but left lastAppendAtNs,
lastWriteAppendAtNs, lastWriteNeedleKey, lastWriteDeleted and
lastModifiedTsSeconds describing the rejected write; recovery and
compaction could read a watermark for a record that no longer exists.
Snapshot and restore them on failure, as processBatchRun already does.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-20 12:58:02 -07:00
Chris LuandDevin 0de6347b11 s3: mask anonymous misses behind AccessDenied unless the bucket is listable (#11857)
* s3: mask anonymous misses behind AccessDenied unless the bucket is listable

Deferred anonymous GET/HEAD authorization only runs once an entry is
selected, so misses returned 404 while existing keys returned 403 — an
unauthenticated existence oracle for bucket names and object keys.

Route not-found responses in Get/HeadObjectHandler through a check that
only discloses 404 when anonymous listing is allowed (policy, identity or
a public-read bucket); otherwise the miss is indistinguishable from a
denied read, matching AWS and 4.48 behavior.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: mask missing-directory misses for anonymous reads too

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: check a fixed request shape when masking anonymous misses

The miss check passed the caller's request through, so query parameters
could retarget the resolved action (?policy) or satisfy an s3:prefix
condition on an unrelated subtree. Each check now runs on a request with a
fixed shape: a ListBucket scoped to the probed key and a plain read of the
key, so the miss only shows when the caller could learn the same fact
directly.

* s3: mask conditional-header misses for anonymous reads

An If-Match conditional on a missing key wrote NoSuchKey directly from
processConditionalHeaders, before the handlers reached the masked miss
path, so anonymous callers could still distinguish a missing key from a
denied read.

* s3: keep the versionId when masking anonymous misses

The read leg must evaluate the action the caller's request resolves to:
a versioned miss is s3:GetObjectVersion, and dropping versionId would let a
plain GetObject grant disclose versions the caller may not read.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-20 12:52:58 -07:00
Chris Lu 6936bbdc3e rust volume: score EC shard placement with the volume's own ratio (#11862)
The auto-select branches of ReceiveFile (is_ec_volume) and
VolumeEcShardsCopy passed the build-default DATA_SHARDS_COUNT into
find_ec_shard_target_location, so its free-slot maths scored every
volume as 10+4 even when the volume's .vif declares a custom ratio
(e.g. 6+2). A wider-ratio volume's shards could be steered onto a disk
too full to hold them all, and a narrower one would be refused slots
it actually has.

Add Store::lookup_ec_data_shards(collection, vid), which answers from
the mounted EC volume first and then from the .vif on whichever disk
holds it (via read_ec_shard_config_across_dirs, so split -dir/-dir.idx
and rebuild-staging layouts are covered), falling back to the default
when no metadata exists yet.

Mirrors the enterprise fix (seaweed-mono c22fa8a15).
2026-10-20 07:40:43 -07:00
Chris Lu 4d06742664 Detect and quarantine duplicate volume ids (#11763) (#11828)
* master: detect and quarantine duplicate volume ids across layouts (#11763)

* shell: volume.check.duplicate_ids audit command (#11763)

* master: persist duplicate-vid quarantine across heartbeats (#11763)

* master: quarantine all copies of a duplicated volume id

Near-simultaneous registrations let each side quarantine only the other,
so arrival order decided which copy stayed writable. A confirmed
cross-layout collision means either copy could be the impostor, so flag
every colliding copy read-only regardless of registration order.

* topology: return a copy from VolumeLayout.Lookup

Callers iterating the returned slice raced with registrations and
removals replacing the location list in place.

* master: release duplicate-vid quarantine when the conflict resolves

A cross-node volume.move briefly shows the same id on two nodes under
different disk layouts; permanent quarantine would pin a valid target
read-only. Keep the first-reported state per quarantined copy and lift
the marks once the conflicting copy unregisters, so only a persistent
collision stays read-only.

* shell: include ec shards in volume.check.duplicate_ids

Ec-only volumes share the global id space. Compare shard collections
against regular ones while allowing shards and regular copies of the
same encoded volume.

* master: keep quarantine release current and cover all removal paths

Track the latest server-reported read-only flag while quarantined so a
release restores the server s true state, and run the release check on
collection deletion and data-node removal too, not only explicit volume
unregistration.

* shell: do not flag ec-only volumes as duplicates

An ec-only volume has no regular copy, so its shard collections had
nothing to compare against and always read as foreign.

* master: quarantine every conflicting layout at the layout level (#11763)

- collect all conflicting layouts, not just the first, so a vid spanning
  three layouts does not leave one writable
- mark quarantine on the location list only: folding it into the stored
  report made the volume digest permanently disagree with the server and
  forced a full volume list every heartbeat
- isAllWritable now honors the location read-only mark so a quarantined
  copy stays out of writables through resyncs and re-registration
- re-check the conflict before keeping marks, and do not clear a node's
  quarantine while it still holds the vid in another layout
- log the full layout keys on both sides of the conflict

* master: push duplicate-vid quarantine to the volume server (#11763)

A quarantined copy stayed writable on the volume server itself for a
client holding an existing file id, since the master-only mark never
reached the server's local write path. Quarantine now also calls
VolumeMarkReadonly (can_delete so expiry still drains), is re-pushed if
the server comes back reporting it writable, and VolumeMarkWritable
runs when the conflict resolves and the server had reported writable.

* master: retry unenforced quarantine marks and restore server writability on release (#11763)

Dial the advertised grpc port, return mark errors, and keep a
quarantine entry until the writable push succeeds so a released
copy does not stay read-only on the server.

* master: treat rp-only layout differences as an in-flight replication change (#11763)

volume.configure.replication rolls through nodes one at a time, so a
volume legitimately reports the old and new replica placement while it
settles. Layouts differing only in rp within the same collection are a
supported transition, not a reused id.

* shell: group ec shards by collection in the duplicate id audit (#11763)

EC shards of one volume legitimately span disk types when
DiskTypePrefer spills, so grouping by disk type reports healthy
volumes as duplicates.
2026-10-20 07:37:33 -07:00
Chris Lu 2874bd05fb Freeze volume growth on topology id mismatch (#11763) (#11820)
* pb: ResumeVolumeGrowth rpc for operator-driven growth resume (#11763)

* master: freeze volume growth on topology id mismatch (#11763)

* shell: volume.growth.resume command (#11763)

* master: replicate the growth freeze and gate it on veteran heartbeats (#11763)

* master: block allocation while the topology id is being minted

DeferAllocation only armed after the topology id persisted, so growth
requests arriving during init ran ahead of the warmup window. Gate
NextVolumeId on an init-pending flag spanning the whole mint.

* master: stash reported topology ids until the local id is minted (#11763)

* drain stashed topology id reports when the id arrives outside the mint path

* persist the growth-freeze flag in the mirror file and raft snapshots

* log hashicorp raft marshal and apply errors in applyGrowthFreeze
2026-10-20 07:34:31 -07:00
Chris LuandDevin 1144bf15d4 rust volume: group-commit fsync uploads on the direct write path (#11860)
* rust volume: group-commit fsync uploads on the direct write path

Each fsync=true upload used to run its own .dat/.idx sync when the write
queue was off, which is the default. Route durable uploads through the
per-volume write workers so a batch shares one sync, matching Go's
writeNeedle2: non-fsync writes, and writes while the server is stopping,
stay on the synchronous path. The queue is now always constructed and
SEAWEED_WRITE_QUEUE is gone.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: document post_handler write routing

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-20 07:32:58 -07:00
Chris Lu 8d2b417c69 iceberg, lance: vend read-only credentials to callers that may only read (#11859)
* s3tables: share the UpdateTable authorization gate

Extract the validation, load, and permission check from handleUpdateTable
into authorizeUpdateTable so the table an update was authorized against is
the same snapshot the mutation writes to. Manager gains
AuthorizeUpdateTable, mirroring AuthorizeCreateTable, so callers can ask the
question without updating anything.

* s3tables: share the UpdateView authorization gate

Extract the lookup, policy load, and permission check from handleUpdateView
into authorizeUpdateView, mirroring authorizeUpdateTable, and expose it as
Manager.AuthorizeUpdateView for callers that need the answer without the
update.

* s3api: scope vended table credentials to read-only on request

VendTableCredentials takes a readOnly flag and tablePrefixSessionPolicy
emits only s3:GetObject for the table prefix when it is set, keeping the
conditioned ListBucket and unconditioned GetBucketLocation statements that
reads still need.

* iceberg: vend read-only credentials when the caller may not write

LoadTable and friends vend storage credentials only for the authority the
caller holds: the vend now checks UpdateTable/UpdateView permission and a
caller that may only read the catalog entry gets a read-only session. Paths
that just passed a write gate (stage-create, view commit) keep the full
session.

* lance: vend read-only credentials when the caller may not write

storage_options vends the same read-only session as the Iceberg catalog
when the caller lacks UpdateTable on the table.

* lance: keep write credentials for callers authorized to declare

A Lance commit is a direct S3 write that never revisits the catalog, so a
caller that just passed CreateTable still gets a writable session; the
UpdateTable gate applies to describe, the read path.

* iceberg, lance: log a failed write check instead of only reading it as denied
2026-10-20 07:16:27 -07:00
7ed26deaae s3: list keys in byte order around directory markers (#11832)
* s3: list keys in byte order around directory markers

* s3: order deferred dir blocks by slash name and hold quota

* s3: fix deferred dir blocks at page and marker boundaries

Review follow-ups on the marker byte-order change:

- A marker inside an emitted "<name>/" block re-descended and re-listed its
  folded contents under delimiter "/" instead of skipping the block.
- A prefix object's "<name>/" common prefix was emitted eagerly at the name
  position, outranking siblings like "a.txt" and repeating across pages; the
  plain key now emits alone at the name position and the block emits the
  common prefix when children exist.
- A recursion starved on quota before listing a deferred subtree returned
  without truncation, dropping keys silently; the block now reports a
  resumable truncation when the subtree still holds entries.
- Clone prefix-object entries with proto.Clone rather than copying the
  protobuf message by value.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: close the hasEntries probe stream on return

* s3: resume delimited listings at the dir block boundary

A deferred directory block emits at its "<name>/" position, but emitDir
left nextMarker at the bare name, which sorts inside the (name, name/)
band and reads as "block pending". Every pass of the listing loop then
re-emitted the same common prefix until maxKeys ran out (999 prefixes,
IsTruncated), and a "<name>/" continuation token trimmed to "<name>" by
adjustMarkerForDelimiter reopened the block and paged forever.

Advance nextMarker to the block boundary the same way the recursive
branch does, and keep a "/" -ending marker intact — it already names the
bound past the block.

---------

Co-authored-by: Devin <devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-20 05:36:22 -07:00
GAURIandGAURI da0d423bd9 Add OLake to README and Admin UI; fix catalog_olake path-style docs (#11858)
* README: mention OLake in the Iceberg lakehouse section

* catalog_olake: fix path-style prose; admin UI: list OLake as a REST catalog client

* admin UI: restore generated file import block, keep only the string change

---------

Co-authored-by: GAURI <174253719+gauriiiiiiiiiiii@users.noreply.github.com>
2026-10-20 04:57:55 -07:00
Chris LuandDevin 2095c905ba server: give heartbeat-options test a topology (#11853)
heartbeatResponse now reads MaxVolumeId and TopologyId off ms.Topo, but
the test built a MasterServer without one and panicked, failing the
package for every PR.

Generated with [Devin](https://devin.ai)

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-19 21:01:31 -07:00
Chris Lu 6b6a268903 volume balance: stop replica-fallback moves between equally full servers (#11851)
* volume balance: unify the per-move convergence guard

Evaluate the post-move utilization comparison through serverUtilization so
the guard also applies when servers lack MaxVolumeCount info (raw-count
mode), where it previously did not run at all.

* volume balance: restrict fallback destinations to improving servers

The score-based fallback inside createBalanceTask could redirect a move to
any valid server, including ones as full as the source — between equally
full servers the move only relocates the imbalance and the next detection
cycle reverses it (#11839). Exclude from eligibleTargets any server whose
post-move utilization would exceed the source's, so a fallback destination
satisfies the same convergence guard as the loop-chosen target.

* volume balance: skip a volume when the chosen target violates placement

When the detection loop's target resolves but cannot host the replica (it
already holds one, or is out of the filtered scope), createBalanceTask now
returns nil so the loop advances to the next volume — matching shell
volume.balance, which continues to the next candidate instead of
redirecting the move to a different server.

The score-based fallback is kept for the case where the target cannot be
resolved at all; its allowed set is already limited to servers the move
would improve.

* volume balance: regression test for replica-constrained targets (#11839)

Reproduces the reported loop: the emptiest server holds the second rp=001
replica of the lowest-ID volumes on the fuller servers, so it is never a
legal target for them. Detection must skip those volumes and send the
movable ones to the under-utilized server, and re-running detection after
applying the plan must never move the same volume twice.
2026-10-19 20:52:30 -07:00
Chris LuandDevin 79daffb2d2 test: retry retryable 503s in cold-start assign test (#11852)
The master intentionally sheds an assign that triggered volume growth
once its wait window elapses (503 + Retry-After); on a slow CI box
cold-start growth can outlast it, so the single-shot assertion flaked.
Honor the documented retry contract like a real client, within a 60s
deadline.

Generated with [Devin](https://devin.ai)

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-19 20:51:13 -07:00
github-actions[bot] 207d06863a docs: regenerate star history chart 2026-10-20 02:55:29 +00:00
Chris LuandDevin 7e348951c2 s3: strip internal metadata headers on auth entry (#11850)
* s3: strip internal object-lock and version headers on auth entry

putToFiler persists Seaweed-X-Amz-Object-Lock-Mode, -Retention-Until-Date,
-Legal-Hold and -Version-Id request headers into authoritative entry
metadata without a permission check, so an s3:PutObject-only writer could
pin COMPLIANCE retention and legal hold on its own uploads, bypassing the
dedicated-permission gate on the standard x-amz-object-lock-* headers.
The server only sets these headers from validated inputs, so drop them in
authenticateRequestInternal alongside the identity headers already
scrubbed there.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: strip internal SSE headers on auth entry

X-SeaweedFS-SSE-S3-Key-Data, -SSE-S3-Base-IV and -SSE-KMS-Base-IV are read
from the request to drive multipart encryption, so a client could inject
its own key material and IVs. The server sets them itself between handler
stages of a request; drop inbound values in authenticateRequestInternal.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: scrub internal headers before the auth-enabled early return

Auth and AuthPostPolicy skip authenticateRequestInternal when IAM is
disabled, so client-supplied internal headers still reached putToFiler.
Hoist the scrub into scrubInternalHeaders and run it unconditionally in
both wrappers as well as the shared auth entry point.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: test internal header scrub through the auth-disabled wrapper

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-19 18:35:20 -07:00
Chris LuandDevin 8b09c96cec s3: keep a signed content-length in the canonical request (#11849)
Co-authored-by: Devin <devin-ai-integration[bot]@users.noreply.github.com>
2026-10-19 18:32:44 -07:00
Chris Lu 0fb6c0e77d filer.sync: unstick sync on chunks the source no longer has (#11845)
* filer.sync: keep the needle-404 clock from resetting on live volumes

The give-up clock for a source chunk that cannot be produced is kept per
volume so a vacuumed volume's files all inherit one wait. But a needle
404 and a volume lookup failure shared that clock, and sourceServed
cleared it whenever any needle on the same volume read fine, so on a
busy cluster the grace never elapsed for a single deleted needle on a
healthy volume and filer.sync retried it forever (#11841).

Give needle not-found answers their own per-volume clock that serving
siblings does not reset.

* filer.sync: an answered 404 needs no liveness probe before skipping

After the missing-chunk gate gives up, onReplicateChunkError only skips
the entry when a re-read of the most recently served chunk succeeds.
That probe can never have been armed on a fresh sync, and its single
stored file id can itself be vacuumed since — either way the entry stays
pinned and the offset never advances.

A needle 404 was answered by a live volume server, which is itself proof
the source is serving requests, so skip on it directly. Keep the probe
for the lookup-failure flavor, where no server answered and the silence
can still mean a source outage.

* filer.sync: skip a replayed version the source rewrote in place

Supersession was decided by entry mtime alone, so a source entry whose
chunks were swapped without an mtime bump — remote cache eviction and
re-cache, an in-place repair — never read as superseded, and its
vacuumed chunks were retried indefinitely.

When a chunk is missing, compare the replayed chunk's file id against
the live entry's resolved chunk set: absent means the replayed version
is obsolete and its event is skipped on the first failure instead of
after the give-up window. The same live-entry comparison now also
decides the entry-level skip in onCorruptChunk/onReplicateChunkError.

* filer.sync: give each missing needle its own grace clock

A 404 is one needle's answer, but the clock was kept per volume: once the
first needle's wait expired, every other needle on that volume inherited an
already-elapsed clock and could be written off on its first sighting. Track
the wait per file id and clear it only when that needle itself serves again,
so a live volume's healthy needles neither shorten nor restart it.

* filer.sync: keep the probe for filer-proxied 404s

ReadPart proxied through the source filer collapses "no volume locations"
into the same 404 a missing needle returns, so that answer cannot stand in
for the liveness probe the way a direct volume read's can. Only treat 404 as
an answered needle when the sink reads volume servers directly.

* filer.sync: only a not-older live entry can supersede a replay

Chunk-list comparison must respect the same ordering the mtime check does:
a live entry older than the replayed version means the source was rolled
back, so its differing chunks say nothing about whether the replayed data
is still needed. Gate the content check on live mtime >= replayed mtime in
both sourceMovedOn and sourceDroppedChunk.

* filer.sync: compare chunk ids without resolving manifests

resolvedChunkFileIds downloaded manifest files to expand chunk lists, so a
manifest sitting on a dead volume made the supersession check itself fail
and left the offset pinned behind a replica the source cannot read. File
ids are immutable needle identities — the unresolved sets already compare
chunk content exactly, and the check no longer performs I/O.

* filer.sync: judge nested chunks by their top-level chunk

A manifest child is not in the entry's chunk list even when the manifest
still references it, so checking the live entry for the child id reported
it dropped and returned the raw 404 — unwrapped, so the offset still
pinned. Thread the entry-level chunk id down the manifest recursion and
consult it for the dropped-chunk check instead.

* filer.sync: let the manifest-child test exit through the gate

Run the worker with a shortened grace period and assert the returned error
is marked errSourceChunkMissing, instead of racing the worker still running
against a restored RetryWaitTime.
2026-10-19 18:11:51 -07:00
Jay HemnaniandChris Lu 60b06b6881 s3: sort ListMultipartUploads by key and honor key-marker (#11833)
* s3: sort ListMultipartUploads by key and honor key-marker

* s3: skip upload folders without an object key in ListMultipartUploads

* s3: order same-key multipart uploads by initiation time

The upload id suffix is now a time-ordered UUIDv7, so sorting by upload id lists a key's uploads oldest first, like S3. Pagination is unchanged: ordering and the upload-id-marker comparison share the same key.

selectMultipartUploads now builds the whole result, including NextKeyMarker and NextUploadIdMarker, and the pagination test pages with the response markers.

---------

Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-19 18:10:06 -07:00
Chris LuandDevin dbcf6850b6 ci: fail the awscli multipart retry loop on the last attempt (#11848)
Co-authored-by: Devin <devin-ai-integration[bot]@users.noreply.github.com>
2026-10-19 17:59:50 -07:00
Chris LuandDevin 04eff7d613 ci: retry the awscli multipart download on transient failures (#11846)
The HEAD on the completed multipart object occasionally gets a 403 from a
signature verification mismatch before the object read settles; retry the
download instead of failing the run.

Co-authored-by: Devin <devin-ai-integration[bot]@users.noreply.github.com>
2026-10-19 17:09:58 -07:00
Chris Lu 4feb30deff Federated volume-id bound via heartbeat stamps (#11763) (#11819)
* pb: heartbeat fields for the federated volume-id bound (#11763)

Heartbeat carries max_volume_id_seen and last_seen_topology_id up, and
HeartbeatResponse echoes max_volume_id and topology_id down, so the
allocation bound is replicated to every volume server.

* master: echo the volume-id bound and fold reported bounds in (#11763)

Every heartbeat response carries the leader current max volume id and
topology id; every heartbeat reported max_volume_id_seen is folded into
the topology bound via UpAdjustMaxVolumeId.

* volume: persist a per-dir vid stamp of the highest id ever seen (#11763)

The store keeps max_volume_id_seen + last_seen_topology_id in a
.vid_stamp file per volume dir, fed by heartbeat echoes, AllocateVolume,
and local volume/EC creation, seeded at boot from stamps and local
holdings, and reported in every heartbeat. A rebuilt master recovers
the bound from the first surviving veteran report.

* volume-rust: mirror the vid stamp and heartbeat echo handling (#11763)

Same mechanism as the Go volume server: .vid_stamp per dir, bound folded
in from add_location scans and allocate/mount calls, echoed fields
applied on heartbeat responses, reported on every heartbeat.

* volume: serialize vid stamp writes and fsync the directory (#11763)

* volume-rust: serialize vid stamp writes and fsync the directory (#11763)

* master: echo the allocation bound to connected servers on change

The bound was only sent at connect time or on digest mismatch, so a
healthy stream never learned later increases. If the high-id server then
went offline and the master lost state, survivors restored a stale bound
and the id could be reissued. Resend the heartbeat response whenever the
bound or topology id moved.

* volume: retry failed vid stamp saves on later updates

A transient write failure left the in-memory bound raised but the stamp
stale, and repeat updates at the same bound never retried. Track a dirty
flag so any later update re-attempts the save.

* volume: cover stamp bootstrap for pre-stamp dirs in test

* volume-rust: retry failed vid stamp saves, mirroring Go

* volume-rust: stamp every dir with the combined bound on add_location

A dir whose own stamp sat below the combined bound kept its stale value
after fold; losing the higher-stamped disk then restarted the server
below ids it had already learned. Rewrite all stamps after each location
folds in.

* volume: stamp mounted volume and ec shard ids (#11763)

* volume-rust: stamp mounted volume ids, mirroring Go (#11763)

* volume: retry dirty vid stamp save on unchanged topology id (#11763)

* volume-rust: retry dirty vid stamp save on unchanged topology id (#11763)

* master: record echoed bound from the sent heartbeat response (#11763)
2026-10-19 17:01:19 -07:00
Chris Lu 117ea58a83 s3tables: accept .puffin deletion vectors in table data/ (#11844)
* s3tables: accept .puffin data files for Iceberg v3 deletion vectors

Iceberg v3 writes deletion vectors as Puffin files under a table's data/
directory, so merge-on-read DELETE and UPDATE on v3 tables were refused
with AccessDenied by the table-bucket upload validator.

Fixes #11842

* s3tables: name all accepted data formats in rejection error

The message stopped at .avro even though .lance and now .puffin are
accepted.

* s3tables: cover deletion-vector filenames in data layout test

Uses the names real engines emit: Spark/Iceberg Java
{seq}-{task}-{uuid}-{counter}-deletes.puffin, UUID-suffixed names, and
the same under a partition directory.
2026-10-19 16:59:08 -07:00
dependabot[bot] ce93660abb build(deps): bump github.com/linxGnu/grocksdb from 1.10.8 to 1.11.1 (#11836)
Bumps [github.com/linxGnu/grocksdb](https://github.com/linxGnu/grocksdb) from 1.10.8 to 1.11.1.
- [Release notes](https://github.com/linxGnu/grocksdb/releases)
- [Commits](https://github.com/linxGnu/grocksdb/compare/v1.10.8...v1.11.1)

---
updated-dependencies:
- dependency-name: github.com/linxGnu/grocksdb
  dependency-version: 1.11.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-19 16:28:08 -07:00
dependabot[bot] 73391a2e2f build(deps): bump github.com/tidwall/gjson from 1.19.0 to 1.20.0 (#11834)
Bumps [github.com/tidwall/gjson](https://github.com/tidwall/gjson) from 1.19.0 to 1.20.0.
- [Commits](https://github.com/tidwall/gjson/compare/v1.19.0...v1.20.0)

---
updated-dependencies:
- dependency-name: github.com/tidwall/gjson
  dependency-version: 1.20.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-19 16:27:54 -07:00
dependabot[bot] 3a30ee177a build(deps): bump github.com/klauspost/compress from 1.20.1 to 1.20.3 (#11837)
Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.20.1 to 1.20.3.
- [Release notes](https://github.com/klauspost/compress/releases)
- [Commits](https://github.com/klauspost/compress/compare/v1.20.1...v1.20.3)

---
updated-dependencies:
- dependency-name: github.com/klauspost/compress
  dependency-version: 1.20.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-19 16:27:28 -07:00
dependabot[bot] 109dd24f52 build(deps): bump github/codeql-action from 4.38.3 to 4.39.0 (#11840)
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.38.3 to 4.39.0.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/v4.38.3...v4.39.0)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-version: 4.39.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-19 16:05:07 -07:00
dependabot[bot] a2f8f4943c build(deps): bump gocloud.dev/pubsub/rabbitpubsub from 0.46.0 to 0.47.0 (#11835)
---
updated-dependencies:
- dependency-name: gocloud.dev/pubsub/rabbitpubsub
  dependency-version: 0.47.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-19 16:03:18 -07:00
dependabot[bot] 4909843748 build(deps): bump modernc.org/sqlite from 1.57.0 to 1.60.1 (#11838)
Bumps [modernc.org/sqlite](https://gitlab.com/cznic/sqlite) from 1.57.0 to 1.60.1.
- [Changelog](https://gitlab.com/cznic/sqlite/blob/master/CHANGELOG.md)
- [Commits](https://gitlab.com/cznic/sqlite/compare/v1.57.0...v1.60.1)

---
updated-dependencies:
- dependency-name: modernc.org/sqlite
  dependency-version: 1.60.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-19 16:03:12 -07:00
Chris Lu 0a96972a99 s3: verify X-Forwarded-Prefix only from trusted proxies (#11843)
* s3api: check the direct peer against the trusted-proxy allowlist

* s3: verify X-Forwarded-Prefix only from trusted proxies

* s3: note that trusted proxies must sanitize forwarded headers
2026-10-19 15:55:18 -07:00
Jay Hemnani 4e5fed9e9e s3: list the size of a directory marker that has a body (#11830) 2026-10-19 04:35:23 -07:00
Chris LuandDevin e7923c14d1 s3tables: honor resource policy Deny on remaining operations (#11825)
* s3tables: honor a table policy Deny for maintenance operations

authorizeMaintenanceTable evaluated only the bucket policy, so a Deny in the table policy never vetoed PutTableMaintenanceConfiguration, GetTableMaintenanceConfiguration or GetTableMaintenanceJobStatus even though they act on the table ARN. The stored table policy is now checked for a matching Deny first via deniedByPolicy, a veto helper that keeps the admin exemption, and resource and bucket tags are loaded into the policy context so tag-conditioned Denies evaluate. readBucketPolicy is renamed to readPolicy and reused for the table path.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3tables: honor the stored policy's Deny on policy management operations

PutTablePolicy, GetTablePolicy and DeleteTablePolicy only evaluated the bucket policy, so a Deny in the table's own policy was ignored and a bucket-level Allow could strip or replace the policy guarding a table. The existing table policy is now loaded and a matching Deny vetoes the call. PutTableBucketPolicy likewise consults the existing bucket policy for a Deny before replacing it. Resource and bucket tags populate the policy context so tag-conditioned Denies evaluate, and policy writes assert the policy authorized against is still in force, returning a conflict when it changed mid-request.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3tables: apply a table policy Deny to tag operations

TagResource, ListTagsForResource and UntagResource evaluated only the bucket policy; a Deny in the table's own policy is now honored when the target ARN is a table. ListTagsForResource and UntagResource also returned early when the resource carried no tags attribute, skipping the permission check entirely; the check now always runs. Policy evaluation uses the canonical resource ARN rebuilt from stored metadata, so a request ARN with a forged account or region cannot dodge the Deny.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3tables: regression tests for resource policy denies

Cover the case where a bucket policy grants an identity s3tables:* while the table policy denies it: the policy, tag and maintenance operations must stay denied so the protection cannot be stripped. Also cover a bucket policy that denies PutTableBucketPolicy under the open fallback, and a tag-conditioned Deny, which vetoes only while the stored tag matches.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-19 03:59:18 -07:00
Chris Lu 51a4c092c8 rust volume: release the store write lock during needle I/O (#11823)
* volume: give Volume interior locks mirroring Go's

Volume's mutable state moves behind interior locks matching Go's
per-volume lock domains, so the store can hand out shared handles
instead of requiring exclusive &mut access:

- io: RwLock<VolumeIo> owns the .dat/remote handles, needle map,
  append cursor, compaction bookkeeping and write watermarks —
  Go's dataFileAccessLock-adjacent mutable set; helpers take the
  guard as &mut VolumeIo so nothing re-locks.
- super_block and volume_info each get a RwLock
  (superBlockAccessLock, volumeInfoRWLock); dir_idx gets its own
  so path building never contends with I/O.
- no_write packs Go's noWriteLock pair into AtomicU8;
  last_modified_ts_seconds and compaction_byte_per_second become
  atomics.

The data-file write lease (dataFileAccessLock) now covers every
mutating entry that previously relied on borrow exclusivity:
writes, deletes, blob writes, set_writable's flag/index swap,
remote-file updates, local-backend swaps, index relocation,
replica-placement rewrite, put_needle_index, close and destroy.
set_read_only_persist leases only its can_delete idx-attach branch
because mark_io_unavailable reaches it under a held lease.
commit_compact takes try_write_lock — callers hold the store lock,
so blocking would stall every volume behind one slow reader;
WouldBlock surfaces as COMMIT_WAITS_FOR_READS for the caller to
retry.

Lock order is fixed: lease -> io -> volume_info/super_block/dir_idx.

* storage: mount volumes as shared Arc<Volume> handles

DiskLocation's map owns Arc<Volume> so the store can hand a caller a
handle that stays valid across store-guard releases, instead of
borrowing &mut Volume under the store write lock.

- volumes: HashMap<VolumeId, Arc<Volume>>; find_volume returns a
  cloned Arc.
- find_volume_mut and volumes_mut are gone — every Volume method is
  &self now, so shared handles cover what exclusive borrows did.
- writable_volume_mut -> writable_volume(&self) -> Result<Arc<Volume>>
  keeps the same admission checks (exists, disk-space-low).
- write_volume_needle/write_volume_needles/delete_volume_needle are
  &self; lifecycle methods that mutate the map or mounted state stay
  &mut self.

* server: run needle I/O with the store guard released

The store write lock previously wrapped needle writes and deletes
end-to-end: process_batch held it across every grouped fsync, and the
HTTP/gRPC fallbacks, tail receiver and fetch-and-write paths did the
same. One volume's slow I/O then stalled admission and metadata on
every other volume.

- write_queue resolves each group's handle via writable_volume under
  a brief store read, then runs write_needles_grouped after the guard
  drops.
- HTTP write/DELETE fallbacks, the gRPC batch-delete EC recheck +
  delete, tail receiver and fetch_and_write local writes, blob writes
  and the quarantine/sync/set_writable/remote-file paths all run
  under store.read() now that every volume method they reach is
  &self.
- Lifecycle calls that mutate the volume map or mounted state
  (mount/unmount/delete, EC shard ops, compaction commits,
  load_new_volumes, heartbeat volume actions) keep store.write().

* test: cross-volume admission proceeds during another volume's write

Regression coverage for the store-lock fix:

- write_queue's test parks a volume-1 write inside its .dat append via
  a new needle_write_hook seam (mirroring needle_read_hook), then
  proves the store still hands out a lifecycle writer and a volume-2
  handle — while volume 1's data-file lease still excludes same-volume
  writers.
- tests/store_lock_repro.rs exercises the public API: concurrent
  read-guarded writes on two volumes, and a stale Arc<Volume> handle
  failing closed after unmount instead of writing into torn-down
  files.

* test: drop store write guards that shared handles made read-only

Test setup sites held store.write() purely to reach find_volume_mut;
with Arc<Volume> and &self methods the guard is read-only (clippy
readonly_write_lock). The volume lease still serializes the writes
inside.

* volume: drop a windows-only read_exact_at that shadows the io helper

cargo's windows build failed E0255: the free fn copied with the
interior-lock conversion collided with the unconditional
crate::storage::io::read_exact_at import, which already carries the
platform branches internally.

* server: release the store read guard before needle I/O

The read guard still ran through write_needle_blob_and_index, the tail
receiver write, the fetch-and-write local write, and the HTTP write and
DELETE fallbacks, so a slow append blocked mount/unmount writers (and,
on writer-preferring locks, readers on other volumes). Resolve the
Arc<Volume> under a short guard and write through the handle after it
drops — the same scoping process_batch already used.

* volume: serialize the read-only mark and its .vif save under the lease

set_read_only_persist only leased the idx-attach branch, so concurrent
mark-readonly and mark-writable calls could interleave flag updates and
.vif writes — including two save_vif calls truncating and writing the
file at once. The lease now spans the whole flag change and save, with
a _held variant for mark_io_unavailable, which reaches this from write
paths already holding the lease (Go: markVolumeReadonly runs under
dataFileAccessLock).

* volume,server: span the tier-down transition under one write lease

update_remote_files cleared the remote-only write restriction and
released its lease before open_local_dat_backend attached the local
file, so a write could slip into the gap — pass check_writable, then
fail appending to a missing dat_file. The handler now holds the lease
across the reference trim, .vif save, and backend swap (and the rollback
re-insert), via _held variants; the store read guard is scoped to the
handle lookup. The swap still lands after the .vif trim — the ordering
that keeps a crash from pointing the volume at a deleted remote object.

* server: run a batch's volume groups on separate scoped threads

A single batch can hold writes for several volumes; running the groups
sequentially made every later volume wait behind a stalled append on an
earlier one — the same head-of-line blocking the store guard release
fixed, one level up. Each group now gets a scoped thread, preserving
order within a group while independent volumes progress in parallel.

* storage: close a volume handle a mount slot replaces

With shared handles, inserting a new Volume over an existing map entry
left the old handle alive in callers — two Volume instances could then
write the same files with separate in-memory indexes. set_volume and
create_volume now close() the replaced handle so its writers fail
closed, matching what unload_volume already does.

* test: key the write hook off a reserved needle id

The hook registry is process-global and keyed by needle id, and other
parallel tests write NeedleId(1) — an unrelated write could enter the
barriers and leave the test's real writer parked forever.

* server: wait out readers for vacuum commit without the store guard

commit_compact try-locked the write lease because callers held the store
write lock, returning COMMIT_WAITS_FOR_READS while a download was in
flight — and the master reads that as a failed vacuum, discarding
completed compaction work. The commit RPC now resolves the handle under
a brief store read and blocks on the volume's lease, matching Go's
CommitCompact under dataFileAccessLock: a slow reader stalls only this
volume's vacuum. The try-lock path stays for callers that still hold the
store lock.

* server: scope the remaining store read guards to volume lookup

Same class as the write-path fix: the batch-delete cookie check and
tombstone, the query read, volume_needle_status, and the local chunk
read all kept the read guard through the needle operation, so slow I/O
there could still stall store writers. Each now resolves the Arc<Volume>
under a short guard; the EC rechecks keep their guards.

* seaweed-volume: fail closed on detached volume handles

A handle resolved under a brief store read can outlive its mount: an
unmount+remount between lookup and lease leaves the Arc pointing at a
closed Volume while a new instance owns the files. Mark the handle in
close() and reject it under the write lease in commit_compact_held,
delete_needle, check_writable, set_read_only_persist and
update_remote_files_held; the tier-down handler also checks the flag
before deleting a remote object its replacement still reads.

* seaweed-volume: drop the store guard before mark-readonly waits on the lease

mark_volume_readonly held a store read across set_read_only_persist, which waits out the volume's leased readers; a slow download would stall every lifecycle operation behind the guard. Resolve the handle under a short read, then mark off the guard; a detached handle now maps to not_found.

* seaweed-volume: retry batch deletes that raced an EC mount

The store guard now drops before delete_needle, so ec.encode can mount shards between the pre-delete recheck and the tombstone; the encode then removes the .dat and the needle resurrects from pre-tombstone shards. Re-check has_ec_volume after each successful delete and return the retriable 503 so the filer requeues onto the EC path.

* seaweed-volume: one write-queue worker per volume

A stalled batch kept later batches for other volumes queued behind it: the single worker awaited process_batch before receiving again. Go's volume_write.go keeps one batching worker per volume, so give each volume its own FIFO worker spawned lazily, bounded by a global concurrent grouped-I/O semaphore and a global queued-request bound; workers exit only when their channel closes, and submit refuses unknown volume ids before spawning.

* seaweed-volume: move the queued-write permit inside the request

submit held the global queued-slot permit across the send and the response wait, so a canceled submit freed the slot while the worker still held the needle; repeated cancels could overflow the bound. The request now owns the permit until the worker drops it.

* seaweed-volume: retire write workers whose volume left the store

worker_for kept a sender per volume forever, so servers that mount and retire volumes accumulated idle workers. A worker now polls while idle and retires once its volume leaves the store: close the channel so racing senders fail fast, drain accepted writes, then release the map slot — a replacement worker can never start while this one still holds queued writes, so same-volume FIFO survives retirement.
2026-10-19 03:00:26 -07:00
Chris Lu 0b3ed4502f Design + master mirror of max volume id outside raft (#11763) (#11818)
* design: durable volume-id allocation boundary (#11763)

* master: mirror max volume id to a file outside raft (#11763)

The allocation bound lived only in raft state, so a master that lost it
rebuilt the bound from whatever volume servers were connected and could
reissue ids held by offline servers.

NextVolumeId now persists the bound to <mdir>/max_volume_id before the
raft apply, and both raft implementations persist on apply and recovery
so follower mirrors stay current. The file also carries the topology id,
which survives raft state cleanup the same way the snapshot recovery did.

* master: serialize and fully sync the max volume id mirror (#11763)

* master: rewrite the mirror when the topology id changes at the same bound

A same-bound raft apply carrying a new topology id was skipped by the
lastPersistedMaxVolumeId check, leaving the mirrored topology id empty
until the next allocation. Track the mirrored topology id too and
rewrite when either value differs.

* drop the design doc from the PR stack
2026-10-19 02:42:42 -07:00
Chris Lu ab559bca76 storage: hash index-metric bloom filter with xxhash
The reverse index walk dedupes seen NeedleIds with a BoomFilters bloom
filter. Its default kernel is FNV-1, whose final step only XORs the last
input byte into the hash's low bits, so keys sharing the high 7 bytes of
a big-endian NeedleId land on correlated probe positions and false
positives arrive in bursts of hundreds instead of the ~0.001 target
rate. Each false positive hides a live-latest key and inflates the
rebuilt deletion counters, which is what made
TestFastLoadingNeedleMapMetrics flake past its +256 bound in CI
(observed DeletedCount drift of 267).

Swap the kernel for xxhash, which fully diffuses the key bytes; the
observed false-positive count in 300 local runs of the test drops to
max 2 (avg 0.23).
2026-10-19 01:47:25 -07:00
d04d838ebd ec: compare live idx NeedleId sets before encode (#11812)
* ec: compare live idx NeedleId sets before encode

Refuse encode when replicas' live .idx NeedleId sets differ. Walk only
live rows (skip deleted and Size<=0) and compare NeedleId, ignoring Size
so MIME/flags drift does not abort a matching id set. Mark the volume
read-only for encode and reject writes after that.

* fix(ec): compare all replicas then pick capacity source

Log matching live idx sets with "replicas are consistent" for the
shell CI check. After readonly, compare every replica's live NeedleId
set, then require the encode source to have FreeVolumeCount>=2.
Empty live-idx vs non-empty refuses encode instead of skipping
superblock stubs. Drop the FileCount rebind after the idx gate.

* volume: mark readonly under the write lock so checked appends finish first

* ec: fix readonly deadlock and restore worker-host source preference

MarkVolumeReadonly held dataFileAccessLock while calling
reopenIdxForWrite, which re-acquires the same mutex and deadlocks
TestMarkVolumeReadonlyCanDelete. Split out reopenIdxForWriteLocked for
callers already holding the lock, matching the existing Locked pattern.

The worker task lost its worker-host source preference when the idx gate
replaced the replica union. Late-bind the source to a replica on the
worker host using only the checked location set, with no FileCount
rebinding. Drop the now-unused helpers, trim comments added by the
earlier commits, and rename tests whose names still claimed convergence
while asserting refusal.

---------

Co-authored-by: Chepurnov Simon <195125934+CHEPURNOV-S-S@users.noreply.github.com>
Co-authored-by: Devin <devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-19 00:56:17 -07:00
Chris Lu 10ce19cb17 test: poll for TopologyId convergence in multi-master failover tests
A restarted goraft master wipes its raft state and only learns the
TopologyId after the leader re-admits it and replays the entry. That
trails HTTP readiness by several seconds, so the one-shot check in
assertTopologyIdConsistent raced the propagation (CI run 39199033687:
TopologyId applied on the follower ~2s after the test gave up).
2026-10-19 00:55:41 -07:00
Mohd Quamar TyagiandMohd Quamar Tyagi e20640acb4 fix(admin): improve Total Volumes count contrast (#11817)
fix(admin): improve Total Volumes count contrast on dashboard

Co-authored-by: Mohd Quamar Tyagi <Tyagiquamar@users.noreply.github.com>
2026-10-18 21:34:09 -07:00
Chris Lu 8ac5133abe admin: default to port 12535, keep 23646 as a best-effort compat listener (#11816)
* admin: default to port 12535, keep 23646 as a best-effort compat listener

The default admin HTTP port 23646 derives a worker gRPC port of 33646
(port + GrpcPortOffset), inside the Linux default ephemeral range
(32768+). One of our own startup dials can claim 33646 before the admin
binds it, killing startup. This is the same failure #10209 fixed for
the test port allocator, still reachable via the mini default (#10838)
and via standalone `weed admin` defaults.

Move the default admin port to 12535 (gRPC 22535) everywhere, keeping
the http+10000 convention so worker -admin derivation, the Rust worker
and host:http.grpc addressing keep working:

- weed admin -port, weed mini -admin.port
- weed worker -admin, the plugin worker fallback, Rust worker and
  weed-lance-worker defaults
- help text, README, helm values, docker compose, tests

So old workers, bookmarks and scripts keep working, admin also serves
the same handler and the worker gRPC server on the previous default
(-port.legacy / -admin.port.legacy, default 23646, 0 disables). The
legacy gRPC port binds best-effort: it lives in the ephemeral range, so
a failed bind only warns instead of reintroducing the startup failure.
weed mini reserves it early like the primary port and keeps it out of
its own port relocation. grpc.Server tracks every listener handed to
Serve, so the compat listener stops with the primary.

Also warn when an explicitly configured admin port puts the derived
gRPC port at or above the ephemeral floor, the other fix suggested in
the issue.

Supersedes the pinned-grpc approach in #11814: pinning -admin.port.grpc
off the +10000 convention breaks external workers that derive the admin
gRPC port from the HTTP address. ClickHouse/ClickHouse#115512 pins
-admin.port on their side; this makes that pin unnecessary.

Fixes #10838

* admin: let services win the legacy ports, close compat listener on failure

Review findings on the compat listeners:

- Do not put the legacy ports in mini's reservedPorts: an explicit
  -s3.port=23646 or -master.port.grpc=33646 must be honored, not rejected
  or stolen by the early reservation. Mini now checks the finalized port
  set and skips only the conflicting compat listener, so HTTP and gRPC
  compat degrade independently and the service keeps its port.
- Also skip the legacy HTTP listener when a mini service owns the port:
  a wildcard-bound service can still leave loopback bindable, which would
  let the compat listener shadow it for loopback clients.
- Close legacyServer's raw listener in the deferred cleanup: Serve may
  not have registered it yet when startAdminServer returns early.
- Helm: add admin.legacyPort (default 23646) and expose the legacy
  http/grpc ports through the admin Service, container ports and
  NetworkPolicy so older workers resolved via the Service keep working.

* helm: only pass -port.legacy when it differs from the binary default

ct install deploys the released image with the new chart, and 4.48 does
not know -port.legacy, so the admin pod crash-loops. Omitting the flag at
the default value is behavior-identical on new binaries (their default
is also 23646) and keeps older releases installable; legacyPort: 0 or a
custom port still renders the flag.
2026-10-18 21:21:05 -07:00
github-actions[bot] d3ee817b2e docs: regenerate star history chart 2026-10-19 00:47:21 +00:00
863efb3d35 webdav: delete a folder with its contents (#11811)
* webdav: delete a folder with its contents

* webdav: guard the recursive delete against stale and confined paths

- propagate stat lookup errors instead of returning an error-bearing
  FileInfo: OpenFile and DELETE would otherwise take the recursive
  removeAll on a path whose type was never confirmed, wiping a
  directory on a transient filer error
- refuse wrappedFs.Rename from or to the served root, matching the
  RemoveAll guard and webdav.Dir
- normalize trailing slashes in removeAll so the delete names the
  entry itself, and refuse the root for every caller

* webdav: keep file-only deletes shallow and refuse ancestor destinations

- removeAll used by OpenFile and failed-upload cleanup now deletes
  non-recursively, so a folder swapped in after the stat check is not
  emptied by a pending file delete
- MOVE and COPY whose destination contains the source are refused
  before the handler removes the destination, which would otherwise
  take the source down with it

* webdav: also refuse a destination inside the source

a MOVE or COPY into its own subtree recurses into the destination it
just created; refuse it with the same guard instead of burning the
recursion budget for a 500

* webdav: allow a shallow copy into the source subtree

a COPY with Depth: 0 creates only the destination collection and never
reads children, so it is safe inside the source; only deep copies and
moves are refused there

* webdav: refuse a copy/move whose source is unreadable

* webdav: cover the missing-source refusal in the guard test

* webdav: 500 on a source lookup failure, 404 only when absent

---------

Co-authored-by: Chris Lu <chris.lu@gmail.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-10-18 12:57:10 -07:00
Chris Lu 8cb42cd69f build(deps): bump github.com/seaweedfs/raft from 1.2.1 to 1.2.2 (#11815)
Brings in the raft snapshot-recovery fixes: a recovered follower returns
to Follower so it can campaign when the leader goes away (masters on
ephemeral -mdir no longer wedge leaderless), and currentTerm stays
monotonic and persisted through recovery.
2026-10-18 12:56:29 -07:00
Chris Lu 6a594da739 filer: free orphaned hard-link chunks and remote objects in delete sweeps (#11809)
* filer: free shared chunks when a folder delete drops the last hard link

doBatchDeleteFolderMetaAndData skipped chunk collection for every hard-linked child, relying on maybeDeleteHardLinks to retire the shared inode record. When the delete removes the last link the record is gone and nobody else will ever free those chunks. After the link callback runs, check whether the record survived; if it did not, queue the entry's chunks for deletion.

* filer: free orphaned hard-link chunks in the replayed delete sweep

The subtree sweep on a replayed directory delete skipped every hard-linked child, so a link that was the inode's last owner leaked its shared chunks. Check the inode record after the store delete; when it is gone, queue the child's chunks like any other swept entry.

* filer: mirror remote-object cleanup in the replayed delete sweep

Children a lagging peer alone still holds under a replayed directory delete were dropped with plain store deletes: their remote objects leaked and lazy remote fetches could resurrect them. The sweep now records event-stamped tombstones and deletes swept non-directory children through maybeDeleteFromRemoteIfUnchanged, which carries the same stat guard the write-back daemon's processDeleteEvent applies plus a modification-time check — a delayed replay cannot erase an object another writer replaced, even a same-size one. A failed remote delete fails the replay so the event is retried instead of orphaning the object. The directory's own tombstone comes from the event pipeline at the event timestamp and covers the subtree. Metadata-only deletes and cross-cluster events keep the origin's semantics.

* filerstore: return real errors from KvGet on MongoDB, ArangoDB, Elastic

These stores mapped every lookup failure to ErrKvNotFound, so a database error looked identical to a missing key: hard-link records could appear gone while links still owned their chunks, and path timestamp reads could treat committed state as absent. Return the underlying error and keep ErrKvNotFound for genuinely missing keys.

* filer: bind remote-object deletes to the observed object version

A guarded stat-then-delete still lets a replacement landing between the two requests go down with the deleted object. Add an optional conditional-delete operation to remote storage clients — S3 If-Match on the live ETag, GCS generation match, Azure IfMatch — and use it everywhere a delete is guarded by a stored stamp: the write-back daemon's delete, rename-source, and pending-rename paths, and the replayed-delete sweep. Backends without it keep the guarded stat-then-delete.

* filer: restamp the remote entry after a metadata-only remote update

Changing Content-Encoding rewrites the object in place on S3 and advances its LastModified, while the recorded stamp keeps the old mtime. A version-checked delete would then mistake the object for a replacement and keep it forever. Re-stat after UpdateFileMetadata so the saved stamp describes the live object.

* filer: mirror the remote-delete ownership checks in the replayed sweep

The sweep deleted swept children's remote objects on the listing's snapshot
alone. Two gaps: a remote-only entry that took the path since — one a commit
record may not cover, such as a lazy fetch materialized from the remote —
reads the object as its only copy, and an entry removed while its rename
copy was pending leaves the object at the source key orphaned. The per-child
delete now runs under the child's entry lock with the commit-time check
repeated, keeps a remote-only entry that holds the path, and releases the
pending rename's source under the same ownership and stamp conditions the
write-back daemon's releasePendingSource applies.

* remote.sync: restamp only the object the metadata update touched

The live stat after UpdateFileMetadata was saved unconditionally, so a
replacement written between the update and the stat stamped the entry with
an object it never described — and a later delete would then pass the
freshness guard on the replacement's data. The restamp now requires the live
object to still match the stored stamp on size and ETag; a different object
keeps the old stamp, which the delete guard reads as replaced and keeps.

* filer: skip unbudgeted reader-cache test on 32-bit

TestReaderCacheUnboundedWithoutBudget proves the reader cache has no
implicit memory budget by keeping 96 concurrent 4 MiB downloads in flight
(>256 MiB). On linux/386 the weed/filer test binary already runs near the
4 GiB address-space limit, so the burst reliably crashes it with
"out of memory" mid-suite.

* filer: bail out of a swept child lock instead of waiting on it

Replay already holds the event's path locks when the sweep reaches a
descendant. A local rename can hold that descendant while waiting on one
of the replayed paths, so blocking on the child lock deadlocks both. The
sweep now tries the lock and reports ErrLockContended — transient, so the
replication retry replays the event once the local write settles.

* remote: report the stamp a metadata update produced

A metadata write can rewrite the object it touches — S3 changes
Content-Encoding through an in-place copy, which can mint a new ETag —
so restamping from a fresh stat can reject the very object the update
produced, or describe a replacement that landed in between.
UpdateFileMetadata now returns the RemoteEntry of the object it wrote
(S3 from the copy result, GCS from the update attributes, Azure from a
post-update stat); nil means the object was untouched and callers keep
the stored stamp.

* remote: restamp Azure from the metadata write's own etag
2026-10-18 12:23:30 -07:00
eea3f232da iam: scope CreateServiceAccount's ParentUser by policy resource (#11736)
* iam: scope CreateServiceAccount's ParentUser by policy resource

AuthorizeIamAction evaluated the iam:CreateServiceAccount grant against a
hardcoded arn:aws:iam:::* resource, so the policy's Resource never had a
say; #11218 then bound the target to the caller outright. Evaluate
self-target actions against the target identity's ARN instead: a wildcard
resource keeps the deliberate any-target grant, a scoped resource confines
issuance to the named parent identity, anything else is denied.

buildResourceARN gains the ARN pass-through buildS3ResourceArn already has,
so iam: ARNs reach the policy engine intact on the attached-policies route.

Closes #11735

* ci: retrigger after unrelated weed/storage metrics flake

* iam: fold IAM-ARN matching into awsIAMMatch, drop debug test

- remove zz_debug_test.go (local debugging leftover)
- fold awsIAMMatchResource into awsIAMMatch: the arn:aws:iam:: value
  prefix selects case-sensitive matching, so no separate name is needed
- trim comments

---------

Co-authored-by: batonogov <batonogov@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-18 11:48:01 -07:00
Chris Lu 4a57432488 s3: list children of the directory a ListObjects marker names (#11805)
* s3: resume into the directory a listing marker names

* s3: keep delimited marker resumes exclusive

* s3: descend a marker-named directory without repeating its key

* s3: emit marker-named dir objects but keep internal folders hidden
2026-10-18 11:44:29 -07:00
Jay HemnaniandChris Lu 006d4150f0 webdav: create a folder under its own name (#11810)
* webdav: create a folder under its own name

* webdav: reuse NormalizePath for the MKCOL entry split

---------

Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-18 11:43:19 -07:00
Chris Lu ab39e3a51b Rust volume: stamp ExpireAtSec into the live volume_info on .vif persist (#11813)
The worker fix reads the source's TTL destroy time through
ReadVolumeFileStatus, which returns the in-memory volume_info. Go's
SaveVolumeInfo stamps volumeInfo.ExpireAtSec before writing; Rust's
save_vif wrote it only to the serialized file, so a Rust source kept
reporting the stale creation-time value and the EC .vif would carry a
wrong expiry. Stamp the live volume_info first, mirroring Go and
save_volume_info.
2026-10-18 04:29:48 -07:00
Chris Lu 8c55425d55 s3tables: tighten legacy OIDC subject re-anchoring (#11808)
* sts: recognize issuer-scoped parent user ids

ComputeParentUser output has a fixed shape (base64url SHA-256); let
callers distinguish it from bare subject strings.

* s3tables: keep legacy owner re-anchoring off local and scoped account ids

Re-anchoring a stored bare subject under the single trusted issuer must
only apply to strings that could actually be a legacy OIDC subject. A
stored owner naming the reserved or a configured local account, or an
already issuer-scoped id, can otherwise be claimed by any caller whose
token asserts that string as sub.

* s3tables: re-anchor legacy policy principals in map and array forms

A stored bare subject can appear as {"AWS": sub} or inside a principal
list, not only as a top-level string; re-anchoring only the string form
left legacy Deny statements unable to match the caller.

* s3tables: keep built-in legacy exclusions independent of the account checker

A missing local-account predicate must not disable re-anchoring entirely:
the reserved accounts, ARN-shaped values, and scoped-id shapes are never
OIDC subjects regardless, while legacy Deny statements for plain subjects
still need to resolve so a denied caller cannot fall through to an
identity grant.

* s3tables: exercise legacy Deny against an owner grant

The previous case denied the caller regardless of whether the statement
principal matched; make the caller the owner so a missed match shows.
Cover the array Deny form alongside the AWS map.
2026-10-18 02:43:03 -07:00
Chris Lu 88d9926b35 s3api: recheck suspended-version condition under the lock when the filer ignores condition_alt_key (#11806)
* s3api: recheck condition under lock when routed filer ignores condition_alt_key

A filer that predates condition_alt_key silently drops the field and
evaluates a suspended-versioning condition against the empty null slot
alone, so a matching If-Match still gets a false 412. Recheck under the
object lock instead of trusting that response: the lock path resolves the
same entry the read path serves. Genuine precondition failures still
return 412 from the recheck.

* s3api: cover suspended If-Match against a filer ignoring condition_alt_key

* s3api: resend against the resolved entry instead of the lock-path recheck

Rechecking under the gateway object lock does not serialize against
routed writers, which commit under the filer's path lock; a concurrent
routed PUT could land between the recheck and the create and be
overwritten with a stale precondition. Resend the same transaction with
condition_key set to the resolved version file so the condition and the
write stay under the same filer lock. When the resolved entry is the
null slot, a delete marker, or unresolvable, the resend targets the
object path and fails closed.
2026-10-18 02:23:00 -07:00
Chris Lu e596be10f0 s3api: evaluate object-tag policy conditions on delete, ACL and lock operations (#11807)
* s3api: resolve the target version in tag-condition policy lookups

* s3api: apply tag-conditioned policy to object deletes

* s3api: apply tag-conditioned policy to object ACL reads and writes

* s3api: apply tag-conditioned policy to retention and legal hold

* s3api: cover tag-conditioned delete, ACL and object-lock rechecks

* s3api: let tag-conditioned grants redecide denied batch deletes

* s3api: resolve the tagged version in delete policy tests
2026-10-17 22:30:32 -07:00
Chris Lu 2ff7888546 filer: honor a storage rule's rack/data-center constraint instead of falling back (#11804)
filer: stop widening a storage rule's location on assign failure
2026-10-17 22:28:53 -07:00
Nabil RagabandClaude Opus 5.5 6ab6926c94 s3: read chunks from HTTPS volume servers with the global HTTP client (#11800)
readChunkRange sent the volume lookup's http:// URL through a bare
http.Client, so with [https.client]/[https.volume] configured every read of
an SSE-KMS object larger than one chunk (and every ranged read through this
path) failed with "client sent an HTTP request to an HTTPS server" and the
GET returned 500. Normalise the URL's scheme and use util_http's global
client, which carries the [https.client] TLS configuration, as the other
volume reads already do. Drop the now-unused volumeServerHTTPClient.

Fixes #11799

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 21:37:23 -07:00
Chris Lu e6a0534e3b s3api: make ambiguous routed-PUT recovery atomic and stop transaction replay (#11787)
* filer_pb: map PRECONDITION_FAILED to a sentinel error

A conditional CreateEntry that the owner's path lock refuses comes back as
FilerError_PRECONDITION_FAILED; mapping it to ErrPreconditionFailed lets the
S3 gateway tell a rejected precondition apart from a transport failure and
report 412 instead of retrying the write.

* s3api: make the ambiguous-PUT recovery create conditional under the lock

createAfterAmbiguousRoute verified state and then issued an unconditional
create, so a write landing between the lookup and the create was still
clobbered by the stale re-commit (issue #11631). The recovery create now
carries a WriteCondition — IF_NOT_EXISTS for a proven-absent entry,
IF_ENTRY_EQUAL for a directory it may need to write onto — which the owner
filer evaluates under its per-path lock, making the check and the write
atomic. A refused precondition loops back to re-read the state instead of
overwriting whatever won the race.

* s3api: do not fail over a routed transaction past a possibly-sent request

withFilerClientFailover retried the callback on the next filer on any
transport error, so an ObjectTransaction whose response was lost could be
re-applied over a write that committed in between (issue #11631). A gRPC
error cannot distinguish "never sent" from "committed", but channel
readiness can: a non-replayable callback now runs only on a connection that
just reached READY, and failover continues only over channels that never
came up — the one failure mode that provably sent nothing.

* s3api: run ambiguous-route recovery for empty PUTs too

The recovery was gated on a nonempty chunk list, so a zero-byte PUT whose
routed transaction lost its response still fell back to an unconditional
create that could overwrite a newer write and delete its chunks. Chunk-count
mismatch on an empty upload list is enough to distinguish a superseding
entry from this PUT's — there is nothing to save by skipping the check.

* s3api: keep conditional recovery creates off the replay path

The conditional create in the ambiguous-PUT fallback rode the replayable
WithFilerClient failover, so a committed create whose response was lost
could be re-sent to another filer — where a delete in between would let
IF_NOT_EXISTS pass again and resurrect an entry pointing at reclaimed
chunks. Conditional creates now run through the non-replayable failover;
an ambiguous result falls through to confirmCreateLanded, which resolves
the committed entry rather than resending.

* s3api: do not mark a filer unhealthy for an answered request

A reply that reached the callback — including an application-level failure
like a refused write precondition — proves the filer is alive. Checking
received before the health bookkeeping keeps definitive application errors
from tripping the failure circuit or flagging a live routed owner as
unreachable.

* s3api: do not claim an empty stored entry as an ambiguous PUT's own

Two chunkless entries compare equal under sameFileChunks, so an older
empty object satisfies "the route landed" even though it predates this
write — recovery would return success while leaving the request's
Content-Type and metadata unapplied. A landed PUT stores the entry
exactly as built, so require mime/size/md5/content and the extended
attributes to match before claiming it; anything else stays uncertain
and the client retries.

* s3api: prove ambiguous empty PUTs by write id and pin recovery to the owner

- Conditional recovery creates are sent to the routed owner alone instead of failing over: the condition only serializes with routed writes under the owner's per-path lock, and another filer's forwarding is only as correct as its ring view. An unreachable owner now yields an uncertain result rather than a create under a different serialization boundary.
- A proven-absent path no longer satisfies an empty, chunkless PUT: the routed commit may have landed and been deleted, and there are no chunks to verify, so IF_NOT_EXISTS could resurrect a removed object. Recovery returns uncertain for the client to retry.
- Chunkless identity moves from metadata equality to a per-write id (x-seaweedfs-write-id) stamped into Extended: an identical older object can match mime/md5/content but not the id. Entries without this write's id are not claimed, and confirmCreateLanded can now positively confirm a committed empty object instead of always reporting failure.
2026-10-17 21:36:39 -07:00
Chris Lu e6633146d8 filer: shrink zip spool threshold in tests to fix 32-bit OOM (#11801)
* filer: shrink zip spool threshold in tests to fix 32-bit OOM

TestSpoolZipReaderTempFile and TestSpoolZipReaderLimit each allocate
~33-70 MiB to cross the 32 MiB in-memory spool threshold. On linux/386 the
weed/filer test binary already runs near the 4 GiB address-space limit, so
these allocations reliably crash it with "out of memory" mid-suite.

Make zipSpoolMemoryLimit a var and drop it to 1 KiB in those tests, which
exercises the same memory-vs-temp-file spool paths with KiB-sized buffers.

* s3 lifecycle: update stale runShard and WalkerFunc call sites in tests

Recent master changes added a walks param to runShard (#11797) and made WalkerFunc take []int, but replay_view_test.go and run_bounded_test.go still use the old signatures, breaking go vet and test builds on merge.
2026-10-17 20:30:21 -07:00
Chris Lu fe8221c453 fs.meta.load: wait for in-flight entry creation before returning (#11776)
* fs.meta.load: wait for in-flight entry creation before returning

On io.EOF and error paths the command returned while CreateEntry
goroutines were still running, so trailing entries could be silently
dropped once the shell exited. Wait for them on every exit path and
guard the shared error with a mutex.

* fs.meta.load: surface worker errors on read/unmarshal failure paths

The ReadFull and proto.Unmarshal error paths waited for in-flight
CreateEntry workers but returned only the local error, hiding a failed
create from the caller. Check loadErr after wg.Wait() like the other
exit paths so the worker error is reported.

Extract the load loop into loadEntries so the concurrency and error
propagation can be exercised by unit tests, and add regression tests
covering delayed creates at EOF and on corrupt input plus worker error
reporting.
2026-10-17 20:28:43 -07:00
github-actions[bot] e32d8ca2a6 docs: regenerate star history chart 2026-10-18 00:47:40 +00:00
Chris Lu 602ddb31f2 s3: count stored versions as bucket contents on delete (#11775)
* s3: count stored versions as bucket contents

A <key>.versions directory holding version entries is user data; skipping
it let bucketHasUserObjects report an empty bucket while the recursive
delete dropped every stored version.

* test: rename stale reserved-folders case in bucket delete test

.versions directories are now descended into, so the case only holds
because the folder is empty.
2026-10-17 15:25:56 -07:00
Chris Lu 44bb697495 s3api: hydrate runtime policies on propagated IAM updates (#11784)
* s3api: hydrate runtime policies on pushed identity upserts

PutIdentity propagation (used on every IAM write when the credential
store has a master client) carries the persisted identity proto. That
proto records inline policies only as flattened legacy Actions; their
runtime __inline_policy__ names live in the policy store and were
dropped by the merge. For the interval until the next full reload, an
identity whose policy denied e.g. DeleteObject routed to legacy
CanDo, where "Write:bucket" authorizes deletes.

Hydrate the pushed identity the same way a full load does, so the
merged state always carries its runtime policy names and documents.

* s3api: hydrate runtime policies on pushed group upserts

PutGroup propagation has the same gap as identity pushes: the wire
proto does not carry the group's runtime __inline_group_policy__ names,
so members' attached-policy check lost them until the next reload.

* s3api: attach only runtime inline policy names on pushes

Listing the pushed identity's or group's own inline policies is enough
to keep it on the attached-policy route; a whole-store hydration per
pushed proto is not. Pushes now never write iam.policies, so a stale
read can no longer overwrite a newer committed policy document.

* s3api: carry inline policy documents on pushed upserts

Attaching only runtime policy names left a window where a pushed identity or group referenced a document the local cache had never loaded, denying allowed requests until the next reload. Fetch the pushed entity own inline documents and publish them with the update; reads stay scoped to the pushed entity rather than the whole store, and inline runtime names are only ever populated from this store so a read cannot install a document older than the committed one. Resync the IAM manager only when a push actually installs documents.

* s3api: order pushed inline policy reads against their installs

A pushed UpsertIdentity or PutGroup read inline policy documents from
the credential store and installed them without holding the ordering
guard that full reloads already take, so a delayed push could overwrite
a newer committed document with its stale read and restore a revoked
grant until the next update or reload.

Pushes now hold reloadMu across the store read and the install, the same
snapshot-through-commit serialization the credential-manager load uses.
MergeS3ApiConfiguration also applies config.Policies to the live
iam.policies at commit instead of swapping in its earlier snapshot, so a
policy installed while a merge prepared (a pushed inline document or a
PutPolicy update) is no longer discarded.
2026-10-17 15:25:00 -07:00
74ed12d6b3 s3 lifecycle: replay only the rules that fit the retention window (#11789)
* s3 lifecycle: replay only the rules that fit the retention window

runShard handed drainShardEvents the full snapshot, so rules the retention
window assigns to the walk were also matched during meta-log replay. Their
matches are not due yet, and drainShardEvents freezes the cursor at the first
event with a not-yet-due match. With a 3-day and a 30-day expiration rule on one
bucket and RetentionWindow of 8 days, every shard's cursor stopped at the first
30-day object's PUT and stayed there until that object came due. Each pass
then re-read the log from that point and re-dispatched every 3-day expiration
already handled, which surfaces as NOOP_RESOLVED dispatches, a growing
cursor_lag_max, and passes that still report status=ok.

Snapshot.ReplayView drops the replay-kind actions promoted to the walk and
keeps every other action as compiled, and the drain uses it. Promoted rules
are served by the walk as before. RulesForShard, PromotedHash and ReplayView
share one isPromoted predicate. With RetentionWindow unset no active rule is
promoted, so behaviour without a configured window is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: scale the replay-view test with DaysToDuration

The test took its retention window from DaysToDuration but dated its PUTs
in hours, so under the s3tests build tag (one lifecycle day = 10s) every
event fell before the seeded cursor and replay never ran. Express the
event ages in DaysToDuration too, so the test exercises replay at both
scales.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3 lifecycle: rehabilitate scan_only mode in the replay drain view

ReplayView shared the base CompiledAction pointers, so a replay-kind
rule locked into ModeScanOnly by a prior compile stayed invisible to
router.Route's event_driven gate even when the runtime retention window
kept it in the replay partition - and it is absent from the walk view
too, so the rule would be served by neither.

Clone non-promoted replay-kind actions the way RulesForShard does
(Mode forced to event_driven), reactivating the scan_only case whose
bootstrap rendezvous already completed, while preserving the base
active bit for genuinely pre-bootstrap actions. Disabled actions are
excluded for parity with the other views, and DESIGN.md now documents
the view.

* s3 lifecycle: keep drain activation observable, require walker with retention

ReplayView now shares pointers for event-driven actions so a mid-pass
MarkActive flip remains visible to the drain; only a scan_only-locked
action is cloned to event_driven, and reactivated only when its
bootstrap rendezvous completed (tracked via a new bootstrapComplete
field on CompiledAction).

validate() rejects RetentionWindow without Walker: promoted rules
would lose their only dispatch path while the drain cursor still
advanced past their events.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-17 15:24:35 -07:00
Chris Lu 443ae1cd83 s3 lifecycle: share daily bucket walk and prune listings (#11797)
* s3 lifecycle: share the daily bucket walk across shards

The daily pass invoked the walker once per shard, and every shard's
walk listed the full bucket tree while keeping only its own 1/16 of
the entries — 16 full listings per pass by default (#11791).

Run now groups the shards by the walk branch they'll take (recovery
vs steady, using the same cursor data and hash checks runShard applies)
and starts one shared walk per group, so a pass lists each bucket at
most twice. WalkerFunc and WalkBuckets take a shard set instead of a
single shard. runShard keeps its own decision and waits on its group's
walk; a shard whose freshly-loaded cursor disagrees with the plan falls
back to a per-shard walk, so mid-pass cursor rewrites can't under-cover
a shard.

* s3 lifecycle: skip the .versions/ pre-pass on non-versioned buckets

walkBucketTree listed every directory twice — once looking for
.versions/ dirs, once for regular entries. On a bucket that never had
versioning enabled the first pass can only find coincidentally-named
user folders, so the extra listing is pure cost (#11791).

FilerListFunc now takes the walk view and consults
Snapshot.BucketVersioned; non-versioned buckets skip pass 1 entirely.
A *-suffixed .versions user folder on such a bucket is recursed into
as a regular directory so real objects inside still surface; a nil
view keeps the two-pass behavior.

* s3 lifecycle: prune walk directories to active rule prefixes

The daily walk listed every directory under a bucket even when no
rule prefix could reach it. Collect the active rule prefixes of each
bucket from the walk view (new Snapshot.BucketPrefixes) and skip
recursing into directories whose keys can never match; a bucket
with no active actions is skipped entirely, saving the listing.

.uploads stays in scope regardless of prefixes because AbortMPU
rules match DestKey, and .versions expansion is pruned only
conservatively (the logical key is a string-prefix of the dir key).

* s3 lifecycle: join planned walks before Run returns

planWalks starts a shared walk goroutine per group, but a shard whose
authoritative cursor load then fails exits runShard without waiting on
it. Run returned while the walk still listed or dispatched deletes
against connections the caller closes on return. Join every planned
walk after the shard waitgroup.

* s3 lifecycle: treat object-locked buckets as versioned in the walk

Object Lock requires versioning and every writer sets both extended
attributes, but IsBucketVersioned only read ExtVersioningKey. A locked
bucket missing that attribute would have its .versions dirs walked as
ordinary folders, hiding version siblings from lifecycle. Classify
ExtObjectLockEnabledKey as versioned too.
2026-10-17 15:21:53 -07:00
Chris Lu bc29ffbfbd helm: apply placement values to the chart's hook Jobs (#11795)
* helm: honor placement values in the bucket creation hook

The post-install/post-upgrade bucket Job ignored nodeSelector,
tolerations and affinity, so it could land on nodes the operator
kept clear, or fail to schedule at all on a fully tainted cluster.

The hook now accepts nodeSelector, tolerations and affinity under
each createBucketsHook (filer.s3, s3, allInOne.s3), read from the
same values section that supplies createBuckets. Unset nodeSelector
and tolerations fall back to the workload the hook already borrows
its security context from; affinity is explicit-only since the
anti-affinity rules of a long-lived server do not suit a one-off
Job.

* helm: honor placement values in the volume resize hook

The pre-install/pre-upgrade resize Job inherited nothing from the
volume server, so on a cluster whose storage nodes are all tainted
the Job could not be scheduled and the upgrade timed out.

The hook now falls back to the volume server's nodeSelector and
tolerations, and accepts explicit nodeSelector, tolerations and
affinity under volume.resizeHook. Affinity stays explicit-only: the
default podAntiAffinity against component=volume would keep the Job
off every node already running a volume pod, which is most of them
during an upgrade. Placement strings are rendered through tpl with
$volumeName bound, same as the StatefulSet.
2026-10-17 12:40:30 -07:00
Chris Lu 01d8160323 mini/server/filer: let embedded s3/sftp gateways use their own -dataCenter (#11798)
* mini: stop forcing -dataCenter onto the embedded S3 gateway

-dataCenter names the embedded volume server's data center, but it was
also copied into the S3 gateway options, overwriting an explicit
-s3.dataCenter and making every S3 write carry an explicit dataCenter
pin in its assign request. That pin outranks fs.configure dataCenter
rules, so a mini node in one DC could never steer S3 writes to volume
servers in another.

With the copy removed, assign requests carry no DC by default: the
filer's own -dataCenter still provides the write fallback, and
fs.configure rules now apply to S3 traffic. -s3.dataCenter still pins
reads and writes when set explicitly.

* server: give embedded s3/sftp gateways their own -dataCenter flags

-dataCenter names the embedded volume server's data center, but it was
also copied into the S3 and SFTP gateway options, and neither gateway
had a flag of its own to override it. The injected DC lands in every
S3 assign request as an explicit pin, outranking fs.configure
dataCenter rules.

Add -s3.dataCenter and -sftp.dataCenter (same names as weed filer) and
stop copying -dataCenter into them, so gateway writes go through the
filer's rules and defaults unless a gateway DC is set explicitly.

* filer: stop defaulting embedded gateway dataCenter to -dataCenter

The embedded S3 and SFTP gateways inherited -dataCenter whenever their
own flag was unset. That made every S3 assign request carry the DC as
an explicit pin, outranking fs.configure dataCenter rules, and left no
way to let rules steer writes while -dataCenter was set.

Let -s3.dataCenter and -sftp.dataCenter stay empty unless set, matching
standalone weed s3 and weed sftp. The filer's own -dataCenter still
provides the write fallback.
2026-10-17 12:38:35 -07:00
Chris LuandDevin 8d9c7e76a5 sftp: enforce file ACLs on rename targets (#11770)
* sftp: check delete permission on an existing rename target

renameEntry authorized only the source path and the target directory,
so renaming over a file with a more restrictive ACL destroyed it even
though Remove on the same file was denied.

* sftp: deny a rename that would strand a carved-out ACL

A rename moves data without moving its path-based ACL: renaming an
ancestor directory left a more specific read-only entry behind, so the
relocated file fell under a permissive ancestor ACL and became
writable. Deny the rename when a configured ACL under the source would
grant less write/delete access than the relocated path would get.

* sftp: cover inherited and destination-side ACLs on rename

The rename ACL check compared only configured paths under the source,
so it missed restrictions inherited from ancestors above the source and
deeper grants configured under the destination. It now compares the
moved root plus every configured path under either tree mapped to its
source counterpart, and skips paths whose entries do not actually move.
The target delete check no longer depends on an existence lookup, so a
protected target created during a rename cannot be overwritten; a no-op
rename skips it. Root bypasses the comparison like every other check.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* sftp: fail closed on rename ACL lookups and restore new-name renames

- checkRenameKeepsAcl returns lookup errors instead of treating them as a missing entry, so a transient filer failure cannot skip the ACL comparison; it also compares read and list gains, not only write and delete, and evaluates each pair with the moved entry's own IsDirectory.

- The target delete check now distinguishes an existing target (still requires delete) from a fresh name (no delete needed) while still enforcing a configured ACL on the target name itself, so a concurrent create cannot slip a protected target under the rename.

* filer: add fail_if_target_exists option to AtomicRenameEntry

Callers that authorize a rename while the target is absent cannot see a
target created in the gap before the move runs. The flag makes the filer
reject the rename when the target exists, checked under the rename path
locks that CreateEntry on the same filer also takes.

* sftp: guard rename against concurrent targets and file-child ACL gains

A rename onto a name that inherits its directory's ACL still needs no
delete access, but now asks the filer to fail if a target appears before
the move, so a concurrently created file cannot be overwritten without
delete permission. Moved directories are also compared in the file ACL
shape, where list no longer counts as read, so their file children cannot
gain read access through the relocation.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-17 12:36:20 -07:00
Chris Lu 887b1aac76 ec: carry the source volume's TTL expiry into the worker-written .vif (#11777)
* ec: carry the source volume's TTL expiry into the worker-written .vif

ReadVolumeFileStatus now reports the volume's ExpireAtSec (last write +
TTL), and the admin worker stamps it into the .vif it distributes with
the shards. Previously every worker-encoded .vif had expireAtSec 0, so
EC volumes of TTL collections never expired.

* ec: report expiry from the Rust volume server and cover the .vif carry

The seaweed-volume drop-in server omitted expire_at_sec from its
ReadVolumeFileStatus response, so worker-encoded volumes sourced from it
lost the TTL. Fall back to the nested VolumeInfo expiry for servers that
predate the top-level field, and pin the nonzero/zero carry with a test.
2026-10-17 12:34:55 -07:00
Chris Lu ac996c0ad1 filer: shrink zip spool threshold in tests to fix 32-bit OOM (#11796)
TestSpoolZipReaderTempFile and TestSpoolZipReaderLimit each allocate
~33-70 MiB to cross the 32 MiB in-memory spool threshold. On linux/386 the
weed/filer test binary already runs near the 4 GiB address-space limit, so
these allocations reliably crash it with "out of memory" mid-suite.

Make zipSpoolMemoryLimit a var and drop it to 1 KiB in those tests, which
exercises the same memory-vs-temp-file spool paths with KiB-sized buffers.
2026-10-17 11:43:55 -07:00
Chris Lu 1155e65fc5 admin: regenerate the gzipped static mirror
admin.js changed in 161310d83 without rerunning go generate, leaving
static_gz/js/admin.js.gz stale and TestStaticGzMirror failing on master.
2026-10-17 10:49:51 -07:00
joma1022 161310d834 admin: fix Create Bucket sending POST /api/s3/buckets twice (#11790)
admin: fix Create Bucket submitting twice

   admin.js and the Buckets page template both bound a submit listener to #createBucketForm, so one click sent POST /api/s3/buckets twice. Keep only the page handler.
2026-10-16 23:47:48 -07:00
Peter Dodd 42ff535511 vacuum: bound concurrent commits per volume server (#11658)
CommitCompact holds the volume's write lock for the whole commit, and
reads of that volume wait on it. Volumes compacted together tend to
finish together, so their commits land at once and their fsyncs
contend: each commit's lock hold grows with how many run alongside it,
even when the disk is far from its throughput limit.

Add -compactionMaxParallelCommits (volume.compactionMaxParallelCommits
for server/mini). A commit waits for a slot before CommitCompactVolume,
so a queued commit holds no volume lock and reads keep flowing. The
wait counts against the caller's deadline and returns the context
error if it expires; a slot granted to an already-expired context is
handed back, so a request the caller gave up on never starts a commit,
and maintenance mode is checked again once the slot is held.
The slot is released whether the commit succeeds or fails. Compaction
itself is not gated, so -maxParallelVacuumPerServer still sets reclaim
throughput. 0, the default, keeps today's behaviour.

The limit sits in the volume server so it covers every commit caller:
the master's vacuum sweep, the vacuum worker task and volume.tier.compact.

The Rust volume server mirrors the flag and behaviour: queued commits
wait on a semaphore keyed to the grpc-timeout deadline, hold no store
lock, recheck maintenance mode once a permit is held, and release it
on every exit path.
2026-10-16 23:46:36 -07:00
Chris Lu 42ff7f2b8e s3: report an empty probed directory as a common prefix, not a key (#11779)
The "<dir>/" existence probe emitted the empty directory as a
FolderMimeType Contents entry, so delimited listings returned a fake
readable key that breaks clients treating every key as an object.
Delimited probes now report the directory as a CommonPrefix -- which
the probe accepts for existence -- while the recursive (no delimiter)
probe keeps the directory marker.
2026-10-16 23:45:37 -07:00
Chris Lu 3e55971e0e ec.shard.unmount: treat same-node shard copies as one target (#11778)
A shard held on two disks of one server produced two identical live
ShardRefs, so resolution reported "ambiguous" even for a fully
node-qualified -shardId. Ambiguity now means the shard lives on more
than one node; duplicate refs on the same node resolve to it.
2026-10-16 23:45:10 -07:00
3c6700c310 volume balance: stop planning moves onto disk types a server does not have (#11762)
* fix(topology): capacity queries no longer create empty disk-type entries

AvailableSpaceFor and CapacityFor called getOrCreateDisk, so asking a
node whether it had room for a disk type it lacks created a zeroed
DiskUsageCounts entry for that type. Volume growth walks every data
center, rack and data node, and ToDataNodeInfo publishes diskUsages, so
every node ended up advertising an empty disk (MaxVolumeCount 0,
DiskId 0) of every type ever grown: HDD-only servers showed an "ssd"
disk and SSD-only servers an "hdd" disk.

Read the counters through a non-creating getDisk; a missing entry
answers 0, the same numbers as before.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq

* fix(admin): don't let an empty disk-type entry shadow a node's real disk

UpdateTopology keys disks by node and disk id. The master published an
empty entry (no capacity, nothing on it, DiskId 0) for every disk type a
capacity check had asked about, and it collided with the node's real
disk 0; map iteration order decided which survived. An SSD-only node
intermittently showed an empty HDD disk, and an HDD node lost its disk
-- and with it its volumes from the volume index, hence "Could not find
volume N on source server" in balance detection.

IsShadowedEmptyDiskEntry matches an empty entry whose disk id another
DiskInfo of the node, one with capacity or data, already uses.
UpdateTopology and CountTopologyResources skip it. A lone empty entry
is still an empty disk of unknown capacity and is kept, as
GetDisksWithEffectiveCapacity expects.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq

* fix(balance): don't seed balance candidates from shadowed empty disk entries

detectForDiskType registered every node with a DiskInfos entry of the
type as a zero-volume candidate. An empty entry the master made up for
a type the node lacks made that node the least-loaded destination, and
its zero max switched the whole disk type to raw-count balancing. The
volume server then refused every move: "no space left hdd" / "no space
left ssd".

Skip entries IsShadowedEmptyDiskEntry matches when seeding candidates.
A server that holds volumes of the type but reports no max still enters
through the metrics and still triggers the raw-count fallback.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq

* fix(admin,balance): ignore made-up disk-type entries at any disk id

The made-up entry is published with DiskId 0, but it need not collide
with a real disk: when location 0 is unavailable (not reported) the
node's usable disk is location 1. IsShadowedEmptyDiskEntry required a
collision, so such an entry survived both UpdateTopology -- where the
unknown-capacity fallback offers it to planners -- and balance
detection, which planned moves onto it again.

Its disk id cannot tell it apart: a DiskInfo with no volumes, no EC
shards and no per-disk max is published with DiskId 0 whichever disk
it stands for. IsPhantomDiskTypeEntry (renamed, since nothing needs to
be shadowed now) matches an empty entry on any node that has a disk of
another type with capacity or data. A real disk looks the same only
while it offers no volume slot, when it can take no volume either. A
node whose entries are all empty keeps them as disks of unknown
capacity, as before.

Reported by review on #11762.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq

* test(balance): assert where the real disks land in the empty-entry test

Check through GetNodeDisks that each server's real disk sits at the
subtest's disk id before planning. The id-1 case already landed on
physical disk 1 (SplitByPhysicalDisk folds all-zero record ids onto the
entry's DiskId), so this only makes the setup explicit; on the first
revision of the fix it shows the real disk at 1 beside the made-up
entry at 0.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Et288CnR1mxDyQNdYNMBNq

* fix(topology,balance,ec): don't publish or trust usage-only disk-type entries

ToDataNodeInfo started from the raw disk-usage counters, so a node that
was asked for capacity of a disk type it lacks kept advertising an empty
DiskInfo for that type even though no such disk was ever reported. Skip
usage entries that have no disk child and hold all-zero counters.

The same stale entries still misled planners that read DiskInfos
directly: the balance plugin's move precondition treated a phantom entry
as proof a target has that disk type, and the EC balancer counted it as
a matching disk. Filter them with IsPhantomDiskTypeEntry.

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-16 23:41:26 -07:00
Chris LuandDevin 5a0be275cc s3: keep the gateway up when an auxiliary listener fails to bind (#11768)
* s3: keep the gateway up when the Iceberg or Lance listener fails to bind

The optional catalog servers ran glog.Fatalf on a bind error, so an
occupied port (e.g. the default Lance port 9101 taken by another process)
killed the entire S3 server. Log the error and run without the auxiliary
listener instead; the S3 API itself still fails hard on bind errors.

* mini: only probe and advertise catalogs whose listeners bound

With aux listener failures no longer fatal, the configured catalog port may
answer as an unrelated process; mini's readiness probe accepts any HTTP
response and would announce that foreign service as the catalog. Record the
bind outcome on S3Options (final by the time the S3 port serves, since the
aux binds happen first) and gate mini's readiness wait and admin UI
advertisement on it.

* s3: test that a failed aux listener bind is non-fatal

Hold the Iceberg and Lance ports open and assert both starters return no
server and leave the bound flags false, covering the new nonfatal path.

* command: range the endpoint test table by index

S3Options now holds atomic.Bool flags, so copying a test case trips
go vet's lock-copy check.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-16 23:10:09 -07:00
Chris Lu cabaf9c38d iceberg: collapse doubled /v1 prefix in catalog requests (#11767)
* s3: verify SigV4 against the request path the client sent

Internal routers that rewrite URL.Path (the Iceberg catalog doubled-/v1
collapse) broke signature verification: verifyV4Signature re-derived the
canonical URI from the rewritten path while the client had signed the
original one, so valid signed catalog requests were rejected. Stash the
client-sent path in the request context at rewrite time and prefer it when
computing pathForSignature.

* iceberg: collapse repeated /v1 prefixes before route matching

Move the doubled-prefix collapse from the catch-all into a middleware so it
also covers paths that match the {prefix} routes (/v1/v1/namespaces would
otherwise resolve "v1" as the table bucket; the name is unreachable anyway
since bucket names require 3-63 chars). Collapse all leading repetitions in
a single pass and re-dispatch once instead of recursing per segment, and
record the original path in context so signed requests still verify.

* iceberg: slice off repeated /v1 prefixes instead of concatenating

Stripping path[len("/v1"):] per iteration reuses the original string
instead of allocating a fresh copy of the remaining path each pass.
2026-10-16 23:01:57 -07:00
ad2486919c iam: accept AWS action names for listing multipart uploads and parts (#11765)
* iam: accept AWS action names for listing multipart uploads and parts

PutUserPolicy rejected s3:ListBucketMultipartUploads and
s3:ListMultipartUploadParts as invalid actions, so no inline policy could
grant these operations by their AWS names. Map both to Read, which is the
action their routes are registered under.

Fixes #11738

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y1b55sEL1gA1c3FCMcQCbm

* test: check each multipart listing action separately

Granting both actions in one policy let a correct mapping for one hide a
wrong mapping for the other. Test each action on its own and assert it
grants only Read.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y1b55sEL1gA1c3FCMcQCbm

* iam: scope multipart listing grants to their resolved action

A policy naming s3:ListBucketMultipartUploads or s3:ListMultipartUploadParts
projected to a Read:<bucket> grant, which the legacy route check honored for
every object in the bucket. Map the AWS names to dedicated action constants
and let VerifyActionPermission fall back to the resolved request action, so
the grant covers only the listing operation it names.

* iam: narrow legacy ListParts and ListMultipartUploads mappings

Both policy names projected to Write:<bucket>, granting full object writes
on any bucket where listing was intended. They now map to the same scoped
listing actions as their AWS names.

* iam: bind the multipart listing fallback to the listing routes

The legacy-action fallback in VerifyActionPermission resolved the
operation with ResolveS3Action, which reads the promoted authorization
object. For GET /bucket?uploads&prefix=foo/ the prefix becomes the
object, so the request resolved to GetObject and a
ListBucketMultipartUploads grant was denied. Resolving from query
priority also did not match how the router picks a handler.

Only fall back for the two listing routes, identified from the route
vars the same way the router dispatches them, and add the scoped
actions to MapToIdentitiesAction so GetUserPolicy in the standalone
IAM API rebuilds valid policy actions.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y1b55sEL1gA1c3FCMcQCbm

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-10-16 21:11:10 -07:00
Chris LuandDevin 2521c202aa s3: accept signed streaming uploads when IAM is disabled (#11766)
* s3: skip seed-signature verification in chunked reader when IAM is off

When no credentials are configured there is nothing to verify streaming
chunk signatures against. Leave the credential unset so the reader decodes
the aws-chunked body and skips per-chunk verification, matching the
unsigned-streaming path. This also fixes SigV4-signed unsigned-streaming
uploads on open gateways, which previously failed in calculateSeedSignature.

setupIam() now sets isAuthEnabled to match the credentials it installs, so
the signature-verification tests still exercise the enabled path.

* s3: decode signed streaming uploads when IAM is disabled

AWS SDK v2 clients (Trino, Spark, modern aws-cli) send PutObject bodies as
STREAMING-AWS4-HMAC-SHA256-PAYLOAD. On a gateway without S3 auth this was
rejected with ErrAuthNotSetup, breaking writes to table buckets and regular
buckets alike. The auth check already short-circuits when IAM is off, so
route both streaming payload types through the chunked reader.

* s3: cover signed trailer streaming uploads without IAM

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-16 21:10:38 -07:00
Chris Lu b0ab4c4e1e shell: make every -collection flag parse the same way (#11794)
* shell: make every -collection flag parse the same way

The filter vocabulary (comma-separated names, wildcards, regexes,
"_default" for the empty-named collection) already existed in
wildcard.CompileCollectionMatcher, but several commands still did their
own exact-match comparison or sent the raw flag value to servers.

fs.mergeVolumes used a literal string compare, so -collection=_default
silently produced an empty merge plan; it now uses the shared matcher
while keeping "*" as the match-all default. volume.fsck and volume.mark
filter through the matcher too, and volume.vacuum resolves the filter
client-side into per-collection and per-volume requests, since no
Collection value scopes the VacuumVolume RPC to the unnamed collection
alone.

Commands that put one concrete collection name on the wire normalize
"_default" to "" (volume.grow, fs.configure, mq.topic.compact,
collection.delete, ec.rebuild, ec.balance), and the tier -volumeId
commands resolve the volume's actual collection from topology first so
patterns and named-collection requests pass the volume server's
collection check.

* shell: spell the default-collection keyword via CollectionDefault

* shell: address review findings on collection filter handling

volume.fsck keeps a set of every volume the master knows before any
filtering, so a filer entry whose volume was excluded by -collection or
-volumeId is skipped instead of reported "volume not found" and deleted
by -reallyDeleteFilerEntries. The collection filter also no longer
scopes the filer walk to /buckets/<collection>: fs.configure can bind a
collection to any path prefix, and a narrower walk could mark live
chunks as orphans under -reallyDeleteFromVolume.

ec.rebuild and ec.balance resolve -collection through the new
wildcard.ExpandCollectionFilter, which passes literal names through even
when the master does not list them — ec.rebuild must be able to recover
a collection whose shards all lost their index and never registered.

fs.configure and mq.topic.compact now reject _default outright: an empty
Collection means "unset" in both the path-rule merge and the filer's
storage-option fallback chain, so the keyword would have been silently
ignored.

The unnamed-collection vacuum sweep keeps the master's low-disk-space
exception: a volume whose every replica is read-only is probed with
VacuumVolumeCheck and stays eligible when a replica reports
DiskSpaceLow, instead of being dropped outright.
2026-10-16 21:07:51 -07:00
Chris Lu 268424a5e1 s3: extract zip members on PutObject (#11786)
* filer: add zip archive extraction helpers

SpoolZipReader makes an upload body seekable for archive/zip,
SanitizeZipMemberName rejects zip-slip and non-regular members, and
CheckZipLimits caps member count and total uncompressed size.

For seaweedfs/seaweedfs#11722

* filer: extract zip members on ?extract=zip upload

POST/PUT to a directory path with ?extract=zip (or X-Seaweedfs-Extract:
zip) writes each archive member as its own entry, so a bundle of small
files lands in one request instead of one roundtrip each. Members keep
their mtime, mode and mime-by-extension; directories are created;
unsafe names are skipped and reported. extract.keep=true also stores
the archive itself.

For seaweedfs/seaweedfs#11722

* filer: bound zip spool memory by actual bytes

A chunked upload has no Content-Length, so the old size hint let an unbounded body go through io.ReadAll. Read up to the memory threshold plus one byte first, then spill the prefix and the rest to a temp file, capped at ZipExtractMaxBytes total.

* filer: guard zip member size total against overflow

Declared member sizes summing past math.MaxUint64 wrapped the running total and slipped past the 1 TiB cap.

* server: only take the inline fast path at a short first read

uploadReaderToChunks read at most chunkSize bytes, then treated any read below SaveToFilerLimit as the whole file. When SaveToFilerLimit exceeded chunkSize, longer uploads were stored truncated.

* server: resolve storage options per extracted zip member

A member can land under a storage rule more specific than the request path. Re-running detectStorageOption0 per destination applies that rule's collection, replication, TTL, filename limit, and read-only guard to files and directories alike, while explicit query overrides still win.

* server: accept raw request body for POST extract uploads

Every POST was forced through MultipartReader, so a raw zip body always got a 400. Only parse multipart when the Content-Type is actually multipart.

* server: extract zip directories before files and stop on cancel

Processing directory members first keeps their stored mtime/mode from being overwritten by parents auto-created for an earlier file member. Checking ctx.Err() between members keeps a disconnected client from driving thousands more writes.

* server: assert stored mtime and mode for extracted members

* s3: add zip extraction header constants

* s3: extract zip members on PutObject with extract header

x-amz-meta-snowball-auto-extract or x-seaweedfs-extract: zip on a PutObject fans out the archive to one object per member under the object's directory, reusing putToFiler so chunking, collections, storage rules, versioning, and bucket-default encryption apply to each member. The archive itself is not stored. Explicit SSE request headers are rejected since one encryption config cannot represent independently created member objects.

* s3: bound member fan-out by request cancellation

Stop starting new member writes once the request context is done, matching the filer extract loop.

* filer: bound compressed zip spool and reject non-local member names

The 1 TiB uncompressed limit was also capping the compressed bytes
spooled to the host's temp directory, so a handful of concurrent
extractions could fill the filer's disk before any ZIP validation ran.
SpoolZipReader now takes a separate compressed-size limit (the filer
passes its TUS max upload size) with a 5 GiB default.

SanitizeZipMemberName gains a filepath.IsLocal check: the segment loop
already rejects "..", but CodeQL only treats member names as sanitized
through a recognized guard, and the extra check also covers reserved
Windows names.

* server: honor destination rules for zip members and clamp maxMB

An x-seaweedfs-destination header picks the storage rules for a logical
destination while members still land under the request path, matching
PostHandler. Directory members now also enforce the actual path's
read-only rule instead of relying on option resolution to fail.

maxMB up to 2048 overflowed the int32 chunk size, so an oversized query
value made extraction (and ordinary autoChunk uploads) store members
empty yet report success. The parsing is shared and clamped at 2047 MB.

* s3: scope zip member writes through the per-key checks PutObject applies

The extraction loop bypassed most of what a standalone PutObject would
face: members were written without IAM authorization for their key,
directory members replaced markers without entry lock checks, and lock
headers persisted retention without the s3:PutObjectRetention or
s3:PutObjectLegalHold permissions. authorizeMember now runs IAM,
table-bucket path rules, bucket policy, lock-header validation, and
non-versioned lock enforcement per member key, and directory members
take the same guarded mkdir path as an explicit dir marker.

The archive's own Content-Md5 and x-amz-checksum-* values are verified
while the body spools instead of being silently dropped, members stop
inheriting the archive's Content-Encoding, and non-versioned members
resolve their lifecycle TTL like normal uploads.

* s3: fail malformed lock headers before spooling the archive

Per-member re-validation turned a bad object-lock header into one error
entry per member inside a 200 response. Validate once up front like
PutObjectHandler does, and keep only the dedicated retention/legal-hold
permission checks scoped to each member key.

* filer: return 413 for oversized archives and honor saveInside in extract

Map filer.ErrZipTooLarge to 413 instead of 500 for client-side archive
size violations. Extracted members written under saveInside now store
as inline entry content instead of unreachable volume chunks.

* s3: write extracted members under their own request context

Point member writes at the member key via mux.SetURLVars so SSE-KMS
builds the encryption context from the member ARN, not the archive's.
Map the spool limit error to ErrEntityTooLarge.
2026-10-16 21:07:05 -07:00
Chris Lu a8affc851d filer: extract zip members on ?extract=zip upload (#11782)
* filer: add zip archive extraction helpers

SpoolZipReader makes an upload body seekable for archive/zip,
SanitizeZipMemberName rejects zip-slip and non-regular members, and
CheckZipLimits caps member count and total uncompressed size.

For seaweedfs/seaweedfs#11722

* filer: extract zip members on ?extract=zip upload

POST/PUT to a directory path with ?extract=zip (or X-Seaweedfs-Extract:
zip) writes each archive member as its own entry, so a bundle of small
files lands in one request instead of one roundtrip each. Members keep
their mtime, mode and mime-by-extension; directories are created;
unsafe names are skipped and reported. extract.keep=true also stores
the archive itself.

For seaweedfs/seaweedfs#11722

* filer: bound zip spool memory by actual bytes

A chunked upload has no Content-Length, so the old size hint let an unbounded body go through io.ReadAll. Read up to the memory threshold plus one byte first, then spill the prefix and the rest to a temp file, capped at ZipExtractMaxBytes total.

* filer: guard zip member size total against overflow

Declared member sizes summing past math.MaxUint64 wrapped the running total and slipped past the 1 TiB cap.

* server: only take the inline fast path at a short first read

uploadReaderToChunks read at most chunkSize bytes, then treated any read below SaveToFilerLimit as the whole file. When SaveToFilerLimit exceeded chunkSize, longer uploads were stored truncated.

* server: resolve storage options per extracted zip member

A member can land under a storage rule more specific than the request path. Re-running detectStorageOption0 per destination applies that rule's collection, replication, TTL, filename limit, and read-only guard to files and directories alike, while explicit query overrides still win.

* server: accept raw request body for POST extract uploads

Every POST was forced through MultipartReader, so a raw zip body always got a 400. Only parse multipart when the Content-Type is actually multipart.

* server: extract zip directories before files and stop on cancel

Processing directory members first keeps their stored mtime/mode from being overwritten by parents auto-created for an earlier file member. Checking ctx.Err() between members keeps a disconnected client from driving thousands more writes.

* server: assert stored mtime and mode for extracted members

* filer: bound compressed zip spool and reject non-local member names

The 1 TiB uncompressed limit was also capping the compressed bytes
spooled to the host's temp directory, so a handful of concurrent
extractions could fill the filer's disk before any ZIP validation ran.
SpoolZipReader now takes a separate compressed-size limit (the filer
passes its TUS max upload size) with a 5 GiB default.

SanitizeZipMemberName gains a filepath.IsLocal check: the segment loop
already rejects "..", but CodeQL only treats member names as sanitized
through a recognized guard, and the extra check also covers reserved
Windows names.

* server: honor destination rules for zip members and clamp maxMB

An x-seaweedfs-destination header picks the storage rules for a logical
destination while members still land under the request path, matching
PostHandler. Directory members now also enforce the actual path's
read-only rule instead of relying on option resolution to fail.

maxMB up to 2048 overflowed the int32 chunk size, so an oversized query
value made extraction (and ordinary autoChunk uploads) store members
empty yet report success. The parsing is shared and clamped at 2047 MB.

* filer: return 413 for oversized archives and honor saveInside in extract

Map filer.ErrZipTooLarge to 413 instead of 500 for client-side archive
size violations. Extracted members written under saveInside now store
as inline entry content instead of unreachable volume chunks.
2026-10-16 21:04:54 -07:00
Chris LuandDevin 6da432b8bf filer: make a retried ObjectTransaction replay its recorded outcome (#11788)
* filer_pb: add request_id to ObjectTransactionRequest

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: replay recorded outcome for a retried ObjectTransaction

A failover replay carries the same request_id; under the same path lock
the owner replays the recorded outcome instead of re-applying the
mutations over whatever superseded them. Only application-level
outcomes are recorded — a lookup error leaves the request free to
re-run.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: stamp routed object transactions with a request id

One id per logical transaction so failover retries replay the recorded
outcome instead of re-applying the mutations.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: scope transaction replay tokens per path and keep them for their TTL

The dedup key includes the lock path so a token reused on a different
object cannot replay this outcome. Records are never evicted early —
an early eviction would let a replay re-apply — so the expiry sweep is
amortized and the map is bounded by the insert rate over the TTL.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: bound the transaction replay cache at capacity

* filer: do not record indeterminate transaction outcomes

A mutation error can be transient or follow a partial apply; recording
it replays that failure to a retry for the whole TTL instead of letting
the same idempotent mutations converge. Only successful transactions
and precondition failures are definitive outcomes worth replaying.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: replay recorded mutation failures instead of re-applying

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-16 20:05:57 -07:00
Chris LuandDevin da3b7feeae admin: move dashboard progress labels beside the bar (#11781)
* s3: keep suspended-versioning conditional writes off the routed condition

A suspended PUT writes the null-version slot at the regular path while the
entry the conditional headers evaluate lives under .versions/. The routed
WriteCondition checked the request's etag against that empty slot, so a
correct If-Match failed with 412. Passing uniqueWritePath sends conditional
requests through the lock path, whose precondition resolves the same entry
the read path serves; unconditional writes still route.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* admin: keep the volumes progress label readable at low fill

The dashboard's Volumes and tier-usage cells rendered their text inside the
progress bar, so a low fill left white label text on the light track and the
count was unreadable. Render the label beside the bar instead.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* admin: keep a minimum width on dashboard progress tracks

A nonwrapping label beside a flex track could squeeze it to zero in a
narrow table, hiding the bar entirely.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-16 20:03:23 -07:00
Chris LuandDevin 00804b32fa s3tables: let an explicit policy Deny veto fallback and owner allows (#11772)
* s3tables: let an explicit policy Deny veto fallback and owner allows

Table operations evaluated the table or namespace policy and the bucket
policy as an OR, so an explicit Deny in one policy was overridden by an
Allow in the other. Owner and identity grants also short-circuited
before the resource policy was read. Deny statements are now evaluated
first at both levels, matching AWS explicit-deny semantics.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3tables: fail closed when a resource side has no owner

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-16 19:54:46 -07:00
Chris LuandDevin 44657988ea s3: apply destination retention on copies instead of inheriting it (#11773)
* s3: apply destination retention on copies instead of inheriting it

CopyObject carried the source object's retention mode, retain-until date
and legal hold into the destination entry, so a copy onto a bucket with
default COMPLIANCE retention produced an unlocked object that could be
deleted early. Copied objects now take their retention from the copy
request's lock headers or the destination bucket's default, matching
PUT semantics; a versioned self-copy gets the same treatment while an
unversioned metadata-only self-copy keeps the object's existing lock.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: validate lock headers on metadata-only self-copies

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-16 19:52:10 -07:00
Chris LuandDevin 2d06e2d6c1 s3: evaluate tag conditions on copy-source and tagging subresources (#11771)
* s3: evaluate tag conditions on copy-source and tagging subresources

Policy evaluation ran without the object entry on several paths, so a
Deny conditioned on s3:ExistingObjectTag never matched:

- CopyObject, UploadPartCopy and RenameObject checked the source key
  before its entry was fetched; repeat the check on a synthesized GET
  once the entry resolves.
- GetObjectAttributes and Get/Put/DeleteObjectTagging never supplied
  the entry; recheck with the resolved entry.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3: fail closed on tag-policy lookups and honor tag-conditioned grants

- recheckTagConditionedPolicyWithLookup returned success when the entry
  lookup errored, letting a tag-gated Deny slip past a transient miss;
  only ErrNotFound still falls through to the handler's own NoSuchKey.
- A read or delete denied in the tagless first phase now gets one more
  evaluation with the resolved entry's tags in hand, so an
  s3:ExistingObjectTag-conditioned Allow can still grant a copy-source
  read or rename. The deferred recheck requires an explicit Allow;
  indeterminate results keep the original verdict, and the existence of
  the key never leaks past a deny.
- RenameObject now applies the same tag-aware recheck to the source
  removal, so a tag-gated Deny on s3:DeleteObject blocks a rename the
  way it blocks a direct DELETE.
- GetObjectAttributes evaluates the object-entry policy recheck before
  conditional headers, so a 304/412 response can no longer bypass a
  tag-based Deny.

* s3: honor identity denies in the tag-aware source recheck

The deferred recheck re-evaluates only the bucket policy, so a bucket
Allow could lift an explicit Deny in the caller's attached or session
policies on a copy/rename source. Apply the same applicable-policy Deny
check the first phase uses.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-16 19:50:36 -07:00
Chris LuandDevin 446e6f3444 s3tables: scope OIDC-derived account IDs to the token issuer (#11774)
* s3tables: scope OIDC-derived account IDs to the token issuer

Table ownership keyed off the bare sub claim, so two identity providers
issuing the same sub collided on one account - a caller could act on
tables owned by a different provider's user. OIDC bearer tokens now
carry the same issuer-namespaced (sub, iss) hash STS sessions already
expose as aws:userid, and getAccountID prefers it over the raw claims.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* s3tables: keep legacy owners working and scrub client-subject claims

- OIDC bearer tokens without a subject are rejected, and a token-supplied
  aws:userid claim is always overwritten with the verified (sub, iss)
  hash, so a forged claim can no longer stand in for the principal.
- Owners written before issuer scoping hold a bare subject; permission
  checks now re-anchor that subject under the caller's issuer, so the
  same (sub, iss) pair keeps its ownership while scoped owners remain
  issuer-isolated. The token issuer is asserted into claims alongside
  sub and carried through PolicyContext.

* s3tables: anchor legacy subjects to the configured issuer set

* s3: enumerate trusted issuers from registered providers too

* s3: fail closed when the OIDC provider store cannot be listed

trustedOIDCIssuers kept the registered issuers when ListOIDCProviders
failed, presenting a partial set as complete. A deployment trusting one
registered issuer A plus a stored issuer B would, on a listing error,
report only A — the single-issuer shape under which a legacy bare
subject re-anchors — letting a caller from A claim a stored subject
that belongs to B and gain owner access.

With no provider store the registered providers are provably complete,
so keep returning them; with a store, a failed listing means the real
set is unknown, so return none.

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-16 19:48:32 -07:00
Chris LuandDevin 348116b2c4 filer: sweep the local subtree on a replayed directory delete (#11785)
* filer: sweep the local subtree on a replayed directory delete

A recursive delete emits only the directory's own key, so peers applied
it with a single DeleteEntry. Children the peer held that the origin had
not listed stayed behind unlisted but still readable, and their needles
were never deleted. Sweep the local children on a replayed directory
delete and drop their chunks when the origin deleted chunks.

* filer: walk the subtree on a replayed directory delete

DeleteFolderChildren removes only direct children in stores keyed by
immediate parent, so the sweep now lists and deletes each descendant
entry itself. Entries the local filer committed after the event survive
— with the directories that hold them — and hard-linked files keep
their chunks for the origin's own events to settle.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: key replayed subtree chunk deletes to each entry's path

The sweep queued every descendant's chunks under the deleted directory's
storage rule and skipped directory entries entirely. Chunks now collect
keyed by their owning path, so per-path rules apply to each file — and a
directory carrying its own chunks (a file promoted by an S3 child) frees
them too.

* filer: fence kept replayed dirs and flush collected chunks on error

A directory kept because a newer descendant survived still needs its
delete timestamp recorded, or a stale replayed create can resurrect
another child under it. And chunks collected for deleted children must
be flushed even when the sweep fails partway — the entries are gone and
a retry cannot list them again. Collection now also runs only after the
entry delete succeeds.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: flush swept chunk references in batches

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-16 19:47:20 -07:00
Chris LuandDevin 5b4a707c43 shell: keep volume.tier.compact replicas on the same remote object (#11783)
* shell: keep volume.tier.compact replicas on the same remote object

The compact cycle ran on a single replica: it deleted the shared remote
object up front and never touched the siblings, leaving their .vif/.idx
pointing at a removed object. It also remounted the working replica into
a writable local volume mid-operation.

Freeze all replicas first, keep the shared object while siblings still
reference it, replicate the new metadata to every replica, and let the
last tiered replica retire the old object.

* storage: surface remote object delete failures from volume destroy

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* shell: keep tier.compact replicas frozen until every copy lands

A delete between the source unfreeze and a sibling's index copy could
land past the copied snapshot, leaving the replica serving deleted
data. Unfreeze all replicas only after every resync completes; a remote
volume's readonly mark is inert, so the freeze adds no extra cost.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* storage: abort volume destroy when remote delete fails

destroyLocked removed local files even when the backend delete errored,
leaving a registered replica with no .dat/.idx and a leaked remote
object. Run the remote delete before local cleanup and bail out on
failure so the volume stays fully usable and the delete can be retried.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: mirror destroy ordering for remote delete failures

Propagate a failed remote-object delete instead of warning past it, and
reinsert the volume into the disk location registry so a still-intact
replica is not unregistered.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* shell: recover partial tier.compact freezes and report held replicas

A failed freeze left already-marked replicas readonly with nothing
changed, and later stage failures returned without saying the replicas
stay frozen or how to resume. Roll back the partial mark when freezing
fails, note in every mid-flight error that replicas remain readonly and
rerun is the recovery path, and report how many replicas a partial
unfreeze released.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* shell: retire abandoned remote objects during tier.compact resync

A resync overwrote each replica's .vif, orphaning its previous remote
object; an interrupted run could leave replicas on different keys, and
only the last tiered replica's object was deleted. Read every replica's
remote file list up front, count holders per object, and delete-before-
copy whichever replica is an object's last holder, so shared and
stranded objects alike are retired instead of leaked.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: keep collection members registered when destroy fails

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-16 19:45:17 -07:00
Chris LuandDevin d3a97f76bd s3: fix PutObject If-Match under suspended versioning (#11780)
* s3: keep suspended-versioning conditional writes off the routed condition

A suspended PUT writes the null-version slot at the regular path while the
entry the conditional headers evaluate lives under .versions/. The routed
WriteCondition checked the request's etag against that empty slot, so a
correct If-Match failed with 412. Passing uniqueWritePath sends conditional
requests through the lock path, whose precondition resolves the same entry
the read path serves; unconditional writes still route.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer_pb: add condition_alt_key to ObjectTransactionRequest

* filer: evaluate a transaction condition against a fallback entry

condition_alt_key names the entry consulted when the primary is absent,
so a suspended-versioning write can check 'the null slot if present,
else the latest version' atomically under the lock.

* s3: route suspended-versioning conditional writes with a fallback key

The gateway names the latest version file as condition_alt_key, so the
transaction checks the entry the read path serves — null slot first,
latest version when it is absent — under the same lock that applies
the write. Conditional suspended PUTs route again instead of splitting
check and write across a gateway lock that routed writes skip.

* filer: resolve the condition fallback under the transaction lock

The gateway named the pointed-at version file as condition_alt_key,
resolved before the transaction ran: a concurrent version delete could
move the latest pointer between the gateway read and the locked check,
and a delete-marker latest read as an existing object. The alt key is
now the .versions directory and the filer resolves the latest-version
pointer under the same lock that serializes the write — a delete marker
resolves to absent, and unresolvable pointer state fails closed.

* filer: fail closed on alt-key lookups and skip directory primaries

A transient FindEntry error returning a nil entry read as "versions dir
absent" and let If-None-Match writes pass. Check the error before the
entry. And a bare directory at the object path satisfied the primary
lookup, so the .versions fallback never ran — directories are not the
object, treat them as an absent slot.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: scope directory-as-absent conditions to alt-key callers

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-16 19:44:26 -07:00
github-actions[bot] cdc6976e3a docs: regenerate star history chart 2026-10-17 00:45:25 +00:00
Chris Lu 921535411b s3: default allowDeleteBucketNotEmpty to false (#11769)
* s3: default allowDeleteBucketNotEmpty to false

DeleteBucket on a non-empty bucket silently destroyed every object,
while AWS returns 409 BucketNotEmpty. Clients like 'aws s3 rb' rely on
that refusal as the 'delete only if empty' safety net.

Deployments that need the recursive delete can opt back in with
-allowDeleteBucketNotEmpty=true.

* test: pin allowDeleteBucketNotEmpty=true on versioning config1

The config1 run relied on the flag's old true default to contrast with
config2's false. Pin it so the two modes stay distinct.

* test: drop redundant allowDeleteBucketNotEmpty=false in s3 integration test

False is the default now, so TestS3DeleteBucketNotEmpty exercises the
out-of-box behavior.
2026-10-15 23:45:36 -07:00
Chris LuandDevin 651546ae74 mount: release flock locks before the close-time flush (#11726)
* mount: release flock locks before the close-time flush

Release ran the dirty-handle flush before dropping the file
description's flock locks. A flush parked on a dead filer kept a dead
holder's lock in the table, so a restarted process taking flock on the
same file saw "file already locked" until the mount restarted.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* mount: finish the blocked-flush release test deterministically

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* mount: let the release test's parked flush talk to a live filer

Marking the handle deleted raced with the flush's unlocked early read
of isDeleted. Pointing the fixture at a live fake filer lets the flush
complete on its own once the handle lock frees, no flag write needed.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-15 22:38:03 -07:00
dependabot[bot]andChris Lu 82f0a37efe build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob from 1.8.0 to 1.8.2 (#11740)
* build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob

Bumps [github.com/Azure/azure-sdk-for-go/sdk/storage/azblob](https://github.com/Azure/azure-sdk-for-go) from 1.8.0 to 1.8.2.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.8.0...sdk/azidentity/v1.8.2)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
  dependency-version: 1.8.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* test: restore loadBucketMetadataFromFiler stub after TestGetBucketMetadata

TestGetBucketMetadata replaced the package-level loadBucketMetadataFromFiler
with a stub that sleeps one second per load and never restored it. Every
test running after it in the same binary paid that sleep on each bucket
metadata cache miss: TestPutObjectUploadACL (~245s) and
TestCopyObjectDestinationACL (~46s) each created a fresh BucketRegistry per
subtest, so the stub fired hundreds of times. In CI this stretched the
Test job past 11 minutes and left it exposed to runner cancellation.

Restore the real loader via t.Cleanup so the slow stub is scoped to the
test that needs it.

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-15 21:53:16 -07:00
dependabot[bot] 9aae0bb702 build(deps): bump helm/kind-action from 1.15.0 to 1.15.1 (#11747)
Bumps [helm/kind-action](https://github.com/helm/kind-action) from 1.15.0 to 1.15.1.
- [Release notes](https://github.com/helm/kind-action/releases)
- [Commits](https://github.com/helm/kind-action/compare/v1.15.0...v1.15.1)

---
updated-dependencies:
- dependency-name: helm/kind-action
  dependency-version: 1.15.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-15 21:53:01 -07:00
dependabot[bot] 80060e1bd6 build(deps): bump github.com/a-h/templ from 0.3.1020 to 0.3.1070 (#11742)
Bumps [github.com/a-h/templ](https://github.com/a-h/templ) from 0.3.1020 to 0.3.1070.
- [Release notes](https://github.com/a-h/templ/releases)
- [Commits](https://github.com/a-h/templ/compare/v0.3.1020...v0.3.1070)

---
updated-dependencies:
- dependency-name: github.com/a-h/templ
  dependency-version: 0.3.1070
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-15 21:52:43 -07:00
dependabot[bot] 72aa17ff02 build(deps): bump github.com/aws/aws-sdk-go-v2 from 1.47.1 to 1.47.3 (#11741)
Bumps [github.com/aws/aws-sdk-go-v2](https://github.com/aws/aws-sdk-go-v2) from 1.47.1 to 1.47.3.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.47.1...v1.47.3)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2
  dependency-version: 1.47.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-15 21:52:20 -07:00
Chris Lu e6cf190596 ec: re-plan balance moves when a target reports no space left (#11759)
* ecbalancer: add Node.SetNoReceive to exclude a node as a move target

A node that has refused a shard copy at runtime (e.g. 'no space left' on the actual disk, issue #11753) can no longer be trusted as a destination for the remainder of a balance run, but it may still hold shards that must move away. SetNoReceive excludes the node from target selection and from receiving-side capacity accounting, while leaving it eligible as a move source.

* ec: re-plan balance moves when a target reports no space left

Moves are planned from a heartbeat snapshot, but concurrent copies let the volume server's real disk state diverge from it, so a planned target can refuse the copy with 'no space left' mid-run and abort the whole balance (issue #11753). Track the refusing targets during execution, mark them no-receive, re-collect the topology, and plan the remaining moves again. The loop terminates because a marked node can never be picked as a target again.

* volume server: return ResourceExhausted when no disk can take an EC shard

A failed EcShardsCopy carried code=Unknown even though the cause is exhausted local capacity; ResourceExhausted describes the refusal and lets clients distinguish it from internal errors. The message text is kept so message-based matching still works (the Rust volume server reports the same text under Internal).

* ec: replay applied moves onto the re-collected snapshot and surface failures

Addresses review on the re-plan loop: a fresh topology read can still describe an applied move while heartbeats lag, so journal each applied move and replay it onto the next snapshot; retry only when every failure was a 'no space left' refusal (an unrelated failure aborts instead of being hidden); and exclude refused nodes' slots from the free-space precheck so a balance with no remaining receiver fails instead of reporting success for an empty plan.

* ec: carry per-disk state when replaying applied moves

A move that landed on an explicit TargetDisk loses that attribution when replayed onto a stale snapshot, leaving the disk's free slots and shard count unchanged; the next plan could pick the same disk, hit another refusal, and over-exclude the node. Replay the shard onto the target disk's entry and move the disk-level counters, and restore the source disk's counters on removal.

* ecbalancer: add Node.SetDiskNoReceive to exclude a single disk as a target

A copy with an explicit disk_id can be refused while the node's other disks still have room; a disk_id 0 refusal already means the server searched every disk, so the scopes differ. disk.noReceive removes that disk from receiving capacity and disk selection while leaving the node and its sibling disks eligible.

* ec: record explicit-disk refusals as disk-scoped no-receive

recordMoveResult now distinguishes the two refusal scopes: TargetDisk>0 marks only (node, disk), while TargetDisk==0 marks the whole node since the server's auto-pick found no usable disk. The free-slot precheck subtracts refused disk slots, and the replan log names refused destinations as node:diskN.

* ecbalancer: skip no-receive disks in pickBestDiskOnNode

The balance phases pick the destination disk through pickBestDiskOnNode, which ranked disks by score only; a refused disk with a low shard count would be re-picked, fail again, and abort the run since no new destination was rejected.

* volume_move: export IsNoSpaceLeftError for move callers

Both the ec.balance replanner and the worker ec_balance task need to recognize the volume server's no-space refusal; keep one matcher next to the move RPCs instead of a private copy in weed/ec.

* worker: retry pinned-disk EC moves with a server-picked disk on no space

ec_balance detection always pins TargetDisk, so a full disk fails the job even when a sibling disk has room. On a no-space refusal to the pinned disk, retry the move once with TargetDisk 0 so the volume server picks a disk on the same node; the planner's node choice stands.
2026-10-15 18:47:07 -07:00
Youngju KIMandChris Lu ee2d98f664 mini: note mini.options and Docker filer.toml pitfalls in help (#11755)
* mini: note mini.options and Docker filer.toml pitfalls in help

Two behaviors of "weed mini" surprise users and are not mentioned in
its help text.

Flags are saved to <dir>/mini.options and applied again on the next
start to every flag that still has its default value. Removing a flag,
or passing it with its default value, therefore keeps the saved value.

The official Docker image ships /etc/seaweedfs/filer.toml, which sets
the leveldb2 store to /data/filerldb2. When filer.toml is found, the
filer ignores the default store dir derived from -dir, so moving -dir
away from /data leaves the metadata in /data. If /data is not
persisted, buckets look empty after the container is recreated.

Only the help text changes.

Signed-off-by: YoungjuKim <fjvbn2003@gmail.com>

* mini: update help notes for computed-only mini.options

---------

Signed-off-by: YoungjuKim <fjvbn2003@gmail.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-15 18:44:31 -07:00
Chris Lu 18b235da0d mini: surface saved options and warn when filer store escapes -dir (#11758)
mini: persist only computed options, and warn when filer store escapes -dir

- saveMiniConfiguration no longer writes command-line flags to
  mini.options; the file now only pins values mini computed itself
  (currently master.volumeSizeLimitMB), so a dropped flag can no longer
  resurrect on the next start. Existing files still apply once and are
  then rewritten without the saved flags.
- applyConfigFileOptions skips any flag explicitly provided on the
  command line or via WEED_* env vars, so passing a flag always wins
  over a saved value, including at its default value.
- mini prints the saved options it applies at startup (secret values
  redacted), since quietMiniLogs hides the previous V(1)/V(2) messages.
- warn once at startup when a filer.toml points a local filer store
  (leveldb/leveldb2/leveldb3, including path-specific <store>.<id>
  sections) outside -dir and -master.dir: the filer skips the
  -dir-derived store dir whenever a filer.toml exists, and the metadata
  is lost on container recreate if that path is not persisted.
2026-10-15 18:36:46 -07:00
github-actions[bot] 4c81015bd3 docs: regenerate star history chart 2026-10-16 00:43:27 +00:00
Chris Lu a5ccefb661 filer: escape LIKE wildcards in SQL list prefixes (#11757)
* filer: escape LIKE wildcards in SQL list prefixes

ListDirectoryPrefixedEntries bound prefix+"%" straight into a LIKE
clause, so '_' and '%' in object names (e.g. Langfuse keys like
call_03_...) acted as wildcards: wrong results and a ~40x slower
range scan because (dirhash, name) can no longer be narrowed.

Escape '!'/'%'/'_' in the prefix and declare ESCAPE '!' in the
generated list SQL so every abstract_sql backend (mysql, mysql2,
postgres, postgres2, sqlite) matches literally with identical
semantics. '!' is used instead of '\' because SQLite shares
SqlGenMysql but would reject the MySQL-style '\\' literal, and
'\' would still need escaping itself.

* filer: escape LIKE wildcards in YDB list prefixes

Same wildcard-in-prefix bug as the abstract_sql backends: YQL LIKE
treated '_' and '%' in the prefix as wildcards. Apply the shared
LikePattern helper and declare ESCAPE '!' (YQL allows any escape
char except '%', '_' and '\').

* test: cover literal '_'/'%'/'!' prefixes in sqlite filer listing

End-to-end regression test through SqliteStore against a real
database; runs in CI under the sqlite build tag.
2026-10-15 16:51:15 -07:00
103328e70f filer: create parent directories for lazily fetched remote entries (#11727)
* filer: create parent directories for lazily fetched remote entries

A remote-mounted object first read through the filer was persisted without
its parent directories, so remote.uncache and the remote cache evictor, which
walk down from the mount directory, never reached it: its cached chunks could
not be evicted.

Persist the missing parents with the entry. With those directories in place,
the empty-folder cleaner leaves folders under a remote mount alone: deleting
one removes its whole prefix on the remote, which can still hold objects
that were never read locally.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: keep remote objects at parent keys and remote prefixes of empty folders

Look up a lazily fetched entry's missing parents through lazy fetch before
creating them, so a remote object stored at a parent's key is materialized
and promoted to a prefix object rather than hidden behind a new directory.

Deleting a folder that is empty in the filer is metadata-only under a remote
mount, both for a non-recursive delete and for the expiry cleanup of empty
parents: the mount is populated lazily, so the folder can still hold remote
objects never read, and removing its remote directory deletes every object
under that prefix. This replaces the empty-folder cleaner's own guard, which
covered only one of those paths.

A metadata-only delete records no remote-deletion tombstone, locally or from
its metadata event: nothing is owed to the remote, so lazy reads stay open.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: stop the lazy parent lookup at a store error

Only a parent the store reports as not found is missing; a parent it cannot
read ends the walk, so the lookup does not stat ancestors above it on the
remote.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: stat each missing remote parent once

A lazy fetch that finds k missing parents asked FindEntry for each, and every nested fetch re-walked its own still-missing ancestors, re-stating remote keys already proven absent: 1+k(k+1)/2 remote stats for one first read. The flagged parent lookups now skip their own walk; their entries are created by the same CreateEntry that already persists the chain.

* filer: materialize parents for lazily listed remote entries

maybeLazyListFromRemote persisted children and the listing directory with skipCreateParentDir, so a first access that goes through the list path still left entries unreachable to a walk from the mount directory. Materialize the missing ancestors the same way the lazy fetch does, and let the synced_at directory create its parent chain.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-15 16:23:33 -07:00
Jay HemnaniandChris Lu 1562621c5e filer: record the size of a multipart upload saved inside the entry (#11734)
* filer: record the size of a multipart upload saved inside the entry

* filer: guard save-inside multipart append and check Content-MD5

Addressing review feedback:

- Reject a save-inside multipart append when the target is chunk
  backed. The buffered content cannot be attached to existing chunks,
  so previously the request grew FileSize without storing the appended
  bytes, and reads returned zeros for the added range.
- Compare the request Content-MD5 against the buffered content before
  saving, matching the regular multipart path, so a bad digest no
  longer commits the file.
- Extend the size test to check the saved entry MD5, and cover the
  rejected append against chunk-backed and inline targets plus the
  bad-digest rejection.

* filer: return 400 for unsupported append modes

The save-inside append rejection and the existing inline append and offset write rejections are deterministic client errors, but the autoChunk default case mapped them to 500. Give them shared constants and return 400 instead, and assert the status in the tests.

---------

Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-10-15 14:20:42 -07:00
Jay Hemnani 3659c369e0 webdav: read small files stored inline in the entry (#11733) 2026-10-15 06:38:53 -07:00
Robin Tweedie e12ec94747 fix(shell): compare fractional evacuation destination ratios (#11752) 2026-10-15 06:36:37 -07:00
dependabot[bot] 9e56651808 build(deps): bump github.com/go-redsync/redsync/v4 from 4.17.0 to 4.18.0 (#11743)
Bumps [github.com/go-redsync/redsync/v4](https://github.com/go-redsync/redsync) from 4.17.0 to 4.18.0.
- [Commits](https://github.com/go-redsync/redsync/compare/v4.17.0...v4.18.0)

---
updated-dependencies:
- dependency-name: github.com/go-redsync/redsync/v4
  dependency-version: 4.18.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-15 06:22:09 -07:00
dependabot[bot] d167521270 build(deps): bump helm/chart-testing-action from 2.8.0 to 2.9.0 (#11745)
Bumps [helm/chart-testing-action](https://github.com/helm/chart-testing-action) from 2.8.0 to 2.9.0.
- [Release notes](https://github.com/helm/chart-testing-action/releases)
- [Commits](https://github.com/helm/chart-testing-action/compare/v2.8.0...v2.9.0)

---
updated-dependencies:
- dependency-name: helm/chart-testing-action
  dependency-version: 2.9.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-15 06:20:36 -07:00
dependabot[bot] 8608a4e75d build(deps): bump github.com/ydb-platform/ydb-go-sdk/v3 from 3.151.1 to 3.155.1 (#11739)
build(deps): bump github.com/ydb-platform/ydb-go-sdk/v3

Bumps [github.com/ydb-platform/ydb-go-sdk/v3](https://github.com/ydb-platform/ydb-go-sdk) from 3.151.1 to 3.155.1.
- [Release notes](https://github.com/ydb-platform/ydb-go-sdk/releases)
- [Changelog](https://github.com/ydb-platform/ydb-go-sdk/blob/master/CHANGELOG.md)
- [Commits](https://github.com/ydb-platform/ydb-go-sdk/compare/v3.151.1...v3.155.1)

---
updated-dependencies:
- dependency-name: github.com/ydb-platform/ydb-go-sdk/v3
  dependency-version: 3.155.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-15 06:19:29 -07:00
dependabot[bot] 896e4dd192 build(deps): bump github/codeql-action from 4.38.2 to 4.38.3 (#11746)
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.38.2 to 4.38.3.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/v4.38.2...v4.38.3)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-version: 4.38.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-15 06:18:54 -07:00
github-actions[bot] 84512927ab docs: regenerate star history chart 2026-10-15 00:43:00 +00:00
github-actions[bot] 590a75b9d6 docs: regenerate star history chart 2026-10-14 00:42:11 +00:00
github-actions[bot] 8093ecf4c8 docs: regenerate star history chart 2026-10-13 00:52:42 +00:00
6840371774 sftp: keep a file's existing bytes when it is opened for write without truncate (#11723)
* sftp: keep a file's existing bytes when it is opened for write without truncate

* sftp: preserve sparse gaps and refuse unreadable entries when copying existing content

The copy used a single chunk-stream read, which ends at the first sparse
gap: the tail past a hole was dropped, and a remote-only entry with no
local chunks copied zero bytes, so closing the handle uploaded an empty
file. Read through PrepareStreamContent, which fills unstored regions
with zeros, and refuse entries that carry no local bytes at all.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: report a failed destination write from a completed chunk fetch

retriedStreamFetchChunkData broke out of the retry loop on a clean HTTP
end before checking writeErr, so a destination write that failed mid-
stream returned a short write and nil error. Check writeErr first.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* sftp: cover remote and all-holes sizes when copying existing content

Take the declared size from RemoteEntry when it exceeds FileSize, pad
inline content to the declared size, and treat a chunkless local file as
all holes rather than unreadable. A remote-only entry still refuses the
copy, now checked with a zero FileSize as well.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* sftp: size the existing-content copy the way filer.FileSize does

A locally shortened file still names its older, larger remote object;
taking RemoteSize unconditionally padded the copy back out to it.
filer.FileSize only honors the remote size when the remote is newer,
and also covers the chunk-extent size. A remote-only entry still
refuses the copy whenever it points at remote bytes.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-13 07:20:20 +08:00
dependabot[bot] d67fc47a61 build(deps): bump rustls from 0.23.37 to 0.23.45 in /seaweed-volume (#11707)
Bumps [rustls](https://github.com/rustls/rustls) from 0.23.37 to 0.23.45.
- [Release notes](https://github.com/rustls/rustls/releases)
- [Changelog](https://github.com/rustls/rustls/blob/main/CHANGELOG.md)
- [Commits](https://github.com/rustls/rustls/compare/v/0.23.37...v/0.23.45)

---
updated-dependencies:
- dependency-name: rustls
  dependency-version: 0.23.45
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-13 05:57:00 +08:00
mortezaandMorteza d0f3fad133 ec: avoid redundant needle.GetActualSize call in LocateEcShardNeedle (#11629)
LocateEcShardNeedle passed needle.GetActualSize(size, version) to
LocateEcShardNeedleInterval, which calls needle.GetActualSize internally.
This caused header and padding to be added twice to needle size.

Fixes #3864

Co-authored-by: Morteza <m.mohammadi@rahkar.team>
2026-10-13 05:50:09 +08:00
Чепурнов СемёнandChepurnov Simon 624a13ecc6 EC worker: pin jobs to replica-host workers, late-bind local source (#11718)
* feat: prefer same-host worker and local EC replica

Admin executor pick ranked idle workers only, so a job whose volume
lived on full-03 could still run on full-01 and pull the whole .dat
across the LAN. Rank workers whose Hello host matches any replica
first (fallback to others if the local worker is full).

Detection named Sources[0] / the lowest server id; execute then
SyncAndSelectBestReplica and could switch the copy source to another
rack. After sync, if a replica on this machine has a complete file
count, copy from that URL so affinity and pull stay on one host.

* feat: pin scheduler jobs to replica-host workers until load K

Reserve prefers the least-loaded replica-host worker and waits while
that load is below 5 instead of spilling to a free remote worker.
K=0 restores immediate spill. Waterfill uses reservation load so four
idle preferred workers each get a job.

* feat: park scheduled jobs on per-worker admin queues

Assign a detection batch to the least-loaded replica-host worker in
one pass, then let ExecutionConcurrency goroutines pop a home-runnable
job or steal when that home queue reaches K. Dispatch no longer holds
a job in reserve-backoff, so a free worker is not blocked by another
home's pin. Tracked jobs stay pending/queued and record the home
worker for the admin UI.

* fix: rehome queued jobs when the home worker dies

A live home still pins below K. A home missing from live executors
is stolen immediately onto a free worker so a short orphaned queue
does not sit until the dispatch window ends.

* feat: wait before stealing a dead home worker queue

Orphaned queues are held for 30s so a worker reconnect blink does
not dump jobs onto other machines. After the grace, steal without K
as before. Zero disables the wait. A home that comes back during
grace keeps its queue.

* fix: late-bind EC from to worker hostname

After replicas are consistent (equal FileCount), rebind encode source
to a volume server on the same hostname as worker -address. Any port
on that host is local. Diverged FileCount still uses the max replica.

* feat: assign homeless jobs to global FIFO

Jobs whose replica hostname has no registered worker skip pin and orphan grace.
Free workers take own home, then steal at K/orphan, then leftover FIFO.

* fix: take homeless FIFO before steal for no-home workers

Workers without a replica-host match take leftover first; steal (K/orphan)
only when FIFO is empty. Home workers keep own-queue, steal, then leftover.

---------

Co-authored-by: Chepurnov Simon <195125934+CHEPURNOV-S-S@users.noreply.github.com>
2026-10-13 05:31:34 +08:00
492e700d1f Volume grow avg fill threshold (#11725)
* master: add conditional grow by average fill ratio to VolumeGrow gRPC

Callers that pre-grow volumes on fill level currently have to list all
volumes of a collection and sum sizes themselves, which is expensive on
clusters with millions of volumes. The master already tracks each
writable volume's effective size in memory, so it can answer the
question directly.

Add grow_avg_fill_threshold to VolumeGrowRequest: when set (e.g. 0.6),
growth only happens once the layout's writable volumes have reached
that average fill ratio. A layout with no writable volumes skips the
check, since that is the case growth exists to fix. VolumeGrowResponse
now reports the observed avg_fill and writable_count whether or not
growth was triggered.

* master: unit tests for WritableAvgFill and FindVolumeLayout

Covers the average-fill computation (mixed fills, no writables, a
writable volume without a size-tracking entry yet) and that
FindVolumeLayout resolves existing layouts without creating missing
collections.

* address review: report avg_fill and writable_count for every VolumeGrow

The stats were only read when grow_avg_fill_threshold was set, so a
request without a threshold got AvgFill=0 and WritableCount=0 even when
writable volumes existed. Read the layout stats unconditionally and use
the threshold only to decide whether growth happens.

* master: scope the grow fill check to the requested target

A VolumeGrow aimed at a data center, rack or node must not skip growth
because fill elsewhere keeps the layout average below the threshold.
Measure only writables serving the target; a target with none still
grows.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-13 05:30:03 +08:00
b5ee5b871e sftp: fail a directory listing on a stream error, and page past hidden entries (#11724)
* sftp: fail a directory listing on a stream error, and page past hidden entries

* sftp: test a fully hidden listing page

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 19:34:46 +08:00
6fb21c9ec0 filer: copy a remote-only entry on the remote when renaming it (#11709)
* filer: copy a remote-only entry on the remote when renaming it

Renaming a file under a remote.mount whose content exists only on the
remote (never cached, or remote.uncache'd) lost it: the new entry kept
the old RemoteEntry, deleting the old entry deleted the old object, and
with no chunks nothing ever wrote the new key. filer.remote.sync could
not finish the rename either, since the old object was already gone.

moveSelfEntry now copies such an entry's object to the new key before
it touches anything, and the new entry gets the RemoteEntry of the
copy. The copy is a new optional RemoteStorageObjectCopier interface,
implemented for S3 with CopyObject up to 5 GB and UploadPartCopy above
that; the multipart copy carries the source's metadata and, where the
remote supports tagging, its tags. When an existing target is
overwritten, its delete keeps the remote object (the copy replaced it)
and its event is marked metadata-only so the sync leaves it alone too.

A remote-only rename that leaves its mount, or whose remote cannot
copy, is refused instead of deleting the only copy. Empty entries and
entries with local data rename as before.

UpdateFileMetadata builds its copy source with the same helper, which
escapes each key segment and keeps the slashes.

* filer: delete a copied rename's old object only after the commit

Review follow-ups:

- The old key of a copied remote-only entry, and the old directory of a
  rename that copied one, are now deleted once the rename has committed.
  A rename that fails after the copy leaves the source entry and its
  object in place, plus an extra object at the new key.
- Every part of a multipart copy is conditioned on the ETag the copy
  started from, so an object overwritten mid-copy fails the copy
  instead of mixing versions.
- Both copy paths keep the source's server-side encryption settings.
- filer.remote.sync no longer deletes a key that a remote-only entry
  holds by the time the delete event is processed: a file deleted and
  then replaced by a remote-only rename has its content only there.

* filer: do not sweep the old prefix of a directory rename that copied

Removing the old directory on the remote deletes everything under its
prefix. Run after the commit, that could also take an object another
rename copied in under the same path meanwhile. When a child was copied,
leave the prefix alone and delete only the copied children's old keys.

* filer: keep an old key a remote-only entry has retaken by cleanup time

A directory rename does not lock its children's paths, so between the
commit and the deferred delete of a copied child's old key another
rename can put a remote-only entry there, whose content is then only in
that object. Check the store before deleting.

* filer: keep a renamed entry's old object when the path lookup fails

The deferred delete must only run once the lookup shows no remote-only
entry at the old path. A failed lookup shows nothing, and an extra
object loses nothing.

* filer: copy over a remote-only rename target only after the commit

Copying the source object onto the target's key before the transaction
committed destroyed the target's only content, which a rolled-back
commit cannot restore. When the target is remote-only the copy now runs
after the commit: the new entry keeps the old key's stamp until the copy
lands and re-stamps it, the target's object is kept meanwhile, and a
failed copy keeps the old object so remote.sync can finish it.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* remote.sync: match the delete event's etag before deleting the object

A rename can copy another entry's content onto a key between the filer's
check and the remote delete, and the delete then removes the copy. The
delete event carries the object's etag; delete only when the live object
still matches it.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: finish a rename's remote work outside the committed transaction

The SQL stores keep the transaction in the context and refuse every
call through it once it has committed. The work that runs after
CommitTransaction used that context, so on postgres2 the deferred
old-key delete could not look up the old path and kept the object, and
finishRemoteCopies could not read or stamp the entry. Both failed with
"sql: transaction has already been committed or rolled back" on every
copied rename; remote.sync cleaned up after them.

Keep the context from before the transaction for the work after the
commit. The rename test store gets a strict mode that fails calls
through a committed transaction the way database/sql does.

* filer: carry a deferred copy's stamp in the rename event; delete orphaned uploads

- The deferred copy stamps the entry with a plain UpdateEntry, which
  publishes no event, so the queued rename event still carried the old
  key's stamp to peer filers and subscribers. Give it the copy's
  RemoteEntry before it is sent. The two notify loops share the code now.
- remote.sync kept any object whose ETag changed since the delete. With
  no entry at the path that is an upload which finished after the
  delete, and keeping it leaves it on the remote for good. Look up the
  path first and keep a changed object only when an entry holds it.

* remote.sync: keep a changed object even with no entry at its path

Deleting a changed object when no entry holds the path also deletes a
copy a directory rename has written but not yet made visible, while a
delete of that path waits in the sync; the rename then removes the old
key and the content is gone. Keep any object whose ETag changed since
the delete, as before. An upload that finishes after the delete leaks
its object instead, which is the cheaper failure.

* filer: read a pending remote rename copy from the source key

A remote-only rename over a remote-only target commits before its copy
lands, so reads on the new path served the target's old object until the
copy stamped the entry — and indefinitely when the copy failed without
remote.sync running. The committed entry now records the renamed-from
path, and remote-only reads resolve that object's key while the stamp
is pending; the copy's stamp and the sync's completion stamp both clear
the marker.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: stamp the rename event with the deferred copy's result

A remote-only rename over a remote-only target queued the new entry's
event with the source's stamp before the deferred copy ran. Peers
replaying it saved a stamp the copy had already superseded, and the
sync serving such a peer rejected the destination as a match and tried
to copy the deleted source. The event now carries the copy's result
and drops the pending-source marker once it lands.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* remote.sync: delete only the object the event's stamp describes

An object at a deleted key is not always the one the delete meant: a
rename copy can land before its filer entry is visible, and a rewrite
can upload over it. When the event recorded no remote stamp, or the
live stat cannot confirm the recorded etag, the delete now keeps the
object instead of treating it as a late upload.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: tighten the pending rename source marker

Follow-ups to reading a pending remote rename copy from its source key:

- Only the filer sets the marker. The HTTP upload and tagging handlers
  store Seaweed-* request headers as extended attributes, so they skip
  this one, and a marker that leads out of the entry's own mount is
  ignored, so it cannot point reads at another remote.
- A pending entry renamed again copies from the marker's source, not
  from its own key, which still holds the old target's bytes. A further
  deferred rename keeps naming the original source; an immediate copy
  clears the marker.
- When remote.sync finds the destination already holding the content,
  it stamps the entry (clearing the marker) before deleting the source,
  and deletes the source only once the entry no longer reads it.
- A read that resolved the source just before the copy finished and
  the source went away retries with the entry as it is now.

* filer, remote.sync: follow a pending rename's source across renames

When a remote-only entry is renamed over another remote-only file and
the deferred copy fails, it reads the source key through its marker.
Renamed again the same way, the marker keeps naming that source, but:

- remote.sync completed each rename from the event's old key. For the
  second one that is the first target's old object, which it copied
  over the destination before clearing the marker. It now copies from
  the source the entry's marker names.
- For the first rename it found no entry at the new path and deleted
  the old key, which the later entry still reads. When the event's new
  entry carried a marker and the path is gone, it now leaves the key to
  the entry that moved on.
- When the filer copies a pending entry immediately on a later rename,
  it now also releases the source the copy read, after the commit and
  with the same checks as the old key.

* remote.sync: release a pending rename's source, the filer leaves it

The filer deleted a pending entry's source after copying it on a later
rename, but that rename locks only its own paths: another rename can
copy onto the source key before its entry is visible, and the cleanup
then deleted that copy. And deleting a pending entry never released its
source at all.

The filer now leaves marked sources alone. remote.sync releases one
when the pending entry is deleted, or when a later rename was copied
right away, and only while the live object is still the one the
entry's stamp describes and no remote-only entry holds the path: a
copy that landed there since has a different ETag and is kept.

* remote.sync: release a pending source when the renamed entry is cached

A rename copied right away left its pending source to remote.sync,
which released it only while the destination was still remote-only.
Cached before the sync got there, the destination took the other path
and the source stayed on the remote for good. Release it there too,
with the same checks.

* remote.sync: release a pending source only after the destination is safe

e6d6438 released the source a renamed entry's old version read before
the destination was uploaded. A cached destination that still carried
the pending marker then had nothing behind it: had its upload failed,
remote.uncache could drop its chunks and leave it reading a deleted
source. Release the source only after the upload is stamped, and only
when the entry no longer reads through a marker.

* remote.sync: delete a rename's old key only if it is still the renamed content

completeRemoteOnlyRename deleted the old key once the destination held
the content. Another rename onto the vacated path can copy its object
there before the sync reaches the first rename, and that copy was then
deleted with it. Delete the old key only while the live object is still
the one the renamed entry's stamp describes.

Also keep the pending-source marker out of DELETE ?tagging, which
clears every Seaweed-* attribute: the marker is not a tag, and losing it
points reads back at the old target's bytes.

* remote.sync: compare a rename's old key against the source's own stamp

A retry can find the renamed entry already stamped with the copy, and a
copy of a multipart source has another ETag, so the unchanged source
was kept and stayed on the remote. Carry the stamp the event recorded
for the source (or the pending entry's, for a marked source) and compare
against that.

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 19:21:54 +08:00
41260a55f6 master: record writableVolumeCount on HTTP /dir/assign like gRPC Assign (#11716)
* master: extract the periodic per-layout grow decision

Move the must-grow and crowded-step decision of the periodic growth scan
in ProcessGrowRequest into periodicGrowCount, a pure function of the
layout's lastGrowCount, writable and crowded counts, so it can be tested
without running the scan loop or growing real volumes.

No behavior change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* master: record writableVolumeCount on HTTP /dir/assign

The periodic growth scan (#5781) keeps a layout supplied ahead of
demand only when it knows how many writable volumes clients want:
lastGrowCount, which gRPC Assign records with SetLastGrowCount. HTTP
/dir/assign parsed writableVolumeCount but only used it to size the
emergency grow when no writable volume was left, so the scan always saw
lastGrowCount 0 for HTTP-only writers.

Since #5821 removed the per-assign threshold grow, such a layout only
grew by the rack-aware step of 2, and only once every writable volume
was crowded. When a batch of volumes filled in lockstep, those 2 took
every write and could fill before the next scan, leaving no writable
volumes until an assign-triggered emergency grow landed, with 503s for
every other assign meanwhile.

Record writableVolumeCount on HTTP assigns as gRPC Assign does. Clients
that do not send it are unaffected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* master: reject unknown rack/dataNode before recording grow targets

An assign naming a rack or data node outside the topology can never be
served, but lastGrowCount was recorded first. The periodic growth scan
drops the requested placement, so it kept topping the layout up with
volumes the requester could not use. Both assign paths now validate the
placement like they already validate the data center.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* master: scope placement existence checks to the requested path

A rack in another data center or a data node under another rack passed
validation yet could never serve the request, and the grow count was
still recorded. The rack and data-node lookups now take the requested
parents, matching the filters the placement engine applies.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 18:10:41 +08:00
3031ab5d97 EC placer: walk extra shards on freeSlots, keep 2+2 (#11714)
* fix: rotate EC extra shards on pickTarget count ties

pickTarget used to keep the first sorted key when shard counts tied,
so 10+4 onto 4 racks always put the two fours on full-01/full-02.
Tie-break on remaining freeSlots (same idea as pickNodeInRackEligible)
so the leftover shards walk across racks over a series of volumes.

* fix: pay within-rack EC shard debt on Place

Score nodes by live EcShards plus this Place's assignments, not
this-volume counts or EC-volume presence. Allow SameRackCount+1
only after a sibling already holds a shard, so 4-on-2 becomes 3+1
while 3-on-2 stays 2+1. Rack pickTarget tie-break is unchanged.

* fix: even-spread EC disks, pay debt with remainders only

Do not pack 3+1: three shards on one disk is one disk-loss from
unrecoverable. Place this volume evenly first; leftover shards
(2+1) go to the thinner live count. 4-on-2 stays 2+2.

* fix: score Place extras by freeSlots then liveShards

Even-per-volume and SameRackCount stay; remainder prefers capacity,
then shard debt. No 3+1.

* fix: keep shard-debt fixture capacity-consistent

freeSlots = cap-shards so extras keep paying the thinner disk
across volumes instead of oscillating after the first pair.

* refactor: document dropped mFree; switch EcBalance node pick

Keep host even (mCount). Remainder is disk freeSlots then liveShards so
thin disks fill first. SameRackCount / 2+2 unchanged. pickTarget unchanged.

* test: cover Place 10+4 extras on thin disks/racks

Seed liveShards/freeSlots imbalance on 1-1-{4,6,7,8,15}, 4-2-3, 5-2-2, 6-1-2, 7-1-2. Remainder tails must miss fat first ids; remainder 0 freezes the live gap.

* fix(ecbalancer): count only matching-type slots when picking a move target

Aggregate freeSlots let a rack, machine, or node whose free capacity sat
on another disk type win placement; pickBestDiskOnNode then found no
eligible disk and the move went out with TargetDisk 0, which the copy
endpoint reads as automatic selection onto any disk.

Score candidates on freeSlotsForType — per-disk metadata when present,
the node aggregate when absent — so a wrong-type destination is never
chosen, and cover pickBestNodeForVolume's tie-breaks and the wrong-type
plan case.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ec_balance: spread free slots per disk type, keeping the remainder

Dividing the node's free slots evenly over all its non-full disks floored
a leftover slot to zero on every disk — one free slot across two disks
made the node look full. It also gave disks of other types a share of
slots counted only for the requested type. Each type's slots now spread
over that type's disks with ceil division, so the total is preserved and
every usable slot stays visible to the planner.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Chepurnov Simon <195125934+CHEPURNOV-S-S@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 18:09:53 +08:00
acb44a68a9 fix(volume): skip unnecessary needle data reads on delete (#11711)
* fix(volume): do not read a needle's data to delete it

A delete read the needle's whole payload from disk only to discard it.
The delete handler and BatchDelete need the needle's cookie and its
chunk-manifest flag, which live in the header and trailer, but fetched
them with a full needle read, the same call a GET uses.

readNeedle now reads the header and trailer only when MustMetaOnly is
set, whatever the needle's size, and the delete paths ask for that. The
Rust volume server honours must_meta_only the same way for its delete
paths. A compressed needle or a chunk manifest still gets the full read,
as before, because their callers need the bytes.

HEAD is unchanged. It used to set MustMetaOnly, which nothing read; it
now asks for AttemptMetaOnly alone, so it keeps reading a needle up to
PagedReadLimit whole. On a spinning disk that one read is cheaper than a
header read followed by a trailer read.

Version 1 needles are always read in full. They have no data size field
and no metadata after the data, which ReadNeedleMeta relies on, so a
metadata-only read of one failed with "invalid needle meta size". That
already happened for version 1 needles above PagedReadLimit read with
AttemptMetaOnly.

A delete therefore no longer verifies the data CRC of the needle it
removes. The cookie check is unchanged.

* fix(volume): read transformed images fully on HEAD too

A HEAD carrying resize or crop params reports the transformed length,
which a metadata read cannot produce. Go and Rust both answered such
HEADs from the source header, returning the stored size instead of the
cropped or resized one.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 18:08:19 +08:00
Chris Lu 6927de6003 seaweed-worker: mint the admin bearer token like the Go worker
The admin control stream now requires mTLS or a token signed by
jwt.filer_signing.key; the Rust worker gets --filer-signing-key and
--filer-signing-key-expires-after-seconds, mirroring the Go worker's
config names, and attaches the token to the stream metadata.
2026-10-12 03:06:57 -07:00
Chris Lu 613a286cde worker: present the admin bearer token on control streams
Workers mint a SeaweedFilerAdminClaims token when jwt.filer_signing.key
is in their configuration, matching the authentication the admin server
now requires.
2026-10-12 02:55:50 -07:00
Chris Lu 55ac46f7a9 admin: authenticate worker and plugin gRPC streams
Worker registration previously ran unauthenticated on any interface the
listener bound. The stream now requires a verified mTLS client
certificate or a Bearer token signed by jwt.filer_signing.key, and falls
back to local clients only when no key is configured.
2026-10-12 02:55:50 -07:00
Chris Lu 3f8035deb7 security: share the admin gRPC peer authentication check
The local-peer fallback and bearer-token validation used by the filer IAM
gRPC path move into weed/security so every admin-plane gRPC service can
enforce the same contract. A client certificate verified by the server's
TLS config now also counts as authenticated.
2026-10-12 02:55:50 -07:00
Chris Lu a4c8d31cb8 s3api: ignore response header overrides on anonymous requests
AWS requires response-* query parameters to be signed; an anonymous
caller must not choose how a stored object is served.
2026-10-12 02:49:55 -07:00
Chris LuandDevin 6bac506842 sftp: terminate the missing-ancestor permission walk at the root
A path whose ancestors are all absent recursed on the root forever,
since Dir("/") is "/".

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 02:41:28 -07:00
Chris LuandDevin 2c031d9d63 sftp: check write permission on the target file, not only its parent
newFileWriter authorized uploads against the directory, so a file with
a more specific read-only ACL or mode could still be overwritten. The
same parent-directory fallback used for nonexistent paths now applies
through CheckFilePermission on the target itself.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 02:41:28 -07:00
Chris LuandDevin d69f45ee83 s3: validate multipart initiate ACLs like PUT
CreateMultipartUpload accepts canned and custom ACLs on initiation;
run them through the same authorization path as PutObject so an
explicit ACL requires s3:PutObjectAcl, and persist the validated
grants on the upload entry for completion to apply.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 02:41:28 -07:00
Chris LuandDevin 3aaf80b33b s3: stop trusting internal ACL headers in ParseS3Metadata
Multipart initiation stored client-supplied Seaweed-X-Amz-Acl and
Seaweed-X-Amz-Owner request headers onto the upload entry, and
CompleteMultipartUpload copied them to the finished object, letting a
caller forge ownership or a public ACL without s3:PutObjectAcl.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 02:41:28 -07:00
Chris LuandDevin 85f04b54fd s3tables: check data-path ancestors for catalog markers on DeleteTable
The decoupled-delete guard only inspects the resolved data path itself,
but the catalog marker lives on a table's name path, so a location that
resolves inside another table's directory (its marker-less data
subdirectory) slips through and the recursive purge destroys the other
table's data. Walk the data path's ancestors within the bucket and
refuse when any carries a table or view catalog entry.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 02:41:28 -07:00
Chris LuandDevin 6f93f87446 s3tables: confine caller-supplied metadataLocation to the table's namespace
CreateTable/RegisterTable/UpdateTable (and the view equivalents) validate
the location only for bucket and traversal, so a tenant can register a
table whose location resolves into another namespace's directory and
DeleteTable will purge it. Require the location to live under the
caller's namespace subtree, the boundary per-namespace delegation uses.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 02:41:28 -07:00
scooby87 73949daa98 filer: keep sequential prefetch within prefetchCount chunks (#11674)
ChunkReadAt.ReadAt asks the ReaderCache to prefetch after every read,
and MaybeCache counted only the downloads it newly started toward the
window, so each call reached prefetchCount chunks further than the last.
A reader consuming a chunk in several reads, as the S3 GET path does
with a 256 KiB copy buffer over 4 MiB chunks, ran the read-ahead up to
the ReaderCache downloader limit and kept most of a large object in
memory, enough to get an S3 gateway OOM-killed mid-response.

Count every chunk in the window, downloading or not, so the read-ahead
stays at prefetchCount chunks past the reader.


Assisted-by: LLM

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-12 17:41:01 +08:00
Chris LuandDevin 603a1221a7 filer: converge peer metadata replication on restart (#11698)
* filer: skip replayed peer events older than the local entry

Replay deleted and re-inserted every received event unconditionally, so
a restarted filer that rewound its subscription offset re-applied stale
peer writes over newer local state (values reverted, deleted entries
came back and pointed at already-vacuumed chunks). Compare the event's
TsNs against the local entry's Mtime and skip per-path, so a move event
can still remove its source while keeping a newer destination.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: add replay last-writer-wins regression tests

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: stop peer subscriptions and flush offsets on shutdown

Filer.Shutdown closed the store while subscription goroutines were still
mid-stream; their deferred offset save was never guaranteed to run, so a
restart rewound each peer's stream back to the last periodic flush and
replayed stale events. MetaAggregator.Shutdown now closes every peer's
stop channel and waits for the loops so the saved offset covers what was
replayed, and the reconnect sleep is interruptible by the stop channel.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: test that shutdown stops and waits for peer loops

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: order replay by persisted per-path commit timestamps

Review feedback found the mtime comparison still lets stale replay win:
a local delete removes the entry it would have been compared against, an
entry mtime is client-controllable, and the check ran unlocked so a local
write could land between it and the store mutation.

- Record the event timestamp per mutated path in the store KV on every
  emitted mutation, plus a delete record for sources of deletes and
  renames. The record survives the entry, so a replayed create older than
  a local delete no longer resurrects it; ancestor delete records cover
  children swept without per-child events.
- Replay compares the event timestamp against that record under the same
  per-path entry locks local RPC mutations hold, falling back to entry
  mtime only for entries written before records existed. Lookup errors
  now abort the event instead of being treated as an absent entry.
- MetaAggregator.Shutdown marks the aggregator stopped under peerChansLock
  so a master update cannot start a subscription that outlives the wait,
  and waits for the loops without the five-second cap now that every
  blocking stage — signature RPC, bootstrap traversal, retry sleeps, and
  per-event processing — cancels on the stop channel.

* filer: pin peer offset at failed replay, fix commit-record context and key size

A replayed event that exhausted its retries (or was interrupted by
shutdown) was still followed by advancing the peer watermark, so the
persisted offset skipped the unapplied event forever. replicateMetadataChange
now reports the outcome: errPeerSubscriptionStopped aborts the subscription
so the deferred save keeps the last completed event, and an exhausted retry
records a barrier the persisted offset and reconnect resume point cannot
pass while the stream keeps flowing.

Commit records also failed on two fronts: NotifyUpdateEvent runs after
CommitTransaction with the same context, so SQL stores wrote MpTs/MpDl
through an already-committed *sql.Tx; and metaPathKey embedded the whole
path, overflowing the 766-char name column SQL KV stores use, which the
KvPut fallback update then silently dropped. The writes now use a fresh
context and a sha256-hashed fixed-size key.

* filer: lock HTTP delete handler and replay against ancestor paths

The HTTP DeleteHandler called DeleteEntryMetaAndData without the shared
entry lock, so a peer replay could still interleave with a local delete and
its commit record. It now holds the same lock the gRPC DeleteEntry handler
does.

Replay also locked only the exact event paths, but its ordering check
consults ancestor delete records: a local recursive delete holds only the
ancestor's lock, so a stale child create could check before the tombstone
landed and then reinsert. lockReplayPaths now locks every ancestor of both
event paths in one lexicographic order — parents sort before children, as
the rename handler's two-path order already does — while never locking the
root, which cannot be deleted and would serialize all replays.

* filer: record replay commit before the replacement insert

If the same-path update DeleteEntry succeeded but InsertEntry failed, the path sat deleted with no commit record until the retry arrived, and a stale peer create could resurrect it in the gap. The commit record now lands immediately after the delete — without a delete record, since an update is not a delete — and again after the successful insert.

* filer: lock replay ancestors shared, not exclusive

Exclusive locks on every ancestor serialized all replays under a shared
parent — unrelated objects in the same bucket could no longer apply in
parallel. Ancestors only need to exclude an exclusive holder (a local
delete or rename of the ancestor), so they now take shared locks; a path
the event changes stays exclusive even when it is also the other path's
ancestor.

* filer: order peer bootstrap on the sender's commit timestamp

traversePeerMetadata compared localCommitTsNs — a commit-clock value —
against the entry's mtime, which a client can set freely: a peer change
committed recently with an old mtime was skipped, and a stale entry
mtime-stamped in the future overwrote a newer local commit.

TraverseBfsMetadataResponse now carries the sender's per-path commit
timestamp (recorded MpTs record, entry mtime as the unrecorded fallback).
The bootstrap applies an entry only when its commit is newer than the
local one and records the applied timestamp, so replayed events order
against it — closing the gap between bootstrap content and the event
stream. A peer without the field sends 0 and keeps the mtime fallback,
preserving rolling-upgrade behavior.

* filer: keep failed commit records pending until they land

notePathChange ran after the mutation committed and only logged a KvPut
failure: a delete whose MpTs/MpDl record never landed left no tombstone,
so a stale replayed create could resurrect the path while the store was
unreachable.

Failed records now stay in metaTsPending and a repair pass retries them
until they land. localCommitTsNs, deletedAfterTsNs and CommitTsNs
consult the pending record, so ordering holds in the window between the
committed mutation and the durable record. A per-path flag bounds the
repair to one goroutine, and respawns cover merges landing between its
last pass and the flag release.

* filer: keep only unwritten work in the pending commit record

The pending record merged every obligation into one timestamp and flag,
and a later successful write left it untouched: a repair pass could then
write an older timestamp over a newer one, and a merged delete inherited
the newer change's time, which the ancestor check reads as a newer
delete. The record now holds a change and a delete timestamp of its own,
both discharged under the record's lock alongside the store writes that
cover them, so a repair writes only what is still owed.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: fail traversal on a commit-record read error

CommitTsNs answered a store read failure with the entry mtime, a value
clients can set arbitrarily far ahead; a peer bootstrapping on it would
record the minted time and reject real updates below it. The lookup now
returns the error so the traversal restarts instead.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 17:39:34 +08:00
f11cecc697 helm: keep the S3 config checksum stable across the first upgrade (fixes #11719) (#11720)
* helm: keep the S3 config checksum stable across the first upgrade (fixes #11719)

checksum/s3config hashed the rendered s3-secret.yaml. That template
keeps its generated S3 keys with lookup, so rendering it again for the
checksum drew new keys whenever lookup found nothing: on install the
filer, S3 and all-in-one pods each carried a checksum of a different
set of keys than the Secret got, and the first upgrade with unchanged
values corrected it and restarted them.

Hash the values s3-secret.yaml is built from instead: the enable and
auth switches, existingConfigSecret, credentials, the legacy-secret
settings, the audit log configs and the Secret name. Only those, so
scaling the S3 gateway or changing its resources does not restart the
filer. A change to any of them still restarts the pods. Keys that the
chart generated stay out of the hash; they only change if the Secret
is deleted by hand.

The new render test fails on the old template (two checksums for one
Secret, different on every render) and passes now; it also checks that
the checksum follows the credentials and filer.s3.auditLogConfig and
ignores s3.replicas and s3.resources. On kind, an upgrade with unchanged
values no longer replaces the filer pod.

* helm: check the S3 checksum per workload, all-in-one included

Each workload's checksum/s3config is now read on its own (--show-only),
so a workload that loses the annotation fails the test instead of being
hidden by sort -u, and the all-in-one deployment gets the same checks.
Verified against two broken copies: one without the filer annotation,
one with the old all-in-one checksum; the test fails on both.

The helper comment now says what happens when the Secret is deleted to
rotate generated keys.

* ci(helm): fail the S3 checksum test on a missing or unrendered checksum

A failed helm template or a render without the annotation produced an
empty checksum, and the inequality assertions then passed against it.
s3_checksum now returns non-zero unless it found a checksum, and the
changed-value renders are assigned before comparing so failures cannot
hide inside test.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Ege Zambelli <egezamb@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 15:43:53 +08:00
521 changed files with 41546 additions and 5982 deletions

No files matched your search

+3 -3
View File
@@ -27,7 +27,7 @@ jobs:
# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@v4.38.2
uses: github/codeql-action/init@v4.39.0
# Override language selection by uncommenting this and choosing your languages
with:
languages: go
@@ -35,7 +35,7 @@ jobs:
# Autobuild attempts to build any compiled languages (C/C++, C#, or Java).
# If this step fails, then you should remove it and run the build manually (see below).
- name: Autobuild
uses: github/codeql-action/autobuild@v4.38.2
uses: github/codeql-action/autobuild@v4.39.0
# ℹ️ Command-line programs to run using the OS shell.
# 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun
@@ -49,4 +49,4 @@ jobs:
# make release
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v4.38.2
uses: github/codeql-action/analyze@v4.39.0
+1 -1
View File
@@ -405,7 +405,7 @@ jobs:
output: trivy-results.sarif
exit-code: '0'
- name: Upload Trivy scan results to GitHub Security
uses: github/codeql-action/upload-sarif@v4.38.2
uses: github/codeql-action/upload-sarif@v4.39.0
if: always()
with:
sarif_file: trivy-results.sarif
@@ -460,7 +460,7 @@ jobs:
- name: Upload Trivy scan results to GitHub Security
if: always()
uses: github/codeql-action/upload-sarif@v4.38.2
uses: github/codeql-action/upload-sarif@v4.39.0
with:
sarif_file: trivy-results.sarif
category: trivy-${{ matrix.variant }}
+37 -2
View File
@@ -39,7 +39,7 @@ jobs:
check-latest: true
- name: Set up chart-testing
uses: helm/chart-testing-action@v2.8.0
uses: helm/chart-testing-action@v2.9.0
- name: Run chart-testing (list-changed)
id: list-changed
@@ -95,6 +95,41 @@ jobs:
grep -A 5 "^ lifecycle:" /tmp/s3-rollout.yaml | grep -q -- "- sleep"
echo "S3 rollout settings render correctly"
echo "=== Testing the S3 config checksum is stable across renders ==="
# s3-secret.yaml keeps its generated keys with lookup. Hashing the rendered
# Secret drew new keys for the checksum whenever lookup found nothing (on
# install), so the first upgrade changed the checksum and restarted the
# filer and S3 pods although nothing had changed.
s3_checksum() { # <template> <helm args...>: the checksum/s3config of one workload
local template=$1; shift
local sum
sum=$(helm template test $CHART_DIR --show-only "$template" "$@" \
| grep 'checksum/s3config:' | awk '{print $2}')
test -n "$sum" || { echo "no checksum/s3config in $template" >&2; return 1; }
echo "$sum"
}
s3_auth_args=(--set s3.enabled=true --set s3.enableAuth=true)
s3_sum=$(s3_checksum templates/s3/s3-deployment.yaml "${s3_auth_args[@]}")
filer_sum=$(s3_checksum templates/filer/filer-statefulset.yaml "${s3_auth_args[@]}")
test "$s3_sum" = "$filer_sum" # one Secret, one checksum
test "$s3_sum" = "$(s3_checksum templates/s3/s3-deployment.yaml "${s3_auth_args[@]}")"
s3_creds_sum=$(s3_checksum templates/s3/s3-deployment.yaml "${s3_auth_args[@]}" \
--set s3.credentials.admin.accessKey=a --set s3.credentials.admin.secretKey=b)
test "$s3_sum" != "$s3_creds_sum"
filer_audit_sum=$(s3_checksum templates/filer/filer-statefulset.yaml "${s3_auth_args[@]}" \
--set-json 'filer.s3.auditLogConfig={"fluent_port":24224}')
test "$filer_sum" != "$filer_audit_sum"
# Settings the Secret does not read must not restart the S3 and filer pods.
test "$filer_sum" = "$(s3_checksum templates/filer/filer-statefulset.yaml "${s3_auth_args[@]}" \
--set s3.replicas=3 --set s3.resources.limits.cpu=1)"
all_in_one_args=(--set allInOne.enabled=true --set allInOne.s3.enabled=true --set allInOne.s3.enableAuth=true)
aio_sum=$(s3_checksum templates/all-in-one/all-in-one-deployment.yaml "${all_in_one_args[@]}")
test "$aio_sum" = "$(s3_checksum templates/all-in-one/all-in-one-deployment.yaml "${all_in_one_args[@]}")"
aio_creds_sum=$(s3_checksum templates/all-in-one/all-in-one-deployment.yaml "${all_in_one_args[@]}" \
--set s3.credentials.admin.accessKey=a --set s3.credentials.admin.secretKey=b)
test "$aio_sum" != "$aio_creds_sum"
echo "S3 config checksum is stable and follows only what the S3 Secret reads"
echo "=== Testing S3 credentials from an existing secret ==="
credential_args=(
--set s3.credentials.admin.existingSecret=minio-root
@@ -1834,7 +1869,7 @@ jobs:
echo "IMAGE_TAG=$TAG" >> $GITHUB_ENV
- name: Create kind cluster
uses: helm/kind-action@v1.15.0
uses: helm/kind-action@v1.15.1
- name: Verify volume resize hook security contexts
run: |
@@ -86,7 +86,11 @@ jobs:
set -e
dd if=/dev/urandom of=generated bs=1M count=32
aws --no-verify-ssl s3 cp --no-progress generated s3://bucket/test-multipart
aws --no-verify-ssl s3 cp --no-progress s3://bucket/test-multipart downloaded
for i in 1 2 3 4 5; do
aws --no-verify-ssl s3 cp --no-progress s3://bucket/test-multipart downloaded && break
if [ "$i" -eq 5 ]; then exit 1; fi
sleep 2
done
diff -q generated downloaded
rm -f generated downloaded
+2 -1
View File
@@ -225,7 +225,7 @@ The full operation list is in [Amazon S3 API][AmazonS3API], and [Supported APIs
SeaweedFS is a lakehouse in one system. [S3 Table Buckets][S3TableBucket] hold Apache Iceberg tables by default, or [Lance][LanceCatalog] tables for vectors and multimodal data, and the built-in [Iceberg REST Catalog][IcebergCatalog] and Lance namespace serve them directly. There is no Hive Metastore, Glue, or separate catalog service to deploy, secure, and back up.
* Query engines operate on the same tables at the same time: [Spark][SparkIceberg], [Trino][TrinoIceberg], [Dremio][DremioIceberg], [DuckDB][DuckDBIceberg], [Apache Doris][DorisIceberg], [RisingWave][RisingWaveIceberg], ClickHouse, and [LanceDB][LanceDB]. Catalog commits are atomic compare-and-swap, so concurrent writers are safe. [Lakekeeper][Lakekeeper] can front the same storage with STS-vended credentials.
* Query engines operate on the same tables at the same time: [Spark][SparkIceberg], [Trino][TrinoIceberg], [Dremio][DremioIceberg], [DuckDB][DuckDBIceberg], [Apache Doris][DorisIceberg], [RisingWave][RisingWaveIceberg], ClickHouse, and [LanceDB][LanceDB]. Catalog commits are atomic compare-and-swap, so concurrent writers are safe. [Lakekeeper][Lakekeeper] can front the same storage with STS-vended credentials. [OLake Go][OLake] is a high-performance, open-source EL (Extract–Load) platform that bridges operational databases and open lakehouse storage - writing Apache Iceberg tables or Parquet files on Amazon S3.
* [Automated table maintenance][IcebergMaintenance]: compaction, snapshot expiration, orphan file removal, and manifest rewriting, configured per bucket or table through the S3 Tables maintenance APIs, and the same for [Lance][LanceMaintenance].
* IAM at the bucket, namespace, and table level with standard bucket policies, see [S3 Tables Security][S3TablesSecurity].
* A [Hadoop compatible file system][Hadoop] for Spark, Flink, and HBase.
@@ -487,6 +487,7 @@ Your support will be really appreciated by me and other supporters!
[RisingWaveIceberg]: https://github.com/seaweedfs/seaweedfs/wiki/RisingWave-Iceberg-Integration
[LanceDB]: https://github.com/seaweedfs/seaweedfs/wiki/LanceDB-Integration
[Lakekeeper]: https://github.com/seaweedfs/seaweedfs/wiki/Lakekeeper-Iceberg-Integration
[OLake]: https://github.com/seaweedfs/seaweedfs/wiki/OLake-Go-Iceberg-Integration
[IcebergMaintenance]: https://github.com/seaweedfs/seaweedfs/wiki/Iceberg-Table-Maintenance
[LanceMaintenance]: https://github.com/seaweedfs/seaweedfs/wiki/Lance-Maintenance-Worker
[S3TablesSecurity]: https://github.com/seaweedfs/seaweedfs/wiki/S3-Tables-Security
+1
View File
@@ -169,6 +169,7 @@ seaweed-volume/
- `--memprofile`
- `--compactionMBps` (default 0)
- `--maintenanceMBps` (default 0)
- `--compactionMaxParallelCommits` (default 0)
- `--fileSizeLimitMB` (default 256)
- `--concurrentUploadLimitMB` (default 0)
- `--concurrentDownloadLimitMB` (default 0)
+1 -1
View File
@@ -543,7 +543,7 @@ a language-agnostic gRPC stream that external maintenance workers connect on: th
opens `WorkerStream`, sends `WorkerHello` with the job types it can `detect` and `execute`,
answers `RequestConfigSchema` with a `JobTypeDescriptor`, replies to `RunDetectionRequest`
with `JobProposal`s and to `ExecuteJobRequest` with `JobProgressUpdate`s and `JobCompleted`.
`weed worker -admin=host:23646` is the Go reference implementation of exactly that contract,
`weed worker -admin=host:12535` is the Go reference implementation of exactly that contract,
from outside the admin process.
Nothing in it is Go-specific, and the Rust toolchain is already in the tree.
@@ -21,7 +21,7 @@ The testing environment uses **official SeaweedFS commands** and includes:
- **1 Master Server** (port 9333) - Coordinates the cluster with 50MB volume size limit
- **6 Volume Servers** (ports 8080-8085) - Distributed across 2 data centers and 3 racks for diversity
- **1 Filer** (port 8888) - Provides file system interface
- **1 Admin Server** (port 23646) - Detects volumes needing EC and manages workers using official `admin` command
- **1 Admin Server** (port 12535) - Detects volumes needing EC and manages workers using official `admin` command
- **3 EC Workers** - Execute erasure coding tasks using official `worker` command with task-specific working directories
- **1 Load Generator** - Continuously writes and deletes files using SeaweedFS shell commands
- **1 Monitor** - Tracks cluster health and EC progress using shell scripts
@@ -47,7 +47,7 @@ This provides:
- Docker and Docker Compose installed
- GNU Make installed
- At least 4GB RAM available for containers
- Ports 8080-8085, 8888, 9333, 23646 available
- Ports 8080-8085, 8888, 9333, 12535 available
### Start the Environment
@@ -127,7 +127,7 @@ Run `make help` to see all available targets:
|---------|-----|-------------|
| Master UI | http://localhost:9333 | Cluster status and topology |
| Filer | http://localhost:8888 | File operations |
| Admin Server | http://localhost:23646/ | Task management |
| Admin Server | http://localhost:12535/ | Task management |
| Monitor | http://localhost:9999/status | Complete cluster monitoring |
| Volume Servers | http://localhost:8080-8085/status | Individual volume server stats |
+15 -15
View File
@@ -26,7 +26,7 @@ build-and-restart: build ## Build with latest changes and restart services
@echo "2️⃣ Recreating workers to reconnect..."
@docker-compose -f $(COMPOSE_FILE) up -d worker1 worker2 worker3
@echo "✅ All services recreated with latest changes!"
@echo "🌐 Admin UI: http://localhost:23646/"
@echo "🌐 Admin UI: http://localhost:12535/"
@echo "💡 Workers will reconnect to the new admin server"
restart-workers: ## Restart all workers to reconnect to admin server
@@ -64,7 +64,7 @@ start: ## Start the complete SeaweedFS cluster with admin and workers
@echo "✅ Cluster started!"
@echo ""
@echo "📊 Access points:"
@echo " • Admin UI: http://localhost:23646/"
@echo " • Admin UI: http://localhost:12535/"
@echo " • Master UI: http://localhost:9333/"
@echo " • Filer: http://localhost:8888/"
@echo ""
@@ -107,7 +107,7 @@ start-staged: ## Start services in proper order with delays
@echo "✅ All services started!"
@echo ""
@echo "📊 Access points:"
@echo " • Admin UI: http://localhost:23646/"
@echo " • Admin UI: http://localhost:12535/"
@echo " • Master UI: http://localhost:9333/"
@echo " • Filer: http://localhost:8888/"
@echo ""
@@ -136,7 +136,7 @@ status: ## Check the status of all services
@echo "Master:"
@curl -s http://localhost:9333/cluster/status | jq '.IsLeader' 2>/dev/null || echo " ❌ Master not ready"
@echo "Admin:"
@curl -s http://localhost:23646/ | grep -q "Admin" && echo " ✅ Admin ready" || echo " ❌ Admin not ready"
@curl -s http://localhost:12535/ | grep -q "Admin" && echo " ✅ Admin ready" || echo " ❌ Admin not ready"
logs: ## Show logs from all services
@echo "📜 Following logs from all services..."
@@ -157,7 +157,7 @@ master-logs: ## Show logs from master server
admin-ui: ## Open admin UI in browser (macOS)
@echo "🌐 Opening admin UI in browser..."
@open http://localhost:23646/ || echo "💡 Manually open: http://localhost:23646/"
@open http://localhost:12535/ || echo "💡 Manually open: http://localhost:12535/"
test: ## Run integration test to verify task assignment and completion
@echo "🧪 Running Admin-Worker Integration Test"
@@ -166,11 +166,11 @@ test: ## Run integration test to verify task assignment and completion
@echo "1️⃣ Checking cluster health..."
@sleep 5
@curl -s http://localhost:9333/cluster/status | jq '.IsLeader' > /dev/null && echo "✅ Master healthy" || echo "❌ Master not ready"
@curl -s http://localhost:23646/ | grep -q "Admin" && echo "✅ Admin healthy" || echo "❌ Admin not ready"
@curl -s http://localhost:12535/ | grep -q "Admin" && echo "✅ Admin healthy" || echo "❌ Admin not ready"
@echo ""
@echo "2️⃣ Checking worker registration..."
@sleep 10
@echo "💡 Check admin UI for connected workers: http://localhost:23646/"
@echo "💡 Check admin UI for connected workers: http://localhost:12535/"
@echo ""
@echo "3️⃣ Generating load to trigger EC tasks..."
@echo "📝 Creating test files to fill volumes..."
@@ -193,13 +193,13 @@ test: ## Run integration test to verify task assignment and completion
@echo " • EC shards being distributed"
@echo ""
@echo "✅ Integration test setup complete!"
@echo "📊 Monitor progress at: http://localhost:23646/"
@echo "📊 Monitor progress at: http://localhost:12535/"
quick-test: ## Quick verification that core services are running
@echo "⚡ Quick Health Check"
@echo "===================="
@echo "Master: $$(curl -s http://localhost:9333/cluster/status | jq -r '.IsLeader // "not ready"')"
@echo "Admin: $$(curl -s http://localhost:23646/ | grep -q "Admin" && echo "ready" || echo "not ready")"
@echo "Admin: $$(curl -s http://localhost:12535/ | grep -q "Admin" && echo "ready" || echo "not ready")"
@echo "Workers: $$(docker-compose -f $(COMPOSE_FILE) ps worker1 worker2 worker3 | grep -c Up) running"
validate: ## Validate integration test configuration
@@ -223,7 +223,7 @@ demo: start ## Start cluster and run demonstration
@echo " • Load generator (creates files continuously)"
@echo ""
@echo "📊 Watch the process:"
@echo " 1. Visit: http://localhost:23646/"
@echo " 1. Visit: http://localhost:12535/"
@echo " 2. Observe workers connecting"
@echo " 3. Watch tasks being created and assigned"
@echo " 4. See tasks progress from pending → completed"
@@ -244,13 +244,13 @@ vacuum-test: ## Create test data with garbage and verify vacuum detection
@echo ""
@echo "1️⃣ Checking cluster health..."
@curl -s http://localhost:9333/cluster/status | jq '.IsLeader' > /dev/null && echo "✅ Master ready" || (echo "❌ Master not ready. Run 'make start' first." && exit 1)
@curl -s http://localhost:23646/ | grep -q "Admin" && echo "✅ Admin ready" || (echo "❌ Admin not ready. Run 'make start' first." && exit 1)
@curl -s http://localhost:12535/ | grep -q "Admin" && echo "✅ Admin ready" || (echo "❌ Admin not ready. Run 'make start' first." && exit 1)
@echo ""
@echo "2️⃣ Creating test data with garbage..."
@docker-compose -f $(COMPOSE_FILE) exec vacuum-tester go run create_vacuum_test_data.go -files=25 -delete=0.5 -size=200
@echo ""
@echo "3️⃣ Configuration Instructions:"
@echo " Visit: http://localhost:23646/maintenance/config/vacuum"
@echo " Visit: http://localhost:12535/maintenance/config/vacuum"
@echo " Set for testing:"
@echo " • Enable Vacuum Tasks: ✅ Checked"
@echo " • Garbage Threshold: 0.20 (20%)"
@@ -258,7 +258,7 @@ vacuum-test: ## Create test data with garbage and verify vacuum detection
@echo " • Min Volume Age: [0] [Minutes]"
@echo " • Max Concurrent: 2"
@echo ""
@echo "4️⃣ Monitor vacuum tasks at: http://localhost:23646/maintenance"
@echo "4️⃣ Monitor vacuum tasks at: http://localhost:12535/maintenance"
@echo ""
@echo "💡 Use 'make vacuum-status' to check volume garbage ratios"
@@ -335,8 +335,8 @@ vacuum-help: ## Show vacuum testing help and examples
@echo " make vacuum-demo # Full guided demonstration"
@echo ""
@echo "Configuration:"
@echo " Visit: http://localhost:23646/maintenance/config/vacuum"
@echo " Monitor: http://localhost:23646/maintenance"
@echo " Visit: http://localhost:12535/maintenance/config/vacuum"
@echo " Monitor: http://localhost:12535/maintenance"
@echo ""
@echo "Custom Parameters:"
@echo " make vacuum-data FILES=50 DELETE=0.8 SIZE=200"
@@ -243,7 +243,7 @@ func printTestingInstructions() {
fmt.Println("🧪 Testing Instructions:")
fmt.Println()
fmt.Println("1. Configure Vacuum for Testing:")
fmt.Println(" Visit: http://localhost:23646/maintenance/config/vacuum")
fmt.Println(" Visit: http://localhost:12535/maintenance/config/vacuum")
fmt.Println(" Set:")
fmt.Printf(" - Garbage Percentage Threshold: 20 (20%% - lower than default 30)\n")
fmt.Printf(" - Scan Interval: [30] [Seconds] (faster than default)\n")
@@ -253,7 +253,7 @@ func printTestingInstructions() {
fmt.Println()
fmt.Println("2. Monitor Vacuum Tasks:")
fmt.Println(" Visit: http://localhost:23646/maintenance")
fmt.Println(" Visit: http://localhost:12535/maintenance")
fmt.Println(" Watch for vacuum tasks to appear in the queue")
fmt.Println()
@@ -7,7 +7,7 @@ echo ""
# Check if SeaweedFS is running
echo "📋 Checking SeaweedFS status..."
MASTER_URL="${MASTER_HOST:-master:9333}"
ADMIN_URL="${ADMIN_HOST:-admin:23646}"
ADMIN_URL="${ADMIN_HOST:-admin:12535}"
if ! curl -s http://$MASTER_URL/cluster/status > /dev/null; then
echo "❌ SeaweedFS master not running at $MASTER_URL"
@@ -42,7 +42,7 @@ echo ""
# Phase 3: Configure vacuum
echo "⚙️ Phase 3: Vacuum configuration instructions..."
echo " 1. Visit: http://localhost:23646/maintenance/config/vacuum"
echo " 1. Visit: http://localhost:12535/maintenance/config/vacuum"
echo " 2. Set these values for testing:"
echo " - Enable Vacuum Tasks: ✅ Checked"
echo " - Garbage Threshold: 0.30"
@@ -57,7 +57,7 @@ echo ""
# Phase 4: Monitor tasks
echo "🎯 Phase 4: Monitoring vacuum tasks..."
echo " Visit: http://localhost:23646/maintenance"
echo " Visit: http://localhost:12535/maintenance"
echo " You should see vacuum tasks appear within 30 seconds"
echo ""
@@ -93,9 +93,9 @@ echo ""
echo "🎉 Demo Complete!"
echo ""
echo "🔍 Things to check:"
echo " 1. Maintenance Queue: http://localhost:23646/maintenance"
echo " 1. Maintenance Queue: http://localhost:12535/maintenance"
echo " 2. Volume Status: http://localhost:9333/vol/status"
echo " 3. Admin Dashboard: http://localhost:23646"
echo " 3. Admin Dashboard: http://localhost:12535"
echo ""
echo "💡 Next Steps:"
echo " - Try different garbage thresholds (0.10, 0.50, 0.80)"
@@ -114,9 +114,9 @@ services:
admin:
image: chrislusf/seaweedfs:local
ports:
- "23646:23646" # HTTP admin interface (default port)
- "33646:33646" # gRPC worker communication (23646 + 10000)
command: "-v=2 admin -port=23646 -masters=master:9333 -dataDir=/data"
- "12535:12535" # HTTP admin interface (default port)
- "22535:22535" # gRPC worker communication (12535 + 10000)
command: "-v=2 admin -port=12535 -masters=master:9333 -dataDir=/data"
depends_on:
- master
- filer
@@ -127,7 +127,7 @@ services:
worker1:
image: chrislusf/seaweedfs:local
command: "-v=2 worker -admin=admin:23646 -capabilities=erasure_coding,vacuum -maxConcurrent=2"
command: "-v=2 worker -admin=admin:12535 -capabilities=erasure_coding,vacuum -maxConcurrent=2"
depends_on:
- admin
volumes:
@@ -139,7 +139,7 @@ services:
worker2:
image: chrislusf/seaweedfs:local
command: "-v=2 worker -admin=admin:23646 -capabilities=erasure_coding,vacuum -maxConcurrent=2"
command: "-v=2 worker -admin=admin:12535 -capabilities=erasure_coding,vacuum -maxConcurrent=2"
depends_on:
- admin
volumes:
@@ -151,7 +151,7 @@ services:
worker3:
image: chrislusf/seaweedfs:local
command: "-v=2 worker -admin=admin:23646 -capabilities=erasure_coding,vacuum -maxConcurrent=2"
command: "-v=2 worker -admin=admin:12535 -capabilities=erasure_coding,vacuum -maxConcurrent=2"
depends_on:
- admin
volumes:
@@ -199,7 +199,7 @@ services:
curl -s http://master:9333/cluster/status | jq '.IsLeader, .Peers' || echo 'Master not ready';
echo;
echo 'Admin status:';
curl -s http://admin:23646/ | grep -o 'Admin.*Interface' || echo 'Admin not ready';
curl -s http://admin:12535/ | grep -o 'Admin.*Interface' || echo 'Admin not ready';
echo;
echo 'Volume count by server:';
curl -s http://master:9333/vol/status | jq '.Volumes | length' || echo 'Volumes not ready';
@@ -237,4 +237,4 @@ services:
- seaweed_net
environment:
- MASTER_HOST=master:9333
- ADMIN_HOST=admin:23646
- ADMIN_HOST=admin:12535
+2 -2
View File
@@ -23,7 +23,7 @@ else
fi
echo -e "${BLUE}2. Checking if required ports are available...${NC}"
for port in 9333 8080 8081 8082 8083 8084 8085 8888 23646; do
for port in 9333 8080 8081 8082 8083 8084 8085 8888 12535; do
if lsof -i :$port > /dev/null 2>&1; then
echo -e "${YELLOW}⚠️ Port $port is in use${NC}"
else
@@ -41,7 +41,7 @@ else
fi
echo -e "${BLUE}4. Verifying admin server configuration...${NC}"
if docker-compose -f docker-compose-ec-test.yml config | grep -q "admin:23646"; then
if docker-compose -f docker-compose-ec-test.yml config | grep -q "admin:12535"; then
echo -e "${GREEN}✅ Admin server port configuration is correct${NC}"
else
echo -e "${RED}❌ Admin server port configuration is incorrect${NC}"
+1 -1
View File
@@ -45,7 +45,7 @@ services:
admin:
image: chrislusf/seaweedfs:dev # use a remote dev image
ports:
- 23646:23646
- 12535:12535
command: 'admin -master=master:9333'
depends_on:
- master
+41 -40
View File
@@ -23,8 +23,8 @@ require (
github.com/facebookgo/stack v0.0.0-20160209184415-751773369052 // indirect
github.com/facebookgo/stats v0.0.0-20151006221625-1b76add642e4
github.com/facebookgo/subset v0.0.0-20200203212716-c811ad88dec4 // indirect
github.com/fsnotify/fsnotify v1.9.0 // indirect
github.com/go-redsync/redsync/v4 v4.17.0
github.com/fsnotify/fsnotify v1.10.1 // indirect
github.com/go-redsync/redsync/v4 v4.18.0
github.com/go-sql-driver/mysql v1.10.1
github.com/go-zookeeper/zk v1.0.4 // indirect
github.com/golang/protobuf v1.5.4
@@ -32,7 +32,7 @@ require (
github.com/google/btree v1.1.3
github.com/google/uuid v1.6.0
github.com/google/wire v0.7.0 // indirect
github.com/googleapis/gax-go/v2 v2.24.0 // indirect
github.com/googleapis/gax-go/v2 v2.24.1 // indirect
github.com/gorilla/mux v1.8.1
github.com/hashicorp/errwrap v1.1.0 // indirect
github.com/hashicorp/go-multierror v1.1.1 // indirect
@@ -44,10 +44,10 @@ require (
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/json-iterator/go v1.1.12
github.com/karlseguin/ccache/v2 v2.0.8
github.com/klauspost/compress v1.20.1
github.com/klauspost/compress v1.20.3
github.com/klauspost/reedsolomon v1.14.2
github.com/kurin/blazer v0.5.3
github.com/linxGnu/grocksdb v1.10.8
github.com/linxGnu/grocksdb v1.11.1
github.com/mailru/easyjson v0.9.2 // indirect
github.com/mattn/go-isatty v0.0.24 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
@@ -65,7 +65,7 @@ require (
github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 // indirect
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
github.com/seaweedfs/goexif v1.0.3
github.com/seaweedfs/raft v1.2.1
github.com/seaweedfs/raft v1.2.2
github.com/sirupsen/logrus v1.9.4 // indirect
github.com/spf13/afero v1.15.0 // indirect
github.com/spf13/cast v1.10.0 // indirect
@@ -73,7 +73,7 @@ require (
github.com/stretchr/testify v1.12.1
github.com/stvp/tempredis v0.0.0-20181119212430-b82af8480203
github.com/syndtr/goleveldb v1.0.1-0.20190318030020-c3a204f8e965
github.com/tidwall/gjson v1.19.0
github.com/tidwall/gjson v1.20.0
github.com/tidwall/match v1.2.0
github.com/tidwall/pretty v1.2.0 // indirect
github.com/tsuna/gohbase v0.0.0-20201125011725-348991136365
@@ -87,11 +87,11 @@ require (
go.etcd.io/etcd/client/v3 v3.7.1
go.mongodb.org/mongo-driver v1.17.9
go.opencensus.io v0.24.0 // indirect
gocloud.dev v0.46.0
gocloud.dev v0.47.0
gocloud.dev/pubsub/natspubsub v0.46.0
gocloud.dev/pubsub/rabbitpubsub v0.46.0
gocloud.dev/pubsub/rabbitpubsub v0.47.0
golang.org/x/crypto v0.58.0
golang.org/x/exp v0.0.0-20260709172345-9ea1abe57597
golang.org/x/exp v0.0.0-20260813180055-c1d0aacb2297
golang.org/x/image v0.46.0
golang.org/x/net v0.61.0
golang.org/x/oauth2 v0.37.0
@@ -100,14 +100,14 @@ require (
golang.org/x/tools v0.51.0 // indirect
golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da // indirect
google.golang.org/api v0.297.0
google.golang.org/genproto v0.0.0-20260715232425-e75dac1f907d // indirect
google.golang.org/genproto v0.0.0-20260904194346-d0f1323225a4 // indirect
google.golang.org/grpc v1.85.0-dev.0.20260915183914-4e49413dcab7
google.golang.org/protobuf v1.36.12
gopkg.in/inf.v0 v0.9.1 // indirect
modernc.org/b v1.0.0 // indirect
modernc.org/mathutil v1.7.1 // indirect
modernc.org/memory v1.11.0 // indirect
modernc.org/sqlite v1.57.0
modernc.org/memory v1.12.1 // indirect
modernc.org/sqlite v1.60.1
)
require (
@@ -116,13 +116,13 @@ require (
github.com/DATA-DOG/go-sqlmock v1.5.2
github.com/Jille/raft-grpc-transport v1.6.1
github.com/ThreeDotsLabs/watermill v1.5.2
github.com/a-h/templ v0.3.1020
github.com/a-h/templ v0.3.1070
github.com/apache/cassandra-gocql-driver/v2 v2.1.2
github.com/apache/iceberg-go v0.7.0
github.com/apple/foundationdb/bindings/go v0.0.0-20250911184653-27f7192f47c3
github.com/arangodb/go-driver v1.6.9
github.com/armon/go-metrics v0.4.1
github.com/aws/aws-sdk-go-v2 v1.47.1
github.com/aws/aws-sdk-go-v2 v1.47.3
github.com/aws/aws-sdk-go-v2/config v1.33.6
github.com/aws/aws-sdk-go-v2/credentials v1.20.6
github.com/aws/aws-sdk-go-v2/service/s3 v1.114.1
@@ -157,7 +157,7 @@ require (
github.com/twmb/avro v1.9.0
github.com/xeipuuv/gojsonschema v1.2.0
github.com/ydb-platform/ydb-go-sdk-auth-environ v0.5.3
github.com/ydb-platform/ydb-go-sdk/v3 v3.151.1
github.com/ydb-platform/ydb-go-sdk/v3 v3.155.1
go.etcd.io/etcd/client/pkg/v3 v3.7.1
go.uber.org/atomic v1.12.0
golang.org/x/sync v0.24.0
@@ -169,9 +169,10 @@ require github.com/k0kubun/colorstring v0.0.0-20150214042306-9440f1994b88 // ind
require (
cloud.google.com/go/longrunning v1.2.0 // indirect
cloud.google.com/go/pubsub/v2 v2.6.0 // indirect
cloud.google.com/go/pubsub/v2 v2.7.0 // indirect
dario.cat/mergo v1.0.2 // indirect
github.com/Azure/azure-sdk-for-go/sdk/keyvault/internal v0.7.1 // indirect
github.com/Azure/azure-sdk-for-go/sdk/storage/internal v0.1.0 // indirect
github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c // indirect
github.com/FilenCloudDienste/filen-sdk-go v0.0.39 // indirect
github.com/ProtonMail/gopenpgp/v3 v3.4.1 // indirect
@@ -231,7 +232,7 @@ require (
github.com/jtolds/gls v4.20.0+incompatible // indirect
github.com/kr/pretty v0.3.1 // indirect
github.com/kr/text v0.2.0 // indirect
github.com/lib/pq v1.12.0 // indirect
github.com/lib/pq v1.12.3 // indirect
github.com/lithammer/shortuuid/v3 v3.0.7 // indirect
github.com/lpar/calendar v0.2.0 // indirect
github.com/magiconair/properties v1.8.10 // indirect
@@ -274,8 +275,8 @@ require (
github.com/xeipuuv/gojsonpointer v0.0.0-20190905194746-02993c407bfb // indirect
github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415 // indirect
github.com/zeebo/xxh3 v1.1.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.45.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.45.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.46.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.46.0 // indirect
go.opentelemetry.io/otel/exporters/zipkin v1.45.0 // indirect
go.opentelemetry.io/proto/otlp v1.11.0 // indirect
go.uber.org/mock v0.5.2 // indirect
@@ -290,20 +291,20 @@ require (
cloud.google.com/go/auth v0.23.2 // indirect
cloud.google.com/go/auth/oauth2adapt v0.2.8 // indirect
cloud.google.com/go/compute/metadata v0.9.0 // indirect
cloud.google.com/go/iam v1.12.0 // indirect
cloud.google.com/go/iam v1.13.0 // indirect
cloud.google.com/go/monitoring v1.30.0 // indirect
filippo.io/edwards25519 v1.2.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.23.1
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.23.2
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.14.1
github.com/Azure/azure-sdk-for-go/sdk/internal v1.12.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.8.0
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.8.2
github.com/Azure/azure-sdk-for-go/sdk/storage/azfile v1.7.0 // indirect
github.com/Azure/go-ntlmssp v0.1.1 // indirect
github.com/AzureAD/microsoft-authentication-library-for-go v1.8.0 // indirect
github.com/AzureAD/microsoft-authentication-library-for-go v1.9.0 // indirect
github.com/Files-com/files-sdk-go/v3 v3.3.194 // indirect
github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.35.0 // indirect
github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/metric v0.57.0 // indirect
github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/resourcemapping v0.57.0 // indirect
github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.37.0 // indirect
github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/metric v0.61.0 // indirect
github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/resourcemapping v0.61.0 // indirect
github.com/IBM/go-sdk-core/v5 v5.23.1 // indirect
github.com/Max-Sum/base32768 v0.0.0-20230304063302-18e6ce5945fd // indirect
github.com/Microsoft/go-winio v0.6.3-0.20251027160822-ad3df93bed29 // indirect
@@ -313,7 +314,7 @@ require (
github.com/ProtonMail/go-srp v0.0.7 // indirect
github.com/PuerkitoBio/goquery v1.12.0 // indirect
github.com/abbot/go-http-auth v0.4.0 // indirect
github.com/andybalholm/brotli v1.2.3 // indirect
github.com/andybalholm/brotli v1.2.6 // indirect
github.com/andybalholm/cascadia v1.3.4 // indirect
github.com/appscode/go-querystring v0.0.0-20170504095604-0126cfb3f1dc // indirect
github.com/arangodb/go-velocypack v0.0.0-20200318135517-5af53c29c67e // indirect
@@ -327,12 +328,12 @@ require (
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.11.5 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.14.4 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.20.4 // indirect
github.com/aws/aws-sdk-go-v2/service/sns v1.39.14 // indirect
github.com/aws/aws-sdk-go-v2/service/sqs v1.42.24 // indirect
github.com/aws/aws-sdk-go-v2/service/sns v1.46.0 // indirect
github.com/aws/aws-sdk-go-v2/service/sqs v1.51.0 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.38.1 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.43.1 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.51.1
github.com/aws/smithy-go v1.28.1
github.com/aws/smithy-go v1.28.5
github.com/boltdb/bolt v1.3.1 // indirect
github.com/bradenaw/juniper v0.15.3 // indirect
github.com/buengese/sgzip v0.1.1 // indirect
@@ -363,7 +364,7 @@ require (
github.com/geoffgarside/ber v1.2.0 // indirect
github.com/go-chi/chi/v5 v5.3.1 // indirect
github.com/go-darwin/apfs v0.0.0-20211011131704-f84b94dbf348 // indirect
github.com/go-jose/go-jose/v4 v4.1.4 // indirect
github.com/go-jose/go-jose/v4 v4.1.5 // indirect
github.com/go-logr/logr v1.4.4 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-ole/go-ole v1.3.0 // indirect
@@ -384,7 +385,7 @@ require (
github.com/gorilla/securecookie v1.1.2 // indirect
github.com/gorilla/sessions v1.4.0
github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.29.0 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0 // indirect
github.com/hashicorp/go-cleanhttp v0.5.2 // indirect
github.com/hashicorp/go-hclog v1.6.3 // indirect
github.com/hashicorp/go-immutable-radix v1.3.1 // indirect
@@ -480,25 +481,25 @@ require (
go.etcd.io/bbolt v1.5.0 // indirect
go.etcd.io/etcd/api/v3 v3.7.1 // indirect
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
go.opentelemetry.io/contrib/detectors/gcp v1.45.0 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.69.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.70.0 // indirect
go.opentelemetry.io/contrib/detectors/gcp v1.46.0 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.71.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.71.0 // indirect
go.opentelemetry.io/otel v1.46.0 // indirect
go.opentelemetry.io/otel/metric v1.46.0 // indirect
go.opentelemetry.io/otel/sdk v1.46.0 // indirect
go.opentelemetry.io/otel/sdk/metric v1.46.0 // indirect
go.opentelemetry.io/otel/trace v1.46.0 // indirect
go.uber.org/multierr v1.11.0 // indirect
go.uber.org/zap v1.27.1 // indirect
go.uber.org/zap v1.28.0 // indirect
golang.org/x/term v0.47.0
golang.org/x/time v0.16.0
google.golang.org/genproto/googleapis/api v0.0.0-20260817212433-ac3dfec99bb1 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20260819154853-08b0e4226688 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20260904194346-d0f1323225a4 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20260904194346-d0f1323225a4 // indirect
gopkg.in/natefinch/lumberjack.v2 v2.2.1 // indirect
gopkg.in/validator.v2 v2.0.1 // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
modernc.org/libc v1.74.4 // indirect
modernc.org/libc v1.77.1 // indirect
moul.io/http2curl/v2 v2.3.0 // indirect
sigs.k8s.io/yaml v1.6.0 // indirect
storj.io/common v0.0.0-20260629224719-ba1bff0a7846 // indirect
+104 -102
View File
@@ -10,20 +10,20 @@ cloud.google.com/go/auth/oauth2adapt v0.2.8 h1:keo8NaayQZ6wimpNSmW5OPc283g65QNIi
cloud.google.com/go/auth/oauth2adapt v0.2.8/go.mod h1:XQ9y31RkqZCcwJWNSx2Xvric3RrU88hAYYbjDWYDL+c=
cloud.google.com/go/compute/metadata v0.9.0 h1:pDUj4QMoPejqq20dK0Pg2N4yG9zIkYGdBtwLoEkH9Zs=
cloud.google.com/go/compute/metadata v0.9.0/go.mod h1:E0bWwX5wTnLPedCKqk3pJmVgCBSM6qQI1yTBdEb3C10=
cloud.google.com/go/iam v1.12.0 h1:Aki3bX9aHUDKPHfnRJfDcTdVedvy6quGBQcTqx3DRXk=
cloud.google.com/go/iam v1.12.0/go.mod h1:FEZ4lXpADAC2AIpQY7LANNjjwyQ2jK439CI2VaD+sLY=
cloud.google.com/go/iam v1.13.0 h1:ufT3FPT5rFFXu6UtLkNoxaOaV5EuA1dsSkmemCSTo6U=
cloud.google.com/go/iam v1.13.0/go.mod h1:gHXdDEiPDvqd1q1KwBDGQlgZY/BwY760zU2LhOZS5w0=
cloud.google.com/go/kms v1.35.0 h1:nJ/ktaqspx1nPM9vIcO0SHbhqCAm8nvAxL1siuVgKm0=
cloud.google.com/go/kms v1.35.0/go.mod h1:0++71pIHvJL+GmMa8K4jOWFq7gNOX3jm2PRMSJwTKJw=
cloud.google.com/go/logging v1.19.0 h1:NCqhdVUg3wQ8Cobdf16FDSuTGi3+6+hdSBHrY5TsR6Q=
cloud.google.com/go/logging v1.19.0/go.mod h1:i40NZCHC9Gqvod4yE+yQfDWwlgwW/SrshkkGibCHxcA=
cloud.google.com/go/logging v1.19.1 h1:7SsLhyTDBDrJw+Ll6Ns3I2mByqHXvJUc3rGjSlwiWgU=
cloud.google.com/go/logging v1.19.1/go.mod h1:2IkQ/d8jVJqV2qW8ZUGUiMjdZG1gkLD2JReGbZ8isqg=
cloud.google.com/go/longrunning v1.2.0 h1:WjYH3YHBGCxGJP9M4dWGHBfXr/cFIjMkNgWcJj7/iMM=
cloud.google.com/go/longrunning v1.2.0/go.mod h1:5KMQALFGOCtFoi2xSOA1u3H7WKlhmckgiyFw7+LGQp0=
cloud.google.com/go/monitoring v1.30.0 h1:r/d+JUbyKmJ8b07iznuKfzVzrIXTWxHQ3lBRm3x2LlY=
cloud.google.com/go/monitoring v1.30.0/go.mod h1:htlUR0QWVMrjFzZmN4LGnMAve9xB/eduwjmINxVZ8RM=
cloud.google.com/go/pubsub v1.51.1 h1:R3G1wCOxBO7jRpL8x2pdZMv1GAJDF6ax/m2zPOtvTNE=
cloud.google.com/go/pubsub v1.51.1/go.mod h1:y2T0IKtW1iWwVvazYaRpqOAFO4gy2+O7dTDt9TWY/5U=
cloud.google.com/go/pubsub/v2 v2.6.0 h1:8pjR0id+GTB+krKx5G6AGJoYrHog58w2Q89PCOrfM64=
cloud.google.com/go/pubsub/v2 v2.6.0/go.mod h1:4anqvV/w8Pcgu2tO0qr2XgsF3GXHowzryfQ5gOnVmWY=
cloud.google.com/go/pubsub/v2 v2.7.0 h1:MFrBTZZa6PDWZzCi4NJRsHKMm2w0a4oAaYNqwjgbQTE=
cloud.google.com/go/pubsub/v2 v2.7.0/go.mod h1:JaFvWNVRk3Knoil/4M1ECeLOaI9D8drbmJWypQlK5aM=
cloud.google.com/go/storage v1.67.0 h1:8xD3NvKuHE4c+b/kmSIVBdTc0BiOjplx+KecKGR9fr8=
cloud.google.com/go/storage v1.67.0/go.mod h1:UsS9OgFg/XHOSYakQ8ZtLWWeyGkk1WnmD/GsGfN0BHM=
cloud.google.com/go/trace v1.16.0 h1:GmQovzFc5F0CNfl0VLgL64aoTtu7xsM0YajW2GlG9+E=
@@ -34,8 +34,8 @@ filippo.io/edwards25519 v1.2.0 h1:crnVqOiS4jqYleHd9vaKZ+HKtHfllngJIiOpNpoJsjo=
filippo.io/edwards25519 v1.2.0/go.mod h1:xzAOLCNug/yB62zG1bQ8uziwrIqIuxhctzJT18Q77mc=
github.com/AdaLogics/go-fuzz-headers v0.0.0-20240806141605-e8a1dd7889d6 h1:He8afgbRMd7mFxO99hRNu+6tazq8nFF9lIwo9JFroBk=
github.com/AdaLogics/go-fuzz-headers v0.0.0-20240806141605-e8a1dd7889d6/go.mod h1:8o94RPi1/7XTJvwPpRSzSUedZrtlirdB3r9Z20bi2f8=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.23.1 h1:zvXfGJCWvywnCA814d8ZiVyt+fm9nnTE8xSb99zRyfo=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.23.1/go.mod h1:iptorS+VYKFL2N6PnebpS91dubG35eAOEERnT4PJbQU=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.23.2 h1:utpeoEeZjd+A8J41zvoLsOOrqXHhX1Kx/X/tCW9dEYQ=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.23.2/go.mod h1:iptorS+VYKFL2N6PnebpS91dubG35eAOEERnT4PJbQU=
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.14.1 h1:u93s+zU2JD62im61Bm5CZIc1ZrOJaIAWEg0WOrMVkEo=
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.14.1/go.mod h1:oXtinPO4OLj9d1DOTrqrL1oRwGhcqadvAmrl6wTeGlk=
github.com/Azure/azure-sdk-for-go/sdk/azidentity/cache v0.4.0 h1:xFaZZ+IubdftrDHnGGwZ6QvQ3KHTtWl2MCK+GMt2vxs=
@@ -48,18 +48,20 @@ github.com/Azure/azure-sdk-for-go/sdk/keyvault/internal v0.7.1 h1:FbH3BbSb4bvGlu
github.com/Azure/azure-sdk-for-go/sdk/keyvault/internal v0.7.1/go.mod h1:9V2j0jn9jDEkCkv8w/bKTNppX/d0FVA1ud77xCIP4KA=
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage v1.8.1 h1:/Zt+cDPnpC3OVDm/JKLOs7M2DKmLRIIp3XIx9pHHiig=
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage v1.8.1/go.mod h1:Ng3urmn6dYe8gnbCMoHHVl5APYz2txho3koEkV2o2HA=
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.8.0 h1:irsmOWwkp0KCTTNS5e2hdFeIvSQClQo2No3IaNmL3Vw=
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.8.0/go.mod h1:GWcBkQj3MqN7ozHKLaCCAuNLiXoIGv2RtanfAwSjY/Y=
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.8.2 h1:tQJdgQjbKKffEiMOKJmcQPkY01eFHRqWIZVefqP44nI=
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.8.2/go.mod h1:kTkpJchXAdO7LGkKZP97a+Ou8TgaF4NT6MHwOIy5uSQ=
github.com/Azure/azure-sdk-for-go/sdk/storage/azfile v1.7.0 h1:cuiKf1UVyWHu+XSQghPZR/qEF43JIcuk2CDqMlPiT6M=
github.com/Azure/azure-sdk-for-go/sdk/storage/azfile v1.7.0/go.mod h1:9JSyvgXLPAOC4jfhgZg58XeU2FJHsGmxuYSclCFQ4ZY=
github.com/Azure/azure-sdk-for-go/sdk/storage/internal v0.1.0 h1:X/bTwe0qpvmvPPutc+LWo7T4cKMqDP9JHP4k+Co9ZB4=
github.com/Azure/azure-sdk-for-go/sdk/storage/internal v0.1.0/go.mod h1:+kX7ahyFelQ42F3RScM2CFgHJljp0phc/FieKLYVpFI=
github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c h1:udKWzYgxTojEKWjV8V+WSxDXJ4NFATAsZjh8iIbsQIg=
github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c/go.mod h1:xomTg63KZ2rFqZQzSB4Vz2SUXa1BpHTVz9L5PTmPC4E=
github.com/Azure/go-ntlmssp v0.1.1 h1:l+FM/EEMb0U9QZE7mKNEDw5Mu3mFiaa2GKOoTSsNDPw=
github.com/Azure/go-ntlmssp v0.1.1/go.mod h1:NYqdhxd/8aAct/s4qSYZEerdPuH1liG2/X9DiVTbhpk=
github.com/AzureAD/microsoft-authentication-extensions-for-go/cache v0.1.1 h1:WJTmL004Abzc5wDB5VtZG2PJk5ndYDgVacGqfirKxjM=
github.com/AzureAD/microsoft-authentication-extensions-for-go/cache v0.1.1/go.mod h1:tCcJZ0uHAmvjsVYzEFivsRTN00oz5BEsRgQHu5JZ9WE=
github.com/AzureAD/microsoft-authentication-library-for-go v1.8.0 h1:Nljr4q1GRA/5vCrMONS+g4u4LRHNgOXVSh3O43J2CnI=
github.com/AzureAD/microsoft-authentication-library-for-go v1.8.0/go.mod h1:Y33QHnf0FfdVewFFISOGe20mkZbxX4H839o955/PoeI=
github.com/AzureAD/microsoft-authentication-library-for-go v1.9.0 h1:MDT4FxAPve5FnYn6vOL1r7RCRDG+l9cI7a5LlCuHsqA=
github.com/AzureAD/microsoft-authentication-library-for-go v1.9.0/go.mod h1:Y33QHnf0FfdVewFFISOGe20mkZbxX4H839o955/PoeI=
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
github.com/Codefor/geohash v0.0.0-20140723084247-1b41c28e3a9d h1:iG9B49Q218F/XxXNRM7k/vWf7MKmLIS8AcJV9cGN4nA=
github.com/Codefor/geohash v0.0.0-20140723084247-1b41c28e3a9d/go.mod h1:RVnhzAX71far8Kc3TQeA0k/dcaEKUnTDSOyet/JCmGI=
@@ -73,14 +75,14 @@ github.com/FilenCloudDienste/filen-sdk-go v0.0.39 h1:tgV5jYL6dsXop9TpDTIQU6UwJjw
github.com/FilenCloudDienste/filen-sdk-go v0.0.39/go.mod h1:0cBhKXQg49XbKZZfk5TCDa3sVLP+xMxZTWL+7KY0XR0=
github.com/Files-com/files-sdk-go/v3 v3.3.194 h1:dtOFxSTWWRpkmvXa6ycNiw8dVDu1wkgzcXyVV1VafNc=
github.com/Files-com/files-sdk-go/v3 v3.3.194/go.mod h1:rl0WumSN9gSo775DgvQv+wMQ8rlb0ES/1hU5jkMtLXg=
github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.35.0 h1:bN1gA3of5bXtbnLsRPrwfmbbe7A5UWFlcTHseujLnpc=
github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.35.0/go.mod h1:Yj5vHEz/aAepZGliRJsA6uvHAVAQyEwajq9ORCHPxzM=
github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/metric v0.57.0 h1:jLdiS1vO+XJFyDSWRHBx56r4s/NNtcl5J6KyCcWUX/w=
github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/metric v0.57.0/go.mod h1:8lmpHY+1VRoteiOwyrQMDt1YGXOrFKCz+1wJW7n3ODY=
github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/cloudmock v0.57.0 h1:cSjUzZ7KU8hicTgzaSv9NmSyM9fTVK3y5lsBUl3wOis=
github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/cloudmock v0.57.0/go.mod h1:dzcEjy1WJ0Q4u9twNR3LcLhNoYMRCrMCMafpxa0TjPQ=
github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/resourcemapping v0.57.0 h1:RoO5+d7uCmDqovLrHCr2/BuViUXvdcrNxyNM1pN9dDQ=
github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/resourcemapping v0.57.0/go.mod h1:YqwkQPrWSC7+byyc1VlKbWLBF5JsW5IoL6xUkemYSXk=
github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.37.0 h1:edOuLWehuDlQ68roBF900Y09jvBoJP5W5d/wusz9nWY=
github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.37.0/go.mod h1:iIyrdhveU5Ow84ipRlxtjCOQnIM0DiqSMxuxm50xwZY=
github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/metric v0.61.0 h1:BeC1Bub7Ttk33TF8v3ZeQ2L8acvz4ZLqtQuLyJAZquc=
github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/metric v0.61.0/go.mod h1:UgG/Xn9+NXb1/ueSczIFxyuya3hs3srFDiZG2JrpJwY=
github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/cloudmock v0.61.0 h1:eSgSuKdcpY7RTN1oXSzC6S39rPBImn03ubLMsWK4RRg=
github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/cloudmock v0.61.0/go.mod h1:ifhe5teRNcbxg9p353DhOkOgczzL/IBLYFqT1cm1zpM=
github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/resourcemapping v0.61.0 h1:KhpL26/GviYITHMCEGGqT/l50siFaBCBBKrjv6Z0G/k=
github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/resourcemapping v0.61.0/go.mod h1:sg6Wqbl0BKKZ5zKnQj2q0qic/3ziF894SCx1LpbX37Y=
github.com/IBM/go-sdk-core/v5 v5.23.1 h1:fxLusCG+GlGY1SM7BYAAiiSFhzKUs5zqs0P8knDqeFU=
github.com/IBM/go-sdk-core/v5 v5.23.1/go.mod h1:yO+OQpByKDLTvpEcsFFexgzpeR8eRfCFWAYzxkAu4bk=
github.com/Jille/raft-grpc-transport v1.6.1 h1:gN3sjapb+fVbiebS7AfQQgbV2ecTOI7ur7NPPC7Mhoc=
@@ -118,8 +120,8 @@ github.com/ThreeDotsLabs/watermill v1.5.2 h1:0ES33Eq1jEsP/pWvtE4n8bE0bs+9Jq7boT7
github.com/ThreeDotsLabs/watermill v1.5.2/go.mod h1:i9/968UriGphWfEbfMuYSD1qFbYRjb0mE0r+rV0FPp4=
github.com/TomiHiltunen/geohash-golang v0.0.0-20150112065804-b3e4e625abfb h1:wumPkzt4zaxO4rHPBrjDK8iZMR41C1qs7njNqlacwQg=
github.com/TomiHiltunen/geohash-golang v0.0.0-20150112065804-b3e4e625abfb/go.mod h1:QiYsIBRQEO+Z4Rz7GoI+dsHVneZNONvhczuA+llOZNM=
github.com/a-h/templ v0.3.1020 h1:ypAT/L5ySWEnZ6Zft/5yfoWXYYkhFNvEFOeeqecg4tw=
github.com/a-h/templ v0.3.1020/go.mod h1:A2DlK61v+K+NRoGnhmYbNYVmtYHcFO5/AisMvBdDxTM=
github.com/a-h/templ v0.3.1070 h1:3iCOXHXqFXa99OQzgkUOLJbNzv/QjFGLwKZkBAg+ADA=
github.com/a-h/templ v0.3.1070/go.mod h1:iIqB3g2RHBe/WlnPwa+Bvhh1TKXHWLRnBD9NWHOLm78=
github.com/a1ex3/zstd-seekable-format-go/pkg v0.10.0 h1:iLDOF0rdGTrol/q8OfPIIs5kLD8XvA2q75o6Uq/tgak=
github.com/a1ex3/zstd-seekable-format-go/pkg v0.10.0/go.mod h1:DrEWcQJjz7t5iF2duaiyhg4jyoF0kxOD6LtECNGkZ/Q=
github.com/aalpar/deheap v1.1.2 h1:MABHLcnjqsffb8GLkUFDigqpBBxOMz0DoKM9QfELeTw=
@@ -142,8 +144,8 @@ github.com/alexbrainman/sspi v0.0.0-20250919150558-7d374ff0d59e h1:4dAU9FXIyQktp
github.com/alexbrainman/sspi v0.0.0-20250919150558-7d374ff0d59e/go.mod h1:cEWa1LVoE5KvSD9ONXsZrj0z6KqySlCCNKHlLzbqAt4=
github.com/anchore/go-lzo v0.1.1 h1:IwL/fvkdtlIrYIXck6WxZ3nb8WjjHziYYmGxlooyOnM=
github.com/anchore/go-lzo v0.1.1/go.mod h1:3kLx0bve2oN1iDwgM1U5zGku1Tfbdb0No5qp1eL1fIk=
github.com/andybalholm/brotli v1.2.3 h1:8H1qwOkl2LPfjf3YezB90JnCliZb6SInJ/OJkEbA5NQ=
github.com/andybalholm/brotli v1.2.3/go.mod h1:rzTDkvFWvIrjDXZHkuS16NPggd91W3kUSvPlQ1pLaKY=
github.com/andybalholm/brotli v1.2.6 h1:ftYnfj6usCp+UGV5kSJ3+chpMQgU+gJf/AxsUQ52REI=
github.com/andybalholm/brotli v1.2.6/go.mod h1:rzTDkvFWvIrjDXZHkuS16NPggd91W3kUSvPlQ1pLaKY=
github.com/andybalholm/cascadia v1.3.4 h1:vM2lgh0Vru9Vwyfm4cQqWP2HHMW0u0+2PAW7Q38Qufg=
github.com/andybalholm/cascadia v1.3.4/go.mod h1:BLRmbRjpEtNKieZOCCvYj4RqN+KRA41GBe/5O+G93kM=
github.com/antihax/optional v1.0.0/go.mod h1:uupD/76wgC+ih3iEmQUL+0Ugr19nfwCT1kdvxnR2qWY=
@@ -171,8 +173,8 @@ github.com/armon/go-metrics v0.4.1 h1:hR91U9KYmb6bLBYLQjyM+3j+rcd/UhE+G78SFnF8gJ
github.com/armon/go-metrics v0.4.1/go.mod h1:E6amYzXo6aW1tqzoZGT755KkbgrJsSdpwZ+3JqfkOG4=
github.com/aws/aws-sdk-go v1.55.8 h1:JRmEUbU52aJQZ2AjX4q4Wu7t4uZjOu71uyNmaWlUkJQ=
github.com/aws/aws-sdk-go v1.55.8/go.mod h1:ZkViS9AqA6otK+JBBNH2++sx1sgxrPKcSzPPvQkUtXk=
github.com/aws/aws-sdk-go-v2 v1.47.1 h1:uOIZnp4PK3ZhKI0dNrJrhTEsLxbpXHTAJlwoS1pvAtw=
github.com/aws/aws-sdk-go-v2 v1.47.1/go.mod h1:bttEH6JqnUL8LepvDVfdrds/fZ5bCIxzpe3abyUrhDU=
github.com/aws/aws-sdk-go-v2 v1.47.3 h1:HRoT/MkNi3bjO5xF4dCzLzvVmkvRGHj3/cLINDnJejk=
github.com/aws/aws-sdk-go-v2 v1.47.3/go.mod h1:xSl+OSI5ZUw6bSMupWc218zE5Pl+65MCCu3ySWFY99A=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.20 h1:GPRlPwz40I2B2VrBEASOA3Bi77NyeqejNLkifosX0rs=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.20/go.mod h1:g7PNzKcsOKWb4fkSRBA7BZVAS6Y8IcxzN+nRohhQ1Q8=
github.com/aws/aws-sdk-go-v2/config v1.33.6 h1:MBjkSTLczek/UgiK+EYPIoRTqE7gP8vtW3OFbFo7Nug=
@@ -183,8 +185,8 @@ github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.20.1 h1:8gALAAmacnIXh+z6VkdDanv
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.20.1/go.mod h1:Z7IJhJU+poOdJjUR2wpyY21ossQ1XS/R3Lk9Msq5kM4=
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.22.34 h1:Pn7OsMwBLbkZ6OnCxWHAjf0L/22H8cnhxZC0uPwtMtg=
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.22.34/go.mod h1:eToXR/Gk1uqpn04eSmdgVXwfS0WvH8aG4eBFr8ygbpU=
github.com/aws/aws-sdk-go-v2/feature/s3/transfermanager v0.4.1 h1:I3mWvASaICc5c8vJ3ftYjroh7LT3jG0q/KtzSu5wW/s=
github.com/aws/aws-sdk-go-v2/feature/s3/transfermanager v0.4.1/go.mod h1:VwSN8piv62OyyxYWtJzj6j7gECyLc0WwMo30L5NQZlE=
github.com/aws/aws-sdk-go-v2/feature/s3/transfermanager v0.4.3 h1:vsFkmFrfzODkxVbyuPLscv5j5lC8KbG48LZwV+p56Y0=
github.com/aws/aws-sdk-go-v2/feature/s3/transfermanager v0.4.3/go.mod h1:pjQYVtvVBEtlsf2p6nXMsjPuDaWs9GM44FMdHbmnmYU=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.5.4 h1:CLq4+8UHCI+ZZYl/EuJxXovaIVN2xeeT8JV+dsApQ5E=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.5.4/go.mod h1:Wv4q5sAM04xAMkoOedxLx2inVf6K5FdxYp+A61L+q/0=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.8.4 h1:dD4MR81I7YkpEBRk6UP9rocC2QnT3qVuXwzlYTtfGEs=
@@ -203,18 +205,18 @@ github.com/aws/aws-sdk-go-v2/service/s3 v1.114.1 h1:9T6eO8/WXhij4PQaa2WaYsdekSCm
github.com/aws/aws-sdk-go-v2/service/s3 v1.114.1/go.mod h1:9APRWGLFITKD+xzWSIyT9V7QV4bNlEuIieWlzXgGFlI=
github.com/aws/aws-sdk-go-v2/service/signin v1.10.1 h1:DzCCWLzcIRQ77F3DEUljud7bEjTgFOIKXP52NmVRyhU=
github.com/aws/aws-sdk-go-v2/service/signin v1.10.1/go.mod h1:xpo/geVldu8payT375WekctUzopG/hBU7miiqItMUlw=
github.com/aws/aws-sdk-go-v2/service/sns v1.39.14 h1:p8WdWDh5AwSZdp19Haa3XMyPCICi9Z375a/Nu3IIEZY=
github.com/aws/aws-sdk-go-v2/service/sns v1.39.14/go.mod h1:NKVY7DER6VXHkt2I/ycmHakALNboi3Rqwt4eEf/1Cnk=
github.com/aws/aws-sdk-go-v2/service/sqs v1.42.24 h1:JP2wjWGmUp8lTCZb13Dv0Eciyc1jbO8pd0HZVMHFlrc=
github.com/aws/aws-sdk-go-v2/service/sqs v1.42.24/go.mod h1:Ql9ziDutk8ERAN9HMaYANCW3lop451ppebkxEJMLCTM=
github.com/aws/aws-sdk-go-v2/service/sns v1.46.0 h1:fD02zeo2Kc4hlFUvGX3FfA+gGOwd4BBcYgPDHDEzx9Y=
github.com/aws/aws-sdk-go-v2/service/sns v1.46.0/go.mod h1:/lB2ILjoT5/q0zSwtHnrJPjXBvjalnEIl0IgSEdvHec=
github.com/aws/aws-sdk-go-v2/service/sqs v1.51.0 h1:FghdY9ebeZf9iIan9J0xfBEID8arYibNBeZ1XHvs4oA=
github.com/aws/aws-sdk-go-v2/service/sqs v1.51.0/go.mod h1:wu7uOJ/2nAdenjvhHikeF6j0al99kfPVG6ZDU4e76OY=
github.com/aws/aws-sdk-go-v2/service/sso v1.38.1 h1:Umtl/0YZhng4xndfW3lKJrYYP7NLEjI6bGXVomwLcs0=
github.com/aws/aws-sdk-go-v2/service/sso v1.38.1/go.mod h1:rRD/dnm7q0HYE/I5TMaPgkWyyUGLcwuxHLABsLnQ3e0=
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.43.1 h1:orIWdNiLgzrhu/11RcPPKO/SBzUUymbUQuZbSPImghg=
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.43.1/go.mod h1:skwM/xsbR/1ReUTesv9BhpJp1VjajR7DWQnuVLwiXsQ=
github.com/aws/aws-sdk-go-v2/service/sts v1.51.1 h1:0HOqZXRvMytH6bFHVIc0oJX07sZjfhz0zXtjs6gdE8s=
github.com/aws/aws-sdk-go-v2/service/sts v1.51.1/go.mod h1:26zA0GhDrLo+yiLI2yXWxqB1PdsShfLikoI7GOEgugM=
github.com/aws/smithy-go v1.28.1 h1:R/nXH00c8qcfCzQVELtRw+eLQWtzv+VAIEFJ1/xxXlQ=
github.com/aws/smithy-go v1.28.1/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
github.com/aws/smithy-go v1.28.5 h1:OqbqKtzY9NUnr2yAe9+hiTdFb20tNaK/lI0PMOGXX9k=
github.com/aws/smithy-go v1.28.5/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
github.com/bahlo/generic-list-go v0.2.0 h1:5sz/EEAK+ls5wF+NeqDpk5+iNdMDXrh3z3nPnH1Wvgk=
github.com/bahlo/generic-list-go v0.2.0/go.mod h1:2KvAjgMlE5NNynlg/5iLrrCCZ2+5xWbdbCW3pNTGyYg=
github.com/bazelbuild/rules_go v0.46.0 h1:CTefzjN/D3Cdn3rkrM6qMWuQj59OBcuOjyIp3m4hZ7s=
@@ -462,8 +464,8 @@ github.com/fsnotify/fsevents v0.2.0 h1:BRlvlqjvNTfogHfeBOFvSC9N0Ddy+wzQCQukyoD7o
github.com/fsnotify/fsevents v0.2.0/go.mod h1:B3eEk39i4hz8y1zaWS/wPrAP4O6wkIl7HQwKBr1qH/w=
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
github.com/fsnotify/fsnotify v1.6.0/go.mod h1:sl3t1tCWJFWoRz9R8WJCbQihKKwmorjAbSClcnxKAGw=
github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
github.com/fsnotify/fsnotify v1.10.1 h1:b0/UzAf9yR5rhf3RPm9gf3ehBPpf0oZKIjtpKrx59Ho=
github.com/fsnotify/fsnotify v1.10.1/go.mod h1:TLheqan6HD6GBK6PrDWyDPBaEV8LspOxvPSjC+bVfgo=
github.com/fvbommel/sortorder v1.1.0 h1:fUmoe+HLsBTctBDoaBwpQo5N+nrCp8g/BjKb/6ZQmYw=
github.com/fvbommel/sortorder v1.1.0/go.mod h1:uk88iVf1ovNn1iLfgUVU2F9o5eO30ui720w+kxuqRs0=
github.com/gabriel-vasile/mimetype v1.4.13 h1:46nXokslUBsAJE/wMsp5gtO500a4F3Nkz9Ufpk2AcUM=
@@ -489,8 +491,8 @@ github.com/go-errors/errors v1.5.1 h1:ZwEMSLRCapFLflTpT7NKaAc7ukJ8ZPEjzlxt8rPN8b
github.com/go-errors/errors v1.5.1/go.mod h1:sIVyrIiJhuEF+Pj9Ebtd6P/rEYROXFi3BopGUQ5a5Og=
github.com/go-git/go-billy/v5 v5.9.0 h1:jItGXszUDRtR/AlferWPTMN4j38BQ88XnXKbilmmBPA=
github.com/go-git/go-billy/v5 v5.9.0/go.mod h1:jCnQMLj9eUgGU7+ludSTYoZL/GGmii14RxKFj7ROgHw=
github.com/go-jose/go-jose/v4 v4.1.4 h1:moDMcTHmvE6Groj34emNPLs/qtYXRVcd6S7NHbHz3kA=
github.com/go-jose/go-jose/v4 v4.1.4/go.mod h1:x4oUasVrzR7071A4TnHLGSPpNOm2a21K9Kf04k1rs08=
github.com/go-jose/go-jose/v4 v4.1.5 h1:RjgjO2LOtWOJKUC5wpwY9LR3B3vwVAz6JS2YHfYU6eA=
github.com/go-jose/go-jose/v4 v4.1.5/go.mod h1:x4oUasVrzR7071A4TnHLGSPpNOm2a21K9Kf04k1rs08=
github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
github.com/go-kit/kit v0.9.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
github.com/go-kit/log v0.1.0/go.mod h1:zbhenjAZHb184qTLMA9ZjW7ThYL0H2mk7Q6pNt4vbaY=
@@ -527,8 +529,8 @@ github.com/go-redis/redis/v7 v7.4.1 h1:PASvf36gyUpr2zdOUS/9Zqc80GbM+9BDyiJSJDDOr
github.com/go-redis/redis/v7 v7.4.1/go.mod h1:JDNMw23GTyLNC4GZu9njt15ctBQVn7xjRfnwdHj/Dcg=
github.com/go-redis/redis/v8 v8.11.5 h1:AcZZR7igkdvfVmQTPnu9WE37LRrO/YrBH5zWyjDC0oI=
github.com/go-redis/redis/v8 v8.11.5/go.mod h1:gREzHqY1hg6oD9ngVRbLStwAWKhA0FEgq8Jd4h5lpwo=
github.com/go-redsync/redsync/v4 v4.17.0 h1:FFJ+uxZs44y4Sq10//IFKic9T94AYl+u3Sog98AHzBo=
github.com/go-redsync/redsync/v4 v4.17.0/go.mod h1:CKVA6qwT07S/916i+Yd9h1/8YFQhCCpPYTQhvvYytJo=
github.com/go-redsync/redsync/v4 v4.18.0 h1:DDM09PEhmNTf/vziolf2x6391y1hHxF5vQ4ytHl9xQc=
github.com/go-redsync/redsync/v4 v4.18.0/go.mod h1:yuoqcQ55FS1VypSeVTMqFsJqTdWpEH5eA4MQGKJ9c4M=
github.com/go-resty/resty/v2 v2.17.2 h1:FQW5oHYcIlkCNrMD2lloGScxcHJ0gkjshV3qcQAyHQk=
github.com/go-resty/resty/v2 v2.17.2/go.mod h1:kCKZ3wWmwJaNc7S29BRtUhJwy7iqmn+2mLtQrOyQlVA=
github.com/go-sql-driver/mysql v1.10.1 h1:arlSnNLq6a5yxGxV7qg9lF4j0C+KwD6NbQyKr9QL6ME=
@@ -604,8 +606,8 @@ github.com/google/go-cmp v0.5.6/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/
github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/google/go-replayers/grpcreplay v1.3.0 h1:1Keyy0m1sIpqstQmgz307zhiJ1pV4uIlFds5weTmxbo=
github.com/google/go-replayers/grpcreplay v1.3.0/go.mod h1:v6NgKtkijC0d3e3RW8il6Sy5sqRVUwoQa4mHOGEy8DI=
github.com/google/go-replayers/grpcreplay v1.5.1 h1:R90FMYaA6GQNhi/GG0D4zS14HEQ+wpO8bQAV0Qwyt08=
github.com/google/go-replayers/grpcreplay v1.5.1/go.mod h1:42YCe31fkUrhRo+haVAfLse+gNqGI1iO1wdNR+sWY9U=
github.com/google/go-replayers/httpreplay v1.2.0 h1:VM1wEyyjaoU53BwrOnaf9VhAyQQEEioJvFYxYcLRKzk=
github.com/google/go-replayers/httpreplay v1.2.0/go.mod h1:WahEFFZZ7a1P4VM1qEeHy+tME4bwyqPcwWbNlUI1Mcg=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
@@ -629,8 +631,8 @@ github.com/google/wire v0.7.0 h1:JxUKI6+CVBgCO2WToKy/nQk0sS+amI9z9EjVmdaocj4=
github.com/google/wire v0.7.0/go.mod h1:n6YbUQD9cPKTnHXEBN2DXlOp/mVADhVErcMFb0v3J18=
github.com/googleapis/enterprise-certificate-proxy v0.3.21 h1:OFdQ3tnCX/zaQ0Cedur3D3z7kI6HiLX9g3TiAN4/DFU=
github.com/googleapis/enterprise-certificate-proxy v0.3.21/go.mod h1:L3D/IQExI6LqEjBdXcZQ1WluSgigQmSwBboFstVPM4w=
github.com/googleapis/gax-go/v2 v2.24.0 h1:myMaPYyF9MecEmvQqMqomIwn9t/4KCZN9qnwsS76wlg=
github.com/googleapis/gax-go/v2 v2.24.0/go.mod h1:IaTHBDd7NHxSCiu0vEs8pQZu4dGZrWwuSoxCnk16OFM=
github.com/googleapis/gax-go/v2 v2.24.1 h1:AtqTN21IXMMWo99LiEVAiBfNNQmO40d8xUfZI640mc0=
github.com/googleapis/gax-go/v2 v2.24.1/go.mod h1:bWeBei0NVwaNZKb2y1HUBS7gLXIF3/Tu3pq7j8D2Tb0=
github.com/gopherjs/gopherjs v1.17.2 h1:fQnZVsXk8uxXIStYb0N4bGk7jeyTalG/wsZjQ25dO0g=
github.com/gopherjs/gopherjs v1.17.2/go.mod h1:pRRIvn/QzFLrKfvEz3qUuEhtE/zLCWfreZ6J5gM2i+k=
github.com/gorilla/mux v1.8.1 h1:TuBL49tXwgrFYWhqrNgrUNEY92u81SPhu7sTdzQEiWY=
@@ -647,8 +649,8 @@ github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 h1:UH//fgunKIs4JdUbpDl1VZCDa
github.com/grpc-ecosystem/go-grpc-middleware v1.4.0/go.mod h1:g5qyo/la0ALbONm6Vbp88Yd8NsDy6rZz+RcrMPxvld8=
github.com/grpc-ecosystem/grpc-gateway v1.16.0 h1:gmcG1KaJ57LophUzW0Hy8NmPhnMZb4M0+kPpLofRdBo=
github.com/grpc-ecosystem/grpc-gateway v1.16.0/go.mod h1:BDjrQk3hbvj6Nolgz8mAMFbcEtjT1g+wF4CSlocrBnw=
github.com/grpc-ecosystem/grpc-gateway/v2 v2.29.0 h1:5VipnvEpbqr2gA2VbM+nYVbkIF28c5ZQfqCBQ5g2xfk=
github.com/grpc-ecosystem/grpc-gateway/v2 v2.29.0/go.mod h1:Hyl3n6Twe1hvtd9XUXDec4pTvgMSEixRuQKPTMH2bNs=
github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0 h1:/Tnpcb2E0Pz/tN9s3bfEY2Q8ePCEX9iuS+cneUwncnw=
github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0/go.mod h1:zOBXOsUaBSjKgmH4OGzV1esUpR3oUSCPYVd2cUBjKYY=
github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
github.com/hashicorp/errwrap v1.1.0 h1:OxrOeh75EUXMY8TBjag2fzXGZ40LB6IKw45YeGUDY2I=
github.com/hashicorp/errwrap v1.1.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
@@ -782,8 +784,8 @@ github.com/keybase/go-keychain v0.0.1/go.mod h1:PdEILRW3i9D8JcdM+FmY6RwkHGnhHxXw
github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
github.com/kisielk/sqlstruct v0.0.0-20201105191214-5f3e10d3ab46/go.mod h1:yyMNCyc/Ib3bDTKd379tNMpB/7/H5TjM2Y9QJ5THLbE=
github.com/klauspost/compress v1.20.1 h1:T7kKElXUMXrUJ2E9QhQhxFtcK5rPyLdsGZvdbLMPdiQ=
github.com/klauspost/compress v1.20.1/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI=
github.com/klauspost/compress v1.20.3 h1:zWNoFqq/lSuYQRim3dUXlXeA0IqwuZGicdG5ZTfY5j4=
github.com/klauspost/compress v1.20.3/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI=
github.com/klauspost/cpuid/v2 v2.4.0 h1:S6Hrbc7+ywsr0r+RLapfGBHfyefhCTwEh3A0tV913Dw=
github.com/klauspost/cpuid/v2 v2.4.0/go.mod h1:19jmZ9mjzoF//ddRSUsv0zfBTJWh3QJh9FNxZTMrGxU=
github.com/klauspost/reedsolomon v1.14.2 h1:SafJYwpBBQBI6amHUygcjxZjXeN2HpiENHQDwuPWCCQ=
@@ -813,12 +815,12 @@ github.com/lanrat/extsort v1.4.2 h1:akbLIdo4PhNZtvjpaWnbXtGMmLtnGzXplkzfgl+XTTY=
github.com/lanrat/extsort v1.4.2/go.mod h1:hceP6kxKPKebjN1RVrDBXMXXECbaI41Y94tt6MDazc4=
github.com/leodido/go-urn v1.4.0 h1:WT9HwE9SGECu3lg4d/dIA+jxlljEa1/ffXKmRjqdmIQ=
github.com/leodido/go-urn v1.4.0/go.mod h1:bvxc+MVxLKB4z00jd1z+Dvzr47oO32F/QSNjSBOlFxI=
github.com/lib/pq v1.12.0 h1:mC1zeiNamwKBecjHarAr26c/+d8V5w/u4J0I/yASbJo=
github.com/lib/pq v1.12.0/go.mod h1:/p+8NSbOcwzAEI7wiMXFlgydTwcgTr3OSKMsD2BitpA=
github.com/lib/pq v1.12.3 h1:tTWxr2YLKwIvK90ZXEw8GP7UFHtcbTtty8zsI+YjrfQ=
github.com/lib/pq v1.12.3/go.mod h1:/p+8NSbOcwzAEI7wiMXFlgydTwcgTr3OSKMsD2BitpA=
github.com/linkedin/goavro/v2 v2.15.0 h1:pDj1UrjUOO62iXhgBiE7jQkpNIc5/tA5eZsgolMjgVI=
github.com/linkedin/goavro/v2 v2.15.0/go.mod h1:KXx+erlq+RPlGSPmLF7xGo6SAbh8sCQ53x064+ioxhk=
github.com/linxGnu/grocksdb v1.10.8 h1:Nau01Hhm/0kaVTR6d4viwD6npYbnDvZAfzwJCLzKRYo=
github.com/linxGnu/grocksdb v1.10.8/go.mod h1:OLQKZwiKwaJiAVCsOzWKvwiLwfZ5Vz8Md5TYR7t7pM8=
github.com/linxGnu/grocksdb v1.11.1 h1:/gjcsviJimrQCDDlQCVuvzmeVAvgapQKaFQkQSe48bQ=
github.com/linxGnu/grocksdb v1.11.1/go.mod h1:WaN+XviOp90uf+bYQ0s4y6DxXedPPMb4QwIsqMd3LdU=
github.com/lithammer/shortuuid/v3 v3.0.7 h1:trX0KTHy4Pbwo/6ia8fscyHoGA+mf1jWbPJVuvyJQQ8=
github.com/lithammer/shortuuid/v3 v3.0.7/go.mod h1:vMk8ke37EmiewwolSO1NLW8vP4ZaKlRuDIi8tWWmAts=
github.com/lpar/calendar v0.2.0 h1:A1kxv6sbvBHFUkd2XotanIRqEXQGreQOeuGhkJqIaRA=
@@ -1063,10 +1065,10 @@ github.com/rdleal/intervalst v1.5.0 h1:SEB9bCFz5IqD1yhfH1Wv8IBnY/JQxDplwkxHjT6ha
github.com/rdleal/intervalst v1.5.0/go.mod h1:xO89Z6BC+LQDH+IPQQw/OESt5UADgFD41tYMUINGpxQ=
github.com/redis/go-redis/v9 v9.22.0 h1:laDvpYXTJtZLloinw1fA5Kqd6HAEH2XKxOkG/PDq2F0=
github.com/redis/go-redis/v9 v9.22.0/go.mod h1:y2g0Wj8rQvuK0ELM+oxSudcLtC09JScs98I/X9gRWY4=
github.com/redis/rueidis v1.0.76 h1:RdDWuvlYBSp+bTrBvaXqJnNEL3VVzsnjo+0psPFgLc4=
github.com/redis/rueidis v1.0.76/go.mod h1:UsfHPSbomB6QAVMk4iiFkzRy0nh9o7scDGa+SitvBY4=
github.com/redis/rueidis/rueidiscompat v1.0.76 h1:7LikbiqCQqCsZXeZ+akgZMnjIV/J0VHih9PIX4gGZC4=
github.com/redis/rueidis/rueidiscompat v1.0.76/go.mod h1:UatQQLVj4QMIsZtpvRWY28qm6r2d72idhcS+C/RM+Zg=
github.com/redis/rueidis v1.0.77 h1:ZR41bgJcm7oRFb3aSDPrRhC0eonDSrPzjvvZvHIlNjM=
github.com/redis/rueidis v1.0.77/go.mod h1:L8mnCQJJaSNL6I4pIR6Rz732HTGS9vmuXm0yT9dRvjo=
github.com/redis/rueidis/rueidiscompat v1.0.77 h1:S1xiLQsFv8XQ46e0LelC8ihGqbDzVRocXXZLzVAeh64=
github.com/redis/rueidis/rueidiscompat v1.0.77/go.mod h1:aXCvRVUTvTHZY0owHuz9JjveO1hyuzyDWrpJh8/E3Ds=
github.com/rekby/fixenv v0.6.1 h1:jUFiSPpajT4WY2cYuc++7Y1zWrnCxnovGCIX72PZniM=
github.com/rekby/fixenv v0.6.1/go.mod h1:/b5LRc06BYJtslRtHKxsPWFT/ySpHV+rWvzTg+XWk4c=
github.com/relvacode/iso8601 v1.7.0 h1:BXy+V60stMP6cpswc+a93Mq3e65PfXCgDFfhvNNGrdo=
@@ -1104,8 +1106,8 @@ github.com/seaweedfs/go-fuse/v2 v2.9.4 h1:ACyloiuopdhRSjdLLeSWbsVaemMPskORaRF01T
github.com/seaweedfs/go-fuse/v2 v2.9.4/go.mod h1:zABdmWEa6A0bwaBeEOBUeUkGIZlxUhcdv+V1Dcc/U/I=
github.com/seaweedfs/goexif v1.0.3 h1:ve/OjI7dxPW8X9YQsv3JuVMaxEyF9Rvfd04ouL+Bz30=
github.com/seaweedfs/goexif v1.0.3/go.mod h1:Oni780Z236sXpIQzk1XoJlTwqrJ02smEin9zQeff7Fk=
github.com/seaweedfs/raft v1.2.1 h1:QgFl/aaPnagpUxYB6Bx+fFss1NyetVVcJmraMmnaQ5Q=
github.com/seaweedfs/raft v1.2.1/go.mod h1:fgs/rAVEzjQ7e04XMzG3eJhwZZRmBW+2uRtjakeCGeU=
github.com/seaweedfs/raft v1.2.2 h1:5OiCQbia7J83WE0whlRGj5oBCEfixqfoOTOeY1IQDXM=
github.com/seaweedfs/raft v1.2.2/go.mod h1:fgs/rAVEzjQ7e04XMzG3eJhwZZRmBW+2uRtjakeCGeU=
github.com/secure-systems-lab/go-securesystemslib v0.11.0 h1:iuCR9kcMFD4QurdKrGvPLoKZLv9YvwPYVr0473BdtFs=
github.com/secure-systems-lab/go-securesystemslib v0.11.0/go.mod h1:+PMOTjUGwHj2vcZ+TFKlb1tXRbrdWE1LYDT5i9JC80Q=
github.com/sergi/go-diff v1.0.0/go.mod h1:0CfEIISq7TuYL3j771MWULgwwjU+GofnZX9QAmXWZgo=
@@ -1206,8 +1208,8 @@ github.com/the42/cartconvert v0.0.0-20131203171324-aae784c392b8 h1:I4DY8wLxJXCrM
github.com/the42/cartconvert v0.0.0-20131203171324-aae784c392b8/go.mod h1:fWO/msnJVhHqN1yX6OBoxSyfj7TEj1hHiL8bJSQsK30=
github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a h1:J/YdBZ46WKpXsxsW93SG+q0F8KI+yFrcIDT4c/RNoc4=
github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a/go.mod h1:h4xBhSNtOeEosLJ4P7JyKXX7Cabg7AVkWCK5gV2vOrM=
github.com/tidwall/gjson v1.19.0 h1:xwxm7n691Uf3u5OFjzngavjGTh55KX5q/9w9xHW88JU=
github.com/tidwall/gjson v1.19.0/go.mod h1:V37/opeE/JbLUOfH0QTXiNez2l0RUjYUhpT4szFQAfc=
github.com/tidwall/gjson v1.20.0 h1:+agJ3rEzKcCXKDKo0ml26UROcpcAlnZyjq+TjbY5Fto=
github.com/tidwall/gjson v1.20.0/go.mod h1:V37/opeE/JbLUOfH0QTXiNez2l0RUjYUhpT4szFQAfc=
github.com/tidwall/match v1.2.0 h1:0pt8FlkOwjN2fPt4bIl4BoNxb98gGHN2ObFEDkrfZnM=
github.com/tidwall/match v1.2.0/go.mod h1:eRSPERbgtNPcGhD8UCthc6PmLEQXEWd3PRB5JTxsfmM=
github.com/tidwall/pretty v1.2.0 h1:RWIZEg2iJ8/g6fDDYzMpobmaoGh5OLl4AXtGUGPcqCs=
@@ -1317,8 +1319,8 @@ github.com/ydb-platform/ydb-go-genproto v0.0.0-20260810122915-65bfd5c4b705 h1:7V
github.com/ydb-platform/ydb-go-genproto v0.0.0-20260810122915-65bfd5c4b705/go.mod h1:Er+FePu1dNUieD+XTMDduGpQuCPssK5Q4BjF+IIXJ3I=
github.com/ydb-platform/ydb-go-sdk-auth-environ v0.5.3 h1:A48PcmIETWwbEMIGwndv52uME6J1eJ28MBgF7HVR74o=
github.com/ydb-platform/ydb-go-sdk-auth-environ v0.5.3/go.mod h1:oRGF+67GWLbtNzqGygsPIUsjXmsRXD1KS5m3o5ppm2k=
github.com/ydb-platform/ydb-go-sdk/v3 v3.151.1 h1:T+fB2ZDHpYIGC7DWjK+rzZHLoexBF0zG/n3Q9DGNskY=
github.com/ydb-platform/ydb-go-sdk/v3 v3.151.1/go.mod h1:dJXJ1u00IqO8Vsph8fWmYj8b1E7jyphnvJPqA69emBY=
github.com/ydb-platform/ydb-go-sdk/v3 v3.155.1 h1:ZMByYe53zZ9R1o23fxJsbNhjzKFQkzpXm6uGj8gWHXQ=
github.com/ydb-platform/ydb-go-sdk/v3 v3.155.1/go.mod h1:dJXJ1u00IqO8Vsph8fWmYj8b1E7jyphnvJPqA69emBY=
github.com/ydb-platform/ydb-go-yc v0.12.5 h1:qYIeU1N6a92FaDA0+/vR55l5IzRF50MeLZwUWwvy2xQ=
github.com/ydb-platform/ydb-go-yc v0.12.5/go.mod h1:DlSltVQKEdXJotaAYwoHzMGDF9EJw5S3hg3LXe/5yYw=
github.com/ydb-platform/ydb-go-yc-metadata v0.6.2 h1:CZFawwHjLcjsxcl/g0dGIXbI2r41okPGLCydF4qTHcc=
@@ -1361,28 +1363,28 @@ go.opencensus.io v0.24.0 h1:y73uSU6J157QMP2kn2r30vwW1A2W2WFwSCGnAVxeaD0=
go.opencensus.io v0.24.0/go.mod h1:vNK8G9p7aAivkbmorf4v+7Hgx+Zs0yY+0fOtgBfjQKo=
go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
go.opentelemetry.io/contrib/detectors/gcp v1.45.0 h1:9jR0ZPRok9ryaOQ2Wx8rg5F7Aon59mxrqbVI60/vlBk=
go.opentelemetry.io/contrib/detectors/gcp v1.45.0/go.mod h1:VSme3o2fvSg5bVg0dRzyHaj4Z5EVhG+g2Fde6LKzmQA=
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.69.0 h1:2yEATaop1/a1I4psnSLgWVPLWwCzkqWakgJy7xTDVy0=
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.69.0/go.mod h1:D7J12YRapIekYyPWgGPlA/23pRmpSEZC5xJC/TTLI9U=
go.opentelemetry.io/contrib/detectors/gcp v1.46.0 h1:PI8dGkqDaQkwJ8kOopqMhDTbrnK3UIeG/RCHH4HErbo=
go.opentelemetry.io/contrib/detectors/gcp v1.46.0/go.mod h1:nsrN5c/sOLoY2vsPxN/rQ0V0nvGrWJCqcW4UXLtqNG8=
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.71.0 h1:B2h3uqicet1CT2N5TOFhS+Gq++9i0/CLmaxvhmhtP5s=
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.71.0/go.mod h1:dylvB+ZiiwMvsDij9O84Uy7SijLgHMX4mbkncds+4Sw=
go.opentelemetry.io/contrib/instrumentation/net/http/httptrace/otelhttptrace v0.69.0 h1:MCcYL7J6Vt/X0kjqbMZkekCmwsurbQRbL69vkiye2lk=
go.opentelemetry.io/contrib/instrumentation/net/http/httptrace/otelhttptrace v0.69.0/go.mod h1:3jnStNwSufK+f5ktjL4EPcwtig4rtd81NS70lqHuXl8=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.70.0 h1:LMuyCAyfalSjDyjdC65nK6N0zoTT63+E/u95X0JovZI=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.70.0/go.mod h1:085m8qbm4hgc8rZWGDEa4vmyyo2c3nPxUslYUKUIU04=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.71.0 h1:3g7B90UzBltIDKq1/5mrTGxTnOFDV0ICOhLoxiZ8jlg=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.71.0/go.mod h1:Ef8SuTh59BT7+ofpDxN9z+yOlc4t2GjLmKDgYNJL/NU=
go.opentelemetry.io/otel v1.46.0 h1:FHt5/CDyVxi/8IM1CH7VE/rRgq3kLHa2mSTVMO8AWyc=
go.opentelemetry.io/otel v1.46.0/go.mod h1:Gj3SEScelsNC45tp4nSxRYlS+f5iez7W8XPMCt905kE=
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.44.0 h1:SUplec5dp06reu1zaXmOXdvqH398taqrDXqUl99jxSc=
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.44.0/go.mod h1:ho2g4N+ane+swq5I/VBkKWnRDY4kUINH3FuqyZqX/Ug=
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.46.0 h1:qkDYCAFiZXLcs1L4aY+tP2wguQ4kURANqHOQMA2et2s=
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.46.0/go.mod h1:tkipS4DRzmpAmvg+Gw4++O1IdDq6TVDnvnYU6cmbQVs=
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v1.44.0 h1:RuynHbfU8JUEw7DyONgkVYg2SVtsoF28y0LGIr69jgA=
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v1.44.0/go.mod h1:qZF+/lBs71APw8mlnEZcqZHMzqrYrsFiJOv83lX1OGo=
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.45.0 h1:QRefszxJmfPdjXUUm3j6iDzY03mTPXMjqErFqQ67vUg=
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.45.0/go.mod h1:Tiz03lTBVBrm7eWZBOidzEaYaJa8tjwGUGv6d8mlTyk=
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.45.0 h1:fG5MCxGz8+2VtrN/WgqSpJFctVz24gpxj8CxkKmc8Ww=
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.45.0/go.mod h1:BmAYTn+3ysbRe+IU2msxmf5Rx3g6DHvex+tWI3LdhYI=
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.46.0 h1:OFnwLJr+pF3iHrlGSzbxyuo6/6HyBlnlN1CWEJmBVcw=
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.46.0/go.mod h1:716wFneO0ov19A2beH5hjfh9AK5z/VWNAtDijp1Y0/g=
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.46.0 h1:w53CDeOA/Kurp7yRsegSr6pbbr759dOvJ+yNmWM6Hxs=
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.46.0/go.mod h1:BOmGMCbAtvcJiSJ+hLuhgPLdDbimnraSl8irz3iY8sY=
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.45.0 h1:QBajQ2SrwQijzHyZbQlPsuIzpl/ll8DY6wPWsajeGcI=
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.45.0/go.mod h1:08ZQLjrPLQ6R4kAXvuOvODEer5Yh4CoFvll5qB2BCI8=
go.opentelemetry.io/otel/exporters/stdout/stdoutmetric v1.44.0 h1:hqxVTu/GtBF+vJ8d1fzW7fRxZFvgoDjWcxwwCaFDYpU=
go.opentelemetry.io/otel/exporters/stdout/stdoutmetric v1.44.0/go.mod h1:z5fVEF4X5v0ESvlJqBrrFlBVoj5EQuefZpzsu7R+x5Q=
go.opentelemetry.io/otel/exporters/stdout/stdoutmetric v1.46.0 h1:PR9eAf7o0dQs3hshZNZpE9aW2dXWX/KdDf6pJilVD3U=
go.opentelemetry.io/otel/exporters/stdout/stdoutmetric v1.46.0/go.mod h1:2Z4KyNdH1uuzivdinyfGsxzNNT/Rl45pwtVwfYVI0xk=
go.opentelemetry.io/otel/exporters/zipkin v1.45.0 h1:KN3btaILMTxR4QDHVGAO87lq5ButzK7l+kIfLuxQ1oA=
go.opentelemetry.io/otel/exporters/zipkin v1.45.0/go.mod h1:yNcodmUclM4InyWoOwX/YW4Jri0Gj5FWAlM+NqCrtqY=
go.opentelemetry.io/otel/metric v1.46.0 h1:yBnkXvgV7AXFILZc5K6IZe/CBFF3OS7BJ8ov6/lj0K8=
@@ -1415,20 +1417,20 @@ go.uber.org/multierr v1.11.0 h1:blXXJkSxSSfBVBlC76pxqeO+LN3aDfLQo+309xJstO0=
go.uber.org/multierr v1.11.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y=
go.uber.org/zap v1.18.1/go.mod h1:xg/QME4nWcxGxrpdeYfq7UvYrLh66cuVKdrbD1XF/NI=
go.uber.org/zap v1.19.0/go.mod h1:xg/QME4nWcxGxrpdeYfq7UvYrLh66cuVKdrbD1XF/NI=
go.uber.org/zap v1.27.1 h1:08RqriUEv8+ArZRYSTXy1LeBScaMpVSTBhCeaZYfMYc=
go.uber.org/zap v1.27.1/go.mod h1:GB2qFLM7cTU87MWRP2mPIjqfIDnGu+VIO4V/SdhGo2E=
go.uber.org/zap v1.28.0 h1:IZzaP1Fv73/T/pBMLk4VutPl36uNC+OSUh3JLG3FIjo=
go.uber.org/zap v1.28.0/go.mod h1:rDLpOi171uODNm/mxFcuYWxDsqWSAVkFdX4XojSKg/Q=
go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ=
go.yaml.in/yaml/v2 v2.4.4/go.mod h1:gMZqIpDtDqOfM0uNfy0SkpRhvUryYH0Z6wdMYcacYXQ=
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
go.yaml.in/yaml/v4 v4.0.0-rc.6 h1:1h7H1ohdUh93/FyE4YaDa1Zh64K6VVbjF4K6WUxMtH4=
go.yaml.in/yaml/v4 v4.0.0-rc.6/go.mod h1:aZqd9kCMsGL7AuUv/m/PvWLdg5sjJsZ4oHDEnfPPfY0=
gocloud.dev v0.46.0 h1:niIuZwSjMtBx8K+ITB2s5kZullB13PGOS2ZoQPZxQ4Q=
gocloud.dev v0.46.0/go.mod h1:ACQe+2qO+hEO+pdcvvsM+RB63r8TyGD1W3ESCLFyzvM=
gocloud.dev v0.47.0 h1:qN4/0vtutn97f1X5ziisf3WXuaFL9AR0FIRnyJRYzgA=
gocloud.dev v0.47.0/go.mod h1:OdMnyXQRhM17U0a0DwU57pe0c01qN3lJNmaeeRYW0D0=
gocloud.dev/pubsub/natspubsub v0.46.0 h1:cTmJnWOmhb3dYRzX7PvNkq2m5mQdBLFFxZbl/21pfjk=
gocloud.dev/pubsub/natspubsub v0.46.0/go.mod h1:hbEEsNt6mGQoonGQi39VOjHfx2f13DOmWouh4hGsfrU=
gocloud.dev/pubsub/rabbitpubsub v0.46.0 h1:uWJ8z/F8xYxoTlBafTweoKvX2V85cwtN+/+nyRRtjZ0=
gocloud.dev/pubsub/rabbitpubsub v0.46.0/go.mod h1:Ef5XHxv17zV9B3U9xoYAqJ8Rutslxt5Emb5e1cScuXc=
gocloud.dev/pubsub/rabbitpubsub v0.47.0 h1:iDSspbLlDbPL2avESbDfMNBK4r3B6H04Ejx+/hqh9JE=
gocloud.dev/pubsub/rabbitpubsub v0.47.0/go.mod h1:2tSZnGnGLPw2ayW4pVaUmnA1LCjP7rekpXPkaBUZt0o=
golang.org/x/arch v0.20.0 h1:dx1zTU0MAE98U+TQ8BLl7XsJbgze2WnNKF/8tGp/Q6c=
golang.org/x/arch v0.20.0/go.mod h1:bdwinDaKcfZUGpH09BB7ZmOfhalA8lQdzl62l8gGWsk=
golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
@@ -1447,8 +1449,8 @@ golang.org/x/crypto v0.14.0/go.mod h1:MVFd36DqK4CsrnJYDkBA3VC4m2GkXAM0PvzMCn4JQf
golang.org/x/crypto v0.58.0 h1:COkYLr4k7nDI4r6QtMML2AfzeIswWHAJthLS6K/M/54=
golang.org/x/crypto v0.58.0/go.mod h1:Xh+kl5A+M0gMiWDytH2Hhr5Z7WbNCpv/eB/EFgfSk+w=
golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20260709172345-9ea1abe57597 h1:qLvzZeaANDgyVOA8pyHCOStGlXn0rseXma+GQjeuv2g=
golang.org/x/exp v0.0.0-20260709172345-9ea1abe57597/go.mod h1:EdfpwwqSu+0Li0mzskwHU6FWDV3t9Q+RZDo3QMUtL3Q=
golang.org/x/exp v0.0.0-20260813180055-c1d0aacb2297 h1:YXnL44eJ77R+ji4/ooy8UsXIhz+lbi2Qgdlc8iRN0gY=
golang.org/x/exp v0.0.0-20260813180055-c1d0aacb2297/go.mod h1:Mkmymgv+uMpSQ/XxJ/7GpdrdYoqm3u72jEbpCLiJmNk=
golang.org/x/image v0.46.0 h1:b1+oYj0Jbp6K5MDT4i4/eZpYlk3V8SJhhDKh6LBHAyQ=
golang.org/x/image v0.46.0/go.mod h1:3B3W05VGVQyuXucLINLjXKrqISASfi4Xj+iCVkLMwew=
golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
@@ -1629,12 +1631,12 @@ google.golang.org/genproto v0.0.0-20200423170343-7949de9c1215/go.mod h1:55QSHmfG
google.golang.org/genproto v0.0.0-20200513103714-09dca8ec2884/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo=
google.golang.org/genproto v0.0.0-20210903162649-d08c68adba83/go.mod h1:eFjDcFEctNawg4eG61bRv87N7iHBWyVhJu7u1kqDUXY=
google.golang.org/genproto v0.0.0-20260715232425-e75dac1f907d h1:C9v1o0/4quuhOAfmRXA2j+we0PqZIp8traLdeogF3Ms=
google.golang.org/genproto v0.0.0-20260715232425-e75dac1f907d/go.mod h1:Wz2wFJntZFmLGo7pLDXZ3wYk5hyc0Mb+SkHhDDXT+lU=
google.golang.org/genproto/googleapis/api v0.0.0-20260817212433-ac3dfec99bb1 h1:lrupDmKL3p5kEX1M92oan027eCKcouzjuPbH6YBK+Rs=
google.golang.org/genproto/googleapis/api v0.0.0-20260817212433-ac3dfec99bb1/go.mod h1:q/3oV3jAi5vwelxsVAprMBC8BcM2zmNe+IjRGd+9/ks=
google.golang.org/genproto/googleapis/rpc v0.0.0-20260819154853-08b0e4226688 h1:cYNAzI2sUwhmCcoj9TxvihSrqsxt6uIkj3rDRhSDmW4=
google.golang.org/genproto/googleapis/rpc v0.0.0-20260819154853-08b0e4226688/go.mod h1:DjtHYE8FKJLivXcBEjGwndXfIC23G0VpXiXKqG179uA=
google.golang.org/genproto v0.0.0-20260904194346-d0f1323225a4 h1:6QRNo34FxJcE1LG6XfnmaQ17m6w21oXaKlKoBaF+UqE=
google.golang.org/genproto v0.0.0-20260904194346-d0f1323225a4/go.mod h1:eifJ91nefXnfP/bMkZsACVGm14ZtEL4eUXRLTryfbY4=
google.golang.org/genproto/googleapis/api v0.0.0-20260904194346-d0f1323225a4 h1:NCe/UiklGd/9xjT+ROBVhJ1kf6TRQaFedsR+z7u1gvo=
google.golang.org/genproto/googleapis/api v0.0.0-20260904194346-d0f1323225a4/go.mod h1:fJ2lYaWjqNknJyQBOCd0fA3HnEElJqGplH71a2txi+g=
google.golang.org/genproto/googleapis/rpc v0.0.0-20260904194346-d0f1323225a4 h1:5t+ZydAFj5kGVLrgCvLmpmCf9ylGRd64hpEronfRaws=
google.golang.org/genproto/googleapis/rpc v0.0.0-20260904194346-d0f1323225a4/go.mod h1:DjtHYE8FKJLivXcBEjGwndXfIC23G0VpXiXKqG179uA=
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg=
google.golang.org/grpc v1.25.1/go.mod h1:c3i+UQWmh7LiEpx4sFZnkU36qjEYZ0imhYfXVyQciAY=
@@ -1705,31 +1707,31 @@ honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWh
honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
modernc.org/b v1.0.0 h1:vpvqeyp17ddcQWF29Czawql4lDdABCDRbXRAS4+aF2o=
modernc.org/b v1.0.0/go.mod h1:uZWcZfRj1BpYzfN9JTerzlNUnnPsV9O2ZA8JsRcubNg=
modernc.org/cc/v4 v4.29.1 h1:MKgdCV3WykTSPqpVrnxdEDS0HEd2FHpKZDzxzU5LyeI=
modernc.org/cc/v4 v4.29.1/go.mod h1:OnovgIhbbMXMu1aISnJ0wvVD1KnW+cAUJkIrAWh+kVI=
modernc.org/ccgo/v4 v4.34.6 h1:sBgfIwyN0TQ9C5hwIeuqyeAKyMWnbvj2fvpF4L11uzU=
modernc.org/ccgo/v4 v4.34.6/go.mod h1:SZ8YcN9NG7XVsQYdm6jYBvi8PQP1qi+kqB6OhjqI3Fk=
modernc.org/cc/v4 v4.29.7 h1:q+NXGJ0bK3b4TXFYQQVr9pYETGnmwFWkrUzJnMya/Tg=
modernc.org/cc/v4 v4.29.7/go.mod h1:OnovgIhbbMXMu1aISnJ0wvVD1KnW+cAUJkIrAWh+kVI=
modernc.org/ccgo/v4 v4.36.1 h1:ZNIUZAryN0UgnJwtyxrdEzcFc3yD4Cu4AzjfPXsLsIE=
modernc.org/ccgo/v4 v4.36.1/go.mod h1:rrtGc2QkS239nYb/mQNuBMyjq3/y3ZXWbBjPoV3wqzA=
modernc.org/fileutil v1.4.0 h1:j6ZzNTftVS054gi281TyLjHPp6CPHr2KCxEXjEbD6SM=
modernc.org/fileutil v1.4.0/go.mod h1:EqdKFDxiByqxLk8ozOxObDSfcVOv/54xDs/DUHdvCUU=
modernc.org/gc/v2 v2.6.5 h1:nyqdV8q46KvTpZlsw66kWqwXRHdjIlJOhG6kxiV/9xI=
modernc.org/gc/v2 v2.6.5/go.mod h1:YgIahr1ypgfe7chRuJi2gD7DBQiKSLMPgBQe9oIiito=
modernc.org/gc/v3 v3.1.4 h1:2g65LGVSmFQrXeITAw97x7hCRvZFcyE1uDP+7Vng7JI=
modernc.org/gc/v3 v3.1.4/go.mod h1:HFK/6AGESC7Ex+EZJhJ2Gni6cTaYpSMmU/cT9RmlfYY=
modernc.org/gc/v3 v3.1.5 h1:21ldfPfRYE31Tb7B3mwAK8gy1AxP4+dKjrOQPfqakoc=
modernc.org/gc/v3 v3.1.5/go.mod h1:HFK/6AGESC7Ex+EZJhJ2Gni6cTaYpSMmU/cT9RmlfYY=
modernc.org/goabi0 v0.2.0 h1:HvEowk7LxcPd0eq6mVOAEMai46V+i7Jrj13t4AzuNks=
modernc.org/goabi0 v0.2.0/go.mod h1:CEFRnnJhKvWT1c1JTI3Avm+tgOWbkOu5oPA8eH8LnMI=
modernc.org/libc v1.74.4 h1:fX1Omw4o2/1C2iRkkIsrQTasJQldLhRmuPreXLoWs9k=
modernc.org/libc v1.74.4/go.mod h1:eeQAS9W3sZeKYMFubydxJpII9ybHWshk+7or7bLG9co=
modernc.org/libc v1.77.1 h1:Ct8j47QtiZ1Enj2DtFXQtUqrPCAjdCmPjtCuvrYQ0Hs=
modernc.org/libc v1.77.1/go.mod h1:87/pZ4L6nD1zqW4nItuS12YO7hN1igAah34xjnQo/W0=
modernc.org/mathutil v1.1.1/go.mod h1:mZW8CKdRPY1v87qxC/wUdX5O1qDzXMP5TH3wjfpga6E=
modernc.org/mathutil v1.7.1 h1:GCZVGXdaN8gTqB1Mf/usp1Y/hSqgI2vAGGP4jZMCxOU=
modernc.org/mathutil v1.7.1/go.mod h1:4p5IwJITfppl0G4sUEDtCr4DthTaT47/N3aT6MhfgJg=
modernc.org/memory v1.11.0 h1:o4QC8aMQzmcwCK3t3Ux/ZHmwFPzE6hf2Y5LbkRs+hbI=
modernc.org/memory v1.11.0/go.mod h1:/JP4VbVC+K5sU2wZi9bHoq2MAkCnrt2r98UGeSK7Mjw=
modernc.org/memory v1.12.1 h1:nFMiWrpStgZczNl6XI9GnIk/rWhYIyHGUaR04pGbp9g=
modernc.org/memory v1.12.1/go.mod h1:/JP4VbVC+K5sU2wZi9bHoq2MAkCnrt2r98UGeSK7Mjw=
modernc.org/opt v0.2.0 h1:tGyef5ApycA7FSEOMraay9SaTk5zmbx7Tu+cJs4QKZg=
modernc.org/opt v0.2.0/go.mod h1:03fq9lsNfvkYSfxrfUhZCWPk1lm4cq4N+Bh//bEtgns=
modernc.org/sortutil v1.2.1 h1:+xyoGf15mM3NMlPDnFqrteY07klSFxLElE2PVuWIJ7w=
modernc.org/sortutil v1.2.1/go.mod h1:7ZI3a3REbai7gzCLcotuw9AC4VZVpYMjDzETGsSMqJE=
modernc.org/sqlite v1.57.0 h1:qNQP6xnx5M0ISNtlnxoOX0+cD5bJ0/gr9aMmndFczzg=
modernc.org/sqlite v1.57.0/go.mod h1:yCJ2cmAaIkHQ25oXWrF8H4O1lIfPYPR26yCEDj2P3pQ=
modernc.org/sqlite v1.60.1 h1:/blz53O951KWFOso4QQvEs/Fq6cDBKLtMVrYNSeJVKw=
modernc.org/sqlite v1.60.1/go.mod h1:1dIoEagfDE72QytD5scH1lxARtaUgKgHC/NuApA27r0=
modernc.org/strutil v1.1.0/go.mod h1:lstksw84oURvj9y3tn8lGvRxyRC1S2+g5uuIzNfIOBs=
modernc.org/strutil v1.2.1 h1:UneZBkQA+DX2Rp35KcM69cSsNES9ly8mQWD71HKlOA0=
modernc.org/strutil v1.2.1/go.mod h1:EHkiggD70koQxjVdSBM3JKM7k6L0FbGE5eymy9i3B9A=
+1 -1
View File
@@ -350,7 +350,7 @@ main() {
info " weed volume -mserver=localhost:9333 # Start Go volume server"
info " weed-volume -mserver localhost:9333 # Start Rust volume server"
if [ "$WORKER_INSTALLED" = true ]; then
info " weed-worker --admin localhost:23646 # Start the Rust maintenance worker"
info " weed-worker --admin localhost:12535 # Start the Rust maintenance worker"
fi
}
+11 -8
View File
@@ -345,8 +345,9 @@ To enable the admin interface, add the following to your values.yaml:
```yaml
admin:
enabled: true
port: 23646
grpcPort: 33646 # For worker connections
port: 12535
grpcPort: 22535 # For worker connections
legacyPort: 23646 # Compat: also serve http + gRPC (+10000) for pre-12535 clients
secret:
adminUser: "admin"
adminPassword: "your-secure-password" # Leave empty to disable auth
@@ -364,7 +365,9 @@ admin:
className: "nginx"
```
The admin interface will be available at `http://<admin-service>:23646` (or via ingress). Workers connect to the admin server via gRPC on port `33646`.
The admin interface will be available at `http://<admin-service>:12535` (or via ingress). Workers connect to the admin server via gRPC on port `22535`.
For backward compatibility with older clients and workers, admin also serves the previous default ports: HTTP on `23646` and gRPC on `33646` (legacyPort + 10000). The Service and NetworkPolicy expose both. Set `admin.legacyPort: 0` to disable.
### Admin Authentication
@@ -436,7 +439,7 @@ worker:
# Auto-discovery only works if the admin is in the same namespace and same Helm release.
# For cross-namespace or separate release scenarios, explicitly set this value.
# Example: If main SeaweedFS is deployed in "production" namespace:
# adminServer: "seaweedfs-admin.production.svc:33646"
# adminServer: "seaweedfs-admin.production.svc:22535"
adminServer: ""
# Workers need storage for task execution
@@ -502,8 +505,8 @@ worker:
# REQUIRED: Point to the admin service of your main SeaweedFS release
# Replace <namespace> with the namespace where your main seaweedfs is deployed
# Example: If deploying in namespace "production":
# adminServer: "seaweedfs-admin.production.svc:33646"
adminServer: "seaweedfs-admin.<namespace>.svc:33646"
# adminServer: "seaweedfs-admin.production.svc:22535"
adminServer: "seaweedfs-admin.<namespace>.svc:22535"
```
**values-worker-balance.yaml** (for balance operations):
@@ -528,8 +531,8 @@ worker:
# REQUIRED: Point to the admin service of your main SeaweedFS release
# Replace <namespace> with the namespace where your main seaweedfs is deployed
# Example: If deploying in namespace "production":
# adminServer: "seaweedfs-admin.production.svc:33646"
adminServer: "seaweedfs-admin.<namespace>.svc:33646"
# adminServer: "seaweedfs-admin.production.svc:22535"
adminServer: "seaweedfs-admin.<namespace>.svc:22535"
```
Deploy the specialized workers as separate releases:
@@ -37,6 +37,16 @@ spec:
nodePort: {{ $nodePorts.grpc }}
{{- end }}
protocol: TCP
{{- if gt (int .Values.admin.legacyPort) 0 }}
- name: "legacy-http"
port: {{ .Values.admin.legacyPort }}
targetPort: {{ .Values.admin.legacyPort }}
protocol: TCP
- name: "legacy-grpc"
port: {{ add (int .Values.admin.legacyPort) 10000 }}
targetPort: {{ add (int .Values.admin.legacyPort) 10000 }}
protocol: TCP
{{- end }}
selector:
app.kubernetes.io/name: {{ template "seaweedfs.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
@@ -170,6 +170,9 @@ spec:
-ip={{ .Values.admin.ip | default "0.0.0.0" }} \
-port={{ .Values.admin.port }} \
-port.grpc={{ .Values.admin.grpcPort }} \
{{- if ne (int .Values.admin.legacyPort) 23646 }}
-port.legacy={{ .Values.admin.legacyPort }} \
{{- end }}
{{- if or (eq .Values.admin.data.type "hostPath") (eq .Values.admin.data.type "persistentVolumeClaim") (eq .Values.admin.data.type "emptyDir") (eq .Values.admin.data.type "existingClaim") }}
-dataDir=/data \
{{- else if .Values.admin.dataDir }}
@@ -233,6 +236,12 @@ spec:
name: http
- containerPort: {{ .Values.admin.grpcPort }}
name: grpc
{{- if gt (int .Values.admin.legacyPort) 0 }}
- containerPort: {{ .Values.admin.legacyPort }}
name: legacy-http
- containerPort: {{ add (int .Values.admin.legacyPort) 10000 }}
name: legacy-grpc
{{- end }}
{{- if .Values.admin.readinessProbe.enabled }}
readinessProbe:
httpGet:
@@ -43,7 +43,7 @@ spec:
{{- $configSecret := (lookup "v1" "Secret" .Release.Namespace $existingS3ConfigSecret) | default dict }}
{{- $_ := set $podAnnotations "checksum/s3config" ($configSecret | toYaml | sha256sum) }}
{{- else }}
{{- $_ := set $podAnnotations "checksum/s3config" (include (print .Template.BasePath "/s3/s3-secret.yaml") . | sha256sum) }}
{{- $_ := set $podAnnotations "checksum/s3config" (include "seaweedfs.s3.configChecksum" .) }}
{{- end }}
{{- $_ := set $podAnnotations "checksum/master-config" (include (print .Template.BasePath "/master/master-configmap.yaml") . | sha256sum) }}
annotations:
@@ -48,7 +48,7 @@ spec:
{{- $configSecret := (lookup "v1" "Secret" .Release.Namespace .Values.filer.s3.existingConfigSecret) | default dict }}
{{- $_ := set $podAnnotations "checksum/s3config" ($configSecret | toYaml | sha256sum) }}
{{- else }}
{{- $_ := set $podAnnotations "checksum/s3config" (include (print .Template.BasePath "/s3/s3-secret.yaml") . | sha256sum) }}
{{- $_ := set $podAnnotations "checksum/s3config" (include "seaweedfs.s3.configChecksum" .) }}
{{- end }}
annotations:
{{- toYaml $podAnnotations | nindent 8 }}
@@ -44,7 +44,7 @@ spec:
{{- $configSecret := (lookup "v1" "Secret" .Release.Namespace .Values.s3.existingConfigSecret) | default dict }}
{{- $_ := set $podAnnotations "checksum/s3config" ($configSecret | toYaml | sha256sum) }}
{{- else }}
{{- $_ := set $podAnnotations "checksum/s3config" (include (print .Template.BasePath "/s3/s3-secret.yaml") . | sha256sum) }}
{{- $_ := set $podAnnotations "checksum/s3config" (include "seaweedfs.s3.configChecksum" .) }}
{{- end }}
annotations:
{{- toYaml $podAnnotations | nindent 8 }}
@@ -692,3 +692,28 @@ Usage: {{ include "seaweedfs.service.loadBalancerFields" .Values.s3.service }}
{{- end }}
{{- end }}
{{- end -}}
{{/*
Checksum that restarts the S3-serving pods when the generated S3 config changes. It hashes the
values s3-secret.yaml is built from, not the rendered Secret: that template keeps its generated keys
with lookup, so rendering it again for a checksum draws new random keys whenever lookup finds
nothing (on install), and the first upgrade then changed the checksum and restarted the pods
although nothing had changed. Only the values that template reads are hashed, so scaling the S3
gateway or changing its resources does not restart the filer. Keep this list in step with
s3-secret.yaml.
Keys the chart generated are not part of it. Deleting the Secret to rotate them regenerates them on
the next upgrade without restarting the pods; restart them afterwards (kubectl rollout restart).
Folding the stored keys in would bring the first-upgrade restart back, since lookup finds nothing
on install.
*/}}
{{- define "seaweedfs.s3.configChecksum" -}}
{{- dict
"secretName" (include "seaweedfs.fullname" .)
"enabled" (list .Values.s3.enabled .Values.filer.s3.enabled .Values.allInOne.enabled .Values.allInOne.s3.enabled)
"enableAuth" (list .Values.s3.enableAuth .Values.filer.s3.enableAuth .Values.allInOne.s3.enableAuth)
"existingConfigSecret" (list .Values.s3.existingConfigSecret .Values.filer.s3.existingConfigSecret .Values.allInOne.s3.existingConfigSecret)
"credentials" .Values.s3.credentials
"legacySecret" (list .Values.s3.reuseLegacySecret .Values.s3.legacySecretName)
"auditLogConfig" (list .Values.s3.auditLogConfig .Values.filer.s3.auditLogConfig)
| toJson | sha256sum -}}
{{- end -}}
@@ -93,7 +93,11 @@
ClusterRole under global.seaweedfs.createClusterRole is a leftover from a
migration and is not exercised by any component. */}}
{{- if .Values.admin.enabled }}
{{- $targets = append $targets (dict "component" "admin" "ports" (list .Values.admin.port .Values.admin.grpcPort)) }}
{{- $ports := list .Values.admin.port .Values.admin.grpcPort }}
{{- if gt (int .Values.admin.legacyPort) 0 }}
{{- $ports = concat $ports (list .Values.admin.legacyPort (add (int .Values.admin.legacyPort) 10000)) }}
{{- end }}
{{- $targets = append $targets (dict "component" "admin" "ports" $ports) }}
{{- end }}
{{- if .Values.worker.enabled }}
{{- $ports := list }}
@@ -18,9 +18,17 @@
{{- $clusterFilerKey := printf "WEED_CLUSTER_%s_FILER" $clusterUpper }}
{{- $podSecurityContext := .Values.filer.podSecurityContext }}
{{- $containerSecurityContext := .Values.filer.containerSecurityContext }}
{{- $nodeSelector := .Values.filer.nodeSelector }}
{{- $tolerations := .Values.filer.tolerations }}
{{- $createBucketsHook := .Values.filer.s3.createBucketsHook }}
{{- if .Values.allInOne.enabled }}
{{- $podSecurityContext = .Values.allInOne.podSecurityContext }}
{{- $containerSecurityContext = .Values.allInOne.containerSecurityContext }}
{{- $nodeSelector = .Values.allInOne.nodeSelector }}
{{- $tolerations = .Values.allInOne.tolerations }}
{{- $createBucketsHook = .Values.allInOne.s3.createBucketsHook }}
{{- else if .Values.s3.createBuckets }}
{{- $createBucketsHook = .Values.s3.createBucketsHook }}
{{- end }}
{{- /* Check allInOne mode first */}}
@@ -76,10 +84,18 @@ spec:
app.kubernetes.io/instance: {{ .Release.Name | quote }}
app.kubernetes.io/component: bucket-hook
spec:
{{- with $createBucketsHook.affinity }}
affinity:
{{ tpl . $ | indent 8 | trim }}
{{- end }}
restartPolicy: Never
{{- if $podSecurityContext.enabled }}
securityContext: {{- omit $podSecurityContext "enabled" | toYaml | nindent 8 }}
{{- end }}
{{- with coalesce $createBucketsHook.tolerations $tolerations }}
tolerations:
{{ tpl . $ | indent 8 | trim }}
{{- end }}
{{- include "seaweedfs.imagePullSecrets" $ | nindent 6 }}
containers:
- name: post-install-job
@@ -274,4 +290,8 @@ spec:
secretName: {{ include "seaweedfs.fullname" . }}-client-cert
{{- end }}
{{- end }}
{{- with coalesce $createBucketsHook.nodeSelector $nodeSelector }}
nodeSelector:
{{ tpl . $ | indent 8 | trim }}
{{- end }}
{{- end }}
@@ -30,11 +30,19 @@ spec:
app.kubernetes.io/instance: {{ .Release.Name | quote }}
app.kubernetes.io/component: volume-resize-hook
spec:
{{- with .Values.volume.resizeHook.affinity }}
affinity:
{{ tpl (printf "{{ $volumeName := \"volume\" }}%s" .) $ | indent 8 | trim }}
{{- end }}
serviceAccountName: {{ $seaweedfsName }}-volume-resize-hook
restartPolicy: Never
{{- if .Values.volume.podSecurityContext.enabled }}
securityContext: {{- omit .Values.volume.podSecurityContext "enabled" | toYaml | nindent 8 }}
{{- end }}
{{- with coalesce .Values.volume.resizeHook.tolerations .Values.volume.tolerations }}
tolerations:
{{ tpl (printf "{{ $volumeName := \"volume\" }}%s" .) $ | indent 8 | trim }}
{{- end }}
containers:
- name: resize
image: {{ .Values.volume.resizeHook.image }}
@@ -58,6 +66,10 @@ spec:
volumes:
{{- include "seaweedfs.tmpDirVolume" (list . .Values.volume.containerSecurityContext "" "" false) | nindent 8 }}
{{- end }}
{{- with coalesce .Values.volume.resizeHook.nodeSelector .Values.volume.nodeSelector }}
nodeSelector:
{{ tpl (printf "{{ $volumeName := \"volume\" }}%s" .) $ | indent 8 | trim }}
{{- end }}
---
apiVersion: v1
kind: ServiceAccount
+24 -2
View File
@@ -437,6 +437,10 @@ volume:
resizeHook:
enabled: true
image: alpine/k8s:1.28.4
# nodeSelector and tolerations fall back to the volume server's when unset
nodeSelector: ""
tolerations: ""
affinity: ""
# idx (the volume index) may use a separate PERSISTENT volume:
#
@@ -1083,6 +1087,10 @@ filer:
# anonymousRead: false
createBucketsHook:
resources: {}
# nodeSelector and tolerations fall back to filer's when unset
nodeSelector: ""
tolerations: ""
affinity: ""
s3:
enabled: false
@@ -1305,6 +1313,10 @@ s3:
createBucketsHook:
resources: {}
# nodeSelector and tolerations fall back to filer's when unset
nodeSelector: ""
tolerations: ""
affinity: ""
ingress:
enabled: false
@@ -1499,8 +1511,14 @@ admin:
imageOverride: null
restartPolicy: null
replicas: 1
port: 23646 # Default admin port
grpcPort: 33646 # Default gRPC port for worker connections
port: 12535 # Default admin port
grpcPort: 22535 # Default gRPC port for worker connections
# Legacy compat port: admin also serves HTTP and worker gRPC (legacyPort +
# 10000) on this port so workers and clients from before the 23646 -> 12535
# default change keep connecting during migration. 0 disables. The flag is
# only passed to the binary when the value differs from its 23646 default,
# so the chart keeps installing on releases that predate -port.legacy.
legacyPort: 23646
# IP address the admin server listens on. Since `weed admin` 4.46 defaults to
# loopback (127.0.0.1), the chart must bind a non-loopback address for the
# kubelet's httpGet readiness/liveness probes (which dial the pod IP) to ever
@@ -1927,6 +1945,10 @@ allInOne:
# anonymousRead: false
createBucketsHook:
resources: {}
# nodeSelector and tolerations fall back to allInOne's when unset
nodeSelector: ""
tolerations: ""
affinity: ""
# SFTP server configuration
# Note: Most parameters below default to null, which means they inherit from
+1099 -1059
View File
File diff suppressed because it is too large. Load diff

Before

Width:  |  Height:  |  Size: 54 KiB

After

Width:  |  Height:  |  Size: 55 KiB

@@ -527,6 +527,10 @@ message AtomicRenameEntryRequest {
string new_directory = 3;
string new_name = 4;
repeated int32 signatures = 5;
// When true, the rename fails with ErrEntryAlreadyExists if the target
// already exists. Lets a caller that authorized the rename without delete
// access on an absent target stay safe if one is created concurrently.
bool fail_if_target_exists = 6;
}
message AtomicRenameEntryResponse {
+15 -8
View File
@@ -234,9 +234,9 @@ dependencies = [
[[package]]
name = "aws-lc-rs"
version = "1.16.2"
version = "1.18.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a054912289d18629dc78375ba2c3726a3afe3ff71b4edba9dedfca0e3446d1fc"
checksum = "b281d307588d634de920874890732659e2e7672f72b5e10e81badc1a8a83621e"
dependencies = [
"aws-lc-sys",
"zeroize",
@@ -244,14 +244,15 @@ dependencies = [
[[package]]
name = "aws-lc-sys"
version = "0.39.1"
version = "0.45.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "83a25cf98105baa966497416dbd42565ce3a8cf8dbfd59803ec9ad46f3126399"
checksum = "9bff6c3b54fad79a2e60b8102caf565819711497c1f5f092f49508e2f5c31b27"
dependencies = [
"cc",
"cmake",
"dunce",
"fs_extra",
"pkg-config",
]
[[package]]
@@ -2677,6 +2678,12 @@ dependencies = [
"spki 0.7.3",
]
[[package]]
name = "pkg-config"
version = "0.3.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f6b464fbc74e149a392436b17d523f769e057cb6877f6a5c4618bc6f11800548"
[[package]]
name = "png"
version = "0.18.1"
@@ -3389,9 +3396,9 @@ dependencies = [
[[package]]
name = "rustls"
version = "0.23.37"
version = "0.23.45"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "758025cb5fccfd3bc2fd74708fd4682be41d99e5dff73c377c0646c6012c73a4"
checksum = "0d41d731c7d2f962d1ccc364cec258de3c0e93b38c2fb3ba97ac74513048d634"
dependencies = [
"aws-lc-rs",
"log",
@@ -3436,9 +3443,9 @@ dependencies = [
[[package]]
name = "rustls-webpki"
version = "0.103.13"
version = "0.103.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "61c429a8649f110dddef65e2a5ad240f747e85f7758a6bccc7e5777bd33f756e"
checksum = "f3c3cf1d8b1e7d4927e2d154c3fcb02979afb9939629c62cd9048d4f07b60ac2"
dependencies = [
"aws-lc-rs",
"ring",
+5 -1
View File
@@ -64,10 +64,14 @@ seaweed-volume --port 8080 --master localhost:9333 --dir /data/vol1 --max 7 \
| `--whiteList` | | Comma-separated IPs with write permission |
| `--preStopSeconds` | `10` | Graceful drain period before shutdown |
| `--compactionMBps` | `0` (unlimited) | Compaction I/O rate limit |
| `--compactionMaxParallelCommits` | `0` (unlimited) | Max concurrent vacuum commits |
| `--pprof` | `false` | Enable pprof HTTP handlers |
Set `RUST_LOG=debug` (or `trace`, `info`, `warn`) for log level control.
Set `SEAWEED_WRITE_QUEUE=1` to enable batched async write processing.
Durable (`fsync=true`) uploads always go through per-volume batching workers
that share one `.dat`/`.idx` sync per batch, matching the Go volume server;
non-fsync writes take the synchronous path.
## Features
+31
View File
@@ -59,6 +59,8 @@ service Seaweed {
}
rpc RaftLeadershipTransfer (RaftLeadershipTransferRequest) returns (RaftLeadershipTransferResponse) {
}
rpc ResumeVolumeGrowth (ResumeVolumeGrowthRequest) returns (ResumeVolumeGrowthResponse) {
}
rpc VolumeGrow (VolumeGrowRequest) returns (VolumeGrowResponse) {
}
rpc CollectionStatistics (CollectionStatisticsRequest) returns (CollectionStatisticsResponse) {
@@ -119,6 +121,12 @@ message Heartbeat {
// place of `volumes`. A master that does not understand this never sets
// volume_digest_supported, so it keeps being sent the whole list.
repeated VolumeInformationMessage changed_volumes = 28;
// Highest volume id this server has ever observed, persisted locally. Fed
// by HeartbeatResponse.max_volume_id echoes and by AllocateVolume requests.
uint32 max_volume_id_seen = 29;
// Topology id of the master cluster this server last heartbeated to.
string last_seen_topology_id = 30;
}
message HeartbeatResponse {
@@ -135,6 +143,12 @@ message HeartbeatResponse {
// The master compares volume digests, so a server that reports one may send
// changed_volumes in place of its whole list.
bool volume_digest_supported = 9;
// The leader's current allocation bound; servers persist it and report it
// back as max_volume_id_seen so the bound survives master state loss.
uint32 max_volume_id = 10;
// The leader's cluster identity; a different value reported back later
// means the master's state was rebuilt.
string topology_id = 11;
}
message VolumeInformationMessage {
@@ -305,6 +319,18 @@ message VolumeGrowRequest {
string data_node = 7;
uint32 memory_map_max_size_mb = 8;
string disk_type = 9;
// If > 0, grow only when the average fill ratio of the layout's writable
// volumes has reached this threshold (e.g. 0.6). Lets callers pre-grow on
// fill level with one call, instead of listing volumes to compute the
// ratio themselves. Ignored when there are no writable volumes.
float grow_avg_fill_threshold = 10;
}
message ResumeVolumeGrowthRequest {
}
message ResumeVolumeGrowthResponse {
bool was_frozen = 1;
}
message AssignResponse {
@@ -611,4 +637,9 @@ message RaftLeadershipTransferResponse {
}
message VolumeGrowResponse {
// Average fill ratio of the layout's writable volumes at request time
// (0 when there are none). Reported whether or not growth was triggered.
float avg_fill = 1;
// Number of writable volumes at request time.
uint32 writable_count = 2;
}
+1
View File
@@ -571,6 +571,7 @@ message ReadVolumeFileStatusResponse {
string disk_type = 9;
VolumeInfo volume_info = 10;
uint32 version = 11;
uint64 expire_at_sec = 12; // TTL expiry counted from the last write; 0 when the volume never expires
}
message DiskStatus {
+8 -4
View File
@@ -125,6 +125,11 @@ pub struct Cli {
#[arg(long = "maintenanceMBps", default_value_t = 0)]
pub maintenance_mb_per_second: u32,
/// Limit how many vacuum commits run at once on this server; queued
/// commits hold no volume lock. 0 means unlimited.
#[arg(long = "compactionMaxParallelCommits", default_value_t = 0)]
pub max_parallel_vacuum_commits: usize,
/// Limit file size to avoid out of memory.
#[arg(long = "fileSizeLimitMB", default_value_t = 256)]
pub file_size_limit_mb: u32,
@@ -263,6 +268,8 @@ pub struct VolumeServerConfig {
pub mem_profile: String,
pub compaction_byte_per_second: i64,
pub maintenance_byte_per_second: i64,
/// Max concurrent vacuum commits; 0 means unlimited.
pub max_parallel_vacuum_commits: usize,
pub file_size_limit_bytes: i64,
pub concurrent_upload_limit: i64,
pub concurrent_download_limit: i64,
@@ -296,8 +303,6 @@ pub struct VolumeServerConfig {
pub grpc_allowed_wildcard_domain: String,
pub grpc_volume_allowed_common_names: Vec<String>,
pub tls_policy: TlsPolicy,
/// Enable batched write queue for improved throughput under load.
pub enable_write_queue: bool,
/// Path to security.toml — stored for SIGHUP reload.
pub security_file: String,
/// If true, FetchAndWriteNeedle skips remote S3 endpoint validation
@@ -927,6 +932,7 @@ fn resolve_config_with_env(cli: Cli, env: EnvLookup<'_>) -> VolumeServerConfig {
mem_profile: cli.mem_profile,
compaction_byte_per_second: cli.compaction_mb_per_second as i64 * 1024 * 1024,
maintenance_byte_per_second: cli.maintenance_mb_per_second as i64 * 1024 * 1024,
max_parallel_vacuum_commits: cli.max_parallel_vacuum_commits,
file_size_limit_bytes: cli.file_size_limit_mb as i64 * 1024 * 1024,
concurrent_upload_limit: cli.concurrent_upload_limit_mb as i64 * 1024 * 1024,
concurrent_download_limit: cli.concurrent_download_limit_mb as i64 * 1024 * 1024,
@@ -960,8 +966,6 @@ fn resolve_config_with_env(cli: Cli, env: EnvLookup<'_>) -> VolumeServerConfig {
grpc_allowed_wildcard_domain: sec.grpc_allowed_wildcard_domain,
grpc_volume_allowed_common_names: sec.grpc_volume_allowed_common_names,
tls_policy: sec.tls_policy,
enable_write_queue: env_string(env, "SEAWEED_WRITE_QUEUE")
.is_some_and(|v| v == "1" || v == "true"),
security_file: cli.security_file,
allow_untrusted_remote_endpoints: cli.allow_untrusted_remote_endpoints,
}
+13 -6
View File
@@ -340,6 +340,14 @@ async fn run(
rack: config.rack.clone(),
file_size_limit_bytes: config.file_size_limit_bytes,
maintenance_byte_per_second: config.maintenance_byte_per_second,
vacuum_commit_slots: if config.max_parallel_vacuum_commits > 0 {
Some(tokio::sync::Semaphore::new(
config.max_parallel_vacuum_commits,
))
} else {
None
},
vacuum_commits_waiting: std::sync::atomic::AtomicU32::new(0),
// Go sets isHeartbeating: true unconditionally at startup
is_heartbeating: std::sync::atomic::AtomicBool::new(true),
has_master: !config.masters.is_empty(),
@@ -425,12 +433,11 @@ async fn run(
}
}
// Initialize the batched write queue if enabled
if config.enable_write_queue {
info!("Batched write queue enabled");
let wq = WriteQueue::new(state.clone(), 128);
let _ = state.write_queue.set(wq);
}
// The write queue is always on: durable (fsync) uploads go through it so
// a batch shares one .dat/.idx sync, the way Go's per-volume async writer
// group-commits. Non-fsync writes stay on the synchronous path.
let wq = WriteQueue::new(state.clone(), 128);
let _ = state.write_queue.set(wq);
// Set initial metric gauges for concurrent limits and max volumes
metrics::CONCURRENT_UPLOAD_LIMIT.set(state.concurrent_upload_limit);
+1 -1
View File
@@ -397,6 +397,7 @@ mod tests {
mem_profile: String::new(),
compaction_byte_per_second: 0,
maintenance_byte_per_second: 0,
max_parallel_vacuum_commits: 0,
file_size_limit_bytes: 0,
concurrent_upload_limit: 0,
concurrent_download_limit: 0,
@@ -430,7 +431,6 @@ mod tests {
grpc_allowed_wildcard_domain: String::new(),
grpc_volume_allowed_common_names: vec![],
tls_policy: TlsPolicy::default(),
enable_write_queue: false,
security_file: String::new(),
}
}
File diff suppressed because it is too large. Load diff
+251 -52
View File
@@ -161,9 +161,10 @@ impl SourceReadRequest {
/// HEAD, a range, or a stream.
fn served_from_source(&self, n: &Needle) -> bool {
// HEAD is meta-only unless the payload is needed to expand a manifest.
if self.is_head {
return !(n.is_chunk_manifest() && !self.bypass_cm);
// HEAD is meta-only unless the payload is needed to expand a
// manifest or to answer an image transform's length.
return !self.has_image_ops && !(n.is_chunk_manifest() && !self.bypass_cm);
}
self.direct(n) && (self.has_range || n.data_size > STREAMING_THRESHOLD)
}
@@ -171,8 +172,10 @@ impl SourceReadRequest {
/// Meta first only if the needle could be served from source; its data
/// size is below its index size.
fn try_meta_first(&self, size: Size) -> bool {
self.is_head
|| (!self.has_image_ops && (self.has_range || size.0 > STREAMING_THRESHOLD as i32))
if self.has_image_ops {
return false;
}
self.is_head || self.has_range || size.0 > STREAMING_THRESHOLD as i32
}
}
@@ -600,7 +603,7 @@ async fn batch_delete_file_ids(
let locations = lookup_volume(
&state.http_client,
&state.outgoing_http_scheme,
&state.master_url,
&state.current_master().await,
volume_id,
)
.await
@@ -672,7 +675,7 @@ async fn do_replicated_request(
let locations = lookup_volume(
&state.http_client,
&state.outgoing_http_scheme,
&state.master_url,
&state.current_master().await,
vid,
)
.await
@@ -680,9 +683,9 @@ async fn do_replicated_request(
let copy_count = {
let store = state.store.read().unwrap();
store
.find_volume(VolumeId(vid))
.map_or(1, |(_, v)| v.super_block.replica_placement.get_copy_count())
store.find_volume(VolumeId(vid)).map_or(1, |(_, v)| {
v.super_block().replica_placement.get_copy_count()
})
};
let allow_delete = method == axum::http::Method::DELETE;
let eligible_locations: Vec<_> = locations
@@ -829,7 +832,7 @@ async fn proxy_or_redirect_to_target(
let locations = match lookup_volume(
&state.http_client,
&state.outgoing_http_scheme,
&state.master_url,
&state.current_master().await,
vid.0,
)
.await
@@ -1235,7 +1238,7 @@ async fn proxy_missing_volume(
let query_string = uri.query().unwrap_or("").to_string();
if is_proxied_query(&query_string)
|| state.read_mode == ReadMode::Local
|| state.master_url.is_empty()
|| state.current_master().await.is_empty()
{
return StatusCode::NOT_FOUND.into_response();
}
@@ -1279,11 +1282,11 @@ async fn check_download_limit(
// Go tries proxy to replica once before the wait loop
// (checkDownloadLimit); it does not retry on each wakeup.
let should_try_replica =
!is_proxied_query(&query_string) && !state.master_url.is_empty() && {
!is_proxied_query(&query_string) && !state.current_master().await.is_empty() && {
let store = state.store.read().unwrap();
store
.find_volume(vid)
.is_some_and(|(_, vol)| vol.super_block.replica_placement.has_replication())
.is_some_and(|(_, vol)| vol.super_block().replica_placement.has_replication())
};
if should_try_replica
&& let Some(info) = build_proxy_request_info(path, request_headers, &query_string)
@@ -2430,6 +2433,9 @@ fn is_zero_u32(v: &u32) -> bool {
*v == 0
}
/// Store an uploaded needle on the local volume — durable (fsync) uploads
/// through the per-volume write queue, the rest synchronously — and then
/// replicate it if the volume's placement calls for more copies.
pub async fn post_handler(
State(state): State<Arc<VolumeServerState>>,
request: Request<Body>,
@@ -2950,21 +2956,18 @@ pub async fn post_handler(
let needle_etag = n.etag();
let needle_has_name = n.has_name();
let write_result = if let Some(wq) = state.write_queue.get() {
wq.submit(vid, n, fsync).await
} else {
let mut store = state.store.write().unwrap();
store.write_volume_needle(vid, &mut n, fsync)
};
let write_result = write_upload_needle(&state, vid, n, fsync).await;
// Replicate to remote volume servers if this volume has replicas.
// Matches Go's GetWritableRemoteReplications: skip if copy_count == 1.
if !is_replicate && write_result.is_ok() && !state.master_url.is_empty() {
// No leader-known gate: when the lookup cannot run, the write must fail
// rather than be acknowledged on fewer copies than required.
if !is_replicate && write_result.is_ok() {
let needs_replication = {
let store = state.store.read().unwrap();
store
.find_volume(vid)
.is_some_and(|(_, v)| v.super_block.replica_placement.get_copy_count() > 1)
.is_some_and(|(_, v)| v.super_block().replica_placement.get_copy_count() > 1)
};
if needs_replication {
let state_clone = state.clone();
@@ -3049,6 +3052,28 @@ pub async fn post_handler(
resp
}
/// Append an uploaded needle the way Go's `writeNeedle2` does: a durable
/// write goes to the volume's batching worker, which group-commits the
/// batch's .dat/.idx syncs, while anything else — a non-durable write, or a
/// durable one landing while the server is stopping and the workers are
/// winding down — takes the synchronous path and fsyncs inline.
async fn write_upload_needle(
state: &Arc<VolumeServerState>,
vid: VolumeId,
mut n: Needle,
fsync: bool,
) -> Result<(u64, Size, bool), crate::storage::volume::VolumeError> {
let queued = fsync && !*state.is_stopping.read().unwrap();
if queued && let Some(wq) = state.write_queue.get() {
return wq.submit(vid, n, fsync).await;
}
let vol = {
let store = state.store.read().unwrap();
store.writable_volume(vid)
};
vol.and_then(|vol| vol.write_needle(&mut n, true, fsync))
}
// ============================================================================
// Delete Handler
// ============================================================================
@@ -3058,6 +3083,20 @@ struct DeleteResult {
size: i64,
}
/// Read option for deletes: read only the needle's header and trailer.
///
/// A delete needs to know that the needle exists, what its cookie is, and
/// whether it is a chunk manifest. None of that requires the payload. If the
/// needle turns out to be a chunk manifest, the read loads its data as well,
/// because the delete needs the manifest to delete the chunks.
pub(crate) fn delete_read_option() -> ReadOption {
ReadOption {
attempt_meta_only: true,
must_meta_only: true,
..ReadOption::default()
}
}
pub async fn delete_handler(
State(state): State<Arc<VolumeServerState>>,
request: Request<Body>,
@@ -3225,8 +3264,14 @@ pub async fn delete_handler(
// Read needle first to validate cookie (matching Go behavior)
let original_cookie = cookie;
{
let store = state.store.read().unwrap();
match store.read_volume_needle(vid, &mut n) {
let vol = {
let store = state.store.read().unwrap();
store.find_volume(vid).map(|(_, vol)| vol)
};
match vol
.ok_or(crate::storage::volume::VolumeError::NotFound)
.and_then(|vol| vol.read_needle_with_option(&mut n, &mut delete_read_option()))
{
Ok(_) => {}
Err(_) => {
let result = DeleteResult { size: 0 };
@@ -3297,17 +3342,30 @@ pub async fn delete_handler(
}
let delete_result = {
let mut store = state.store.write().unwrap();
store.delete_volume_needle(vid, &mut n)
let vol = {
let store = state.store.read().unwrap();
store
.find_volume(vid)
.map(|(_, vol)| vol)
.ok_or(crate::storage::volume::VolumeError::NotFound)
};
match vol {
Err(e) => Err(e),
// Matches Store::delete_volume_needle: noWriteOrDelete rejects first.
Ok(vol) if vol.is_no_write_or_delete() => {
Err(crate::storage::volume::VolumeError::ReadOnly(vid))
}
Ok(vol) => vol.delete_needle(&mut n),
}
};
let is_replicate = del_query.split('&').any(|p| p == "type=replicate");
if !is_replicate && delete_result.is_ok() && !state.master_url.is_empty() {
if !is_replicate && delete_result.is_ok() {
let needs_replication = {
let store = state.store.read().unwrap();
store
.find_volume(vid)
.is_some_and(|(_, v)| v.super_block.replica_placement.get_copy_count() > 1)
.is_some_and(|(_, v)| v.super_block().replica_placement.get_copy_count() > 1)
};
if needs_replication
&& let Err(e) = do_replicated_request(
@@ -3394,7 +3452,7 @@ pub async fn status_handler(
);
vol_info.insert(
"CompactRevision".to_string(),
serde_json::Value::from(vol.super_block.compaction_revision),
serde_json::Value::from(vol.super_block().compaction_revision),
);
vol_info.insert(
"ModifiedAtSecond".to_string(),
@@ -3405,7 +3463,7 @@ pub async fn status_handler(
serde_json::Value::from(loc.disk_type.to_string()),
);
let replica = &vol.super_block.replica_placement;
let replica = &vol.super_block().replica_placement;
let mut replica_value = serde_json::Map::new();
if replica.diff_data_center_count > 0 {
replica_value.insert(
@@ -3430,7 +3488,7 @@ pub async fn status_handler(
serde_json::Value::Object(replica_value),
);
let ttl = vol.super_block.ttl;
let ttl = vol.super_block().ttl;
let mut ttl_value = serde_json::Map::new();
if ttl.count > 0 {
ttl_value.insert("Count".to_string(), serde_json::Value::from(ttl.count));
@@ -4097,20 +4155,12 @@ async fn read_chunk_needle(
enum Placement {
Ec,
Remote,
Local(std::sync::Arc<crate::storage::volume::Volume>),
}
let placement = {
let store = state.store.read().unwrap();
if store.find_volume(vid).is_some() {
let mut n = Needle {
id: nid,
cookie,
..Needle::default()
};
return store
.read_volume_needle(vid, &mut n)
.map_err(|e| format!("{}", e))
.and_then(|_| cookie_checked_chunk(n, cookie))
.map(|data| ChunkBytes { origin: 0, data });
if let Some((_, vol)) = store.find_volume(vid) {
Placement::Local(vol)
} else if store.find_ec_volume(vid).is_some() {
Placement::Ec
} else {
@@ -4119,6 +4169,17 @@ async fn read_chunk_needle(
};
match placement {
Placement::Local(vol) => {
let mut n = Needle {
id: nid,
cookie,
..Needle::default()
};
vol.read_needle(&mut n)
.map_err(|e| format!("{}", e))
.and_then(|_| cookie_checked_chunk(n, cookie))
.map(|data| ChunkBytes { origin: 0, data })
}
Placement::Ec => {
match crate::server::store_ec::read_ec_shard_needle_distributed(state, vid, nid).await {
Ok(Some(n)) => {
@@ -4173,7 +4234,7 @@ async fn read_remote_chunk_needle(
let locations = lookup_volume(
&state.http_client,
&state.outgoing_http_scheme,
&state.master_url,
&state.current_master().await,
vid.0,
)
.await?;
@@ -5268,6 +5329,8 @@ mod tests {
rack: String::new(),
file_size_limit_bytes: 0,
maintenance_byte_per_second: 0,
vacuum_commit_slots: None,
vacuum_commits_waiting: std::sync::atomic::AtomicU32::new(0),
is_heartbeating: AtomicBool::new(false),
has_master: false,
pre_stop_seconds: 0,
@@ -5457,7 +5520,7 @@ mod tests {
use crate::storage::volume::{Volume, VolumeSpec};
let dir = tmp.path().to_str().unwrap();
let mut v = Volume::new(
let v = Volume::new(
dir,
dir,
VolumeId(2),
@@ -5523,6 +5586,17 @@ mod tests {
(status, headers, body)
}
/// The public router does not accept DELETE requests, so this calls the
/// handler directly, the same way the admin router does.
async fn send_delete(state: &Arc<VolumeServerState>, path: &str) -> StatusCode {
let request = Request::builder()
.method(Method::DELETE)
.uri(path)
.body(Body::empty())
.unwrap();
delete_handler(State(state.clone()), request).await.status()
}
/// A GET whose needle read is parked must not hold the store lock: a
/// writer, here a real append to the same volume, must get through.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
@@ -5627,6 +5701,43 @@ mod tests {
);
}
/// A delete needs the needle's cookie and its chunk-manifest flag. Both
/// are stored in the header and trailer, so the delete must not read the
/// payload.
#[tokio::test]
async fn test_delete_reads_needle_meta_only() {
use crate::storage::volume::needle_read_hook;
use std::sync::Mutex;
const ID: u64 = 0x6e7a_0a01;
let tmp = tempfile::TempDir::new().unwrap();
let state = volume_test_state(&tmp);
let data: Vec<u8> = (0..512 * 1024).map(|i| (i % 251) as u8).collect();
let path = put_test_needle(&state, ID, &data);
let reads = Arc::new(Mutex::new(Vec::new()));
let _hook = needle_read_hook::register(NeedleId(ID), {
let reads = reads.clone();
move |len| reads.lock().unwrap().push(len)
});
let wrong_cookie = format!("/1,{:x}{:08x}", ID, TEST_COOKIE + 1);
assert_eq!(
send_delete(&state, &wrong_cookie).await,
StatusCode::BAD_REQUEST
);
assert_eq!(send_delete(&state, &path).await, StatusCode::ACCEPTED);
let reads = reads.lock().unwrap().clone();
assert!(
!reads.is_empty() && reads.iter().all(|&len| len < 4096),
"a delete must not read the needle's payload ({} bytes): {reads:?}",
data.len()
);
let (status, _, _) = send_read(&state, Method::GET, &path, None).await;
assert_eq!(status, StatusCode::NOT_FOUND);
}
/// HEAD and ranged reads keep answering from the needle meta and the
/// data file, and a missing or wrong-cookie needle is still a 404.
#[tokio::test]
@@ -5715,7 +5826,7 @@ mod tests {
let state = volume_test_state(&tmp);
let dir = tmp.path().to_str().unwrap();
let data: Vec<u8> = (0..4096u32).map(|i| (i % 251) as u8).collect();
let mut v = Volume::new(
let v = Volume::new(
dir,
dir,
VolumeId(2),
@@ -6105,6 +6216,29 @@ mod tests {
assert_eq!(taken(&reads), vec![1]);
}
/// A HEAD asking for an image transform needs the payload too: a
/// meta-only read could not produce the transformed length.
#[tokio::test]
async fn test_head_with_image_op_reads_full() {
let tmp = tempfile::TempDir::new().unwrap();
let state = volume_test_state(&tmp);
let png = solid_png(8, 8);
let stored = put_test_needle(&state, 0x6e7a_0a30, &png);
let crop = format!(
"{}.png?crop_x1=0&crop_y1=0&crop_x2=2&crop_y2=2",
stored
);
let (get_status, _, cropped) = send_read(&state, Method::GET, &crop, None).await;
assert_eq!(get_status, StatusCode::OK);
assert_ne!(cropped.len(), png.len(), "the crop must change the length");
let (status, headers, body) = send_read(&state, Method::HEAD, &crop, None).await;
assert_eq!(status, StatusCode::OK);
assert!(body.is_empty());
assert_eq!(headers[header::CONTENT_LENGTH], cropped.len().to_string());
}
/// Go takes the transform extension from the URL first — including a fid
/// suffix like /vid,fid.png — before the resolved filename's.
#[tokio::test]
@@ -6539,7 +6673,8 @@ mod tests {
let s = Arc::get_mut(&mut state).unwrap();
// Empty self_url makes `url.contains(self_url)` true for every peer.
s.self_url = "198.51.100.8:8080".to_string();
s.master_url = master_url;
s.master_url = master_url.clone();
s.current_master_url = tokio::sync::RwLock::new(master_url);
state
}
@@ -6969,8 +7104,8 @@ mod tests {
);
{
let mut store = state.store.write().unwrap();
let (_, vol) = store.find_volume_mut(VolumeId(1)).unwrap();
let store = state.store.read().unwrap();
let (_, vol) = store.find_volume(VolumeId(1)).unwrap();
vol.fail_next_fsync_for_test(true);
vol.fail_next_truncate_for_test(true);
let mut n = Needle {
@@ -6983,7 +7118,7 @@ mod tests {
store
.write_volume_needle(VolumeId(1), &mut n, true)
.unwrap_err();
let (_, vol) = store.find_volume_mut(VolumeId(1)).unwrap();
let (_, vol) = store.find_volume(VolumeId(1)).unwrap();
vol.fail_next_fsync_for_test(false);
vol.fail_next_truncate_for_test(false);
assert!(vol.unavailable_error().is_some());
@@ -7001,8 +7136,9 @@ mod tests {
}
}
/// A vacuum committed mid-stream must not change the bytes served: the
/// stream's handle pins the inode its offset was resolved against.
/// A leased stream blocks the compaction commit instead of racing the
/// file swap, and still delivers the needle's bytes; the commit goes
/// through once the stream releases its read lease.
#[cfg(unix)]
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_streaming_body_survives_a_vacuum_that_moves_the_needle() {
@@ -7048,13 +7184,76 @@ mod tests {
{
let mut store = state.store.write().unwrap();
store.compact_volume(VolumeId(1), 0, 0, |_| true).unwrap();
store.commit_compact_volume(VolumeId(1)).unwrap();
let err = store.commit_compact_volume(VolumeId(1)).unwrap_err();
assert!(
err.to_string()
.contains(crate::storage::volume::COMMIT_WAITS_FOR_READS),
"the leased stream must hold the commit back: {err}"
);
}
assert_ne!(data_offset(), before, "the vacuum must move the needle");
while let Some(chunk) = stream.next().await {
got.extend_from_slice(&chunk.expect("the stream failed across a vacuum"));
}
assert_eq!(got, data);
drop(stream);
{
let mut store = state.store.write().unwrap();
store.commit_compact_volume(VolumeId(1)).unwrap();
}
assert_ne!(
data_offset(),
before,
"the vacuum moves the needle once the stream drains"
);
}
/// Issue #801: an fsync upload takes the per-volume worker so its sync is
/// shared across the batch (Go's `writeNeedle2` split); a non-fsync write,
/// or an fsync write landing while the server is stopping, takes the
/// synchronous path instead.
#[tokio::test]
async fn test_write_upload_needle_routes_fsync_to_queue() {
let tmp = tempfile::TempDir::new().unwrap();
let state = volume_test_state(&tmp);
assert!(
state
.write_queue
.set(crate::server::write_queue::WriteQueue::new(
state.clone(),
128,
))
.is_ok()
);
let queue = state.write_queue.get().unwrap();
let needle = |id: u64| {
let mut n = Needle {
id: NeedleId(id),
cookie: Cookie(TEST_COOKIE),
data: vec![id as u8; 8],
data_size: 8,
..Needle::default()
};
n.checksum = CRC::new(&n.data);
n
};
write_upload_needle(&state, VolumeId(1), needle(0x6e7a_0b01), true)
.await
.unwrap();
assert_eq!(queue.submits_for_test(), 1);
write_upload_needle(&state, VolumeId(1), needle(0x6e7a_0b02), false)
.await
.unwrap();
assert_eq!(queue.submits_for_test(), 1);
*state.is_stopping.write().unwrap() = true;
write_upload_needle(&state, VolumeId(1), needle(0x6e7a_0b03), true)
.await
.unwrap();
assert_eq!(queue.submits_for_test(), 1);
}
}
+96 -29
View File
@@ -155,7 +155,11 @@ pub async fn run_heartbeat_with_state(
}
PostAction::SleepPulse => {
new_leader = None;
tokio::time::sleep(pulse).await;
// The lookup spends part of the retry delay rather than
// adding to it.
let retry_at = tokio::time::Instant::now() + pulse;
refresh_leader_from_peers(&config, &state, pulse).await;
tokio::time::sleep_until(retry_at).await;
}
}
@@ -254,10 +258,56 @@ pub async fn try_get_master_configuration(
Ok(resp.into_inner())
}
/// Publish the leader the configured masters report, so lookups stop going
/// to the master the heartbeat just lost while the loop retries. Without an
/// answer inside `timeout` the published master is cleared: the lost master
/// is dead or no longer leader, and a follower answers lookups from its own
/// cache rather than refusing them. Mirrors Go's refreshLeaderFromPeers.
async fn refresh_leader_from_peers(
config: &HeartbeatConfig,
state: &Arc<VolumeServerState>,
timeout: Duration,
) {
let lookup = lookup_leader_from_peers(
&config.master_addresses,
state.outgoing_grpc_tls.as_ref(),
);
match tokio::time::timeout(timeout, lookup).await {
Ok(Some(leader)) => publish_current_master(state, &leader).await,
_ => state.current_master_url.write().await.clear(),
}
}
/// Ask each seed master for the raft leader and return the first answer, or
/// None when none replies with one. Mirrors Go's
/// `operation.LookupRaftLeaderMaster`.
pub async fn lookup_leader_from_peers(
master_addrs: &[String],
tls: Option<&super::grpc_client::OutgoingGrpcTlsConfig>,
) -> Option<String> {
for master_addr in master_addrs {
let grpc_addr = to_grpc_address(master_addr);
if let Ok(resp) = try_get_master_configuration(&grpc_addr, tls).await
&& !resp.leader.is_empty()
{
return Some(resp.leader);
}
}
None
}
fn is_stopping(state: &VolumeServerState) -> bool {
*state.is_stopping.read().unwrap()
}
/// Store the master lookups should go to, in the address form it was
/// announced in — `host:port` or `host:port.grpcPort` — so an explicit gRPC
/// port survives to the gRPC callers, mirroring Go's setCurrentMaster.
/// The heartbeat calls this only for a master known to lead.
async fn publish_current_master(state: &Arc<VolumeServerState>, master: &str) {
*state.current_master_url.write().await = master.to_string();
}
fn duplicate_uuid_retry_delay(retry_count: u32) -> Duration {
Duration::from_secs((1u64 << retry_count) * 2)
}
@@ -430,14 +480,6 @@ async fn do_heartbeat(
let mut response_stream = client.send_heartbeat(stream).await?.into_inner();
info!("Heartbeat stream established with {}", grpc_addr);
// Publish the master we're now talking to in canonical http host:port
// form so Ping admission can recognise it once a leader change moves us
// off the seed list. Mirrors Go's vs.setCurrentMaster(masterAddress).
{
let normalised = super::volume_server::to_http_address(current_master).into_owned();
let mut guard = state.current_master_url.write().await;
*guard = normalised;
}
if is_stopping(state) {
state.is_heartbeating.store(false, Ordering::Relaxed);
send_deregister_heartbeat(config, state, &tx).await;
@@ -451,13 +493,31 @@ async fn do_heartbeat(
volume_tick.tick().await;
ec_tick.tick().await;
let mut confirmed_leader = false;
loop {
tokio::select! {
resp = response_stream.message() => {
match resp {
Ok(Some(hb_resp)) => {
// Match Go ordering: DuplicatedUuids first, then volume
// options, then leader redirect.
// A reply names the leader or comes from it, so it
// settles where lookups go before anything else on the
// reply is handled: a follower reply carries no other
// fields, and a leader stays the lookup target even
// when it rejects this registration. Mirrors Go.
if !hb_resp.leader.is_empty() {
if super::volume_server::to_http_address(current_master)
!= super::volume_server::to_http_address(&hb_resp.leader)
{
publish_current_master(state, &hb_resp.leader).await;
return Ok(Some(hb_resp.leader));
}
} else if !confirmed_leader {
// The leader answers every new stream without
// naming a leader, so this peer is the one lookups
// go to.
confirmed_leader = true;
publish_current_master(state, current_master).await;
}
if !hb_resp.duplicated_uuids.is_empty() {
let uuids = hb_resp.duplicated_uuids.clone();
let duplicate_dirs = off_runtime(config, state, move |_, state| {
@@ -474,6 +534,14 @@ async fn do_heartbeat(
)
.into());
}
let echo_max_volume_id = hb_resp.max_volume_id;
let echo_topology_id = hb_resp.topology_id.clone();
off_runtime(config, state, move |_, state| {
let store = state.store.read().unwrap();
store.update_max_volume_id_seen(echo_max_volume_id);
store.update_last_seen_topology_id(&echo_topology_id);
})
.await?;
let options_changed =
volume_options_pass(config, state, &hb_resp).await?;
if options_changed && maybe_adjust_volume_max(config, state).await? {
@@ -494,11 +562,6 @@ async fn do_heartbeat(
if metrics_changed {
state.metrics_notify.notify_waiters();
}
// Match Go: only redirect if leader is non-empty AND
// different from the current master we're connected to.
if !hb_resp.leader.is_empty() && current_master != hb_resp.leader {
return Ok(Some(hb_resp.leader));
}
}
Ok(None) => return Ok(None),
Err(e) => return Err(Box::new(e)),
@@ -966,7 +1029,7 @@ fn apply_volume_actions_to(store: &mut Store, actions: VolumeActions) {
if let Some(vol) = store
.locations
.get_mut(disk_id)
.and_then(|loc| loc.find_volume_mut(vid))
.and_then(|loc| loc.find_volume(vid))
&& vol.should_quarantine()
{
vol.set_no_write_or_delete(true);
@@ -1207,10 +1270,10 @@ fn build_heartbeat_with_ec_status(
deleted_byte_count: vol.deleted_size(),
read_only: vol.is_read_only(),
read_only_can_delete: vol.is_no_write_can_delete(),
replica_placement: vol.super_block.replica_placement.to_byte() as u32,
version: vol.super_block.version.0 as u32,
ttl: vol.super_block.ttl.to_u32(),
compact_revision: vol.super_block.compaction_revision as u32,
replica_placement: vol.super_block().replica_placement.to_byte() as u32,
version: vol.super_block().version.0 as u32,
ttl: vol.super_block().ttl.to_u32(),
compact_revision: vol.super_block().compaction_revision as u32,
// The .dat mtime, as Go reports: the shell's quiet-period
// gates read this as "last touched", which a delete has to
// count towards even though the TTL clock ignores it.
@@ -1360,6 +1423,8 @@ fn build_heartbeat_with_ec_status(
grpc_port: config.grpc_port as u32,
location_uuids,
disk_tags,
max_volume_id_seen: store.max_volume_id_seen(),
last_seen_topology_id: store.last_seen_topology_id(),
..Default::default()
};
(heartbeat, volumes, actions)
@@ -1485,6 +1550,8 @@ mod tests {
rack: String::new(),
file_size_limit_bytes: 0,
maintenance_byte_per_second: 0,
vacuum_commit_slots: None,
vacuum_commits_waiting: std::sync::atomic::AtomicU32::new(0),
is_heartbeating: std::sync::atomic::AtomicBool::new(false),
has_master: true,
pre_stop_seconds: 0,
@@ -1876,7 +1943,7 @@ mod tests {
.store(true, Ordering::Relaxed);
{
let (_, volume) = store.find_volume_mut(VolumeId(17)).unwrap();
let (_, volume) = store.find_volume(VolumeId(17)).unwrap();
volume.set_read_only().unwrap();
volume
.update_remote_files(|files| files.push(Default::default()))
@@ -1975,7 +2042,7 @@ mod tests {
)
.unwrap();
{
let (_, volume) = store.find_volume_mut(VolumeId(21)).unwrap();
let (_, volume) = store.find_volume(VolumeId(21)).unwrap();
volume.set_read_only().unwrap();
}
@@ -2153,7 +2220,7 @@ mod tests {
)
.unwrap();
let dat_path = {
let (_, volume) = store.find_volume_mut(VolumeId(41)).unwrap();
let (_, volume) = store.find_volume(VolumeId(41)).unwrap();
volume.set_last_io_error_for_test(None);
volume.set_last_modified_ts_for_test(
SystemTime::now()
@@ -2207,7 +2274,7 @@ mod tests {
},
)
.unwrap();
let (_, volume) = store.find_volume_mut(VolumeId(51)).unwrap();
let (_, volume) = store.find_volume(VolumeId(51)).unwrap();
volume.set_last_io_error_for_test(Some("input/output error"));
let heartbeat = build_heartbeat(&test_config(), &mut store);
@@ -2217,7 +2284,7 @@ mod tests {
// advertised to the master.
assert!(heartbeat.volumes.is_empty());
assert!(store.has_volume(VolumeId(51)));
let (_, volume) = store.find_volume_mut(VolumeId(51)).unwrap();
let (_, volume) = store.find_volume(VolumeId(51)).unwrap();
assert!(volume.is_no_write_or_delete());
}
@@ -2513,7 +2580,7 @@ mod tests {
},
)
.unwrap();
let (_, volume) = store.find_volume_mut(VolumeId(id)).unwrap();
let (_, volume) = store.find_volume(VolumeId(id)).unwrap();
volume.set_last_io_error_for_test(None);
volume.set_last_modified_ts_for_test(now.saturating_sub(60 * 60));
std::fs::OpenOptions::new()
@@ -2533,7 +2600,7 @@ mod tests {
},
)
.unwrap();
let (_, volume) = store.find_volume_mut(VolumeId(64)).unwrap();
let (_, volume) = store.find_volume(VolumeId(64)).unwrap();
volume.set_last_io_error_for_test(Some("input/output error"));
let (heartbeat, _, actions) =
@@ -2785,7 +2852,7 @@ mod tests {
},
)
.unwrap();
let (_, volume) = store.find_volume_mut(VolumeId(71)).unwrap();
let (_, volume) = store.find_volume(VolumeId(71)).unwrap();
volume
.update_remote_files(|files| {
files.push(crate::storage::volume::PbRemoteFile {
+1 -1
View File
@@ -136,6 +136,7 @@ mod tests {
mem_profile: String::new(),
compaction_byte_per_second: 0,
maintenance_byte_per_second: 0,
max_parallel_vacuum_commits: 0,
file_size_limit_bytes: 0,
concurrent_upload_limit: 0,
concurrent_download_limit: 0,
@@ -169,7 +170,6 @@ mod tests {
grpc_allowed_wildcard_domain: String::new(),
grpc_volume_allowed_common_names: vec![],
tls_policy: TlsPolicy::default(),
enable_write_queue: false,
security_file: String::new(),
}
}
+2 -9
View File
@@ -1139,16 +1139,9 @@ async fn cached_lookup_ec_shard_locations(
state: &Arc<VolumeServerState>,
vid: VolumeId,
) -> io::Result<HashMap<ShardId, Vec<String>>> {
let master = {
let live = state.current_master_url.read().await.clone();
if !live.is_empty() {
live
} else {
state.master_url.clone()
}
};
let master = state.current_master().await;
if master.is_empty() {
return Err(io::Error::other("no master configured for ec shard lookup"));
return Err(io::Error::other("no master leader known for ec shard lookup"));
}
let grpc_addr =
+1 -1
View File
@@ -400,7 +400,7 @@ fn collect_ui_data(
file_count: volume.file_count(),
delete_count: volume.deleted_count(),
deleted_byte_count: volume.deleted_size(),
ttl: volume.super_block.ttl.to_string(),
ttl: volume.super_block().ttl.to_string(),
read_only: volume.is_read_only(),
version: volume.version().0 as u32,
remote_storage_name,
+59 -7
View File
@@ -63,6 +63,12 @@ pub struct VolumeServerState {
pub file_size_limit_bytes: i64,
/// Default IO rate limit for maintenance copy/replication work.
pub maintenance_byte_per_second: i64,
/// Bounds concurrent vacuum commits; None means unlimited. Queued commits
/// wait on the semaphore, not on the store lock, so reads keep flowing.
/// Mirrors Go's `vacuumCommitSlots`.
pub vacuum_commit_slots: Option<tokio::sync::Semaphore>,
/// Vacuum commits currently waiting for a slot.
pub vacuum_commits_waiting: AtomicU32,
/// Whether the server is connected to master (heartbeat active).
pub is_heartbeating: AtomicBool,
/// Whether master addresses are configured.
@@ -71,14 +77,17 @@ pub struct VolumeServerState {
pub pre_stop_seconds: u32,
/// Notify heartbeat to send an immediate update when volume state changes.
pub volume_state_notify: tokio::sync::Notify,
/// Optional batched write queue for improved throughput under load.
/// Per-volume write workers that group-commit durable (fsync) uploads.
/// Set once at startup in production; a test state may leave it unset,
/// which sends every write down the synchronous path.
pub write_queue: std::sync::OnceLock<WriteQueue>,
/// Read mode: local, proxy, or redirect for non-local volumes.
pub read_mode: ReadMode,
/// If true, FetchAndWriteNeedle skips remote S3 endpoint validation,
/// allowing arbitrary (incl. loopback / link-local / metadata) hosts.
pub allow_untrusted_remote_endpoints: bool,
/// First master address for volume lookups (e.g., "localhost:9333").
/// First configured master address (e.g., "localhost:9333"). A seed, not a
/// lookup target: lookups go to `current_master_url`.
pub master_url: String,
/// Seed master addresses for UI rendering.
pub master_urls: Vec<String>,
@@ -86,10 +95,12 @@ pub struct VolumeServerState {
/// Built once at construction so Ping admission stays O(1). Mirrors
/// Go's `seedMasterSet` on `VolumeServer`.
pub seed_master_set: std::collections::HashSet<String>,
/// Current master this server is heartbeating with, in canonical http
/// `host:port` form. Empty when no heartbeat connection is active. The
/// heartbeat goroutine writes; admission reads — the lock keeps them
/// from racing on a leader change. Mirrors Go's `currentMaster` plus
/// Master this server takes as leader, in its announced
/// `host:port[.grpcPort]` form — callers pick http or gRPC form per use.
/// Empty until the heartbeat confirms one — a peer that answered a
/// stream without naming a leader, or the leader a follower named — and
/// cleared again when no master reports one. The heartbeat writes;
/// admission and lookups read. Mirrors Go's `currentMaster` plus
/// `currentMasterLock`.
pub current_master_url: tokio::sync::RwLock<String>,
/// This server's own address (ip:port) for filtering self from lookup results.
@@ -132,6 +143,13 @@ pub struct VolumeServerState {
}
impl VolumeServerState {
/// The master lookups should go to: the leader the heartbeat confirmed,
/// or empty while none is known, in its announced `host:port[.grpcPort]`
/// form. Mirrors Go's `vs.getCurrentMaster()`.
pub async fn current_master(&self) -> String {
self.current_master_url.read().await.clone()
}
/// Check if the server is in maintenance mode; return gRPC error if so.
pub fn check_maintenance(&self) -> Result<(), tonic::Status> {
if self.maintenance.load(Ordering::Relaxed) {
@@ -144,6 +162,40 @@ impl VolumeServerState {
Ok(())
}
/// Wait for a vacuum commit slot without touching the store lock; the wait
/// counts against the caller's `grpc-timeout`. An already-expired deadline
/// is refused even when a slot is free, matching Go's post-acquire
/// `ctx.Err()` check in `acquireVacuumCommitSlot`.
pub async fn acquire_vacuum_commit_slot(
&self,
deadline: Option<tokio::time::Instant>,
) -> Result<Option<tokio::sync::SemaphorePermit<'_>>, tonic::Status> {
let Some(slots) = &self.vacuum_commit_slots else {
return Ok(None);
};
self.vacuum_commits_waiting.fetch_add(1, Ordering::Relaxed);
let acquired = match deadline {
Some(d) => match tokio::time::timeout_at(d, slots.acquire()).await {
Ok(permit) => permit.map_err(|e| tonic::Status::internal(e.to_string())),
Err(_) => Err(tonic::Status::deadline_exceeded(
"timed out waiting for a vacuum commit slot",
)),
},
None => slots
.acquire()
.await
.map_err(|e| tonic::Status::internal(e.to_string())),
};
self.vacuum_commits_waiting.fetch_sub(1, Ordering::Relaxed);
let permit = acquired?;
if deadline.is_some_and(|d| tokio::time::Instant::now() >= d) {
return Err(tonic::Status::deadline_exceeded(
"timed out waiting for a vacuum commit slot",
));
}
Ok(Some(permit))
}
/// Build the seed master set from a list of raw `host:port[.grpcPort]`
/// addresses, normalised the same way Go's `pb.ServerAddress.ToHttpAddress`
/// does (drop the `.grpcPort` suffix, preserve everything else).
@@ -167,7 +219,7 @@ impl VolumeServerState {
return false;
}
let current = self.current_master_url.read().await.clone();
if !current.is_empty() && current == key {
if !current.is_empty() && to_http_address(&current) == key {
return true;
}
self.seed_master_set.contains(&key)
+407 -70
View File
@@ -1,14 +1,19 @@
//! Async batched write processing for the volume server.
//!
//! Instead of each upload handler directly calling `write_needle`, writes are
//! submitted to a queue. A background worker drains the queue in batches (up to
//! 128 entries), groups them by volume ID, and processes them together under a
//! single store lock. Durable writes to a volume share their .dat and .idx
//! flushes (see `Volume::write_needles_grouped`).
//! Durable (fsync) uploads are submitted here instead of each handler calling
//! `write_needle` directly — Go's `writeNeedle2` makes the same split, so the
//! queue is always constructed. Each volume gets its own FIFO worker (Go's
//! volume_write.go keeps one batching worker per volume), which drains its
//! channel in batches of up to 128 entries or 4 MiB and shares the .dat/.idx
//! flushes across durable writes (see `Volume::write_needles_grouped`).
//! Writes to different volumes proceed in parallel; a volume's ordering and
//! exclusion still come from its data-file write lease.
use std::sync::Arc;
use std::collections::HashMap;
use std::collections::hash_map::Entry;
use std::sync::{Arc, Mutex};
use tokio::sync::{mpsc, oneshot};
use tokio::sync::{OwnedSemaphorePermit, Semaphore, mpsc, oneshot};
use tracing::debug;
use crate::storage::needle::needle::Needle;
@@ -29,6 +34,10 @@ pub struct WriteRequest {
pub needle: Needle,
pub fsync: bool,
pub response_tx: oneshot::Sender<WriteResult>,
/// The global queued-slot permit, owned by the request itself: a caller
/// that cancels `submit` after the send must not free the slot while the
/// worker still holds the needle.
pub queue_permit: Option<OwnedSemaphorePermit>,
}
/// Maximum number of write requests to batch together.
@@ -38,102 +47,219 @@ const MAX_BATCH_SIZE: usize = 128;
/// This prevents large writes from accumulating unbounded latency.
const MAX_BATCH_BYTES: usize = 4 * 1024 * 1024;
/// Handle for submitting write requests to the background worker.
/// Cap on volumes doing grouped I/O at once, so many writers cannot crowd
/// readers off a slow disk the way unlimited parallel fsyncs would.
const MAX_CONCURRENT_VOLUME_WRITES: usize = 8;
/// How often an idle worker re-checks that its volume is still mounted.
/// Workers live for their volume's queue lifetime (Go parity), so the only
/// retirement is the volume leaving the store.
const IDLE_WORKER_POLL: std::time::Duration = std::time::Duration::from_secs(60);
/// Handle for submitting write requests to the per-volume workers.
#[derive(Clone)]
pub struct WriteQueue {
tx: mpsc::Sender<WriteRequest>,
state: Arc<VolumeServerState>,
/// Per-volume channel capacity; backpressure kicks in when full.
capacity: usize,
io_permits: Arc<Semaphore>,
/// Bounds queued+in-flight writes across all volumes — per-volume channels
/// alone would multiply the old single queue's capacity by volume count.
queue_permits: Arc<Semaphore>,
workers: Arc<Mutex<HashMap<VolumeId, mpsc::Sender<WriteRequest>>>>,
/// Submissions accepted, for tests asserting which writes route here.
#[cfg(test)]
submits: Arc<std::sync::atomic::AtomicU64>,
}
impl WriteQueue {
/// Create a new write queue and spawn the background worker.
///
/// `capacity` controls the channel buffer size (backpressure kicks in when full).
/// The worker holds a reference to `state` for accessing the store.
/// Create a new write queue. Per-volume workers are spawned lazily on the
/// first submission for that volume.
pub fn new(state: Arc<VolumeServerState>, capacity: usize) -> Self {
let (tx, rx) = mpsc::channel(capacity);
let worker = WriteQueueWorker { rx, state };
tokio::spawn(worker.run());
WriteQueue { tx }
WriteQueue {
state,
capacity,
io_permits: Arc::new(Semaphore::new(MAX_CONCURRENT_VOLUME_WRITES)),
queue_permits: Arc::new(Semaphore::new(capacity)),
workers: Arc::new(Mutex::new(HashMap::new())),
#[cfg(test)]
submits: Arc::new(std::sync::atomic::AtomicU64::new(0)),
}
}
/// How many writes were submitted, for tests asserting upload routing.
#[cfg(test)]
pub fn submits_for_test(&self) -> u64 {
self.submits.load(std::sync::atomic::Ordering::Relaxed)
}
/// Sender for the volume's worker, spawning one on first use.
fn worker_for(&self, volume_id: VolumeId) -> mpsc::Sender<WriteRequest> {
let mut workers = self.workers.lock().unwrap();
match workers.entry(volume_id) {
Entry::Occupied(e) => e.get().clone(),
Entry::Vacant(e) => {
let (tx, rx) = mpsc::channel(self.capacity);
tokio::spawn(
VolumeWriteWorker {
rx,
vid: volume_id,
state: self.state.clone(),
io_permits: self.io_permits.clone(),
workers: self.workers.clone(),
}
.run(),
);
e.insert(tx.clone());
tx
}
}
}
/// Submit a write request and wait for the result.
///
/// Returns `Err` if the worker has shut down or the response channel was dropped.
pub async fn submit(&self, volume_id: VolumeId, needle: Needle, fsync: bool) -> WriteResult {
// Workers spawn only for mounted volumes — a bogus id never creates
// a channel, so volume-id sprays can't grow the worker map.
#[cfg(test)]
self.submits
.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
{
let store = self.state.store.read().unwrap();
if store.find_volume(volume_id).is_none() {
return Err(VolumeError::NotFound);
}
}
// The permit travels inside the request, so the global queued bound
// covers the request until the worker drops it — including after a
// canceled submit.
let queue_permit = self
.queue_permits
.clone()
.acquire_owned()
.await
.map_err(|_| broken_pipe("write queue shut down"))?;
let (response_tx, response_rx) = oneshot::channel();
let request = WriteRequest {
let mut request = WriteRequest {
volume_id,
needle,
fsync,
response_tx,
queue_permit: Some(queue_permit),
};
// Send to queue; this awaits if the channel is full (backpressure).
if self.tx.send(request).await.is_err() {
return Err(VolumeError::Io(std::io::Error::new(
std::io::ErrorKind::BrokenPipe,
"write queue worker has shut down",
)));
// A send fails only against a retiring worker's closed channel; the
// request was never accepted, so retrying it behind a fresh worker
// keeps same-volume order. The volume recheck bounds the spin.
loop {
let tx = self.worker_for(volume_id);
match tx.send(request).await {
Ok(()) => break,
Err(mpsc::error::SendError(returned)) => {
request = returned;
{
let store = self.state.store.read().unwrap();
if store.find_volume(volume_id).is_none() {
return Err(VolumeError::NotFound);
}
}
tokio::task::yield_now().await;
}
}
}
// Wait for the worker to process our request.
match response_rx.await {
Ok(result) => result,
Err(_) => Err(VolumeError::Io(std::io::Error::new(
std::io::ErrorKind::BrokenPipe,
"write queue worker dropped response channel",
))),
Err(_) => Err(broken_pipe("write queue worker dropped response channel")),
}
}
}
/// Background worker that drains write requests and processes them in batches.
struct WriteQueueWorker {
rx: mpsc::Receiver<WriteRequest>,
state: Arc<VolumeServerState>,
fn broken_pipe(msg: &'static str) -> VolumeError {
VolumeError::Io(std::io::Error::new(std::io::ErrorKind::BrokenPipe, msg))
}
impl WriteQueueWorker {
async fn run(mut self) {
debug!("write queue worker started");
/// FIFO worker draining one volume's write channel in batches. Like Go's
/// per-volume async writer it runs for the volume's queue lifetime — exiting
/// only when the channel closes or its volume leaves the store — so no
/// handoff can ever split one volume's write order across two workers.
struct VolumeWriteWorker {
rx: mpsc::Receiver<WriteRequest>,
vid: VolumeId,
state: Arc<VolumeServerState>,
io_permits: Arc<Semaphore>,
workers: Arc<Mutex<HashMap<VolumeId, mpsc::Sender<WriteRequest>>>>,
}
impl VolumeWriteWorker {
async fn run(mut self) {
loop {
// Wait for the first request (blocks until one arrives or channel closes).
let first = match self.rx.recv().await {
Some(req) => req,
None => {
debug!("write queue channel closed, worker exiting");
let first = match tokio::time::timeout(IDLE_WORKER_POLL, self.rx.recv()).await {
Ok(Some(req)) => req,
// recv() yields None only once every sender is gone — the map
// holds a sender per volume, so that is exactly queue shutdown.
Ok(None) => return,
Err(_) if self.volume_gone() => {
self.retire().await;
return;
}
Err(_) => continue,
};
// Drain as many additional requests as available, up to MAX_BATCH_SIZE
// or MAX_BATCH_BYTES (matches Go: 128 requests or 4MB, whichever comes first).
let mut batch = Vec::with_capacity(MAX_BATCH_SIZE);
let mut batch_bytes: usize = first.needle.data.len();
batch.push(first);
while batch.len() < MAX_BATCH_SIZE && batch_bytes < MAX_BATCH_BYTES {
match self.rx.try_recv() {
Ok(req) => {
batch_bytes += req.needle.data.len();
batch.push(req);
}
Err(_) => break,
}
}
let batch_size = batch.len();
debug!("processing write batch of {} requests", batch_size);
// Process the batch in spawn_blocking since write_needle does file I/O.
let state = self.state.clone();
let _ = tokio::task::spawn_blocking(move || {
process_batch(state, batch);
})
.await;
self.process_from(first).await;
}
}
/// The volume's handle left the store, so the worker can go too; an idle
/// worker for a mounted volume stays (Go keeps the writer for the mount).
fn volume_gone(&self) -> bool {
let store = self.state.store.read().unwrap();
store.find_volume(self.vid).is_none()
}
/// Retire in a fixed order that keeps same-volume FIFO: close the channel
/// so racing senders fail fast (they retry and wait for a fresh worker),
/// drain what was already accepted, and only then give up the map slot —
/// a replacement worker can never start while this one still holds queued
/// writes.
async fn retire(&mut self) {
self.rx.close();
while let Some(first) = self.rx.recv().await {
self.process_from(first).await;
}
self.workers.lock().unwrap().remove(&self.vid);
}
/// Collect a batch around `first` and process it under the I/O bound.
async fn process_from(&mut self, first: WriteRequest) {
// Drain as many additional requests as available, up to MAX_BATCH_SIZE
// or MAX_BATCH_BYTES (matches Go: 128 requests or 4MB, whichever comes first).
let mut batch = Vec::with_capacity(MAX_BATCH_SIZE);
let mut batch_bytes: usize = first.needle.data.len();
batch.push(first);
while batch.len() < MAX_BATCH_SIZE && batch_bytes < MAX_BATCH_BYTES {
match self.rx.try_recv() {
Ok(req) => {
batch_bytes += req.needle.data.len();
batch.push(req);
}
Err(_) => break,
}
}
debug!("processing write batch of {} requests", batch.len());
// Bound the number of volumes doing grouped I/O at once, then run
// the batch in spawn_blocking since write_needle does file I/O.
let Ok(_permit) = self.io_permits.clone().acquire_owned().await else {
return;
};
let state = self.state.clone();
let _ = tokio::task::spawn_blocking(move || {
process_batch(state, batch);
})
.await;
}
}
/// Process a batch of write requests, grouped by volume ID.
@@ -155,15 +281,28 @@ fn process_batch(state: Arc<VolumeServerState>, batch: Vec<WriteRequest>) {
}
}
// Process each volume group under a single store lock.
let mut store = state.store.write().unwrap();
// Resolve each group's volume handle under a brief store read, then run
// the group's I/O with the guard released — one volume's write must not
// block admission on another volume's.
for (vid, entries) in groups {
let (mut writes, senders): (Vec<_>, Vec<_>) = entries
.into_iter()
.map(|(needle, fsync, response_tx)| ((needle, fsync), response_tx))
.unzip();
let results = store.write_volume_needles(vid, &mut writes);
let vol = {
let store = state.store.read().unwrap();
store.writable_volume(vid)
};
let results = match vol {
Ok(vol) => vol.write_needles_grouped(&mut writes),
Err(e) => writes
.iter()
.map(|_| match e {
VolumeError::ReadOnly(vid) => Err(VolumeError::ReadOnly(vid)),
_ => Err(VolumeError::NotFound),
})
.collect(),
};
for (response_tx, result) in senders.into_iter().zip(results) {
// Send result back; ignore error if receiver dropped.
let _ = response_tx.send(result);
@@ -206,6 +345,8 @@ mod tests {
rack: String::new(),
file_size_limit_bytes: 0,
maintenance_byte_per_second: 0,
vacuum_commit_slots: None,
vacuum_commits_waiting: std::sync::atomic::AtomicU32::new(0),
is_heartbeating: AtomicBool::new(false),
has_master: false,
pre_stop_seconds: 0,
@@ -363,6 +504,7 @@ mod tests {
},
fsync: true,
response_tx,
queue_permit: None,
}
})
.collect();
@@ -397,4 +539,199 @@ mod tests {
let result = queue2.submit(VolumeId(1), needle, false).await;
assert!(result.is_err()); // NotFound is fine -- the point is it doesn't panic
}
/// Issue #791: a write parked inside its .dat append must not hold the
/// store guard, so another volume still resolves and writes. The hook
/// runs under the volume's data-file lease, so the assertion is that the
/// store — not the per-volume serialization — stays open.
#[test]
fn test_process_batch_does_not_block_other_volumes() {
use crate::config::MinFreeSpace;
use crate::storage::types::{DiskType, NeedleId};
use crate::storage::volume::{VolumeSpec, needle_write_hook};
use std::sync::Barrier;
let tmp = tempfile::TempDir::new().unwrap();
let dir = tmp.path().to_str().unwrap();
let state = make_test_state();
{
let mut store = state.store.write().unwrap();
store
.add_location(
dir,
dir,
10,
DiskType::HardDrive,
MinFreeSpace::Percent(1.0),
Vec::new(),
)
.unwrap();
for vid in [VolumeId(1), VolumeId(2)] {
store
.add_volume(vid, DiskType::HardDrive, &VolumeSpec::default())
.unwrap();
}
}
let in_append = Arc::new(Barrier::new(2));
let release = Arc::new(Barrier::new(2));
let _reg = needle_write_hook::register(NeedleId(101), {
let in_append = in_append.clone();
let release = release.clone();
move |_| {
in_append.wait();
release.wait();
}
});
let (response_tx, mut response_rx) = oneshot::channel();
let batch = vec![WriteRequest {
volume_id: VolumeId(1),
needle: Needle {
id: NeedleId(101),
cookie: 0x1111.into(),
data: vec![7u8; 8],
data_size: 8,
..Needle::default()
},
fsync: false,
response_tx,
queue_permit: None,
}];
let writer = {
let state = state.clone();
std::thread::spawn(move || process_batch(state, batch))
};
in_append.wait(); // volume 1's write is inside its .dat append.
// The store still admits a lifecycle writer and a volume-2 write.
drop(state.store.try_write().unwrap());
let vol2 = state
.store
.read()
.unwrap()
.writable_volume(VolumeId(2))
.unwrap();
let mut n2 = Needle {
id: NeedleId(9),
cookie: 0x2222.into(),
data: vec![9u8; 8],
data_size: 8,
..Needle::default()
};
vol2.write_needle(&mut n2, true, false).unwrap();
// Same-volume exclusion is intact: volume 1's data-file lease is held.
let (_, vol1) = state.store.read().unwrap().find_volume(VolumeId(1)).unwrap();
assert!(vol1.instance().try_write_lock().is_none());
release.wait();
writer.join().unwrap();
assert!(response_rx.try_recv().unwrap().is_ok());
assert!(vol1.instance().try_write_lock().is_some());
}
/// A write parked inside volume 1's .dat append must not keep later
/// batches for other volumes queued behind it: each volume drains through
/// its own FIFO worker, while same-volume writes still serialize.
/// NeedleId(102) is reserved for this test — hooks are keyed by needle id
/// process-wide and parallel tests must not share one.
#[tokio::test]
async fn test_write_queue_does_not_block_other_volumes() {
use crate::config::MinFreeSpace;
use crate::storage::types::{DiskType, NeedleId};
use crate::storage::volume::{VolumeSpec, needle_write_hook};
use std::sync::Barrier;
let tmp = tempfile::TempDir::new().unwrap();
let dir = tmp.path().to_str().unwrap();
let state = make_test_state();
{
let mut store = state.store.write().unwrap();
store
.add_location(
dir,
dir,
10,
DiskType::HardDrive,
MinFreeSpace::Percent(1.0),
Vec::new(),
)
.unwrap();
for vid in [VolumeId(1), VolumeId(2)] {
store
.add_volume(vid, DiskType::HardDrive, &VolumeSpec::default())
.unwrap();
}
}
let queue = WriteQueue::new(state, MAX_BATCH_SIZE);
let in_append = Arc::new(Barrier::new(2));
let release = Arc::new(Barrier::new(2));
let _reg = needle_write_hook::register(NeedleId(102), {
let in_append = in_append.clone();
let release = release.clone();
move |_| {
in_append.wait();
release.wait();
}
});
let needle_a = Needle {
id: NeedleId(102),
cookie: 0x1111.into(),
data: vec![7u8; 8],
data_size: 8,
..Needle::default()
};
let a1 = tokio::spawn({
let queue = queue.clone();
async move { queue.submit(VolumeId(1), needle_a, false).await }
});
// The hook parks on a blocking thread, so wait on the barrier off the
// runtime worker.
let parked = {
let in_append = in_append.clone();
tokio::task::spawn_blocking(move || in_append.wait())
};
parked.await.unwrap(); // volume 1's write is inside its .dat append.
// Volume 2's worker is independent: its submit completes now.
let needle_b = Needle {
id: NeedleId(9),
cookie: 0x2222.into(),
data: vec![9u8; 8],
data_size: 8,
..Needle::default()
};
queue.submit(VolumeId(2), needle_b, true).await.unwrap();
// Same-volume FIFO is preserved: a second volume-1 write stays queued.
let needle_a2 = Needle {
id: NeedleId(2),
cookie: 0x3333.into(),
data: vec![8u8; 8],
data_size: 8,
..Needle::default()
};
let mut a2 = tokio::spawn({
let queue = queue.clone();
async move { queue.submit(VolumeId(1), needle_a2, false).await }
});
assert!(
tokio::time::timeout(std::time::Duration::from_millis(200), &mut a2)
.await
.is_err(),
"a same-volume write must queue behind the stalled one"
);
let released = {
let release = release.clone();
tokio::task::spawn_blocking(move || release.wait())
};
released.await.unwrap();
assert!(a1.await.unwrap().is_ok());
assert!(a2.await.unwrap().is_ok());
}
}
+47 -30
View File
@@ -37,7 +37,7 @@ pub struct DiskLocation {
pub tags: Vec<String>,
pub max_volume_count: AtomicI32,
pub original_max_volume_count: i32,
volumes: HashMap<VolumeId, Volume>,
volumes: HashMap<VolumeId, Arc<Volume>>,
ec_volumes: HashMap<VolumeId, EcVolume>,
pub is_disk_space_low: Arc<AtomicBool>,
pub available_space: AtomicU64,
@@ -237,6 +237,7 @@ impl DiskLocation {
}
for (collection, vid, v) in self.open_volumes(to_load, needle_map_kind) {
let v = Arc::new(v);
crate::metrics::VOLUME_GAUGE
.with_label_values(&[&collection, "volume"])
.inc();
@@ -549,20 +550,23 @@ impl DiskLocation {
None
}
/// Find a volume by ID.
pub fn find_volume(&self, vid: VolumeId) -> Option<&Volume> {
self.volumes.get(&vid)
}
/// Find a volume by ID (mutable).
pub fn find_volume_mut(&mut self, vid: VolumeId) -> Option<&mut Volume> {
self.volumes.get_mut(&vid)
/// Find a volume by ID. Returns a shared handle like Go's findVolume:
/// the lookup borrows the map briefly and I/O runs on the handle after.
pub fn find_volume(&self, vid: VolumeId) -> Option<Arc<Volume>> {
self.volumes.get(&vid).cloned()
}
/// Add a volume to this location.
pub fn set_volume(&mut self, vid: VolumeId, volume: Volume) {
let collection = volume.collection.clone();
self.volumes.insert(vid, volume);
if let Some(old) = self.volumes.insert(vid, Arc::new(volume)) {
// A held handle would keep writing the same files against a stale
// in-memory index; close it so the stale handle fails closed.
crate::metrics::VOLUME_GAUGE
.with_label_values(&[&old.collection, "volume"])
.dec();
old.close();
}
crate::metrics::VOLUME_GAUGE
.with_label_values(&[&collection, "volume"])
.inc();
@@ -586,13 +590,20 @@ impl DiskLocation {
crate::metrics::VOLUME_GAUGE
.with_label_values(&[spec.collection, "volume"])
.inc();
self.volumes.insert(vid, v);
if let Some(old) = self.volumes.insert(vid, Arc::new(v)) {
// Same fail-closed rule as set_volume: a stale handle must not
// keep writing the files the replacement just reopened.
crate::metrics::VOLUME_GAUGE
.with_label_values(&[&old.collection, "volume"])
.dec();
old.close();
}
Ok(())
}
/// Remove and close a volume.
pub fn unload_volume(&mut self, vid: VolumeId) -> Option<Volume> {
if let Some(mut v) = self.volumes.remove(&vid) {
pub fn unload_volume(&mut self, vid: VolumeId) -> Option<Arc<Volume>> {
if let Some(v) = self.volumes.remove(&vid) {
crate::metrics::VOLUME_GAUGE
.with_label_values(&[&v.collection, "volume"])
.dec();
@@ -619,12 +630,21 @@ impl DiskLocation {
{
return Err(v.compacting_error());
}
if let Some(mut v) = self.volumes.remove(&vid) {
crate::metrics::VOLUME_GAUGE
.with_label_values(&[&v.collection, "volume"])
.dec();
v.destroy(only_empty, only_garbage, keep_remote_data)?;
Ok(())
if let Some(v) = self.volumes.remove(&vid) {
match v.destroy(only_empty, only_garbage, keep_remote_data) {
Ok(()) => {
crate::metrics::VOLUME_GAUGE
.with_label_values(&[&v.collection, "volume"])
.dec();
Ok(())
}
Err(e) => {
// A failed destroy left the local files in place, so the
// replica stays registered rather than becoming a phantom.
self.volumes.insert(vid, v);
Err(e)
}
}
} else {
Err(VolumeError::NotFound)
}
@@ -640,12 +660,14 @@ impl DiskLocation {
.collect();
for vid in vids {
if let Some(mut v) = self.volumes.remove(&vid) {
crate::metrics::VOLUME_GAUGE
.with_label_values(&[&v.collection, "volume"])
.dec();
if let Some(v) = self.volumes.remove(&vid) {
if let Err(e) = v.destroy(false, false, false) {
warn!(volume_id = vid.0, error = %e, "delete collection: failed to destroy volume");
self.volumes.insert(vid, v);
} else {
crate::metrics::VOLUME_GAUGE
.with_label_values(&[&v.collection, "volume"])
.dec();
}
}
}
@@ -686,7 +708,7 @@ impl DiskLocation {
/// Iterate over all volumes.
pub fn iter_volumes(&self) -> impl Iterator<Item = (&VolumeId, &Volume)> {
self.volumes.iter()
self.volumes.iter().map(|(k, v)| (k, v.as_ref()))
}
/// Number of free volume slots.
@@ -707,12 +729,7 @@ impl DiskLocation {
/// Iterate over all volumes.
pub fn volumes(&self) -> impl Iterator<Item = (&VolumeId, &Volume)> {
self.volumes.iter()
}
/// Iterate over all volumes (mutable).
pub fn volumes_mut(&mut self) -> impl Iterator<Item = (&VolumeId, &mut Volume)> {
self.volumes.iter_mut()
self.volumes.iter().map(|(k, v)| (k, v.as_ref()))
}
/// Sum of unused space in writable volumes (volumeSizeLimit - actual size per volume).
@@ -502,7 +502,7 @@ mod tests {
let dir = tmp.path().to_str().unwrap();
// Create volume with data
let mut v = Volume::new(
let v = Volume::new(
dir,
dir,
VolumeId(1),
@@ -848,7 +848,7 @@ mod tests {
let dir = tmp.path().to_str().unwrap();
// Create a volume with some data
let mut v = Volume::new(
let v = Volume::new(
dir,
dir,
VolumeId(1),
@@ -895,7 +895,7 @@ mod tests {
fn make_volume_with_needles(n: u64) -> TempDir {
let tmp = TempDir::new().unwrap();
let dir = tmp.path().to_str().unwrap();
let mut v = Volume::new(
let v = Volume::new(
dir,
dir,
VolumeId(1),
@@ -970,7 +970,7 @@ mod tests {
// the dir path so a test can drop/truncate shards and rebuild.
fn encode_sample_volume(tmp: &TempDir) -> String {
let dir = tmp.path().to_str().unwrap().to_string();
let mut v = Volume::new(
let v = Volume::new(
&dir,
&dir,
VolumeId(1),
@@ -1167,7 +1167,7 @@ mod tests {
use crate::storage::volume::{Volume, VolumeSpec};
let tmp = TempDir::new().unwrap();
let dir = tmp.path().to_str().unwrap().to_string();
let mut v = Volume::new(
let v = Volume::new(
&dir,
&dir,
VolumeId(2),
@@ -1216,7 +1216,7 @@ mod tests {
use crate::storage::volume::{Volume, VolumeSpec};
let tmp = TempDir::new().unwrap();
let dir = tmp.path().to_str().unwrap().to_string();
let mut v = Volume::new(
let v = Volume::new(
&dir,
&dir,
VolumeId(3),
@@ -1316,7 +1316,7 @@ mod tests {
let idx_dir = idx_tmp.path().to_str().unwrap();
// Create a volume with separate data and index directories
let mut v = Volume::new(
let v = Volume::new(
dat_dir,
idx_dir,
VolumeId(1),
@@ -1396,7 +1396,7 @@ mod tests {
let idx_dir = idx_tmp.path().to_str().unwrap();
let wrong_dir = wrong_tmp.path().to_str().unwrap();
let mut v = Volume::new(
let v = Volume::new(
dat_dir,
idx_dir,
VolumeId(1),
@@ -1430,7 +1430,7 @@ mod tests {
/// a caller-chosen directory, which is what a split-disk test needs: the
/// shards have to be scattered out of the directory they were encoded into.
fn seed_encoded_volume(dir: &str, vid: VolumeId) {
let mut v = Volume::new(
let v = Volume::new(
dir,
dir,
vid,
@@ -2382,7 +2382,7 @@ mod tests {
let tmp = TempDir::new().unwrap();
let dir = tmp.path().to_str().unwrap();
let mut v = Volume::new(
let v = Volume::new(
dir,
dir,
VolumeId(2074),
@@ -2457,7 +2457,7 @@ mod tests {
let tmp = TempDir::new().unwrap();
let dir = tmp.path().to_str().unwrap();
let mut v = Volume::new(
let v = Volume::new(
dir,
dir,
VolumeId(1),
@@ -2511,7 +2511,7 @@ mod tests {
let tmp = TempDir::new().unwrap();
let dir = tmp.path().to_str().unwrap();
let mut v = Volume::new(
let v = Volume::new(
dir,
dir,
VolumeId(1),
@@ -2600,7 +2600,7 @@ mod tests {
let tmp = TempDir::new().unwrap();
let dir = tmp.path().to_str().unwrap();
let mut v = Volume::new(
let v = Volume::new(
dir,
dir,
VolumeId(1),
@@ -2688,7 +2688,7 @@ mod tests {
let tmp = TempDir::new().unwrap();
let dir = tmp.path().to_str().unwrap();
let mut v = Volume::new(
let v = Volume::new(
dir,
dir,
VolumeId(1),
@@ -2783,7 +2783,7 @@ mod tests {
let tmp = TempDir::new().unwrap();
let dir = tmp.path().to_str().unwrap();
let mut v = Volume::new(
let v = Volume::new(
dir,
dir,
VolumeId(1),
@@ -4201,7 +4201,7 @@ mod tests {
use crate::storage::needle_map::NeedleMapKind;
use crate::storage::volume::{Volume, VolumeSpec};
let mut v = Volume::new(
let v = Volume::new(
dir,
dir,
vid,
@@ -5014,7 +5014,7 @@ mod uniform_layout_tests {
let dir = tmp.path().to_str().unwrap();
let vid = VolumeId(8);
let mut v = Volume::new(
let v = Volume::new(
dir,
dir,
vid,
+2
View File
@@ -11,9 +11,11 @@ pub mod store_ec_mirror;
pub mod store_ec_reconcile;
pub mod super_block;
pub mod types;
pub mod vid_stamp;
pub mod volume;
pub mod volume_idx_rebuild;
pub mod volume_idx_repair;
pub mod volume_open;
pub mod volume_report;
pub mod volume_report_hash;
pub mod volume_tail_repair;
+144 -48
View File
@@ -6,15 +6,15 @@
use std::collections::HashSet;
use std::io;
use std::sync::Mutex;
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
use std::sync::atomic::{AtomicBool, AtomicU32, AtomicU64, Ordering};
use std::sync::{Arc, Mutex};
use crate::config::MinFreeSpace;
use crate::pb::master_pb;
use crate::storage::disk_location::DiskLocation;
use crate::storage::erasure_coding::ec_shard::{EcVolumeShard, MAX_SHARD_COUNT, ShardId};
use crate::storage::erasure_coding::ec_volume::{
ECJ_COMPACT_TMP_EXT, EcVolume, is_usable_ecx_file,
ECJ_COMPACT_TMP_EXT, EcVolume, is_usable_ecx_file, read_ec_shard_config_across_dirs,
};
use crate::storage::needle::needle::{Needle, get_actual_size};
use crate::storage::needle_map::NeedleMapKind;
@@ -43,12 +43,12 @@ fn ensure_compact_volume_space(v: &Volume, preallocate: u64) -> Result<(), Volum
if dir_idx.is_empty() || dir_idx == dir {
check(dir, data_bytes + index_bytes)
} else if !same_filesystem(dir, dir_idx) {
} else if !same_filesystem(dir, &dir_idx) {
check(dir, data_bytes)?;
check(dir_idx, index_bytes)
check(&dir_idx, index_bytes)
} else {
check(dir, data_bytes + index_bytes)?;
check(dir_idx, index_bytes)
check(&dir_idx, index_bytes)
}
}
@@ -72,6 +72,10 @@ pub struct Store {
/// reporting the heartbeat that saw it.
pub reported_collections: Mutex<HashSet<String>>,
pub reported_ec_collections: Mutex<HashSet<String>>,
pub max_volume_id_seen: AtomicU32,
pub last_seen_topology_id: Mutex<String>,
pub vid_stamp_lock: Mutex<()>,
pub vid_stamp_dirty: std::sync::atomic::AtomicBool,
}
impl Store {
@@ -91,6 +95,10 @@ impl Store {
rack: String::new(),
reported_collections: Mutex::new(HashSet::new()),
reported_ec_collections: Mutex::new(HashSet::new()),
max_volume_id_seen: AtomicU32::new(0),
last_seen_topology_id: Mutex::new(String::new()),
vid_stamp_lock: Mutex::new(()),
vid_stamp_dirty: std::sync::atomic::AtomicBool::new(false),
}
}
@@ -152,6 +160,12 @@ impl Store {
// (read-only target, partial copy).
self.reconcile_ec_shards_across_disks();
self.fold_location_into_vid_stamp(self.locations.last().unwrap());
// Write the combined bound back to every dir, including ones whose
// own stamp or volumes sit below it — a lost disk must not restart
// the server below the bound its remaining dirs once reached.
self.save_vid_stamps();
Ok(())
}
@@ -170,8 +184,13 @@ impl Store {
// ---- Volume lookup ----
/// Find which location contains a volume.
pub fn find_volume(&self, vid: VolumeId) -> Option<(usize, &crate::storage::volume::Volume)> {
/// Find which location contains a volume, returning a shared handle.
/// Mirroring Go's findVolume: the lookup only borrows the volume map,
/// so callers can run volume I/O after releasing the store guard.
pub fn find_volume(
&self,
vid: VolumeId,
) -> Option<(usize, Arc<crate::storage::volume::Volume>)> {
for (i, loc) in self.locations.iter().enumerate() {
if let Some(v) = loc.find_volume(vid) {
return Some((i, v));
@@ -180,19 +199,6 @@ impl Store {
None
}
/// Find which location contains a volume (mutable).
pub fn find_volume_mut(
&mut self,
vid: VolumeId,
) -> Option<(usize, &mut crate::storage::volume::Volume)> {
for (i, loc) in self.locations.iter_mut().enumerate() {
if let Some(v) = loc.find_volume_mut(vid) {
return Some((i, v));
}
}
None
}
/// Check if a volume exists.
pub fn has_volume(&self, vid: VolumeId) -> bool {
self.find_volume(vid).is_some()
@@ -202,10 +208,10 @@ impl Store {
/// the volume in place. A no-op when the location has no separate index
/// directory. Mirrors Go's `Store::ConsolidateVolumeIndex`.
pub fn consolidate_volume_index(&mut self, vid: VolumeId) -> Result<(), VolumeError> {
for loc in self.locations.iter_mut() {
for loc in self.locations.iter() {
let idx_dir = loc.idx_directory.clone();
let data_dir = loc.directory.clone();
if let Some(v) = loc.find_volume_mut(vid) {
if let Some(v) = loc.find_volume(vid) {
if idx_dir == data_dir {
return Ok(());
}
@@ -410,6 +416,39 @@ impl Store {
.collect()
}
/// Look up the per-volume EC data-shard count, preferring a mounted EC
/// volume's own config and then the `.vif` on whichever disk holds it.
/// Falls back to the default when neither exists yet (e.g. before the
/// first shard streams in during ec.rebuild), so
/// `find_ec_shard_target_location`'s free-slot maths scores custom-ratio
/// volumes against their real width instead of the build default.
pub fn lookup_ec_data_shards(&self, collection: &str, vid: VolumeId) -> u32 {
use crate::storage::erasure_coding::ec_shard::DATA_SHARDS_COUNT;
for loc in &self.locations {
if let Some(ec_vol) = loc.find_ec_volume(vid) {
return ec_vol.data_shards;
}
}
let Some(first) = self.locations.first() else {
return DATA_SHARDS_COUNT as u32;
};
let other_dirs: Vec<String> = self.locations[1..]
.iter()
.flat_map(|loc| [loc.directory.clone(), loc.idx_directory.clone()])
.filter(|d| !d.is_empty())
.collect();
read_ec_shard_config_across_dirs(
&first.directory,
&first.idx_directory,
&other_dirs,
collection,
vid,
)
.map(|(data_shards, _, _)| data_shards)
.unwrap_or(DATA_SHARDS_COUNT as u32)
}
/// Create a new volume, placing it on the location with the most free space.
pub fn add_volume(
&mut self,
@@ -417,6 +456,7 @@ impl Store {
disk_type: DiskType,
spec: &VolumeSpec<'_>,
) -> Result<(), VolumeError> {
self.stamp_allocated_vid(vid);
if self.find_volume(vid).is_some() {
return Err(VolumeError::AlreadyExists);
}
@@ -514,7 +554,7 @@ impl Store {
vid
))));
}
return loc.create_volume(
let result = loc.create_volume(
vid,
self.needle_map_kind,
&VolumeSpec {
@@ -522,6 +562,10 @@ impl Store {
..Default::default()
},
);
if result.is_ok() {
self.stamp_allocated_vid(vid);
}
return result;
}
}
Err(VolumeError::Io(io::Error::new(
@@ -611,7 +655,10 @@ impl Store {
..Default::default()
},
) {
Ok(()) => return Ok(()),
Ok(()) => {
self.stamp_allocated_vid(vid);
return Ok(());
}
Err(e) => {
last_err = Some(e);
continue;
@@ -667,7 +714,10 @@ impl Store {
..Default::default()
},
) {
Ok(()) => return Ok(()),
Ok(()) => {
self.stamp_allocated_vid(vid);
return Ok(());
}
Err(e) => {
last_err = Some(e);
continue;
@@ -736,6 +786,20 @@ impl Store {
vol.read_needle(n)
}
/// Read a needle from a volume using the given read option.
///
/// On return, `read_option.is_meta_only` tells the caller whether the
/// payload was skipped.
pub fn read_volume_needle_with_option(
&self,
vid: VolumeId,
n: &mut Needle,
read_option: &mut ReadOption,
) -> Result<i32, VolumeError> {
let (_, vol) = self.find_volume(vid).ok_or(VolumeError::NotFound)?;
vol.read_needle_with_option(n, read_option)
}
/// Read a needle from a volume, optionally reading deleted needles.
pub fn read_volume_needle_opt(
&self,
@@ -761,22 +825,22 @@ impl Store {
/// Write a needle to a volume. With `fsync` the volume flushes its .dat
/// before returning, so the caller can ack a durable write.
pub fn write_volume_needle(
&mut self,
&self,
vid: VolumeId,
n: &mut Needle,
fsync: bool,
) -> Result<(u64, Size, bool), VolumeError> {
self.writable_volume_mut(vid)?.write_needle(n, true, fsync)
self.writable_volume(vid)?.write_needle(n, true, fsync)
}
/// Write a batch of needles to one volume, sharing the syncs of its
/// durable writes. See `Volume::write_needles_grouped`.
pub fn write_volume_needles(
&mut self,
&self,
vid: VolumeId,
writes: &mut [(Needle, bool)],
) -> Vec<Result<(u64, Size, bool), VolumeError>> {
match self.writable_volume_mut(vid) {
match self.writable_volume(vid) {
Ok(vol) => vol.write_needles_grouped(writes),
// The lookup fails only with NotFound or the disk-space ReadOnly.
Err(e) => writes
@@ -789,9 +853,12 @@ impl Store {
}
}
fn writable_volume_mut(&mut self, vid: VolumeId) -> Result<&mut Volume, VolumeError> {
/// Resolve a mounted volume handle for writing, checking disk space like
/// Go's writableVolume. The returned Arc stays valid for I/O after the
/// store guard is released; if the volume is unmounted meanwhile the I/O
/// fails on the volume's closed state, not on a stale pointer.
pub(crate) fn writable_volume(&self, vid: VolumeId) -> Result<Arc<Volume>, VolumeError> {
// Check disk space on the location containing this volume.
// We do this before the mutable borrow to avoid borrow conflicts.
let loc_idx = self
.find_volume(vid)
.map(|(i, _)| i)
@@ -803,13 +870,13 @@ impl Store {
return Err(VolumeError::ReadOnly(vid));
}
let (_, vol) = self.find_volume_mut(vid).ok_or(VolumeError::NotFound)?;
let (_, vol) = self.find_volume(vid).ok_or(VolumeError::NotFound)?;
Ok(vol)
}
/// Delete a needle from a volume.
pub fn delete_volume_needle(
&mut self,
&self,
vid: VolumeId,
n: &mut Needle,
) -> Result<Size, VolumeError> {
@@ -818,8 +885,6 @@ impl Store {
if vol.is_no_write_or_delete() {
return Err(VolumeError::ReadOnly(vid));
}
let (_, vol) = self.find_volume_mut(vid).ok_or(VolumeError::NotFound)?;
vol.delete_needle(n)
}
@@ -945,6 +1010,7 @@ impl Store {
collection: &str,
shard_ids: &[ShardId],
) -> Result<(), VolumeError> {
self.stamp_allocated_vid(vid);
// Find the location where the EC files live
let loc_idx = self.find_ec_location(vid, collection).ok_or_else(|| {
VolumeError::Io(io::Error::new(
@@ -967,6 +1033,7 @@ impl Store {
shard_id: ShardId,
source_disk_type: &str,
) -> Result<(), VolumeError> {
self.stamp_allocated_vid(vid);
// The .ecx may live on a different disk than the shard being mounted
// (ec.balance / ec.rebuild can spread shards across sibling disks), so
// look up its owner once and point EcVolume::new at the directory that
@@ -1590,10 +1657,10 @@ impl Store {
let (_, v) = self
.find_volume(vid)
.ok_or(VolumeError::VolumeNotFound(vid))?;
ensure_compact_volume_space(v, preallocate)?;
ensure_compact_volume_space(&v, preallocate)?;
let (_, v) = self
.find_volume_mut(vid)
.find_volume(vid)
.ok_or(VolumeError::VolumeNotFound(vid))?;
v.begin_compact_by_index()
}
@@ -1611,8 +1678,8 @@ impl Store {
collection,
..VolumeSpec::default()
};
let mut v = Volume::new(dir, dir_idx, vid, needle_map_kind, &spec)?;
let mut compact = || -> Result<(), VolumeError> {
let v = Volume::new(dir, dir_idx, vid, needle_map_kind, &spec)?;
let compact = || -> Result<(), VolumeError> {
ensure_compact_volume_space(&v, 0)?;
v.compact_by_index(0, 0, |_| true)?;
v.commit_compact()
@@ -1632,7 +1699,7 @@ impl Store {
/// Commit a completed compaction: swap files and reload.
pub fn commit_compact_volume(&mut self, vid: VolumeId) -> Result<(bool, u64), VolumeError> {
let (_, v) = self
.find_volume_mut(vid)
.find_volume(vid)
.ok_or(VolumeError::VolumeNotFound(vid))?;
let is_read_only = v.is_read_only();
v.commit_compact()?;
@@ -1643,7 +1710,7 @@ impl Store {
/// Clean up leftover compaction files.
pub fn cleanup_compact_volume(&mut self, vid: VolumeId) -> Result<(), VolumeError> {
let (_, v) = self
.find_volume_mut(vid)
.find_volume(vid)
.ok_or(VolumeError::VolumeNotFound(vid))?;
v.cleanup_compact()
}
@@ -2776,7 +2843,7 @@ mod tests {
}
let revision = {
let (_, v) = store.find_volume(vid).unwrap();
v.super_block.compaction_revision
v.super_block().compaction_revision
};
let job = store
@@ -2789,20 +2856,20 @@ mod tests {
assert!(store.delete_volume(vid, false, false, false).is_err());
store.delete_collection("").unwrap();
assert!(store.cleanup_compact_volume(vid).is_err());
let (_, v) = store.find_volume_mut(vid).unwrap();
let (_, v) = store.find_volume(vid).unwrap();
assert!(v.relocate_index_to(other.path().to_str().unwrap()).is_err());
// Go parity: a commit that finds the volume compacting is a no-op.
store.commit_compact_volume(vid).unwrap();
let (_, v) = store.find_volume(vid).expect("still mounted");
assert!(v.is_compacting());
assert_eq!(v.super_block.compaction_revision, revision);
assert_eq!(v.super_block().compaction_revision, revision);
job.run(|_| true).unwrap();
assert!(!store.find_volume(vid).unwrap().1.is_compacting());
store.commit_compact_volume(vid).unwrap();
let (_, v) = store.find_volume(vid).unwrap();
assert_eq!(v.super_block.compaction_revision, revision + 1);
assert_eq!(v.super_block().compaction_revision, revision + 1);
assert!(store.unmount_volume(vid).unwrap());
}
@@ -2842,7 +2909,7 @@ mod tests {
store.delete_volume_needle(vid, &mut del).unwrap();
let revision = {
let (_, v) = store.find_volume(vid).unwrap();
v.super_block.compaction_revision
v.super_block().compaction_revision
};
let job = store.begin_compact_volume(vid, 0).unwrap().unwrap();
@@ -2871,7 +2938,7 @@ mod tests {
store.commit_compact_volume(vid).unwrap();
let (_, v) = store.find_volume(vid).unwrap();
assert_eq!(v.super_block.compaction_revision, revision + 1);
assert_eq!(v.super_block().compaction_revision, revision + 1);
assert_eq!(read_test_needle(&store, vid, 99).unwrap(), b"late-write");
assert_eq!(read_test_needle(&store, vid, 3).unwrap(), b"overwritten");
assert!(read_test_needle(&store, vid, 4).is_err());
@@ -2946,6 +3013,35 @@ mod tests {
);
}
/// A volume whose `.vif` declares a custom EC ratio must feed its own
/// data-shard count into the free-slot maths — not the build default —
/// wherever the `.vif` happens to sit. With no `.vif` anywhere (a
/// brand-new volume mid-ec.rebuild) the answer is the default.
#[test]
fn test_lookup_ec_data_shards_reads_vif_on_any_disk() {
use crate::storage::erasure_coding::ec_shard::DATA_SHARDS_COUNT;
let (store, _tmp) = make_ec_target_test_store(3);
let collection = "custom-ratio";
let vid = VolumeId(4242);
assert_eq!(
store.lookup_ec_data_shards(collection, vid),
DATA_SHARDS_COUNT as u32,
"no .vif anywhere must answer the default"
);
// A 6+2 .vif lands on disk 2 only — the rebuild staging disk.
let base = volume_file_name(&store.locations[2].directory, collection, vid);
std::fs::write(
format!("{}.vif", base),
br#"{"ecShardConfig":{"dataShards":6,"parityShards":2}}"#,
)
.unwrap();
assert_eq!(store.lookup_ec_data_shards(collection, vid), 6);
}
/// Deleting a disk's last shard removes that disk's now-orphaned `.ecsum`;
/// a disk this delete never changed keeps its sidecar. Disk 0 models
/// ec.rebuild's index-prep state (`.ecsum` landed, no shards yet) — an
+232
View File
@@ -0,0 +1,232 @@
//! Persisted volume-id high-water mark ("vid stamp"), mirroring Go's
//! `weed/storage/vid_stamp.go`.
//!
//! Each volume dir carries a `.vid_stamp` JSON file with the highest volume
//! id the server has ever observed plus the topology id it last heartbeated
//! to. The bound is fed by heartbeat echoes, AllocateVolume, and local
//! volume/EC creation, is reported back in `Heartbeat.max_volume_id_seen`,
//! and survives master state loss so rebuilt masters cannot reissue ids an
//! offline server still holds.
use std::fs;
use std::io;
use std::path::Path;
use serde::{Deserialize, Serialize};
use crate::storage::types::VolumeId;
use super::store::Store;
const VID_STAMP_FILE_NAME: &str = ".vid_stamp";
#[derive(Serialize, Deserialize, Default)]
struct VidStamp {
#[serde(rename = "maxVolumeId")]
max_volume_id: u32,
#[serde(rename = "topologyId", default, skip_serializing_if = "String::is_empty")]
topology_id: String,
}
impl Store {
pub fn max_volume_id_seen(&self) -> u32 {
self.max_volume_id_seen.load(std::sync::atomic::Ordering::Relaxed)
}
pub fn last_seen_topology_id(&self) -> String {
self.last_seen_topology_id
.lock()
.unwrap()
.clone()
}
pub fn update_max_volume_id_seen(&self, vid: u32) {
use std::sync::atomic::Ordering::Relaxed;
let prev = self.max_volume_id_seen.fetch_max(vid, Relaxed);
if vid > prev || self.vid_stamp_dirty.load(Relaxed) {
self.save_vid_stamps();
}
}
pub fn update_last_seen_topology_id(&self, id: &str) {
if id.is_empty() {
return;
}
{
let mut guard = self.last_seen_topology_id.lock().unwrap();
if *guard == id && !self.vid_stamp_dirty.load(std::sync::atomic::Ordering::Relaxed) {
return;
}
*guard = id.to_string();
}
self.save_vid_stamps();
}
/// Seed the bound from a freshly added location: its persisted stamp,
/// its volumes, and its EC shards. Called per location from
/// `add_location` after the EC reconcile passes have run.
pub(crate) fn fold_location_into_vid_stamp(&self, loc: &super::disk_location::DiskLocation) {
if let Ok(stamp) = read_vid_stamp(Path::new(&loc.directory)) {
self.update_max_volume_id_seen(stamp.max_volume_id);
if !stamp.topology_id.is_empty() {
self.update_last_seen_topology_id(&stamp.topology_id);
}
}
let mut max_vid: u32 = 0;
for vid in loc.volume_ids() {
max_vid = max_vid.max(vid.0);
}
for (vid, _) in loc.ec_volumes() {
max_vid = max_vid.max(vid.0);
}
if max_vid > 0 {
self.update_max_volume_id_seen(max_vid);
}
}
pub(crate) fn save_vid_stamps(&self) {
let _guard = self.vid_stamp_lock.lock().unwrap();
let stamp = match serde_json::to_vec(&VidStamp {
max_volume_id: self.max_volume_id_seen(),
topology_id: self.last_seen_topology_id(),
}) {
Ok(s) => s,
Err(_) => return,
};
let mut failed = false;
for loc in &self.locations {
if let Err(e) = write_vid_stamp(Path::new(&loc.directory), &stamp) {
tracing::warn!("write vid stamp in {}: {}", loc.directory, e);
failed = true;
}
}
self.vid_stamp_dirty
.store(failed, std::sync::atomic::Ordering::Relaxed);
}
}
fn read_vid_stamp(dir: &Path) -> io::Result<VidStamp> {
let data = fs::read(dir.join(VID_STAMP_FILE_NAME))?;
serde_json::from_slice(&data).map_err(|e| io::Error::new(io::ErrorKind::InvalidData, e))
}
fn write_vid_stamp(dir: &Path, stamp: &[u8]) -> io::Result<()> {
let tmp = dir.join(format!("{}.tmp", VID_STAMP_FILE_NAME));
{
use std::io::Write;
let mut f = fs::File::create(&tmp)?;
f.write_all(stamp)?;
f.sync_all()?;
}
fs::rename(&tmp, dir.join(VID_STAMP_FILE_NAME))?;
if cfg!(unix) {
fs::File::open(dir)?.sync_all()?;
}
Ok(())
}
/// Stamps the volume id of a newly created volume or EC shard set. Called
/// from `Store::add_volume` / `Store::mount_ec_shards*` so the bound covers
/// ids before their files exist.
impl Store {
pub(crate) fn stamp_allocated_vid(&self, vid: VolumeId) {
self.update_max_volume_id_seen(vid.0);
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::config::MinFreeSpace;
use crate::storage::needle_map::NeedleMapKind;
use crate::storage::types::DiskType;
use tempfile::TempDir;
fn make_store(dir: &str) -> Store {
let mut store = Store::new(NeedleMapKind::InMemory);
store
.add_location(dir, dir, 10, DiskType::HardDrive, MinFreeSpace::Percent(1.0), Vec::new())
.unwrap();
store
}
#[test]
fn stamp_written_on_bound_increase() {
let tmp = TempDir::new().unwrap();
let dir = tmp.path().to_str().unwrap();
let store = make_store(dir);
store.update_max_volume_id_seen(42);
let stamp = read_vid_stamp(tmp.path()).unwrap();
assert_eq!(stamp.max_volume_id, 42);
}
#[test]
fn stamp_survives_restart() {
let tmp = TempDir::new().unwrap();
let dir = tmp.path().to_str().unwrap();
let store = make_store(dir);
store.update_max_volume_id_seen(42);
store.update_last_seen_topology_id("cluster-a");
drop(store);
let restarted = make_store(dir);
assert_eq!(restarted.max_volume_id_seen(), 42);
assert_eq!(restarted.last_seen_topology_id(), "cluster-a");
}
#[test]
fn later_disk_gets_combined_stamp() {
let tmp1 = TempDir::new().unwrap();
let tmp2 = TempDir::new().unwrap();
// dir2 carries a stale stamp from an older run
write_vid_stamp(
tmp2.path(),
&serde_json::to_vec(&VidStamp {
max_volume_id: 10,
topology_id: String::new(),
})
.unwrap(),
)
.unwrap();
let mut store = Store::new(NeedleMapKind::InMemory);
store
.add_location(
tmp1.path().to_str().unwrap(),
tmp1.path().to_str().unwrap(),
10,
DiskType::HardDrive,
MinFreeSpace::Percent(1.0),
Vec::new(),
)
.unwrap();
store.update_max_volume_id_seen(100);
store
.add_location(
tmp2.path().to_str().unwrap(),
tmp2.path().to_str().unwrap(),
10,
DiskType::HardDrive,
MinFreeSpace::Percent(1.0),
Vec::new(),
)
.unwrap();
let stamp2 = read_vid_stamp(tmp2.path()).unwrap();
assert_eq!(stamp2.max_volume_id, 100);
}
#[test]
fn bound_is_monotonic() {
let tmp = TempDir::new().unwrap();
let dir = tmp.path().to_str().unwrap();
let store = make_store(dir);
store.update_max_volume_id_seen(10);
store.update_max_volume_id_seen(5);
assert_eq!(store.max_volume_id_seen(), 10);
}
}
File diff suppressed because it is too large. Load diff
@@ -142,7 +142,7 @@ mod tests {
let old_idx = old_idx_dir.to_str().unwrap();
let new_idx = new_idx_dir.to_str().unwrap();
let mut v = Volume::new(
let v = Volume::new(
data,
old_idx,
VolumeId(1),
@@ -196,7 +196,7 @@ mod tests {
let root = TempDir::new().unwrap();
let dir = root.path().to_str().unwrap();
let mut v = Volume::new(
let v = Volume::new(
dir,
dir,
VolumeId(1),
@@ -242,7 +242,7 @@ mod tests {
let root = TempDir::new().unwrap();
let dir = root.path().to_str().unwrap();
let mut v = Volume::new(
let v = Volume::new(
dir,
dir,
VolumeId(1),
@@ -291,7 +291,7 @@ mod tests {
let root = TempDir::new().unwrap();
let dir = root.path().to_str().unwrap();
let mut v = Volume::new(
let v = Volume::new(
dir,
dir,
VolumeId(1),
+15 -12
View File
@@ -12,7 +12,7 @@ use crate::storage::idx;
use crate::storage::needle::Needle;
use crate::storage::needle::needle::needle_body_length;
use crate::storage::types::*;
use crate::storage::volume::{Volume, VolumeError, fsync_dir};
use crate::storage::volume::{Volume, VolumeError, VolumeIo, fsync_dir};
/// Needles found in the head of .dat, keyed by id, plus the ids in .dat order.
type DatHeadNeedles = (HashMap<NeedleId, (Offset, Size)>, Vec<NeedleId>);
@@ -45,15 +45,15 @@ impl Volume {
/// restores .idx to .dat append order, which several readers lean on: the
/// fingerprint is gone so a later load stops after one row, and the last row
/// is the .dat-tail needle again.
pub(crate) fn repair_idx_head_tombstones(&self) -> Result<usize, VolumeError> {
if !self.has_data_backend() {
pub(crate) fn repair_idx_head_tombstones(&self, io: &VolumeIo) -> Result<usize, VolumeError> {
if !self.has_data_backend_io(io) {
return Ok(0);
}
let version = self.version();
if !version.is_supported() {
return Ok(0);
}
let first_needle_offset = self.super_block.block_size() as i64;
let first_needle_offset = self.super_block().block_size() as i64;
let idx_path = self.file_name(".idx");
let clobbered = idx_head_tombstone_count(&idx_path)?;
@@ -68,7 +68,7 @@ impl Volume {
"idx starts with offset-0 tombstones, recovering the rows they overwrote from .dat"
);
let (mut lost, order) = self.scan_dat_head(version, first_needle_offset, clobbered)?;
let (mut lost, order) = self.scan_dat_head(io, version, first_needle_offset, clobbered)?;
drop_indexed_keys(&idx_path, &mut lost)?;
if lost.is_empty() {
return Ok(0);
@@ -93,6 +93,7 @@ impl Volume {
/// being resurrected.
fn scan_dat_head(
&self,
io: &VolumeIo,
version: Version,
first_needle_offset: i64,
limit: usize,
@@ -103,7 +104,7 @@ impl Volume {
for _ in 0..limit {
let mut header = [0u8; NEEDLE_HEADER_SIZE];
match self.read_exact_at_backend(&mut header, offset as u64) {
match self.read_exact_at_backend(io, &mut header, offset as u64) {
Ok(()) => {}
Err(VolumeError::Io(ref e)) if e.kind() == io::ErrorKind::UnexpectedEof => break,
Err(e) => return Err(e),
@@ -245,7 +246,7 @@ mod tests {
}
fn write_test_volume(dir: &str, needle_count: u64) -> Vec<Vec<u8>> {
let mut v = open_volume(dir);
let v = open_volume(dir);
let mut written = Vec::new();
for i in 1..=needle_count {
let data = format!("needle-{}-payload", i).into_bytes();
@@ -355,7 +356,7 @@ mod tests {
let rows = idx_rows(&idx_path);
assert_eq!(
rows[0].1,
v.super_block.block_size() as i64,
v.super_block().block_size() as i64,
"first row does not index the first needle in .dat"
);
for i in 1..4 {
@@ -380,7 +381,7 @@ mod tests {
let idx_path = format!("{}/1.idx", dir);
write_test_volume(dir, 6);
let mut v = open_volume(dir);
let v = open_volume(dir);
v.delete_needle(&mut Needle {
id: NeedleId(3),
..Needle::default()
@@ -415,7 +416,7 @@ mod tests {
let idx_path = format!("{}/1.idx", dir);
write_test_volume(dir, 8);
let mut v = open_volume(dir);
let v = open_volume(dir);
v.delete_needle(&mut Needle {
id: NeedleId(2),
..Needle::default()
@@ -456,8 +457,10 @@ mod tests {
drop(dat);
let v = open_volume(dir);
let first = v.super_block.block_size() as i64;
let err = v.scan_dat_head(v.version(), first, 10).unwrap_err();
let first = v.super_block().block_size() as i64;
let err = v
.scan_dat_head(&v.io_read_for_test(), v.version(), first, 10)
.unwrap_err();
assert!(
matches!(&err, VolumeError::Io(e) if e.kind() == io::ErrorKind::InvalidData),
"expected a corrupt-data error, got {err:?}"
File diff suppressed because it is too large. Load diff
+7 -5
View File
@@ -107,6 +107,8 @@ fn build_test_state(
rack: String::new(),
file_size_limit_bytes: 0,
maintenance_byte_per_second: 0,
vacuum_commit_slots: None,
vacuum_commits_waiting: std::sync::atomic::AtomicU32::new(0),
is_heartbeating: std::sync::atomic::AtomicBool::new(true),
ec_decodes_in_flight: std::sync::Mutex::new(std::collections::HashSet::new()),
ec_decode_tail: std::sync::Mutex::new(std::collections::HashSet::new()),
@@ -509,7 +511,7 @@ async fn write_and_delete_on_read_only_volume_say_is_read_only() {
.store
.write()
.unwrap()
.find_volume_mut(VolumeId(1))
.find_volume(VolumeId(1))
.unwrap()
.1
.set_no_write_or_delete(true);
@@ -537,7 +539,7 @@ async fn write_and_delete_on_read_only_volume_say_is_read_only() {
.store
.write()
.unwrap()
.find_volume_mut(VolumeId(1))
.find_volume(VolumeId(1))
.unwrap()
.1
.set_no_write_or_delete(false);
@@ -1042,7 +1044,7 @@ async fn chunk_manifest_expands_chunk_stored_on_ec_volume() {
// Build regular volume 2 holding the chunk, then EC-encode it and mount all
// 14 shards locally so reconstruct-on-read is a pure local read.
{
let mut v = Volume::new(
let v = Volume::new(
dir,
dir,
VolumeId(2),
@@ -1082,7 +1084,7 @@ async fn chunk_manifest_expands_chunk_stored_on_ec_volume() {
let manifest_nid = NeedleId(0x7);
let manifest_cookie = Cookie(0x55667788);
{
let mut store = state.store.write().unwrap();
let store = state.store.read().unwrap();
let mut n = Needle {
id: manifest_nid,
cookie: manifest_cookie,
@@ -1146,7 +1148,7 @@ async fn delete_on_ec_volume_succeeds_when_the_needles_shard_is_not_mounted() {
// Build regular volume 4, then EC-encode it. The volume is standalone and
// never registered in the store, so afterwards it exists only as shards.
{
let mut v = Volume::new(
let v = Volume::new(
dir,
dir,
VolumeId(4),
+89
View File
@@ -0,0 +1,89 @@
//! Issue #791: the shared `RwLock<Store>` guards the volume map only.
//! `find_volume` hands out an `Arc<Volume>` so needle I/O runs without the
//! guard; a handle that outlives its mount must fail closed rather than I/O
//! a torn-down volume — the same end Go reaches through nil needle-map/dat
//! checks under the per-volume dataFileAccessLock.
use std::sync::{Arc, RwLock};
use seaweed_volume::config::MinFreeSpace;
use seaweed_volume::storage::needle::needle::Needle;
use seaweed_volume::storage::needle_map::NeedleMapKind;
use seaweed_volume::storage::store::Store;
use seaweed_volume::storage::types::{DiskType, NeedleId, VolumeId};
use seaweed_volume::storage::volume::VolumeSpec;
use tempfile::TempDir;
fn store_with_two_volumes() -> (Arc<RwLock<Store>>, TempDir) {
let tmp = TempDir::new().unwrap();
let dir = tmp.path().to_str().unwrap();
let mut store = Store::new(NeedleMapKind::InMemory);
store
.add_location(dir, dir, 10, DiskType::HardDrive, MinFreeSpace::Percent(1.0), Vec::new())
.unwrap();
store
.add_volume(VolumeId(1), DiskType::HardDrive, &VolumeSpec::default())
.unwrap();
store
.add_volume(VolumeId(2), DiskType::HardDrive, &VolumeSpec::default())
.unwrap();
(Arc::new(RwLock::new(store)), tmp)
}
fn needle(id: u64) -> Needle {
Needle {
id: NeedleId(id),
cookie: 0x1234.into(),
data: vec![7u8; 16],
data_size: 16,
..Needle::default()
}
}
#[test]
fn store_read_guard_admits_cross_volume_io() {
let (store, _tmp) = store_with_two_volumes();
// Read-guarded callers can write on different volumes concurrently —
// what used to require store.write() now only resolves a handle.
let t = {
let store = store.clone();
std::thread::spawn(move || {
let store = store.read().unwrap();
store
.write_volume_needle(VolumeId(1), &mut needle(1), false)
.unwrap();
})
};
{
let store = store.read().unwrap();
store
.write_volume_needle(VolumeId(2), &mut needle(9), false)
.unwrap();
}
t.join().unwrap();
let store = store.read().unwrap();
let mut n = needle(9);
n.data.clear();
assert_eq!(store.read_volume_needle(VolumeId(2), &mut n).unwrap(), 16);
let mut n = needle(1);
n.data.clear();
assert_eq!(store.read_volume_needle(VolumeId(1), &mut n).unwrap(), 16);
}
#[test]
fn stale_volume_handle_fails_after_unmount() {
let (store, _tmp) = store_with_two_volumes();
// A handle resolved before unmount keeps the Volume alive but must not
// be able to write into its closed files.
let (_, vol1) = store.read().unwrap().find_volume(VolumeId(1)).unwrap();
store.write().unwrap().unmount_volume(VolumeId(1)).unwrap();
assert!(vol1.write_needle(&mut needle(1), true, false).is_err());
let mut n = needle(1);
n.data.clear();
assert!(vol1.read_needle(&mut n).is_err());
}
+11 -2
View File
@@ -485,6 +485,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ce2b2dcc879c3bae0d371e77c99f2238400ef24ec001394befa67b6e543add9e"
dependencies = [
"aws-lc-sys",
"untrusted 0.7.1",
"zeroize",
]
@@ -5124,7 +5125,7 @@ dependencies = [
"cfg-if",
"getrandom 0.2.17",
"libc",
"untrusted",
"untrusted 0.9.0",
"windows-sys 0.52.0",
]
@@ -5260,7 +5261,7 @@ dependencies = [
"aws-lc-rs",
"ring",
"rustls-pki-types",
"untrusted",
"untrusted 0.9.0",
]
[[package]]
@@ -5342,7 +5343,9 @@ version = "0.1.0"
dependencies = [
"anyhow",
"async-trait",
"aws-lc-rs",
"axum",
"base64 0.22.1",
"prometheus",
"prost",
"prost-types",
@@ -6213,6 +6216,12 @@ version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b4ac048d71ede7ee76d585517add45da530660ef4390e49b098733c6e897f254"
[[package]]
name = "untrusted"
version = "0.7.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a156c684c91ea7d62626509bce3cb4e1d9ed5c4d978f7b4352658f96a4c26b4a"
[[package]]
name = "untrusted"
version = "0.9.0"
+4 -4
View File
@@ -2,7 +2,7 @@
`weed/pb/plugin.proto` is a language-agnostic contract: a maintenance worker
connects out to admin, announces the job types it can detect and execute, and
answers requests on that one stream. `weed worker -admin=host:23646` is the Go
answers requests on that one stream. `weed worker -admin=host:12535` is the Go
implementation of it from outside the admin process. This workspace is the Rust
one.
@@ -35,7 +35,7 @@ binary for the runner's platform, resolved from the version in `Cargo.lock`.
## Running
cargo run -p weed-lance-worker -- --admin 127.0.0.1:23646
cargo run -p weed-lance-worker -- --admin 127.0.0.1:12535
The admin's *HTTP* address is what an operator has; the gRPC port is derived
from it the way the Go side does. Dialling the HTTP port fails as "frame with
@@ -49,7 +49,7 @@ Released builds do not need a toolchain. The worker ships inside the SeaweedFS
image, beside the Rust volume server, under the verb that mirrors
`volume-rust`:
docker run chrislusf/seaweedfs worker-rust --admin admin:23646
docker run chrislusf/seaweedfs worker-rust --admin admin:12535
and as `weed-worker_linux_{amd64,arm64}.tar.gz` on each GitHub release. Both are
linux amd64/arm64 only — lance, arrow and datafusion make every extra target an
@@ -59,7 +59,7 @@ entrypoint says so rather than failing as "not found".
## Metrics
cargo run -p weed-lance-worker -- --admin 127.0.0.1:23646 --metrics-port 9328
cargo run -p weed-lance-worker -- --admin 127.0.0.1:12535 --metrics-port 9328
Serves `/health`, `/ready` and `/metrics` on that port, the same three the Go
worker serves under `weed worker -metricsPort`, so one scrape config covers
+4
View File
@@ -12,6 +12,10 @@ name = "seaweed_worker_core"
# Helpers the Rust volume server needs as well. A path dependency because
# the two trees are separate cargo workspaces with no common root manifest.
seaweed-common = { path = "../../../seaweed-common" }
# Already in the graph through tonic's `tls-aws-lc` feature; named here so the
# admin token can sign with the same HMAC the Go worker uses.
aws-lc-rs = "1"
base64 = "0.22"
anyhow.workspace = true
async-trait.workspace = true
prost.workspace = true
+13 -2
View File
@@ -3,7 +3,7 @@ use std::time::Duration;
/// How one worker process connects and how much work it will take on.
#[derive(Clone, Debug)]
pub struct WorkerOptions {
/// Admin gRPC address, e.g. "localhost:23646".
/// Admin gRPC address, e.g. "localhost:12535".
pub admin_address: String,
pub worker_id: String,
pub worker_version: String,
@@ -17,6 +17,15 @@ pub struct WorkerOptions {
/// section of security.toml. None means plaintext, which is the default the
/// Go worker also takes when no certificates are configured.
pub tls: Option<TlsOptions>,
/// The shared secret the admin stream requires when the cluster configures
/// `jwt.filer_signing.key`; mirrored from that config value. Empty means no
/// token is attached, which only works against admins that either bound
/// their worker listener to loopback or authenticate with mTLS instead.
pub signing_key: Option<String>,
/// Seconds the minted admin token lives, mirroring
/// `jwt.filer_signing.expires_after_seconds`. Zero mints a token without an
/// expiry, matching the Go default.
pub signing_key_expires_after_seconds: u64,
}
/// Certificates for the control stream. All three are required together: the
@@ -35,7 +44,7 @@ pub struct TlsOptions {
impl Default for WorkerOptions {
fn default() -> Self {
Self {
admin_address: "localhost:23646".to_string(),
admin_address: "localhost:12535".to_string(),
worker_id: String::new(),
worker_version: env!("CARGO_PKG_VERSION").to_string(),
worker_address: String::new(),
@@ -44,6 +53,8 @@ impl Default for WorkerOptions {
max_detection_concurrency: 1,
max_execution_concurrency: 1,
tls: None,
signing_key: None,
signing_key_expires_after_seconds: 0,
}
}
}
+2 -1
View File
@@ -2,7 +2,7 @@
//!
//! `weed/pb/plugin.proto` is a language-agnostic gRPC stream: a worker connects
//! out to admin, announces the job types it can detect and execute, and then
//! answers requests on that one stream. `weed worker -admin=host:23646` is the
//! answers requests on that one stream. `weed worker -admin=host:12535` is the
//! Go implementation of the same contract from outside the admin process; this
//! is the Rust one, and it knows nothing about any particular job.
@@ -13,6 +13,7 @@ pub mod metrics;
pub mod registry;
pub mod senders;
pub mod stream;
pub mod token;
/// Generated plugin.proto types.
pub mod pb {
+13 -4
View File
@@ -176,10 +176,19 @@ async fn serve_once(
metadata: Default::default(),
}))?;
let mut inbound = client
.worker_stream(UnboundedReceiverStream::new(rx))
.await?
.into_inner();
let mut request = tonic::Request::new(UnboundedReceiverStream::new(rx));
if let Some(token) = crate::token::admin_token(
options.signing_key.as_deref().unwrap_or("").as_bytes(),
options.signing_key_expires_after_seconds,
) {
request.metadata_mut().insert(
"authorization",
format!("Bearer {token}")
.parse()
.context("minted admin token is not valid metadata")?,
);
}
let mut inbound = client.worker_stream(request).await?.into_inner();
let heartbeat = spawn_heartbeat(
sender.clone(),
+61
View File
@@ -0,0 +1,61 @@
use std::time::{SystemTime, UNIX_EPOCH};
use aws_lc_rs::hmac;
use base64::{engine::general_purpose::URL_SAFE_NO_PAD, Engine};
/// Mints the HS256 Bearer token the admin control stream requires when
/// jwt.filer_signing.key is configured, matching the SeaweedFilerAdminClaims
/// the Go worker sends: registered claims only, no session id.
pub fn admin_token(signing_key: &[u8], expires_after_secs: u64) -> Option<String> {
if signing_key.is_empty() {
return None;
}
let claims = if expires_after_secs > 0 {
let exp = SystemTime::now()
.duration_since(UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or_default()
+ expires_after_secs;
format!("{{\"exp\":{exp}}}")
} else {
"{}".to_string()
};
let signing_input = format!(
"{}.{}",
URL_SAFE_NO_PAD.encode(b"{\"alg\":\"HS256\",\"typ\":\"JWT\"}"),
URL_SAFE_NO_PAD.encode(claims.as_bytes())
);
let key = hmac::Key::new(hmac::HMAC_SHA256, signing_key);
let signature = hmac::sign(&key, signing_input.as_bytes());
Some(format!(
"{signing_input}.{}",
URL_SAFE_NO_PAD.encode(signature.as_ref())
))
}
#[cfg(test)]
mod tests {
use super::admin_token;
#[test]
fn no_key_no_token() {
assert!(admin_token(b"", 0).is_none());
}
#[test]
fn minted_token_verifies_shape() {
let token = admin_token(b"secret", 60).unwrap();
let parts: Vec<&str> = token.split('.').collect();
assert_eq!(parts.len(), 3);
assert!(!parts.iter().any(|p| p.is_empty()));
}
#[test]
fn no_expiry_has_empty_claims() {
let token = admin_token(b"secret", 0).unwrap();
let payload = token.split('.').nth(1).unwrap();
use base64::{engine::general_purpose::URL_SAFE_NO_PAD, Engine};
let claims = URL_SAFE_NO_PAD.decode(payload).unwrap();
assert_eq!(claims, b"{}");
}
}
+15 -1
View File
@@ -15,7 +15,7 @@ use weed_lance_worker::metrics::LanceMetrics;
#[command(name = "weed-worker", about = "SeaweedFS maintenance worker")]
struct Args {
/// Admin server gRPC address.
#[arg(long, default_value = "localhost:23646", env = "WEED_ADMIN")]
#[arg(long, default_value = "localhost:12535", env = "WEED_ADMIN")]
admin: String,
/// Worker identity reported to admin.
@@ -61,6 +61,18 @@ struct Args {
#[arg(long)]
tls_server_name: Option<String>,
/// Shared secret for the admin control stream, the `jwt.filer_signing.key`
/// value the Go worker reads from its config file. Required by admins that
/// authenticate the worker stream with a signing key.
#[arg(long, env = "WEED_FILER_SIGNING_KEY")]
filer_signing_key: Option<String>,
/// Lifetime of the token minted from the signing key, mirroring
/// `jwt.filer_signing.expires_after_seconds`. Zero mints a token with no
/// expiry.
#[arg(long, default_value = "0")]
filer_signing_key_expires_after_seconds: u64,
/// Serve /health, /ready and /metrics on this port, the way
/// `weed worker -metricsPort` does. Zero, the default, serves nothing.
#[arg(long, default_value = "0", env = "WEED_METRICS_PORT")]
@@ -131,6 +143,8 @@ async fn main() -> Result<()> {
max_detection_concurrency: args.max_concurrency,
max_execution_concurrency: args.max_concurrency,
tls,
signing_key: args.filer_signing_key,
signing_key_expires_after_seconds: args.filer_signing_key_expires_after_seconds,
..Default::default()
};
+1 -1
View File
@@ -1,6 +1,6 @@
# Cluster security group + the SeaweedFS port matrix.
# Intra-cluster: all TCP allowed within the SG (master 9333/19333, volume
# 8080/18080, filer 8888/18888, admin 33646, ...). Client-facing: only S3 (8333)
# 8080/18080, filer 8888/18888, admin 22535, ...). Client-facing: only S3 (8333)
# and filer (8888). Metrics ports (9327) are never opened to clients.
resource "aws_security_group" "cluster" {
@@ -18,7 +18,7 @@ import (
)
const (
AdminUrl = "http://localhost:23646"
AdminUrl = "http://localhost:12535"
MasterUrl = "http://localhost:9333"
FilerUrl = "http://localhost:8888"
)
@@ -147,13 +147,13 @@ func ensureEnvironment(t *testing.T) {
debugPort := 6060 + i
dir, _ := filepath.Abs(filepath.Join("tmp", workerName))
os.MkdirAll(dir, 0755)
startWeed(t, workerName, "worker", "-admin=localhost:23646", "-workingDir="+dir, fmt.Sprintf("-metricsPort=%d", metricsPort), fmt.Sprintf("-debug.port=%d", debugPort))
startWeed(t, workerName, "worker", "-admin=localhost:12535", "-workingDir="+dir, fmt.Sprintf("-metricsPort=%d", metricsPort), fmt.Sprintf("-debug.port=%d", debugPort))
}
// 6. Start Admin
os.RemoveAll(filepath.Join("tmp", "admin"))
os.MkdirAll(filepath.Join("tmp", "admin"), 0755)
startWeed(t, "admin", "admin", "-master=localhost:9333", "-port=23646", "-dataDir=./tmp/admin")
startWeed(t, "admin", "admin", "-master=localhost:9333", "-port=12535", "-dataDir=./tmp/admin")
// Admin is started after master, 14 volume servers, filer and 2 workers,
// so under cold CI conditions the wait here has to absorb the tail of
// every earlier subprocess coming up. 60s is too tight and has flaked;
+1 -1
View File
@@ -24,7 +24,7 @@ require (
github.com/jcmturner/gofork v1.7.6 // indirect
github.com/jcmturner/gokrb5/v8 v8.4.4 // indirect
github.com/jcmturner/rpc/v2 v2.0.3 // indirect
github.com/klauspost/compress v1.18.0 // indirect
github.com/klauspost/compress v1.18.7 // indirect
github.com/kr/text v0.2.0 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/pierrec/lz4/v4 v4.1.22 // indirect
+2 -2
View File
@@ -38,8 +38,8 @@ github.com/jcmturner/gokrb5/v8 v8.4.4 h1:x1Sv4HaTpepFkXbt2IkL29DXRf8sOfZXo8eRKh6
github.com/jcmturner/gokrb5/v8 v8.4.4/go.mod h1:1btQEpgT6k+unzCwX1KdWMEwPPkkgBtP+F6aCACiMrs=
github.com/jcmturner/rpc/v2 v2.0.3 h1:7FXXj8Ti1IaVFpSAziCZWNzbNuZmnvw/i6CqLNdWfZY=
github.com/jcmturner/rpc/v2 v2.0.3/go.mod h1:VUJYCIDm3PVOEHw8sgt091/20OJjskO/YJki3ELg/Hc=
github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ=
github.com/klauspost/compress v1.18.7 h1:aUyZsS4kH3QTKurYhAOwAHxllVPnOthb3vPfnF1Ehjw=
github.com/klauspost/compress v1.18.7/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
@@ -5,6 +5,6 @@ go 1.24.7
require github.com/segmentio/kafka-go v0.4.49
require (
github.com/klauspost/compress v1.15.9 // indirect
github.com/klauspost/compress v1.18.7 // indirect
github.com/pierrec/lz4/v4 v4.1.15 // indirect
)
@@ -1,7 +1,7 @@
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/klauspost/compress v1.15.9 h1:wKRjX6JRtDdrE9qwa4b/Cip7ACOshUI4smpCQanqjSY=
github.com/klauspost/compress v1.15.9/go.mod h1:PhcZ0MbTNciWF3rruxRgKxI5NkcHHrHUDtV4Yw2GlzU=
github.com/klauspost/compress v1.18.7 h1:aUyZsS4kH3QTKurYhAOwAHxllVPnOthb3vPfnF1Ehjw=
github.com/klauspost/compress v1.18.7/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
github.com/pierrec/lz4/v4 v4.1.15 h1:MO0/ucJhngq7299dKLwIMtgTfbkoSPF6AoMYDd8Q4q0=
github.com/pierrec/lz4/v4 v4.1.15/go.mod h1:gZWDp/Ze/IJXGXf23ltt2EXimqmTUXEy0GFuRQyBid4=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
+2 -2
View File
@@ -1,10 +1,10 @@
module simple-consumer
go 1.21
go 1.24
require github.com/segmentio/kafka-go v0.4.47
require (
github.com/klauspost/compress v1.17.0 // indirect
github.com/klauspost/compress v1.18.7 // indirect
github.com/pierrec/lz4/v4 v4.1.15 // indirect
)
+2 -2
View File
@@ -2,8 +2,8 @@ github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSs
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/klauspost/compress v1.15.9/go.mod h1:PhcZ0MbTNciWF3rruxRgKxI5NkcHHrHUDtV4Yw2GlzU=
github.com/klauspost/compress v1.17.0 h1:Rnbp4K9EjcDuVuHtd0dgA4qNuv9yKDYKK1ulpJwgrqM=
github.com/klauspost/compress v1.17.0/go.mod h1:ntbaceVETuRiXiv4DpjP66DpAtAGkEQskQzEyD//IeE=
github.com/klauspost/compress v1.18.7 h1:aUyZsS4kH3QTKurYhAOwAHxllVPnOthb3vPfnF1Ehjw=
github.com/klauspost/compress v1.18.7/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
github.com/pierrec/lz4/v4 v4.1.15 h1:MO0/ucJhngq7299dKLwIMtgTfbkoSPF6AoMYDd8Q4q0=
github.com/pierrec/lz4/v4 v4.1.15/go.mod h1:gZWDp/Ze/IJXGXf23ltt2EXimqmTUXEy0GFuRQyBid4=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
+49 -32
View File
@@ -12,41 +12,51 @@ import (
"time"
"google.golang.org/grpc"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/credentials/insecure"
"google.golang.org/grpc/status"
"github.com/seaweedfs/seaweedfs/weed/pb"
"github.com/seaweedfs/seaweedfs/weed/pb/master_pb"
)
// The very first write against a fresh cluster must succeed in one shot: the
// assign that triggers volume growth waits for it instead of shedding itself
// with "volume growth in progress". No client retries — a single request per
// assign, like a plain HTTP client.
// The very first write against a fresh cluster must succeed: the assign that
// triggers volume growth waits for it, and when growth outlasts that wait the
// master sheds with a retryable 503/Retry-After. A normal client honors the
// hint and retries, so the test does the same within a deadline.
func TestColdStartFirstWrite(t *testing.T) {
c := StartCluster(t, 3)
// Each subtest uses its own replication so its volume layout starts cold.
t.Run("http_assign_and_write", func(t *testing.T) {
client := &http.Client{Timeout: 15 * time.Second}
resp, err := client.Get(c.MasterURL() + "/dir/assign?replication=002")
if err != nil {
t.Fatalf("dir/assign: %v", err)
}
body, _ := io.ReadAll(resp.Body)
resp.Body.Close()
if resp.StatusCode != http.StatusOK {
t.Fatalf("dir/assign returned %d: %s", resp.StatusCode, body)
}
var assign struct {
Fid string `json:"fid"`
Url string `json:"url"`
Error string `json:"error"`
}
if err := json.Unmarshal(body, &assign); err != nil {
t.Fatalf("parse assign response %q: %v", body, err)
}
if assign.Error != "" || assign.Fid == "" {
t.Fatalf("assign failed: %s", body)
deadline := time.Now().Add(60 * time.Second)
for {
resp, err := client.Get(c.MasterURL() + "/dir/assign?replication=002")
if err != nil {
t.Fatalf("dir/assign: %v", err)
}
body, _ := io.ReadAll(resp.Body)
resp.Body.Close()
if resp.StatusCode == http.StatusServiceUnavailable && time.Now().Before(deadline) {
time.Sleep(time.Second)
continue
}
if resp.StatusCode != http.StatusOK {
t.Fatalf("dir/assign returned %d: %s", resp.StatusCode, body)
}
if err := json.Unmarshal(body, &assign); err != nil {
t.Fatalf("parse assign response %q: %v", body, err)
}
if assign.Error != "" || assign.Fid == "" {
t.Fatalf("assign failed: %s", body)
}
break
}
content := []byte("cold start write")
@@ -57,21 +67,28 @@ func TestColdStartFirstWrite(t *testing.T) {
t.Run("grpc_assign", func(t *testing.T) {
grpcDialOption := grpc.WithTransportCredentials(insecure.NewCredentials())
err := pb.WithMasterClient(context.Background(), false, pb.ServerAddress(c.MasterAddress()), grpcDialOption, false,
func(client master_pb.SeaweedClient) error {
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
defer cancel()
resp, err := client.Assign(ctx, &master_pb.AssignRequest{Count: 1, Replication: "001"})
if err != nil {
return err
}
if resp.Error != "" || resp.Fid == "" {
return fmt.Errorf("assign failed: %+v", resp)
}
return nil
})
var err error
for deadline := time.Now().Add(60 * time.Second); ; {
err = pb.WithMasterClient(context.Background(), false, pb.ServerAddress(c.MasterAddress()), grpcDialOption, false,
func(client master_pb.SeaweedClient) error {
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
defer cancel()
resp, err := client.Assign(ctx, &master_pb.AssignRequest{Count: 1, Replication: "001"})
if err != nil {
return err
}
if resp.Error != "" || resp.Fid == "" {
return fmt.Errorf("assign failed: %+v", resp)
}
return nil
})
if err == nil || status.Code(err) != codes.ResourceExhausted || time.Now().After(deadline) {
break
}
time.Sleep(time.Second)
}
if err != nil {
t.Fatalf("single grpc assign on cold cluster: %v", err)
t.Fatalf("grpc assign on cold cluster: %v", err)
}
})
}
+23 -1
View File
@@ -290,9 +290,31 @@ func TestLeaderConsistencyAcrossNodes(t *testing.T) {
assertTopologyIdConsistent(t, mc, topologyId)
}
// assertTopologyIdConsistent verifies that all running nodes report the expected TopologyId.
// assertTopologyIdConsistent waits until all running nodes report the expected
// TopologyId. A restarted node only discovers it after the leader has admitted
// it back into raft — /dir/status is proxied to the leader, so a node that has
// not yet learned the leader reports its own empty id. That admission trails
// HTTP readiness by several seconds, so a one-shot check races the propagation.
func assertTopologyIdConsistent(t *testing.T, mc *MasterCluster, expectedId string) {
t.Helper()
deadline := time.Now().Add(waitTimeout)
for time.Now().Before(deadline) {
consistent := true
for i := range 3 {
if !mc.IsNodeRunning(i) {
continue
}
if id, err := mc.GetTopologyId(i); err != nil || id != expectedId {
consistent = false
break
}
}
if consistent {
return
}
time.Sleep(waitTick)
}
// Timed out: report what each node actually sees.
for i := range 3 {
if !mc.IsNodeRunning(i) {
continue
+1 -1
View File
@@ -12,7 +12,7 @@ FILER_PORT := 8888
# Pin the filer gRPC port off the FILER_PORT+10000 convention so any
# code path that assumes the offset breaks here, not in production.
FILER_GRPC_PORT := 18890
ADMIN_PORT := 23646
ADMIN_PORT := 12535
ACCESS_KEY ?= some_access_key1
SECRET_KEY ?= some_secret_key1
TEST_TIMEOUT := 10m
+1 -1
View File
@@ -41,7 +41,7 @@ const (
defaultS3Endpoint = "http://localhost:8333"
defaultS3GrpcEndpoint = "localhost:18333"
defaultMasterEndpt = "http://localhost:9333"
defaultAdminEndpoint = "http://localhost:23646"
defaultAdminEndpoint = "http://localhost:12535"
// Pinned off the FILER_PORT+10000 convention; see Makefile.
defaultFilerGRPC = "localhost:18890"
bucketLifecycleXMLKey = "s3-bucket-lifecycle-configuration-xml"
+2 -2
View File
@@ -114,7 +114,7 @@ func TestS3Integration(t *testing.T) {
// findAvailablePort finds an available port by binding to port 0
// startMiniCluster starts a weed mini instance directly without exec.
// Extra flags (e.g. "-s3.allowDeleteBucketNotEmpty=false") can be appended via extraArgs.
// Extra flags can be appended via extraArgs.
func startMiniCluster(t *testing.T, extraArgs ...string) (*TestCluster, error) {
// Allocate non-colliding ports (including gRPC offsets) for weed mini.
ports := testutil.MustFreeMiniPorts(t, []string{"Master", "Volume", "Filer", "S3"})
@@ -872,7 +872,7 @@ func TestS3DeleteBucketNotEmpty(t *testing.T) {
t.Skip("Skipping integration test in short mode")
}
cluster, err := startMiniCluster(t, "-s3.allowDeleteBucketNotEmpty=false")
cluster, err := startMiniCluster(t)
require.NoError(t, err)
defer cluster.Stop()
+2 -2
View File
@@ -705,10 +705,10 @@ func uniqueName(prefix string) string {
func startMiniCluster(t *testing.T) (*TestCluster, error) {
// Allocate two extra ports for admin HTTP + gRPC. Without explicit
// values mini falls back to 23646/33646, which collide across the
// values mini falls back to 12535/22535, which collide across the
// sequential subtests in this package: each subtest spins up a fresh
// mini in a goroutine, but the previous mini's admin goroutine is
// still binding 23646 — so the next test's admin can never become
// still binding 12535 — so the next test's admin can never become
// ready and mini.go fatals on its readiness check.
ports := testutil.MustAllocatePorts(t, 10)
masterPort, masterGrpcPort := ports[0], ports[1]
+2 -2
View File
@@ -228,8 +228,8 @@ test-with-server: start-server
# Test with different configurations
test-versioning-with-configs: check-deps
@echo "Testing with different S3 configurations..."
@echo "Testing with empty folder allowed..."
@$(WEED_BINARY) mini -s3.port=$(S3_PORT) > weed-test-config1.log 2>&1 & echo $$! > weed-config1.pid
@echo "Testing with delete bucket not empty allowed..."
@$(WEED_BINARY) mini -s3.port=$(S3_PORT) -s3.allowDeleteBucketNotEmpty=true > weed-test-config1.log 2>&1 & echo $$! > weed-config1.pid
@sleep 5
@go test -v -timeout=5m -run "TestVersioningBasicWorkflow" . || true
@if [ -f weed-config1.pid ]; then kill -TERM $$(cat weed-config1.pid) 2>/dev/null || true; rm -f weed-config1.pid; fi
+4 -3
View File
@@ -3,7 +3,7 @@
An integration test for [OLake](https://github.com/datazip-inc/olake) against
SeaweedFS's Iceberg REST Catalog, in the same shape as `catalog_clickhouse`.
## Why OLake, given we already test five engines
## Why OLake, given we already test other engines
Two things here are covered by nothing else in this directory.
@@ -73,8 +73,9 @@ Nothing in the destination config is SeaweedFS-specific:
```
`catalog_type` is the generic `rest`, auth is the standard OAuth2
client-credentials flow, and `s3_path_style` does not even need setting —
OLake turns it on by itself whenever `s3_endpoint` is non-empty. As of
client-credentials flow, and the test config sets `s3_path_style` explicitly;
OLake also defaults to path style whenever `s3_endpoint` is non-empty, so
either form works. As of
OLake v0.10.1 this works with no change on either side.
## Running it
@@ -376,8 +376,8 @@ func (env *TestEnvironment) execSQL(t *testing.T, sqlText string) {
// writeOLakeConfigs writes the source and destination configs. The destination
// is the point of this test: nothing in it is SeaweedFS-specific. catalog_type
// is the generic "rest", auth is the standard OAuth2 client-credentials flow,
// and path-style access is not even set here because OLake turns it on by
// itself whenever s3_endpoint is non-empty.
// and path-style access is set explicitly here; OLake also turns it on by
// itself whenever s3_endpoint is non-empty, so either form works.
func (env *TestEnvironment) writeOLakeConfigs(t *testing.T) {
t.Helper()
@@ -0,0 +1,139 @@
package s3tables
import (
"bytes"
"context"
"errors"
"fmt"
"io"
"testing"
"time"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/credentials"
"github.com/aws/aws-sdk-go-v2/service/s3"
"github.com/aws/smithy-go"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func newObjectClient(t *testing.T) *s3.Client {
t.Helper()
cfg, err := config.LoadDefaultConfig(context.Background(),
config.WithRegion(testRegion),
config.WithCredentialsProvider(credentials.NewStaticCredentialsProvider(testAccessKey, testSecretKey, "")),
)
require.NoError(t, err)
return s3.NewFromConfig(cfg, func(o *s3.Options) {
o.BaseEndpoint = aws.String(sharedCluster.s3Endpoint)
o.UsePathStyle = true
})
}
func apiErrorCode(err error) string {
var apiErr smithy.APIError
if errors.As(err, &apiErr) {
return apiErr.ErrorCode()
}
return ""
}
// The external-volume verification object round-trips at a table bucket's
// root the way the check uses it: PUT, HEAD, GET, list, DELETE. Any other key
// at the root is still refused.
func TestTableBucketExternalVolumeVerificationRoundTrip(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
ctx := context.Background()
tables := NewS3TablesClient(sharedCluster.s3Endpoint, testRegion, testAccessKey, testSecretKey)
bucket := "verify-" + randomString(8)
created, err := tables.CreateTableBucket(bucket, nil)
require.NoError(t, err)
defer tables.DeleteTableBucket(created.ARN)
client := newObjectClient(t)
body := []byte("verification payload!") // 21 bytes
key := fmt.Sprintf("verify_%d_%d", time.Now().UnixMilli(), 1081846483)
// the control: an ordinary key at the root is refused
_, err = client.PutObject(ctx, &s3.PutObjectInput{Bucket: aws.String(bucket), Key: aws.String("notes.parquet"), Body: bytes.NewReader(body)})
require.Error(t, err)
assert.Equal(t, "AccessDenied", apiErrorCode(err))
_, err = client.PutObject(ctx, &s3.PutObjectInput{Bucket: aws.String(bucket), Key: aws.String(key), Body: bytes.NewReader(body), ContentLength: aws.Int64(int64(len(body)))})
require.NoError(t, err)
head, err := client.HeadObject(ctx, &s3.HeadObjectInput{Bucket: aws.String(bucket), Key: aws.String(key)})
require.NoError(t, err)
assert.Equal(t, int64(len(body)), aws.ToInt64(head.ContentLength))
got, err := client.GetObject(ctx, &s3.GetObjectInput{Bucket: aws.String(bucket), Key: aws.String(key)})
require.NoError(t, err)
data, err := io.ReadAll(got.Body)
got.Body.Close()
require.NoError(t, err)
assert.Equal(t, body, data)
list, err := client.ListObjectsV2(ctx, &s3.ListObjectsV2Input{Bucket: aws.String(bucket), Prefix: aws.String("verify_")})
require.NoError(t, err)
var keys []string
for _, o := range list.Contents {
keys = append(keys, aws.ToString(o.Key))
}
assert.Equal(t, []string{key}, keys)
_, err = client.DeleteObject(ctx, &s3.DeleteObjectInput{Bucket: aws.String(bucket), Key: aws.String(key)})
require.NoError(t, err)
_, err = client.HeadObject(ctx, &s3.HeadObjectInput{Bucket: aws.String(bucket), Key: aws.String(key)})
require.Error(t, err, "the verification object must be gone after DELETE")
}
// A namespace can carry a name of the verification key's shape. The key
// then names the namespace's directory: a PUT or DELETE through it must not
// replace or remove the namespace.
func TestTableBucketVerificationKeyLeavesNamespaceAlone(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration test in short mode")
}
ctx := context.Background()
tables := NewS3TablesClient(sharedCluster.s3Endpoint, testRegion, testAccessKey, testSecretKey)
bucket := "verify-ns-" + randomString(8)
created, err := tables.CreateTableBucket(bucket, nil)
require.NoError(t, err)
defer tables.DeleteTableBucket(created.ARN)
client := newObjectClient(t)
body := []byte("verification payload!")
listed := func(name string) bool {
resp, err := tables.ListNamespaces(created.ARN, "", "", 0)
require.NoError(t, err)
for _, ns := range resp.Namespaces {
if len(ns.Namespace) == 1 && ns.Namespace[0] == name {
return true
}
}
return false
}
for i, op := range []string{"PUT", "DELETE"} {
t.Run(op, func(t *testing.T) {
name := fmt.Sprintf("verify_%d_%d", time.Now().UnixMilli(), 7+i)
_, err := tables.CreateNamespace(created.ARN, []string{name})
require.NoError(t, err)
defer tables.DeleteNamespace(created.ARN, []string{name})
require.True(t, listed(name), "setup: the namespace is listed")
if op == "PUT" {
_, err = client.PutObject(ctx, &s3.PutObjectInput{Bucket: aws.String(bucket), Key: aws.String(name), Body: bytes.NewReader(body), ContentLength: aws.Int64(int64(len(body)))})
assert.Error(t, err, "a PUT over a namespace must be refused")
} else {
_, err = client.DeleteObject(ctx, &s3.DeleteObjectInput{Bucket: aws.String(bucket), Key: aws.String(name)})
assert.Error(t, err, "a DELETE of a namespace must be refused")
}
_, err = tables.GetNamespace(created.ARN, []string{name})
assert.NoError(t, err, "the namespace survives the %s", op)
assert.True(t, listed(name), "the namespace is still listed after the %s", op)
})
}
}
+27
View File
@@ -3,6 +3,7 @@ package sftp
import (
"bytes"
"io"
"os"
"path"
"testing"
@@ -602,6 +603,32 @@ func TestFileContent(t *testing.T) {
sftpClient.Remove("/binary.bin")
})
t.Run("WriteAtOffsetKeepsExistingBytes", func(t *testing.T) {
file, err := sftpClient.Create("/resume.txt")
require.NoError(t, err)
_, err = file.Write([]byte("hello world"))
require.NoError(t, err)
require.NoError(t, file.Close())
// Reopen without O_TRUNC and write past the start, as a resumed upload does
file, err = sftpClient.OpenFile("/resume.txt", os.O_WRONLY)
require.NoError(t, err)
_, err = file.WriteAt([]byte("WORLD!"), 6)
require.NoError(t, err)
require.NoError(t, file.Close())
readFile, err := sftpClient.Open("/resume.txt")
require.NoError(t, err)
content, err := io.ReadAll(readFile)
require.NoError(t, err)
readFile.Close()
require.Equal(t, "hello WORLD!", string(content))
// Clean up
sftpClient.Remove("/resume.txt")
})
t.Run("EmptyFile", func(t *testing.T) {
file, err := sftpClient.Create("/empty.txt")
require.NoError(t, err)
+1 -1
View File
@@ -40,7 +40,7 @@ var miniDefaultPorts = []int{
8181, // s3.port.iceberg
9101, // s3.port.lance
7333, // webdav.port
23646, // admin.port
12535, // admin.port
}
func reservedMiniPorts() map[int]bool {
+1 -1
View File
@@ -4,7 +4,7 @@ import "testing"
// AllocateMiniPorts must never hand out a port that weed mini will reserve
// for one of its default services (or that default's gRPC offset). A real
// failure: Filer was given 33646 (Admin default 23646 + GrpcPortOffset),
// failure: Filer was given 22535 (Admin default 12535 + GrpcPortOffset),
// which mini then refused as "reserved for gRPC calculation".
func TestAllocateMiniPortsAvoidsMiniDefaults(t *testing.T) {
reserved := reservedMiniPorts()
@@ -16,6 +16,18 @@ import (
func AllocateVolume(t testing.TB, client volume_server_pb.VolumeServerClient, volumeID uint32, collection string) {
t.Helper()
allocateVolume(t, client, volumeID, collection, "")
}
// AllocateVolumeWithTTL allocates a volume whose needles expire after ttl
// (e.g. "10m").
func AllocateVolumeWithTTL(t testing.TB, client volume_server_pb.VolumeServerClient, volumeID uint32, collection string, ttl string) {
t.Helper()
allocateVolume(t, client, volumeID, collection, ttl)
}
func allocateVolume(t testing.TB, client volume_server_pb.VolumeServerClient, volumeID uint32, collection string, ttl string) {
t.Helper()
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
@@ -24,6 +36,7 @@ func AllocateVolume(t testing.TB, client volume_server_pb.VolumeServerClient, vo
VolumeId: volumeID,
Collection: collection,
Replication: "000",
Ttl: ttl,
Version: uint32(needle.GetCurrentVersion()),
})
if err != nil {
+10
View File
@@ -14,6 +14,7 @@ import (
"sync"
"time"
"github.com/google/uuid"
"github.com/seaweedfs/seaweedfs/weed/admin/maintenance"
adminplugin "github.com/seaweedfs/seaweedfs/weed/admin/plugin"
"github.com/seaweedfs/seaweedfs/weed/cluster"
@@ -1155,6 +1156,7 @@ func (s *AdminServer) SetBucketLifecycle(bucketName string, rules []BucketLifecy
bucketPath := filerConfig.BucketsPath + "/" + bucketName
resp, err := client.ObjectTransaction(context.Background(), &filer_pb.ObjectTransactionRequest{
LockKey: bucketPath,
RequestId: uuid.NewString(),
RouteKey: s3_constants.ObjectWriteRouteKeyPrefix + bucketPath,
Mutations: []*filer_pb.ObjectMutation{bucketLifecycleMutation(filerConfig.BucketsPath, bucketName, lifecycleXML)},
})
@@ -1640,6 +1642,14 @@ func (s *AdminServer) WorkerGrpcMTLSEnabled() bool {
return s.workerGrpcServer != nil && s.workerGrpcServer.mtlsEnabled
}
// ServeWorkerGrpcOn serves the worker gRPC API on an additional already-bound
// listener, e.g. a legacy compat port kept for backward compatibility.
func (s *AdminServer) ServeWorkerGrpcOn(listener net.Listener) {
if s.workerGrpcServer != nil {
s.workerGrpcServer.ServeOn(listener)
}
}
// StopWorkerGrpcServer stops the worker gRPC server
func (s *AdminServer) StopWorkerGrpcServer() error {
if s.workerGrpcServer != nil {
+2
View File
@@ -6,6 +6,7 @@ import (
"errors"
"fmt"
"github.com/google/uuid"
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
"github.com/seaweedfs/seaweedfs/weed/s3api"
"github.com/seaweedfs/seaweedfs/weed/s3api/policy_engine"
@@ -126,6 +127,7 @@ func (s *AdminServer) writeBucketPolicy(bucketName string, policyJSON []byte) er
bucketPath := filerConfig.BucketsPath + "/" + bucketName
resp, err := client.ObjectTransaction(context.Background(), &filer_pb.ObjectTransactionRequest{
LockKey: bucketPath,
RequestId: uuid.NewString(),
RouteKey: s3_constants.ObjectWriteRouteKeyPrefix + bucketPath,
Mutations: []*filer_pb.ObjectMutation{bucketPolicyMutation(filerConfig.BucketsPath, bucketName, policyJSON)},
})
+15
View File
@@ -133,6 +133,17 @@ func (s *WorkerGrpcServer) StartWithTLS(bindIp string, port int, listener net.Li
return nil
}
// ServeOn serves the worker gRPC API on an additional already-bound listener,
// e.g. a legacy compat port kept alongside the primary one. grpc.Server tracks
// every listener passed to Serve, so Stop closes it together with the primary.
func (s *WorkerGrpcServer) ServeOn(listener net.Listener) {
go func() {
if err := s.grpcServer.Serve(listener); err != nil && s.running {
glog.Warningf("Worker gRPC compat listener on %s stopped: %v", listener.Addr(), err)
}
}()
}
// ListenPort returns the currently bound worker gRPC listen port.
func (s *WorkerGrpcServer) ListenPort() int {
if s == nil || s.listener == nil {
@@ -188,6 +199,10 @@ func (s *WorkerGrpcServer) Stop() error {
func (s *WorkerGrpcServer) WorkerStream(stream worker_pb.WorkerService_WorkerStreamServer) error {
ctx := stream.Context()
if err := security.CheckGrpcAdminPeer(ctx, security.SigningKey(util.GetViper().GetString("jwt.filer_signing.key"))); err != nil {
return err
}
// get client address
address := findClientAddress(ctx)
+3 -3
View File
@@ -180,9 +180,9 @@ Phase 4 status (starter)
Run example:
- Start admin: `weed admin -master=localhost:9333`
- Start worker: `weed worker -admin=localhost:23646`
- Optional explicit job type: `weed worker -admin=localhost:23646 -jobType=vacuum`
- Optional stable worker ID persistence: `weed worker -admin=localhost:23646 -workingDir=/var/lib/seaweedfs-plugin`
- Start worker: `weed worker -admin=localhost:12535`
- Optional explicit job type: `weed worker -admin=localhost:12535 -jobType=vacuum`
- Optional stable worker ID persistence: `weed worker -admin=localhost:12535 -workingDir=/var/lib/seaweedfs-plugin`
Phase 5
- Migrate remaining job types and deprecate old mechanism.
+66
View File
@@ -0,0 +1,66 @@
package plugin
import (
"strings"
"github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb"
"github.com/seaweedfs/seaweedfs/weed/pb/worker_pb"
pluginworker "github.com/seaweedfs/seaweedfs/weed/plugin/worker"
"github.com/seaweedfs/seaweedfs/weed/util"
"google.golang.org/protobuf/proto"
)
// JobReplicaHosts are the volume-replica hosts a job will read from.
// Used so executor pick prefers a worker on one of those machines.
// Sources with ShardIds are EC-cleanup rows, not encode replicas.
func JobReplicaHosts(job *plugin_pb.JobSpec) []string {
if job == nil {
return nil
}
var addrs []string
addrs = append(addrs, pluginworker.ReadStringListConfig(job.Parameters, "replica_servers")...)
if src := strings.TrimSpace(pluginworker.ReadStringConfig(job.Parameters, "source_server", "")); src != "" {
addrs = append(addrs, src)
}
if payload := pluginworker.ReadBytesConfig(job.Parameters, "task_params_pb"); len(payload) > 0 {
params := &worker_pb.TaskParams{}
if err := proto.Unmarshal(payload, params); err == nil {
for _, s := range params.GetSources() {
if s == nil || s.Node == "" || len(s.ShardIds) > 0 {
continue
}
addrs = append(addrs, s.Node)
}
}
}
return util.HostKeysFromAddresses(addrs)
}
// partitionByPreferredHost splits candidates into (host matches, others).
// No preferred hosts, or no matches, returns (nil, all) so callers keep
// the existing slots+round-robin path.
func partitionByPreferredHost(candidates []*WorkerSession, preferredHosts []string) (pref, rest []*WorkerSession) {
want := util.HostKeysFromAddresses(preferredHosts)
if len(want) == 0 || len(candidates) == 0 {
return nil, candidates
}
set := make(map[string]struct{}, len(want))
for _, h := range want {
set[h] = struct{}{}
}
for _, c := range candidates {
if c == nil {
continue
}
if _, ok := set[util.HostKey(c.Address)]; ok {
pref = append(pref, c)
continue
}
rest = append(rest, c)
}
if len(pref) == 0 {
return nil, candidates
}
return pref, rest
}
+7 -1
View File
@@ -14,6 +14,8 @@ import (
"github.com/seaweedfs/seaweedfs/weed/glog"
"github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb"
"github.com/seaweedfs/seaweedfs/weed/security"
"github.com/seaweedfs/seaweedfs/weed/util"
"google.golang.org/protobuf/proto"
"google.golang.org/protobuf/types/known/timestamppb"
)
@@ -267,6 +269,10 @@ func (r *Plugin) Shutdown() {
}
func (r *Plugin) WorkerStream(stream plugin_pb.PluginControlService_WorkerStreamServer) error {
if err := security.CheckGrpcAdminPeer(stream.Context(), security.SigningKey(util.GetViper().GetString("jwt.filer_signing.key"))); err != nil {
return err
}
first, err := stream.Recv()
if err != nil {
if errors.Is(err, io.EOF) {
@@ -609,7 +615,7 @@ func (r *Plugin) ExecuteJob(
releaseGate := r.acquireDetectionExecutionGate(jobType, true)
defer releaseGate()
executor, err := r.registry.PickExecutor(jobType)
executor, err := r.registry.PickExecutor(jobType, JobReplicaHosts(job)...)
if err != nil {
return nil, err
}
+31
View File
@@ -887,6 +887,37 @@ func (r *Plugin) trackExecutionQueued(job *plugin_pb.JobSpec) {
})
}
// bindQueuedJobWorker records the home worker while the job stays pending/queued.
func (r *Plugin) bindQueuedJobWorker(job *plugin_pb.JobSpec, workerID string) {
if job == nil || strings.TrimSpace(job.JobId) == "" || strings.TrimSpace(workerID) == "" {
return
}
now := time.Now().UTC()
r.jobsMu.Lock()
tracked := r.jobs[job.JobId]
if tracked == nil {
r.jobsMu.Unlock()
return
}
if isTerminalTrackedJobState(tracked.State) {
r.jobsMu.Unlock()
return
}
tracked.WorkerID = workerID
tracked.Message = "queued on worker " + workerID
tracked.UpdatedAt = timeToPtr(now)
snap := cloneTrackedJob(*tracked)
r.dirtyJobs = true
r.jobsMu.Unlock()
r.persistJobDetailSnapshot(job.JobId, func(detail *TrackedJob) {
detail.WorkerID = snap.WorkerID
detail.Message = snap.Message
detail.UpdatedAt = snap.UpdatedAt
})
}
func (r *Plugin) cancelQueuedJob(job *plugin_pb.JobSpec, cause error) {
reason := "job canceled"
if cause != nil {
+206 -125
View File
@@ -4,6 +4,7 @@ import (
"context"
"errors"
"fmt"
"sort"
"strings"
"sync"
"sync/atomic"
@@ -28,12 +29,18 @@ const (
defaultScheduledMaxResults int32 = 1000
defaultScheduledExecutionConcurrency = 1
defaultScheduledPerWorkerConcurrency = 1
maxScheduledExecutionConcurrency = 128
defaultScheduledRetryBackoff = 5 * time.Second
defaultClusterContextTimeout = 10 * time.Second
defaultWaitingBacklogFloor = 8
defaultWaitingBacklogMultiplier = 4
maxEstimatedRuntimeCap = 8 * time.Hour
// Hold a job on replica-host workers until their reservation load
// reaches this; 0 disables the pin and spills immediately.
defaultLocalQueuePin = 5
// Wait this long after a home worker disappears before stealing
// its queue; 0 steals immediately. Covers a reconnect blink.
defaultOrphanRehomeGrace = 30 * time.Second
maxScheduledExecutionConcurrency = 128
defaultScheduledRetryBackoff = 5 * time.Second
defaultClusterContextTimeout = 10 * time.Second
defaultWaitingBacklogFloor = 8
defaultWaitingBacklogMultiplier = 4
maxEstimatedRuntimeCap = 8 * time.Hour
// How long started jobs may drain past the run window close.
scheduledExecutionDrainGrace = 30 * time.Minute
)
@@ -49,6 +56,12 @@ type schedulerPolicy struct {
PerWorkerConcurrency int
RetryLimit int
ExecutorReserveBackoff time.Duration
// LocalQueuePin is K: do not spill off replica-host workers while
// min(load(pref)) < K. Zero is the old immediate-spill path.
LocalQueuePin int
// OrphanRehomeGrace is how long a missing home's queue is held
// before steal-without-K. Zero steals as soon as the worker is gone.
OrphanRehomeGrace time.Duration
}
// laneSchedulerLoop is the main scheduling goroutine for a single lane.
@@ -466,6 +479,8 @@ func (r *Plugin) loadSchedulerPolicy(jobType string) (schedulerPolicy, bool, err
PerWorkerConcurrency: int(adminRuntime.PerWorkerExecutionConcurrency),
RetryLimit: int(adminRuntime.RetryLimit),
ExecutorReserveBackoff: 200 * time.Millisecond,
LocalQueuePin: defaultLocalQueuePin,
OrphanRehomeGrace: defaultOrphanRehomeGrace,
}
if policy.DetectionInterval < r.schedulerTick {
@@ -496,6 +511,9 @@ func (r *Plugin) loadSchedulerPolicy(jobType string) (schedulerPolicy, bool, err
if policy.ExecutionTimeout < defaultScheduledExecutionTimeout {
policy.ExecutionTimeout = defaultScheduledExecutionTimeout
}
if policy.LocalQueuePin < 0 {
policy.LocalQueuePin = 0
}
return policy, true, nil
}
@@ -813,19 +831,22 @@ func (r *Plugin) dispatchScheduledProposals(
// whole dispatch lets manual shell operations interleave between jobs.
jobsRequireLock := LaneRequiresLock(JobTypeLane(jobType))
jobQueue := make(chan *plugin_pb.JobSpec, len(proposals))
jobs := make([]*plugin_pb.JobSpec, 0, len(proposals))
for index, proposal := range proposals {
job := buildScheduledJobSpec(jobType, proposal, index)
r.trackExecutionQueued(job)
select {
case <-r.shutdownCh:
close(jobQueue)
return 0, 0, 0
for _, job := range jobs {
r.cancelQueuedJob(job, errSchedulerShutdown)
}
return 0, 0, len(jobs)
default:
jobQueue <- job
}
job := buildScheduledJobSpec(jobType, proposal, index)
r.trackExecutionQueued(job)
jobs = append(jobs, job)
}
close(jobQueue)
queues := r.assignJobsToWorkerQueues(jobType, jobs, policy)
var wg sync.WaitGroup
var statsMu sync.Mutex
@@ -842,130 +863,110 @@ func (r *Plugin) dispatchScheduledProposals(
wg.Add(1)
go func() {
defer wg.Done()
jobLoop:
for job := range jobQueue {
for {
select {
case <-r.shutdownCh:
return
default:
}
if ctx.Err() != nil {
r.cancelQueuedJob(job, ctx.Err())
statsMu.Lock()
canceledCount++
statsMu.Unlock()
return
}
if queues.isEmpty() {
return
}
item, targetID := r.takeQueuedJob(queues, jobType, policy)
if item == nil {
if queues.isEmpty() {
return
}
if !waitForShutdownOrTimerWithContext(r.shutdownCh, ctx, policy.ExecutorReserveBackoff) {
return
}
continue
}
if ctx.Err() != nil {
queues.pushFront(item)
return
}
executor, release, reserveErr := r.reserveExecutorByID(jobType, policy, targetID)
if reserveErr != nil {
queues.pushFront(item)
if ctx.Err() != nil {
return
}
if !waitForShutdownOrTimerWithContext(r.shutdownCh, ctx, policy.ExecutorReserveBackoff) {
return
}
continue
}
for {
select {
case <-r.shutdownCh:
return
default:
var releaseJobLock func()
if jobsRequireLock {
var lockErr error
releaseJobLock, lockErr = r.acquireAdminLock(fmt.Sprintf("plugin scheduler %s job %s", jobType, item.job.JobId))
if lockErr != nil {
release()
statsMu.Lock()
errorCount++
statsMu.Unlock()
r.appendActivity(JobActivity{
JobID: item.job.JobId,
JobType: item.job.JobType,
Source: "admin_scheduler",
Message: fmt.Sprintf("scheduled execution lock failed: %v", lockErr),
Stage: "failed",
OccurredAt: timeToPtr(time.Now().UTC()),
})
continue
}
if ctx.Err() != nil {
r.cancelQueuedJob(job, ctx.Err())
releaseJobLock()
release()
queues.pushFront(item)
return
}
}
err := r.executeScheduledJobWithExecutor(ctx, executor, item.job, clusterContext, policy)
if releaseJobLock != nil {
releaseJobLock()
}
release()
if errors.Is(err, errExecutorAtCapacity) {
queues.pushFront(item)
if !waitForShutdownOrTimerWithContext(r.shutdownCh, ctx, policy.ExecutorReserveBackoff) {
return
}
continue
}
if err != nil {
if ctx.Err() != nil || errors.Is(err, context.DeadlineExceeded) || errors.Is(err, context.Canceled) {
r.cancelQueuedJob(item.job, err)
statsMu.Lock()
canceledCount++
statsMu.Unlock()
continue jobLoop
}
executor, release, reserveErr := r.reserveScheduledExecutor(ctx, jobType, policy)
if reserveErr != nil {
if ctx.Err() != nil {
r.cancelQueuedJob(job, ctx.Err())
statsMu.Lock()
canceledCount++
statsMu.Unlock()
continue jobLoop
}
statsMu.Lock()
errorCount++
statsMu.Unlock()
r.appendActivity(JobActivity{
JobType: jobType,
Source: "admin_scheduler",
Message: fmt.Sprintf("scheduled execution reservation failed: %v", reserveErr),
Stage: "failed",
OccurredAt: timeToPtr(time.Now().UTC()),
})
break
}
var releaseJobLock func()
if jobsRequireLock {
var lockErr error
releaseJobLock, lockErr = r.acquireAdminLock(fmt.Sprintf("plugin scheduler %s job %s", jobType, job.JobId))
if lockErr != nil {
release()
statsMu.Lock()
errorCount++
statsMu.Unlock()
r.appendActivity(JobActivity{
JobID: job.JobId,
JobType: job.JobType,
Source: "admin_scheduler",
Message: fmt.Sprintf("scheduled execution lock failed: %v", lockErr),
Stage: "failed",
OccurredAt: timeToPtr(time.Now().UTC()),
})
break
}
// The lock may have been held by an operator for a
// while; re-check the execution window.
if ctx.Err() != nil {
releaseJobLock()
release()
r.cancelQueuedJob(job, ctx.Err())
statsMu.Lock()
canceledCount++
statsMu.Unlock()
continue jobLoop
}
}
err := r.executeScheduledJobWithExecutor(ctx, executor, job, clusterContext, policy)
if releaseJobLock != nil {
releaseJobLock()
}
release()
if errors.Is(err, errExecutorAtCapacity) {
r.trackExecutionQueued(job)
if !waitForShutdownOrTimerWithContext(r.shutdownCh, ctx, policy.ExecutorReserveBackoff) {
return
}
continue
}
if err != nil {
if ctx.Err() != nil || errors.Is(err, context.DeadlineExceeded) || errors.Is(err, context.Canceled) {
r.cancelQueuedJob(job, err)
statsMu.Lock()
canceledCount++
statsMu.Unlock()
continue jobLoop
}
statsMu.Lock()
errorCount++
statsMu.Unlock()
r.appendActivity(JobActivity{
JobID: job.JobId,
JobType: job.JobType,
Source: "admin_scheduler",
Message: fmt.Sprintf("scheduled execution failed: %v", err),
Stage: "failed",
OccurredAt: timeToPtr(time.Now().UTC()),
})
break
}
statsMu.Lock()
successCount++
errorCount++
statsMu.Unlock()
break
r.appendActivity(JobActivity{
JobID: item.job.JobId,
JobType: item.job.JobType,
Source: "admin_scheduler",
Message: fmt.Sprintf("scheduled execution failed: %v", err),
Stage: "failed",
OccurredAt: timeToPtr(time.Now().UTC()),
})
continue
}
statsMu.Lock()
successCount++
statsMu.Unlock()
}
}()
}
@@ -976,8 +977,8 @@ func (r *Plugin) dispatchScheduledProposals(
if drainErr == nil {
drainErr = errSchedulerShutdown
}
for job := range jobQueue {
r.cancelQueuedJob(job, drainErr)
for _, item := range queues.drain() {
r.cancelQueuedJob(item.job, drainErr)
canceledCount++
}
@@ -988,6 +989,7 @@ func (r *Plugin) reserveScheduledExecutor(
ctx context.Context,
jobType string,
policy schedulerPolicy,
preferredHosts []string,
) (*WorkerSession, func(), error) {
if ctx == nil {
ctx = context.Background()
@@ -1015,7 +1017,7 @@ func (r *Plugin) reserveScheduledExecutor(
return nil, nil, fmt.Errorf("timed out waiting for executor capacity for %s", jobType)
}
executors, err := r.registry.ListExecutors(jobType)
executors, err := r.registry.ListExecutors(jobType, preferredHosts...)
if err != nil {
if !waitForShutdownOrTimerWithContext(r.shutdownCh, ctx, policy.ExecutorReserveBackoff) {
if ctx.Err() != nil {
@@ -1026,11 +1028,19 @@ func (r *Plugin) reserveScheduledExecutor(
continue
}
for _, executor := range executors {
release, ok := r.tryReserveExecutorCapacity(executor, jobType, policy)
if !ok {
continue
pref, rest := partitionByPreferredHost(executors, preferredHosts)
lane := JobTypeLane(jobType)
loadOf := func(e *WorkerSession) int { return r.executorReservationLoad(e, lane) }
if len(pref) > 0 {
sortExecutorsByLoad(pref, loadOf)
if executor, release, ok := r.tryReserveFirst(pref, jobType, policy); ok {
return executor, release, nil
}
}
sortExecutorsByLoad(rest, loadOf)
if executor, release, ok := r.tryReserveFirst(rest, jobType, policy); ok {
return executor, release, nil
}
@@ -1051,6 +1061,75 @@ func (r *Plugin) tryReserveExecutorCapacity(
return r.tryReserveExecutorCapacityForLane(executor, jobType, policy, JobTypeLane(jobType))
}
func (r *Plugin) tryReserveFirst(executors []*WorkerSession, jobType string, policy schedulerPolicy) (*WorkerSession, func(), bool) {
for _, executor := range executors {
release, ok := r.tryReserveExecutorCapacity(executor, jobType, policy)
if ok {
return executor, release, true
}
}
return nil, nil, false
}
func (r *Plugin) reserveExecutorByID(jobType string, policy schedulerPolicy, workerID string) (*WorkerSession, func(), error) {
workerID = strings.TrimSpace(workerID)
if workerID == "" {
return nil, nil, fmt.Errorf("executor id is empty")
}
executors, err := r.registry.ListExecutors(jobType)
if err != nil {
return nil, nil, err
}
for _, executor := range executors {
if executor == nil || executor.WorkerID != workerID {
continue
}
release, ok := r.tryReserveExecutorCapacity(executor, jobType, policy)
if !ok {
return nil, nil, errExecutorAtCapacity
}
return executor, release, nil
}
return nil, nil, fmt.Errorf("executor %s is not available for %s", workerID, jobType)
}
func (r *Plugin) executorReservationLoad(executor *WorkerSession, lane SchedulerLane) int {
if executor == nil {
return 0
}
load := 0
if executor.Heartbeat != nil && executor.Heartbeat.ExecutionSlotsUsed > 0 {
load = int(executor.Heartbeat.ExecutionSlotsUsed)
}
workerID := strings.TrimSpace(executor.WorkerID)
if ls := r.lanes[lane]; ls != nil {
ls.execMu.Lock()
load += ls.execRes[workerID]
ls.execMu.Unlock()
return load
}
r.schedulerExecMu.Lock()
load += r.schedulerExecReservations[workerID]
r.schedulerExecMu.Unlock()
return load
}
func sortExecutorsByLoad(executors []*WorkerSession, load func(*WorkerSession) int) {
sort.SliceStable(executors, func(i, j int) bool {
if executors[i] == nil {
return false
}
if executors[j] == nil {
return true
}
li, lj := load(executors[i]), load(executors[j])
if li != lj {
return li < lj
}
return executors[i].WorkerID < executors[j].WorkerID
})
}
// tryReserveExecutorCapacityForLane reserves an execution slot on the
// per-lane reservation pool so that lanes cannot starve each other.
func (r *Plugin) tryReserveExecutorCapacityForLane(
@@ -1577,6 +1656,8 @@ func (r *Plugin) DispatchProposals(
ExecutionTimeout: defaultScheduledExecutionTimeout,
RetryBackoff: defaultScheduledRetryBackoff,
ExecutorReserveBackoff: 200 * time.Millisecond,
LocalQueuePin: defaultLocalQueuePin,
OrphanRehomeGrace: defaultOrphanRehomeGrace,
}
}
+58 -5
View File
@@ -61,6 +61,12 @@ func TestLoadSchedulerPolicyUsesAdminConfig(t *testing.T) {
if policy.JobTypeMaxRuntime != 30*time.Minute {
t.Fatalf("unexpected max runtime: got=%v", policy.JobTypeMaxRuntime)
}
if policy.LocalQueuePin != defaultLocalQueuePin {
t.Fatalf("unexpected local queue pin: got=%d", policy.LocalQueuePin)
}
if policy.OrphanRehomeGrace != defaultOrphanRehomeGrace {
t.Fatalf("unexpected orphan rehome grace: got=%v", policy.OrphanRehomeGrace)
}
}
func TestLoadSchedulerPolicyUsesDescriptorDefaultsWhenConfigMissing(t *testing.T) {
@@ -138,13 +144,13 @@ func TestReserveScheduledExecutorRespectsPerWorkerLimit(t *testing.T) {
ExecutorReserveBackoff: time.Millisecond,
}
executor1, release1, err := pluginSvc.reserveScheduledExecutor(context.Background(), "balance", policy)
executor1, release1, err := pluginSvc.reserveScheduledExecutor(context.Background(), "balance", policy, nil)
if err != nil {
t.Fatalf("reserve executor 1: %v", err)
}
defer release1()
executor2, release2, err := pluginSvc.reserveScheduledExecutor(context.Background(), "balance", policy)
executor2, release2, err := pluginSvc.reserveScheduledExecutor(context.Background(), "balance", policy, nil)
if err != nil {
t.Fatalf("reserve executor 2: %v", err)
}
@@ -266,7 +272,7 @@ func TestReserveScheduledExecutorTimesOutWhenNoExecutor(t *testing.T) {
start := time.Now()
pluginSvc.Shutdown()
_, _, err = pluginSvc.reserveScheduledExecutor(context.Background(), "missing-job-type", policy)
_, _, err = pluginSvc.reserveScheduledExecutor(context.Background(), "missing-job-type", policy, nil)
if err == nil {
t.Fatalf("expected reservation shutdown error")
}
@@ -297,7 +303,7 @@ func TestReserveScheduledExecutorWaitsForWorkerCapacity(t *testing.T) {
ExecutorReserveBackoff: 5 * time.Millisecond,
}
_, release1, err := pluginSvc.reserveScheduledExecutor(context.Background(), "balance", policy)
_, release1, err := pluginSvc.reserveScheduledExecutor(context.Background(), "balance", policy, nil)
if err != nil {
t.Fatalf("reserve executor 1: %v", err)
}
@@ -308,7 +314,7 @@ func TestReserveScheduledExecutorWaitsForWorkerCapacity(t *testing.T) {
}
secondReserveCh := make(chan reserveResult, 1)
go func() {
_, release2, reserveErr := pluginSvc.reserveScheduledExecutor(context.Background(), "balance", policy)
_, release2, reserveErr := pluginSvc.reserveScheduledExecutor(context.Background(), "balance", policy, nil)
if release2 != nil {
release2()
}
@@ -992,3 +998,50 @@ func TestLaneStatus_LifecycleNextDetectionSurvivesDefaultLanePrune(t *testing.T)
status.NextDetectionAt, expected)
}
}
func TestReserveWaterfillsPreferredHosts(t *testing.T) {
t.Parallel()
pluginSvc, err := New(Options{})
if err != nil {
t.Fatalf("New: %v", err)
}
defer pluginSvc.Shutdown()
ids := []string{"worker-a", "worker-b", "worker-c", "worker-d"}
for _, id := range ids {
pluginSvc.registry.UpsertFromHello(&plugin_pb.WorkerHello{
WorkerId: id,
Address: "10.0.0.1:9334",
Capabilities: []*plugin_pb.JobTypeCapability{
{JobType: "erasure_coding", CanExecute: true, MaxExecutionConcurrency: 8},
},
})
}
policy := schedulerPolicy{
PerWorkerConcurrency: 8,
ExecutorReserveBackoff: time.Millisecond,
LocalQueuePin: 5,
}
preferred := []string{"10.0.0.1:8080"}
seen := map[string]int{}
var releases []func()
defer func() {
for _, release := range releases {
release()
}
}()
for i := 0; i < 4; i++ {
exec, release, err := pluginSvc.reserveScheduledExecutor(context.Background(), "erasure_coding", policy, preferred)
if err != nil {
t.Fatalf("reserve %d: %v", i, err)
}
releases = append(releases, release)
seen[exec.WorkerID]++
}
if len(seen) != 4 {
t.Fatalf("want one job per worker, got %v", seen)
}
}
+23 -5
View File
@@ -277,14 +277,24 @@ func (r *Registry) PickDetector(jobType string) (*WorkerSession, error) {
}
// PickExecutor picks one executor worker for a job type.
func (r *Registry) PickExecutor(jobType string) (*WorkerSession, error) {
return r.pickByKind(jobType, false)
// preferredHosts, when set, ranks workers whose Hello Address host matches a
// replica/source host ahead of others (same-machine EC pull). Empty means the
// previous slots+round-robin order.
func (r *Registry) PickExecutor(jobType string, preferredHosts ...string) (*WorkerSession, error) {
list, err := r.ListExecutors(jobType, preferredHosts...)
if err != nil {
return nil, err
}
return list[0], nil
}
// ListExecutors returns sorted executor candidates for one job type.
// Ordering is by most available execution slots, then lexical worker ID.
// The top tie group is rotated round-robin to prevent sticky assignment.
func (r *Registry) ListExecutors(jobType string) ([]*WorkerSession, error) {
// preferredHosts, when any worker matches, are listed first (then the rest);
// rotation applies only inside the preferred group so a local worker is not
// rotated away in favor of a remote one.
func (r *Registry) ListExecutors(jobType string, preferredHosts ...string) ([]*WorkerSession, error) {
r.mu.Lock()
defer r.mu.Unlock()
@@ -293,8 +303,16 @@ func (r *Registry) ListExecutors(jobType string) ([]*WorkerSession, error) {
return nil, fmt.Errorf("no executor worker available for job_type=%s", jobType)
}
sortByKind(candidates, jobType, false)
r.rotateTopCandidatesLocked(candidates, jobType, false)
pref, rest := partitionByPreferredHost(candidates, preferredHosts)
if len(pref) > 0 {
sortByKind(pref, jobType, false)
r.rotateTopCandidatesLocked(pref, jobType, false)
sortByKind(rest, jobType, false)
candidates = append(pref, rest...)
} else {
sortByKind(candidates, jobType, false)
r.rotateTopCandidatesLocked(candidates, jobType, false)
}
out := make([]*WorkerSession, 0, len(candidates))
for _, candidate := range candidates {
+67
View File
@@ -401,3 +401,70 @@ func TestRegistryHasCapableWorkerIgnoresNilCapability(t *testing.T) {
t.Fatalf("expected HasCapableWorker to return false when capability has no detect or execute")
}
}
func TestListExecutorsPrefersMatchingReplicaHost(t *testing.T) {
t.Parallel()
r := NewRegistry()
r.UpsertFromHello(&plugin_pb.WorkerHello{
WorkerId: "worker-remote",
Address: "10.87.0.4:33646",
Capabilities: []*plugin_pb.JobTypeCapability{
{JobType: "erasure_coding", CanExecute: true, MaxExecutionConcurrency: 8},
},
})
r.UpsertFromHello(&plugin_pb.WorkerHello{
WorkerId: "worker-local",
Address: "10.87.0.3:33646",
Capabilities: []*plugin_pb.JobTypeCapability{
{JobType: "erasure_coding", CanExecute: true, MaxExecutionConcurrency: 1},
},
})
r.UpdateHeartbeat("worker-remote", &plugin_pb.WorkerHeartbeat{
WorkerId: "worker-remote", ExecutionSlotsUsed: 0, ExecutionSlotsTotal: 8,
})
r.UpdateHeartbeat("worker-local", &plugin_pb.WorkerHeartbeat{
WorkerId: "worker-local", ExecutionSlotsUsed: 0, ExecutionSlotsTotal: 1,
})
// Without preference the emptier remote worker wins on slots.
plain, err := r.ListExecutors("erasure_coding")
if err != nil {
t.Fatalf("ListExecutors: %v", err)
}
if plain[0].WorkerID != "worker-remote" {
t.Fatalf("without preference want remote first, got %s", plain[0].WorkerID)
}
// Replica on :8080 of the same host as worker-local.
pref, err := r.ListExecutors("erasure_coding", "10.87.0.3:8080")
if err != nil {
t.Fatalf("ListExecutors preferred: %v", err)
}
if pref[0].WorkerID != "worker-local" {
t.Fatalf("preferred host want worker-local first, got %s", pref[0].WorkerID)
}
if pref[1].WorkerID != "worker-remote" {
t.Fatalf("remote should still be fallback, got %s", pref[1].WorkerID)
}
}
func TestJobReplicaHostsFromSourcesAndReplicaServers(t *testing.T) {
t.Parallel()
job := &plugin_pb.JobSpec{
Parameters: map[string]*plugin_pb.ConfigValue{
"source_server": {
Kind: &plugin_pb.ConfigValue_StringValue{StringValue: "10.87.0.1:8080"},
},
"replica_servers": {
Kind: &plugin_pb.ConfigValue_StringList{
StringList: &plugin_pb.StringList{Values: []string{"10.87.0.1:8080", "10.87.0.2:8081"}},
},
},
},
}
got := JobReplicaHosts(job)
if len(got) != 2 {
t.Fatalf("got %v", got)
}
}
Loaded 100 of 521 files, more files were not shown because too many files have changed in this diff. Show more