The owner index read the raw filer entry off BucketConfig, which no
longer carries one. Cache Crtime alongside IdentityId, and pass the
owner id rather than the entry through the ListBuckets visibility
checks so the scan and granted-bucket paths share one implementation.
* s3: paginate ListBuckets with max-buckets, continuation-token, and prefix
ListBuckets buffered every bucket entry into one slice and one XML body,
which falls over with very large bucket counts. Page through the filer
listing instead, cap each response at 10000 buckets like AWS, and honor
max-buckets, prefix, and an opaque keyset continuation-token.
* s3: maintain a bucket owner index under /buckets/.system/owners
Map each bucket owner to its buckets as zero-length entries at
/buckets/.system/owners/<owner>/<bucket>, with Crtime mirroring the
bucket's creation time. The bucket handlers write the index
synchronously, the /buckets metadata subscription reconciles changes
made elsewhere (weed shell, other gateways, direct filer operations),
and a startup backfill indexes pre-existing buckets before writing a
ready marker. Owner names are path-escaped so no identity name can
escape the index directory.
* s3: serve ListBuckets from the bucket owner index
Once the owner index is ready, non-admin identities list their owned
buckets straight from it, merged with any buckets their legacy actions
name explicitly, so ListBuckets costs O(own buckets) instead of a scan
of the global /buckets directory. Admins, identities with a bare List
grant or wildcard action patterns, and policy-authorized identities
whose grants cannot be enumerated keep the paged scan; policy-routed
identities get their owned buckets, matching AWS ListBuckets returning
only the caller's buckets.
* s3: keep dot-prefixed names under /buckets out of bucket surfaces
Dot-prefixed entries (.system) can never be valid bucket names, so
refuse to resolve them as buckets and skip them in the shell bucket
listing, matching what ListBuckets and the admin UI already do.
* test: cover ListBuckets pagination and the owner index end to end
* s3: fail closed on a nil identity when routing ListBuckets
* s3: decide the IAM authorization mechanism in one place
VerifyActionPermission and the ListBuckets owner-index routing each
re-derived the session-token / attached-policy / legacy-actions split;
extract the decision so the two cannot drift.
* s3: heal the owner index on concurrent bucket recreation too
The mkdir-lost-the-race path answers BucketAlreadyOwnedByYou just like
the up-front existence check, so give it the same index repair.
* s3: drop owner-index records for buckets deleted during backfill
A bucket removed between the backfill reading its page and writing the
index record became a permanent phantom in its owner's listing: the
delete's own cleanup ran before the record existed. After indexing each
page, re-list the same name range and remove records whose bucket is
gone; deletes landing after the re-list find the record and remove it
themselves.
* s3: add ContinuationToken and Prefix to the ListBuckets schema
Keep AmazonS3.xsd aligned with the generated ListAllMyBucketsResult so
a regeneration does not drop the pagination fields.
* s3: stop warming every bucket's config at startup
Listing all buckets in BucketRegistry.init() made S3 gateway startup
O(buckets) and pinned every bucket's metadata and config resident, which
does not scale past a few hundred thousand buckets. Both caches already
have lazy miss paths, so load on first access instead and let the
metadata subscription refresh only entries already resident; cold
buckets cost one filer round-trip on their first request.
* s3: bound the per-bucket caches with LRU eviction
The bucket config cache, bucket registry, and their negative caches were
plain maps that only ever grew: the config cache TTL made Get miss but
never evicted the entry, and the not-found sets grew on every probe of a
nonexistent bucket name. Cap all four at 65536 entries with LRU eviction
so a gateway keeps its hot working set and evicted buckets reload from
the filer on next access.
* s3: cache parsed bucket config instead of the full filer entry
Each cached BucketConfig retained the whole bucket entry (extended
attribute map plus raw content bytes) alongside the fields parsed from
it, roughly doubling per-bucket cache cost and keeping data the read
path never looks at. Parse everything up front in
newBucketConfigFromEntry - now also the creator identity, tags,
encryption config, and stored lifecycle XML - and drop the entry.
updateBucketConfig now reads the entry fresh from the filer and diffs
the mapped extended attributes against it, so the patch is computed
against current state instead of a cached copy; the config clone
helpers that existed for that path go away.
* s3: dedup cold bucket-registry loads per bucket
The registry's notFound lock doubled as the load serializer, holding one
global mutex across the filer round-trip so first-touch requests for
different buckets queued behind each other; the cache fill also happened
after the lock was released, so two concurrent misses for the same
bucket could both reach the filer. Replace it with a singleflight per
bucket that fills the cache inside the flight: different buckets load
concurrently, the same bucket loads once.
* util: don't let the activity timeout clobber externally-set conn deadlines
util.Conn extended the connection deadline at the start of every
Read/Write. net/http's server also sets deadlines directly on the same
conn - abortPendingRead sets one in the past to interrupt the pending
background read after each response. The activity extension raced with
and silently overwrote that interrupt, leaving the read blocked (and the
server's conn.serve goroutine stuck in abortPendingRead) until the full
-idleTimeout (default 30s) expired.
Wedged connections count as active, so the volume server's graceful HTTP
drain waited out its whole 30s StopTimeout on shutdown - observed as
weed mini taking ~30s to exit in the FUSE integration tests after any
filer->volume traffic.
Track externally-set deadlines, suspend the activity extension while one
is in force, and serialize deadline updates with a mutex. Activity still
extends both directions at once: a long write-only response must keep
the read deadline alive too, or the server's background read would time
out and cancel the in-flight request.
* util: extend read/write deadlines independently when one side is external
A server-configured WriteTimeout keeps an external write deadline in
force for the whole request, which previously suspended the activity
extension entirely - leaving the read deadline stale from before the
request and letting net/http's background read time out mid-response.
Extend each direction independently instead.
* fix(master): let the growth initiator wait for the growth it triggered
The growth-in-flight shed also fired on the request that initiated the
growth: it sets the pending flag right before the shed check, so a
cold-start assign enqueued growth and immediately failed itself with
"volume growth in progress". With no concurrent assigns around to pick
up the freshly grown volume, a single writer against an empty cluster
never completes a write despite ample free space.
Claim the pending flag with a compare-and-swap so exactly one request
becomes the initiator, triggering growth at most once, and let it wait
for that growth to land. Everyone else still sheds retryably instead of
pinning a goroutine: followers behind an in-flight growth, an initiator
whose growth concluded without yielding a writable volume, and an
initiator whose growth outlives the 10s wait budget, which previously
surfaced a non-retryable error (gRPC Unknown, HTTP 406) even though a
retry would have succeeded moments later.
* fix(master): stop assign waits when the request is cancelled
The assign retry loops slept through client cancellation, keeping a
goroutine spinning for the rest of the 10s budget after the caller had
gone; StreamAssign also ran assigns on a background context detached
from the stream. Wait on the request context and pass the stream
context through.
* topology: drop the unconditional grow-request setter
Growth is only claimed through AddGrowRequestIfAbsent's compare-and-swap
now; keeping the raw Store(true) around invites the check-then-set race
back.
* test: cover cold-start first write with a real cluster
Boot a fresh master plus three empty volume servers and require the very
first assign - HTTP and gRPC, each on a cold volume layout, no client
retries - to complete a write. The assign that triggers volume growth
must wait for it rather than answering "volume growth in progress";
unit tests stub the topology, so only a real cluster exercises the
assign-grow-wait path end to end.
* s3: surface multipart-list store errors instead of masking them
listMultipartUploads masked a filer/metadata-store list error as an empty
200 response, and listObjectParts masked it as NoSuchUpload. A client
resuming a multipart upload (the Docker Registry S3 driver resolves an
in-progress upload via ListMultipartUploads, then ListParts) reads either
as "upload gone" and fails the upload permanently with a non-retryable
error. Return ErrInternalError so a transient store error stays retryable,
keeping ErrNotFound as an empty list / NoSuchUpload respectively.
* s3: return retryable error for transient CopyObject source lookups
Resolving the copy source is reported as InvalidArgument ("Copy Source must
mention the source bucket and key") whenever the lookup returns any error,
including a transient store error. Clients don't retry a 400, so a resumable
blob commit fails permanently. Keep the client error for a missing, invalid,
directory, or delete-marker source; map a transient store error to
ErrInternalError.
* s3: mark versioned lookup miss with the not-found sentinel
recoverLatestVersionWithoutPointer's terminal miss (a .versions directory
with no pointer, no versions, and no null object) returned a plain error,
so errors.Is-based callers could not tell this designed NoSuchKey state
from a store failure and reported it as an internal error.
* s3: reject a delete-marker copy source with NoSuchKey
getLatestObjectVersion returns the delete-marker entry when the latest
version is a delete marker, and the copy handlers copied its empty stub
into the destination. Every other handler detects ExtDeleteMarkerKey and
answers NoSuchKey; do the same for CopyObject and UploadPartCopy.
* s3: align copy-source not-found detection with the grpc status idiom
The lookup path canonicalizes text-form not-found into the sentinel in
filer_pb.LookupEntry and wraps with %w after that, so matching sentinel
text here was unreachable -- and risky, since a store error whose message
merely mentions the sentinel would be downgraded to a terminal 400.
Match the raw grpc NotFound code instead, like the other version-lookup
sites, and give transient filer errors the 503 that the upload-entry
lookup in this file already returns.
* s3: keep source-bucket versioning lookup errors retryable in copy
CopyObject and UploadPartCopy mapped any source-bucket versioning-state
lookup error to a terminal InvalidCopySource, including transient store
errors; getVersioningState signals a missing bucket with the not-found
sentinel, so split on that and let real store errors surface as 500, the
same mapping the destination-bucket lookup already uses.
* s3: match grpc-transported not-found in multipart list error paths
The list client path returns raw grpc status errors without the sentinel
reconstruction that lookups get in filer_pb.LookupEntry, so errors.Is on
the not-found sentinel never matched a store-reported missing directory;
match the sentinel text as well via a shared helper. The common missing-
directory case still lists as empty with no error and is unaffected.
* s3: complete and abort multipart surface store errors
prepareMultipartCompletionState mapped any upload-directory list or
lookup error to NoSuchUpload, so a transient store error at completion
time made the client discard a fully-uploaded object as gone. Split on
not-found like the sibling listing paths. abortMultipartUpload did the
same on s3a.exists, whose errors are never not-found (filer_pb.Exists
reports that as false with no error) -- a store failure there answered
NoSuchUpload and silently leaked the uploaded parts.
* s3: reject part numbers below 1 in UploadPartCopy
Only the upper bound was checked, unlike PutObjectPart; partNumber=0
passed the route regex and validation and wrote an out-of-range
0000_copy.part into the upload directory.
* filer: stream offloaded metadata-log entries instead of buffering whole files
The client metadata-chunks read path (ReadLogFileRefs, used by the meta
aggregator to consume peer filers and by mounts) decoded every entry of a
log file into a slice before handing it to the consumer, and prefetched the
next whole file the same way. Peak memory scaled with log-file size: under
heavy concurrent writes the per-event chunk lists grow and minute-files reach
hundreds of MB to GBs, so a filer aggregating a few peers held many GBs of
decoded entries at once (heap dominated by readLogFileEntries ->
consumeBytesNoZero) and OOMed.
Stream entries through a bounded channel: a producer decodes one entry at a
time and the next file's read overlaps processing via the channel buffer, so
peak memory is bounded by the channel depth rather than O(file size). In a
synthetic replay peak live heap dropped from ~1.3x the file size to a flat
few MB regardless of file size.
* filer: tighten offload replay tests
Share one ordered-replay assertion between the merge-order and single-filer
tests, assert the callback's own error is what propagates, and drop atomic
counters from callbacks that run on a single goroutine.
* filer: abort offloaded log replay promptly instead of joining wedged readers
Collapse the single-filer fast path into the merged reader: it was a second
copy of the producer/stop lifecycle with its own subtler synchronization, and
a one-stream merge does the same job.
On abort (processing error or a fatal read error from one filer), the
consumer used to drain channels and join every producer. A producer blocked
in an uncancellable chunk read cannot observe stop until that read returns,
so an abort could stall the caller's retry loop behind a dead volume-server
connection. Closing stop is now the only cleanup: producers check it at every
send and file boundary and exit on their own, and the merge loop's blocking
receives also escape on stop. Producers also check stop before opening each
file, so an aborted replay no longer keeps reading remaining files whose
entries never reach the channel.
A mid-file chunk-not-found still skips to the next file, but the log line now
reports how many entries were delivered first instead of pretending the whole
file was skipped; the redundant error log before setFatal is gone since the
error propagates to callers that already log it.
* filer: cap offloaded log entry allocation against corrupt size prefix
A garbage 4-byte size prefix (torn chunk or stream desync) drove
make([]byte, size) up to 4GiB per entry. Reject sizes above the same 1GiB
bound the filer-side log readers enforce.
Unmount, and optionally delete, EC volume shards from the shell, so broken
or over-replicated shards can be handled without stopping volume servers and
deleting files by hand.
Default action is unmount; --delete also removes the shard files. Targets
resolve against the live topology, with shardId:host:port to disambiguate
co-located or over-replicated shards. Dry-run by default; pass --apply.
* s3: embedded IAM inline policy honors prefix-scoped resources
getActions stripped the trailing wildcard from a resource like
arn:aws:s3:::bucket/prefix/*, producing a non-wildcard action
(Write:bucket/prefix) that CanDo only ever matched at bucket level, so
PutObject under the prefix was denied. Preserve the object path with its
wildcard (Write:bucket/prefix/*) to match objects under the prefix,
matching the standalone iamapi behavior.
* s3: prune bucket-confined wildcard actions on bucket delete
actionScopedToBucket treated any wildcard as multi-bucket, so a
prefix-scoped action like Write:bucket/prefix/* survived deletion of its
own bucket and could re-grant access if the bucket was recreated. Scope
the wildcard check to the bucket segment only: a wildcard in the object
path stays scoped to its bucket, while one in the bucket segment does
not.
filer: count reader on cacher waitgroup under the map lock
The read's wg.Add(1) ran after ReadChunkAt released the ReaderCache lock, so a
concurrent destroy() (error eviction, LRU, or UnCache) could start wg.Wait() on
a zero counter and then race the Add - a WaitGroup reuse that trips -race and
can panic. Move the Add under the lock, before the cacher can leave the
downloaders map, so a destroy is always ordered against a counted read.
* s3: add IncompleteBody error code
* operation: tag a truncated source read as ErrTruncatedBody
The chunked uploader wraps both source-read failures and volume-upload failures,
and a mid-write volume server drop also carries io.ErrUnexpectedEOF. Tag only the
source read so callers can tell a truncated input apart from a server-side fault.
* s3: return IncompleteBody for a truncated PUT body
A client abort or reverse-proxy timeout truncates the request body mid-upload.
putToFiler mapped every streaming-upload failure to InternalError (500), which a
reverse proxy relays as a 502. Classify a source-read truncation as IncompleteBody
(400) so the response matches AWS and passes through. All S3 write paths share
putToFiler, so they all benefit.
* volume: drop stale volume-location cache on under-replication
A replicated write looks up the volume's locations and caches them for 10
minutes. When the master briefly reports fewer replicas than the copy count
(e.g. a stale heartbeat drops a just-added volume), that under-replicated
result got cached, so every write failed with "replicating operations is less
than replication copy count" until the entry expired -- long after the master
re-registered the replica.
Invalidate the cached entry when the location count is below the copy count, so
the next write re-queries the master and recovers as soon as it heals.
* volume: mirror the replication copy-count guard in seaweed-volume
do_replicated_request accepted a write even when the master reported fewer
locations than the volume's copy count, silently under-replicating. Reject it,
matching Go's GetWritableRemoteReplications. lookup_volume is uncached, so the
next write recovers as soon as the missing replica re-registers.
* remote.meta.sync: materialize directory entries, including empty ones
Pull metadata by walking the remote tree one directory level at a time
with a delimiter, so subdirectories, including empty ones, are listed as
their own entries and created locally. The previous flat listing only
returned files, so empty remote directories never appeared locally and
non-empty ones only existed as filer-synthesized parents.
* remote.meta.sync: remove local metadata for entries deleted from remote
After reconciling each directory, drop local entries whose remote source
is gone: files are deleted outright, and a directory removed from the
remote is descended into so its remote-backed children are cleaned while
local-only entries are kept. remote.meta.sync exposes -delete (default
on) and remote.mount.buckets reconciles the same way; a plain
remote.mount stays additive.
* remote.meta.sync: reconcile type swaps and prune emptied directories
- when the remote swaps an entry's type (file <-> directory), drop the
stale local entry and recreate it with the right type; local-only
entries are left alone
- mark synced directories remote-backed and clean a directory removed
from the remote locally, deleting it once it holds no local-only
entries, instead of re-listing the missing remote path
- treat a differing remote size or mtime, not only a newer mtime, as a
change worth pulling
* Respect filerGroup in admin discovery
Admin discovery previously queried master cluster nodes with an empty filer group, so filers registered under a non-default group could not appear in the admin UI. Add an admin filerGroup flag and carry it through cluster-node discovery requests while preserving the empty default behavior.
Constraint: SeaweedFS master ListClusterNodes filters by exact filer_group.
Rejected: Discover all groups implicitly | no existing admin or shell behavior exposes cross-group discovery.
Confidence: high
Scope-risk: narrow
Directive: Keep admin cluster discovery scoped to the configured filerGroup unless an explicit all-groups API is added.
Tested: docker run --rm -v "$PWD:/src" -w /src golang:1.25 go test ./weed/admin/dash -run TestListClusterNodesRequest -count=1
Tested: docker run --rm -v "$PWD:/src" -w /src golang:1.25 go test ./weed/command -run '^$' -count=1
Not-tested: full repository test suite
* mini: pass filer group to admin cluster discovery
miniAdminOptions.filerGroup was never initialized, so startAdminServer
dereferenced a nil *string. Share the filer.filerGroup flag pointer so the
co-located admin queries the same group the filer registers under.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* feat(filer.sync.verify): reclassify chunk-slice-order ETag diffs as CHUNK_REORDER
filer.ETagChunks concatenates per-chunk MD5s in stored slice order without
normalising by offset, so byte-identical content written by two different paths
(S3 multipart part-completion order on the source vs filer.backup replication
arrival order on the destination) yields different file ETags. filer.sync.verify
reported these as ETAG_MISMATCH even though the files are equal.
Add a second-pass check on every ETAG_MISMATCH: when both sides derive their ETag
from chunks (no attr.Md5) and hold a manifest-free, non-overlapping chunk set
that, once sorted by offset, matches element-wise on (offset, size, ETag),
classify the file as CHUNK_REORDER. Such files are content-equal, so they are
not counted as errors and do not affect the exit code; they are listed only at
higher verbosity (weed -v=1), while the summary always shows their count.
The check stays conservative: a stored attr.Md5 (order-independent content
hash), a differing chunk count, an overlapping/duplicate offset (whose visible
bytes are resolved by timestamp), or a manifest chunk all remain ETAG_MISMATCH.
* filer.sync.verify: decline chunk-reorder fast path on empty per-chunk ETag
An empty or undecodable per-chunk ETag is not a content fingerprint, so
element-wise (offset, size, ETag) equality can't prove the bytes match.
Treating "" == "" as content-equal could reclassify a genuine divergence as
CHUNK_REORDER and drop it from the error count. Decline such chunk sets so they
stay ETAG_MISMATCH.
* filer.sync.verify: emit CHUNK_REORDER in JSON output regardless of -v
The -v=1 gate belongs to the human text report only. Applying it before the
jsonOutput branch dropped the per-file CHUNK_REORDER records from NDJSON while
the summary still counted them, so a machine consumer saw a non-zero count with
no records to reconcile it. Gate the text path only; JSON always emits.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* ec: expose force_deleted_needles_check in ScrubEcVolume RPC and shell
FULL EC scrubs can opt into strict deleted-needle verification via the
-forceDeletedNeedlesCheck shell flag, off by default since it can report
false positives when EC indexes disagree. Rejected for non-FULL modes.
The Rust volume server parses the new field and ignores it: its FULL
scrub verifies shards via RS parity, not per-needle reads.
* volume: require admin auth for ScrubEcVolume
ScrubEcVolume ran unauthenticated while its sibling ScrubVolume, and the
rest of the mutating volume handlers, gate on checkGrpcAdminAuth. Close
the gap so an EC scrub can't be triggered anonymously.
* shell: reject ec.scrub -forceDeletedNeedlesCheck outside full mode
Fail in the client before fanning out to every volume server, instead of
erroring halfway through once the servers reject the request.
* balance: extract the bytes-aware density metric to weed/topology/balancer
The shell's volume.balance ranks servers by a bytes-aware density (used volume
equivalents over free capacity). Move that math into the shared balancer package
(VolumeDensity / DensityRatio / DensityNextRatio) so the maintenance worker can
adopt the same metric next. Shell behavior is unchanged.
* balance: rank a server with no free capacity as the fullest
DensityRatio/DensityNextRatio divided by capacity, so a server past its slot
limit (negative capacity) returned a negative ratio and sorted as the emptiest
under ascending consumers — the opposite of reality. Treat any non-positive
capacity (full, or overfull mid-run after receiving volumes) as the fullest
(+Inf) so it is a move source, never a target. Covered by negative-capacity and
ordering tests.
* kms: cache decrypted data keys so cache_enabled/cache_ttl take effect
cache_enabled/cache_ttl were parsed into KMSConfig but never consulted,
so every SSE-KMS read repeated a KMS Decrypt round-trip. Add a
CachedKMSProvider decorator keyed on (ciphertext, encryption context)
and wrap providers with it wherever one is created. Decrypt is
deterministic for that pair; GenerateDataKey/DescribeKey/GetKeyID pass
through untouched.
Cache hits and stores return private copies so the read path's
ClearSensitiveData can't wipe the cached key. TTL and max_cache_size
bound the cache.
* kms: zero superseded data keys and pool cache-key hash states
Wipe the old plaintext when a cache entry is overwritten so a key
material buffer left behind by two readers racing on the same miss does
not linger in memory. Reuse sha256 states via a sync.Pool so the read
hot path stops allocating a fresh hash on every Decrypt.
* kms: drop cache writes after Close
An in-flight Decrypt miss can call set after Close scrubbed the map,
repopulating it with a data key that would then never be cleared. Guard
set with a closed flag so post-Close stores wipe their copy and no-op.
* admin: never close the worker outgoing channel while senders are live
conn.outgoing has multiple concurrent senders (heartbeat, task assignment,
log request, registration handlers). Closing it on connection teardown raced
a sender and paniced with "send on closed channel" — reliably reproduced when
a laptop goes idle: heartbeats stall past the 2-minute stale cutoff, the
cleanup routine closes the channel, and the resumed worker's heartbeat is
received and handled at the same moment.
The connection context is already the sole teardown signal, so stop closing
the channel entirely. handleOutgoingMessages exits on conn.ctx.Done(), and the
buffered channel is GC'd once the connection drops. Route sends through a
sendToWorker helper that also selects on conn.ctx.Done() so they bail on
teardown instead of blocking for the full timeout.
* admin: bail on ctx.Done() while waiting for a worker log response
* refactor: centralize genUploadUrl in UploadOption
Replace inline genFileUrlFn closures with operation.GenUploadUrl field:
- Add GenUploadUrl func(host, fileId) string to UploadOption struct
- Add GenUploadUrlProxy(filerAddress string) utility function
- Remove genFileUrlFn parameter from UploadWithRetry signature
- Update all callers: mount, gateway, mq, filer_copy, filer_sync
This matches the weed mount -filerProxy pattern exactly,
factorizing the URL generation logic across all consumers.
Co-Authored-By: Athena 🏛️ <hermes-agent@local> (custom / Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf)
* docker release: run all platform jobs in one wave, cache rocksdb compile
Drop max-parallel so the 13 per-platform builds run together instead of two
waves of 8 (rocksdb was queuing behind the cap and starting ~8 min late).
Keep cache-to mode=max for rocksdb: its RocksDB static_lib compile is
sha-independent, so it caches across releases and stops being the ~16-min
long-pole that gates the merge fan-in. go-build variants stay mode=min.
Co-Authored-By: Athena 🏛️ <hermes-agent@local> (custom / Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf)
* refactor: centralize genUploadUrl in UploadOption
Replace inline genFileUrlFn closures with operation.GenUploadUrl field:
- Add GenUploadUrl func(host, fileId) string to UploadOption struct
- Add GenUploadUrlProxy(filerAddress string) utility function
- Remove genFileUrlFn parameter from UploadWithRetry signature
- Update all callers: mount, gateway, mq, filer_copy, filer_sync
This matches the weed mount -filerProxy pattern exactly,
factorizing the URL generation logic across all consumers.
Co-Authored-By: Athena 🏛️ <hermes-agent@local> (custom / Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf)
* Remove accidental ROCmFPX submodule reference
* gofmt chunk upload option block
* Preserve broker cipher and re-read proxy filer per upload attempt
Chunk uploads must keep the configured Cipher, and both the mount and broker current filer can change on failover, so build the proxy upload URL inside the closure instead of capturing the address once.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* proto: add EC bitrot checksum messages + CHECKSUM scrub mode
Mirror weed/pb/volume_server.proto byte-for-byte (field numbers + types) so the
.ecsum sidecar payload is wire-identical across the Go and Rust binaries:
EcBitrotProtection / EcShardChecksums / ChecksumAlgorithm, VolumeScrubMode.CHECKSUM=4,
and VolumeEcShardsCopyRequest.copy_ecsum_file. No code uses them yet — the .ecsum
format, producer, mount-load, copy, and scrub land in following commits.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat(ec): port the .ecsum bitrot checksum module
ec_bitrot.rs mirrors weed/storage/erasure_coding/ec_bitrot.go: the .ecsum sidecar
format (14-byte big-endian ECSU header + CRC32C over a prost-serialized
EcBitrotProtection payload), the per-shard per-block CRC32C producer
(ShardChecksumBuilder), save/load with payload self-integrity, manifest
validation, status resolution, and verify_shard_file_blocks for the CHECKSUM
scrub. A byte-exact test pins the serialized bytes against the Go reference's
identical constant so a format drift in either binary fails loudly.
Producer wiring (encode/vacuum), mount-load, copy, and the mode-4 dispatch land
in following commits.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* test(ec): pin .ecsum sidecar bytes for cross-binary interop
Deterministic EcBitrotProtection -> exact on-disk bytes, asserted against a
canonical constant on BOTH sides (this test and ec_bitrot.rs), so a format drift
in either binary fails its own suite rather than silently desyncing a Go-written
.ecsum from a Rust-written one.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat(ec): write the .ecsum bitrot sidecar during EC encode
write_ec_files now feeds each shard's bytes through a per-shard
ShardChecksumBuilder as it writes them, then persists the generation-0 sidecar
(<base>.ecsum) alongside the shards — mirroring weed's WriteEcFiles +
SaveBitrotSidecar. Best-effort: a failed sidecar write leaves the generation
unprotected rather than failing the encode. A test confirms the produced sidecar
validates and its per-block CRCs match every on-disk shard.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat(ec): load the .ecsum at mount + EcVolume::checksum_scrub
EcVolume now loads and validates its generation-0 .ecsum sidecar at mount,
caching the parsed protection + BitrotStatus (Off/On/Invalid), and exposes
bitrot_protection() mirroring Go's EcVolume.BitrotProtection(). checksum_scrub()
verifies every locally-held shard's raw bytes against the sidecar block CRCs —
the only path that exercises cold parity shards — reporting mismatched shards
without mutating anything; a wholesale mismatch beyond parity is flagged as a
suspect sidecar rather than mass shard corruption. Mirrors Go's ChecksumScrub.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat(scrub): dispatch EC CHECKSUM (mode 4) to checksum_scrub
Accept VolumeScrubMode.CHECKSUM=4 and route it to EcVolume::checksum_scrub,
accumulating blocks scanned + mismatched shards into the scrub response, plus the
CHECKSUM scrub-mode metric label. Read-only bitrot verification over local shards.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat(ec): copy the .ecsum sidecar during VolumeEcShardsCopy
Honor copy_ecsum_file: when set, copy the generation-0 .ecsum alongside the
shards so protection travels with them, mirroring Go's non-2PC copy path.
Tolerant of a missing source (empty stream) — the 0-byte file is dropped so
mount sees no sidecar (protection off) rather than a truncated/invalid one.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat(ec): remove the .ecsum sidecars when destroying an EC volume
remove_ec_volume_files now clears <base>.ecsum (and any versioned .ecsum.v<N>)
from the data and idx dirs, so a vid reuse can't load a stale sidecar. Mirrors
Go's removeBitrotSidecars.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* style(ec): align bitrot comments and test setup for merge-cleanliness
Match the shared bitrot code (write_ec_files, encode_one_batch, checksum_scrub,
the encode sidecar test) to the canonical wording/layout so the volume-server
Rust port stays line-aligned across trees, keeping periodic merges conflict-free.
No behavior change.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
The disk-fullness gate only rejected destinations already at/above the mark, so a
server just under it could take a large volume and overshoot. Project the selected
volume's bytes onto the candidate: if the move would cross the mark, drop that
destination for the rest of the cycle and re-pick instead of overshooting. Also
note the per-location capacity-summing assumption on the Rust heartbeat side, to
match the Go store.go comment.
The replica-placement rule (data-center/rack/same-node limits plus host
anti-affinity) existed three times: the shell's satisfyReplicaPlacement/isGoodMove
used by volume.balance, fix.replication, and tier.move, and a line-for-line port
in the maintenance balance worker. Move the canonical logic into
weed/topology/balancer on a shared Location type; the shell and worker keep thin
adapters that convert their own location representation and call it. Behavior is
unchanged (the shared IsGoodMove keeps the shell's reject-move-to-self guard, and
all four replica test suites pass).
EC placement scored destinations purely by free EC shard slots (derived from
maxVolumeCount) and shard counts, blind to real disk fullness — the same defect
as volume balancing. A disk that is physically full but still shows free EC slots
kept being chosen, and EC shard bytes are captured by statfs free space yet not
by any slot accounting, so the slot math is exactly the metric that can't see EC
fullness.
Treat a disk at/above 90% physical usage as having zero free EC slots at
snapshot-build time, so every existing freeSlots>0 placement predicate excludes
it. Applied in all three snapshot builders (shell countFreeShardSlots, the shared
ecbalancer FromActiveTopology, and the worker ec_balance buildBalancerTopology)
via the shared balancer.DiskTooFullAfter gate. Servers not reporting disk bytes
fall back to slot-only behavior. ec.rebuild recovery is left ungated so shard
recovery can still complete onto fuller disks.
* shell: add volume.balance -byDiskUsage to balance by actual data
The default balancer ranks servers by slot density, dividing used volumes by
MaxVolumeCount. When MaxVolumeCount is configured higher than the disk can hold,
a physically near-full server looks nearly empty and gets picked as the move
target, so balancing drains less-full servers onto an already-full one.
-byDiskUsage ranks servers by the actual data they hold (sum of volume sizes)
instead, so the fullest-by-data server is treated as full and balancing drains
it. It assumes comparable disk sizes per disk type and still respects each
server's free volume slots. Default behavior is unchanged.
* plumb physical disk usage into topology, gate volume.balance on it
Volume servers now report each disk's filesystem total/free bytes in the
heartbeat, and the master stores them in DiskInfo. volume.balance uses them to
skip any move target whose disk is already near full (-maxDiskUsagePercent,
default 90), so an over-configured maxVolumeCount can no longer make a
physically full server look empty and get drained onto. The gate judges each
server against its own disk, so heterogeneous disk sizes are fine; servers that
do not report bytes fall back to slot-only behavior.
Rust seaweed-volume mirrors the heartbeat reporting.
* admin: report real physical disk capacity when volume servers provide it
The dashboard estimated server capacity as maxVolumeCount * volumeSizeLimit,
which overstates it when maxVolumeCount is set higher than the disk holds.
Prefer the filesystem capacity now reported per disk, falling back to the
estimate for servers that do not report it.
* worker: gate automatic balance on physical disk fullness too
The maintenance balance worker selects the least slot-utilized server as the
move destination, so an over-configured maxVolumeCount makes a physically full
server look empty and get drained onto — the same defect as the shell command.
Now that DiskInfo carries real disk bytes, skip any destination whose disk is
at/above 90% used (per server, against its own disk); a full server can still be
a source. When every candidate destination is full, create no tasks. Servers
that do not report disk bytes are not gated.
* balance: share the physical-disk-fullness gate between shell and worker
The shell volume.balance command and the maintenance balance worker each grew
their own copy of the disk-fullness gate (targetDiskTooFull / destinationDiskTooFull)
and a maxDiskUsagePercent=90 constant. Pull both into weed/topology/balancer
(DiskTooFullAfter + DefaultMaxDiskUsagePercent) so the policy has one home and the
two balancers can't drift.
* balance: harden the physical-disk gate
Guard against a nil DiskInfo in the byte/slot lookups. Let a zero disk-capacity
report clear previously stored bytes (0 means "not reported" for bytes, unlike
maxVolumeCount), so a server that stops reporting falls back to slot-only instead
of trusting stale capacity. In the worker, charge each planned move's bytes to
its destination within a detection cycle so the gate sees a target fill up rather
than only its heartbeat-time free space. Note the per-location capacity summing
assumes one location per filesystem (the used ratio the gate relies on stays
correct regardless; absolute capacity can over-report).
* fix(topology): keep physical disk 0 distinct in SplitByPhysicalDisk
DiskId 0 doubles as the first physical disk (Locations[0]) and the
protobuf "unset" default. SplitByPhysicalDisk folded every DiskId-0
record onto the aggregate DiskId whenever that was non-zero, so on a
multi-disk node the first disk's volumes merged into whichever disk
held volumes[0]: the node reported one fewer disk, the sibling showed
~2x volumes, and per-disk max was smeared across the survivors. This
surfaced as cluster.status and volume.list undercounting disks.
Only treat 0 as unset when no record carries a non-zero DiskId; with a
mix, 0 is a real disk and keeps its own entry.
* fix(admin): resolve physical disk 0 in active-topology indexes
rebuildIndexes re-derived each volume/EC record's physical disk id with
the same "DiskId 0 means unset" heuristic SplitByPhysicalDisk used, so
the two agreed only by sharing the bug. Now that SplitByPhysicalDisk
keeps disk 0 distinct, the duplicated heuristic would fold disk-0 records
onto a sibling while at.disks kept them on disk 0; GetVolumeLocations and
GetECShardLocations then matched no record and silently dropped every
volume and EC shard on the first disk, starving balance and EC tasks.
Build the indexes from the same SplitByPhysicalDisk reconstruction that
builds at.disks, so the keys always resolve. One source of truth instead
of a parallel normalize.
* fix(ec): allow physical disk 0 as preferred EC shard target
pickBestDiskOnNode gated its result on bestDiskId != 0, but 0 is both a
valid physical disk and the uint32 zero value, so a best-scoring disk 0
was discarded and the non-matching fallback returned instead. Gate on
bestScore.
* test(admin): cover EC-shard index resolution for physical disk 0
rebuildIndexes builds ecShardIndex the same way as volumeIndex; pin the EC
path too so a shard on disk 0 keeps resolving via GetECShardLocations.
* proto: per-disk type/capacity in DiskTag, DiskInfo.physical_disks
DiskTag gains type + max_volume_count so the heartbeat can describe every
physical disk, including ones holding no volumes or EC shards. DiskInfo
gains physical_disks so the master can hand the full per-type disk set to
per-physical-disk consumers.
* feat(volume): report each physical disk's type and capacity
CollectHeartbeat fills DiskTag.type and the per-disk effective max for
every location, so the master can account for disks that hold no volumes
or EC shards yet. Rust heartbeat mirrors it.
* feat(master): surface empty disks in the per-physical-disk view
The master records each disk's type and max from DiskTags and lists them
on DiskInfo.physical_disks per type, including disks with no volumes or
EC shards. SplitByPhysicalDisk enumerates that full set and gives each
disk its exact max, so cluster.status, volume.list and the admin
topology count and can target empty disks. Without physical_disks the
even-split fallback is unchanged.
* fix(master): clamp per-disk free at zero for over-allocated disks
In the exact-max path FreeVolumeCount could go negative when a disk holds
more volumes than its max; a negative would reduce the node's summed free
and block placement on healthy disks. Clamp at 0.
* fix(master): rebuild disk tags fresh each heartbeat
DiskTags is the full authoritative per-disk list every heartbeat, so
rebuild dn.diskTags from scratch like dn.diskBackends; merging left stale
entries for removed disks.
* fix(master): keep zero-capacity disks in physical_disks
A disk reporting max 0 (an unavailable disk) is a valid physical disk,
not a signal to drop it. List every disk of the type, but only emit
physical_disks when the node reports real per-disk capacity, so an older
server sending all zeros still falls back to the aggregate split.
* test(volume): cover disk-space-low per-disk max in heartbeat
Assert DiskTag.max_volume_count follows the used-slots override when a
location is low on space, matching the per-type max_volume_counts.
* chore: trim comments on the empty-disk change
Drop narration; keep only the non-obvious why (disk-0 sentinel, exact-max
free clamp, EC slots not subtracted, all-zeros fallback).
* refactor(master): merge per-disk tags and capacity into one map
diskTags and diskBackends were parallel maps keyed by the same DiskId and
filled together from DiskTags. Fold them into one diskMetas map of
{tags, type, max}.
* refactor(proto): per-disk max as a map keyed by disk id
physical_disks was a repeated {disk_id, max_volume_count} whose fields
duplicated DiskInfo's own disk_id/max_volume_count. A map<uint32,int64>
keyed by disk id expresses "max per disk" directly, drops the extra
PhysicalDiskInfo message, and the consumer reads it as the disk set.
* docs(proto): note DiskInfo.disk_id's two meanings
Identity on a per-physical-disk DiskInfo (from SplitByPhysicalDisk),
representative fallback on the type-keyed aggregate.
* fix(topology): keep physical disk 0 distinct in SplitByPhysicalDisk
DiskId 0 doubles as the first physical disk (Locations[0]) and the
protobuf "unset" default. SplitByPhysicalDisk folded every DiskId-0
record onto the aggregate DiskId whenever that was non-zero, so on a
multi-disk node the first disk's volumes merged into whichever disk
held volumes[0]: the node reported one fewer disk, the sibling showed
~2x volumes, and per-disk max was smeared across the survivors. This
surfaced as cluster.status and volume.list undercounting disks.
Only treat 0 as unset when no record carries a non-zero DiskId; with a
mix, 0 is a real disk and keeps its own entry.
* fix(admin): resolve physical disk 0 in active-topology indexes
rebuildIndexes re-derived each volume/EC record's physical disk id with
the same "DiskId 0 means unset" heuristic SplitByPhysicalDisk used, so
the two agreed only by sharing the bug. Now that SplitByPhysicalDisk
keeps disk 0 distinct, the duplicated heuristic would fold disk-0 records
onto a sibling while at.disks kept them on disk 0; GetVolumeLocations and
GetECShardLocations then matched no record and silently dropped every
volume and EC shard on the first disk, starving balance and EC tasks.
Build the indexes from the same SplitByPhysicalDisk reconstruction that
builds at.disks, so the keys always resolve. One source of truth instead
of a parallel normalize.
* fix(ec): allow physical disk 0 as preferred EC shard target
pickBestDiskOnNode gated its result on bestDiskId != 0, but 0 is both a
valid physical disk and the uint32 zero value, so a best-scoring disk 0
was discarded and the non-matching fallback returned instead. Gate on
bestScore.
* test(admin): cover EC-shard index resolution for physical disk 0
rebuildIndexes builds ecShardIndex the same way as volumeIndex; pin the EC
path too so a shard on disk 0 keeps resolving via GetECShardLocations.
* fix(shell): honor explicit fs.mergeVolumes from/to direction
mergeVolumes only ever merged a smaller volume into a larger one. When the
user named both -fromVolumeId and -toVolumeId with the source larger than the
target, the planner produced an empty plan and the command printed just
"max volume size: N MB" and moved nothing.
Build the requested pair directly when both ids are given, instead of routing
through the size-descending heuristic. Read-only, empty, and wrong-collection
endpoints are rejected with a clear error rather than a silent no-op.
* fix(shell): allow fs.mergeVolumes into an empty target volume
Merging chunks into an empty volume is valid, e.g. consolidating data into a
freshly created or recently vacuumed volume. Only reject an empty source, which
has nothing to move.
* fix(shell): reject self-map in directed mergeVolumes planner
createMergePlan with from == to returned a {vid: vid} self-merge when called
directly. Guard it in the planner so it is correct independent of the Do
entrypoint.
* s3: invalidate stale reader cache locations on chunk read failure
* filer: share the chunk-read self-heal across reader cache and streaming paths
The reader cache retry added a third copy of the invalidate-relookup-compare-retry
dance already inlined in PrepareStreamContentWithThrottler and duplicated in
retryWithCacheInvalidation. Extract retryFetchWithFreshLocations and route all
three through it, parameterized by the refetch primitive.
* filer: drop redundant completedTimeNew store in reader cache success path
startCaching already stamps completedTimeNew unconditionally before the
fetchErr branch; the second store inside the success branch is dead.
* filer: make NewReaderCache cache invalidator an explicit parameter
The variadic ...CacheInvalidator only ever read the first element, so a caller
could pass two and silently get one. Take a single explicit argument and have
the non-S3 callers pass nil.
* filer: inject reader cache chunk fetch as a struct field
Replace the process-global readerCacheFetchChunkData test seam with a
per-instance fetchChunkDataFn field defaulted in NewReaderCache, matching how
lookupFileIdFn is already wired. Tests set the field on the cache instead of
swapping a shared global.
* filer: log the location count, not full URLs, on self-heal retry
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Concurrent bucket deletion across multiple filer replicas races on the
per-bucket DROP TABLE. The first replica drops the table; the rest hit
an undefined-table error (postgres 42P01, mysql 1051) which propagates
out of DeleteFolderChildren and panics the filer. On restart the same
pending DROP re-runs and the filer crash-loops.
Make the drop idempotent. Same defect class in all SQL backends, so fix
postgres, postgres2, mysql, mysql2, and sqlite together.
The EC volumes, EC shards, and collection details pages each rendered a
repair (wrench) button for incomplete EC volumes. Its handler POSTed to a
/repair endpoint that the admin server never registers, so every click
returned "404 page not found" (the collection details page only had a
placeholder handler).
Remove the buttons and their JavaScript handlers, and regenerate the
templ output. Manual EC shard recovery remains available from weed shell
via ec.rebuild.
EC volumes do not propagate deletions to all shard indexes, so it is possible
to run scrubbing on a volume where a deleted needle is still present in the
index, or a needle deleted from the index is still present on the volume.
On either scenario, scrubbing will fail due to size mismatch errors.
This PR reworks the scrubbing logic so needle size mismatches are
ignored in such scenarios.
Scrubbing can still be forced to check deleted needles (f.ex. to discover
index inconsistencies); this option will be exposed in RPCs and `weed shell`
on a follow-up PR.
* fix(scrub): don't flag offset-0 logical tombstones in volume scrub
A remote-tier delete records a tombstone at .idx offset 0 with no physical .dat
bytes. Full scrub double-flagged a healthy remote-tiered volume with deletes:
scrubVolumeData counted the tombstone's GetActualSize(-1)=32 toward totalRead
(want > physical .dat), and CheckIndexFile treated it as occupying [0,31] and
flagged the first live needle as overlapping. Skip offset-0 logical tombstones
from both the size reconcile and the overlap check; they are still counted for
the index-size check. Local deletes (offset != 0) are unaffected.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* fix(scrub): mirror offset-0 logical tombstone handling into Rust
Same fix as the Go volume_checking.go + idx/check.go change: Volume::scrub skips
offset-0 logical tombstones from total_read, and check_index_file excludes them
from the overlap check (still counted for the index-size check).
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* fix(ec): suppress deleted-needle size mismatch in EC LOCAL scrub
EcVolume.ScrubLocal reassembles each fully-local needle and ReadBytes-checks
it, but appended every error unconditionally. A needle the .ecx still reports
live while its reassembled on-disk header carries size 0 (delete state
disagrees between index and header) is not corruption — the LOCAL twin of the
#10130 fix for the FULL path. Suppress the ErrorSizeMismatch in that case;
genuine (non-zero) size mismatches and CRC/tail errors are still reported.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* fix(ec): mirror EC LOCAL scrub deleted-needle suppression into Rust
Same suppression as the Go EcVolume.ScrubLocal change: a NeedleError::SizeMismatch
whose on-disk header size is 0 against a live index entry is a delete-state
disagreement, not corruption.
Claude-Session: https://claude.ai/code/session_015EE9Sc9EvNp8BCVva4RKdo
* feat: add collection pattern to delete empty volumes
Co-authored-by: Codex <noreply@openai.com>
* shell: match collection pattern with wildcard matcher
Use wildcard.MatchesWildcard in the shared collection-pattern helper,
matching command_volume_fix_replication's matchCollectionPattern. The
flag only advertises '*' and '?', which is exactly what the matcher
supports.
---------
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3: support AWS object form for bucket policy Principal, add NotPrincipal
Bucket policy statements only accepted a bare string or array of strings for
the Principal element, so the AWS-documented object form was rejected:
"Principal": { "AWS": "arn:aws:iam::123456789012:root" }
"Principal": { "AWS": ["arn:...", "999999999999"] }
Add a PolicyPrincipal type that parses the bare string, the bare array
(retained for backward compatibility), and the object form keyed by AWS,
Service, Federated or CanonicalUser (each value a string or array). All keyed
values are flattened for principal matching, and the original JSON is preserved
so PutBucketPolicy/GetBucketPolicy returns the exact shape submitted - keeping
infrastructure-as-code tools (Terraform, Ansible) idempotent.
Also add NotPrincipal support (a statement applies to every principal except the
ones named), compiled and evaluated in both policy evaluators, and reject
statements that specify both Principal and NotPrincipal.
* s3: address review - validate principal object form, honor dynamic NotPrincipal
- Reject unsupported Principal object keys (only AWS/Service/Federated/
CanonicalUser) and empty values, so a form like {"AWS":[]} no longer compiles
to zero matchers and silently relies on the match-all fallback.
- Detect both Principal and NotPrincipal by field presence, not by flattened
length, so a present-but-empty field is still rejected.
- Honor dynamic (policy-variable) NotPrincipal/Principal patterns in the
compiled evaluator; previously a NotPrincipal made only of variables was
treated as absent and its exclusion bypassed.
- Add regression tests for the object-form validation and dynamic NotPrincipal.
* Review comment removed unnecessary success and failure count
* fix: use Gather.Gather() with seeded counter for EC rebuild registration test
- Restore Gather.Gather() to verify MustRegister calls as requested in review
- Seed VolumeServerECRebuildCounter before gathering because CounterVec
only appears after at least one label value is observed
- Use correct fully-qualified metric names (SeaweedFS_volumeServer_*)
* fix: remove preflight checkEcVolumeStatus failure from ec_rebuild_total counter
ec_rebuild_total should only reflect actual rebuild execution failures
(from RebuildEcFiles / RebuildEcxFile), not scan/precheck failures in
the volume status loop. The error is still returned to the caller;
only the misleading counter increment was removed.
* Review comment removed unnecessary observe
* label EC rebuild duration histogram by result
Without a result label, fast failures pull down the success-latency
quantiles shown on the EC Rebuild Duration panel. Make the histogram a
HistogramVec keyed by result, record success/failure through one
recordEcRebuild helper, and split the Grafana quantiles by (le, result).
* reset EC rebuild metric vecs in registration test
The HistogramVec needs a child before Gather emits it, so the test must
observe once; reset both vecs in cleanup so that sample doesn't leak into
other tests.
---------
Co-authored-by: Ubuntu User <ubuntu@example.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
An empty or truncated tasks/*.pb file unmarshals into a TaskStateFile
with a nil Task, and protobufToMaintenanceTask dereferenced it
immediately, panicking the whole admin process on startup. Guard the
nil case so the loader logs a warning and skips the bad file.
Under a herd of concurrent assigns with no writable volume, Assign spun
PickForWrite for the full 10s timeout, pinning a goroutine per request and
starving the master of the cycles it needs to process growth and answer
heartbeats. When growth is the relevant remedy and already in flight, stop
spinning: if free space exists, shed with a fast retryable error so clients
back off and retry once growth lands; if the cluster is out of space, fail fast
with the real out-of-space error instead of masking it as retryable.
The gRPC shed uses ResourceExhausted, not Unavailable: operation.Assign retries
it, but the client connection layer doesn't treat it as a dead channel, so a
per-request shed across a herd doesn't tear down the shared master connection
and cancel every other in-flight assign. The HTTP dirAssignHandler sheds with
503 + Retry-After.
* volume server: route VolumeMarkReadonly to raft leader
After a master raft election, volume servers may still heartbeat a follower
while admin paths such as weed shell volume.mark call notifyMasterVolumeReadonly
via vs.GetMaster(). Followers reject VolumeMarkReadonly with NotLeader, which
breaks tiering and other mark-readonly workflows until the heartbeat loop
reconnects.
Resolve the leader through GetMasterConfiguration on configured -master peers
(same Leader field filer/master clients already use) before calling
VolumeMarkReadonly. When the leader differs from the heartbeat peer, update
currentMaster so the heartbeat loop converges faster.
Adds operation.LookupRaftLeaderMaster with unit tests.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: address review feedback on volume.mark raft leader routing
Do not update currentMaster during leader lookup — heartbeat owns that
field and uses stream GetLeader() to reconnect. Try the heartbeat peer
first and only resolve the raft leader after a NotLeader rejection.
Add ctx.Err() early exit and quieter logging for context cancellation.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(operation): thread the lookup timeout ctx into connection invalidation
The 5s timeout drove only the RPC; WithMasterServerClient saw the
unbounded outer ctx, so a self-inflicted timeout (slow GetMasterConfiguration
during an election) was treated as a stale channel and tore down the shared
master connection. Pass the timeout ctx into the helper so its own expiry
leaves ctx.Err() set and spares the connection.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* fix(filer.sync.verify): sort listings client-side before merge
The merge walks both filers' directory listings in lockstep and needs
them in the same byte order. A filer before 4.32 with a locale SQL
collation lists case-insensitively while a 4.32+ peer lists byte-ordered,
so comparing two such clusters returns the same names in a different
order and the merge desyncs into spurious MISSING / ONLY_IN_B.
Buffer and sort each directory client-side so both sides agree on order
regardless of filer version or store backend. Trades the streaming
source's O(buffer) memory for O(directory) per side, fine for a one-shot
verify CLI; both sides still load concurrently.
Claude-Session: https://claude.ai/code/session_01BKsBdKYFNCEjeHLjJfumPF
* fix(filer.sync.verify): surface listing errors before merging
A listing that fails mid-stream leaves a partial, unsorted buffer. Now
that both sides are fully buffered anyway, check each side's error right
after the loads finish and before the merge, so partial entries can't
emit spurious MISSING / ONLY_IN_B before the error aborts the run.
Claude-Session: https://claude.ai/code/session_01BKsBdKYFNCEjeHLjJfumPF
* fix(shell): correct volume.list -writable filter unit and comparison
* fix(shell): correct volume.list -writable filter unit and comparison
* chore(shell): fix typo in EC shard helper param names
* fix(shell): use exact match for volume.balance -racks/-nodes filter
The old strings.Contains-based filter quietly included any id that was a
substring of the user-supplied flag value (e.g. -racks=rack10 also matched
rack1). Replace it with an exact-match set parsed from the comma-separated
flag value, and add regression tests for both -racks and -nodes paths.
Also fix a small typo in the "remote storage" error returned by
maybeMoveOneVolume.
* fix(shell): use exact match for volume.balance -racks/-nodes filter
The old strings.Contains-based filter quietly included any id that was a
substring of the user-supplied flag value (e.g. -racks=rack10 also matched
rack1). Replace it with an exact-match set parsed from the comma-separated
flag value, and add regression tests for both -racks and -nodes paths.
Also fix a small typo in the "remote storage" error returned by
maybeMoveOneVolume.
* refactor(shell): drop nil sentinel in splitCSVSet, use len() in callers
* fix: apply collectionPattern during detection in volume.fix.replication
* use existing wildcard.MatchesWildcard for collection matching
It returns a plain bool, so drop the up-front filepath.Match validation
and the path/filepath import that only existed to handle its error.
* trim verbose comments to terse one-liners
* drop redundant per-path collection guards
Detection already filters by replicas[0].info.Collection. The repair guard
re-checked pickOneReplicaToCopyFrom's collection (a different replica), so a
mixed-collection volume could pass detection yet be skipped in repair without
decrementing the counter, spinning the -apply loop. deleteOneVolume keeps its
collectionIsMismatch safety.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
mount: move directory cache state to a side map to shrink InodeEntry
The mount keeps an InodeEntry alive for every inode the kernel references.
On a mount that is almost entirely regular files, each entry carried the full
directory readdir-cache bookkeeping (four time.Time fields plus counters),
bloating it to 152 bytes whether or not the inode was a directory.
Move that state into a dirState held in a side map keyed by inode, and drop the
isDirectory bool: an inode is a directory iff it has a dirState. InodeEntry is
now just paths + nlookup at 32 bytes, landing in a smaller Go allocator size
class; on a mount with tens of millions of cached file inodes that is several GB
less resident heap. As a side effect the readdir-cache scan helpers iterate only
directories instead of every inode.
* ec: recover EC shards whose .ecx index lives only on a peer server
A volume server that boots with EC shard files on disk but no .ecx index
on any local disk cannot mount the shards, so the master never learns
about them. ec.rebuild works off master-registered shards, so it sees the
volume as short and gives up even though the shard data is intact.
Add an operator-triggered recovery: VolumeEcShardsMount gains a
recover_missing_index flag that makes the volume server fetch the missing
.ecx (plus .ecj/.vif) from a peer holding it and mount the on-disk shards.
ec.rebuild runs this across the cluster before planning, so orphaned
shards register and the rebuild sees the true shard set.
.ecx is an immutable encode-time index, identical on every holder. .ecj
is a per-holder deletion journal that differs across holders, so the
recovered node adopts the source peer's deletion view, like a balanced or
rebuilt shard does.
* ec: mirror missing-index recovery into the Rust volume server
Port the #10104 recovery to seaweed-volume so the Rust volume server
self-heals the same layout: EC shards on disk with the .ecx index only on
a peer. Adds collect_ec_volumes_missing_index / mount_recovered_ec_shards
to the store, recover_missing_ec_indexes (master LookupEcVolume + peer
CopyFile fetch + mount) to the server, and the recover_missing_index flag
on VolumeEcShardsMount.
.ecx is the immutable encode-time index, identical on every holder. .ecj
is a per-holder deletion journal, so the recovered node adopts the source
peer's deletion view, matching the Go path.
Chris Lu