* wdclient: bound the wait for a master leader by the caller's context
WithClient waited on GetMaster with context.Background(), so a caller that
arrived while no master leader was known parked in a 200ms poll loop until one
appeared, whatever deadline it had already set on the RPC. Each retry above it
then left another goroutine in the same wait.
Take the context in WithClient and WithClientCustomGetMaster and hand it to
GetMaster, and stop the retry loop once it is done. The dial keeps
context.Background(): fn brings its own RPC context, so a cancellation seen
here cannot be attributed to the shared connection.
Call sites pass whatever they hold: the request context in the filer's
CollectionList, DeleteCollection and Statistics handlers and in the credential
store's propagation, the operation context in the shell's s3.bucket.delete and
the kafka gateway's broker and filer discovery, and context.Background() where
there is none - the shell commands, the admin dashboard wrapper, and the
exclusive locker's initial lease. The locker's release keeps its own
uncancelled context so a slow unlock cannot turn into a ghost lock.
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* wdclient: test that WithClient gives up with the caller's context
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* wdclient: cut the master retry backoff short when the caller gives up
util.Retry sleeps unconditionally between attempts, so a transient error
arriving just before the caller's deadline still cost it a full backoff step.
Use the context-aware util.RetryWithBackoff, the same helper the volume lookup
in this file already uses.
Two call sites went with it: the shell's lock-holder lookup builds its three
second bound before WithClient so it also covers finding the leader, as its
comment already promised, and the filer's post-delete collection cleanup goes
back to an uncancelled context - the entry is already gone, so a caller that
hung up must not leave the collection behind.
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* wdclient: test that a cancel during backoff ends the retry
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* s3: enforce bucket quota on logical size, not un-vacuumed physical size
A bucket full of deleted/overwritten objects awaiting vacuum went
read-only while its live data stayed under quota, because enforcement
used the raw single-copy volume size with garbage included. Subtract
DeletedByteCount via a LogicalSize() helper in the auto-enforce loop,
the s3.bucket.quota.enforce command, and the bucket_size_bytes metric
(labeled logical but counting garbage too). Deleting objects now
relieves quota immediately and enforcement matches the UI usage figure.
* admin: surface bucket read-only state in the S3 buckets UI
Read the read-only flag quota enforcement writes to filer.conf and show
it as a badge in the bucket list and a Status row in the details modal,
so an operator can see why writes are being rejected.
* fix(s3/shell): include EC volumes in bucket size metrics and collection.list
S3 bucket size metrics exported to Prometheus (and fed through
stats.UpdateBucketSizeMetrics) are computed by
collectCollectionInfoFromTopology, which only walked diskInfo.VolumeInfos.
As soon as a volume was encoded to EC it dropped out of every aggregate,
so Grafana showed bucket sizes shrinking while physical disk usage kept
climbing. The shell helper collectCollectionInfo — used by collection.list
and s3.bucket.quota.enforce — had the same gap, with the EC branch left as
a commented-out TODO.
Fold EC shards into both paths using the same approach the admin dashboard
already uses (PR #9093):
- PhysicalSize / Size sum across shard holders: EC shards are node-local
(not replicas), so per-node TotalSize() and MinusParityShards().TotalSize()
sum to the whole-volume physical and logical sizes respectively.
- FileCount is deduped via max across reporters (every shard holder reports
the same .ecx count; a slow node with a not-yet-loaded .ecx reports 0 and
must not pin the aggregate).
- DeleteCount is summed (each delete tombstones exactly one node's .ecj).
- VolumeCount increments once per unique EC volume id.
Adds regression tests covering pure-EC, mixed regular+EC, and the
slow-reporter FileCount dedupe case.
Refs #9086
* Address PR review feedback: EC size helpers, composite key, VolumeCount dedupe
- Add EcShardsTotalSize / EcShardsDataSize helpers in the erasure_coding
package that walk the shard bitmap directly instead of materializing a
ShardsInfo and copying it via MinusParityShards(). Keeps the
DataShardsCount dependency encapsulated in one place and avoids the
per-shard allocation/copy overhead in the metrics hot path.
- Switch shell collectCollectionInfo ecVolumes map to a composite
{collection, volumeId} key, matching the bucket_size_metrics collector
and defending against any cross-collection volume id aliasing.
- Dedupe VolumeCount in shell addToCollection by volume id so regular
volumes aren't counted once per replica presence. Aligns the shell's
collection.list output with the S3 metrics collector and the EC branch,
all of which now report logical volume counts.
- Add unit tests for the new helpers and for the regular-volume
VolumeCount dedupe.
* Parameterize EcShardsDataSize with dataShards for custom EC ratios
Add a dataShards parameter to EcShardsDataSize so forks with per-volume
ratio metadata (e.g. the enterprise data_shards field carried on an
extended VolumeEcShardInformationMessage) can pass the configured value
and get accurate logical sizes under custom EC policies like 6+3 or 16+6.
Passing 0 or a negative value falls back to the upstream DataShardsCount
default, which is correct for the fixed 10+4 layout — so OSS callers in
s3api and shell pass 0 and keep their current behavior.
Added table cases covering the custom 6+3 and 16+6 paths so the
parameterization is pinned by tests.
Chris Lu