* filer: self-heal fetchWholeChunk on stale volume locations
Upstream #10156/#10800 wired cache invalidation into the buffer-based
read paths, but manifest resolution still goes through fetchWholeChunk,
which returns the raw error on failure. When cached volume locations
are stale (volume tiered to remote storage, server rolled), resolving
a large multipart file fails permanently even though other locations
are healthy.
Thread the ChunkGroup's cacheInvalidator through ResolveChunkManifest /
ResolveOneChunkManifest / fetchWholeChunk, and on failure invalidate,
re-lookup and retry once via the existing retryFetchWithFreshLocations
helper. The streaming bytesBuffer is reset before the retry so partial
bytes from the failed attempt cannot corrupt the manifest
proto.Unmarshal. Non-mount callers pass nil and keep their semantics.
* filer: move the manifest self-heal tests in with the other manifest tests
Also make the stale server stream a prefix and then abort mid-body, which is
what actually leaves partial bytes in the buffer: an HTTP error status returns
before ReadUrlAsStream ever calls the writer, so a 500 never exercised the
Reset the tests claimed to cover.
Claude-Session: https://claude.ai/code/session_01FK3oGC5ZVeJYvNBWgb9JUD
* filer: keep the cached volume locations when a manifest read is cancelled
A cancelled or timed-out read says nothing about where the volume lives, so
dropping the location and going back to the master only costs the next reader
a round trip. PrepareStreamContentWithThrottler already guards its self-heal
this way. The guard also goes inside retryFetchWithFreshLocations, since the
caller can be cancelled between its own check and the invalidation, and that
covers the reader cache and prefetch paths too.
fetchWholeChunk returns the context error rather than the stream failure it
provoked, and ResolveOneChunkManifest wraps with %w so errors.Is still sees it.
That matters even where no invalidator is passed: volume.fsck resolves
manifests with nil and tells its own abort from a corrupt manifest that way,
so the cancellation check sits ahead of the nil-invalidator return.
Claude-Session: https://claude.ai/code/session_01FK3oGC5ZVeJYvNBWgb9JUD
* filer: self-heal manifest reads on the filer and s3 paths too
Every caller that already holds the location cache backing its lookup function
can hand it over: the filer's read, copy and deletion paths and the log cache
have the MasterClient right there, and s3api has the FilerClient. MinusChunks
takes one for the same reason, since the deletion path resolves manifests
through it. Only the shell tools and the replication sinks, whose lookup
functions cache privately with nothing to invalidate, keep passing nil.
Claude-Session: https://claude.ai/code/session_01FK3oGC5ZVeJYvNBWgb9JUD
---------
Co-authored-by: bruce-zzz <bruce.zou@hhy-data.com>
* filer: store TUS sub-chunks through the regular chunk writer
A TUS sub-chunk was written with one assigned file id, retried up to
three times against that same id, and abandoned on failure: an attempt
that had landed on some replicas left a needle no session record and no
entry ever references, unreclaimable by vacuum.
dataToChunkWithSSE, which the regular write path uses per chunk, assigns
a fresh file id per attempt and hands back the file ids of failed
attempts, which are now freed the way the regular write path frees them.
* filer: retry a chunk write on a fresh volume when the server 5xxs
The filer's chunk writer assigns a fresh file id per attempt but only
retried transient network errors, so a volume filling up and turning
read-only mid-write failed the whole request even though the very next
assignment would have landed elsewhere. Every other write client already
routes this through ShouldReassignUpload; the filer's own write path now
does the same, for regular uploads and TUS sub-chunks alike.
* filer: export the chunk deletion queue
The filer test harness in weed/server builds filer.Filer as a struct
literal, so any code path reaching DeleteChunks dereferenced a nil
queue. Exported like the neighboring DeletionRetryQueue so the harness
can arm it.
* filer: complete a TUS upload whose chunk records overlap
A PATCH retried while its predecessor was still storing a sub-chunk -
a proxy timeout with an immediate retry is enough - records the same
range twice. HEAD computes Upload-Offset as the covered watermark and
reported the upload fully received, but completion demanded exactly
adjacent records and failed every attempt: the client concluded success
from offset == length, no entry was created, and the session eventually
expired, turning the entire upload into deleted needles for the vacuum
to chew through.
Completion now validates gapless coverage with the same watermark HEAD
uses. A record extending coverage joins the entry - the read path
resolves partial overlaps by ModifiedTsNs, and the raced copies carry
identical bytes - while a fully covered duplicate is freed once the
entry lands.
* filer: allow one mutating TUS request per session at a time
Nothing stopped two PATCHes from writing the same range concurrently:
both loaded the same offset, both passed the conflict check, and both
recorded their sub-chunks. A client whose request timed out in a proxy
retries immediately while the server side is still storing the buffered
sub-chunk, which is exactly that race.
A session now accepts one PATCH or DELETE at a time, the way tusd locks
uploads; a concurrent one is refused with 423 Locked, which TUS clients
retry, and HEAD keeps answering so progress polling is unaffected. The
chunk state is loaded under the claim, so a retried PATCH sees every
record its predecessor left and conflicts cleanly instead of duplicating
data.
* test: cover a TUS PATCH raced by its own retry
Stalls a PATCH mid-body over a raw connection, retries the same range
while it is in flight, and expects the retry refused with 423 Locked;
the upload then resumes from the reported offset and the final content
must be intact.
* filer: never free a TUS duplicate the entry still references
Coverage is computed from ranges, so a record fully covered by another
is treated as a duplicate no matter which needle it names. A malformed
record naming a file id the entry keeps would have had that needle freed
right after the entry landed - the corruption this change set exists to
stop. The duplicates are now freed in one batch, skipping any file id
the entry references; their records go with the session directory.
* test: bound the raw TUS connection reads
http.ReadResponse on the stalled PATCH's connection blocked until the
whole go test timeout if the filer never answered.
* filer: free the needles of chunk write attempts a retry replaced
A volume server stores the needle locally and only then fans out to the
replicas, so a replication failure 5xxs with the data already written.
Each attempt assigns its own file id, so once a later attempt lands
elsewhere nothing references the earlier ones: the caller only sees the
chunk that succeeded, and the failed ids were dropped.
They are now freed the way the caller frees them when the whole write
fails. Retrying on a 5xx makes this reachable on every read-only or full
volume, which is exactly the condition that filled the reporter's
volumes.
* util: match transient error messages case-insensitively, expose the message form
The same condition reaches different layers capitalized differently -- a
volume server relays its idle timeout as "I/O timeout" inside a JSON string --
and the callers that grew their own substring lists all lower-case first.
Also split out IsTransientErrorMessage for the paths that carry only the text,
such as the per-file status strings in a batch delete response, and pick up
"no route to host" and "network is unreachable" from the gRPC classifier.
* filersink: classify transient network errors through util.IsTransientError
The local list caught i/o timeout, connection reset, and broken pipe but not
connection refused, no such host, unexpected EOF, the syscall errnos, or the
gRPC and S3 overload codes. Keep only the bare io.EOF case, which is transient
here -- a truncated chunk read -- but a clean stream end elsewhere.
* filer deletion: reuse util.IsTransientErrorMessage for the network patterns
Six of the sixteen patterns were already covered. Keep the ones specific to
this pipeline -- read-only volumes, lookup failures, backpressure -- and note
why context cancellation stays retryable here: it decides whether to requeue
the deletion, not whether to retry a call.
* wdclient: fold the shared classifier into the volume lookup retry check
The string tail duplicated the shared list and missed the syscall errnos and
net.Error timeouts. Keep "connection" and "timeout", which are broader than
the shared classifier on purpose: a volume lookup is a cheap read-only call.
* Filer: Add retry mechanism for failed file deletions
Implement a retry queue with exponential backoff for handling transient
deletion failures, particularly when volumes are temporarily read-only.
Key features:
- Automatic retry for retryable errors (read-only volumes, network issues)
- Exponential backoff: 5min → 10min → 20min → ... (max 6 hours)
- Maximum 10 retry attempts per file before giving up
- Separate goroutine processing retry queue every minute
- Enhanced logging with retry/permanent error classification
This addresses the issue where file deletions fail when volumes are
temporarily read-only (tiered volumes, maintenance, etc.) and these
deletions were previously lost.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Update weed/filer/filer_deletion.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Filer: Add retry mechanism for failed file deletions
Implement a retry queue with exponential backoff for handling transient
deletion failures, particularly when volumes are temporarily read-only.
Key features:
- Automatic retry for retryable errors (read-only volumes, network issues)
- Exponential backoff: 5min → 10min → 20min → ... (max 6 hours)
- Maximum 10 retry attempts per file before giving up
- Separate goroutine processing retry queue every minute
- Map-based retry queue for O(1) lookups and deletions
- Enhanced logging with retry/permanent error classification
- Consistent error detail limiting (max 10 total errors logged)
- Graceful shutdown support with quit channel for both processors
This addresses the issue where file deletions fail when volumes are
temporarily read-only (tiered volumes, maintenance, etc.) and these
deletions were previously lost.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Filer: Replace magic numbers with named constants in retry processor
Replace hardcoded values with package-level constants for better
maintainability:
- DeletionRetryPollInterval (1 minute): interval for checking retry queue
- DeletionRetryBatchSize (1000): max items to process per iteration
This improves code readability and makes configuration changes easier.
* Filer: Optimize retry queue with min-heap data structure
Replace map-based retry queue with a min-heap for better scalability
and deterministic ordering.
Performance improvements:
- GetReadyItems: O(N) → O(K log N) where K is items retrieved
- AddOrUpdate: O(1) → O(log N) (acceptable trade-off)
- Early exit when checking ready items (heap top is earliest)
- No full iteration over all items while holding lock
Benefits:
- Deterministic processing order (earliest NextRetryAt first)
- Better scalability for large retry queues (thousands of items)
- Reduced lock contention duration
- Memory efficient (no separate slice reconstruction)
Implementation:
- Min-heap ordered by NextRetryAt using container/heap
- Dual index: heap for ordering + map for O(1) FileId lookups
- heap.Fix() used when updating existing items
- Comprehensive complexity documentation in comments
This addresses the performance bottleneck identified in GetReadyItems
where iterating over the entire map with a write lock could block
other goroutines in high-failure scenarios.
* Filer: Modernize heap interface and improve error handling docs
1. Replace interface{} with any in heap methods
- Addresses modern Go style (Go 1.18+)
- Improves code readability
2. Enhance isRetryableError documentation
- Acknowledge string matching brittleness
- Add comprehensive TODO for future improvements:
* Use HTTP status codes (503, 429, etc.)
* Implement structured error types with errors.Is/As
* Extract gRPC status codes
* Add error wrapping for better context
- Document each error pattern with context
- Add defensive check for empty error strings
Current implementation remains pragmatic for initial release while
documenting a clear path for future robustness improvements. String
matching is acceptable for now but should be replaced with structured
error checking when refactoring the deletion pipeline.
* Filer: Refactor deletion processors for better readability
Extract large callback functions into dedicated private methods to
improve code organization and maintainability.
Changes:
1. Extract processDeletionBatch method
- Handles deletion of a batch of file IDs
- Classifies errors (success, not found, retryable, permanent)
- Manages retry queue additions
- Consolidates logging logic
2. Extract processRetryBatch method
- Handles retry attempts for previously failed deletions
- Processes retry results and updates queue
- Symmetric to processDeletionBatch for consistency
Benefits:
- Main loop functions (loopProcessingDeletion, loopProcessingDeletionRetry)
are now concise and focused on orchestration
- Business logic is separated into testable methods
- Reduced nesting depth improves readability
- Easier to understand control flow at a glance
- Better separation of concerns
The refactored methods follow the single responsibility principle,
making the codebase more maintainable and easier to extend.
* Update weed/filer/filer_deletion.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Filer: Fix critical retry count bug and add comprehensive error patterns
Critical bug fixes from PR review:
1. Fix RetryCount reset bug (CRITICAL)
- Problem: When items are re-queued via AddOrUpdate, RetryCount
resets to 1, breaking exponential backoff
- Solution: Add RequeueForRetry() method that preserves retry state
- Impact: Ensures proper exponential backoff progression
2. Add overflow protection in backoff calculation
- Check shift amount > 63 to prevent bit-shift overflow
- Additional safety: check if delay <= 0 or > MaxRetryDelay
- Protects against arithmetic overflow in extreme cases
3. Expand retryable error patterns
- Added: timeout, deadline exceeded, context canceled
- Added: lookup error/failed (volume discovery issues)
- Added: connection refused, broken pipe (network errors)
- Added: too many requests, service unavailable (backpressure)
- Added: temporarily unavailable, try again (transient errors)
- Added: i/o timeout (network timeouts)
Benefits:
- Retry mechanism now works correctly across restarts
- More robust against edge cases and overflow
- Better coverage of transient failure scenarios
- Improved resilience in high-failure environments
Addresses feedback from CodeRabbit and Gemini Code Assist in PR #7402.
* Filer: Add persistence docs and comprehensive unit tests
Documentation improvements:
1. Document in-memory queue limitation
- Acknowledge that retry queue is volatile (lost on restart)
- Document trade-offs and future persistence options
- Provide clear path for production hardening
- Note eventual consistency through main deletion queue
Unit test coverage:
1. TestDeletionRetryQueue_AddAndRetrieve
- Basic add/retrieve operations
- Verify items not ready before delay elapsed
2. TestDeletionRetryQueue_ExponentialBackoff
- Verify exponential backoff progression (5m→10m→20m→40m→80m)
- Validate delay calculations with timing tolerance
3. TestDeletionRetryQueue_OverflowProtection
- Test high retry counts (60+) that could cause overflow
- Verify capping at MaxRetryDelay
4. TestDeletionRetryQueue_MaxAttemptsReached
- Verify items discarded after MaxRetryAttempts
- Confirm proper queue cleanup
5. TestIsRetryableError
- Comprehensive error pattern coverage
- Test all retryable error types (timeout, connection, lookup, etc.)
- Verify non-retryable errors correctly identified
6. TestDeletionRetryQueue_HeapOrdering
- Verify min-heap property maintained
- Test items processed in NextRetryAt order
- Validate heap.Init() integration
All tests passing. Addresses PR feedback on testing requirements.
* Filer: Add code quality improvements for deletion retry
Address PR feedback with minor optimizations:
- Add MaxLoggedErrorDetails constant (replaces magic number 10)
- Pre-allocate slices and maps in processRetryBatch for efficiency
- Improve log message formatting to use constant
These changes improve code maintainability and runtime performance
without altering functionality.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* refactoring retrying
* use constant
* assert
* address comment
* refactor
* address comments
* dedup
* process retried deletions
* address comment
* check in-flight items also; dedup code
* refactoring
* refactoring
* simplify
* reset heap
* more efficient
* add DeletionBatchSize as a constant;Permanent > Retryable > Success > Not Found
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: chrislu <chris.lu@gmail.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* batch deletion operations to return individual error results
Modify batch deletion operations to return individual error results instead of one aggregated error, enabling better tracking of which specific files failed to delete (helping reduce orphan file issues).
* Simplified logging logic
* Optimized nested loop
* handles the edge case where the RPC succeeds but connection cleanup fails
* simplify
* simplify
* ignore 'not found' errors here
