* wdclient: bound the wait for a master leader by the caller's context
WithClient waited on GetMaster with context.Background(), so a caller that
arrived while no master leader was known parked in a 200ms poll loop until one
appeared, whatever deadline it had already set on the RPC. Each retry above it
then left another goroutine in the same wait.
Take the context in WithClient and WithClientCustomGetMaster and hand it to
GetMaster, and stop the retry loop once it is done. The dial keeps
context.Background(): fn brings its own RPC context, so a cancellation seen
here cannot be attributed to the shared connection.
Call sites pass whatever they hold: the request context in the filer's
CollectionList, DeleteCollection and Statistics handlers and in the credential
store's propagation, the operation context in the shell's s3.bucket.delete and
the kafka gateway's broker and filer discovery, and context.Background() where
there is none - the shell commands, the admin dashboard wrapper, and the
exclusive locker's initial lease. The locker's release keeps its own
uncancelled context so a slow unlock cannot turn into a ghost lock.
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* wdclient: test that WithClient gives up with the caller's context
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* wdclient: cut the master retry backoff short when the caller gives up
util.Retry sleeps unconditionally between attempts, so a transient error
arriving just before the caller's deadline still cost it a full backoff step.
Use the context-aware util.RetryWithBackoff, the same helper the volume lookup
in this file already uses.
Two call sites went with it: the shell's lock-holder lookup builds its three
second bound before WithClient so it also covers finding the leader, as its
comment already promised, and the filer's post-delete collection cleanup goes
back to an uncancelled context - the entry is already gone, so a caller that
hung up must not leave the collection behind.
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* wdclient: test that a cancel during backoff ends the retry
Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU
* fix: admin UI bucket delete now properly deletes collection and checks Object Lock
Fixes#7711
The admin UI's DeleteS3Bucket function was missing two critical behaviors:
1. It did not delete the collection from the master (unlike s3.bucket.delete
shell command), leaving orphaned volume data that caused fs.verify errors.
2. It did not check for Object Lock protections before deletion, potentially
allowing deletion of buckets with locked objects.
Changes:
- Add shared Object Lock checking utilities to object_lock_utils.go:
- EntryHasActiveLock: standalone function to check if an entry has active lock
- HasObjectsWithActiveLocks: shared function to scan bucket for locked objects
- Refactor S3 API entryHasActiveLock to use shared EntryHasActiveLock function
- Update admin UI DeleteS3Bucket to:
- Check Object Lock using shared HasObjectsWithActiveLocks utility
- Delete the collection before deleting filer entries (matching s3.bucket.delete)
* refactor: S3 API uses shared Object Lock utilities
Removes 114 lines of duplicated code from s3api_bucket_handlers.go by
having hasObjectsWithActiveLocks delegate to the shared HasObjectsWithActiveLocks
function in object_lock_utils.go.
Now both S3 API and Admin UI use the same shared utilities:
- EntryHasActiveLock
- HasObjectsWithActiveLocks
- recursivelyCheckLocksWithClient
- checkVersionsForLocksWithClient
* feat: s3.bucket.delete shell command now checks Object Lock
Add Object Lock protection to the s3.bucket.delete shell command.
If the bucket has Object Lock enabled and contains objects with active
retention or legal hold, deletion is prevented.
Also refactors Object Lock checking utilities into a new s3_objectlock
package to avoid import cycles between shell, s3api, and admin packages.
All three components now share the same logic:
- S3 API (DeleteBucketHandler)
- Admin UI (DeleteS3Bucket)
- Shell command (s3.bucket.delete)
* refactor: unified Object Lock checking and consistent deletion parameters
1. Add CheckBucketForLockedObjects() - a unified function that combines:
- Bucket entry lookup
- Object Lock enabled check
- Scan for locked objects
2. All three components now use this single function:
- S3 API (via s3api.CheckBucketForLockedObjects)
- Admin UI (via s3api.CheckBucketForLockedObjects)
- Shell command (via s3_objectlock.CheckBucketForLockedObjects)
3. Aligned deletion parameters across all components:
- isDeleteData: false (collection already deleted separately)
- isRecursive: true
- ignoreRecursiveError: true
* fix: properly handle non-EOF errors in Recv() loops
The Recv() loops in recursivelyCheckLocksWithClient and
checkVersionsForLocksWithClient were breaking on any error, which
could hide real stream errors and incorrectly report 'no locks found'.
Now:
- io.EOF: break loop (normal end of stream)
- any other error: return it so caller knows the stream failed
* fix: address PR review comments
1. Add path traversal protection - validate entry names before building
subdirectory paths. Skip entries with empty names, '.', '..', or
containing path separators.
2. Use exact match for .versions folder instead of HasSuffix() to avoid
mismatching unrelated directories like 'foo.versions'.
3. Replace path.Join with simple string concatenation since we now
validate entry names.
* refactor: extract paginateEntries helper to reduce duplication
The recursivelyCheckLocksWithClient and checkVersionsForLocksWithClient
functions shared significant structural similarity. Extracted a generic
paginateEntries helper that:
- Handles pagination logic (lastFileName tracking, Limit)
- Handles stream receiving with proper EOF vs error handling
- Validates entry names (path traversal protection)
- Calls a processEntry callback for business logic
This centralizes pagination logic and makes the code more maintainable.
* feat: add context propagation for timeout and cancellation support
All Object Lock checking functions now accept context.Context parameter:
- paginateEntries(ctx, client, dir, processEntry)
- recursivelyCheckLocksWithClient(ctx, client, dir, hasLocks, currentTime)
- checkVersionsForLocksWithClient(ctx, client, versionsDir, hasLocks, currentTime)
- HasObjectsWithActiveLocks(ctx, client, bucketPath)
- CheckBucketForLockedObjects(ctx, client, bucketsPath, bucketName)
This enables:
- Timeout support for large bucket scans
- Cancellation propagation from HTTP requests
- The S3 API handler now uses r.Context() for proper request lifecycle
* fix: address PR review comments
1. Add DefaultBucketsPath constant in admin_server.go instead of
hardcoding "/buckets" in multiple places.
2. Add defensive normalization in EntryHasActiveLock:
- TrimSpace to handle whitespace around values
- ToUpper for case-insensitive comparison of legal hold and
retention mode values
- TrimSpace on retention date before parsing
* fix: use ctx variable consistently instead of context.Background()
In both DeleteS3Bucket and command_s3_bucket_delete, use the ctx
variable defined at the start of the function for all gRPC calls
instead of creating new context.Background() instances.
Chris Lu