mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-19 12:34:29 +02:00
Chris Lu
443ae1cd83
* s3 lifecycle: share the daily bucket walk across shards The daily pass invoked the walker once per shard, and every shard's walk listed the full bucket tree while keeping only its own 1/16 of the entries — 16 full listings per pass by default (#11791). Run now groups the shards by the walk branch they'll take (recovery vs steady, using the same cursor data and hash checks runShard applies) and starts one shared walk per group, so a pass lists each bucket at most twice. WalkerFunc and WalkBuckets take a shard set instead of a single shard. runShard keeps its own decision and waits on its group's walk; a shard whose freshly-loaded cursor disagrees with the plan falls back to a per-shard walk, so mid-pass cursor rewrites can't under-cover a shard. * s3 lifecycle: skip the .versions/ pre-pass on non-versioned buckets walkBucketTree listed every directory twice — once looking for .versions/ dirs, once for regular entries. On a bucket that never had versioning enabled the first pass can only find coincidentally-named user folders, so the extra listing is pure cost (#11791). FilerListFunc now takes the walk view and consults Snapshot.BucketVersioned; non-versioned buckets skip pass 1 entirely. A *-suffixed .versions user folder on such a bucket is recursed into as a regular directory so real objects inside still surface; a nil view keeps the two-pass behavior. * s3 lifecycle: prune walk directories to active rule prefixes The daily walk listed every directory under a bucket even when no rule prefix could reach it. Collect the active rule prefixes of each bucket from the walk view (new Snapshot.BucketPrefixes) and skip recursing into directories whose keys can never match; a bucket with no active actions is skipped entirely, saving the listing. .uploads stays in scope regardless of prefixes because AbortMPU rules match DestKey, and .versions expansion is pruned only conservatively (the logical key is a string-prefix of the dir key). * s3 lifecycle: join planned walks before Run returns planWalks starts a shared walk goroutine per group, but a shard whose authoritative cursor load then fails exits runShard without waiting on it. Run returned while the walk still listed or dispatched deletes against connections the caller closes on return. Join every planned walk after the shard waitgroup. * s3 lifecycle: treat object-locked buckets as versioned in the walk Object Lock requires versioning and every writer sets both extended attributes, but IsBucketVersioned only read ExtVersioningKey. A locked bucket missing that attribute would have its .versions dirs walked as ordinary folders, hiding version siblings from lifecycle. Classify ExtObjectLockEnabledKey as versioned too.