4 Commits

Include renames

Author	SHA1	Message	Date
Chris Lu	1854101125	feat(s3/lifecycle): bootstrap re-walk cadence + operator hooks (Phase 8) (#9386) ... * feat(s3/lifecycle): bootstrap re-walk cadence + operator hooks (Phase 8) scan_only actions only fire from the bootstrap walk: the engine classifies a rule as scan_only when its retention horizon exceeds the meta-log retention, so event-driven routing can't be trusted. Today each bucket walks once per process, so a long-running worker never revisits — scan_only retention only catches up when the worker restarts. Replace BucketBootstrapper.known (set) with BucketBootstrapper.lastWalk (name -> completion time). KickOffNew now re-walks a bucket whose last walk completed more than BootstrapInterval ago. Zero interval preserves the legacy walk-once-per-process behavior so existing deployments don't change cadence by default. walkBucket re-stamps on success and clears the stamp on failure (via MarkDirty), so the next KickOffNew picks failed walks back up. Add MarkDirty / MarkAllDirty operator hooks for forced re-walks, and a Now func() for testable time travel. weed shell run-shard grows --bootstrap-interval (cadence knob) and --force-bootstrap (drop in-memory state at startup so every bucket walks again immediately, useful when a config change should take effect without a restart). Tests: cadence respected (skip inside interval, re-walk past it); zero interval keeps once-per-process; MarkDirty forces re-walk under a 24h interval; MarkAllDirty resets every record. The fakeClock helper guards the test clock with a mutex so race-detector runs are clean. * fix(s3/lifecycle): split walk state, thread BootstrapInterval through worker, drop dead flag Three issues with the Phase 8 cadence work as it landed: 1. lastWalk did double duty as both completed-walk timestamp and in-flight debounce. A walk that took longer than BootstrapInterval would have a fresh KickOffNew start a duplicate goroutine on the next refresh tick because the stamp from KickOffNew looked stale against the interval. Split into lastCompleted (set on success) and inFlight (set on dispatch, cleared after the walk goroutine returns success or failure). KickOffNew skips inFlight buckets regardless of cadence. 2. The cadence knob existed on `weed shell` but not on the production path: scheduler.Scheduler constructed BucketBootstrapper without BootstrapInterval, and weed/worker/tasks/s3_lifecycle/Config had no field for it. Add Scheduler.BootstrapInterval, parse `bootstrap_interval_minutes` in ParseConfig (zero = legacy walk- once-per-process; negative clamps to zero), and forward it from the handler. Tests cover default, override, clamp, and explicit-zero. 3. --force-bootstrap was a no-op: BucketBootstrapper is freshly allocated at command start, so MarkAllDirty on empty state does nothing, and the flag couldn't influence an already-running process anyway. Remove it; a real runtime trigger (SIGHUP, control RPC) is a separate change. In-flight regression: a blockingInjector pins the first walk in progress while the test advances the clock past the interval. The second KickOffNew is a no-op (inFlight check). After release, the post-completion KickOffNew within the interval is also a no-op. * test(s3/lifecycle): wait for lastCompleted stamp before advancing fake clock The cadence test polled listedN to know "the walk happened" — but that fires once both list passes are issued, while the success-stamp lands later, after walkBucketDir returns. A clock.Advance(30m) between those two events would record the stamp at clock+30m instead of T0; the next assertion would then see now.Sub(last) < 1h and skip the expected re-walk. Tight in practice but exposed under -race / load. Add a waitForCompleted helper that polls b.lastCompleted directly, and use it before each clock advance in both the cadence and zero- interval tests. * fix(s3/lifecycle): expose bootstrap interval in worker UI; honor MarkDirty during walks Two follow-ups on Phase 8. The worker config descriptor had no bootstrap_interval_minutes field, so the production operator UI couldn't enable the cadence — only the internal ParseConfig + Scheduler wiring knew about it. Add the field to the cadence section (MinValue=0 since 0 is the legacy default) and include the default in DefaultValues so existing deployments see the knob with the right preset. MarkDirty / MarkAllDirty silently lost their effect when a walk was in flight: the methods cleared lastCompleted, but the walk's success path then wrote a fresh timestamp, hiding the operator's invalidation. Track a pendingDirty set; the walk goroutine consumes the flag on exit and skips the success stamp, so the next KickOffNew picks the bucket up immediately. Regression: pin a walk in progress with a blockingInjector, MarkDirty the bucket, release the walk, and assert lastCompleted stayed empty plus the next KickOffNew triggers a new walk inside the BootstrapInterval window. * refactor(s3/lifecycle): drop unused MarkDirty / MarkAllDirty + pendingDirty These methods were the operator-hook half of Phase 8, but the only caller (--force-bootstrap on the shell command) was removed when it turned out to be a no-op against a freshly-allocated bootstrapper. Nothing in production calls them anymore. Strip the dead surface: MarkDirty, MarkAllDirty, the pendingDirty set, the dirty-suppression branch in walkBucket, and the three tests that only exercised those methods. BootstrapInterval-driven re-bootstrap is the live mechanism. A real runtime trigger (SIGHUP, control RPC) is a separate change with a real call site.	2026-05-09 13:42:31 -07:00
Chris Lu	edfa1ce210	feat(s3/lifecycle): pointer-transition routing for live PUTs (Phase 5b/4) (#9385) ... * feat(s3/lifecycle): pointer-transition routing for live PUTs (Phase 5b/4) Bootstrap covers existing versions, but a live PUT that creates a new .versions/<v-new> file and updates the parent's ExtLatestVersionIdKey didn't fire NoncurrentDays / NewerNoncurrent on the displaced prior version until the next bootstrap. Close that runtime gap. The meta-log already emits an Update event for the .versions/ directory itself when the latest pointer changes; the router was dropping it because buildObjectInfo returns nil for directories. New branch in Route detects that shape (versioned bucket, NewEntry + OldEntry both directories with the .versions/ suffix, ExtLatestVersionIdKey changed, ID different from the new ID) and emits a Match against the LOGICAL key with VersionID=oldID. Match.Identity comes from a single LookupVersion RPC for the displaced version file; SuccessorModTime is the directory update's mtime, which is the moment the displaced version became noncurrent. SiblingLister grows LookupVersion(bucket, key, versionID) for that single-RPC fetch. filerSiblingLister implements it; routing path treats NotFound as "displaced version was hard-deleted in the meantime, suppress" rather than an error. The router gates the lookup on at least one active event-driven NoncurrentDays / NewerNoncurrent rule for the bucket, so most buckets pay nothing per directory update. Tests: pointer-flip fires NoncurrentDays with displaced version_id; unchanged pointer skips; empty old pointer skips (first-PUT scenario); displaced-version NotFound suppresses; no-rule skips lookup; NewerNoncurrentVersions retains rank-0; unversioned bucket skips. * fix(s3/lifecycle): SuccessorModTime cache + NewerNoncurrent expansion Two correctness gaps in pointer-transition routing. The .versions/ directory's own Attributes.Mtime is preserved across pointer updates by updateLatestVersionInDirectory: it's a stale clock relative to the freshly-written latest version. Using it as the displaced version's SuccessorModTime made NoncurrentDays compute due = staleMtime + days, which fires immediately on a fresh PUT into an old .versions/ container. Read ExtLatestVersionMtimeKey written by setCachedListMetadata; suppress (return no matches) when the cache is missing rather than fall back to dir mtime. Single-oldID lookup is only enough for pure NoncurrentVersionExpirationDays. Any rule with NewerNoncurrentVersions > 0 cares about the noncurrent ranks, and a pointer flip shifts every prior noncurrent's index by one — the version that just crossed the keep-count threshold needs to be evaluated too. When any matching rule needs ranks, list the full .versions/ container, sort newest- first with mtime + version-id tiebreak, and route every noncurrent with its real index. Identity-CAS dedups against earlier schedules. SiblingLister grows ListVersions(bucket, key); filerSiblingLister's implementation paginates the container fully. Two regression tests: stale dir mtime + correct cached mtime schedules ~30 days out (not immediate); NewerNoncurrentVersions=2 with 4 versions fires on the rank-2 entry that just crossed the threshold while rank-0/1 are retained. * fix(s3/lifecycle): bound pointer-transition expansion to threshold crossings routePointerTransitionExpand emitted a Match for every eligible noncurrent on every PUT. Schedule.Add doesn't dedup, so identity-CAS at dispatch only saved the wasted RPC, not the heap slot. A hot key with many already-eligible versions and a count rule would push O(versions) entries per flip, repeatedly, until dispatch caught up. Bound the emission to versions that newly entered eligibility on this specific flip: rank 0 (the displaced version, for the NoncurrentDays clock) plus rank == rule.NewerNoncurrentVersions for each active count-gated rule (the version that just crossed from kept to expired). Bootstrap still owns full backfill for versions that were already over-threshold. Adds a regression with 6 versions and NewerNoncurrentVersions=2: asserts only the rank-2 entry that just crossed fires, not the already-over-threshold rank-3/rank-4 entries. * fix(s3/lifecycle): suppress pointer-transition expansion when newID missing routePointerTransitionExpand defaulted latestPos to 0 if newID wasn't found in the listing. That made the actual newest sibling latest against the pointer's intent, then misranked every other version. A race between the pointer write and the version write could land us there. Default latestPos to -1, set it only on a real match, and suppress the expansion when the search misses. Bootstrap repairs state on the next walk. The NewerNoncurrentVersions retention test was setting only lookupEntry, so Route never reached the expansion path it claimed to exercise. Repoint to listVersions and assert ListVersions was consulted while LookupVersion was not. Adds a regression covering the missing-newID suppression directly. * fix(s3/lifecycle): include bare null version in pointer-transition routing Bootstrap models the bare-key object as a "null" sibling alongside .versions/ children, but the live pointer-transition path didn't. Two cases lost: 1. oldID == "" was treated as "nothing displaced". A pre-versioning bare object becomes noncurrent when the first versioned PUT lands and the pointer flips to a real id, but live routing skipped it and waited for the next bootstrap. 2. The expansion path's ListVersions returned only .versions/ children. With a bare null in the picture, the noncurrent ranks were wrong, so NewerNoncurrentVersions could keep the wrong versions and delete the right ones (or vice versa). SiblingLister grows LookupNullVersion(bucket, key) returning the bare entry plus an explicit-null flag (matches the bootstrap shape). filerSiblingLister implements it via util.NewFullPath + filer_pb.LookupEntry. routePointerTransitionDisplaced: oldID == "" now consults LookupNullVersion. When the bare entry exists, route it as VersionID="null" against the LOGICAL key. routePointerTransitionExpand: collect .versions/ children plus the null entry into one sibling slice before sorting and ranking. The threshold-crossing logic now sees the same N-version set that bootstrap would compute. Three new tests: oldID == "" with no null is a no-op (one null lookup, no version lookup); oldID == "" with a bare null schedules NoncurrentDays as VersionID="null"; expansion with a bare null between .versions/ siblings places null at its mtime-correct rank and only that rank-N entry fires. * fix(s3/lifecycle): atomic listPageSize so test cleanup doesn't race KickOffNew dispatches walks via `go b.walkBucket(...)`. A test that finishes before its goroutines drain leaves them running into the next test's t.Cleanup, which mutates listPageSize. -race spots the read/write collision intermittently. Convert listPageSize to atomic.Uint32; tests use Load/Store. No production semantics change. * fix(s3/lifecycle): null becomes latest when suspended PUT clears pointer The router treated newID == "" as if the cached ExtLatestVersionMtimeKey were still authoritative — but that cache holds the displaced version's mtime, written by setCachedListMetadata when the prior version became latest. Using it as SuccessorModTime made NoncurrentDays=30 immediately fire on a 100-day-old displaced version even though it became noncurrent today. When newID == "" the bare null is the new latest. Look it up, substitute its mtime as the successor clock, and substitute "null" as the latestPos target for the expansion path's id match. Both displaced and expand paths now derive the right clock. updateIsLatestFlagsForSuspendedVersioning was the upstream cause of the staleness — it cleared ExtLatestVersionIdKey and FileNameKey but left the cached size/mtime/etag/owner/delete-marker behind. Call clearCachedVersionMetadata so the .versions/ container is consistent with "null is latest". The router-side guard is still needed for older deployments that ran the buggy code, but new writes won't exercise the workaround. Two regressions: 100-day-old displaced under NoncurrentDays=30 with a today-null PUT schedules ~30d out (not immediate); same shape with NewerNoncurrentVersions=2 ranks the null at latest and only the rank-2 entry fires.	2026-05-09 12:21:35 -07:00
Chris Lu	2f7ac1d664	feat(s3/lifecycle): NoncurrentVersionExpiration via bootstrap (Phase 5b/3) (#9383) ... * feat(s3/lifecycle): NoncurrentVersionExpiration via bootstrap (Phase 5b/3) Bootstrap now expands every <key>.versions/ directory into one event per version with sibling state pre-computed. The router fires NoncurrentDays / NewerNoncurrent off these events using SuccessorModTime as the noncurrent clock; previously these rules never ran on a versioned bucket because buildObjectInfo couldn't classify version-folder events without the latest pointer. Mechanics walkBucketDir treats a directory ending in .versions and carrying ExtLatestVersionIdKey as a SeaweedFS .versions container — emit it once and skip the recursion. Coincidentally-named directories without the latest pointer recurse normally. BucketBootstrapper.expandVersionsDir lists the children, sorts newest-first by mtime, resolves the latest position from the pointer, and injects a synthesized reader.Event per version with BootstrapVersion populated. NoncurrentIndex is 0-based among noncurrents in newest-first order; SuccessorModTime is the immediate newer sibling's mtime (zero for the latest). Pointer naming a missing or absent version falls back to the newest-by-mtime sibling so a race window can't flag every entry as noncurrent. routeBootstrapVersion uses BootstrapVersion to build ObjectInfo directly (bypassing the version-folder skip in buildObjectInfo) and runs the standard match loop. ABORT_MPU is excluded by kind-shape gate. The schedule clock uses SuccessorModTime for noncurrents and ModTime for the latest, so the dispatcher fires when the rule's days threshold is met. Match.ObjectKey is the LOGICAL key, Match.VersionID is the marker's stored version_id — the dispatcher reaches deleteSpecificObjectVersion or createDeleteMarker correctly. Layer 2 tests cover both sides. Router: latest fires ExpirationDays; noncurrent fires NoncurrentDays; NewerNoncurrentVersions retains the N newest noncurrents; ABORT_MPU never matches. Bootstrap: .versions dir emitted once and not recursed; missing latest pointer falls back to newest; backdated PUT (latest pointer is older by mtime) keeps the right noncurrent index; delete-marker flag propagates. * fix(s3/lifecycle): no VersionID for latest expirations, child-based dir disambig Two correctness gaps in Phase 5b/3. Bootstrap was pinning the version_id on every Match. For EXPIRATION_DAYS / EXPIRATION_DATE on the latest version this is unsafe: between schedule and dispatch a fresh PUT can land, the dispatcher would still identity-match against the original version's bytes (it still exists at that path) and the resulting delete marker would hide the new latest. Drop VersionID for those kinds; an empty VersionID makes the dispatcher fetch the current latest, where identity-CAS resolves to STALE_IDENTITY and bootstrap re-schedules with the new latest's identity. NoncurrentDays / NewerNoncurrent / EXPIRED_DELETE_MARKER still pin the version_id since those are version-targeted. isVersionsDir gating on ExtLatestVersionIdKey lost a race window: createDeleteMarker writes the version file before updating the parent's Extended pointer, so a walk between those two steps would see a .versions/ dir without the pointer, recurse into it, and emit raw version files that the router drops. Match the suffix only and let expandVersionsDir disambiguate by child inspection: if any child carries ExtVersionIdKey it's a real .versions container and we expand; otherwise it's a coincidentally-named user folder and we recurse via the bucket-walk's own callback so nested entries still flow through. Tests: latest-expiration assertion flipped to expect empty VersionID; new tests cover the coincidentally-named-folder recursion and the race-window expansion (children present, pointer absent). * fix(s3/lifecycle): filter directory + missing-version-id children at listing expandVersionsDir's listing callback collected every child with attributes; subdirectories or entries without ExtVersionIdKey would make it past the empty-id skip in the inner loop but still inflate NumVersions and skew NoncurrentIndex (the rank derives from the filtered slice's position, which was wrong when the unfiltered slice was sorted). Drop directories at listing time and partition the file children into a versions slice that's the actual rank source. Test cleanups: out-of-order-mtime test now sets v1 older than v2 so latestPos > 0 actually exercises the rank-skip branch in expandVersionsDir; bootstrapVersionEntry preserves nanosecond precision via MtimeNs to match markerLoneEntry's pattern; drop a leftover unused idx variable. * fix(s3/lifecycle): null version + canonical version-id tiebreak Two correctness gaps in Phase 5b/3 bootstrap. Null versions live at the bare logical path, not under .versions/. Bootstrap previously expanded only .versions/<key>/ children, so: - pre-versioning objects with newer .versions/ history never had their null version expired by NoncurrentDays - suspended-bucket writes (which clear the .versions/ latest pointer so null becomes current) had every .versions/ child wrongly classified as latest by the buildObjectInfo fallback expandVersionsDir now looks up the bare key via NewFullPath + LookupEntry, accepts a regular file or an explicit S3 directory-key marker (Mime set), and folds it into the sibling set with VersionID="null". Latest resolution: pointer present + names a real id wins; pointer absent + null exists makes null latest; otherwise falls back to newest sibling. The walker's regular emission for the bare entry would otherwise duplicate, so walkBucketDir now does a two-pass walk per directory level — .versions/ first, then everything else with a per-walk skipBare set keyed by bucket-relative path that expandVersionsDir populates when it claims a bare null sibling. Sort tiebreak: PUTs only set second-level Mtime, so two versions written in the same second tied. The unstable secondary order let old-format version filenames sort oldest-first and corrupt NoncurrentIndex under NewerNoncurrentVersions retention. Add CompareVersionIds to s3lifecycle/version_time.go (mirrors the canonical comparator in s3api/s3api_version_id.go to avoid the import cycle) and use it as a secondary key after mtime equality. Tests: pre-versioning null-as-noncurrent, suspended null-as-current, directory-key marker as null version, end-to-end claim through walkBucketDir's two-pass ordering, and same-second tiebreak via canonical version-id ordering. fakeFilerClient grows a LookupDirectoryEntry implementation backed by the same in-memory tree. * fix(s3/lifecycle): only treat explicit-null bare entries as current The pointer-missing branch in expandVersionsDir made null latest as soon as a bare object was found. That's correct for suspended-bucket writes (s3api_object_handlers_put.go writes the bare entry with ExtVersionIdKey="null") but wrong for the pre-versioning race window: a brand-new version under .versions/<file> exists before the parent's ExtLatestVersionIdKey update lands, and a pre-versioning bare object has no version-id marker. Marking that older bare object latest hides the real new version and skips noncurrent expiration of the null until the next process restart/bootstrap. Distinguish the two: lookupNullVersion now returns whether the bare entry's Extended map carries ExtVersionIdKey="null" (the suspended write marker). expandVersionsDir's pointer-missing branch only promotes null to latest when explicit; otherwise it falls back to newest-sibling, which is safe for the race window since the new version's mtime is fresher than the bare object's. The existing suspended-null test now uses a new helper that adds the explicit marker. New regression test covers the race window: bare entry without the marker + a fresh .versions/<v1> file + missing parent pointer must keep v1 as latest and the null as noncurrent. * fix(s3/lifecycle): only the newest item can be the explicit-null latest The pointer-missing branch in expandVersionsDir scanned every item for an explicit null and promoted it to latest. After a suspended->enabled transition that's the wrong call: createVersion writes the version file before updating ExtLatestVersionIdKey, so a bootstrap that lands in the race window sees an older bare null with ExtVersionIdKey="null" plus a newer .versions/<v-new> child and no parent pointer. Promoting the null misclassifies v-new as noncurrent and skips both the new version's current-version expiration and the null's noncurrent scheduling until the next bootstrap. Constrain the explicit-null branch to items[0]: if the suspended-null write is genuinely current it'll be the newest by mtime AND tagged. Anything else falls through to the newest-sibling default. Adds a regression test for the suspended->re-enabled race. * fix(s3/lifecycle): paginate bootstrap directory listings SeaweedList(..., limit=0) is a single-page request: the filer caps limit=0 at DirListingLimit (1000 by default) and returns whatever fits in one round trip. expandVersionsDir and walkBucketDir both relied on that, so any directory bigger than the cap silently truncated. For noncurrent retention this is correctness, not just scale — a hot key with more versions than the cap had its rank/sort math computed off the first page only, NumVersions, NoncurrentIndex, SuccessorModTime, and the latest-fallback all wrong, with the older versions never scheduled until a future bootstrap. Add a listAll helper that drives pagination via StartFromFileName + inclusive=false, looping until a page returns fewer entries than the configured page size. Use it in both call sites. Page size is a var (listPageSize, default 1024) so tests can shrink it without generating thousands of entries. The fake filer client now mirrors the real semantics: sort children by name, honor StartFromFileName/InclusiveStartFrom, cap at Limit. New regression tests force a small page size and assert the full result set is processed and the call count matches what pagination should drive. * perf(s3/lifecycle): stream bucket walk in two passes instead of buffering walkBucketDir was paginating into a children slice and then iterating twice (pass 1: .versions/, pass 2: everything else). For flat buckets with millions of entries the buffer is a real memory spike. Drop the materialization: each pass now drives its own listAll over the same directory and acts on entries as they stream in. The skipBare ordering contract is preserved — pass 2 still runs after pass 1 finishes — and the per-pass paging keeps memory bounded by listPageSize. Tradeoff: each directory level is listed twice. For workloads where that matters more than the memory headroom, we can revisit; the correctness/scale dial here is what the noncurrent rules need. Updated three tests for the new call count: each walk now records 2 listings per directory (pass 1 + pass 2). The KickOffNew dedup tests expect 2 calls per bucket; the pagination test expects 6 instead of 3.	2026-05-09 10:48:32 -07:00
Chris Lu	196c41d21a	test(s3/lifecycle): cover scheduler/bootstrap walker + MPU detection (#9380) ... Locks down isMPUInitDir, walkBucketDir (regular/nested/MPU/error/cancel paths), and BucketBootstrapper.KickOffNew (per-bucket fanout and in-process dedup) against a fake SeaweedFilerClient.	2026-05-08 22:14:45 -07:00