The plugin cluster context forwarded filers as gRPC-only addresses
(host:grpcPort). The admin-script worker stored that in
ShellOptions.FilerAddress, whose shell commands re-derive the gRPC port
via ToGrpcAddress() and re-add the +10000 offset, dialing a non-existent
host:28888.
Carry filers in pb.ServerAddress form (host:httpPort.grpcPort) and let
each consumer convert when it dials: the admin shell uses it verbatim,
while the s3_lifecycle and iceberg workers collapse it to a gRPC address.
Rename the proto field filer_grpc_addresses -> filer_addresses so the
name matches the content.
* refactor(s3/lifecycle): drop Per-Run Time Limit knob; use scheduler's Execution Timeout
"Per-Run Time Limit (minutes)" duplicated the admin scheduler's
"Execution Timeout (s)" — both are wall-clock caps on the same
Execute call, stacked via context.WithTimeout. Whichever was shorter
won. Under defaults the scheduler's 90s timeout always clobbered the
worker's 60-min cap, so the "Per-Run Time Limit" knob was effectively
dead unless an operator also raised Execution Timeout, and operators
had to keep two values in agreement.
Remove the worker-side knob and declare a sane scheduler default on
the handler descriptor:
- WorkerConfigForm: nil (was: one section with one field)
- Config.MaxRuntime removed; ParseConfig drops max_runtime_minutes
- Handler no longer wraps ctx in context.WithTimeout(MaxRuntime);
runCtx is just the ctx the scheduler passes
- AdminRuntimeDefaults.ExecutionTimeoutSeconds = 3600 (1h) and
JobTypeMaxRuntimeSeconds = 3600 — the scheduler's global 90s
default would otherwise kill every real run
Tests:
- TestParseConfigDefaults loses the MaxRuntime check; new
TestParseConfigIgnoresWorkerValues documents the contract
- TestDescriptor_WorkerConfigFormIsAbsent pins that the form is gone
so a future re-add forces a conscious revisit
- TestDescriptor_AdminRuntimeDefaultsBoundExecutionTimeout pins the
1h default with a comment about the 90s scheduler floor
* fix(s3/lifecycle): no per-pass timeout by default
Lifecycle is a scheduled batch — its natural duration is "as long as
today's events take." The 1h default ExecutionTimeoutSeconds from the
previous commit was still a footgun: too low truncates legitimate
large-bucket passes; too high makes the value meaningless.
Set both ExecutionTimeoutSeconds and JobTypeMaxRuntimeSeconds to
math.MaxInt32 (~68 years) to say "no timeout in practice" in a
code-review-readable way. Operators who genuinely want a wall-clock
cap can set one in the admin UI; the scheduler's context.WithTimeout
machinery is unchanged (we just hand it an effectively-infinite
duration).
Note: the scheduler floors ExecutionTimeout at 90s
(defaultScheduledExecutionTimeout in weed/admin/plugin/plugin_scheduler.go),
so 0 doesn't mean "unlimited" — it clamps back to 90s. A literal
math.MaxInt32 is the way to express the intent without touching the
shared scheduler code.
Test updated to pin math.MaxInt32 and document the rationale so a
future tighter cap fails the test and forces conscious revisit.
S3 lifecycle is a standard bucket feature — operators set
PutBucketLifecycleConfiguration through the S3 API expecting the
configured expirations to actually fire. With the prior default
(scheduler enabled=false), buckets with lifecycle XML silently
retained data past their declared expiration until an operator
noticed and turned the scheduler on.
The failure mode of enabled-by-default is "worker runs every day
and fast-exits on buckets with no lifecycle rules" — cheap.
The failure mode of disabled-by-default is "data lingers, looks
like it expired, doesn't" — bad. Enabled-by-default matches both
the AWS S3 default behavior and the operator's natural mental
model.
Operators who want the worker off can still disable it via the
admin UI; once a persisted config exists, this descriptor default
no longer applies (the persisted Enabled state wins).
Test pins the choice so a future flip to false fails loud.
feat(s3/lifecycle): remove algorithm flag, daily_replay is the only path (Phase 5a)
With Phase 4b on master the daily_replay path covers every rule kind
and the streaming algorithm has no remaining responsibilities. This
PR retires the algorithm flag from the worker:
- Drop the "Algorithm" enum field from AdminConfigForm and its
DefaultValues entry.
- Drop the if/else routing in Execute — every Execute call now
routes straight into executeDailyReplay.
- Drop the streaming-only worker fields (DispatchTick,
CheckpointTick, RefreshInterval, BootstrapInterval) and their
matching form fields. None of them are read by the daily_replay
path; keeping them in the form would suggest tuning knobs that
don't do anything.
- Drop AlgorithmStreaming / AlgorithmDailyReplay constants and the
Config.Algorithm field.
The streaming-path packages (s3lifecycle/scheduler, s3lifecycle/dispatcher)
remain on the tree; they're now reachable only by the
weed shell s3.lifecycle.run-shard debug command and the few helpers
(LoadCompileInputs, FilerStore, FilerSiblingLister) the daily_replay
worker still uses. Phase 5b deletes the dead code.
Tests prune the cadence-default assertions to the single remaining
field (max_runtime_minutes).
* feat(s3/lifecycle): cluster rate-limit allocation (Phase 3)
Admin computes a per-worker share of cluster_deletes_per_second at
ExecuteJob time and ships it to the worker via
ClusterContext.Metadata. The worker reads the share, constructs a
golang.org/x/time/rate.Limiter, and passes it to dailyrun.Run via
cfg.Limiter (Phase 2 already plumbed the field). Phase 5 deletes the
streaming path; until then streaming ignores the cap.
Why allocate at admin: the cluster cap is a single knob operators
care about. Dividing it locally per worker would either need
out-of-band coordination or accept N× the configured budget. Admin
is the only party that knows how many execute-capable workers there
are, so it owns the math.
Admin side (weed/admin/plugin):
- Registry.CountCapableExecutors(jobType) returns the number of
non-stale workers with CanExecute=true.
- New file cluster_rate_limit.go: decorateClusterContextForJob clones
the input ClusterContext and injects two metadata keys for
s3_lifecycle. cloneClusterContext duplicates Metadata so per-job
decoration doesn't race shared base state.
- executeJobWithExecutor calls the decorator after loading the admin
config; other job types pass through unchanged.
Worker side (weed/worker/tasks/s3_lifecycle):
- New cluster_rate_limit.go declares the constants both sides agree
on (admin-config field names, metadata keys). Plain strings on the
admin side keep weed/admin/plugin free of a dependency on the
s3_lifecycle worker package; the two sets of constants are pinned
to identical values and a mismatch would silently disable rate
limiting.
- handler.go executeDailyReplay reads ClusterContext.Metadata,
builds a rate.Limiter, and passes it into dailyrun.Config{Limiter}.
Missing/empty/non-positive values → no limiter (legacy unlimited
behavior). burst defaults to 2 × rate, clamped to ≥1 to avoid a
bucket that never refills.
- Admin form gains two fields under "Scope": cluster_deletes_per_second
(rate, 0 = unlimited) and cluster_deletes_burst (0 = 2 × rate).
Metric:
- New S3LifecycleDispatchLimiterWaitSeconds histogram observes how
long each Limiter.Wait blocks before a LifecycleDelete RPC.
Operators tune the cap by reading p95 — near-zero means the cap
isn't binding, a long tail at 1/rate means it is.
Tests:
- weed/admin/plugin/cluster_rate_limit_test.go: 9 cases covering
pass-through for non-allocator job types, rps=0 / no-executors
skip, even sharing, burst sharing, burst=0 omit (worker default
kicks in), burst floor of 1, no mutation of input metadata, nil
input.
- weed/worker/tasks/s3_lifecycle/cluster_rate_limit_test.go: 7 cases
covering nil/empty/missing metadata, non-positive/invalid rate,
positive rate builds correctly, burst missing defaults to 2× rate,
tiny rate clamps burst to ≥1.
Build clean. Phase 2 (#9446) and Phase 4 engine (#9447) are the
parents; this branch stacks on Phase 2 since it consumes
dailyrun.Config{Limiter} which lands there.
* fix(s3/lifecycle): divide cluster budget by active workers, not all capable
gemini pointed out that s3_lifecycle has MaxJobsPerDetection=1
(handler.go:189) — it's a singleton job, only one worker is ever active.
Dividing the cluster_deletes_per_second budget by the count of capable
executors gave the single active worker just 1/N of the configured cap.
Pass adminRuntime.MaxJobsPerDetection through to the decorator. Divisor
is now min(executors, maxJobsPerDetection), clamped to >=1. For
s3_lifecycle (maxJobs=1) the active worker gets the full budget; for a
hypothetical parallel-dispatch job (maxJobs>1) the budget divides
across the running-set.
Tests swap the SharedEvenly case for two pinned scenarios:
- SingletonJobGetsFullBudget: maxJobs=1 across 4 executors => 100/1
- SharedEvenlyWhenParallelLimited: maxJobs=4 across 4 executors => 25/worker
- MaxJobsExceedsExecutors: maxJobs=10 across 4 executors => divisor 4
* feat(s3/lifecycle): drop Worker Count knob from admin config form
The "Worker Count" admin field controlled in-process pipeline goroutines
across the 16-shard space — per-worker tuning, not a cluster-wide scope
concern. Operators looking at the form alongside Cluster Delete Rate
reasonably misread it as the number of workers in the cluster.
Drop the form field and DefaultValues entry. cfg.Workers is now hardcoded
to shardPipelineGoroutines (=1) inside ParseConfig; the rest of the
plumbing through dailyrun.Config.Workers stays so a future need can
re-introduce it as a worker-local knob (or just bump the constant).
handler_test.go pins that "workers" must NOT appear in the form so the
removal doesn't silently regress.
Three branches: gRPC error from GetFilerConfiguration must propagate
(else Execute would proceed to dial S3 with an empty buckets path
and never dispatch); a non-empty DirBuckets is honored verbatim so
operators with a non-default layout aren't force-routed to /buckets;
an empty DirBuckets falls back to the documented "/buckets" default
rather than returning empty (which would route to root). stubFilerConfigClient
embeds filer_pb.SeaweedFilerClient so methods other than the one
under test panic if called — keeps the surface narrow.
* test(s3/lifecycle): pin Descriptor structural invariants
Pre-existing handler tests covered Capability and Detect; Descriptor
was previously untested. A drift between the form fields it advertises
and the defaults config.go reads silently breaks the admin UI in two
ways: the form renders blank (admin can't tune) or the worker clamps
to a hardcoded fallback ignoring the admin's edits. The new tests
catch both directions.
Pinned: jobType / DisplayName / Description / DescriptorVersion;
AdminConfigForm exposes a workers field whose default matches
defaultWorkers; WorkerConfigForm has a default and a field for every
cadence knob ParseConfig reads (dispatch_tick / checkpoint_tick /
refresh_interval / bootstrap_interval / max_runtime); AdminRuntime-
Defaults hits a daily cadence with bounded detection timeout and
single job per detection.
* test(s3/lifecycle): tighten Descriptor invariant assertions
Per gemini review on #9401: pin DetectionTimeoutSeconds to its exact
value (60) instead of ">0" so an accidental tweak is caught, and
assert WorkerConfigForm fields are INT64 (matching ParseConfig's
readInt64) so a STRING-type drift can't silently make the worker
ignore admin edits.
7 tests pin the Execute early-return surface that runs without a
filer or S3 dial: nil request / nil Job / nil sender all error;
foreign JobType errors with the offending name in the message; no
S3 endpoints in cluster context errors (Execute is stricter than
Detect — the admin shouldn't have routed the job there); missing
filer_grpc_address parameter errors (proposal must have been tampered
with or dropped); empty JobType is accepted as broadcast routing and
flows through to the next validation step. The dial path itself is
intentionally not covered here — those tests would need an in-process
gRPC server and belong with the integration suite.
* test(s3/lifecycle): cover worker handler Detect + helpers
13 tests pin the worker-handler surface that runs without a live filer
or S3 server. Pure helpers: clusterS3Endpoints (nil context, empty
list, filter empty entries while preserving order, all-valid
passthrough); readString (missing key, nil ConfigValue, wrong kind
falls back, string returned). Capability advertises jobType with
single-job concurrency caps. Detect: nil request / nil sender / wrong
JobType all error; no S3 endpoints emits a 'skipped' activity and
completes with success; no filer addresses behaves the same; the
happy path proposes one job parameterized with the first filer
address; empty JobType is accepted (broadcast detect); a
SendProposals failure propagates without firing complete.
* test(s3/lifecycle): cover SendComplete error propagation in worker Detect
The recordingSender already supported forcing an err on SendComplete
via errOn, but no case exercised it. A SendComplete failure must
propagate so the admin learns the completion signal never landed;
proposals went out before the failure so they remain recorded.
ahalaun