Files
seaweedfs/weed/command/scaffold/admin.toml
T
b2dd93eac8 admin: support [maintenance.s3_lifecycle] in admin.toml (#12661)
* admin: support [maintenance.s3_lifecycle] in admin.toml

The admin.toml overlay covers vacuum, volume_balance and erasure_coding but
not s3_lifecycle, so the bucket lifecycle job's settings can only be changed
from the web UI and are lost whenever the admin data directory does not
persist. That is the worst combination for this job type: its
meta_log_retention_days knob decides whether bucket rules ever apply to
objects written before the job was enabled, and silently reverting it to the
default makes the rules no-op without any error.

Wire the four admin-side knobs into the existing overlay:

  meta_log_retention_days    retention window the meta-log replay can reach
  walker_interval_minutes    minimum gap between full bucket walks
  cluster_deletes_per_second cluster-wide delete budget
  cluster_deletes_burst      token-bucket burst for that budget

All four are admin config values, not worker values: the engine partition is
computed on the admin side and the delete budget is divided across
execute-capable workers at dispatch time. enabled/retry_limit/
retry_backoff_seconds come for free from the shared section handling.

Tested with TestApplyPluginConfigFromTomlS3Lifecycle, which also pins the
retention=0 case: 0 means unbounded and must survive as an explicit 0 rather
than being treated as unset.

* admin: fix meta_log_retention_days guidance in admin.toml scaffold

Both reviewers caught a real error. `0` is not "unbounded": runShard does

    retentionWindow := cfg.RetentionWindow
    if retentionWindow <= 0 {
        retentionWindow = maxTTL
    }

so zero falls back to the longest rule TTL, which keeps every replay rule in
replay and promotes nothing to the walk set. Recommending 0 to force full
scans was wrong, and so was the claim that pre-existing objects are only
reachable through the walk set -- the cold-start recovery walk covers them
regardless of the retention value.

Describe zero as "no retention info supplied" and tell operators to set the
window the filer meta-log actually retains. The code change is unaffected:
it only wires the four keys into the existing overlay.

* admin: tie s3_lifecycle walk-set scans to walker_interval_minutes in admin.toml

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: show the s3_lifecycle retry defaults in admin.toml

The s3_lifecycle descriptor sets no retry limit or backoff, so a failed
run is not retried and the scheduler's 5 s backoff applies; the scaffold
listed the vacuum values instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: shorten the s3_lifecycle toml section comment

The retention and walker knobs are read by the worker from the admin
values, not computed on the admin side.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: cover cluster_deletes_burst in the s3_lifecycle toml test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* admin: check the s3_lifecycle retention zero is a stored int64

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 13:46:14 -08:00

102 lines
4.0 KiB
TOML

# Put this file in one of the locations, with descending priority
# ./admin.toml
# $HOME/.seaweedfs/admin.toml
# /etc/seaweedfs/admin.toml
# this file is read by "weed admin"
# Maintenance task settings declared here are applied every time the admin
# server starts, overriding values saved from the admin UI under
# <dataDir>/conf. Commented-out keys keep their saved or default values.
# Requires -dataDir.
# Each value can also be set via environment variable, e.g.
# export WEED_MAINTENANCE_VACUUM_GARBAGE_THRESHOLD=0.3
[s3]
# The address clients reach the S3 gateway at, including the scheme.
# The file browser uses it to offer copyable object URLs; without it,
# the copy action is hidden.
# public_endpoint = "https://s3.example.com"
[maintenance]
# toggle the entire maintenance system (task detection and execution)
# enabled = true
[maintenance.vacuum]
# enabled = true
# vacuum volumes with more deleted content than this ratio
# garbage_threshold = 0.3
# how often to scan for volumes needing vacuum
# scan_interval_seconds = 7200
# max_concurrent = 2
# only vacuum volumes older than this (rounded up to whole hours)
# min_volume_age_seconds = 86400
# max retry attempts for a failed vacuum job (admin runtime default)
# retry_limit = 1
# seconds to wait between retry attempts
# retry_backoff_seconds = 10
[maintenance.balance]
# enabled = true
# trigger balancing when volume distribution exceeds this ratio
# imbalance_threshold = 0.2
# scan_interval_seconds = 1800
# max_concurrent = 1
# minimum number of volume servers before balancing
# min_server_count = 2
# max retry attempts for a failed balance job
# retry_limit = 1
# seconds to wait between retry attempts
# retry_backoff_seconds = 15
[maintenance.erasure_coding]
# enabled = true
# only erasure code volumes this full
# fullness_ratio = 0.95
# only erasure code volumes not modified for this long
# quiet_for_seconds = 3600
# scan_interval_seconds = 3600
# max_concurrent = 1
# min_size_mb = 30
# only process volumes from these collections, comma separated, "*" and "?" wildcards
# and regex patterns allowed ("" = all collections)
# collection_filter = ""
# disk tags preferred for shard placement
# preferred_tags = ["fast", "ssd"]
# EC shard placement constraint, e.g. "020"; empty uses the master default replication
# replica_placement = ""
# fail planning when the placement constraints cannot be met instead of relaxing them
# strict_placement = false
# volume-slot deadband for shard placement: nodes whose free volume slots differ by at
# most this much tie and fewer live shards wins; applied times data shards (0 = strict)
# free_slots_deadband = 0
# shard copy concurrency. Leave unset for one in-flight copy per destination disk.
# 0 sends shards sequentially. A positive N is a global cap, still one per disk.
# max_shard_transfers = 4
# max retry attempts for a failed erasure coding job
# retry_limit = 1
# seconds to wait between retry attempts
# retry_backoff_seconds = 30
[maintenance.s3_lifecycle]
# enabled = true
# How far back the filer meta-log subscription can reliably reach, in days.
# Rules whose TTL exceeds this window cannot be serviced by replay alone and
# are partitioned into the walk set, which steady-state walks scan in full,
# subject to walker_interval_minutes.
# 0 means "no retention info supplied": the run then falls back to the longest
# rule TTL, which keeps every replay rule in replay and promotes nothing.
# Set this to the window the filer meta-log actually retains.
# meta_log_retention_days = 0
# Minimum minutes between steady-state bucket walks per shard. Cold-start and
# recovery walks ignore it. 0 walks on every run.
# walker_interval_minutes = 0
# Cluster-wide delete budget in delete RPCs per second, divided across
# execute-capable workers at dispatch time. 0 means unlimited.
# cluster_deletes_per_second = 0
# Token-bucket burst for the delete budget. 0 means 2 x per-second.
# cluster_deletes_burst = 0
# max retry attempts for a failed lifecycle run
# retry_limit = 0
# seconds to wait between retry attempts
# retry_backoff_seconds = 5