admin: add bucket lifecycle rule editing (#10860)

* admin: add bucket lifecycle rule editing

* address greptile's comments

* more small fixes

* coderabbit's comments

* more comment fixes

* more fixes

* more

* maybe last

* last ?

* 14850

* 14851

* filer: stamp the content MD5 on every SaveInsideFiler write

An entry's ETag falls back to Attributes.Md5, so conditional writers key
IF_ETAG_MATCH off it. SaveInsideFiler carried the looked-up attributes
forward without refreshing the hash, leaving it describing whatever the
previous writer stored: a later conditional write matched the stale hash
and overwrote content that had already changed.

* s3api: give the bucket lifecycle constants and the write route key one definition each

The extended-attribute keys, the XML size cap and the object-write ring key
prefix were each spelled out in two places, so the admin dashboard's copies
could drift from the gateway's. Move them to the packages both sides already
import and alias them where the short local name reads better.

* admin: patch the bucket entry's lifecycle keys instead of rewriting the entry

The save read the bucket entry, edited its extended map and wrote the whole
entry back, guarded by IF_UNMODIFIED_SINCE. Nothing that writes a bucket
entry advances its mtime - not the S3 gateway's patchBucketEntry, not
SetBucketOwner, not SetBucketQuota - so the guard never fired and the stale
snapshot reverted whatever else had changed since the lookup.

Send the PATCH_EXTENDED mutation the S3 gateway already uses for these keys:
the filer re-reads and merges under the bucket path lock, so only the two
lifecycle keys move. That removes the reason for the mtime snapshot, the
verification retry loop and the compensating restore of the cleared day-TTL
rules, which the migration now logs instead.

* s3api: run the delete-lifecycle day-TTL migration through the shared helper

DeleteBucketLifecycleHandler kept its own copy of the read-strip-write
sequence the put handler now shares, including a missing return that let a
ToText failure persist a truncated filer.conf and write a second response.
It also wrote the whole file back unconditionally, reverting any concurrent
edit; the shared helper writes conditionally.

* admin: answer 404 when a lifecycle request names a bucket that does not exist

Every SetBucketLifecycle failure came back as 500, including the lookup miss
for an unknown bucket, so a client or monitor read a caller error as a server
fault and retried it.

* s3api: emit lifecycle XML a client would recognize

Two changes to what MarshalCanonical writes, both visible through
GetBucketLifecycleConfiguration, which replays the stored bytes verbatim:
stamp the S3 namespace on the root, and put a size range under <And>. A
<Filter> carries one predicate, so two size bounds side by side is a shape
AWS does not document. Parsing still accepts either.

* admin: fix the lifecycle editor's handling of stored status, deletes and empty saves

Four things the editor got wrong:

A stored <Status> the S3 API never validated, say 'enabled', left both radio
buttons unchecked, so reading the form threw on a null querySelector result
and Save did nothing. Collapse anything but an exact 'Enabled' to 'Disabled',
which is what the engine already does with it.

Deleting a rule re-rendered an open edit form from the snapshot taken when
editing began, discarding what had been typed; every other transition folds
the form in first.

The Transition warning only matched a bare <Transition>, missing the form
with attributes, self-closed or namespace-prefixed.

Saving an emptied rule list clears the configuration through a path with no
prompt, next to a Delete-all-rules button that asks.

Also collapses the three divergent copies of formatBytes on this page to one.

* filer: stop the day-TTL migration from deleting an operator's path rule

The migration removed every rule under the bucket's path that carried a day
TTL in the bucket's collection. The add path it is retiring used
AddLocationConf, which merged its TTL onto whatever already sat at the
prefix, so a rule can hold operator settings the lifecycle path never wrote -
a disk type, WORM retention, a read-only flag, a placement pin. Deleting the
whole rule to retire its TTL took those with it, leaving objects under that
prefix on defaults nobody asked for.

Delete only rules shaped like ones the add path created from scratch;
anything else keeps its settings and loses just the TTL.

---------

Co-authored-by: Chris Lu <chris.lu@gmail.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
This commit is contained in:
authored and GitHub committed 2026-08-21 23:42:26 -07:00
1 parent 301d83cc7a
commit df93d01c06
17 files changed
+2313 -185

No files matched your search

+141
View File
@@ -3,6 +3,7 @@ package lifecycle_xml
import (
"bytes"
"encoding/xml"
"sort"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
)
@@ -81,6 +82,146 @@ func ruleToCanonical(r *Rule) *s3lifecycle.Rule {
return out
}
// CanonicalToLifecycle is the inverse of LifecycleToCanonical: it builds a
// marshalable Lifecycle from the engine's flat Rule shape. Only the fields
// s3lifecycle.Rule can represent are populated — there is no way back to
// Transition / NoncurrentVersionTransition, which the canonical form never
// carries.
func CanonicalToLifecycle(rules []*s3lifecycle.Rule) *Lifecycle {
lc := &Lifecycle{
Rules: make([]Rule, 0, len(rules)),
}
for _, r := range rules {
lc.Rules = append(lc.Rules, ruleFromCanonical(r))
}
return lc
}
// s3XMLNamespace is the namespace every S3 client stamps on a lifecycle
// document it PUTs. GetBucketLifecycleConfiguration replays the stored bytes
// verbatim, so a document written here has to carry it too or it would come
// back stripped of a namespace the client-written ones have.
const s3XMLNamespace = "http://s3.amazonaws.com/doc/2006-03-01/"
// MarshalCanonical serializes the canonical rules straight to a
// BucketLifecycleConfiguration XML document, mirroring ParseCanonical.
func MarshalCanonical(rules []*s3lifecycle.Rule) ([]byte, error) {
lc := CanonicalToLifecycle(rules)
out, err := xml.Marshal(struct {
Lifecycle
XMLNS string `xml:"xmlns,attr"`
}{Lifecycle: *lc, XMLNS: s3XMLNamespace})
if err != nil {
return nil, err
}
return append([]byte(xml.Header), out...), nil
}
func ruleFromCanonical(r *s3lifecycle.Rule) Rule {
out := Rule{
ID: r.ID,
Status: RuleStatus(r.Status),
Filter: filterFromCanonical(r.Prefix, r.FilterTags, r.FilterSizeGreaterThan, r.FilterSizeLessThan),
}
if r.ExpirationDays > 0 || !r.ExpirationDate.IsZero() || r.ExpiredObjectDeleteMarker {
out.Expiration = Expiration{
set: true,
Days: r.ExpirationDays,
}
if !r.ExpirationDate.IsZero() {
out.Expiration.Date = ExpirationDate{Time: r.ExpirationDate}
}
if r.ExpiredObjectDeleteMarker {
out.Expiration.DeleteMarker = ExpireDeleteMarker{val: true, set: true}
}
}
if r.NoncurrentVersionExpirationDays > 0 || r.NewerNoncurrentVersions > 0 {
out.NoncurrentVersionExpiration = NoncurrentVersionExpiration{
set: true,
NoncurrentDays: r.NoncurrentVersionExpirationDays,
NewerNoncurrentVersions: r.NewerNoncurrentVersions,
}
}
if r.AbortMPUDaysAfterInitiation > 0 {
out.AbortIncompleteMultipartUpload = AbortIncompleteMultipartUpload{
set: true,
DaysAfterInitiation: r.AbortMPUDaysAfterInitiation,
}
}
return out
}
// filterFromCanonical is the inverse of flattenFilter: it picks the
// narrowest Filter shape that represents the given prefix/tags/size bounds,
// matching what Filter.MarshalXML (single Prefix|Tag branch plus optional
// size bounds) and its And branch can each express.
func filterFromCanonical(prefix string, tags map[string]string, sizeGT, sizeLT int64) Filter {
hasSize := sizeGT > 0 || sizeLT > 0
// <Filter> holds exactly one predicate; anything more goes under <And>,
// two size bounds included, which is the form AWS documents for a range.
discriminants := 0
if prefix != "" {
discriminants++
}
discriminants += len(tags)
if sizeGT > 0 {
discriminants++
}
if sizeLT > 0 {
discriminants++
}
f := Filter{set: true, ObjectSizeGreaterThan: sizeGT, ObjectSizeLessThan: sizeLT}
switch {
case discriminants > 1:
f.andSet = true
f.And = And{
ObjectSizeGreaterThan: sizeGT,
ObjectSizeLessThan: sizeLT,
}
if prefix != "" {
f.And.Prefix = NewPrefix(prefix)
}
if len(tags) > 0 {
keys := make([]string, 0, len(tags))
for k := range tags {
keys = append(keys, k)
}
sort.Strings(keys)
f.And.Tags = make([]Tag, 0, len(keys))
for _, k := range keys {
f.And.Tags = append(f.And.Tags, Tag{Key: k, Value: tags[k]})
}
}
// The And branch carries its own size bounds; the enclosing
// Filter only emits them on the non-And path (see
// Filter.MarshalXML), so clear them here to avoid duplication.
f.ObjectSizeGreaterThan = 0
f.ObjectSizeLessThan = 0
case len(tags) == 1:
f.tagSet = true
for k, v := range tags {
f.Tag = Tag{Key: k, Value: v}
}
case hasSize:
// Single discriminant and it's a size range: the bounds set above
// already cover it — no <Prefix> (not even an empty one; nothing
// was requested) and no <And> (nothing else to combine with).
default:
// Either a single prefix or no discriminant at all (whole-bucket
// filter) — both are expressed as a <Prefix> element, empty or not.
f.Prefix = NewPrefix(prefix)
}
return f
}
func flattenFilter(f *Filter) (prefix string, tags map[string]string, sizeGT, sizeLT int64) {
if !f.set {
return
+323
View File
@@ -4,6 +4,9 @@ import (
"encoding/xml"
"strings"
"testing"
"time"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3lifecycle"
)
func TestLifecycleXMLRoundTrip_NoncurrentVersionExpiration(t *testing.T) {
@@ -274,3 +277,323 @@ func TestLifecycleXMLRoundTrip_CompleteRule(t *testing.T) {
}
}
}
// assertCanonicalRoundTrip drives a canonical rule through
// CanonicalToLifecycle -> MarshalCanonical -> ParseCanonical and checks the
// result matches the input, proving the admin write path (which only ever
// has the canonical form) produces XML the S3 API can read back unchanged.
func assertCanonicalRoundTrip(t *testing.T, in *s3lifecycle.Rule) *s3lifecycle.Rule {
t.Helper()
xmlBytes, err := MarshalCanonical([]*s3lifecycle.Rule{in})
if err != nil {
t.Fatalf("MarshalCanonical: %v", err)
}
out, err := ParseCanonical(xmlBytes)
if err != nil {
t.Fatalf("ParseCanonical(%s): %v", xmlBytes, err)
}
if len(out) != 1 {
t.Fatalf("expected 1 rule after round trip, got %d: %s", len(out), xmlBytes)
}
return out[0]
}
func TestCanonicalRoundTrip_WholeBucket(t *testing.T) {
in := &s3lifecycle.Rule{ID: "whole-bucket", Status: s3lifecycle.StatusEnabled, ExpirationDays: 30}
out := assertCanonicalRoundTrip(t, in)
if out.Prefix != "" {
t.Errorf("expected empty prefix, got %q", out.Prefix)
}
if out.ExpirationDays != 30 {
t.Errorf("expected ExpirationDays=30, got %d", out.ExpirationDays)
}
}
func TestCanonicalRoundTrip_PrefixOnly(t *testing.T) {
in := &s3lifecycle.Rule{ID: "prefix-only", Status: s3lifecycle.StatusEnabled, Prefix: "logs/", ExpirationDays: 7}
out := assertCanonicalRoundTrip(t, in)
if out.Prefix != "logs/" {
t.Errorf("expected prefix 'logs/', got %q", out.Prefix)
}
if len(out.FilterTags) != 0 {
t.Errorf("expected no tags, got %v", out.FilterTags)
}
}
func TestCanonicalRoundTrip_TagOnly(t *testing.T) {
in := &s3lifecycle.Rule{
ID: "tag-only",
Status: s3lifecycle.StatusEnabled,
FilterTags: map[string]string{"env": "dev"},
ExpirationDays: 7,
}
out := assertCanonicalRoundTrip(t, in)
if out.Prefix != "" {
t.Errorf("expected empty prefix, got %q", out.Prefix)
}
if len(out.FilterTags) != 1 || out.FilterTags["env"] != "dev" {
t.Errorf("expected tags {env:dev}, got %v", out.FilterTags)
}
}
func TestCanonicalRoundTrip_PrefixAndTags(t *testing.T) {
in := &s3lifecycle.Rule{
ID: "prefix-and-tags",
Status: s3lifecycle.StatusEnabled,
Prefix: "logs/",
FilterTags: map[string]string{"env": "dev", "tier": "hot"},
ExpirationDays: 7,
}
out := assertCanonicalRoundTrip(t, in)
if out.Prefix != "logs/" {
t.Errorf("expected prefix 'logs/', got %q", out.Prefix)
}
if len(out.FilterTags) != 2 || out.FilterTags["env"] != "dev" || out.FilterTags["tier"] != "hot" {
t.Errorf("expected tags {env:dev, tier:hot}, got %v", out.FilterTags)
}
}
func TestCanonicalRoundTrip_SizeBounds(t *testing.T) {
in := &s3lifecycle.Rule{
ID: "size-bounds",
Status: s3lifecycle.StatusEnabled,
FilterSizeGreaterThan: 512,
FilterSizeLessThan: 1048576,
ExpirationDays: 30,
}
out := assertCanonicalRoundTrip(t, in)
if out.FilterSizeGreaterThan != 512 {
t.Errorf("expected FilterSizeGreaterThan=512, got %d", out.FilterSizeGreaterThan)
}
if out.FilterSizeLessThan != 1048576 {
t.Errorf("expected FilterSizeLessThan=1048576, got %d", out.FilterSizeLessThan)
}
}
func TestCanonicalRoundTrip_PrefixAndTagsWithSizeBounds(t *testing.T) {
// Multiple discriminants (prefix + tags) force the <And> branch, which
// carries its own size bounds distinct from the single-branch Filter.
in := &s3lifecycle.Rule{
ID: "and-with-size",
Status: s3lifecycle.StatusEnabled,
Prefix: "logs/",
FilterTags: map[string]string{"env": "dev"},
FilterSizeGreaterThan: 1024,
FilterSizeLessThan: 2048,
ExpirationDays: 7,
}
out := assertCanonicalRoundTrip(t, in)
if out.Prefix != "logs/" || len(out.FilterTags) != 1 {
t.Errorf("expected prefix 'logs/' and 1 tag, got prefix=%q tags=%v", out.Prefix, out.FilterTags)
}
if out.FilterSizeGreaterThan != 1024 || out.FilterSizeLessThan != 2048 {
t.Errorf("expected size bounds 1024/2048, got %d/%d", out.FilterSizeGreaterThan, out.FilterSizeLessThan)
}
}
func TestCanonicalRoundTrip_ExpirationDate(t *testing.T) {
date := time.Date(2030, 1, 15, 0, 0, 0, 0, time.UTC)
in := &s3lifecycle.Rule{ID: "expire-on-date", Status: s3lifecycle.StatusEnabled, ExpirationDate: date}
out := assertCanonicalRoundTrip(t, in)
if !out.ExpirationDate.Equal(date) {
t.Errorf("expected ExpirationDate=%v, got %v", date, out.ExpirationDate)
}
}
func TestCanonicalRoundTrip_ExpiredObjectDeleteMarker(t *testing.T) {
in := &s3lifecycle.Rule{ID: "delete-marker", Status: s3lifecycle.StatusEnabled, ExpiredObjectDeleteMarker: true}
out := assertCanonicalRoundTrip(t, in)
if !out.ExpiredObjectDeleteMarker {
t.Error("expected ExpiredObjectDeleteMarker=true")
}
}
func TestCanonicalRoundTrip_NoncurrentVersionExpiration(t *testing.T) {
in := &s3lifecycle.Rule{
ID: "noncurrent",
Status: s3lifecycle.StatusEnabled,
NoncurrentVersionExpirationDays: 30,
NewerNoncurrentVersions: 2,
}
out := assertCanonicalRoundTrip(t, in)
if out.NoncurrentVersionExpirationDays != 30 {
t.Errorf("expected NoncurrentVersionExpirationDays=30, got %d", out.NoncurrentVersionExpirationDays)
}
if out.NewerNoncurrentVersions != 2 {
t.Errorf("expected NewerNoncurrentVersions=2, got %d", out.NewerNoncurrentVersions)
}
}
func TestCanonicalRoundTrip_AbortMultipartUpload(t *testing.T) {
in := &s3lifecycle.Rule{ID: "abort-mpu", Status: s3lifecycle.StatusEnabled, AbortMPUDaysAfterInitiation: 7}
out := assertCanonicalRoundTrip(t, in)
if out.AbortMPUDaysAfterInitiation != 7 {
t.Errorf("expected AbortMPUDaysAfterInitiation=7, got %d", out.AbortMPUDaysAfterInitiation)
}
}
func TestCanonicalRoundTrip_Disabled(t *testing.T) {
in := &s3lifecycle.Rule{ID: "disabled-rule", Status: s3lifecycle.StatusDisabled, ExpirationDays: 30}
out := assertCanonicalRoundTrip(t, in)
if out.Status != s3lifecycle.StatusDisabled {
t.Errorf("expected Status=Disabled, got %q", out.Status)
}
}
func TestCanonicalRoundTrip_TagOrderIsStable(t *testing.T) {
// FilterTags is a map; And.Tags must be emitted in a deterministic
// (sorted) order so re-saving an unchanged rule doesn't churn the
// stored XML.
in := &s3lifecycle.Rule{
ID: "stable-order",
Status: s3lifecycle.StatusEnabled,
Prefix: "logs/",
FilterTags: map[string]string{"zeta": "1", "alpha": "2", "mu": "3"},
}
var firstXML []byte
for i := 0; i < 5; i++ {
xmlBytes, err := MarshalCanonical([]*s3lifecycle.Rule{in})
if err != nil {
t.Fatalf("MarshalCanonical: %v", err)
}
if i == 0 {
firstXML = xmlBytes
continue
}
if string(xmlBytes) != string(firstXML) {
t.Fatalf("marshal output is not stable across runs:\n%s\nvs\n%s", firstXML, xmlBytes)
}
}
if !strings.Contains(string(firstXML), "<Tag><Key>alpha</Key>") {
t.Errorf("expected tags sorted alphabetically (alpha first), got: %s", firstXML)
}
}
func TestCanonicalRoundTrip_PrefixWithSizeBoundsUsesAnd(t *testing.T) {
// A size range paired with a prefix is two different attributes, which
// requires <And> — unlike a size range alone (see
// TestCanonicalRoundTrip_SizeBounds), which doesn't need one.
in := &s3lifecycle.Rule{
ID: "prefix-with-size",
Status: s3lifecycle.StatusEnabled,
Prefix: "logs/",
FilterSizeGreaterThan: 1024,
ExpirationDays: 7,
}
xmlBytes, err := MarshalCanonical([]*s3lifecycle.Rule{in})
if err != nil {
t.Fatalf("MarshalCanonical: %v", err)
}
if !strings.Contains(string(xmlBytes), "<And>") {
t.Errorf("expected prefix+size to be wrapped in <And>, got: %s", xmlBytes)
}
out := assertCanonicalRoundTrip(t, in)
if out.Prefix != "logs/" {
t.Errorf("expected prefix 'logs/', got %q", out.Prefix)
}
if out.FilterSizeGreaterThan != 1024 {
t.Errorf("expected FilterSizeGreaterThan=1024, got %d", out.FilterSizeGreaterThan)
}
}
func TestCanonicalRoundTrip_TagWithSizeBoundsUsesAnd(t *testing.T) {
in := &s3lifecycle.Rule{
ID: "tag-with-size",
Status: s3lifecycle.StatusEnabled,
FilterTags: map[string]string{"env": "dev"},
FilterSizeLessThan: 2048,
ExpirationDays: 7,
}
xmlBytes, err := MarshalCanonical([]*s3lifecycle.Rule{in})
if err != nil {
t.Fatalf("MarshalCanonical: %v", err)
}
if !strings.Contains(string(xmlBytes), "<And>") {
t.Errorf("expected tag+size to be wrapped in <And>, got: %s", xmlBytes)
}
out := assertCanonicalRoundTrip(t, in)
if len(out.FilterTags) != 1 || out.FilterTags["env"] != "dev" {
t.Errorf("expected tags {env:dev}, got %v", out.FilterTags)
}
if out.FilterSizeLessThan != 2048 {
t.Errorf("expected FilterSizeLessThan=2048, got %d", out.FilterSizeLessThan)
}
}
func TestCanonicalRoundTrip_SizeOnlyOmitsPrefixElement(t *testing.T) {
// A size-only filter must not stamp a spurious empty <Prefix> — nothing
// was requested, so nothing besides the size bounds should appear.
in := &s3lifecycle.Rule{
ID: "size-only-no-prefix",
Status: s3lifecycle.StatusEnabled,
FilterSizeGreaterThan: 512,
ExpirationDays: 7,
}
xmlBytes, err := MarshalCanonical([]*s3lifecycle.Rule{in})
if err != nil {
t.Fatalf("MarshalCanonical: %v", err)
}
if strings.Contains(string(xmlBytes), "<Prefix>") {
t.Errorf("expected no <Prefix> element for a size-only filter, got: %s", xmlBytes)
}
out := assertCanonicalRoundTrip(t, in)
if out.Prefix != "" {
t.Errorf("expected empty prefix, got %q", out.Prefix)
}
if out.FilterSizeGreaterThan != 512 {
t.Errorf("expected FilterSizeGreaterThan=512, got %d", out.FilterSizeGreaterThan)
}
}
func TestCanonicalRoundTrip_SizeRangeUsesAnd(t *testing.T) {
// <Filter> holds one predicate; a range is two, so it belongs under <And>,
// which is the form AWS documents.
in := &s3lifecycle.Rule{
ID: "size-range",
Status: s3lifecycle.StatusEnabled,
FilterSizeGreaterThan: 512,
FilterSizeLessThan: 1048576,
ExpirationDays: 30,
}
xmlBytes, err := MarshalCanonical([]*s3lifecycle.Rule{in})
if err != nil {
t.Fatalf("MarshalCanonical: %v", err)
}
if !strings.Contains(string(xmlBytes), "<And><ObjectSizeGreaterThan>512</ObjectSizeGreaterThan><ObjectSizeLessThan>1048576</ObjectSizeLessThan></And>") {
t.Errorf("expected both size bounds under <And>, got: %s", xmlBytes)
}
out := assertCanonicalRoundTrip(t, in)
if out.FilterSizeGreaterThan != 512 || out.FilterSizeLessThan != 1048576 {
t.Errorf("expected the size range to survive the round trip, got %+v", out)
}
}
func TestMarshalCanonical_CarriesTheS3Namespace(t *testing.T) {
// GetBucketLifecycleConfiguration replays these bytes verbatim, so a
// document written here has to look like one a client PUT.
xmlBytes, err := MarshalCanonical([]*s3lifecycle.Rule{{
ID: "ns",
Status: s3lifecycle.StatusEnabled,
ExpirationDays: 1,
}})
if err != nil {
t.Fatalf("MarshalCanonical: %v", err)
}
if !strings.Contains(string(xmlBytes), `<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">`) {
t.Errorf("expected the S3 namespace on the root element, got: %s", xmlBytes)
}
if _, err := ParseCanonical(xmlBytes); err != nil {
t.Errorf("the namespaced document must still parse: %v", err)
}
}