* admin: add bucket lifecycle rule editing
* address greptile's comments
* more small fixes
* coderabbit's comments
* more comment fixes
* more fixes
* more
* maybe last
* last ?
* 14850
* 14851
* filer: stamp the content MD5 on every SaveInsideFiler write
An entry's ETag falls back to Attributes.Md5, so conditional writers key
IF_ETAG_MATCH off it. SaveInsideFiler carried the looked-up attributes
forward without refreshing the hash, leaving it describing whatever the
previous writer stored: a later conditional write matched the stale hash
and overwrote content that had already changed.
* s3api: give the bucket lifecycle constants and the write route key one definition each
The extended-attribute keys, the XML size cap and the object-write ring key
prefix were each spelled out in two places, so the admin dashboard's copies
could drift from the gateway's. Move them to the packages both sides already
import and alias them where the short local name reads better.
* admin: patch the bucket entry's lifecycle keys instead of rewriting the entry
The save read the bucket entry, edited its extended map and wrote the whole
entry back, guarded by IF_UNMODIFIED_SINCE. Nothing that writes a bucket
entry advances its mtime - not the S3 gateway's patchBucketEntry, not
SetBucketOwner, not SetBucketQuota - so the guard never fired and the stale
snapshot reverted whatever else had changed since the lookup.
Send the PATCH_EXTENDED mutation the S3 gateway already uses for these keys:
the filer re-reads and merges under the bucket path lock, so only the two
lifecycle keys move. That removes the reason for the mtime snapshot, the
verification retry loop and the compensating restore of the cleared day-TTL
rules, which the migration now logs instead.
* s3api: run the delete-lifecycle day-TTL migration through the shared helper
DeleteBucketLifecycleHandler kept its own copy of the read-strip-write
sequence the put handler now shares, including a missing return that let a
ToText failure persist a truncated filer.conf and write a second response.
It also wrote the whole file back unconditionally, reverting any concurrent
edit; the shared helper writes conditionally.
* admin: answer 404 when a lifecycle request names a bucket that does not exist
Every SetBucketLifecycle failure came back as 500, including the lookup miss
for an unknown bucket, so a client or monitor read a caller error as a server
fault and retried it.
* s3api: emit lifecycle XML a client would recognize
Two changes to what MarshalCanonical writes, both visible through
GetBucketLifecycleConfiguration, which replays the stored bytes verbatim:
stamp the S3 namespace on the root, and put a size range under <And>. A
<Filter> carries one predicate, so two size bounds side by side is a shape
AWS does not document. Parsing still accepts either.
* admin: fix the lifecycle editor's handling of stored status, deletes and empty saves
Four things the editor got wrong:
A stored <Status> the S3 API never validated, say 'enabled', left both radio
buttons unchecked, so reading the form threw on a null querySelector result
and Save did nothing. Collapse anything but an exact 'Enabled' to 'Disabled',
which is what the engine already does with it.
Deleting a rule re-rendered an open edit form from the snapshot taken when
editing began, discarding what had been typed; every other transition folds
the form in first.
The Transition warning only matched a bare <Transition>, missing the form
with attributes, self-closed or namespace-prefixed.
Saving an emptied rule list clears the configuration through a path with no
prompt, next to a Delete-all-rules button that asks.
Also collapses the three divergent copies of formatBytes on this page to one.
* filer: stop the day-TTL migration from deleting an operator's path rule
The migration removed every rule under the bucket's path that carried a day
TTL in the bucket's collection. The add path it is retiring used
AddLocationConf, which merged its TTL onto whatever already sat at the
prefix, so a rule can hold operator settings the lifecycle path never wrote -
a disk type, WORM retention, a read-only flag, a placement pin. Deleting the
whole rule to retire its TTL took those with it, leaving objects under that
prefix on defaults nobody asked for.
Delete only rules shaped like ones the add path created from scratch;
anything else keeps its settings and loses just the TTL.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* feat(s3/lifecycle): filer-backed cursor Persister
FilerPersister persists per-shard cursor maps as JSON to
/etc/s3/lifecycle/cursors/shard-NN.json via filer.SaveInsideFiler.
One file per shard keeps Save atomic — the filer writes the entry
in a single mutation, so a crash mid-write doesn't leak partial
state. Pipeline.Run loads on start; the periodic checkpoint and
graceful-shutdown save go through this implementation.
A small FilerStore interface wraps the SeaweedFilerClient surface
the persister needs, so tests inject an in-memory fake instead of
mocking the full gRPC client.
* refactor(s3/lifecycle): drop BlockerStore — durable cursor IS the block
A frozen cursor doesn't advance, so the durable cursor (FilerPersister)
encodes the blocked state on its own. On worker restart the reader
re-encounters the poison event at MinTsNs, the dispatcher walks the
same retry budget to BLOCKED, and the cursor freezes at the same
EventTs. Other in-flight events between freeze tsNs and prior cursor
positions self-resolve via NOOP_RESOLVED (STALE_IDENTITY) since the
underlying objects were already deleted on the prior pass.
Removed:
- BlockerStore interface + InMemoryBlockerStore + BlockerRecord
- Dispatcher.Blockers + Dispatcher.ReplayBlockers
- the BlockerStore.Put call in handleBlocked
- Pipeline.Blockers field + the ReplayBlockers call on startup
Added a TestDispatchRestartReFreezesNaturally that pins the
self-recovery property: a fresh Dispatcher with a fresh Cursor, fed
the same poison event, reaches the same frozen state at the same
EventTs without any durable blocker store.
Operator visibility: a cursor whose MinTsNs hasn't advanced is the
signal — surfaced via the durable cursor file.
* refactor(filer): SaveInsideFiler accepts ctx
ReadInsideFiler already takes ctx; SaveInsideFiler used context.Background()
internally and silently dropped the caller's ctx. Symmetric API now;
cancellation/deadlines propagate through LookupEntry / CreateEntry /
UpdateEntry. Mechanical update of all callers — most pass
context.Background() since the existing call sites have no ctx in scope.
* fix(s3/lifecycle): deterministic order in cursor save
Iterating Go maps yields random order, so json.Encode produced a different
byte sequence on each save even when the state hadn't changed. Sort
entries by (Bucket, ActionKind, RuleHash) before encoding so the on-disk
file diffs cleanly. New test pins byte-identical output across two saves
of the same map.
* fix(s3/lifecycle): log reason when freezing cursor in handleBlocked
handleBlocked dropped the reason via _ = reason with a comment claiming
the caller logged it; none of the three callers do. A frozen cursor is
the only surface where the operator finds out something stuck, so the
reason has to land somewhere. glog.Warningf with shard, key, eventTs,
and the original reason — same shape the rest of the package uses.
