filer: auto-enable per-path commit records when a peer has its own store (#12062)

* filer: enable per-path commit records only when a peer has its own store

Since seaweedfs#11698 every metadata change writes a per-path commit
record into the filer store (an MpTs key, plus an MpDl key on deletes),
and the records stay after the path is deleted. They order the events a
filer replays from peer filers that keep their own store, since replay
only runs when peerSignature != f.Signature.

Filers that all share one store (one Redis, MySQL, TiKV, ...) never
replay each other's events, so the records are never read, yet each
change still costs one or two extra store writes and the key count grows
with every path ever changed — unbounded growth for in-memory stores.

This turns the records off by default and latches them on the first time
a peer reports a different store signature — no option needed. The latch
is persisted in the store (MpRep) so it survives restarts and converges
filers sharing a store even before they discover the peer themselves.
While off, a filer keeps an MpOff marker so enabling later bumps the
records' generation (MpGen, folded into each record key) past the
unrecorded window: a record older than a change made while off cannot
let an older replayed peer event win. Ordering for unrecorded changes
falls back to the entry's mtime, the same as entries written before the
records existed.

Supersedes the manual option approach in seaweedfs#12060.

* filer: keep delete tombstones while records off, harden enable path

Review on the first commit:

- A delete committed while the records were off left no entry and no
  usable tombstone: after enabling, an older replayed create could
  resurrect the path. Deletes and rename sources are now always recorded
  under the generation-0 MpDl key — they are never fenced by a
  generation bump, and they cannot go stale because every delete is
  recorded. Creates/updates still write nothing while off.

- bumpPathCommitGen could regress MpGen under concurrent bumpers on a
  shared store. It now loops: each pass writes only a generation higher
  than any it has seen, so concurrent bumps converge upward.

- doSubscribeToOneFiler proceeded into replay when
  enablePathCommitRecords failed, running the whole session without
  ordering. It now aborts the subscribe so the reconnect retries with
  the records on.

* filer: bind pending commit-record repairs to their generation

A failed MpTs write kept in metaTsPending could land after a shared-store
generation bump, writing a pre-bump timestamp as a current-generation
record and overriding newer unrecorded state. Each pending change record
now carries the generation it failed under: repairs write it only while
that generation is current, and ordering reads ignore it otherwise.
Pending delete records need no binding — MpDl keys are never fenced.

* filer: never move the local records' generation backward

The converge loop in bumpPathCommitGen seeded from zero and both it and
syncPathCommitGen stored unconditionally: a stale writer could leave the
stored generation below what this filer already adopted, and the next
sync or bump then walked the local value back — orphaning records it
wrote under the higher generation. The bump now seeds from the local
generation and every store only moves it forward.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: name a records generation after the off marker it consumes

Two filers consuming the same off marker each counted one past the
stored generation, so the second could store N+2 after the first
stored N+1 and write records under a key the other could not read
until the next check. The bump now takes the marker's own timestamp —
filers racing on one marker land on one value — a marker older than
the stored generation is cleaned as a leftover, and a stored value
that already passed the bump is adopted instead of counted past. A
newly enabling filer whose marker was already consumed names its own
off window with the current time, so its unrecorded period is still
fenced.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: tell a fresh off marker from one the generation already consumed

A marker whose timestamp reads at or below the stored generation was
dropped as leftover, but a new off period can also read that old — the
off filer's clock lagging, or a host clock moving back — and dropping
it kept records from before that period live, so stale replay could
win.

The stored generation now names the marker it was bumped for: a marker
the store still shows is consumed unless it is that one, however old
it reads, and only the consumed marker itself is cleared without a
bump. The generation trails in the value, so a whole-slice read still
yields it on stores written before the pair was stored. Enabling
fences the filer's own off window with the current time instead of a
marker a sharer may already have consumed.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: cover racing off-marker consumers settling on one generation

* filer: adopt the stored generation, not the consumed marker it names

A filer syncing with no off marker present read the value's first field —
the consumed marker's timestamp — so a restart after a lagging-clock off
period under-counted the generation and kept records the bump dropped.

* filer: re-check the off marker before consuming it

A sync whose marker read ran behind a later off period's consume cycle
still bumped for its stale read: the stored generation already covered a
different marker, so the bump overwrote the consumed name, counted the
generation past records a newer consumption meant to keep, and cleared
whatever marker was live by then. Each pass now re-reads the marker and
only bumps or clears while it is still the one the pass saw.

* filer: keep a failed generation read from resetting the records' generation

The final generation read ignored errors and then stored whatever gen
held — possibly zero — so a transient store failure could hide records
the current generation wrote. Unexpected read errors now return before
the local generation changes.

Also cover a sync whose marker read runs behind another consumer's
delete: it must adopt the stored generation, not bump for the stale read.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: fail a bump that cannot verify the stored generation

A generation read that failed mid-bump was treated as absent: the
write could land below a generation a racing bumper had stored, or
re-store a marker that bumper had already consumed. Read failures
now abort the pass like every other store error on the path.

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: cover generation reads that fail instead of returning a value

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: carry the commit timestamp on the entry itself

The per-path MpTs record only exists because a commit timestamp had
nowhere else to live. Stamping commit_ts_ns on the entry before its
store write - and reusing that stamp as the emitted event's TsNs -
makes the entry its own commit record: replay and bootstrap traversal
order on it with zero extra writes for creates and updates.

Deletes still need a durable key since a removed entry cannot carry a
timestamp, so MpDl tombstones stay (one write per delete, growth
bounded by delete history). Entries predating the field fall back to
mtime, matching what peers and old records already assume.

This replaces the MpTs/MpOff/MpGen/MpRep machinery entirely: shared-
store filers write nothing extra, and dedicated-store replication
keeps last-writer-wins ordering plus tombstone coverage for deletes
and ancestor drops.

* filer: keep a directory update replay from tombstoning its subtree

A same-path update wrote a delete record before the delete-and-reinsert,
so a directory metadata change read as a subtree delete: a later peer
event for a child older than the update was then skipped even though
nobody deleted the children, leaving files missing on this filer. Only
file updates take the tombstone now — the re-inserted directory entry's
own commit stamp orders the path itself.

* filer: order unstamped entries by their recorded commit before mtime

Entries written before stamping existed have no CommitTsNs, so the
ordering check fell back to the client-settable mtime and dropped the
MpTs record those versions wrote: an entry committed at event time 300
with mtime 100 lost to a delayed peer event at 200 after upgrade. Read
the legacy record for unstamped entries in localCommitTsNs, CommitTsNs,
and the traverse stamp (which also stopped falling back to mtime,
regressing entry_ts_ns to 0).

* filer: release queued rename event stamps when the rename fails

Each moved entry carried an in-flight stamp its queued event would
adopt at emit time. When the move succeeded but a later child move or
the commit failed, the handler rolled back without emitting the queued
events or releasing their stamps, pinning the delivered and flushed
watermarks below the aborted rename forever. Release every queued
stamp on both failure exits in both rename handlers.

* filer: apply a directory update replay with an in-place upsert

Deleting the stored entry before inserting its replacement left a
window where the directory had no entry and no ordering record — a
tombstone cannot stand in for it without reading as a subtree delete
that suppresses the children's own records. A stale peer event accepted
in that gap could replay an older update or delete and sweep the
directory's children. Upsert the replacement instead: the entry keeps
its recorded commit order across the whole update, and a directory has
no chunks or hard links for a delete to release.

* filer: update an existing directory entry on replay instead of upserting

InsertEntry is not an upsert on SQL stores configured without upsert
support — PostgreSQL with enableUpsert=false issues a plain INSERT and
its duplicate-key error does not match the MySQL-only fallback, so a
replay updating a present directory failed on every retry. Pick
UpdateEntry when the entry exists and InsertEntry when it is absent;
UpdateEntry touches no row that is not there.

* filer: detect a shared-store delete racing a replayed directory update

UpdateEntry reports success even when the row was deleted between the
find and the update — the UPDATE changes zero rows and the replayed
change is dropped while the subscription offset advances. Re-read the
entry after the update and retry the write, re-checking the recorded
commit order each pass so a newer peer change wins instead of being
resurrected over.

* filer: declare commit_ts_ns in field-number order

Field 6 sits between extended and hard_link_id; the declaration now
reads in the same order. Wire format unchanged. Also syncs the Java
client's checked-in copy of filer.proto, which the pb Makefile copies
on gen.

* filer: option to skip delete tombstones entirely

filer.options.deleteTombstones=false stops writing MpDl records on
deletes - a single filer or shared-store cluster never replays peer
events, so the tombstones are dead weight. The tradeoff is explicit:
if a dedicated-store peer does replicate, a stale event can resurrect
a deleted path.

* filer: drop delete tombstones entirely

Deletes no longer write MpDl records, and the pending-repair machinery
they needed is gone with them: per-mutation metadata overhead is now
zero for every store topology. Ordering information lives only on the
entry itself via commit_ts_ns.

The accepted tradeoff: a dedicated-store peer replaying an event older
than a delete can resurrect the path, and a kept directory no longer
fences stale child creates. Recorded tombstones and MpTs records left
by older builds are still read, so upgrades keep the protections they
already have.

* filer: stop reading legacy per-path records too

Drops the MpTs/MpDl read path along with the writes: meta_path_ts.go
keeps only the entry-field readers (CommitTsNs, EntryCommitTsNs,
localCommitTsNs). Entries written before stamping existed fall back to
mtime, and tombstones or commit records older builds left behind no
longer fence replayed events.

* filer: trim commit timestamp comments

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

other/java/client/src/main/proto/filer.proto

@@ -166,6 +166,10 @@ message Entry {
     repeated FileChunk chunks = 3;
     FuseAttributes attributes = 4;
     map<string, bytes> extended = 5;
+    // filer-local commit timestamp of the version stored here: the meta log
+    // event's ts_ns for the mutation that produced this entry, or 0 for
+    // entries written before the field existed
+    int64 commit_ts_ns = 6;
     bytes hard_link_id = 7;
     int32 hard_link_counter = 8; // only exists in hard link meta data
     bytes content = 9; // if not empty, the file content
@@ -281,6 +285,7 @@ message WriteCondition {
         IF_EXTENDED_NOT_EQUAL = 7; // fail if present and extended[ext_key] == ext_value
         IF_EXTENDED_TIME_ELAPSED = 8; // fail if present and extended[ext_key] (unix seconds) is in the future
         IF_CHUNKS_EQUAL       = 9; // fail unless the stored chunk fid multiset equals fids (absent entry = no chunks)
+        IF_ENTRY_EQUAL        = 10; // fail unless the stored entry equals expected_entry
     }
     // Clause is one primitive comparison. IF_ETAG_MATCH holds when the current
     // entry's ETag equals any value in etags; IF_ETAG_NOT_MATCH holds when it
@@ -309,6 +314,7 @@ message WriteCondition {
         string gate_key = 7;       // IF_EXTENDED_TIME_ELAPSED: only enforce when extended[gate_key] == gate_value
         string gate_value = 8;     // gate value (e.g. retention mode COMPLIANCE for governance bypass)
         repeated string fids = 9;  // chunk fid strings for IF_CHUNKS_EQUAL
+        Entry expected_entry = 10; // whole-entry comparison for IF_ENTRY_EQUAL
     }
     repeated Clause clauses = 1;   // all must hold (logical AND)
 }
@@ -323,6 +329,7 @@ enum FilerError {
     EXISTING_IS_FILE      = 4; // cannot overwrite file with directory
     ENTRY_ALREADY_EXISTS  = 5; // O_EXCL and entry already exists
     PRECONDITION_FAILED   = 6; // WriteCondition not satisfied
+    READ_ONLY             = 7; // path matches a read-only storage rule (e.g. bucket over quota)
 }
 
 // ObjectMutation is one entry-level change applied by ObjectTransaction. All
@@ -351,6 +358,7 @@ message ObjectMutation {
     bytes content = 11;                    // PATCH_EXTENDED: new Entry.content when set_content
     bool touch_mtime = 12;                 // PATCH_EXTENDED: set the entry's Mtime to now (e.g. a metadata-replace copy)
     bool remove_empty_parent = 13;         // DELETE: also remove the parent directory when the delete leaves it empty (best-effort)
+    bool keep_remote_object = 14;          // DELETE: leave the object on the mounted remote storage
 }
 
 // Recompute re-derives a pointer entry (directory/name on the mutation) from the
@@ -388,6 +396,8 @@ message ObjectTransactionRequest {
     string condition_key = 6;          // if set, evaluate the condition against this entry instead of lock_key (still locking lock_key)
     string route_key = 7;              // ring key identifying the owner filer; a non-owner forwards the whole transaction to it
     bool is_moved = 8;                 // set on a forwarded transaction so the receiver applies it locally instead of forwarding again
+    string request_id = 9;             // idempotency token: the owner records the outcome and replays it on a retried request instead of re-applying
+    string condition_alt_key = 10;     // if the condition's entry is absent, evaluate it against this entry instead (e.g. null slot, else latest version)
 }
 
 message ObjectTransactionResponse {
@@ -477,6 +487,7 @@ message UpdateEntryRequest {
     // the write, under the filer's per-path lock. The caller must route the
     // key's writes to this entry's owner filer for the check to be authoritative.
     WriteCondition condition = 6;
+    bool is_moved = 7;             // set on a forwarded request so the owner applies it locally
 }
 message UpdateEntryResponse {
     SubscribeMetadataResponse metadata_event = 1;
@@ -514,6 +525,7 @@ message DeleteEntryRequest {
     bool is_from_other_cluster = 7;
     repeated int32 signatures = 8;
     int64 if_not_modified_after = 9;
+    bool keep_remote_object = 10;
 }
 
 message DeleteEntryResponse {
@@ -573,6 +585,9 @@ message AssignVolumeResponse {
     // fsync is the storage rule's fsync decision for the assigned path, so the
     // client can carry it onto the volume server upload request.
     bool fsync = 11;
+    // error_code classifies error for callers that must branch on it; error
+    // keeps the human-readable text for clients that predate the field.
+    FilerError error_code = 12;
 }
 
 message LookupVolumeRequest {
@@ -709,6 +724,10 @@ message TraverseBfsMetadataRequest {
 message TraverseBfsMetadataResponse {
     string directory = 1;
     Entry entry = 2;
+    // The sender's per-path commit timestamp (see meta_path_ts.go): the newest
+    // event time this filer committed for the path, 0 when unrecorded so
+    // subscribers fall back to the entry's mtime.
+    int64 entry_ts_ns = 3;
 }
 
 message LogEntry {