filer.remote.sync: -allMounts writes back every mounted directory from one process (#12476)

* filer.remote.sync: -allMounts writes back every mounted directory from one process

A filer with many remote mounts needs one filer.remote.sync process per
mount. -allMounts runs the existing -dir write-back once per directory in
the mount table, in one process: each with its own subscription, client id
(the filer keeps one subscription per client id), offset, processor and
retry loop. A supervisor reads the mount table every -mountsCheckInterval
(5s) and starts a write-back for each mounted directory that has none
running, 10 at a time, 500 ms apart. It never stops one because the live
table changed: a write-back ends itself where the -dir process exits, at the
first mount-table version it receives that ends its mount (#12472), and a
directory still mounted is started again from its saved offset, after a
pause that grows from 1s to 30s while it keeps ending within a minute. A
directory mounted while the process runs, without a saved offset, waits up
to 2 minutes for one (a mount followed by a pull sets it after the pull). A
write-back that ends at an unmount 5 times in a row while the table still
mounts it at the same location is logged as a warning.

To run under a supervisor, a follow:

- returns errMountGone at the end of its mount instead of exiting.
  admitUntilMountEnds takes the end action: the -dir process keeps draining
  and exiting as before; a -allMounts follow admits nothing more (events
  after the end can still arrive in the same batch or before the stream
  closes, including remote.unmount's purge and a remount), ends its stream,
  drains, and saves no offset.
- drains its admitted jobs before it returns, however it ended, so a
  successor never races a job still running for the same path.
- saves its offset only over the value it read at its start or last saved
  (offsetGuard), both at its checkpoints and at its end. Otherwise a
  write-back behind in the log can overwrite the offset remote.unmount
  writes after its purge (or a mount after its mapping write), and its
  successor replays the unmount and never writes the remounted directory
  back. Once it sees the offset moved it saves nothing more and keeps no
  resume position. A value whose write reported an error counts as its own
  when read back.
- reads the stored offset once, before the mount table, and uses it as both
  its start and its guard's value: read after the table, an unmount
  completing in between would start a follow past the unmount's table.
- keeps a resume position for its next attempt only with the mount location
  it was earned under and the offset stored at the time; it is dropped when
  the directory is mounted elsewhere or someone else moved the offset.
- turns a panic in a job into that job's error, so one directory's failure
  does not end every directory's write-back.

Fixes #12471

* filer.remote.sync -allMounts: retire a failed follow whose mount is gone

A follow that failed before its stream reached the unmount (or before it
started) was retried every 30s forever: DetectMountInfo reports the
directory not mounted, which is not errMountGone. No stream runs on the
retry path, so a directory the live table no longer maps is retired there
and started again by reconcile if it is mounted again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync -allMounts: drop the restart pause of an unmounted directory

ended() left notBefore and ends behind for every directory ever mounted.
reconcile now drops them once the directory is neither mounted nor
followed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: log a -dir drain held at the mount end

The -dir exit drained with processor.Drain, so a remote call hung during
an unmount or move held the process silently; the follow's own drain
already warns every 5 minutes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: keep the resume position through a failed mount read

A transient DetectMountInfo error dropped the only position behind the
event a follow failed on. With -timeAgo and no checkpoint saved yet, the
retry then started at a later now minus timeAgo and skipped that event.
The next attempt that reads the table still checks the position against
the mount location and the stored offset.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync: judge a kept resume position by the directory's remote location

resumeFor compared the mount's raw location, so a -dir below a mount kept
its position when the same remote was mounted one level up (or down),
although its paths now map to other keys; a replay from that position
skips startAfterMountOffset and writes old events to the new keys. The
position now carries dirRemoteLocation and is checked against it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync -allMounts: warn about a replayed unmount only from an unmoved offset

The warning counted every end at an unmount while the directory stayed
mounted at one location, so write-backs that each saved progress, or
unmount and remount cycles between two reconciles, also tripped it, and
it told the operator to set the offset to now, skipping events. An end
saves no offset, so a run of ends is one replay only while the stored
offset stays put: a moved offset now starts a new run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer.remote.sync -allMounts tests: wait for the supervisor before moving the frozen clock

A reconcile or end recorded after the clock moved would start its wait
or pause at the moved time, which the frozen clock never passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* remote.mount: save the sync offset right after the mount mapping

A new mount had no saved offset, so filer.remote.sync -allMounts waited
the full allMountsOffsetWait before following it and then started from
the directory's creation time anyway. remote.mount and remote.mount.buckets
now save the time taken just before the mapping write, keeping an offset
remote.unmount saved for a remount.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* remote.mount: save the directory's creation time as the sync offset

The mount time skipped local files already in a -nonempty directory, which
a write-back with no offset replays from the directory's creation. Saving
that creation time keeps the start position and only ends the -allMounts
wait.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* remote.mount: warn instead of failing when the sync offset is not saved

The mount mapping is already saved by then, and a mount without an
offset still starts from the same creation time after the -allMounts
wait. Failing stopped remote.mount.buckets before the remaining buckets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

weed/shell/command_remote_mount_buckets.go

@@ -120,6 +120,7 @@ func (c *commandRemoteMountBuckets) Do(args []string, commandEnv *CommandEnv, wr
 			if err = filer.InsertMountMapping(commandEnv, string(dir), remoteStorageLocation); err != nil {
 				return fmt.Errorf("save mount mapping %s to %+v: %v", dir, remoteStorageLocation, err)
 			}
+			saveMountSyncOffset(commandEnv, writer, string(dir))
 
 		}
 	}