mount: keep retrying writeback chunk uploads through a volume outage (#12465)

* mount: keep retrying writeback chunk uploads through a volume outage

With -writebackCache, close() returns before the data is uploaded. When
the volume servers stay unreachable past the upload's own retry (~20 s),
the chunk was dropped and the file never appeared, with no error to the
application. Retry transient upload errors with the same bytes until they
succeed, unless the dirty pages were discarded. Permanent errors and the
synchronous mode keep failing as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: test writeback upload retries and the cases that still fail

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* operation: keep the filer's read-only verdict on assign failures

UploadWithRetry flattened AssignVolumeResponse errors into a string, so
callers could not tell a read-only rule from an outage. Wrap the filer's
coded error so errors.Is(err, filer_pb.ErrReadOnly) works; the message is
unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: retry writeback uploads the upload layer reassigns, never read-only

A filer proxy that cannot reach the volume answers 500 with an empty
body, which util.IsTransientError does not match, so that outage still
dropped the chunk. Also retry what operation.ShouldReassignUpload takes.
A read-only rule stays final even when its path or message contains a
transient word such as "unavailable".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: stop a writeback upload once its dirty pages are discarded

Destroy() was only seen after a failed attempt, so a truncate during the
retry wait still started another upload, and a chunk that finished after
its pages were replaced was attached to the new content. Check before
each attempt, and drop the chunk instead of attaching it; volume.fsck
reclaims it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: keep Destroy from slipping between the discard check and the append

truncateEntry destroys the pages and then installs the new entry, so a
Destroy landing after the check but before AddChunks put the old chunk
into the replacing entry. Hold one lock across the check and the append,
and take it in Destroy while setting the flag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: leave the Windows interrupt drain to the capped filesystem hook

Interrupt hooks run in reverse registration order, so the Windows hook ran
before the filesystem hook and waited on pending writeback flushes without a
bound. With writeback uploads now retrying through an outage, Ctrl-C could
wait until the volume servers came back. Unmount only, as on Linux, and let
the filesystem hook drain for at most 30 s and keep the write cache.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* mount: test a writeback upload whose assign keeps timing out

With every volume server gone the assign RPC times out until the filer
client gives up; that error must be retried like a refused upload.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

weed/command/mount_windows.go

@@ -123,12 +123,9 @@ func RunMount(option *MountOptions, umask os.FileMode) bool {
 	host.Notify(seaweedFileSystem)
 
 	grace.OnInterrupt(func() {
-		// The signal handler exits the process as soon as the hooks return, so
-		// anything still queued has to be flushed here rather than after Serve.
-		// WaitForAsyncFlush is idempotent, so the post-Serve call below is
-		// harmless if both run.
+		// The filesystem's own hook runs next and drains queued writeback
+		// flushes for at most 30 s, keeping the write cache if they are not done.
 		host.Unmount()
-		seaweedFileSystem.WaitForAsyncFlush()
 	})
 
 	serveMountGrpc(seaweedFileSystem, mountSocketListener)