s3: fail over routed object writes when the owner filer is unreachable (#10251)

* s3: fail over routed object writes when the owner filer is unreachable

A routed object write (multipart completion, PUT, delete, versioned
finalize, metadata replace) dialed the ring-selected owner filer
directly with no failover. After a filer restarts onto a new address the
lock ring can still name the old one, so every routed write hangs on the
dead address until the gateway is restarted; CompleteMultipartUpload in
particular exceeds client timeouts.

Route the transaction through withFilerClientFailover, skipping an owner
that recently failed, so a live filer forwards it to the real owner by
route_key. Mirrors the read path's getObjectEntryRoutedByKey.

* s3: fail over bucket-config writes when the owner filer is unreachable

patchBucketEntry dialed the bucket's ring owner directly, so a restarted
filer's stale ring address hung every bucket-config write (versioning,
lifecycle, object lock, ownership, ACL, policy, CORS) - the same failure
as routed object writes. Route it through objectTxnOnFiler so it skips an
unreachable owner and a live filer forwards by route_key.

weed/s3api/s3api_object_routed_write.go

@@ -148,13 +148,29 @@ func singleStrongETag(v string) (string, bool) {
 	return strings.Trim(v, `"`), true
 }
 
+// objectTxnOnFiler applies a routed transaction, preferring the object's owner
+// filer but failing over to a live filer when the owner is unreachable, so a
+// restarted filer's stale ring address (e.g. a rolled K8s pod's old IP) can't hang
+// the write. The reached filer forwards to the real owner by route_key, keeping its
+// per-path lock authoritative. Mirrors getObjectEntryRoutedByKey on the read path.
 func (s3a *S3ApiServer) objectTxnOnFiler(owner pb.ServerAddress, req *filer_pb.ObjectTransactionRequest) (*filer_pb.ObjectTransactionResponse, error) {
 	var resp *filer_pb.ObjectTransactionResponse
-	err := pb.WithFilerClient(false, 0, owner, s3a.option.GrpcDialOption, func(client filer_pb.SeaweedFilerClient) error {
+	txn := func(client filer_pb.SeaweedFilerClient) error {
 		var e error
 		resp, e = client.ObjectTransaction(context.Background(), req)
 		return e
-	})
+	}
+	if s3a.filerClient == nil {
+		err := pb.WithFilerClient(false, 0, owner, s3a.option.GrpcDialOption, txn)
+		return resp, err
+	}
+	// Skip an owner whose recent write hit a transport error so a dead/stale
+	// address isn't re-dialed every request.
+	preferred := owner
+	if s3a.ownerRecentlyUnreachable(owner) {
+		preferred = ""
+	}
+	err := s3a.withFilerClientFailover(preferred, false, txn)
 	return resp, err
 }