s3, filer: bound the wait for an upload slot (#12559)

* util: add WaitUntil, a cond wait bounded by a context and a timeout

The caller holds the cond lock. Cancellation and the timeout wake the
waiter through a broadcast taken under the same lock, so no wakeup is lost.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: bound the wait for an upload slot

A PUT that arrived while the concurrent upload limits were exhausted waited
on the condition variable with no bound. The handler reads nothing from the
body while it waits, so a client that gave up and reset its connection went
unnoticed: the socket and goroutine stayed pinned until a slot freed, and a
stalled volume server grew the gateway's fd count by one per aborted PUT.

The wait now ends on InflightUploadDataTimeout with 503 SlowDown, which S3
SDKs retry, and on request cancellation. An admitted upload takes its slot
under the lock, and a finished upload broadcasts, so freed bytes admit every
waiter that fits instead of one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: bound the wait for an upload slot

The filer's PUT/POST upload limits queued requests the same unbounded way.
Reuse the bounded wait: reject with 429 after InflightUploadDataTimeout, as
the volume server does, and stop on request cancellation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* command: add -inflightUploadDataTimeout for the S3 gateway and the filer

weed s3 -inflightUploadDataTimeout, weed filer -inflightUploadDataTimeout and
-s3.inflightUploadDataTimeout, weed server and weed mini
-filer.inflightUploadDataTimeout and -s3.inflightUploadDataTimeout. The
default of 60s matches the volume server's flag of the same name; 0 keeps
the old unbounded wait.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3: reject a queued upload without reading its body

WriteErrorResponse drains the request body before answering. Behind the
stats recorder the drain has no read deadline, and reading the body sends an
Expect: 100-continue client the go-ahead to upload what is about to be
discarded. Answer the rejection with an empty-body copy of the request so
the drain is skipped; net/http closes the connection when a large unread
body remains.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3, filer: release an upload slot under the cond lock

A waiter checks the counters under the cond lock and then waits. A release
that decremented and broadcast without the lock could land between the two,
so the waiter slept with a slot free until the next release or its timeout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* s3, filer: close the HTTP/1 connection when rejecting a queued upload

net/http reads up to 256 KiB of an unread request body before sending a
keep-alive reply, so a client that stalls its body held back the 503 or
429 until the connection idle timeout. Connection: close skips that read.
HTTP/2 is left alone: there it would shut down the whole connection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

weed/util/cond_wait.go

@@ -30,3 +30,29 @@ func WaitWithTimeout(ctx context.Context, cond *sync.Cond, timer *time.Timer) in
 		return HttpStatusCancelled
 	}
 }
+
+// WaitUntil waits on cond, whose lock the caller holds, until ready returns
+// true, ctx is done, or timeout elapses; a zero timeout waits without a bound.
+func WaitUntil(ctx context.Context, cond *sync.Cond, timeout time.Duration, ready func() bool) error {
+	if ready() {
+		return nil
+	}
+	if timeout > 0 {
+		var cancel context.CancelFunc
+		ctx, cancel = context.WithTimeout(ctx, timeout)
+		defer cancel()
+	}
+	stop := context.AfterFunc(ctx, func() {
+		cond.L.Lock()
+		cond.Broadcast()
+		cond.L.Unlock()
+	})
+	defer stop()
+	for !ready() {
+		if err := ctx.Err(); err != nil {
+			return err
+		}
+		cond.Wait()
+	}
+	return nil
+}

weed/util/cond_wait_test.go

@@ -0,0 +1,53 @@
+package util
+
+import (
+	"context"
+	"errors"
+	"sync"
+	"testing"
+	"time"
+)
+
+func TestWaitUntilReturnsWhenReady(t *testing.T) {
+	cond := sync.NewCond(new(sync.Mutex))
+	ready := false
+	go func() {
+		time.Sleep(20 * time.Millisecond)
+		cond.L.Lock()
+		ready = true
+		cond.L.Unlock()
+		cond.Broadcast()
+	}()
+	cond.L.Lock()
+	err := WaitUntil(context.Background(), cond, time.Minute, func() bool { return ready })
+	cond.L.Unlock()
+	if err != nil {
+		t.Fatalf("WaitUntil: %v", err)
+	}
+}
+
+func TestWaitUntilTimesOutWithoutWakeup(t *testing.T) {
+	cond := sync.NewCond(new(sync.Mutex))
+	start := time.Now()
+	cond.L.Lock()
+	err := WaitUntil(context.Background(), cond, 50*time.Millisecond, func() bool { return false })
+	cond.L.Unlock()
+	if !errors.Is(err, context.DeadlineExceeded) {
+		t.Fatalf("got %v, want deadline exceeded", err)
+	}
+	if elapsed := time.Since(start); elapsed < 50*time.Millisecond || elapsed > 5*time.Second {
+		t.Fatalf("returned after %v", elapsed)
+	}
+}
+
+func TestWaitUntilStopsOnCancel(t *testing.T) {
+	cond := sync.NewCond(new(sync.Mutex))
+	ctx, cancel := context.WithCancel(context.Background())
+	time.AfterFunc(20*time.Millisecond, cancel)
+	cond.L.Lock()
+	err := WaitUntil(ctx, cond, 0, func() bool { return false })
+	cond.L.Unlock()
+	if !errors.Is(err, context.Canceled) {
+		t.Fatalf("got %v, want canceled", err)
+	}
+}