vacuum: report a request dropped because another vacuum is running (#12648)

* vacuum: report a request dropped because another vacuum is running

The master runs one vacuum at a time. A request that arrives while
another one is running, the automatic vacuum or an earlier request, is
dropped: Topology.Vacuum logs "Vacuum is already running" and returns.
The VacuumVolume RPC and /vol/vacuum then answered as if the vacuum had
run, so volume.vacuum printed nothing and exited 0 with no volume
compacted. Where the automatic vacuum is busy for hours, an explicit
volume.vacuum -volumeId could do nothing every time, and nothing said so.

Topology.Vacuum now returns ErrVacuumRunning for a dropped request. The
RPC returns it and /vol/vacuum answers 409 with it. volume.vacuum names
the request that failed, because a list of volumes or collections is
sent as several requests and a vacuum that starts in between fails only
the later ones. Requests are still dropped, not queued, and the
automatic vacuum ignores the error as before.

The integration tests that need their /vol/vacuum request to run ask
again while the master answers 409.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

* docs: note the 409 answer from /vol/vacuum while another vacuum runs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>

test/s3/versioning/s3_copy_versioning_regression_test.go

@@ -44,12 +44,22 @@ func vacuumVolumes(t *testing.T) {
 	require.NotEmpty(t, defaultConfig.MasterEndpoint, "vacuum needs a master endpoint; set MASTER_ENDPOINT")
 	endpoint := strings.TrimRight(defaultConfig.MasterEndpoint, "/") + "/vol/vacuum?garbageThreshold=0.001"
 	httpClient := &http.Client{Timeout: 30 * time.Second}
-	resp, err := httpClient.Get(endpoint)
-	require.NoError(t, err, "vacuum request to %s", endpoint)
-	defer resp.Body.Close()
-	_, err = io.Copy(io.Discard, resp.Body)
-	require.NoError(t, err, "reading the vacuum response")
-	require.Equal(t, http.StatusOK, resp.StatusCode, "vacuum request to %s", endpoint)
+	// The master runs one vacuum at a time and answers 409 to a request it
+	// dropped because its automatic vacuum was running. Ask until ours runs.
+	deadline := time.Now().Add(time.Minute)
+	for {
+		resp, err := httpClient.Get(endpoint)
+		require.NoError(t, err, "vacuum request to %s", endpoint)
+		_, err = io.Copy(io.Discard, resp.Body)
+		resp.Body.Close()
+		require.NoError(t, err, "reading the vacuum response")
+		if resp.StatusCode == http.StatusConflict && time.Now().Before(deadline) {
+			time.Sleep(500 * time.Millisecond)
+			continue
+		}
+		require.Equal(t, http.StatusOK, resp.StatusCode, "vacuum request to %s", endpoint)
+		return
+	}
 }
 
 func requireVersionBody(t *testing.T, client *s3.Client, bucketName, objectKey, versionId string, want []byte, msg string) {