* volume: a delete cancels the writes of its needle still in flight
A write queued behind the volume locks, for example behind another write's
fsync on a busy disk, is not in the needle map yet. A delete of that needle
in the meantime finds nothing, and the write lands afterwards, so the needle
comes back on this copy only while the other replicas have it deleted.
Each volume now tracks its writes from entry until they return. A delete
cancels the ones for its needle, and a cancelled write that landed is
tombstoned before it returns, unless the needle was deleted or rewritten
since.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: a replicate delete that finds nothing cancels the pending write
Rolling back a failed chunk upload sends DELETE ?type=replicate to every
holder, including the slow one whose write timed out on the client. That
holder has not indexed the write yet, so the delete answered 404 and the
write landed live afterwards. Cancel the needle's pending writes, then look
again in case one finished in between. A plain delete that finds nothing
still answers 404 without cancelling: it is not sent on to the replicas.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: a delete cancels the writes of its needle still in flight
Mirrors the Go volume. Volume::track_pending_write registers a write until
its guard drops, delete_needle cancels the needle's pending writes, and
settle_pending_write tombstones a cancelled write that landed unless the
needle was deleted or rewritten since.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: track uploads and cancel them on a replicate delete
An upload is tracked from the handler, and a durable one from submission
to the write queue, where the guard travels with the request: a handler
dropped while the write waits in the queue does not untrack it, and the
worker settles it after the batch. A replicate delete that finds nothing
cancels the needle's pending writes and looks again, like the Go handler.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: take a cancelled write back in the lock span that publishes it
The write tombstoned itself only after it returned, with the volume locks
released. In that gap a compaction could move the record so the offset
check skipped it, a close could drop it, and an identical write sent after
the delete could match the live record and report success before the
tombstone removed it. The check now runs right after the publish, under
the same dataFileSyncLock and dataFileAccessLock hold, in the plain, fsync
and batch write paths.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: a delete cancels only the writes carrying its cookie
A delete with the right needle id but the wrong cookie could cancel an
in-flight write and get it tombstoned. Each pending write now records its
cookie, and a cancel only marks the matching ones.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: take a cancelled write back in the lock span that publishes it
Mirror the Go change: settle_pending_write ran after write_needles_grouped
returned, so a compaction, a close or an identical write later in the same
batch could slip in before the tombstone. drop_cancelled_write now runs
right after the publish, under the same sync and access lock hold, in the
synced run, the unsynced run and write_needle_unsynced_or_refused.
write_pending_needles carries each write's PendingWrite down to that
point.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: a delete cancels only the writes carrying its cookie
Mirror the Go change: a delete with the right needle id and the wrong
cookie no longer cancels an in-flight write. Each tracked write keeps its
cookie next to its cancel flag.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: refuse a queued write whose volume was remounted
The write queue tracks a write on the volume found at submit, but the
worker looks the volume up again by id. After a remount, a delete reaches
the new volume and cannot cancel the write registered on the old one, so
the worker now refuses such a write instead of landing it on the new
volume.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: decode the delete's type parameter like Go's FormValue
The delete handler matched the raw query text for type=replicate, so a
percent-encoded name or value was not treated as a replica delete and did
not cancel the write it undoes. Parse the query the way the upload handler
does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: test a replica delete landing in a write's fsync window
Drive DeleteHandler while a durable write is parked in its .dat fsync,
appended but not indexed: the replica delete answers 404 and the write is
taken back when it publishes, while a plain delete leaves it live.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: flush a take-back tombstone before the index row that points at it
A durable write taken back during its fsync appends the tombstone after the
.dat sync, and the .idx sync that follows made its row durable while the
record itself could still be lost, which reloads the volume read only. The
fsync and batch paths now sync the .dat again first when a take-back
appended a tombstone; a failure stops the volume taking writes like a failed
index sync.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: flush a take-back tombstone before the index row that points at it
A durable write taken back in write_synced_run appends its tombstone after
the .dat sync, so the .idx flush that follows could make the row durable
while the tombstone record itself was still in the page cache. The run now
syncs the .dat again first when a take-back appended a tombstone; a failure
stops the volume taking writes like a failed index flush.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: a skip-cookie batch delete cancels the writes of the file id it names
BatchDelete with SkipCookieCheck left the needle's cookie at zero, so the
delete's cancel matched none of the in-flight writes and a retry queued
behind it could land afterwards. The non-EC delete now carries the file
id's cookie, as the Rust volume server already does; the EC path keeps
cookie 0, which DeleteEcShardNeedle reads as "skip the check".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: release a held test write before the store closes
A failed wait ended the test before release ran, so the writer stayed parked
in its fsync and the store cleanup blocked on its lock until the package
timed out.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(volume): do not read a needle's data to delete it
A delete read the needle's whole payload from disk only to discard it.
The delete handler and BatchDelete need the needle's cookie and its
chunk-manifest flag, which live in the header and trailer, but fetched
them with a full needle read, the same call a GET uses.
readNeedle now reads the header and trailer only when MustMetaOnly is
set, whatever the needle's size, and the delete paths ask for that. The
Rust volume server honours must_meta_only the same way for its delete
paths. A compressed needle or a chunk manifest still gets the full read,
as before, because their callers need the bytes.
HEAD is unchanged. It used to set MustMetaOnly, which nothing read; it
now asks for AttemptMetaOnly alone, so it keeps reading a needle up to
PagedReadLimit whole. On a spinning disk that one read is cheaper than a
header read followed by a trailer read.
Version 1 needles are always read in full. They have no data size field
and no metadata after the data, which ReadNeedleMeta relies on, so a
metadata-only read of one failed with "invalid needle meta size". That
already happened for version 1 needles above PagedReadLimit read with
AttemptMetaOnly.
A delete therefore no longer verifies the data CRC of the needle it
removes. The cookie check is unchanged.
* fix(volume): read transformed images fully on HEAD too
A HEAD carrying resize or crop params reports the transformed length,
which a metadata read cannot produce. Go and Rust both answered such
HEADs from the source header, returning the stored size instead of the
cropped or resized one.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: an already-deleted EC needle is not a delete failure
Deleting a needle that is already gone is what the caller asked for, and the
non-EC paths have always said so: BatchDelete reports StatusNotModified when
DeleteVolumeNeedle finds nothing to do, and DeleteHandler answers 404 from its
ReadVolumeNeedle pre-check. The EC branches had no such case, so ErrorDeleted
fell through to a generic failure -- 500 from both, and DeleteHandler also
counted it in VolumeServerFileWriteFailures, inflating a failure metric on a
replayed or duplicated delete.
The filer already tolerates this by string-matching "already deleted" on the
result, which leaves an error message load-bearing; the status is now right at
the source instead.
Claude-Session: https://claude.ai/code/session_01P3pE6J2UPFp6G3ksfMV4s1
* volume: close the EC fixture's disk location
Close stops the location's disk-space goroutine and releases the mounted EC
volume's file handles, which otherwise live until the test binary exits.
Claude-Session: https://claude.ai/code/session_01P3pE6J2UPFp6G3ksfMV4s1
* fix: reject overflowing needle ID deltas
Problem: Parsing a file ID with a delta can wrap a valid maximum needle ID back to zero without returning an error.
Root cause: Needle.ParsePath added the parsed uint64 delta without checking whether the sum exceeded the needle ID range.
Fix: Compare the delta with the remaining uint64 capacity before addition and return a contextual overflow error when it does not fit.
Validation: go test ./weed/storage/needle -run ^TestNeedleParsePathRejectsDeltaOverflow$ -count=1; go test ./weed/storage/needle -count=1; git diff --check 10cdaf381875492a2c752d1038797e96ff18208f..HEAD
Co-authored-by: Codex <noreply@openai.com>
* fix: propagate needle ID delta parse errors
Co-authored-by: Codex <noreply@openai.com>
* print the needle id in hex in the delta overflow error
* batch delete: keep processing after a cookie mismatch
* rust volume: reject overflowing needle id deltas
---------
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
Run BatchDelete through checkGrpcAdminAuth like the other destructive
volume-server RPCs (VolumeDelete, DeleteCollection, vacuum, EC, ...),
so a whitelist-configured server denies non-admin callers.
* Boostrap persistent state for volume servers.
This PR implements logic load/save persistent state information for storages
associated with volume servers, and reporting state changes back to masters
via heartbeat messages.
More work ensues!
See https://github.com/seaweedfs/seaweedfs/issues/7977 for details.
* Block RPC operations writing to volume servers when maintenance mode is on.
