* volume: validate the file extension in CopyFile and ReceiveFile
CopyFile and ReceiveFile build an on-disk path from the client-supplied
Ext. Both are intentionally ungated for cluster-internal peers, so a
value like "/../../x" is joined onto the volume directory and, once
path-cleaned, resolves outside it -- an EC-shard receive can then write,
and CopyFile read, anywhere the process can reach.
Constrain Ext to a real suffix (a leading dot followed by alphanumerics)
before it is used to build any path, so it can no longer carry a
separator or a parent reference.
* test: use an alphanumeric missing-file extension in the copy variants
The not-found and stop-offset-zero cases used ".definitely-missing" as a
deliberately absent source. The extension is now validated, and the hyphen
makes it invalid, so switch to ".missing" -- still a nonexistent file, but a
real extension shape.
* volume: validate the collection in CopyFile and ReceiveFile
The client-supplied Collection is folded into the on-disk path as
"<collection>_<vid>" by VolumeFileName and EcShardBaseFileName, both joined
with path.Join / util.Join. A Collection carrying a separator, e.g.
"../../x", therefore path-cleans to a target outside the volume directory,
the same escape the extension check just closed. Reject a collection that is
a bare parent reference or holds a separator; ordinary names ('.', '-' and
all) still pass.
The staged-new-volume placement skipped a disk holding the vid's EC shards using only the in-memory ecVolumes map, missing a shard present on disk but not mounted. Also scan the candidate disk for <vid>.ecNN files, so the promise holds regardless of mount state.
Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu
volume: skip a shard-holding disk when staging a decoded volume
ReceiveFile staged-new-volume mode picked any free disk of the target
medium. Skip a disk that already holds the vid's EC shards (Go
DiskLocation.FindEcVolume / Rust ec_volumes), so a decoded .dat never
lands in the same directory as a shard. This lets a caller safely stage
onto a shard host that has a spare disk, instead of requiring a host with
no shard of the vid at all.
Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu
Decoding EC shards back to a normal volume in place reconstructs <vid>.dat
in the shards' own directory, so the vid is momentarily registered as both
an EC and a normal volume in one location — the load/scan path then sees it
as both, risking mount ambiguity and needle loss. VolumeEcShardsToVolume
still supports that in-place path; this adds the primitives to decode onto
a *clean* peer instead:
- ReceiveFile gains a staged-new-volume mode: when the volume does not
exist here and ReceiveFileInfo.disk_type is set, pick a free-slot disk
of that medium and write <base><ext>.copying (not a valid volume name,
so the scanner never half-loads a partial push).
- VolumeEcShardsToVolume gains from_staged: adopt the pushed .dat/.idx/
.vif — rename .copying into place under a .note in-progress marker,
then mount — so <vid> lands on the peer only as a normal volume.
The caller decodes the shards off-box and streams the finished volume to a
peer holding no shard of the vid on the target medium. Go and Rust volume
servers get identical handlers. Proto: ReceiveFileInfo.disk_type (12; 8-11
reserved for versioned-EC), VolumeEcShardsToVolumeRequest.from_staged (3) +
disk_type (4).
Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu
* volume: gate the admin RPCs that only shell and workers call
checkGrpcAdminAuth covered 19 of the 48 VolumeServer RPCs, so an operator who
sets -whiteList expecting it to cover the gRPC surface gets partial coverage.
Extend it to ten that mutate state and are only ever called by the shell or a
worker: SetState, VolumeCopy, the EC generate/rebuild/copy/unmount/to-volume
pair, both tier moves, and VolumeTailReceiver. That is safe because the same
callers already reach gated RPCs today -- VolumeMarkReadonly, VacuumVolume*,
VolumeEcShardsDelete, VolumeDelete -- so a whitelist deployment already lists
those hosts. Nothing here is on a master or peer path, which is what made the
earlier fail-closed gate break multi-host clusters.
The split is by caller rather than by blast radius: the guard matches a peer IP
against the whitelist, and a whitelist holds masters, shell hosts and workers,
not every peer volume server. Gating a call one volume server makes to another
would break replication, EC and tiering, so those stay open.
Two test fakes embedded a nil grpc.ServerStream and only implemented Send;
they now implement Context, which the streaming RPCs read to authorize.
* volume: fail the build when a gRPC method skips the admin gate
The admin gate is an opt-in list in a 48-method service, which is how it
drifted down to covering 19 of them: nothing tied adding an RPC to deciding
whether it needed the gate.
Parse volume_server.proto, walk the AST of every *VolumeServer method, and
require each RPC to either call checkGrpcAdminAuth or appear in
ungatedVolumeServerRPCs with the reason it stays open. A stale entry naming an
RPC that no longer exists fails too, so the list can't quietly stop exempting
anything.
The exemptions are the cluster-internal calls -- replica sync, EC shard
distribution, vacuum reads, backup, tailing -- plus the read-only and liveness
RPCs. Closing the cluster-internal ones needs a peer identity rather than an
IP whitelist; recording them here makes that a visible decision instead of an
omission.
The AST walk also corrects the count: a line-window scan credits
VacuumVolumeCheck and VolumeServerStatus with a neighbouring function's guard.
* volume: copying a remote-backed volume only needs space for the index
VolumeCopy sized its target-location check by the source .dat even when
that .dat lives in a cloud tier and only .idx/.vif land locally, so
re-replicating a tiered volume demanded the full remote size in free
disk. Require the index size instead.
* shell: volume.tier.upload keeps volume replicas
Tiering a replicated volume deleted every replica but the upload
source, leaving one server holding the only .idx and the only .vif
that knows the remote object key — losing that server orphaned the
volume even though its data sat intact in the cloud.
Replicate the uploaded .idx/.vif onto the other replica servers
instead (VolumeCopy skips the .dat for remote-backed volumes), so all
replicas serve reads from the same remote object and the volume keeps
its replica count. An already-tiered replica is preferred as the
upload source, so a rerun after a partial failure reuses the existing
remote object instead of uploading a second copy under a new key.
* shell: group tier upload locations instead of re-prepending
* rust volume: copying a remote-backed volume only needs space for the index
Mirror the Go VolumeCopy change: size the free-location check by the
source .idx when the .dat lives in a cloud tier, since only .idx/.vif
land locally.
* volume: verify before destroy in VolumeCopy and replication repair
Four data-safety fixes around copy/repair paths that could destroy or
resurrect data before verifying the source or survivors.
(a) VolumeCopy no longer deletes a pre-existing local replica up front.
The delete is deferred until ReadVolumeFileStatus on the source succeeds,
so a transient source outage (or a retry after one) can no longer wipe a
healthy destination replica. Gated on source readability only; size/count
comparisons are intentionally not used because they invert legitimately
after divergent vacuum/compaction. Mirrored in the Rust volume server.
(b) volume.check.disk no longer resurrects vacuumed-deleted needles. A
key present-and-live on the source but entirely absent on the target is
ambiguous: it may be a genuine missing write, or a needle deleted on the
target and then vacuumed (its index entry and any tombstone are gone). An
individual needle AppendAtNs has no monotonic relation to a vacuum
watermark, so the old cutoff heuristic could not tell them apart. Without
positive proof the absence is a missing write, the safe default is to NOT
push it back. Tradeoff: a real missing write may go unrepaired until a
tombstone-aware path exists, but we never raise back deleted data.
(c) Over-replication trim no longer resurrects needles or removes the
wrong replica. The pre-delete sync now runs read-only (divergence check
only) instead of writing the doomed replica's needles into the survivor.
pickOneReplicaToDelete only ever removes the smallest of multiple healthy
writable replicas; it refuses the trim when doing so would leave only
read-only/integrity-flagged survivors, since file_count>0 alone cannot
prove the survivor's .dat is readable.
(d) Incomplete-volume (.note) cleanup keeps the shared .vif when an .ecx
for the same vid coexists on the disk, so removing an interrupted regular
copy cannot strip a coexisting EC volume's info file. VolumeCopy now
surfaces .note write/remove errors instead of ignoring them. In the Rust
volume server (where a persisting note is actually reachable) the .note
check moves below the empty-stub sweep and EC validation, keeps the .vif
on EC coexistence, and the mount path fails when a .note still persists.
* shell: scope the over-replication writable-survivor guard to the trim path only
The writable-survivor guard (never trim down to a read-only survivor) lived
inside the shared pickOneReplicaToDelete, so it also gated the misplaced-volume
relocation via pickOneMisplacedVolume -- a misplaced read-only volume (e.g. a
full one) would silently stop being rebalanced. Extract pickSmallestReplica
for the relocation path (which deletes-and-recreates and must act on read-only
replicas), and keep the writable-survivor guard only in pickOneReplicaToDelete
used by the over-replication trim.
* seaweed-volume: recompute keep_vif after invalid-EC cleanup in the .note path
keep_vif used the pre-validation ecx_exists snapshot, so when the EC-validation
step above removed the invalid .ecx/shards, the .note cleanup still preserved a
now-orphaned .vif. Re-check .ecx existence at cleanup time, matching the Go
hasEcxFile re-check.
* shell: keep placement when picking an over-replication victim to delete
The trim picked the smallest writable replica without regard to placement, so
it could delete the only replica in a required failure domain (e.g. with "100"
and replicas dc1 + two in dc2, deleting dc1 leaves both survivors in dc2).
Prefer a writable replica whose removal still satisfies placement, falling back
to the smallest writable only when none does.
A Canceled/DeadlineExceeded from the caller's per-request context was
treated like a dead channel: it closed the shared cached ClientConn and
cancelled every other in-flight RPC on it with "the client connection is
closing". Under a burst of concurrent chunk assigns (e.g. a large S3
multipart upload) one slow assign hitting its 10s attempt timeout could
poison the connection for all the rest, cascading into a flood of 500s.
Thread the caller's context into shouldInvalidateConnection and only
invalidate on Canceled/DeadlineExceeded while that context is still live,
which isolates the genuine stale-channel signal (a peer restart behind a
k8s Service VIP). To carry the context, add a ctx parameter to the
existing WithGrpcClient, WithMasterClient, and WithMasterServerClient; the
master assign and volume-lookup paths pass their per-attempt context and
every other caller passes context.Background().
* fix(ec_bitrot): cap sidecar block_size in ValidateBitrotManifest
A sidecar loaded from disk (or supplied via a backfill/peer RPC) could carry a
huge power-of-two block_size that passed validation, then force a multi-GiB
scratch-buffer allocation in scrub/verify. Add a shared MaxBitrotBlockSize
(64 MiB) constant, enforce it as an upper bound in isPow2MultipleOf1MiB, and
derive the volume flag cap from the same constant so they cannot drift.
* fix(ec_bitrot): don't destroy a valid destination sidecar on an optional copy
writeToFile opened the destination with O_TRUNC before knowing whether the
source had the file, so an optional copy (ignoreSourceFileNotFound) from a source
that lacks the .ecsum truncated and then removed a valid pre-existing destination
sidecar. Stage the optional copy into a temp sibling and commit it with an atomic
rename only when the source actually delivered the file; a missing source is now
a no-op. Mandatory copies keep their in-place behavior.
* fix(ec_distribute): remove partial files on copy stream error
writeToFile opens the destination with O_TRUNC and streams into it. On
a mid-stream receive / write / cancellation error it returned the
failure but left the destination behind in whatever state had been
written so far — typically 0 bytes when the source errored before
sending any FileContent. VolumeEcShardsCopy distributes .ecx by
calling doCopyFile, so this same stub-leaving behaviour produced the
0-byte .ecx files seen on EC encoding failures: the source claims a
non-zero ModifiedTsNs (so the existing "source not found" cleanup
doesn't fire), the stream then errors immediately, and the receiver
ends up with a 0-byte .ecx that downstream code mistook for a valid
empty index.
Clean up the partial file on every error path that returns from the
streaming loop (receive, write, and cancellation). Skip cleanup when
isAppend=true so resumable appends keep their existing content. As
defense in depth, VolumeEcShardsCopy also stats the .ecx after copy
and removes / errors on a 0-byte result so the orchestrator can pick
a different source.
The Rust volume server has only the source side of CopyFile (no
client-side stream-to-disk consumer) and no .ecx subsystem yet, so
this fix has no Rust mirror.
* fix(ec_distribute): close file before remove, fail fast on stat error
Address review feedback:
- writeToFile's mid-stream removeIncomplete called os.Remove while the
destination file handle was still open. On Windows os.Remove fails
while a handle is open, so the cleanup wouldn't run there. Wrap the
handle close in a once-only helper, call it from removeIncomplete
and from the existing "source not found" cleanup, and keep a deferred
close as the safety net for the normal-return path.
- VolumeEcShardsCopy's post-copy .ecx check silently passed when
os.Stat returned an error: doCopyFile had reported success but if
the file was already gone, unreadable, or somehow a directory, the
orchestrator only learned at mount time with no useful context.
Treat any non-nil stat error and any directory result as a copy
failure here and surface it immediately.
* fix(ec): blanket-clean every destination over the full shard range
The previous cleanup pass walked t.sources only, with the shard ids the
topology had reported at detection time. In the wild, a destination can
end up with EC shards mounted that the topology snapshot didn't list —
shards on a sibling disk that hadn't heartbeated, or shards left over
from a concurrent attempt's mount step. FindEcVolume still returns
true, so the next ReceiveFile trips the mounted-volume guard.
Cleanup now unions t.sources (with ShardIds) and t.targets and issues
unmount + delete over [0..totalShards-1] on each. Both RPCs are
idempotent on missing shards, so the wider sweep is free.
Two new tests cover the gap: shards mounted beyond what t.sources
lists, and a target-only destination with no source row.
* log(ec): include disk_id in EC unmount/delete/refusal log lines
The current logs identify the volume and shard but leave disk_id off,
which makes the cross-server cleanup story hard to follow when
multiple disks of one server hold pieces of the same volume:
UnmountEcShards 4121.1 -> add disk_id
ec volume video-recordings_4121 shard delete [1 5] -> add per-loc disk_id
volume server X:Y deletes ec shards from 4121 [...] -> add disk_id
ReceiveFile: ec volume 4121 is mounted; refusing... -> add disk_ids
ReceiveFile's refusal now names the disk_ids actually holding the
mount so operators can see whether the next cleanup pass needs to
target a sibling disk. Added Store.FindEcVolumeDiskIds /
Store::find_ec_volume_disk_ids as the supporting primitive.
Mirrored in seaweed-volume/src/ (unmount log in Store::unmount_ec_shard,
heartbeat delete log in diff_ec_shard_delta_messages, refusal in the
ReceiveFile handler).
* test(ec): stub VolumeEcShardsUnmount/Delete on the fake volume server
The plugin-worker EC tests boot a fake volume server that embeds
UnimplementedVolumeServerServer. After the worker started calling
VolumeEcShardsUnmount + VolumeEcShardsDelete pre-distribute, the
default Unimplemented response surfaced as fourteen "method not
implemented" errors and TestErasureCodingExecutionEncodesShards
failed. Both RPCs are no-ops here — nothing on the fake server has
mounted state or persisted shard files to remove.
* fix(volume): don't fatal on missing .idx for remote-tiered volume
A .vif left behind without its .idx (orphaned by a crashed move, partial
copy, or hand-edit) would trip glog.Fatalf in checkIdxFile and take the
whole volume server down on boot, killing every healthy volume on it
too. For remote-tiered volumes treat it as a per-volume load error so
the server can come up and the operator can clean up the stray .vif.
Refs #9331.
* fix(balance): skip remote-tiered volumes in admin balance detection
The admin/worker balance detector had no equivalent of the shell-side
guard ("does not move volume in remote storage" in
command_volume_balance.go), so it scheduled moves on remote-tiered
volumes. The "move" copies .idx/.vif to the destination and then calls
Volume.Destroy on the source, which calls backendStorage.DeleteFile —
deleting the remote object the destination's new .vif now points at.
Populate HasRemoteCopy on the metrics emitted by both the admin
maintenance scanner and the worker's master poll, then drop those
volumes at the top of Detection.
Fixes#9331.
* Apply suggestion from @gemini-code-assist[bot]
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* fix(volume): keep remote data on volume-move-driven delete
The on-source delete after a volume move (admin/worker balance and
shell volume.move) ran Volume.Destroy with no way to opt out of the
remote-object cleanup. Volume.Destroy unconditionally calls
backendStorage.DeleteFile for remote-tiered volumes, so a successful
move would copy .idx/.vif to the destination and then nuke the cloud
object the destination's new .vif was already pointing at.
Add VolumeDeleteRequest.keep_remote_data and plumb it through
Store.DeleteVolume / DiskLocation.DeleteVolume / Volume.Destroy. The
balance task and shell volume.move set it to true; the post-tier-upload
cleanup of other replicas and the over-replication trim in
volume.fix.replication also set it to true since the remote object is
still referenced. Other real-delete callers keep the default. The
delete-before-receive path in VolumeCopy also sets it: the inbound copy
carries a .vif that may reference the same cloud object as the
existing volume.
Refs #9331.
* test(storage): in-process remote-tier integration tests
Cover the four operations the user is most likely to run against a
cloud-tiered volume — balance/move, vacuum, EC encode, EC decode — by
registering a local-disk-backed BackendStorage as the "remote" tier and
exercising the real Volume / DiskLocation / EC encoder code paths.
Locks in:
- Destroy(keepRemoteData=true) preserves the remote object (move case)
- Destroy(keepRemoteData=false) deletes it (real-delete case)
- Vacuum/compact on a remote-tier volume never deletes the remote object
- EC encode requires the local .dat (callers must download first)
- EC encode + rebuild round-trips after a tier-down
Tests run in-process and finish in under a second total — no cluster,
binary, or external storage required.
* fix(rust-volume): keep remote data on volume-move-driven delete
Mirror the Go fix in seaweed-volume: plumb keep_remote_data through
grpc volume_delete → Store.delete_volume → DiskLocation.delete_volume
→ Volume.destroy, and skip the s3-tier delete_file call when the flag
is set. The pre-receive cleanup in volume_copy passes true for the
same reason as the Go side: the inbound copy carries a .vif that may
reference the same cloud object as the existing volume.
The Rust loader already warns rather than fataling on a stray .vif
without an .idx (volume.rs load_index_inmemory / load_index_redb), so
no counterpart to the Go fatal-on-missing-idx fix is needed.
Refs #9331.
* fix(volume): preserve remote tier on IO-error eviction; fix EC test target
Two review nits:
- Store.MaybeAddVolumes' periodic cleanup pass deleted IO-errored
volumes with keepRemoteData=false, so a transient local fault on a
remote-tiered volume would also nuke the cloud object. Track the
delete reason via a parallel slice and pass keepRemoteData=v.HasRemoteFile()
for IO-error evictions; TTL-expired evictions still pass false.
- TestRemoteTier_ECEncodeDecode_AfterDownload deleted shards 0..3 but
called them "parity" — by the klauspost/reedsolomon convention shards
0..DataShardsCount-1 are data and DataShardsCount..TotalShardsCount-1
are parity. Switch the loop to delete the parity range so the
intent matches the indices.
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* fix(volume_server): pin EC shard auto-select to the .ecx-owning disk (#9212)
ec.rebuild only sets CopyEcxFile=true on the first shard sent to the
rebuilder; subsequent shards rely on VolumeEcShardsCopy / ReceiveFile
auto-select to land on the same disk. The old auto-select used
FindEcVolume (in-memory) to detect the "already has this volume" case.
Mid-rebuild, no EC volume has been mounted yet on the destination, so
FindEcVolume returns nothing and the fallback picks "any HDD with free
space" — which can split shards from their .ecx across disks of the
same node and feed the orphan-shard layout reported in #9212 / fixed
on the loader side in #9244.
Add Store.FindEcShardTargetLocation as the canonical placement
primitive: prefer a mounted EC volume, then a disk that has the .ecx
on disk, then any HDD, then any disk. DiskLocation.HasEcxFileOnDisk is
the new on-disk check, and it looks at IdxDirectory first with a
fallback to Directory to handle .ecx written before -dir.idx was
configured.
Both VolumeEcShardsCopy and ReceiveFile now route through the new
helper, dropping their duplicated 4-level fallback ladder. No protocol
changes; explicit DiskId callers are unaffected.
* fix(volume_server): treat directories named *.ecx as no-match in HasEcxFileOnDisk
os.Stat(".ecx") succeeds for both files and directories. If something
happens to leave a directory named X.ecx in the data or idx folder,
HasEcxFileOnDisk would currently report true and FindEcShardTargetLocation
would route shards to that disk — where NewEcVolume's eventual
OpenFile(O_RDWR) on the same path errors out.
Add a !info.IsDir() check on both stat sites. Cheap and conservative.
Suggested in PR #9245 review by @gemini-code-assist.
* refactor(volume_server): collapse EC placement helper to a single pass
FindEcShardTargetLocation called FindFreeLocation up to four times. Each
call iterates s.Locations and acquires VolumesLen / EcShardCount RLocks
per disk — for a typical 4-disk node that's 32 RLock cycles per
placement decision.
Walk s.Locations once, score each disk by tier (mounted > .ecx-on-disk
> HDD > any-disk), break ties by free count. The free-slot math is
factored into a small helper that mirrors FindFreeLocation's formula
without re-entering the location's locks. Behaviour is unchanged: each
existing tier still wins over later tiers, and within a tier the disk
with the most free count still wins, matching the original max-tracking
in FindFreeLocation.
Suggested in PR #9245 review by @gemini-code-assist.
* refactor(volume_server): thread dataShardCount as a parameter through EC placement
ecFreeShardCount and FindEcShardTargetLocation referenced
erasure_coding.DataShardsCount directly. Take it as a parameter so
custom-ratio builds (e.g. enterprise) can swap the default without
touching the helper itself, and so unit tests can pin a specific ratio
independent of the package constant. Default callsites in
VolumeEcShardsCopy and ReceiveFile now pass the package default
explicitly; tests pass a literal 10 for clarity.
* fix(volume_server): treat MaxVolumeCount=0 as unlimited in EC placement
ecFreeShardCount computed `MaxVolumeCount - VolumesLen()` and went
negative when MaxVolumeCount was 0 — the "unlimited disk" sentinel
already honoured by Store.hasFreeDiskLocation and friends. With a
negative free count, FindEcShardTargetLocation's `freeCount <= 0`
guard skipped the disk entirely, so unlimited disks could never receive
EC shards via the placement helper.
Special-case MaxVolumeCount<=0: report a synthetic large free count
that decrements with current usage, so unlimited disks are eligible
and tie-breaks still prefer the less-loaded one. Added
TestFindEcShardTargetLocation_HonoursUnlimitedDisk as the regression.
Reported in PR #9245 review by @gemini-code-assist.
* fix(volume_server): account in shard slots, not volume slots, in ecFreeShardCount
FindFreeLocation in store.go ends with `free /= DataShardsCount`,
converting "shard slots free" back to "volume-equivalent slots." The
truncation is harmless there, but my new ecFreeShardCount inherited
the same final divide and re-introduced exactly the orphan-shard
hazard #9245 was meant to prevent: with MaxVolumeCount=1,
VolumesLen=0, EcShardCount=1 the formula reports 0 even though the
disk has room for 9 more shards, so subsequent shards route off the
.ecx-owning disk into the HDD-fallback tier.
Drop the trailing divide and return the count directly in shard slots.
Same shape, finer granularity; tie-breaks still order by free count.
The unlimited branch's "used" calculation is updated to match (mix
volume-slots and shard-slots in shard units). Added
TestFindEcShardTargetLocation_TightProvisioningKeepsEcxDisk as the
regression.
Reported in PR #9245 review by @coderabbitai.
* test(volume_server): reproduce #9184 ReceiveFile truncating a mounted shard
ReceiveFile for an EC shard calls os.Create(filePath) which opens the
path with O_TRUNC. When the shard is already mounted, the in-memory
EcVolume holds a file descriptor against the same inode, so a second
ReceiveFile call for the same (volume, shard) truncates the live shard
file beneath the reader.
Reproducer: generate and mount shard 0 for a populated volume, capture
the on-disk size, then send a smaller payload for the same shard via
ReceiveFile. The current handler accepts the overwrite and leaves the
shard truncated in place; this test pins that behavior. When the fix
lands the server should reject (or rename-then-swap) and this test
must be inverted.
* fix(volume_server): refuse ReceiveFile overwrite of mounted EC shard
ReceiveFile used os.Create on EC shard paths, which opens with
O_TRUNC and truncates in place. When an EC shard is already
mounted, the in-memory EcVolume holds file descriptors against the
same inodes, so the truncation corrupts the live shard beneath any
ongoing read. On retries of an EC task this produced the "missing
parts" class of errors in #9184.
The fix rejects any ReceiveFile for an EC volume that currently
has mounted shards. The caller must unmount before retrying —
silent truncation is never an acceptable outcome. Non-EC writes and
ReceiveFile for volumes that have never been mounted on this server
continue to work as before.
Tests:
- TestReceiveFileRejectsOverwriteOfMountedEcShard: mounts a shard,
attempts an overwrite, asserts the error response and that the
on-disk file and live reads are undisturbed.
- TestReceiveFileAllowsEcShardWhenNoMount: pins the common-case
contract that a first write to a target still succeeds.
* fix(volume-rust): refuse ReceiveFile overwrite of mounted EC shard
Mirror the Go-side change: reject receive_file for any EC volume that
currently has mounted shards on this server. std::fs::File::create
truncates in place and the in-memory EcVolume holds fds on the same
inodes, so an overwrite would corrupt live readers.
* test(volume_server): reproduce #9184 EC ReceiveFile disk-placement bug
The plugin-worker EC task sends shards via ReceiveFile, which picks
Locations[0] as the target directory regardless of the admin planner's
TargetDisk assignment. ReceiveFileInfo has no disk_id field, so there
is no wire channel to honor the plan.
Adds StartSingleVolumeClusterWithDataDirs to the integration framework
so tests can launch a volume server with N data directories. The new
repro asserts the current (buggy) behavior: sending three distinct EC
shards via ReceiveFile leaves all three files in dir[0] and the other
dirs empty. When the fix adds disk_id to ReceiveFileInfo, this
assertion must flip to verify the planned placement is respected.
* fix(ec): honor disk_id in ReceiveFile so EC shards respect admin placement
Before this change, VolumeServer.ReceiveFile for EC shards always
selected the first HDD location (Locations[0]). The plugin-worker EC
task had no way to pass the admin planner's per-shard disk
assignment — ReceiveFileInfo carried no disk_id field — so every
received EC shard piled onto a single disk per destination server.
On multi-disk servers this caused uneven load (one disk absorbing all
EC shard I/O), frequent ENOSPC retries, and a growing EC backlog
under sustained ingest (see issue #9184).
Changes:
- proto: add disk_id to ReceiveFileInfo, mirroring
VolumeEcShardsCopyRequest.disk_id.
- worker: DistributeEcShards tracks the planner-assigned disk per
shard; sendShardFileToDestination forwards that disk id. Metadata
files (ecx/ecj/vif) inherit the disk of the first data shard
targeting the same node so they land next to the shards.
- server: ReceiveFile honors disk_id when > 0 with bounds
validation; disk_id=0 (unset) falls back to the same
auto-selection pattern as VolumeEcShardsCopy (prefer disk that
already has shards for this volume, then any HDD with free space,
then any location with free space).
Tests updated:
- TestReceiveFileEcShardHonorsDiskID asserts three shards sent with
disk_id={1,2,0} land on data dirs 1, 2, and 0 respectively.
- TestReceiveFileEcShardRejectsInvalidDiskID pins the out-of-range
disk_id rejection path.
* fix(volume-rust): honor disk_id in ReceiveFile for EC shards
Mirror the Go-side change: when disk_id > 0 place the EC shard on the
requested disk; when unset, auto-select with the same preference order
as volume_ec_shards_copy (disk already holding shards, then any HDD,
then any disk).
* fix(volume): compare disk_id as uint32 to avoid 32-bit overflow
On 32-bit Go builds `int(fileInfo.DiskId) >= len(Locations)` can wrap a
high-bit uint32 to a negative int, bypassing the bounds check before the
index operation. Compare in the uint32 domain instead.
* test(ec): fail invalid-disk_id test on transport error
Previously a transport-level error from CloseAndRecv silently passed the
test by returning early, masking any real gRPC failure. Fail loudly so
only the structured ReceiveFileResponse rejection path counts as a pass.
* docs(test): explain why DiskId=0 auto-selects dir 0 in EC placement test
Documents the load-bearing assumption that shards are never mounted in
this test, so loc.FindEcVolume always returns false and auto-select
falls through to the first HDD. Saves future readers from re-deriving
the expected directory for the DiskId=0 case.
* fix(test): preserve baseDir/volume path for single-dir clusters
StartSingleVolumeClusterWithDataDirs started naming the data directory
volume0 even in the dataDirCount=1 case, which broke Scrub tests that
reach into baseDir/volume via CorruptDatFile / CorruptEcShardFile /
CorruptEcxFile. Keep the legacy name for single-dir clusters; only use
the indexed "volumeN" layout when multiple disks are requested.
* Boostrap persistent state for volume servers.
This PR implements logic load/save persistent state information for storages
associated with volume servers, and reporting state changes back to masters
via heartbeat messages.
More work ensues!
See https://github.com/seaweedfs/seaweedfs/issues/7977 for details.
* Block RPC operations writing to volume servers when maintenance mode is on.
* fix: correctly detect missing source file during volume copy
The previous fix (commit 5c27522) incorrectly used progressedBytes == 0 to
detect if the source file didn't exist. This was wrong because it would also
delete files when the source file exists but is empty.
This fix:
1. Server side: Send ModifiedTsNs even for empty files, so the client knows
the source file exists
2. Client side: Check modifiedTsNs == 0 instead of progressedBytes == 0 to
determine if source file didn't exist
Now the logic correctly handles:
- Source file doesn't exist → No ModifiedTsNs sent → Remove empty file
- Source file exists but is empty → ModifiedTsNs sent → Keep empty file
- Source file exists with content → ModifiedTsNs sent → Keep file with content
Fixes#7777
* Update weed/server/volume_grpc_copy.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
When copying/evacuating empty volumes, the .idx file may not exist on disk
(this is allowed by checkIdxFile for volumes with only super block in .dat).
This fix:
1. Uses os.IsNotExist() instead of err == os.ErrNotExist for proper
wrapped error checking in CopyFile
2. Treats missing source file as success when StopOffset == 0 (empty file)
3. Allows checkCopyFiles to pass when idx file doesn't exist but
IdxFileSize == 0 (empty volume)
Fixes volumeServer.evacuate and volume.fix.replication for empty volumes.
* fix: sync EC volume files before copying to fix deleted needles not being marked when decoding (#7751)
When a file is deleted from an EC volume, the deletion is written to both
the .ecx and .ecj files. However, these writes were not synced to disk
before the files were copied during ec.decode. This caused the copied
files to miss the deletion markers, resulting in 'leaked' space where
deleted files were not properly tracked after decoding.
This fix:
1. Adds a Sync() method to EcVolume that flushes .ecx and .ecj files
to disk without closing them
2. Calls Sync() in CopyFile before copying EC volume files to ensure
all deletions are visible to the copy operation
Fixes#7751
* test: add integration tests for EC volume deletion sync (issue #7751)
Add comprehensive tests to verify that deleted needles are properly
visible after EcVolume.Sync() is called. These tests cover:
1. TestWriteIdxFileFromEcIndex_PreservesDeletedNeedles
- Verifies that WriteIdxFileFromEcIndex preserves deletion markers
from .ecx files when generating .idx files
2. TestWriteIdxFileFromEcIndex_ProcessesEcjJournal
- Verifies that deletions from .ecj journal file are correctly
appended to the generated .idx file
3. TestEcxFileDeletionVisibleAfterSync
- Verifies that MarkNeedleDeleted changes are visible after Sync()
4. TestEcxFileDeletionWithSeparateHandles
- Tests that synced changes are visible across separate file handles
5. TestEcVolumeSyncEnsuresDeletionsVisible
- Integration test for the full EcVolume.DeleteNeedleFromEcx +
Sync() workflow that validates the fix for issue #7751
* refactor: log sync errors in EcVolume.Sync() instead of ignoring them
Per code review feedback: sync errors could reintroduce the bug this PR
fixes, so logging warnings helps with debugging.
* fix: prevent empty .vif files from ec.decode causing parse errors
When ec.decode copies .vif files from EC shard nodes, if a source node
doesn't have the .vif file, an empty .vif file was created on the target
node. This caused volume.configure.replication to fail with 'proto: syntax
error' when trying to parse the empty file.
This fix:
1. In writeToFile: Remove empty files when no data was written (source
file was not found) to avoid leaving corrupted empty files
2. In MaybeLoadVolumeInfo: Handle empty .vif files gracefully by treating
them as non-existent, allowing the system to create a proper one
Fixes#7666
* refactor: remove redundant dst.Close() and add error logging
Address review feedback:
- Remove redundant dst.Close() call since defer already handles it
- Add error logging for os.Remove() failure
Fixes#7619
The writeToFile function had two critical bugs that could cause data loss
during EC shard evacuation when the destination disk is full:
Bug 1: When os.OpenFile fails (e.g., disk full), the error was silently
ignored and nil was returned. This caused the caller to think the copy
succeeded.
Bug 2: When dst.Write fails (e.g., 'no space left on device'), the error
was completely ignored because the return value was not checked.
When evacuating EC shards to a full volume server (especially on BTRFS):
1. OpenFile may succeed (creates 0-byte file inode)
2. Write fails with 'no space left on device'
3. Errors were ignored, function returned nil
4. Caller thinks copy succeeded and deletes source shard
5. Result: 0-byte shard on destination, data loss!
This fix ensures both errors are properly returned, preventing data loss.
Added unit tests to verify the fix.
* initial design
* added simulation as tests
* reorganized the codebase to move the simulation framework and tests into their own dedicated package
* integration test. ec worker task
* remove "enhanced" reference
* start master, volume servers, filer
Current Status
✅ Master: Healthy and running (port 9333)
✅ Filer: Healthy and running (port 8888)
✅ Volume Servers: All 6 servers running (ports 8080-8085)
🔄 Admin/Workers: Will start when dependencies are ready
* generate write load
* tasks are assigned
* admin start wtih grpc port. worker has its own working directory
* Update .gitignore
* working worker and admin. Task detection is not working yet.
* compiles, detection uses volumeSizeLimitMB from master
* compiles
* worker retries connecting to admin
* build and restart
* rendering pending tasks
* skip task ID column
* sticky worker id
* test canScheduleTaskNow
* worker reconnect to admin
* clean up logs
* worker register itself first
* worker can run ec work and report status
but:
1. one volume should not be repeatedly worked on.
2. ec shards needs to be distributed and source data should be deleted.
* move ec task logic
* listing ec shards
* local copy, ec. Need to distribute.
* ec is mostly working now
* distribution of ec shards needs improvement
* need configuration to enable ec
* show ec volumes
* interval field UI component
* rename
* integration test with vauuming
* garbage percentage threshold
* fix warning
* display ec shard sizes
* fix ec volumes list
* Update ui.go
* show default values
* ensure correct default value
* MaintenanceConfig use ConfigField
* use schema defined defaults
* config
* reduce duplication
* refactor to use BaseUIProvider
* each task register its schema
* checkECEncodingCandidate use ecDetector
* use vacuumDetector
* use volumeSizeLimitMB
* remove
remove
* remove unused
* refactor
* use new framework
* remove v2 reference
* refactor
* left menu can scroll now
* The maintenance manager was not being initialized when no data directory was configured for persistent storage.
* saving config
* Update task_config_schema_templ.go
* enable/disable tasks
* protobuf encoded task configurations
* fix system settings
* use ui component
* remove logs
* interface{} Reduction
* reduce interface{}
* reduce interface{}
* avoid from/to map
* reduce interface{}
* refactor
* keep it DRY
* added logging
* debug messages
* debug level
* debug
* show the log caller line
* use configured task policy
* log level
* handle admin heartbeat response
* Update worker.go
* fix EC rack and dc count
* Report task status to admin server
* fix task logging, simplify interface checking, use erasure_coding constants
* factor in empty volume server during task planning
* volume.list adds disk id
* track disk id also
* fix locking scheduled and manual scanning
* add active topology
* simplify task detector
* ec task completed, but shards are not showing up
* implement ec in ec_typed.go
* adjust log level
* dedup
* implementing ec copying shards and only ecx files
* use disk id when distributing ec shards
🎯 Planning: ActiveTopology creates DestinationPlan with specific TargetDisk
📦 Task Creation: maintenance_integration.go creates ECDestination with DiskId
🚀 Task Execution: EC task passes DiskId in VolumeEcShardsCopyRequest
💾 Volume Server: Receives disk_id and stores shards on specific disk (vs.store.Locations[req.DiskId])
📂 File System: EC shards and metadata land in the exact disk directory planned
* Delete original volume from all locations
* clean up existing shard locations
* local encoding and distributing
* Update docker/admin_integration/EC-TESTING-README.md
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* check volume id range
* simplify
* fix tests
* fix types
* clean up logs and tests
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Added context for the MasterClient's methods to avoid endless loops
* Returned WithClient function. Added WithClientCustomGetMaster function
* Hid unused ctx arguments
* Using a common context for the KeepConnectedToMaster and WaitUntilConnected functions
* Changed the context termination check in the tryConnectToMaster function
* Added a child context to the tryConnectToMaster function
* Added a common context for KeepConnectedToMaster and WaitUntilConnected functions in benchmark
* remove old raft servers if they don't answer to pings for too long
add ping durations as options
rename ping fields
fix some todos
get masters through masterclient
raft remove server from leader
use raft servers to ping them
CheckMastersAlive for hashicorp raft only
* prepare blocking ping
* pass waitForReady as param
* pass waitForReady through all functions
* waitForReady works
* refactor
* remove unneeded params
* rollback unneeded changes
* fix
* feat(weed.move): add a speed limit parameter of moving files
* fix(weed.move): set the default value of ioBytePerSecond to vs.compactionBytePerSecond
Co-authored-by: zhihao.qu <zhihao.qu@ly.com>
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Chris Lu