5 Commits

Include renames

Author	SHA1	Message	Date
4 people	9c427b265f	volume tier: accept deletes on a volume as soon as it is tiered (#11894) ... * volume tier: accept deletes on a volume as soon as it is tiered volume.tier.upload marks the volume read-only without deletes before the upload (markVolumeReplicasWritable(..., false, false), i.e. noWriteOrDelete), and nothing clears that until the volume server restarts, when load() gives every tiered volume noWriteCanDelete. Every delete in between is refused with "volume N is read only": the needle stays live in the index, and its bytes in the remote object, until someone notices. After the remote file is loaded, the upload handler now gives the volume the flags a restart would (MarkTieredDeletable: writes refused, deletes accepted), with two exceptions that keep the current behaviour: - a replicated volume: the shell copies the uploaded replica's .idx and .vif to the other replicas after the upload, and a tombstone written before that copy would reach one replica only; - a volume in I/O quarantine: that flag is not the upload's to lift. Tests: TestTierUploadAcceptsDeletesOnASingleCopyVolume fails without the change ("volume 1 is read only") and checks writes stay refused; TestTierUploadKeepsRefusingDeletesOnAReplicatedVolume; TestMarkTieredDeletable (quarantine kept). * volume tier: check the I/O quarantine under noWriteLock MarkTieredDeletable read the quarantine state before taking noWriteLock. CollectHeartbeat could quarantine the volume and set noWriteOrDelete in between, and the upload then cleared that refusal, so a quarantined volume accepted deletes until the next heartbeat. Take noWriteLock first: the heartbeat marks the quarantine before it takes noWriteLock, so either the check sees it or the heartbeat's refusal lands afterwards and stays. Also leave alone a volume past IoErrorTolerance that the next heartbeat will quarantine, as the heartbeat itself does. * volume: read noWriteOrDelete under noWriteLock in DeleteVolumeNeedle DeleteVolumeNeedle checked v.noWriteOrDelete without the lock that MarkVolumeReadonly, CollectHeartbeat and now MarkTieredDeletable hold when they write it, so a delete racing a tier upload was a data race. Copy the flag under the read lock; the delete path is otherwise unchanged. * volume tier: mirror the delete acceptance rules in the Rust volume server The Rust tier upload already promoted a freshly tiered volume to no-write-can-delete via refresh_remote_write_mode, and did so for every volume -- including the replicated and I/O-quarantined ones the Go MarkTieredDeletable deliberately refuses. A live tier-up now goes through mark_tiered_deletable (replicated volumes and quarantines keep their upload-time mark until the next load), while load() keeps promoting every tiered volume unconditionally, as Go does. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: keep a tiered volume delete-only when marked writable MarkVolumeWritable preserved noWriteCanDelete on a remote-tiered volume only when it was already set. A tiered volume still holding the upload-time mark (a replicated volume, or one whose tier upload ran before MarkTieredDeletable existed) had it clear, so marking writable published the volume as fully writable while its .dat lives remotely -- writes then failed against a backend that cannot append, after the master was told the volume is writable again. load() gives every remote-tiered volume noWriteCanDelete; deriving the same flag here keeps mark-writable consistent with a restart: "writable" on a remote volume resolves to can-delete. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: keep a tiered volume delete-only when marked writable (rust) set_writable preserved the remote delete-only mode only when the flag was already set, so a tiered volume holding the upload-time read-only mark (replicated, or quarantined under mark_tiered_deletable) was published fully writable with its .dat remote. Derive no_write_can_delete from has_remote_file() instead, matching load() and the Go MarkVolumeWritable. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume tier: serialize the quarantine check with the flag swap (rust) A heartbeat refusal landing between should_quarantine() and the store could be clobbered until the next heartbeat re-asserted it — the same race the Go side closed by checking under noWriteLock. A compare-exchange gives the same ordering: the swap fails, and the retry sees the quarantine mark the heartbeat sets first. * volume tier: MarkTieredDeletable refuses a non-tiered volume A stray call on a local single-copy volume would pin it read-only-can-delete. The only caller promotes right after LoadRemoteFile, but the guard makes the method safe to reuse; the Rust mark_tiered_deletable gets the same check, though its caller already only reaches it under has_remote. --------- Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com> Co-authored-by: chrislusf <chris.lu@gmail.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-21 14:01:10 -07:00
Chris Lu	75ec5ec193	admin: allow setting volume read-only and read/write modes (#11217) ... * admin: support setting volume read-only and read/write modes * admin: address PR review on volume access-mode persistence Reject trailing JSON values in the SetVolumeReadOnly handler so requests like {"read_only":true}{} no longer pass validation, and add a trailing-value case to the invalid-request test. Propagate .vif persistence failures through the access-mode chain. PersistReadOnly now returns the SaveVolumeInfo error and rolls back the in-memory volumeInfo on failure; Store.MarkVolumeReadonly and Store.MarkVolumeWritable propagate that error and roll back their noWrite flags, so the API reports failure instead of success while restart would revert the mode. * admin: make .vif persistence atomic and preserve error chain SaveVolumeInfo now writes to a .vif.tmp file, syncs it, renames it over the target, and fsyncs the directory. A write/sync/close failure leaves the existing .vif intact, so the PersistReadOnly in-memory rollback matches the durable state instead of diverging from a partially written file that restart would apply. Switch the error wrappers in PersistReadOnly, MarkVolumeReadonly, and MarkVolumeWritable from %v to %w so callers can use errors.Is and errors.As to classify persistence failures. * admin: treat post-rename dir fsync failure as a warning After os.Rename commits the new .vif, the on-disk file already holds the requested mode. A directory fsync failure only risks losing the rename across a crash; returning an error here would make PersistReadOnly roll back in-memory state while the durable file keeps the new mode, splitting the replica. Log the failure as a warning instead, matching the best-effort nature of FsyncDir (already skipped on Windows). * admin: distinguish post-rename durability failures and use unique temp files SaveVolumeInfo now uses os.CreateTemp for the staging file, preventing concurrent saves for the same volume from colliding on a shared .tmp path. A directory fsync failure after os.Rename returns a NotCrashDurableError instead of being silently swallowed. The rename already committed the new metadata to disk, so PersistReadOnly, MarkVolumeReadonly, and MarkVolumeWritable skip the in-memory rollback for this error type (keeping state aligned with the durable file) while still propagating the failure to the API. Pre-commit failures continue to roll back as before. * admin: continue post-commit work after NotCrashDurableError MarkVolumeWritable now clears the EIO quarantine and the gRPC handlers (makeVolumeReadonly step 3, makeVolumeWritable master notification) proceed with their post-commit work when SaveVolumeInfo returns a NotCrashDurableError, instead of aborting and leaving the volume unavailable or the master unaware of the mode change. The durability warning is still propagated to the API caller. Pre-commit failures continue to abort early as before. * admin: handle NotCrashDurableError in tier and EC callers VolumeTierMoveDatFromRemote and VolumeEcShardsGenerate now check for NotCrashDurableError from SaveVolumeInfo. When the rename has already committed the new .vif, they continue with their post-commit work (backend switch, remote deletion, keeping generated EC shards) instead of aborting and leaving the on-disk metadata inconsistent with the file layout. The durability warning is logged for the operator.	2026-09-07 18:40:37 -07:00
3 people	9575032b4c	volume: forward fsync=true to replicas in ReplicatedWrite (#10805) ... * volume: forward fsync=true to replicas in ReplicatedWrite When a write request carries fsync=true, only the primary volume server flushed to disk: the replica fan-out URL in ReplicatedWrite only carried type/ttl/ts/cm, so replicas always wrote without fsync even when the client explicitly requested a durable write. Forward the fsync request parameter to the replica volume servers so a durable write means every replica has flushed to disk, not just the primary. Replicas without fsync are untouched (zero behavior change). * storage: flush a durable write inline while stopping The fsync flag on the write path really selects the async batch worker, and it was switched off once the store is stopping. So a fsync=true write landing during the pre-stop drain got acked without ever being flushed - and now that ReplicatedWrite forwards fsync, that covers replicas too. Flush it inline instead of queueing it. The drain keeps accepting writes, which is the whole point of preStopSeconds, and the ack still means the .dat is on disk. If the fsync fails, the append comes back off the .dat and the needle map goes back to what it pointed at before, so nothing resolves to an offset past the truncated end. * storage: make the store's stopping flag atomic SetStopping runs on the signal handler goroutine while the write and vacuum paths read the flag, so every read of it was racy. Nothing about the shutdown ordering changes; only the flag itself is now safe to read. * topology: check the errors the replication test was dropping The mock replica ignored its response write and the mock master ignored whatever Serve returned, so a broken mock would have shown up as a confusing timeout rather than a failure. Also drops the explicit listener close: grpc.Server.Stop already closes the listener it was given. --------- Co-authored-by: hzsunchao <hzsunchao@corp.netease.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-08-18 17:17:50 -07:00
Chris Lu	c6e1387f59	shell: multi-target fs.mergeVolumes and volume.mark -readonlyCanDelete (#10706) ... * shell: fs.mergeVolumes distributes one volume across multiple -toVolumeId targets * volume: volume.mark -readonlyCanDelete rejects writes but keeps accepting deletes * seaweed-volume: mirror readonlyCanDelete volume state	2026-08-10 16:31:26 -07:00
Chris Lu	7c252e1f16	fix(volume): reopen .idx writable after MarkVolumeWritable (fixes #9515) (#9526) ... * fix(volume): reopen .idx writable after MarkVolumeWritable When .vif has ReadOnly=true, load() opens .idx as O_RDONLY and builds a SortedFileNeedleMap whose Put returns os.ErrInvalid. MarkVolumeWritable only flipped noWriteOrDelete back to false and rewrote .vif, so writes still failed at v.nm.Put. Reopen .idx in O_RDWR and rebuild v.nm in its writable form (in-memory or leveldb small/medium/large) before flipping the flag. Mirror the same fix in seaweed-volume: the Rust load path leaves CompactNeedleMap/RedbNeedleMap with no idx_file writer when the volume boots read-only, so post-MarkVolumeWritable puts silently succeeded in-memory only and were lost on the next restart. set_writable now reattaches an append-mode writer when one is missing. * fix(volume): keep old needle map until replacement is built; defer writable flag Go: build the writable needle map into a local before swapping. A construction failure now leaves v.nm pointing at the original SortedFileNeedleMap so MarkVolumeWritable can roll back, instead of stranding the volume with v.nm == nil. Rust: attach the .idx writer before flipping no_write_or_delete to false. A transient open/metadata failure used to leave the volume marked writable with no writer attached, and subsequent puts would silently skip the on-disk append.	2026-05-18 20:51:04 -07:00