mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-19 20:44:28 +02:00
Chris Lu
51a4c092c8
* volume: give Volume interior locks mirroring Go's Volume's mutable state moves behind interior locks matching Go's per-volume lock domains, so the store can hand out shared handles instead of requiring exclusive &mut access: - io: RwLock<VolumeIo> owns the .dat/remote handles, needle map, append cursor, compaction bookkeeping and write watermarks — Go's dataFileAccessLock-adjacent mutable set; helpers take the guard as &mut VolumeIo so nothing re-locks. - super_block and volume_info each get a RwLock (superBlockAccessLock, volumeInfoRWLock); dir_idx gets its own so path building never contends with I/O. - no_write packs Go's noWriteLock pair into AtomicU8; last_modified_ts_seconds and compaction_byte_per_second become atomics. The data-file write lease (dataFileAccessLock) now covers every mutating entry that previously relied on borrow exclusivity: writes, deletes, blob writes, set_writable's flag/index swap, remote-file updates, local-backend swaps, index relocation, replica-placement rewrite, put_needle_index, close and destroy. set_read_only_persist leases only its can_delete idx-attach branch because mark_io_unavailable reaches it under a held lease. commit_compact takes try_write_lock — callers hold the store lock, so blocking would stall every volume behind one slow reader; WouldBlock surfaces as COMMIT_WAITS_FOR_READS for the caller to retry. Lock order is fixed: lease -> io -> volume_info/super_block/dir_idx. * storage: mount volumes as shared Arc<Volume> handles DiskLocation's map owns Arc<Volume> so the store can hand a caller a handle that stays valid across store-guard releases, instead of borrowing &mut Volume under the store write lock. - volumes: HashMap<VolumeId, Arc<Volume>>; find_volume returns a cloned Arc. - find_volume_mut and volumes_mut are gone — every Volume method is &self now, so shared handles cover what exclusive borrows did. - writable_volume_mut -> writable_volume(&self) -> Result<Arc<Volume>> keeps the same admission checks (exists, disk-space-low). - write_volume_needle/write_volume_needles/delete_volume_needle are &self; lifecycle methods that mutate the map or mounted state stay &mut self. * server: run needle I/O with the store guard released The store write lock previously wrapped needle writes and deletes end-to-end: process_batch held it across every grouped fsync, and the HTTP/gRPC fallbacks, tail receiver and fetch-and-write paths did the same. One volume's slow I/O then stalled admission and metadata on every other volume. - write_queue resolves each group's handle via writable_volume under a brief store read, then runs write_needles_grouped after the guard drops. - HTTP write/DELETE fallbacks, the gRPC batch-delete EC recheck + delete, tail receiver and fetch_and_write local writes, blob writes and the quarantine/sync/set_writable/remote-file paths all run under store.read() now that every volume method they reach is &self. - Lifecycle calls that mutate the volume map or mounted state (mount/unmount/delete, EC shard ops, compaction commits, load_new_volumes, heartbeat volume actions) keep store.write(). * test: cross-volume admission proceeds during another volume's write Regression coverage for the store-lock fix: - write_queue's test parks a volume-1 write inside its .dat append via a new needle_write_hook seam (mirroring needle_read_hook), then proves the store still hands out a lifecycle writer and a volume-2 handle — while volume 1's data-file lease still excludes same-volume writers. - tests/store_lock_repro.rs exercises the public API: concurrent read-guarded writes on two volumes, and a stale Arc<Volume> handle failing closed after unmount instead of writing into torn-down files. * test: drop store write guards that shared handles made read-only Test setup sites held store.write() purely to reach find_volume_mut; with Arc<Volume> and &self methods the guard is read-only (clippy readonly_write_lock). The volume lease still serializes the writes inside. * volume: drop a windows-only read_exact_at that shadows the io helper cargo's windows build failed E0255: the free fn copied with the interior-lock conversion collided with the unconditional crate::storage::io::read_exact_at import, which already carries the platform branches internally. * server: release the store read guard before needle I/O The read guard still ran through write_needle_blob_and_index, the tail receiver write, the fetch-and-write local write, and the HTTP write and DELETE fallbacks, so a slow append blocked mount/unmount writers (and, on writer-preferring locks, readers on other volumes). Resolve the Arc<Volume> under a short guard and write through the handle after it drops — the same scoping process_batch already used. * volume: serialize the read-only mark and its .vif save under the lease set_read_only_persist only leased the idx-attach branch, so concurrent mark-readonly and mark-writable calls could interleave flag updates and .vif writes — including two save_vif calls truncating and writing the file at once. The lease now spans the whole flag change and save, with a _held variant for mark_io_unavailable, which reaches this from write paths already holding the lease (Go: markVolumeReadonly runs under dataFileAccessLock). * volume,server: span the tier-down transition under one write lease update_remote_files cleared the remote-only write restriction and released its lease before open_local_dat_backend attached the local file, so a write could slip into the gap — pass check_writable, then fail appending to a missing dat_file. The handler now holds the lease across the reference trim, .vif save, and backend swap (and the rollback re-insert), via _held variants; the store read guard is scoped to the handle lookup. The swap still lands after the .vif trim — the ordering that keeps a crash from pointing the volume at a deleted remote object. * server: run a batch's volume groups on separate scoped threads A single batch can hold writes for several volumes; running the groups sequentially made every later volume wait behind a stalled append on an earlier one — the same head-of-line blocking the store guard release fixed, one level up. Each group now gets a scoped thread, preserving order within a group while independent volumes progress in parallel. * storage: close a volume handle a mount slot replaces With shared handles, inserting a new Volume over an existing map entry left the old handle alive in callers — two Volume instances could then write the same files with separate in-memory indexes. set_volume and create_volume now close() the replaced handle so its writers fail closed, matching what unload_volume already does. * test: key the write hook off a reserved needle id The hook registry is process-global and keyed by needle id, and other parallel tests write NeedleId(1) — an unrelated write could enter the barriers and leave the test's real writer parked forever. * server: wait out readers for vacuum commit without the store guard commit_compact try-locked the write lease because callers held the store write lock, returning COMMIT_WAITS_FOR_READS while a download was in flight — and the master reads that as a failed vacuum, discarding completed compaction work. The commit RPC now resolves the handle under a brief store read and blocks on the volume's lease, matching Go's CommitCompact under dataFileAccessLock: a slow reader stalls only this volume's vacuum. The try-lock path stays for callers that still hold the store lock. * server: scope the remaining store read guards to volume lookup Same class as the write-path fix: the batch-delete cookie check and tombstone, the query read, volume_needle_status, and the local chunk read all kept the read guard through the needle operation, so slow I/O there could still stall store writers. Each now resolves the Arc<Volume> under a short guard; the EC rechecks keep their guards. * seaweed-volume: fail closed on detached volume handles A handle resolved under a brief store read can outlive its mount: an unmount+remount between lookup and lease leaves the Arc pointing at a closed Volume while a new instance owns the files. Mark the handle in close() and reject it under the write lease in commit_compact_held, delete_needle, check_writable, set_read_only_persist and update_remote_files_held; the tier-down handler also checks the flag before deleting a remote object its replacement still reads. * seaweed-volume: drop the store guard before mark-readonly waits on the lease mark_volume_readonly held a store read across set_read_only_persist, which waits out the volume's leased readers; a slow download would stall every lifecycle operation behind the guard. Resolve the handle under a short read, then mark off the guard; a detached handle now maps to not_found. * seaweed-volume: retry batch deletes that raced an EC mount The store guard now drops before delete_needle, so ec.encode can mount shards between the pre-delete recheck and the tombstone; the encode then removes the .dat and the needle resurrects from pre-tombstone shards. Re-check has_ec_volume after each successful delete and return the retriable 503 so the filer requeues onto the EC path. * seaweed-volume: one write-queue worker per volume A stalled batch kept later batches for other volumes queued behind it: the single worker awaited process_batch before receiving again. Go's volume_write.go keeps one batching worker per volume, so give each volume its own FIFO worker spawned lazily, bounded by a global concurrent grouped-I/O semaphore and a global queued-request bound; workers exit only when their channel closes, and submit refuses unknown volume ids before spawning. * seaweed-volume: move the queued-write permit inside the request submit held the global queued-slot permit across the send and the response wait, so a canceled submit freed the slot while the worker still held the needle; repeated cancels could overflow the bound. The request now owns the permit until the worker drops it. * seaweed-volume: retire write workers whose volume left the store worker_for kept a sender per volume forever, so servers that mount and retire volumes accumulated idle workers. A worker now polls while idle and retires once its volume leaves the store: close the channel so racing senders fail fast, drain accepted writes, then release the map slot — a replacement worker can never start while this one still holds queued writes, so same-volume FIFO survives retirement.