mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-02 18:22:40 +01:00
d431c1b35a57b32b83818aab29ca7bb1120d6448
96
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris LuandClaude Opus 5.5
|
d84ae4f2ab |
volume: don't let an unreachable tier backend hold up volume server startup (#12312)
* s3 tier backend: bound connect and response-header waits The tier client used the SDK default transport: a 30s dial timeout and no limit on waiting for response headers. A blackholed endpoint cost about two minutes per request with the SDK's retries, and an endpoint that accepts connections but never answers held the request forever. Dial with a 10s timeout and give up on a response whose headers do not arrive within a minute. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: stop the disk scan from waiting on a stalled tier backend A volume server registers with the master only after every volume on its disks has loaded, and loading a tiered volume reads its superblock from the remote object. With the S3 endpoint unreachable, each read waited out the client's timeouts and retries on one of the loader workers, so start-up grew with the number of tiered volumes and the local volumes were not served until it finished. A tiered volume already loads without its superblock when the read fails. Give that read 10s, and once it stalls, let the backend's other volumes skip it for a minute instead of each waiting out the timeout. The volume reads from the backend as before once the endpoint answers again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: bound connect and read waits of the S3 tier client Mirror the Go tier client: a 10s connect timeout instead of the SDK's 3.1s default, and a minute to receive a response, so an endpoint that accepts connections but never answers cannot hold a read forever. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: stop volume loading from waiting on a stalled tier backend Mirror the Go loader: the superblock read of a tiered volume gets 10s, and once it stalls the backend's other volumes skip it for a minute. The volume loads with its .vif version as before when the read fails, and reads from the backend once the endpoint answers again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: keep the .vif replication when a remote superblock read fails Go applies the .vif replication to a tiered volume whether or not its superblock read succeeded, so a volume loaded while its backend is down still reports its replica placement. The Rust loader applied it only on success and reported the default placement instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: skip the .idx head repair while the tier backend is unreachable The repair reads needle headers from the remote .dat, so after the superblock read failed it would wait on the same backend again and hold up the load. It runs on the next load that reaches the backend. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: skip the .idx head repair while the tier backend is unreachable Mirrors the Go load: after the remote superblock read failed, the repair's remote header reads would wait on the same backend again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 tier backend: give up a ranged read whose body stops arriving The response-header timeout ends once headers arrive, so an endpoint that answers and then sends no body held ReadAt, and a superblock read's worker, forever. Cancel the request once the body makes no progress for s3ReadTimeout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: test that a tier read whose body stalls gives up The SDK's stalled-stream protection already fails it, matching the Go read's no-progress cancel. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rust): keep the S3 read timeout off upload_part The SDK's read timeout runs from the start of the request, so a 64 MiB part on a slow link would time out while its body is still being sent. Go's ResponseHeaderTimeout only starts after the body is written. upload_part now keeps the connect timeout and drops the read timeout; stalled-stream protection still fails an upload that stops moving. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rust): bound the upload_part response wait from when the part is sent Without a read timeout, an upload_part whose body was accepted but never answered waited forever. The part body now goes to the connection in 64 KiB chunks, and once the last chunk is taken the response must arrive within S3_READ_TIMEOUT, like Go's ResponseHeaderTimeout. A retry rebuilds the body and restarts that clock. The body keeps its in-memory contents, so signing and the checksum header are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rust): keep a retried part's response clock from an earlier attempt The send state was a single flag, and a retry body could be rebuilt and fully sent before the waiter saw the flag drop, so the waiter kept the earlier attempt's deadline. Each body now carries its attempt number, only the latest attempt marks the part sent, and the deadline holds only while no later attempt has started. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
6ebc055a94 |
master, volume, filer: start on a read-only mounted directory and serve existing data read-only (#12081)
* master, volume, filer: start on a read-only mounted directory and serve existing data read-only A read-only mount was treated as writable (permission bits only), so every service failed on its first write: the master panicked writing raft conf.tmp, the filer exited opening the leveldb LOCK, and the volume server skipped every volume while still offering the directory as free capacity. - util.IsReadOnlyMount: access(W_OK) == EROFS. - volume: on a read-only -dir take the existing O_RDONLY/noWriteOrDelete load, index .idx in memory when no .sdx exists, offer no free slots (same branches as isDiskSpaceLow), skip the vid stamp there. - master: on a read-only -mdir run raft from a private copy and refuse new volume ids. - filer: leveldb2/leveldb3 set their existing ReadOnly option. * volume: skip read-only mounts when picking EC shard copy targets A shard copy landing on a read-only mount fails at write time; offer only writable locations like the other placement paths do. * volume: mirror the read-only mount handling in the Rust volume server The .dat open only fell back to read-only on EACCES, so a real read-only mount (EROFS) still failed the load. The index side already degrades once no_write_or_delete is set. Add is_read_only_mount via access(W_OK) == EROFS, mark DiskLocation.read_only_mount, and mirror the Go checks: no free slots, heartbeat max = used, no vid-stamp writes, no EC shard copy targets, and a read-only index dir forces the volume read-only. * volume: write vid stamps while only the index dir is read-only * filer: skip the write-bit check when a leveldb store opens read-only * filer: serve reads of buckets with no leveldb3 dir on a read-only store * volume: derive a stable dir uuid when a read-only mount cannot store one * master: keep the read-only raft copy across restarts so votes survive * volume: index a missing .idx in memory on a read-only mount * volume: release the .idx handle after an in-memory read-only load * volume: mount EC shards read-only when their index dir is on a read-only mount * volume: mark the index dir read-only before the tail repair opens .idx * volume: write vid stamps while only the index dir is read-only (Rust) * volume: derive a stable dir uuid and report no free slots on a read-only mount (Rust) * volume: index a missing .idx in memory on a read-only mount (Rust) * volume: refuse an EC copy whose shard owner is on a read-only disk * master: key the read-only raft copy by the absolute -mdir Two masters sharing the same relative -mdir text from different working directories would have collided on one temp copy and shared raft state. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: mix device and host into the read-only dir uuid Deriving the uuid from the path alone let independent servers at the same mount point report one uuid, which RegisterUuids rejects. Hash in the device id and hostname so restarts stay stable but disks differ. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: vic6996-web <320442181+vic6996-web@users.noreply.github.com> Co-authored-by: user.email <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Javier de la TorreandJavier de la Torre
|
97f8ca13ff |
s3tables, iceberg: log client refusals at V(1), not ERROR (#11896)
* s3tables, iceberg: log client refusals at V(1), not ERROR Requests the S3 Tables and Iceberg REST APIs correctly refuse with a 4xx (access denied, not authorized, no such namespace, already exists, not empty, a commit conflict) were logged with glog.Errorf, so a client probing for what it may do, or retrying an optimistic commit, filled the ERROR log and hid real faults from anyone alerting on it. - s3tables: HandleRequest wraps the ResponseWriter in a statusRecorder (with Unwrap for http.ResponseController) and logs a handler error by the status it answered: 4xx at V(1) as "<op> refused (<status>)", 5xx or no status at ERROR as before. - s3tables: the "handleCreateNamespace called" trace moves from ERROR to V(3). - iceberg: writeManagerError's mapping is split into classifyManagerError, shared with a new logManagerError that logs 4xx-mapped errors at V(1) and everything else at ERROR; used by CreateNamespace, CommitTable and CommitTransaction. - storage: loading a remote-tier volume logged "readSuperBlock remote volume N: <nil>" at ERROR even when the read succeeded; it is now logged only when there is an error. Tests: TestRequestRefusedWith4xxIsNotLoggedAsError, TestLogRequestErrorByStatus, TestStatusRecorderKeepsTheFirstStatus, TestStatusRecorderUnwrapsForResponseController, TestNoTraceLoggedAtErrorLevel (fails on master on the CreateNamespace trace), TestLogManagerErrorRefusalsAreNotErrors, TestRemoteSuperBlockLogOnlyOnError. * s3tables: do not log a malformed CreateNamespace body at ERROR handleCreateNamespace logged a body it could not decode at ERROR before answering 400; the dispatcher already logs that refusal once at V(1). Also drop statusRecorder.Flush: it made the recorder an http.Flusher even over a writer that cannot flush, so http.NewResponseController reported success instead of ErrNotSupported. Unwrap already lets the controller reach the wrapped writer. * iceberg: require exactly one log line per classified manager error The test accepted a refusal that produced no log, or both logs. Require one V(1) refusal and no ERROR for a 4xx, and the reverse for a fault. --------- Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com> |
||
|
Chris LuandDevin
|
1cfd9bbb4c |
volume: run the durable sync outside the reader-visible lock (#11863)
* volume: run the durable sync outside the reader-visible lock Reads of a volume waited on that volume's fsync because the write lock was held across Sync()/sync_all. A new per-volume lock (Go dataFileSyncLock, Rust data_file_sync_lock) now serializes every .dat mutation and lifecycle op across append+publish+sync+rollback, while the fsync itself runs with dataFileAccessLock/the io write guard dropped — so readers proceed during a sync, and a failed sync's rollback still cannot clobber another write or a compaction swap. Same change in Go (volume_write.go syncWrite/processBatch, SyncToDisk) and Rust (write_needles_grouped/write_synced_run). Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: publish after sync; fix Close/CommitCompact lock order Address review on #11863: - Close took dataFileSyncLock before waiting on isCompactionInProgress while CommitCompact claims the flag then takes the lock — a deadlock. Close now claims the flag before taking the locks (flag -> sync -> access). - Go now publishes needle-map entries only after the .dat sync, matching Rust's ordering: appendWriteRequest/appendDeleteRequest stage the bytes, publishWriteRequest/publishDeleteRequest index them after Sync. A failed sync rolls back bytes no reader ever resolved — the streamed-read offset-invalidation window is gone. processBatch splits runs at repeated needle ids (Rust parity) so dedup/cookie checks stay sequential; the old snapshot/restore rollback machinery is removed. - Rust commit_compact (store-lock caller) takes data_file_sync_lock via try_lock -> WouldBlock before the write lease; commit_compact_held and the tier-down _held paths document/require the caller-held sync lock, and the two vol.instance().write_lock() grpc sites take it first. The swap can no longer land between a durable write's append and publish. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: restore write watermarks when an inline fsync fails The syncWrite rollback truncated the .dat but left lastAppendAtNs, lastWriteAppendAtNs, lastWriteNeedleKey, lastWriteDeleted and lastModifiedTsSeconds describing the rejected write; recovery and compaction could read a watermark for a record that no longer exists. Snapshot and restore them on failure, as processBatchRun already does. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
d04d838ebd |
ec: compare live idx NeedleId sets before encode (#11812)
* ec: compare live idx NeedleId sets before encode Refuse encode when replicas' live .idx NeedleId sets differ. Walk only live rows (skip deleted and Size<=0) and compare NeedleId, ignoring Size so MIME/flags drift does not abort a matching id set. Mark the volume read-only for encode and reject writes after that. * fix(ec): compare all replicas then pick capacity source Log matching live idx sets with "replicas are consistent" for the shell CI check. After readonly, compare every replica's live NeedleId set, then require the encode source to have FreeVolumeCount>=2. Empty live-idx vs non-empty refuses encode instead of skipping superblock stubs. Drop the FileCount rebind after the idx gate. * volume: mark readonly under the write lock so checked appends finish first * ec: fix readonly deadlock and restore worker-host source preference MarkVolumeReadonly held dataFileAccessLock while calling reopenIdxForWrite, which re-acquires the same mutex and deadlocks TestMarkVolumeReadonlyCanDelete. Split out reopenIdxForWriteLocked for callers already holding the lock, matching the existing Locked pattern. The worker task lost its worker-host source preference when the idx gate replaced the replica union. Late-bind the source to a replica on the worker host using only the checked location set, with no FileCount rebinding. Drop the now-unused helpers, trim comments added by the earlier commits, and rename tests whose names still claimed convergence while asserting refusal. --------- Co-authored-by: Chepurnov Simon <195125934+CHEPURNOV-S-S@users.noreply.github.com> Co-authored-by: Devin <devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
|
1df165d514 |
fix(volume): keep the TTL clock across a vacuum commit instead of rescanning (#11630)
* fix(volume): keep the TTL clock across a vacuum commit instead of rescanning CommitCompact reloads the swapped files while holding dataFileAccessLock, and for a vacuumed TTL volume that reload re-derived lastModifiedTsSeconds by reading every live needle's append timestamp from the .dat: two random reads per needle, with every read of the volume blocked behind them. The in-memory clock is already current at that point. Every write since the volume loaded moved it, and makeupDiff only replays writes that went through that path. Carry it across the reload instead. This also stops an over-budget scan from falling back to the new .dat's mtime and restarting an expiring volume's TTL at the commit. * volume: carry the append watermark as the TTL clock across a vacuum commit The running append watermark is the clock the reload's recovery scan recomputes, so the commit can keep it directly. Client-supplied needle modified times can run ahead of or behind the append time; keeping lastModifiedTsSeconds itself would let a forged or stale timestamp move expiry through a vacuum, where the scan it replaces used server-side append timestamps. * storage: test that a vacuum commit keeps the append clock A write's client supplied modified time can lie ahead of or behind its append time; the commit must land the TTL clock on the append watermark, the same value the recovery scan would have recomputed. * volume: carry the append watermark as the TTL clock across a vacuum commit Mirrors the Go volume server: the running append watermark is the clock the reload's recovery scan recomputes, so the commit keeps it instead of rescanning live needles under the write lock. * volume: commit carries the last-write append time, not the latest append lastAppendAtNs counts tombstone appends and is reseeded from the .dat tail at every load, so it can sit ahead of the last write -- a delete freshens the commit clock -- or behind it: a restarted vacuumed volume's tail needle is not its newest write, and the commit would move the TTL clock backward into premature expiry. Track lastWriteAppendAtNs instead, bumped only on needle appends and seeded by the recovery scan, so the commit lands the clock on the same live-write maximum the rescan would have recomputed. * volume: rescan at commit when the newest write was deleted lastWriteAppendAtNs can hold a write the index no longer holds, so carrying it extends the TTL clock past what recovery over the compacted index would compute. Remember the key behind the watermark so its tombstone or index rollback can send the reload back through recoverLastModifiedTs, landing on the newest surviving write. * volume: a tombstone retires the write rows beneath it in the last-write scan A needle deleted after the compaction copy leaves its write row followed by a tombstone in the committed index. The reverse scan skipped the tombstone row but then counted the dead write, reseeding the watermark and flag as if it were alive. Track keys whose latest row is a tombstone so their earlier write rows stop counting, and exercise the delete-inside-the-commit-window ordering in the tests. --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Joel Town RoadandChris Lu
|
23893eb378 |
volume: return error instead of panicking when .dat open fails (#11619)
* volume: return error instead of panicking when .dat open fails When backend.OpenVolumeFile returns an error (e.g. disk below -minFreeSpace), dataFile is nil. Calling backend.NewDiskFile(nil) immediately after caused a nil-pointer panic in f.Stat()/f.Name(). Move the existing error check to run right after OpenVolumeFile, before NewDiskFile is called, so the error is returned cleanly. Fixes seaweedfs/seaweedfs#11615 * volume: share .dat load error handling Extract datFileLoadError helper so open and create paths share one check. * volume: trim dat-open-fail test comments * volume(rust): cover unopenable .dat load path --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
ssshr-66andChris Lu
|
b9ad62fc16 |
[Volume] Keep DAT and index state consistent after async batch Sync failure (#11425)
* fix 11400 * persist failed-recovery quarantine and harden rollback - record the unavailable state in a .unavailable marker, fsync it, and re-arm it on load so a restart cannot serve an unverified pair - quarantine the volume so heartbeats stop advertising it - block MarkVolumeWritable while unavailable, rechecked under noWriteLock - fail every request of a failed batch, not only the succeeded ones - restore the needle map and truncate .dat on inline fsync rollback failure - add truncateIndex for the sorted-file needle map - mirror the fail-closed semantics in the Rust volume server * volume: erase rolled-back mappings instead of leaving tombstones A rolled-back batch or failed inline write used Delete() to undo a needle that did not exist beforehand, leaving a tombstoned map entry whose stale offset makes the next write to that needle fail reading a header that no longer exists. Add removeMapping/restoreMapping to the mappers so recovery erases entries that were absent before the batch and reinstates the exact prior offset/size for ones that were, including tombstones. The index row still goes through Delete so a replay forgets the needle. * volume: gate bulk readers on unavailable and fsync the marker's dir - fsync_dir(&self.dir) synced the volume dir's parent, not the dir holding .unavailable; pass the marker path so the create survives a host crash - export UnavailableError and check it in ReadAllNeedles, VolumeTailSender, VolumeIncrementalCopy, and IncrementalBackup so replica-sync paths cannot stream or append data from an unverified .dat/.idx pair; mirror on the Rust side via read_dat_slice, read_all_needles, dat_scan_plan, and the incremental-copy handler * volume: drop issue references from comments near touched code * volume: stop active scans when the volume becomes unavailable The stream entry-point checks ran once per RPC, so a volume quarantined by a failed recovery mid-scan kept serving data. Recheck availability per needle/chunk on the detached read paths: tail scan and heartbeat, read-all, incremental copy, incremental backup writes, and the Rust StreamingBody chunk reads. Rust incremental copy also rejects a quarantined volume before sync_to_disk touches the backend. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
f79d83abf4 |
volume: expire TTL volumes whose only traffic is deletes (#11167)
* volume: count a TTL volume's age from its last write, not the .dat mtime A delete appends a tombstone needle and vacuum rewrites the .dat wholesale, so the file's mtime moves without any write ever landing. The loader read lastModifiedTsSeconds back from that mtime, so every restart of a volume taking delete traffic re-armed expired() for another full TTL: an overwrite-heavy collection kept growing until it hit the max-volume cap. Recover the clock from the newest .idx entry that is not a tombstone and read that needle's append timestamp, falling back to the mtime when no write is recoverable. Only TTL volumes pay for the scan. Fixes #11160 * volume: count the .vif destroy time from the last write too ExpireAtSec is what an EC volume is reclaimed on, and it was recomputed as now+TTL every time the .vif was written. A read-only mark, a tier upload or an EC encode therefore handed an already expiring volume another full TTL, the same way the .dat mtime did. Derive it from the volume's last write, falling back to now for a volume that has not taken one yet so a fresh volume is not born expired. * volume: mirror the last-write TTL clock in the Rust volume server Same recovery as the Go loader: scan the .idx backwards for the newest entry that is not a tombstone and take that needle's append timestamp, leaving the clock on the .dat mtime when no write is recoverable. * volume: mirror the last-write destroy time in the Rust volume server Both .vif writers and the EC encode computed ExpireAtSec as now+TTL, the same way Go did, so the destroy time moved every time the sidecar was rewritten. Route all three through the volume's last write. * volume: report the .dat mtime in the Rust heartbeat, like Go does The Rust server reported its TTL clock as ModifiedAtSecond while Go reports the .dat mtime. The shell's quiet-period gates (volume.tier.move, volume.delete_empty) read that field as "last touched", which a delete has to count towards even though the TTL clock deliberately ignores it -- and with the clock now recovered from the last write, the two drift further apart. * volume: take the newest write by timestamp on a vacuumed volume The reverse .idx scan trusted position, which holds only while the .dat is append ordered. Vacuum rewrites it in key order, and since an overwrite keeps its original key, the highest-key survivor is not necessarily the newest write -- the recovered clock could land up to a TTL early and take the volume with data still inside its TTL. A volume that has been vacuumed (CompactionRevision > 0) now takes the maximum append timestamp over a bounded window of write entries instead. An append-ordered volume still answers in one read. * volume: never guess a vacuumed volume's last write, and resolve wrapped offsets Two holes in the reverse scan, both from review: A vacuumed volume's writes are ordered by key, so any of them can hold the newest timestamp. Reading a capped window sampled the highest keys, which could still miss a recently overwritten low-key needle and expire data inside its TTL. The scan now covers every write a vacuumed volume indexes, and a volume too large to scan keeps the .dat mtime rather than report a partial maximum -- late is recoverable, early is not. A .dat past MaxPossibleVolumeSize wraps the offsets in its .idx, so reading a timestamp at the unwrapped offset picks up an unrelated needle. Resolve the entry against the needle header first and retry one volume size in, the way doCheckAndFixVolumeData already does. * volume: drop GitHub issue references from TTL comments |
||
|
Chris Lu
|
31fb46f693 |
volume: rebuild a missing .idx from the .dat (#11115)
* volume: rebuild a missing .idx from the .dat Pointing -dir.idx at a directory that holds no index aborted the whole volume server: checkIdxFile found no .idx and load() called glog.Fatalf. Every row of the index is derivable from the .dat, so walk it in append order and write the index back, which reproduces byte for byte what the server's own writes had left in the old directory. Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ * volume: keep the index co-located with the data in the Rust server Go's load() drops back to the data directory when an .idx already sits beside the .dat, so naming a --dir.idx does not strand a pre-existing index. Rust had no such adjustment: it opened the new directory with create, and the volume came up on an empty index with every needle invisible. Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ * volume: rebuild a missing .idx from the .dat in the Rust server Mirrors the Go side. Rust did not abort on a missing index the way checkIdxFile did; it opened the new directory with create and mounted the volume on an empty index, so every needle read as missing while the .dat still held the data. Walk the .dat in append order and write the index back, byte for byte what the server's own writes had left behind. Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ * volume: stop the idx rebuild at a zero-padded .dat tail An all-zero needle header is unwritten space, not a record. Go's .dat walk keeps reading past it and would index a truncated data file's tail as millions of needle 0 rows; the Rust walk already stops there. Stop the Go rebuild at the same place. Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ * volume: create the -dir.idx directory when it does not exist Rust's DiskLocation creates the index directory as it takes it; Go only resolved the path, so naming a directory that does not exist yet left every volume unable to open or rebuild its index and took the server down. Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ * volume: stop the idx rebuild at a torn .dat record A crash between writing a needle's header and its body leaves a record whose declared size runs past the end of .dat. Indexing it puts a row in the .idx that points at bytes that do not exist, which fails every read of that needle and trips the past-EOF check on the next load. Stop at the first record that does not fit, in both servers. Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ * volume: stop the idx rebuild at a negative-size header A corrupt header whose size field is negative makes the .dat walk advance backwards: NeedleBodyLength adds the negative size, so the next offset is lower than the current one. The Go walk then reads at a negative offset and the rebuild fails, which puts the volume server right back to exiting at startup; the Rust walk seeks past EOF and truncates the index instead. A negative size is never a record, so stop there. Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ * volume: skip a volume whose index cannot be rebuilt, do not exit glog.Fatalf calls os.Exit(255), so a rebuild that could not write -- a full or read-only index directory -- put the server right back to dying at startup for one bad volume. Return the error instead: loadExistingVolume logs it and skips that volume, which is what the remote-volume branch just above already does and what the Rust loader has always done. Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ * volume: create the index directory from the rebuild too The rebuild is the first thing to write into a fresh -dir.idx, and it runs before the loaders that create the directory on their way to opening .idx. Create it in both rebuilds so the ordering does not matter. Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ * ci: let codespell past the sme variable in the mount tests weedfs_stream_mutate_error_test.go names its *streamMutateError local sme, which codespell reads as a misspelling of same/some. It is an identifier, so exempt it beside the other variable-name entries. Claude-Session: https://claude.ai/code/session_01BYrb2AdJSckq9FdHuqseDJ |
||
|
Chris Lu
|
1996c6aec6 |
volume: open volume files with O_NOATIME (#11055)
* volume server: open volume files with O_NOATIME Nothing reads the atime of .dat, .idx, .sdx, or EC files, but every needle read still dirtied the inode: even relatime writes atime on the first read after each write, so an actively written volume paid a metadata write per read/write cycle, and strictatime mounts paid one per read. Open the serving handles with O_NOATIME, falling back to a plain open when the file belongs to another owner (EPERM). Claude-Session: https://claude.ai/code/session_015uVY4diBgEn3VYQoc2eMuD * seaweed-volume: mirror the O_NOATIME volume file opens Same change as the Go volume server: serving handles for .dat, .idx, .sdx, .ecx, .ecj, and shard files open with O_NOATIME on Linux, with a plain-open fallback on EPERM. Claude-Session: https://claude.ai/code/session_015uVY4diBgEn3VYQoc2eMuD * route the tier-down and recreate .dat opens through the no-atime helper Review caught the Rust tier-down swap opening the local .dat directly. The Go swapToLocalDatBackend and the zero-length read-only .dat recreate in maybeWriteSuperBlock had the same gap: all three install long-lived serving handles. Claude-Session: https://claude.ai/code/session_015uVY4diBgEn3VYQoc2eMuD |
||
|
Chris Lu
|
c6e1387f59 |
shell: multi-target fs.mergeVolumes and volume.mark -readonlyCanDelete (#10706)
* shell: fs.mergeVolumes distributes one volume across multiple -toVolumeId targets * volume: volume.mark -readonlyCanDelete rejects writes but keeps accepting deletes * seaweed-volume: mirror readonlyCanDelete volume state |
||
|
Chris Lu
|
13176b4edd |
volume: recover .idx rows overwritten by tiered deletes (#10474)
* volume: recover .idx rows overwritten by tiered deletes A delete on a read-only volume backed by a remote tier used to write its tombstone row at .idx offset 0 rather than appending it, so each delete overwrote one more row at the front and lost the Put rows indexing the first needles in .dat. Those needles 404 even though .dat still holds them, and rebuilding .idx with weed fix means stopping the server and pulling the whole .dat back from the tier. The damage has a fingerprint -- .idx opening with a run of offset-0 tombstones, which a healthy .idx never does -- and .idx and .dat grow in lockstep, so the lost rows indexed exactly the first N .dat records. Detect it at load and re-derive them from a header-only walk over the head of .dat, cheap even against a remote tier, appending only the keys the .idx no longer names. * rust volume: mirror the .idx head tombstone recovery Port the Go detection and repair: an .idx opening with a run of offset-0 tombstones lost the Put rows indexing the first needles in .dat, so re-derive them at load from a header-only walk over the head of .dat and append the keys the .idx no longer names. * volume: put recovered .idx rows back in front instead of appending Appending left the offset-0 tombstone run at the head, so every later load re-walked .idx to the tail to notice the volume was already recovered, and the rows for the head of .dat sat past the .dat-tail row -- costing CheckVolumeDataIntegrity its O(1) path and breaking the ascending append order BinarySearchByAppendAtNs assumes. Rewrite .idx as the recovered rows followed by its current contents, through a temp file and a rename. .idx is back in .dat append order, so a later load stops after reading one row. * volume: keep the .idx mode when the repair replaces it The recovery renames a fresh temp file over .idx, so a fixed 0644 (Go) or whatever the umask allows (Rust) would silently widen an index an operator had locked down. Carry the mode off the file being replaced. |
||
|
Chris Lu
|
0ae1fdcad2 |
volume: reload a remote-tiered volume without re-entering the data lock (#10266)
LoadRemoteFile now takes dataFileAccessLock (so a live tier upload does not race the heartbeat's DataBackend read). But load() also runs it, and CommitCompact calls load() while already holding that lock, so reloading a remote-tiered volume during a compaction commit re-enters the non-reentrant lock and deadlocks. Split the locked bodies out: swapDataBackendLocked and loadRemoteFileLocked assume the caller holds dataFileAccessLock. load() uses loadRemoteFileLocked; the public LoadRemoteFile keeps taking the lock for the live tier-upload handler that does not hold it. |
||
|
Chris Lu
|
1e91a99f79 |
fix(volume): avoid nil-deref when needle map loader errors (#9694) (#9697)
* fix(volume): avoid nil-deref when needle map loader errors A corrupt .idx whose size is not a multiple of NeedleMapEntrySize sends the read-only load path into NewSortedFileNeedleMap, which returns (*SortedFileNeedleMap)(nil) when reverseWalkIndexFile rejects the file. The multi-value assignment `v.nm, err = NewSortedFileNeedleMap(...)` parks that typed-nil pointer in the v.nm NeedleMapper interface, so the subsequent `v.nm != nil` guard still passes — and the post-load MaxNeedleEnd structural check dispatches through the promoted mapMetric accessor on a nil receiver, segfaulting the whole volume server at load time. Reset v.nm explicitly after every loader failure so the interface is truly nil, and skip the MaxNeedleEnd check when err is non-nil since the value would come from a partial walk anyway. NewLevelDbNeedleMap has the same typed-nil-on-error shape and is fixed the same way. * fix(volume): close indexFile when needle map load errors Pre-fix the typed-nil v.nm path either leaked indexFile silently (SortedFileNeedleMap.Close had a nil-receiver early return) or crashed (LevelDbNeedleMap.Close had no such guard). With v.nm cleared to nil on error, the defer cleanup no longer calls Close at all, so the LoadCompactNeedleMap success-with-error path now also leaks indexFile. Close indexFile explicitly on each loader error to keep ownership balanced. * trim comments |
||
|
Chris Lu
|
7c252e1f16 |
fix(volume): reopen .idx writable after MarkVolumeWritable (fixes #9515) (#9526)
* fix(volume): reopen .idx writable after MarkVolumeWritable When .vif has ReadOnly=true, load() opens .idx as O_RDONLY and builds a SortedFileNeedleMap whose Put returns os.ErrInvalid. MarkVolumeWritable only flipped noWriteOrDelete back to false and rewrote .vif, so writes still failed at v.nm.Put. Reopen .idx in O_RDWR and rebuild v.nm in its writable form (in-memory or leveldb small/medium/large) before flipping the flag. Mirror the same fix in seaweed-volume: the Rust load path leaves CompactNeedleMap/RedbNeedleMap with no idx_file writer when the volume boots read-only, so post-MarkVolumeWritable puts silently succeeded in-memory only and were lost on the next restart. set_writable now reattaches an append-mode writer when one is missing. * fix(volume): keep old needle map until replacement is built; defer writable flag Go: build the writable needle map into a local before swapping. A construction failure now leaves v.nm pointing at the original SortedFileNeedleMap so MarkVolumeWritable can roll back, instead of stranding the volume with v.nm == nil. Rust: attach the .idx writer before flipping no_write_or_delete to false. A transient open/metadata failure used to leave the volume marked writable with no writer attached, and subsequent puts would silently skip the on-disk append. |
||
|
Chris Luandgemini-code-assist[bot]
|
1c0e24f06a |
fix(balance): don't move remote-tiered volumes; don't fatal on missing .idx (#9335)
* fix(volume): don't fatal on missing .idx for remote-tiered volume A .vif left behind without its .idx (orphaned by a crashed move, partial copy, or hand-edit) would trip glog.Fatalf in checkIdxFile and take the whole volume server down on boot, killing every healthy volume on it too. For remote-tiered volumes treat it as a per-volume load error so the server can come up and the operator can clean up the stray .vif. Refs #9331. * fix(balance): skip remote-tiered volumes in admin balance detection The admin/worker balance detector had no equivalent of the shell-side guard ("does not move volume in remote storage" in command_volume_balance.go), so it scheduled moves on remote-tiered volumes. The "move" copies .idx/.vif to the destination and then calls Volume.Destroy on the source, which calls backendStorage.DeleteFile — deleting the remote object the destination's new .vif now points at. Populate HasRemoteCopy on the metrics emitted by both the admin maintenance scanner and the worker's master poll, then drop those volumes at the top of Detection. Fixes #9331. * Apply suggestion from @gemini-code-assist[bot] Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * fix(volume): keep remote data on volume-move-driven delete The on-source delete after a volume move (admin/worker balance and shell volume.move) ran Volume.Destroy with no way to opt out of the remote-object cleanup. Volume.Destroy unconditionally calls backendStorage.DeleteFile for remote-tiered volumes, so a successful move would copy .idx/.vif to the destination and then nuke the cloud object the destination's new .vif was already pointing at. Add VolumeDeleteRequest.keep_remote_data and plumb it through Store.DeleteVolume / DiskLocation.DeleteVolume / Volume.Destroy. The balance task and shell volume.move set it to true; the post-tier-upload cleanup of other replicas and the over-replication trim in volume.fix.replication also set it to true since the remote object is still referenced. Other real-delete callers keep the default. The delete-before-receive path in VolumeCopy also sets it: the inbound copy carries a .vif that may reference the same cloud object as the existing volume. Refs #9331. * test(storage): in-process remote-tier integration tests Cover the four operations the user is most likely to run against a cloud-tiered volume — balance/move, vacuum, EC encode, EC decode — by registering a local-disk-backed BackendStorage as the "remote" tier and exercising the real Volume / DiskLocation / EC encoder code paths. Locks in: - Destroy(keepRemoteData=true) preserves the remote object (move case) - Destroy(keepRemoteData=false) deletes it (real-delete case) - Vacuum/compact on a remote-tier volume never deletes the remote object - EC encode requires the local .dat (callers must download first) - EC encode + rebuild round-trips after a tier-down Tests run in-process and finish in under a second total — no cluster, binary, or external storage required. * fix(rust-volume): keep remote data on volume-move-driven delete Mirror the Go fix in seaweed-volume: plumb keep_remote_data through grpc volume_delete → Store.delete_volume → DiskLocation.delete_volume → Volume.destroy, and skip the s3-tier delete_file call when the flag is set. The pre-receive cleanup in volume_copy passes true for the same reason as the Go side: the inbound copy carries a .vif that may reference the same cloud object as the existing volume. The Rust loader already warns rather than fataling on a stray .vif without an .idx (volume.rs load_index_inmemory / load_index_redb), so no counterpart to the Go fatal-on-missing-idx fix is needed. Refs #9331. * fix(volume): preserve remote tier on IO-error eviction; fix EC test target Two review nits: - Store.MaybeAddVolumes' periodic cleanup pass deleted IO-errored volumes with keepRemoteData=false, so a transient local fault on a remote-tiered volume would also nuke the cloud object. Track the delete reason via a parallel slice and pass keepRemoteData=v.HasRemoteFile() for IO-error evictions; TTL-expired evictions still pass false. - TestRemoteTier_ECEncodeDecode_AfterDownload deleted shards 0..3 but called them "parity" — by the klauspost/reedsolomon convention shards 0..DataShardsCount-1 are data and DataShardsCount..TotalShardsCount-1 are parity. Switch the loop to delete the parity range so the intent matches the indices. --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
654292b57d |
fix(volume): cap leveldb OpenFilesCacheCapacity per index DB (#9139) (#9223)
* fix(volume): cap leveldb OpenFilesCacheCapacity per index DB (#9139) The leveldb opt.Options for NeedleMapLevelDb / Medium / Large never set OpenFilesCacheCapacity, so each leveldb instance defaulted to goleveldb's 500. On servers with thousands of volumes, that ceiling stacks across DBs and exhausts even high ulimits, starving WAL rotation: failed to write leveldb: open .../000006.log: too many open files CompactionTableSizeMultiplier=10 already keeps the SST count low, so a small per-DB cache is sufficient. Cap at 16 / 32 / 64 for the small / medium / large variants so per-DB FD usage is bounded. * storage: hoist leveldb FD-cap values into named constants Per review feedback: replace the inline 16/32/64 literals with LevelDb{,Medium,Large}OpenFilesCacheCapacity, and move the rationale (why 500 is too high per-DB on busy servers, what the tradeoff is) into a package-level comment so future readers see the memory vs. performance picture at the constant declaration instead of inline. |
||
|
Chris Lu
|
45578a42e9 |
fix(volume): keep vacuum running past dangling .idx entries (#9115)
* fix(volume): keep vacuum running past dangling .idx entries Vacuum compaction aborted entirely on the first .idx entry whose offset pointed past the end of the .dat file, surfacing as `cannot hydrate needle from file: EOF` and stalling progress on every other volume. In both Go and Rust: - During compaction, skip an unreadable needle and continue. The bytes it pointed at were already unreachable via reads, so dropping the index reference makes the post-vacuum volume consistent. Real EIO still bails out so a disk fault is not silently papered over. - At volume load, do a single linear scan of the .idx and confirm every (offset + actual size) fits inside .dat. The pre-existing integrity check only looked at the last 10 entries, so deeper corruption (e.g. left over from a crashed batched write) went undetected and only surfaced later as a vacuum EOF. A failure now marks the volume read-only at load time so an operator can react. Refs #8928 * fix(volume): only skip permanent-corruption needle reads during vacuum Address PR review feedback (gemini-code-assist + coderabbit): The original patch skipped any non-EIO read failure, which would silently drop needles on transient errors — Windows hardware bad-sector errors (ERROR_CRC etc.) never surface as syscall.EIO; tiered-storage network timeouts and EROFS would also slip through and shrink the volume. Switch to an explicit whitelist of permanent-corruption shapes: - Add needle.ErrorCorrupted sentinel and wrap CRC and "index out of range" errors with %w so callers can match via errors.Is. - copyDataBasedOnIndexFile now skips only when the read failure is io.EOF, io.ErrUnexpectedEOF, ErrorSizeMismatch, ErrorSizeInvalid, or ErrorCorrupted. Anything else (real disk faults, environmental errors, Windows hardware codes) aborts the compaction so an operator notices. - Mirror the same whitelist in the Rust volume server, matching on io::ErrorKind::UnexpectedEof and the NeedleError corruption variants (SizeMismatch, CrcMismatch, IndexOutOfRange, TailTooShort). Also add `defer v.Close()` in TestVerifyIndexFitsInDat so Windows t.TempDir() cleanup can release the .dat/.idx handles. Refs #8928 * fix(volume): wrap entry-not-found size-mismatch with ErrorSizeMismatch Address PR review: the fallback branch in ReadBytes returned an unwrapped fmt.Errorf, so isSkippableNeedleReadError (and any caller using errors.Is(..., ErrorSizeMismatch)) could not match it. Wrap with %w so the whitelist applies, while leaving the existing direct sentinel return for the OffsetSize==4 / offset<MaxPossibleVolumeSize retry path unchanged so ReadData's `err == ErrorSizeMismatch` retry still triggers. Refs #8928 * fix(volume): integrate dangling-idx check into existing index load walk Address PR review (gemini-code-assist, medium): the structural .idx check used to do a second linear scan of the index file at every volume load, doubling the disk-I/O cost on servers managing many volumes. Track the largest (offset + actual size) seen during the existing needle-map load walks (`LoadCompactNeedleMap`, `NewLevelDbNeedleMap`, `NewSortedFileNeedleMap`'s `newNeedleMapMetricFromIndexFile`, `DoOffsetLoading`) on a new `MaximumNeedleEnd` field on `mapMetric`, exposed as `MaxNeedleEnd()` on the NeedleMapper interface. `volume.load()` then compares `nm.MaxNeedleEnd()` to the .dat size after the load is complete — pure numeric comparison, no extra I/O. The standalone `verifyIndexFitsInDat` helper and its caller in `CheckVolumeDataIntegrity` are removed; the test that used to drive the helper directly now exercises the new path via `LoadCompactNeedleMap`. Mirror the same change in the Rust volume server: track `max_needle_end` on `NeedleMapMetric`, expose via `max_needle_end()` on `CompactNeedleMap`, `RedbNeedleMap`, and the `NeedleMap` enum. The Rust load walk already happens in `load_from_idx` for both map kinds, so the structural check becomes free. Refs #8928 |
||
|
Chris LuandCopilot
|
af68449a26 |
Process .ecj deletions during EC decode and vacuum decoded volume (#8863)
* Process .ecj deletions during EC decode and vacuum decoded volume (#8798) When decoding EC volumes back to normal volumes, deletions recorded in the .ecj journal were not being applied before computing the dat file size or checking for live needles. This caused the decoded volume to include data for deleted files and could produce false positives in the all-deleted check. - Call RebuildEcxFile before HasLiveNeedles/FindDatFileSize in VolumeEcShardsToVolume so .ecj deletions are merged into .ecx first - Vacuum the decoded volume after mounting in ec.decode to compact out deleted needle data from the .dat file - Add integration tests for decoding with non-empty .ecj files * storage: add offline volume compaction helper Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ec: compact decoded volumes before deleting shards Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ec: address PR review comments - Fall back to data directory for .ecx when idx directory lacks it - Make compaction failure non-fatal during EC decode - Remove misleading "buffer: 10%" from space check error message * ec: collect .ecj from all shard locations during decode Each server's .ecj only contains deletions for needles whose data resides in shards held by that server. Previously, sources with no new data shards to contribute were skipped entirely, losing their .ecj deletion entries. Now .ecj is always appended from every shard location so RebuildEcxFile sees the full set of deletions. * ec: add integration tests for .ecj collection during decode TestEcDecodePreservesDeletedNeedles: verifies that needles deleted via VolumeEcBlobDelete are excluded from the decoded volume. TestEcDecodeCollectsEcjFromPeer: regression test for the fix in collectEcShards. Deletes a needle only on a peer server that holds no new data shards, then verifies the deletion survives decode via .ecj collection. * ec: address review nits in decode and tests - Remove double error wrapping in mountDecodedVolume - Check VolumeUnmount error in peer ecj test - Assert 404 specifically for deleted needles, fail on 5xx --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Chris Lu
|
a473278bfa |
Fix: Fail fast on unsupported volume versions (#8047)
* Fix: Fail fast when initializing volume with Version 0 * Fix: Fail fast when loading unsupported volume version (e.g. 0 or 4) * Refactor: Use IsSupportedVersion helper function for version validation |
||
|
Chris Lu
|
3613279f25 |
Add 'weed mini' command for S3 beginners and small/dev use cases (#7831)
* Add 'weed mini' command for S3 beginners and small/dev use cases
This new command simplifies starting SeaweedFS by combining all components
in one process with optimized settings for development and small deployments.
Features:
- Starts master, volume, filer, S3, WebDAV, and admin in one command
- Volume size limit: 64MB (optimized for small files)
- Volume max: 0 (auto-configured based on free disk space)
- Pre-stop seconds: 1 (faster shutdown for development)
- Master peers: none (single master mode by default)
- Includes admin UI with one worker for maintenance tasks
- Clean, user-friendly startup message with all endpoint URLs
Usage:
weed mini # Use default temp directory
weed mini -dir=/data # Custom data directory
This makes it much easier for:
- Developers getting started with SeaweedFS
- Testing and development workflows
- Learning S3 API with SeaweedFS
- Small deployments that don't need complex clustering
* Change default volume server port to 9340 to avoid popular port 8080
* Fix nil pointer dereference by initializing all required volume server fields
Added missing VolumeServerOptions field initializations:
- id, publicUrl, diskType
- maintenanceMBPerSecond, ldbTimeout
- concurrentUploadLimitMB, concurrentDownloadLimitMB
- pprof, idxFolder
- inflightUploadDataTimeout, inflightDownloadDataTimeout
- hasSlowRead, readBufferSizeMB
This resolves the panic that occurred when starting the volume server.
* Fix multiple nil pointer dereferences in mini command
Added missing field initializations for:
- Master options: raftHashicorp, raftBootstrap, telemetryUrl, telemetryEnabled
- Filer options: filerGroup, saveToFilerLimit, concurrentUploadLimitMB,
concurrentFileUploadLimit, localSocket, showUIDirectoryDelete,
downloadMaxMBps, diskType, allowedOrigins, exposeDirectoryData, tusBasePath
- Volume options: id, publicUrl, diskType, maintenanceMBPerSecond, ldbTimeout,
concurrentUploadLimitMB, concurrentDownloadLimitMB, pprof, idxFolder,
inflightUploadDataTimeout, inflightDownloadDataTimeout, hasSlowRead, readBufferSizeMB
- WebDAV options: tlsPrivateKey, tlsCertificate, filerRootPath
- Admin options: master
These initializations are required to avoid runtime panics when starting components.
* Fix remaining S3 option nil pointers in mini command
* Update mini command: 256MB volume size and add S3 access instructions for beginners
* mini: set default master.volumeSizeLimitMB to 128MB and update help/banner text
* mini: shorten S3 help text to a concise pointer to docs/Admin UI
* mini: remove duplicated component bullet list, use concise sentence
* mini: tidy help alignment and update example usage
* mini: default -dir to current directory
* mini: load initial S3 credentials from env and write IAM config
* mini: use AWS env vars for initial S3 creds; instruct to create via Admin UI if absent
* Improve startup synchronization with channel-based coordination
- Replace fragile time.Sleep delays with robust channel-based synchronization
- Implement proper service dependency ordering (Master → Volume → Filer → S3/WebDAV/Admin)
- Add sync.WaitGroup for goroutine coordination
- Add startup readiness logging for better visibility
- Implement 10-second timeout for admin server startup
- Remove arbitrary sleep delays for faster, more reliable startup
- Services now start deterministically based on dependencies, not timing
This makes the startup process more reliable and eliminates race conditions on slow systems or under load.
* Refactor service startup logic for better maintainability
Extract service startup into dedicated helper functions:
- startMiniServices(): Orchestrates all service startup with dependency coordination
- startServiceWithCoordination(): Starts services with readiness signaling
- startServiceWithoutReady(): Starts services without readiness signaling
- startS3Service(): Encapsulates S3 initialization logic
Benefits:
- Reduced code duplication in runMini()
- Clearer separation of concerns
- Easier to add new services or modify startup sequence
- More testable code structure
- Improved readability with explicit service names and logging
* Remove unused serviceStartupInfo struct type
- Delete the serviceStartupInfo struct that was defined but never used
- Improves code clarity by removing dead code
- All service startup is now handled directly by helper functions
* Preserve existing IAM config file instead of truncating
- Use os.Stat to check if IAM config file already exists
- Only create and write configuration if file doesn't exist
- Log appropriate messages for each case:
* File exists: skip writing, preserve existing config
* File absent: create with os.OpenFile and write new config
* Stat error: log error without overwriting
- Set *miniIamConfig only when new file is successfully created
- Use os.O_CREATE|os.O_WRONLY flags for safe file creation
- Handles file operations with proper error checking and cleanup
* Fix CodeQL security issue: prevent logging of sensitive S3 credentials
- Add createdInitialIAM flag to track when initial IAM config is created from env vars
- Set flag in startS3Service() when new IAM config is successfully written
- Update welcome message to inform user of credential creation without exposing secrets
- Print only the username (mini) and config file location to user
- Never print access keys or secret keys in clear text
- Maintain security while keeping user informed of what was created
- Addresses CodeQL finding: Clear-text logging of sensitive information
* Fix three code review issues in weed mini command
1. Fix deadlock in service startup coordination:
- Run blocking service functions (startMaster, startFiler, etc.) in separate goroutines
- This allows readyChan to be closed and prevents indefinite blocking
- Services now start concurrently instead of sequentially blocking the coordinator
2. Use shared grace.StartDebugServer for consistency:
- Replace inline debug server startup with grace.StartDebugServer
- Improves code consistency with other commands (master, filer, etc.)
- Removes net/http import which is no longer needed
3. Simplify IAM config file cleanup with defer:
- Use 'defer f.Close()' instead of multiple f.Close() calls
- Ensures file is closed regardless of which code path is taken
- Improves robustness and code clarity
* fmt
* Fix: Remove misleading 'service is ready' logs
The previous fix removed 'go' from service function calls but left misleading
'service is ready' log messages. The service helpers now correctly:
- Call fn() directly (blocking) instead of 'go fn()' (non-blocking)
- Remove the 'service is ready' message that was printed before the service
actually started running
- Services run as blocking goroutines within the coordinator goroutine,
which keeps them alive while the program runs
- The readiness channels still work correctly because they're closed when
the coordinator finishes waiting for dependencies
* Update mini.go
* Fix four code review issues in weed mini command
1. Use restrictive file permissions (0600) for IAM config:
- Changed from 0644 to 0600 when creating iam_config.json
- Prevents world-readable access to sensitive AWS credentials
- Protects AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
2. Remove unused sync.WaitGroup:
- Removed WaitGroup that was never waited on
- All services run as blocking goroutines in the coordinator
- Main goroutine blocks indefinitely with select{}
- Removes unnecessary complexity without changing behavior
3. Merge service startup helper functions:
- Combined startServiceWithCoordination and startServiceWithoutReady
- Made readyChan optional (nil for services without readiness signaling)
- Reduces code duplication and improves maintainability
- Both services now use single startServiceWithCoordination function
4. Fix admin server readiness check:
- Removed misleading timeout channel that never closed at startup
- Replaced with simple 2-second sleep before worker startup
- startAdminServer() blocks indefinitely, so channel would only close on shutdown
- Explicit sleep is clearer about the startup coordination intent
* Fix three code quality issues in weed mini command
1. Define volume configuration as named constants:
- Added miniVolumeMaxDataVolumeCounts = "0"
- Added miniVolumeMinFreeSpace = "1"
- Added miniVolumeMinFreeSpacePercent = "1"
- Removed local variable assignments in Volume startup
- Improves maintainability and documents configuration intent
2. Fix deadlock in startServiceWithCoordination:
- Changed from 'defer close(readyChan)' with blocking fn() to running fn() in goroutine
- Close readyChan immediately after launching service goroutine
- Prevents deadlock where fn() never returns, blocking defer execution
- Allows dependent services to start without waiting for blocking call
3. Improve admin server readiness check:
- Replaced fixed 2-second sleep with polling the gRPC port
- Polls up to 20 times (10 seconds total) with 500ms intervals
- Uses net.DialTimeout to check if port is available
- Properly handles IPv6 addresses using net.JoinHostPort
- Logs progress and warnings about connection status
- More robust than sleep against server startup timing variations
4. Add net import for network operations (IPv6 support)
Also fixed IAM config file close error handling to properly check error
from f.Close() and log any failures, preventing silent data loss on NFS.
* Document environment variable setup for S3 credentials
Updated welcome message to explain two ways to create S3 credentials:
1. Environment variables (recommended for quick setup):
- Set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
- Run 'weed mini -dir=/data'
- Creates initial 'mini' user credentials automatically
2. Admin UI (for managing multiple users and policies):
- Open http://localhost:23646 (Admin UI)
- Add identities to create new S3 credentials
This gives users clear guidance on the easiest way to get started with S3
credentials while also explaining the more advanced option for multiple users.
* Print welcome message after all services are running
Moved the welcome message printing from immediately after startMiniServices()
to after all services have been started and are ready. This ensures users see
the welcome message only after startup is complete, not mixed with startup logs.
Changes:
- Extract welcome message logic into printWelcomeMessage() function
- Call printWelcomeMessage() after startMiniServices() completes
- Change message from 'are starting' to 'are running and ready to use'
- This provides cleaner startup output without interleaved logs
* Wait for all services to complete before printing welcome message
The welcome message should only appear after all services are fully running and
the worker is connected. This prevents the message from appearing too early before
startup logs complete.
Changes:
- Pass allServicesReady channel through startMiniServices()
- Add adminReadyChan to track when admin/worker startup completes
- Signal allServicesReady when admin service is fully ready
- Wait for allServicesReady in runMini() before printing welcome message
- This ensures clean output: startup logs first, then welcome message once ready
Now the user sees all startup activity, then a clear welcome message when
everything is truly ready to use.
* Fix welcome message timing: print after worker is fully started
The welcome message was printing too early because allServicesReady was being
closed when the Admin service goroutine started, not when it actually completed.
The Admin service launches startMiniAdminWithWorker() which is a blocking call
that doesn't return until the worker is fully connected.
Now allServicesReady is passed through to startMiniWorker() which closes it
after the worker successfully starts and connects to the admin server.
This ensures the welcome message only appears after:
- Master is ready
- Volume server is ready
- Filer is ready
- S3 service is ready
- WebDAV service is ready
- Admin server is ready
- Worker is connected and running
All startup logs appear first, then the clean welcome message at the end.
* Wait for S3 and WebDAV services to be ready before showing welcome message
The welcome message was printing before S3 and WebDAV servers had fully
initialized. Now the readiness flow is:
1. Master → ready
2. Volume → ready
3. Filer → ready
4. S3 → ready (signals s3ReadyChan)
5. WebDAV → ready (signals webdavReadyChan)
6. Admin/Worker → starts, then waits for both S3 and WebDAV
7. Welcome message prints (all services truly ready)
Changes:
- Add s3ReadyChan and webdavReadyChan to service startup
- Pass S3 and WebDAV ready channels through to Admin service
- Admin/Worker waits for both S3 and WebDAV before closing allServicesReady
- This ensures welcome message appears only when all services are operational
* Admin service should wait for Filer, S3, and WebDAV to be ready
Admin service depends on Filer being operational since it uses the filer
for credential storage. It also makes sense to wait for S3 and WebDAV
since they are user-facing services that should be ready before Admin.
Updated dependencies:
- Admin now waits for: Master, Filer, S3, WebDAV
- This ensures all critical services are operational before Admin starts
- Welcome message will print only after all services including Admin are ready
* Add initialization delay for S3 and WebDAV services
S3 and WebDAV servers need extra time to fully initialize and start listening
after their service functions are launched. Added a 1-second delay after
launching S3 and WebDAV goroutines before signaling readiness.
This ensures the welcome message doesn't print until both services have
emitted their startup logs and are actually serving requests.
* Increase service initialization wait times for more reliable startup
- Increase S3 and WebDAV initialization delay from 1s to 2s to ensure they emit startup logs before welcome message
- Add 1s initialization delay for Filer to ensure it's listening
- Increase admin gRPC polling timeout from 10s to 20s to ensure admin server is fully ready
- This ensures welcome message prints only after all services are fully initialized and ready to accept requests
* Increase service wait time to 10 seconds for reliable startup
All services now wait 10 seconds after launching to ensure they are fully initialized and ready before signaling readiness to dependent services. This ensures the welcome message prints only after all services have fully started.
* Replace fixed 10s delay with intelligent port polling for service readiness
Instead of waiting a fixed 10 seconds for each service, now polls the service
port to check if it's actually accepting connections. This eliminates unnecessary
waiting and allows services to signal readiness as soon as they're ready.
- Polls each service port with up to 30 attempts (6 seconds total)
- Each attempt waits 200ms before retrying
- Stops polling immediately once service is ready
- Falls back gracefully if service is unknown
- Significantly faster startup sequence while maintaining reliability
* Replace channel-based coordination with HTTP pinging for service readiness
Instead of using channels to coordinate service startup, now uses HTTP GET requests
to ping each service endpoint to check if it's ready to accept connections.
Key changes:
- Removed all readiness channels (masterReadyChan, volumeReadyChan, etc.)
- Simplified startMiniServices to use sequential HTTP polling for each service
- startMiniService now just starts the service with logging
- waitForServiceReady uses HTTP client to ping service endpoints (max 6 seconds)
- waitForAdminServerReady uses HTTP GET to check admin server availability
- startMiniAdminWithWorker and startMiniWorker simplified without channel parameters
Benefits:
- Cleaner, more straightforward code
- HTTP pinging is more reliable than TCP port probing
- Services signal readiness through their HTTP endpoints
- Eliminates channel synchronization complexity
* log level
* Remove overly specific comment from volume size limit in welcome message
The '(good for small files)' comment is too limiting. The 128MB volume size
limit works well for general use cases, not just small files. Simplified the
message to just show the value.
* Ensure allServicesReady channel is always closed via defer
Add 'defer close(allServicesReady)' at the start of startMiniAdminWithWorker
to guarantee the channel is closed on ALL exit paths (normal and error).
This prevents the caller waiting on <-allServicesReady from ever hanging,
while removing the explicit close() at the successful end prevents panic
from double-close.
This makes the code more robust by:
- Guaranteeing channel closure even if worker setup fails
- Eliminating the possibility of caller hanging on errors
- Following Go defer patterns for resource cleanup
* Enhance health check polling for more robust service coordination
The service startup already uses HTTP health checks via waitForServiceReady()
to verify services are actually accepting connections. This commit improves
the health check implementation:
Changes:
- Elevated success logging to Info level so users see when services become ready
- Improved error messages to clarify that health check timeouts are not fatal
- Services continue startup even if health checks timeout (they may still work)
- Consistent handling of health check results across all services
This provides better visibility into service startup while maintaining the
existing robust coordination via HTTP pinging rather than just TCP port checks.
* Implement stricter error handling for robust mini server startup
Apply all PR review feedback to ensure the mini server fails fast and clearly
when critical components cannot start:
Changes:
1. Remove redundant miniVolumeMinFreeSpacePercent constant
- Simplified util.MustParseMinFreeSpace() call to use single parameter
2. Make service readiness checks fatal errors:
- Master, Volume, Filer, S3, WebDAV health check failures now return errors
- Prevents partially-functional servers from running
- Caller can handle errors gracefully instead of continuing with broken state
3. Make admin server readiness fatal:
- Admin gRPC availability is critical for worker startup
- Use glog.Fatalf to terminate with clear error message
4. Improve IAM config error handling:
- Treat all file operation failures (stat, open, write, close) as fatal
- Prevents silent failures in S3 credential setup
- User gets immediate feedback instead of authentication issues later
5. Use glog.Fatalf for critical worker setup errors:
- Failed to create worker directory, task directories, or worker instance
- Failed to create admin client or start worker
- Ensures mini server doesn't run in broken state
This ensures deterministic startup: services succeed completely or fail with
clear, actionable error messages for the user.
* Make health checks non-fatal for graceful degradation and improve IAM file handling
Address PR feedback to make the mini command more resilient for development:
1. Make health check failures non-fatal
- Master, Volume, Filer, S3, WebDAV health checks now log warnings but allow startup
- Services may still work even if health check endpoints aren't immediately available
- Aligns with intent of a dev-focused tool that should be forgiving of timing issues
- Only prevents startup if startup coordination or critical errors occur
2. Improve IAM config file handling
- Refactored to guarantee file is always closed using separate error variables
- Opens file once and handles write/close errors independently
- Maintains strict error handling while improving code clarity
- All file operation failures still cause fatal errors (as intended)
This makes startup more graceful while maintaining critical error handling for
fundamental failures like missing directories or configuration errors.
* Fix code quality issues in weed mini command
- Fix pointer aliasing: use value copy (*miniBindIp = *miniIp) instead of pointer assignment
- Remove misleading error return from waitForServiceReady() function
- Simplify health check callers to call waitForServiceReady() directly without error handling
- Remove redundant S3 option assignments already set in init() block
- Remove unused allServicesReady parameter from startMiniWorker() function
* Refactor welcome message to use template strings and add startup delay
- Convert welcome message to constant template strings for cleaner code
- Separate credentials instructions into dedicated constant
- Add 500ms delay after worker startup to allow full initialization before welcome message
- Improves output cleanliness by avoiding log interleaving with welcome message
* Fix code style issues in weed mini command
- Fix indentation in IAM config block (lines 424-432) to align with surrounding code
- Remove unused adminServerDone channel that was created but never read
* Address code review feedback for robustness and resource management
- Use defer f.Close() for IAM file handling to ensure file is closed in all code paths, preventing potential file descriptor leaks
- Use 127.0.0.1 instead of *miniIp for service readiness checks to ensure checks always target localhost, improving reliability in environments with firewalls or complex network configurations
- Simplify error handling in waitForAdminServerReady by using single error return instead of separate write/close error variables
* Fix function declaration formatting
- Separate closing brace of startS3Service from startMiniAdminWithWorker declaration with blank line
- Move comment to proper position above function declaration
- Run gofmt for consistent formatting
* Fix IAM config pointer assignment when file already exists
- Add missing *miniIamConfig = iamPath assignment when IAM config file already exists
- Ensures S3 service is properly pointed to the existing IAM configuration
- Retains logging to inform user that existing configuration is being preserved
* Improve pointer assignments and worker synchronization
- Simplify grpcPort and dataDir pointer assignments by directly dereferencing and assigning values instead of taking address of local variables
- Replace time.Sleep(500ms) with proper TCP-based polling to wait for worker gRPC port readiness
- Add waitForWorkerReady function that polls worker's gRPC port with max 6-second timeout
- Add net package import for TCP connection checks
- Improves code idiomaticity and synchronization robustness
* Refactor and simplify error handling for maintainability
- Remove unused error return from startMiniServices (always returned nil)
- Update runMini caller to not expect error from startMiniServices
- Refactor init() into component-specific helper functions:
* initMiniCommonFlags() for common options
* initMiniMasterFlags() for master server options
* initMiniFilerFlags() for filer server options
* initMiniVolumeFlags() for volume server options
* initMiniS3Flags() for S3 server options
* initMiniWebDAVFlags() for WebDAV server options
* initMiniAdminFlags() for admin server options
- Significantly improves code readability and maintainability
- Each component's flags are now in dedicated, focused functions
|
||
|
|
fa35efc076 |
Volume Server: Unexpected Deletion of Remote Tier Data (#7377)
* [Admin UI] Login not possible due to securecookie error * avoid 404 favicon * Update weed/admin/dash/auth_middleware.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * address comments * avoid variable over shadowing * log session save error * When jwt.signing.read.key is enabled in security.toml, the volume server requires JWT tokens for all read operations. * reuse fileId * refactor * fix deleting remote tier * simplify the fix * Update weed/storage/volume_loading.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/storage/volume_loading.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/storage/volume_loading.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Chris Lu
|
69553e5ba6 | convert error fromating to %w everywhere (#6995) | ||
|
chrislu
|
2f1b3d68d7 | pass volume version when creating a volume | ||
|
chrislu
|
be15fee8e7 |
show error if backend is mis-configured
related to https://github.com/seaweedfs/seaweedfs/discussions/6472 |
||
|
Konstantin Lebedev
|
6d4f5c94ac | [volume] remove truncate idx file if size not healthy (#6043) | ||
|
Bruce
|
f9e141a412 | persist readonly state to volume info (#5977) | ||
|
Bruce
|
310d41998d | Skip volume data integrity check for volume which has remote file (#5976) | ||
|
steve.wei
|
0bdf121e51 | rename VolumeServerVolumeGauge (#5504) | ||
|
chrislu
|
6ebe26a765 |
Revert "Revert "Revert "Add disk type to prometheus metrics" (#4777)""
This reverts commit
|
||
|
chrislu
|
249c0e06ef |
Revert "fix compilation"
This reverts commit
|
||
|
chrislu
|
451ec6504d | fix compilation | ||
|
chrislu
|
567d788928 |
Revert "Revert "Add disk type to prometheus metrics" (#4777)"
This reverts commit
|
||
|
|
5b905fb2b7 |
Lazy loading (#3958)
* types packages is imported more than onece * lazy-loading * fix bugs * fix bugs * fix unit tests * fix test error * rename function * unload ldb after initial startup * Don't load ldb when starting volume server if ldbtimeout is set. * remove uncessary unloadldb * Update weed/command/server.go Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> * Update weed/command/volume.go Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: guol-fnst <goul-fnst@fujitsu.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Konstantin Lebedev
|
2f72103c83 |
avoid load volume file with BytesOffset mismatch (#3841)
* avoid load volume file with BytesOffset mismatch https://github.com/seaweedfs/seaweedfs/issues/2966 * set BytesOffset if has not VolumeInfoFile * typos fail => failed * exit if bytesOffset mismatch |
||
|
Guo Lei
|
c57c79a0ab |
optimiz commitig compact (#3388)
* optimiz vacuuming volume * fix bugx * rename parameters * fix conflict * change copyDataBasedOnIndexFile to an instance method * close needlemap * optimiz commiting Vacuum volume for leveldb index * fix bugs * fix leveldb loading bugs * refactor * fix leveldb loading bug * add leveldb recovery * add test case for levelDB * modify test case to cover all the new branches * use one tmpNm instead of two instances * refactor * refactor * move setWatermark to the end * add test for watermark and updating leveldb * fix error logic * refactor, add test * check nil before close needlemapeer add test case fix metric bug * add tests, fix bugs * adjust log level remove wrong test case refactor * avoid duplicate updating metric for leveldb index |
||
|
chrislu
|
26dbc6c905 | move to https://github.com/seaweedfs/seaweedfs | ||
|
chrislu
|
59e58c4b23 |
volume: fix loading old volume format
fix https://github.com/chrislusf/seaweedfs/issues/2487 |
||
|
Chris Lu
|
c46d16b2f3 | skip network error when remote volumes start | ||
|
Chris Lu
|
6e43e8ce17 | add logs for remote file loading | ||
|
Chris Lu
|
015d16f43f | add vif file versions in case loading superblock fails | ||
|
Chris Lu
|
885ca34748 |
volume: fail fast if idx files are missing
fix https://github.com/chrislusf/seaweedfs/issues/1796 |
||
|
bingoohuang
|
94ea3bd3a5 | renaming NeedleMapType to NeedleMapKind | ||
|
Chris Lu
|
4f31c1bb94 | go fmt | ||
|
Chris Lu
|
d171d9f988 |
volume: ensure the volume dat and idx files are always closed
fix https://github.com/chrislusf/seaweedfs/issues/1646 |
||
|
Chris Lu
|
6d30b21b10 |
volume: add "-dir.idx" option for separate index storage
fix https://github.com/chrislusf/seaweedfs/issues/1265 |
||
|
Chris Lu
|
744c5594da | volume: automatically trim out unreachable entries | ||
|
Chris Lu
|
faa5c2e89a | refactoring | ||
|
Chris Lu
|
c8ca234773 | refactoring |