mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-12 23:09:33 +01:00
a3b68aa37cc8a161e232a7135d57dd389353cfaf
8
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris LuandClaude Opus 5.5
|
641ab6204b |
shell: fs.mergeVolumes leaves in-progress multipart parts in place (#12556)
* shell: fs.mergeVolumes leaves in-progress multipart parts in place CompleteMultipartUpload reads the part entries and later commits the object with their fids, guarded only by the upload directory. A merge that moved a part chunk in between deleted the source needle the new object pointed at. Parts stay where they are until the upload completes or is cleaned, and their chunks still count as referenced sources. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: note in fs.mergeVolumes help that upload parts stay in place Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: match upload parts when the buckets folder is the root Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: count chunks behind a part's manifest as referenced sources A skipped part whose chunks on a merge source sit behind a manifest was not counted, so the merge printed no skip line and warned that nothing references the source. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: keep a part's direct source references when its manifest is unreadable Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: count chunks resolved before an unreadable part manifest ResolveChunkManifest returns the data chunks of earlier manifests along with a later manifest's error; record them before returning so the unreferenced-source warning does not fire for a source they reference. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: test the partial manifest result without a timing delay The unreadable manifest is now nested under a readable part manifest, so the first level resolves fully before the failing lookup runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
d13d975b6e |
shell: keep fs.mergeVolumes and fs.distributeChunks from reverting entries written while they run (#12362)
* shell: keep fs.mergeVolumes from reverting entries written during the merge The walk reads a page of entries and moves their chunks one at a time, then wrote each entry back unconditionally. A file overwritten in between got its old version back, and the filer deleted the new version's chunks as garbage, so an acknowledged write was lost without any error. The update now carries IF_ENTRY_EQUAL over the entry as the walk read it. When the filer refuses because the entry changed or was deleted, the copies made on the target volumes are deleted, their plan room is released, and the entry is skipped; the source needles stay where the filer still points. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: keep fs.distributeChunks from reverting a file overwritten while it runs The command looks the entry up once, moves its chunks, and then wrote that entry back unconditionally, so an overwrite during the redistribution was reverted and the filer deleted the new version's chunks as garbage. The update now carries IF_ENTRY_EQUAL over the looked-up entry. When the filer refuses because the file changed or was deleted, the moved copies and any new manifest chunks are deleted and the command reports it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: keep s3.versions.audit -heal from clearing a pointer that just moved The heal cleared the latest-version pointer on the directory entry the walk read, unconditionally. A version written between the audit and the heal has already pointed the directory at a live file, and clearing that pointer hid it. The heal now carries IF_ENTRY_EQUAL over the audited entry and reports a refusal as a failed heal. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: keep fs.meta.changeVolumeId from reverting entries written meanwhile The walk rewrote each entry's volume ids and wrote back the copy it had read, unconditionally, so an overwrite between the read and the write was reverted and its chunks were deleted as garbage. The update now carries IF_ENTRY_EQUAL over the entry as read, and a refused entry is skipped and reported. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: keep the target copies another fs.mergeVolumes run committed moveChunk keeps the key and cookie, so two overlapping merges of the same source into the same target write the same fid. When one run commits, the other's update is refused, and deleting its copies deleted the needle the committed entry now points at. A refused update now re-reads the entry, resolving its manifests, and deletes only the copies it no longer references. If the entry cannot be read, the copies are left for volume.fsck. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: let a later hard-link sibling retry a merge the filer refused The first sibling of a HardLinkId claims the move for all of them. When its update is refused, the claim made every later sibling skip, so the shared chunks stayed on the source volume for the rest of the run. A refused update now releases the claim, and a sibling listed after the change retries the move against the entry as it is now. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: delete the manifest blobs a failed fold in fs.distributeChunks saved When re-manifestizing failed partway, the blobs already uploaded were only counted in a log line and stayed on their volumes with nothing referencing them. They have freshly assigned fids, so they are now deleted with the cookie-checked delete the other orphan cleanups use. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: check the heal clears the pointer file name too Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: require the shell lock for fs.mergeVolumes -apply moveChunk keeps the key and cookie, so two overlapping runs that move the same chunk into the same volume write the same fid. Re-reading the entry before deleting a refused run's copies covers the run that already committed, but not one that commits after that read. With the lock, two applying runs can no longer overlap, as volume.fsck and volume.merge already require. A dry run still works without it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: check unreferencedTargets reads the refused entry itself Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: check unreferencedTargets keeps a copy a manifest references Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: check unreferencedTargets keeps the manifest copy and skips an unreadable manifest Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris Lu
|
b0ab4c4e1e |
shell: make every -collection flag parse the same way (#11794)
* shell: make every -collection flag parse the same way The filter vocabulary (comma-separated names, wildcards, regexes, "_default" for the empty-named collection) already existed in wildcard.CompileCollectionMatcher, but several commands still did their own exact-match comparison or sent the raw flag value to servers. fs.mergeVolumes used a literal string compare, so -collection=_default silently produced an empty merge plan; it now uses the shared matcher while keeping "*" as the match-all default. volume.fsck and volume.mark filter through the matcher too, and volume.vacuum resolves the filter client-side into per-collection and per-volume requests, since no Collection value scopes the VacuumVolume RPC to the unnamed collection alone. Commands that put one concrete collection name on the wire normalize "_default" to "" (volume.grow, fs.configure, mq.topic.compact, collection.delete, ec.rebuild, ec.balance), and the tier -volumeId commands resolve the volume's actual collection from topology first so patterns and named-collection requests pass the volume server's collection check. * shell: spell the default-collection keyword via CollectionDefault * shell: address review findings on collection filter handling volume.fsck keeps a set of every volume the master knows before any filtering, so a filer entry whose volume was excluded by -collection or -volumeId is skipped instead of reported "volume not found" and deleted by -reallyDeleteFilerEntries. The collection filter also no longer scopes the filer walk to /buckets/<collection>: fs.configure can bind a collection to any path prefix, and a narrower walk could mark live chunks as orphans under -reallyDeleteFromVolume. ec.rebuild and ec.balance resolve -collection through the new wildcard.ExpandCollectionFilter, which passes literal names through even when the master does not list them — ec.rebuild must be able to recover a collection whose shards all lost their index and never registered. fs.configure and mq.topic.compact now reject _default outright: an empty Collection means "unset" in both the path-rule merge and the filer's storage-option fallback chain, so the keyword would have been silently ignored. The unnamed-collection vacuum sweep keeps the master's low-disk-space exception: a volume whose every replica is read-only is probed with VacuumVolumeCheck and stays eligible when a replica reports DiskSpaceLow, instead of being dropped outright. |
||
|
Nguyễn Đăng Minh LựcandChris Lu
|
ac03d3fd78 |
shell: warn when fs.mergeVolumes source holds only orphan needles (#11310)
* shell: warn when fs.mergeVolumes source holds only orphan needles fs.mergeVolumes traverses filer entries, so a source volume whose needles are all orphans — filer entries lost to a crashed write or a wiped filer store — produces only the plan header and exits 0: no move, no skip, no error. Operators read that as a successful merge while the real cleanup (volume.fsck) never runs, and dat>idx volumes keep coming back read-only after restarts. Count the source-volume needles seen during traversal and, when a plan source was never seen but its index still reports needles, print a warning pointing at volume.fsck. Dry-run warns too. * shell: make needle counting concurrency-safe and count manifest sub-chunks TraverseBfs runs its callbacks from five workers, so the plain needlesSeen map raced between source-heavy merges (fatal concurrent map writes). All increments now funnel through a mutex-guarded recordSeen closure. Manifest sub-chunks that live on planned source volumes are now recorded too — rewriteManifestChunk visits them (including dry-run and capacity-skipped ones) but previously never marked their source, which produced false 'orphan needles' warnings for sources whose chunks were all reached through manifests. * shell: extract sourceNeedleCounter so the concurrency test covers the production path The orphan-warning recording was a closure local to Do, so TestWarnUnreferencedSources_ConcurrentRecording could only exercise a test-local copy of it — a regression in the production mutex would pass the test. Lift the map and mutex into a sourceNeedleCounter type with record/count methods and use it from Do and the test, so the -race test now drives the actual recording path. Trim the verbose comments added with the warning while here. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
b8049bc633 |
shell: keep fs.mergeVolumes from spinning past the finished moves (#11050)
* filer_pb: walk a re-delivered directory only once in TraverseBfs A directory handed back twice by a listing (a page-boundary race with concurrent renames, or a store whose ordering misbehaves) was enqueued twice; the second walk re-lists the same subtree and can keep the traversal from ever terminating. Claude-Session: https://claude.ai/code/session_01XH7iM88ZqWMEvsLB8tkWPQ * filer_pb: fail a directory listing whose pagination stops advancing A full page ending on the very name the cursor started from re-fetches the same page forever; a store whose listing order does not advance past the cursor turns any full-directory read into a silent infinite loop. Return an error naming the stuck cursor instead. Claude-Session: https://claude.ai/code/session_01XH7iM88ZqWMEvsLB8tkWPQ * shell: skip foreign-collection manifests in fs.mergeVolumes Every manifest chunk in the namespace was resolved, downloading its manifest needle, even when the merge plan only touches one collection. Sub-chunks live in the manifest's own collection, so a manifest on a volume outside the plan's collections cannot reference a source volume; skip it and spare a cluster-wide download pass that looks like a hang after the real moves finish. Claude-Session: https://claude.ai/code/session_01XH7iM88ZqWMEvsLB8tkWPQ |
||
|
Chris Lu
|
c6e1387f59 |
shell: multi-target fs.mergeVolumes and volume.mark -readonlyCanDelete (#10706)
* shell: fs.mergeVolumes distributes one volume across multiple -toVolumeId targets * volume: volume.mark -readonlyCanDelete rejects writes but keeps accepting deletes * seaweed-volume: mirror readonlyCanDelete volume state |
||
|
Chris Lu
|
08f0ba5564 |
topology: clamp the deleted-vs-total subtractions in volume stats (#10633)
VolumeLocationList.Stats subtracts the deleted figures from the totals to report live size and needle count. Both deleted figures are maintained as counters independent of the totals they come off, so either can transiently exceed its total, and neither subtraction was clamped. Unclamped, the size wraps to ~16 EB. The count is signed so it merely goes negative, but VolumeLayout.Stats converts it with uint64(fileCount), which turns it into ~1.8e19 just the same. Either one swamps the cluster totals behind /dir/status, /vol/status and Topology.CollectionVolumeStats. commandFsMergeVolumes.getVolumeSize had the same unclamped subtraction, where a wrapped size reads as a volume far too large to join any merge plan. Clamped to zero, matching the guards already in CollectionInfo.LogicalSize and the admin server's logical-size accumulator. |
||
|
Chris Lu
|
a653a7f72a |
fix(shell): honor explicit fs.mergeVolumes from/to direction (#10159)
* fix(shell): honor explicit fs.mergeVolumes from/to direction
mergeVolumes only ever merged a smaller volume into a larger one. When the
user named both -fromVolumeId and -toVolumeId with the source larger than the
target, the planner produced an empty plan and the command printed just
"max volume size: N MB" and moved nothing.
Build the requested pair directly when both ids are given, instead of routing
through the size-descending heuristic. Read-only, empty, and wrong-collection
endpoints are rejected with a clear error rather than a silent no-op.
* fix(shell): allow fs.mergeVolumes into an empty target volume
Merging chunks into an empty volume is valid, e.g. consolidating data into a
freshly created or recently vacuumed volume. Only reject an empty source, which
has nothing to move.
* fix(shell): reject self-map in directed mergeVolumes planner
createMergePlan with from == to returned a {vid: vid} self-merge when called
directly. Guard it in the planner so it is correct independent of the Do
entrypoint.
|