9817 Commits

Include renames

Author	SHA1	Message	Date
Chris Lu	97b54adcf6	iceberg: sort compaction bins on disk instead of in memory (#11112) ... A sorted rewrite collected every row of a bin into one slice and sorted it there, so a bin larger than the worker's heap could not be sorted at all. sort_max_input_mb existed for that reason and skipped the bins it capped. parquet-go's SortingWriter buffers sort_buffer_rows rows, encodes each buffer as a sorted run, and merges the runs at close; backing those runs with a FileBufferPool keeps them in files rather than on the heap. sort_spill_dir says where, defaulting to the system temp directory — NewFileBufferPool resolves an empty path to the working directory, which is not what an unset setting means. The output now also declares its sorting columns, which the plain writer the sorted path used never did. Claude-Session: https://claude.ai/code/session_015SZkLTUvd1svDu4xdr6Q3y	2026-09-02 20:17:30 -07:00
Chris Lu	8398af3572	filer: route exclusive and conditional creates to the entry's ring owner (#11109) ... * proto: resync the java copy of filer.proto The Makefile keeps other/java/client/src/main/proto/filer.proto a verbatim copy, but AssignVolumeResponse.fsync and SubscribeMetadataResponse.flushed_ts_ns landed without it. Copy them over; no behaviour change. Claude-Session: https://claude.ai/code/session_01Fx1Hx8RqsJqHpbfbgTf4WJ * filer: route exclusive and conditional creates to the entry's ring owner CreateEntry with o_excl is a FindEntry-then-Insert. The per-path lock added for it makes that atomic only on the filer running it, and the store's insert is an upsert on every backend, so two filers both pass the existence check and both report success. mkdir(2) then succeeds twice for the same path. The same hole sits under the condition precondition, whose comment already told callers to route the key's writes to the owner filer themselves. Do it on the server instead, with the mechanism ObjectTransaction already uses: resolve the entry's ring owner and forward one hop, bounded by is_moved. The ring's membership comes from the master, so it tolerates a stale view and reassigns when a filer dies, neither of which a client's configured filer list can do. Every creator gets this — mount, S3, the filer's own HTTP surface, the Java client — not only the ones that opted in. Plain creates are upserts whoever applies them, so they stay local and pay nothing. The route key shares the S3 gateway's namespace so an object's ObjectTransaction and its CreateEntry land on the same filer's per-path lock. Claude-Session: https://claude.ai/code/session_01Fx1Hx8RqsJqHpbfbgTf4WJ	2026-09-02 19:56:11 -07:00
Chris Lu	938a15eb98	filer: keep a moved key on its prior owner while the ring settles (#11108) ... ObjectTransaction forwards to the ring owner so one filer's per-path lock arbitrates every writer of a key. But a ring change hands the key over before the new owner has rebuilt the locks the prior owner still holds, so for the cooling-off window both can grant it. LockRing.PriorOwner exists for exactly this and nothing consulted it. Route to the prior owner while that window is open. LockRing.WriteOwner resolves prior-else-current under one read lock, so the pair cannot come from different rings and name the same filer twice. An unreachable owner fails the request rather than falling back to the current one. gRPC reports a response lost in transit as Unavailable, indistinguishable from a request the owner never saw, so re-sending elsewhere could re-apply what the owner already committed; and an owner unreachable from here may be partitioned rather than down, still serving the key to everyone else — which is the split brain the routing exists to prevent. The window is bounded: once it closes the ring hands the key to its new owner. The owner resolution and the forward move into writeOwner/forwardToWriteOwner so the next routed RPC reuses them rather than copying the block. Claude-Session: https://claude.ai/code/session_01Fx1Hx8RqsJqHpbfbgTf4WJ	2026-09-02 19:40:53 -07:00
Chris Luandbruce-zzz	0f05957bc4	filer: self-heal chunk manifest reads when volume locations go stale (#11107) ... * filer: self-heal fetchWholeChunk on stale volume locations Upstream #10156/#10800 wired cache invalidation into the buffer-based read paths, but manifest resolution still goes through fetchWholeChunk, which returns the raw error on failure. When cached volume locations are stale (volume tiered to remote storage, server rolled), resolving a large multipart file fails permanently even though other locations are healthy. Thread the ChunkGroup's cacheInvalidator through ResolveChunkManifest / ResolveOneChunkManifest / fetchWholeChunk, and on failure invalidate, re-lookup and retry once via the existing retryFetchWithFreshLocations helper. The streaming bytesBuffer is reset before the retry so partial bytes from the failed attempt cannot corrupt the manifest proto.Unmarshal. Non-mount callers pass nil and keep their semantics. * filer: move the manifest self-heal tests in with the other manifest tests Also make the stale server stream a prefix and then abort mid-body, which is what actually leaves partial bytes in the buffer: an HTTP error status returns before ReadUrlAsStream ever calls the writer, so a 500 never exercised the Reset the tests claimed to cover. Claude-Session: https://claude.ai/code/session_01FK3oGC5ZVeJYvNBWgb9JUD * filer: keep the cached volume locations when a manifest read is cancelled A cancelled or timed-out read says nothing about where the volume lives, so dropping the location and going back to the master only costs the next reader a round trip. PrepareStreamContentWithThrottler already guards its self-heal this way. The guard also goes inside retryFetchWithFreshLocations, since the caller can be cancelled between its own check and the invalidation, and that covers the reader cache and prefetch paths too. fetchWholeChunk returns the context error rather than the stream failure it provoked, and ResolveOneChunkManifest wraps with %w so errors.Is still sees it. That matters even where no invalidator is passed: volume.fsck resolves manifests with nil and tells its own abort from a corrupt manifest that way, so the cancellation check sits ahead of the nil-invalidator return. Claude-Session: https://claude.ai/code/session_01FK3oGC5ZVeJYvNBWgb9JUD * filer: self-heal manifest reads on the filer and s3 paths too Every caller that already holds the location cache backing its lookup function can hand it over: the filer's read, copy and deletion paths and the log cache have the MasterClient right there, and s3api has the FilerClient. MinusChunks takes one for the same reason, since the deletion path resolves manifests through it. Only the shell tools and the replication sinks, whose lookup functions cache privately with nothing to invalidate, keep passing nil. Claude-Session: https://claude.ai/code/session_01FK3oGC5ZVeJYvNBWgb9JUD --------- Co-authored-by: bruce-zzz <bruce.zou@hhy-data.com>	2026-09-02 17:43:46 -07:00
Chris Lu	cd064f6eef	shell: clean up the target copy when a merge upload fails (#11104) ... * shell: clean up the target copy when a merge upload fails A replicated write commits the needle to the local volume before it fans out to the other replicas, so an upload that reports failure can still have left a copy on the target. fs.mergeVolumes printed "failed to move" and carried on, so that copy stayed behind forever: the filer is never re-pointed at it, and nothing else knows it exists. One sick replica orphans roughly half the chunks of a merge, two thirds with three copies, since the entry node is picked at random from the replicas and the replica upload uses MaxAttempts 1. A volume with a single copy has no such window: the write is one local append that either succeeds or leaves nothing. Delete the needle we may have written before continuing. The source side already did exactly this, so deleteMovedSourceNeedles is renamed to deleteOrphanedNeedles and reused for both ends. Claude-Session: https://claude.ai/code/session_01XjiMGK72F4Gs3yWNJhnVjy * shell: verify the cookie before deleting a merge target needle The target cleanup deletes a needle an upload may or may not have written, and BatchDelete matches on the needle id alone. Needle ids come from one global sequence, so normally nothing else can hold that id — but a volume restored from elsewhere, or one written either side of a master sequence reset, can, and then a failed move deletes a live needle out from under its filer entry. Have the volume server verify the cookie for those. Source needles keep deleting by id: they are the ones the filer just pointed at, matching every other filer-driven delete. Claude-Session: https://claude.ai/code/session_01XjiMGK72F4Gs3yWNJhnVjy * shell: delete the target copies an abandoned manifest rewrite leaves rewriteManifestChunk moves sub-chunks one at a time and only then uploads the rewritten manifest. Every error after the first successful move — a nested rewrite failing, the marshal, the manifest upload — returned without touching the copies already written to the target volumes. The filer keeps pointing at the old manifest, so those copies orphan, one per sub-chunk moved so far. Track them alongside the sources and delete them on the way out. Nested rewrites hand theirs up so an outer failure clears the whole subtree. A failed UpdateEntry deliberately still leaks its copies: that error can also mean the filer applied the update and lost the response, and deleting there would turn a leak into data loss. Claude-Session: https://claude.ai/code/session_01XjiMGK72F4Gs3yWNJhnVjy * shell: give the plan its room back when a manifest rewrite is abandoned allocate reserves plannedSize against the chosen target for every move a multi-target source makes, and release hands it back when the move fails. Abandoning a manifest rewrite now deletes the copies that did land, so those reservations stopped matching anything on disk: the plan kept counting bytes that are gone and refused later chunks with "no target volume has room". Release them alongside the delete. Nested rewrites hand theirs up so an outer failure unwinds the whole subtree's accounting. Claude-Session: https://claude.ai/code/session_01XjiMGK72F4Gs3yWNJhnVjy	2026-09-02 17:32:11 -07:00
Dmitriy PavlovandChris Lu	1d0b97f4c6	avro: field time.Time <> iceberg.date (#11091) ... * iceberg: normalize foreign day partitions during manifest rewrite * test: cover manifest rewrite with foreign day partitions * iceberg: restore every foreign partition value, not just day transforms iceberg-go takes a partition field's logical type from the last branch of its Avro union, so a writer that spells an optional partition [<type>, null] rather than [null, <type>] leaves the value as whatever the Avro decoder produced. A day or date partition then arrives as a time.Time the manifest writer cannot encode, and a time partition is worse: time.Duration converts to int64 nanoseconds and silently records the wrong value. ReadManifest sits next to ReadManifestList, the other shim for what foreign writers put on the wire, and converts each partition value back to the Iceberg representation for its field type. Claude-Session: https://claude.ai/code/session_01FdQyRuWF9SuCnPn21iH9yR * iceberg: read manifests that carry partition values through the shim Compaction, delete rewrite and their detection passes read entries and write the same partition values back into new manifests, so they fail on a foreign day partition exactly as manifest rewrite does. Where filters see it too: literalMatchesActual falls through to fmt.Sprint, so a time.Time renders as a timestamp and never matches the day the user asked for. The two remaining readers, orphan collection and the admin preview, only look at file paths and stay on iceberg.ReadManifest. Claude-Session: https://claude.ai/code/session_01FdQyRuWF9SuCnPn21iH9yR * iceberg: convert partition values before the writer rebinds logical types Dimonyga checked the manifests of a live Doris table: every input spells the partition union null-first, with the date logical type present, so the union ordering is not what breaks the merge. The conversion is lazy. iceberg-go converts what the Avro decoder returned on the first Partition() call, using the logical types read from the manifest being parsed, and ManifestWriter.addEntry rebinds them to the manifest it is about to write before it makes that call. A day partition is where the two disagree -- iceberg-go's day transform reports an int32 result type, so the manifest it writes carries no date logical type at all -- and an entry nobody looked at in between converts against that and keeps its time.Time. That is why only rewrite_manifests failed: compaction and delete rewrite group entries by partitionKey(df.Partition()) first, which converts them, and a where filter does the same. Reading every entry's partition here converts them all while the manifest's own logical types are still in place. Claude-Session: https://claude.ai/code/session_01FdQyRuWF9SuCnPn21iH9yR --------- Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-09-02 17:24:23 -07:00
3 people	7620e96171	expose whether a volume replica is backed by remote storage, and prefer local replicas (#11105) ... * expose whether a volume replica is backed by remote storage Volume locations returned by lookups do not indicate whether a replica has been tiered to remote storage. Readers cannot distinguish a local replica from a remote-backed one, so they may hit a remote-backed replica first even when a local replica is available. Add DataInRemote to the lookup location message, populate it from the master's volume info, and carry it through the wdclient vid map so clients can prefer local replicas when resolving chunk locations. * wdclient: prefer local volume replicas over remote-tier replicas on lookup LookupFileIdWithFallback (and the publicUrl variant in FilerClient) didn't honor the DataInRemote flag when shuffling URLs, so the DataInRemote patch only took effect in LookupVolumeServerUrl. Apply the same ReorderToFront(localUrls) to sameDcUrls/otherDcUrls so non-remote replicas stay at the front, matching the existing vidMap convention. * wdclient: propagate DataInRemote across tier transitions on existing replicas When a volume is tiered to remote storage or a remote-backed replica is restored locally, the cached DataInRemote on the same volume-server URL stayed at its old value because two pieces of state never updated: * master_grpc_server.go only split newVolumes and (already-tracked) volumes into NewVids vs RemoteVids. ChangedVolumes went straight to NewVids, so the broadcast announced the re-classified volume as a fresh arrival and the client had no way to tell whether its existing cache was stale. * vid_map.addLocationToMap early-returned when an entry already had the same URL. A tier transition reports the same URL with DataInRemote flipped, so the cached entry stayed at the old classification. Wire both sides together: ChangedVolumes now go through the same IsRemote split as newVolumes, and addLocationToMap replaces the existing entry in place when the URL matches but DataInRemote has changed. The server reference key only depends on URL/grpc port, so the refcount does not move across the flip. Adds vid_map_remote_transition_test.go covering the local->remote and remote->local paths so the in-place update and the cache-key stability are pinned by tests. * wdclient: prefer local replicas across data-center boundaries The previous local-first ordering hoisted local URLs to the front of each data-center bucket separately, then concatenated same-DC before other-DC. That meant a same-DC remote replica could still be tried before an other-DC local replica even though the local one would answer cheaply. Reorder once across the full candidate list: concatenate same-DC and other-DC first, then ReorderToFront pulls every local replica to the very front while preserving the DC preference inside each tier. Apply the same ordering in all four lookup paths so the cached vidMap, the LookupFileIdWithFallback provider path, FilerClient.GetLookupFileIdFunction (PublicUrl-preferred variant), and the deprecated filer.LookupFn all agree: - weed/wdclient/vid_map.go (LookupVolumeServerUrl) - weed/wdclient/vidmap_client.go (LookupFileIdWithFallback) - weed/wdclient/filer_client.go (LookupFileId) - weed/filer/reader_at.go (LookupFn) Strengthen the existing local-first tests: vidmap_client_localfirst_test now asserts both endpoints are present (not just the local one is first), and slice_test asserts an exact match instead of accepting two orderings. Add TestLookupFileIdWithFallbackGlobalLocalFirst to pin the cross-DC ordering invariant: any local replica (same or other DC) precedes every remote-tier replica; within each tier DC1 precedes DC2. Add docstrings to ToVolumeLocations, ReorderToFront, LookupVolumeServerUrl, LookupFileId, GetVidLocations, GetLocations, LookupFileIdWithFallback, and updateVidMap so the touched lookup paths are described in one place. * topology: broadcast tier transitions on existing replicas When a volume replica is tiered to remote storage or restored locally, the wdclient's cached DataInRemote went stale: every connected client kept preferring a remote-backed replica over a freshly restored local one, or demoted a freshly tiered remote replica. The fix in commit 116982595 routed ChangedVolumes to NewVids/RemoteVids on the master, but ApplyVolumeChanges returned only fresh arrivals and previously servable replicas. An existing replica whose IsRemote() classification flipped was neither, so it never reached the broadcast loop and the wdclient never learned. Make Disk.doAddOrUpdateVolume return a third signal -- tierTransition -- true exactly when an existing replica's IsRemote() flips. ApplyVolumeChanges treats that as an arrival so the existing SendHeartbeat routing loop now sees it. Add a master-side end-to-end test covering local->remote, remote->local, no-op re-reports, and a mixed heartbeat that only announces the tier transition. Also add docstrings to LookupFileId, wdclientLocationsToPb, and LookupVolume where the prior change touched their bodies. * topology: broadcast tier transitions received through full reconciliation The previous commit added tier-transition routing on the ChangedVolumes delta path, but that is not the only way a re-tiered replica reaches the master. After a digest mismatch the volume server resends a full Volumes list, and SyncDataNodeRegistration applies the new IsRemote() classification silently -- the changedVolumes return value was being thrown away. The master therefore never broadcast NewVids/RemoteVids, and a wdclient connected during the recovery kept the stale DataInRemote until it lost contact with the master. Surface the changed set through UpdateVolumes.changedVolumes (now covering both ReadOnly flips and tier flips) and SyncDataNodeRegistration, then route it through NewVids/RemoteVids in SendHeartbeat the same way the delta path already does. Add an end-to-end test for the full reconciliation path. * master: keep an EC volume's locations in the volume lookup The nodes that answer for an EC volume hold shards, not a volume record, so asking them for one fails. Dropping the location on that failure emptied the result and turned every EC read through the master's HTTP lookup and fid redirect into a 404. Treat an absent volume record as a local read and keep the node in the answer. The per-node conversion moves into topologyLocation so the EC case is covered by a test. Claude-Session: https://claude.ai/code/session_01FcSp6quCaxQ1cb3Vx7o9fW * wdclient: replace a tier-flipped location without writing under a reader GetLocations hands back the entry's own slice and the caller walks it after the read lock is dropped, which is why every other mutation here builds a new slice. Writing the flipped replica into the array in place raced LookupVolumeServerUrl, reported by -race. Copy the slice, swap the one element, and publish it. Claude-Session: https://claude.ai/code/session_01FcSp6quCaxQ1cb3Vx7o9fW * master: keep a remote volume on NewVids for older clients Moving remote-tier volumes out of NewVids and into RemoteVids alone is a wire break in the wrong direction. A master upgraded ahead of its filers and mounts -- the usual order -- announces a tiered volume only on a field the older client ignores, so the volume drops out of that client's vid map entirely and reads for it fail. Announce every volume on NewVids and repeat the remote-tier subset on RemoteVids, so a new client still learns the tier and an old one keeps the location. The routing moves into announceVolume, which the heartbeat paths and their tests now share instead of each restating it. On the client, RemoteVids no longer needs a second write per volume: the tier is settled before anything is added. Claude-Session: https://claude.ai/code/session_01FcSp6quCaxQ1cb3Vx7o9fW * topology: split the volume snapshot by tier without copying the records ToVolumeLocations runs on every KeepConnected, so a filer or mount connecting made the master allocate a full VolumeInfo per volume per node just to read four bytes of id off each one. AppendVolumeIds exists to avoid exactly that. Extend it to fill the remote-tier list alongside the full one, and use it again in the snapshot. Claude-Session: https://claude.ai/code/session_01FcSp6quCaxQ1cb3Vx7o9fW * wdclient: keep the data-center preference ahead of the local-first ordering Hoisting every local replica to the very front puts an other-DC local read ahead of a same-DC remote one. When the remote tier sits in the same region as the replicas -- the common arrangement -- that trades an in-region GET for a WAN round trip and costs more than the remote read it avoids. Reorder inside each data-center bucket instead, so local still wins among equals and the data-center preference still wins overall. Claude-Session: https://claude.ai/code/session_01FcSp6quCaxQ1cb3Vx7o9fW * operation: pick the read replica from one list The local-preferring lookup built a list of local URLs and then branched on whether it was empty, duplicating the random pick. Fall back by filling the same list with every replica instead. Claude-Session: https://claude.ai/code/session_01FcSp6quCaxQ1cb3Vx7o9fW --------- Co-authored-by: Bruce Zou <gift_secondst@msn.com> Co-authored-by: bruce-zzz <bruce.zou@hhy-data.com>	2026-09-02 16:12:46 -07:00
3 people	96242a2be2	s3: ListParts on a completed or unknown upload answers NoSuchUpload (#11081) ... * s3: ListParts on a completed or unknown upload answers NoSuchUpload complete/abort delete the .uploads/<id> directory, but most filer stores list a missing directory as empty rather than erroring, so listObjectParts answered 200 with an empty Parts list for an upload that no longer exists -- the same response an open upload with no parts yet gets. AWS (and Ceph/RGW, MinIO) answer NoSuchUpload, and clients lean on that: tusd derives the resumable upload offset from the ListParts part sizes, so every completed upload read back as zero bytes received. Probe the upload record before listing, the way completeMultipartUpload already does: not found, or a directory a late part write resurrected without the destination key, answers NoSuchUpload. An open upload with no parts keeps answering 200 with an empty list. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GGQEfVUvoATtwRCR8oC2jG * s3: have the ListParts test filer refuse a directory it was not asked about The fake answered the upload lookup on the name alone and the part listing regardless of directory, so a wrong genUploadsFolder or upload-id suffix would still have passed. Both calls now refuse any other directory with an Internal error, which surfaces as ErrInternalError rather than the NoSuchUpload the tests expect. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01StsRz9wbu5dUCMGbFgPoRM --------- Co-authored-by: tomislavcivcija <9787657+tomislavcivcija@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-09-02 12:05:16 -07:00
Chris Lu	eed5e8cdf6	s3: return the multipart object checksum in the CompleteMultipartUpload response (#11101) ... * s3: return the multipart object checksum in the CompleteMultipartUpload body S3 carries the flexible-checksum members of CompleteMultipartUploadResult in the XML body, not in response headers, so every SDK read back an empty checksum from an upload that asked for one. Claude-Session: https://claude.ai/code/session_01Huux1uh7JxAbf8yypMYrMk * s3: echo the checksum algorithm and type from CreateMultipartUpload The upload directory already records both, but the response dropped them, so a client could not confirm which checksum its parts had to carry. Claude-Session: https://claude.ai/code/session_01Huux1uh7JxAbf8yypMYrMk * test: multipart upload reports the object checksum it was asked for Covers every algorithm end to end: the create response echoes the algorithm and type, the complete response carries the checksum, and it matches what a later HEAD reports. Claude-Session: https://claude.ai/code/session_01Huux1uh7JxAbf8yypMYrMk	2026-09-02 11:57:45 -07:00
Chris Lu	5f787a25c3	master: survive a volume layout deleted twice (#11098) ... * master: survive a layout deleted twice Two volume servers dropping the last replica of volumes that share a layout both find it empty and both delete it. The loser's lookup misses, and the single-value type assertion on the result crashed the master before the caller could look at the found flag. Claude-Session: https://claude.ai/code/session_01WmX6Rchx298NQksHDXg7sk * master: remove a layout and read it back in one step DeleteVolumeLayout looked the layout up and then deleted it, so two deleters could each release the lookup ownership of the same layout, or one could find nothing to release at all. Have the map hand back what it removed. Claude-Session: https://claude.ai/code/session_01WmX6Rchx298NQksHDXg7sk	2026-09-02 11:50:48 -07:00
Chris Lu	86761cc7d5	filer: keep empty folders that are s3tables catalog entries (#11102) ... * s3tables: build the catalog attribute keys from one shared prefix Every attribute the catalog stores on a bucket, namespace, table or view entry is spelled out with the same literal prefix. Name it once in s3_constants so code outside the package can recognize a catalog entry without repeating the string. Claude-Session: https://claude.ai/code/session_01GfZsc4cyNB2yr6KYLRv9q1 * filer: keep empty folders that are s3tables catalog entries A namespace, table or view is a directory whose extended attributes are the catalog record. Its files can live elsewhere - a rename moves only the catalog pointer and leaves the data at the old path, and a view has no files at all - so an empty one is still a live entry. Drop a table, then rename another table onto that name: the drop queues the old table's folders, the rename recreates the name path, and two minutes later the cleaner deletes it and cascades into the namespace, losing a table the catalog still lists. Claude-Session: https://claude.ai/code/session_01GfZsc4cyNB2yr6KYLRv9q1 * filer: drop a queued cleanup when the folder is created again A cleanup is queued against the folder that was found empty. If that folder is deleted and a new one takes its name, the queue entry outlives the folder it was about and the next pass deletes the replacement. A drop followed by a rename onto the dropped name does exactly this: the name path comes back as a live catalog entry two minutes before the queue is read. Claude-Session: https://claude.ai/code/session_01GfZsc4cyNB2yr6KYLRv9q1	2026-09-02 11:49:08 -07:00
Chris Lu	23a6b8feb5	filer_pb: walk the whole tree when the BFS start path ends in a slash (#11099) ... * filer_pb: build BFS child paths with FullPath.Child A start path with a trailing slash produced "/dir//sub" for every subdirectory, and the filer only trims a trailing slash, so those listings came back empty and the walk stopped after the first level. Claude-Session: https://claude.ai/code/session_01Jp9tXRpBv9gvh8fkaVFqxQ * filer_pb: normalize the BFS start path Entries directly under the start path were reported with the caller's trailing slash, so filer.meta.backup wrote them under a directory the incremental stream never names again. Claude-Session: https://claude.ai/code/session_01Jp9tXRpBv9gvh8fkaVFqxQ	2026-09-02 11:45:22 -07:00
Chris Lu	9ea52db219	s3: validate the version-id header used as a filer path segment (#11097) ... * s3: reject a version-id header that is not a valid path segment putToFiler stored the client-supplied Seaweed-X-Amz-Version-Id header verbatim into object metadata. That value is later read back and used as a filer path component when building the .versions/v_<id> path, so a value containing "/", "\" or ".." could steer retention/legal-hold writes and remote-cache reads outside the object's own bucket tree. Validate the header with isValidVersionID before storing it, the same check the versioned read paths already apply, and reject the request otherwise. Server-set version ids ("null" and generated hex) pass. Claude-Session: https://claude.ai/code/session_011QqNaxZwnpHgMoAZNp3RkY * s3: validate a stored version-id before using it as a path The retention and legal-hold sinks build a .versions/v_<id> path from a version id read back out of object metadata, and the remote-cache path builder does the same from either the request or the stored id, without the isValidVersionID check the other version-id consumers apply. Guard these so a value that is not a valid path segment falls back to the regular / unversioned path instead of steering the write or read out of the bucket tree. Claude-Session: https://claude.ai/code/session_011QqNaxZwnpHgMoAZNp3RkY	2026-09-02 11:36:58 -07:00
Chris Lu	23adeb37e2	s3: check Object Lock on directory-marker keys before bucket deletion (#11096) ... recursivelyCheckLocksWithClient tested EntryHasActiveLock only on non-directory entries, so a directory-marker object (an S3 key ending in "/") that carries retention or a legal hold was recursed into but never lock-checked. DeleteBucket then saw no locks and removed the bucket, destroying an object under active Object Lock along with the rest of the bucket. DeleteObject already enforces the lock on the same key, so the two paths disagreed. Check the directory entry for an active lock before recursing. Claude-Session: https://claude.ai/code/session_011QqNaxZwnpHgMoAZNp3RkY	2026-09-02 11:35:19 -07:00
Chris Lu	3cdfe648eb	sftp: reject an empty password (#11095) ... ValidatePassword compared the stored and supplied passwords with subtle.ConstantTimeCompare, which returns 1 for two zero-length slices. A user provisioned for public-key-only auth has an empty stored password, so an empty supplied password authenticated as that user whenever "password" was among the enabled auth methods (the default). Treat an empty stored or supplied password as a non-match. Claude-Session: https://claude.ai/code/session_011QqNaxZwnpHgMoAZNp3RkY	2026-09-02 11:33:25 -07:00
Chris Lu	398277a15d	mini: expose -volume.max (#11100) ... mini hardcoded the per-directory volume limit to 0, so the volume server always auto-sized it as free disk space divided by the volume size. That sizing reserves a whole volume size for every writable volume, so a workload spreading small objects over many buckets runs out of slots long before the disk fills and assign starts failing with "no free volumes left". Same name and semantics as the flag weed server already carries, and still 0 (auto) by default. Claude-Session: https://claude.ai/code/session_01BiQLeBvZLzG8XitjiypDKu	2026-09-02 09:46:27 -07:00
Chris Lu	8f2daad338	topology: mirror the writable volume list in a set (#11076) ... Membership was a linear scan over a slice, and ensureCorrectWritables runs it for every volume on every heartbeat, so the master's steady-state cost per volume server is quadratic in that server's volume count. BenchmarkSyncDataNodeRegistration, median of 3: 1000 volumes 565.7us -> 535.0us -5.4% 100000 volumes 1.665s -> 55.3ms -96.7% Allocations are unchanged at both sizes, so the difference is the scan. Claude-Session: https://claude.ai/code/session_01P3pE6J2UPFp6G3ksfMV4s1	2026-09-01 14:21:26 -07:00
Chris Lu	86a189ff80	mount: keep metadata operations working on a removed open directory (#11073) ... * mount: remember the entry of a directory removed while still referenced A directory removed while a descriptor is open on it keeps its inode until the kernel's final forget, but unlike a file it has no handle to live on through: OpenDir hands out only a listing cursor. Keep the last-known entry in memory, keyed by inode, from rmdir until that forget. Claude-Session: https://claude.ai/code/session_01GYqLENjZzbV5hgt4L8cSAK * mount: serve metadata ops on a removed open directory from its remembered entry fchmod, futimens, and the f*xattr calls on a descriptor whose directory was removed failed with ENOENT: maybeReadEntry resolved the inode to a path, and rmdir had already dropped it. Fall back to the remembered entry the same way an unlinked file falls back to its open handle. Mutations publish a changed copy back rather than editing in place, so a concurrent reader never sees a half-applied change, and the empty path keeps nlink 0 in every reply. Claude-Session: https://claude.ai/code/session_01GYqLENjZzbV5hgt4L8cSAK * mount: stash the entry the delete itself returned, not an earlier snapshot A chmod landing between Rmdir's entry load and the delete RPC would be resurrected pre-change: the remembered entry was the earlier local snapshot. The filer serializes the delete against updates under the path lock and hands the entry back in the delete event, so prefer that, keeping the local load for the sticky-bit check and as fallback when no event comes back. Claude-Session: https://claude.ai/code/session_01GYqLENjZzbV5hgt4L8cSAK * mount: drop a remembered entry whose insert lost to the final forget The forget's cleanup runs between RemovePath and the insert when the kernel evicts the inode concurrently, finds nothing, and the entry would sit in the map for the life of the mount. Re-check the inode after inserting and take the entry back out; every interleaving now ends with the map empty. Claude-Session: https://claude.ai/code/session_01GYqLENjZzbV5hgt4L8cSAK * mount: insert the remembered entry under the inode table lock The post-insert HasInode re-check could be fooled by inode number reuse: a lookup landing between the forget and the check makes the number look alive and the stale entry stays, keyed to someone else's inode. Do not check after the fact — RemovePath now runs the retention callback inside its critical section, where the forget that releases under the same lock cannot have run and cannot be missed. Publishes need no such fence: their open descriptor keeps the kernel from issuing the final forget in the first place. Claude-Session: https://claude.ai/code/session_01GYqLENjZzbV5hgt4L8cSAK	2026-09-01 13:21:02 -07:00
Chris Lu	77a9dd4b9e	s3: route per-key object authorization through a shared helper (#11072) ... * s3: share the per-key object authorization across copy and delete AuthorizeCopySource and AuthorizeObjectDelete both authorize a key the request URL does not name by evaluating the bucket policy and IAM against a synthetic per-key request; only the method and action differed. Extract that into authorizeObjectKeyAction and make the two callers thin wrappers. No behavior change. Claude-Session: https://claude.ai/code/session_01Qo7p6VsoWxMo8816ogJFk5 * s3: route POST Object uploads through the shared object authorization POST Object uploads (presigned-POST / HTML form) authorized the write with only the coarse per-identity Write action, unlike the other write paths which also check the resolved object against the bucket policy and IAM. Route POST through authorizeObjectKeyAction via a new AuthorizeObjectWrite so it is authorized like the equivalent PUT. Claude-Session: https://claude.ai/code/session_01Qo7p6VsoWxMo8816ogJFk5 * s3: test POST Object per-key authorization Drives a signed POST upload and checks the per-key authorization decision for a denied, permitted, and admin caller. Claude-Session: https://claude.ai/code/session_01Qo7p6VsoWxMo8816ogJFk5	2026-09-01 13:20:20 -07:00
Chris Lu	34f5442e9b	s3api: push the listing prefix down to the filer in ListObjectVersions (#11070) ... The version walk listed every directory with no prefix, transferring all 1024-entry batches over gRPC and filtering gateway-side - and kept paging past the point where names can no longer match. On wide directories (many sibling orgs/jobs next to the requested prefix) that is most of the transfer, decode, and CPU cost of every page. Derive the next path component of the requested prefix per directory level and hand it to the filer listing. A name holds no slash, so a directory whose name does not start with the component cannot contain a matching key and a file that does not cannot be one; stores with native prefixed listing (sql, leveldb) turn this into a range scan and stop the stream at the end of the prefix zone. Claude-Session: https://claude.ai/code/session_01FquvGtTD2zA3uMZGQHAuV4	2026-09-01 10:37:52 -07:00
Chris Lu	0ba21174bf	volume: an already-deleted EC needle is not a delete failure (#11071) ... * volume: an already-deleted EC needle is not a delete failure Deleting a needle that is already gone is what the caller asked for, and the non-EC paths have always said so: BatchDelete reports StatusNotModified when DeleteVolumeNeedle finds nothing to do, and DeleteHandler answers 404 from its ReadVolumeNeedle pre-check. The EC branches had no such case, so ErrorDeleted fell through to a generic failure -- 500 from both, and DeleteHandler also counted it in VolumeServerFileWriteFailures, inflating a failure metric on a replayed or duplicated delete. The filer already tolerates this by string-matching "already deleted" on the result, which leaves an error message load-bearing; the status is now right at the source instead. Claude-Session: https://claude.ai/code/session_01P3pE6J2UPFp6G3ksfMV4s1 * volume: close the EC fixture's disk location Close stops the location's disk-space goroutine and releases the mounted EC volume's file handles, which otherwise live until the test binary exits. Claude-Session: https://claude.ai/code/session_01P3pE6J2UPFp6G3ksfMV4s1	2026-09-01 10:37:25 -07:00
Chris Lu	81ca5cb6c6	s3api: drop two redundant filer round-trips per listed version entry (#11068) ... * s3api: drop two redundant filer round-trips per listed version entry ListObjectVersions paid two avoidable getEntry calls while walking a bucket, both re-fetching data the walk already held: - getObjectVersionList re-read the .versions directory entry that every caller had just received from listing the parent directory (or from its own sibling probe). Pass the entry down instead: one RPC saved per object listed. - getObjectOwnerFromVersion, on a version with no stamped owner, re-fetched the same version entry its OwnerID had been extracted from. The refetch cannot answer differently, so data written before owners were stamped cost one futile RPC per listed version, forever. All round-trips on this path are sequential, so on large versioned buckets (Veeam-style workloads) they add up to a visible share of per-page latency and gateway CPU. Claude-Session: https://claude.ai/code/session_01FquvGtTD2zA3uMZGQHAuV4 * s3api: treat a nil .versions entry as an empty version list filer_pb.GetEntry's contract permits (nil, nil) for an absent entry, and the old internal lookup answered that case with an empty list. Keep that answer now that the entry arrives from the caller. Claude-Session: https://claude.ai/code/session_01FquvGtTD2zA3uMZGQHAuV4	2026-09-01 10:29:52 -07:00
Chris Lu	4c9cbf72bc	s3api: stop retrying a definitive NotFound in getLatestObjectVersion (#11067) ... The .versions lookup retried every error through the full backoff ladder, so a missing key spent 12.7s (8 attempts, 100ms..6.4s) before the pre-versioning fallback could answer. NotFound is an answer, not a transient failure: gate the retries on isRetryableFilerErr, the same classifier retryFilerOp already uses, which also stops retrying for callers whose context is canceled or past its deadline. GetObject already treats NotFound on .versions/ as definitive; this brings the retention/tagging/ACL/attributes/delete/copy paths that go through getLatestObjectVersion in line with it. Claude-Session: https://claude.ai/code/session_01FquvGtTD2zA3uMZGQHAuV4	2026-09-01 10:11:09 -07:00
Chris Lu	2ef0e60aeb	filer.sync: export replication lag, event counters, and in-flight jobs (#11069) ... * filer.sync: count received, processed, and failed events and export in-flight jobs The metadata processor admits at most -concurrency jobs and blocks the subscription stream past that, so the backlog lives in the source filer's metadata log and cannot be counted here. What can be measured honestly: events read off the stream, replication outcomes, and worker saturation. in_flight_jobs pinned at the concurrency limit means the sync itself is the bottleneck; near zero means it is caught up or starved by the source. Claude-Session: https://claude.ai/code/session_01VGUXS7kdqiaXenTYVCg6vw * filer.sync: export replication lag in seconds Lag is now minus the freshest of the processed watermark and the last idle heartbeat: the watermark stops at the last real event, so a quiet caught-up stream would otherwise show phantom lag. A ticker drives the gauge because the offset callback only fires while events flow and freezes exactly when the workers are saturated. Until the first event or heartbeat the gauge stays unset rather than reporting lag against a zero or stale resume offset. Claude-Session: https://claude.ai/code/session_01VGUXS7kdqiaXenTYVCg6vw * filer.sync: track replicated data sizes alongside event counts An event count hides that 32 in-flight jobs can be 32 KB or 300 GB. Byte counters mirror the event counters, and in_flight_bytes pairs with in_flight_jobs. An event's size is the chunk delta - new chunks the old entry does not already have - so deletes, renames, and attribute-only updates count zero and byte rates reflect data movement, not metadata churn. Claude-Session: https://claude.ai/code/session_01VGUXS7kdqiaXenTYVCg6vw * grafana: chart the new filer.sync metrics The lag panel reads lag_seconds directly instead of deriving it from sync_offset, and the sync row gains event rate, throughput, and the in-flight jobs and bytes gauges. Claude-Session: https://claude.ai/code/session_01VGUXS7kdqiaXenTYVCg6vw * filer.sync: a pinned failure keeps showing as lag An idle heartbeat means the stream is consumed, not that every event replicated. While a permanent failure pins the watermark, letting the heartbeat advance lag_seconds or the sync_offset gauge would report a caught-up stream with an unreplicated event in it, so both now ignore heartbeats until a restart replays the failure. Claude-Session: https://claude.ai/code/session_01VGUXS7kdqiaXenTYVCg6vw * filer.sync: in-flight gauges survive subscription retries A subscription retry builds a new processor sharing the gauge children while the old processor's jobs may still be draining, so setting the gauge from either side's local count clobbers the other. Each job now increments and decrements for itself, keeping the total truthful across generations. Claude-Session: https://claude.ai/code/session_01VGUXS7kdqiaXenTYVCg6vw	2026-09-01 10:10:30 -07:00
Chris Lu	9d5525e747	master: keep periodic volume growth to the data centers a layout lives in (#11060) ... * master: keep the periodic growth scan to data centers hosting the layout The rack-aware scan planned growth for every data center in the topology, so a collection pinned to one DC (fs.configure -dataCenter) sprouted volumes in all the others within one scan cycle. Plan only for data centers already hosting the layout's volumes; an empty DC gets its volumes from the DC-constrained assign that first asks for them. The lastGrowCount divisor likewise counts only the racks the scan can plan for. Claude-Session: https://claude.ai/code/session_01J22TVTyoCMzdHyJirsLMG5 * master: pin periodic must-grow growth to a single-DC layout's data center The must-grow and crowded paths of the periodic loop grow with no DataCenter, so even with the scan fixed a pinned collection's volumes could still land in any DC once lastGrowCount demands more writables. Stamp the grow request with the layout's data center when its volumes all live in one; layouts spanning DCs keep unconstrained growth. Claude-Session: https://claude.ai/code/session_01J22TVTyoCMzdHyJirsLMG5 * master: never pin growth of a cross-DC-replicated layout A layout whose replication spans data centers cannot legitimately live in one DC; observing a single hosting DC there means the other DCs are down. Do not encode that outage as a placement constraint. Claude-Session: https://claude.ai/code/session_01J22TVTyoCMzdHyJirsLMG5 * master: bound the hosting-DC walk by the answer it needs listVolumeDataCenters walked every location of the layout under accessLock — ~190ms for a million volumes, twice per layout per cycle, stalling assigns behind the read lock. Stop once enough distinct DCs answer the caller's question: two for the single-DC check, the topology's DC count for the scan. A spanning million-volume layout now finishes in microseconds; only a layout truly confined to fewer DCs still pays a full walk, the same cost class as the under-replication count this loop already takes each cycle. Claude-Session: https://claude.ai/code/session_01J22TVTyoCMzdHyJirsLMG5	2026-09-01 00:40:50 -07:00
Chris Lu	8873f9775c	shell: reset noLock in the admin script dispatcher too (#11059) ... Three dispatchers reuse one CommandEnv: the interactive shell, the master's maintenance script runner, and the plugin worker's admin script handler. The first two were fixed; this is the third. It changes nothing today -- ForceNoLock already exempts this path -- so it is here to keep the rule the same everywhere rather than resting on that exemption staying in place.	2026-08-31 23:14:37 -07:00
Chris Lu	1996c6aec6	volume: open volume files with O_NOATIME (#11055) ... * volume server: open volume files with O_NOATIME Nothing reads the atime of .dat, .idx, .sdx, or EC files, but every needle read still dirtied the inode: even relatime writes atime on the first read after each write, so an actively written volume paid a metadata write per read/write cycle, and strictatime mounts paid one per read. Open the serving handles with O_NOATIME, falling back to a plain open when the file belongs to another owner (EPERM). Claude-Session: https://claude.ai/code/session_015uVY4diBgEn3VYQoc2eMuD * seaweed-volume: mirror the O_NOATIME volume file opens Same change as the Go volume server: serving handles for .dat, .idx, .sdx, .ecx, .ecj, and shard files open with O_NOATIME on Linux, with a plain-open fallback on EPERM. Claude-Session: https://claude.ai/code/session_015uVY4diBgEn3VYQoc2eMuD * route the tier-down and recreate .dat opens through the no-atime helper Review caught the Rust tier-down swap opening the local .dat directly. The Go swapToLocalDatBackend and the zero-length read-only .dat recreate in maybeWriteSuperBlock had the same gap: all three install long-lived serving handles. Claude-Session: https://claude.ai/code/session_015uVY4diBgEn3VYQoc2eMuD	2026-08-31 21:41:50 -07:00
Chris Lu	0c59c0fb05	master: scope the startup capacity shed to a truly empty topology (#11058) ... * master: scope the startup capacity shed to a truly empty topology The retryable "no volume server capacity registered yet" shed checked capacity for the requested disk type, so a cluster serving only other media -- where that capacity will never register -- shed every assign until the client's deadline instead of failing fast. An unsteered write to such a cluster hung for its full HTTP deadline and surfaced "context deadline exceeded" in place of "No writable volumes". Shed only while no disk type has any registered capacity, and name the unserved medium in the fast failure. Claude-Session: https://claude.ai/code/session_01TF7FQghfDkpdoZgakTMX4R * master: name the unserved medium for every fail-fast caller The diagnostic sat in the growth-initiator block, so a follower joining an in-flight growth and a growth-disabled master failed the same way with only the generic pick error. Wrap at the fail-fast break instead, which every caller reaches, and cover all three paths in the test. Claude-Session: https://claude.ai/code/session_01TF7FQghfDkpdoZgakTMX4R	2026-08-31 21:29:27 -07:00
github-actions[bot]	79b8720213	4.45	2026-08-31 23:21:01 +00:00
Chris Lu	0f4a7d0803	shell: keep noLock to the command that set it (#11052) ... noLock says "this invocation changes nothing" -- volume.balance, volume.move, volume.copy, volume.merge and volume.fix.replication all set it for a dry run, and none clears it. The CommandEnv is created once and reused by both dispatchers, the interactive shell and the master's maintenance script runner, so a simulation left every later command unlocked: volume.balance -noLock # changes nothing volume.move ... # mutates, and skips its lock Reset before dispatch in both, where the invocation begins. forceNoLock is untouched: that is set once, deliberately, for a trusted path.	2026-08-31 11:36:23 -07:00
Chris Lu	87474c2f21	s3: let attached policies authorize CreateBucket (#11049) ... * s3: resolve admin bucket subresources to their specific S3 actions Encryption, requestPayment, publicAccessBlock and ownershipControls requests reached the policy engines as s3:*, so only a policy granting all of s3 could authorize them. Map each subresource to its AWS action, with DELETE sharing the PUT permission as AWS does. Claude-Session: https://claude.ai/code/session_01XH7iM88ZqWMEvsLB8tkWPQ * s3: authorize CreateBucket as s3:CreateBucket in the policy engine A plain bucket-level PUT is registered with ACTION_ADMIN, which resolved to s3:*, so no attached policy short of s3:* could match it. Federated sessions whose policy explicitly allowed s3:CreateBucket were always denied while the same policy worked for object operations. Resolve it to s3:CreateBucket, like DeleteBucket already resolves. Claude-Session: https://claude.ai/code/session_01XH7iM88ZqWMEvsLB8tkWPQ	2026-08-31 10:24:22 -07:00
Chris Lu	b8049bc633	shell: keep fs.mergeVolumes from spinning past the finished moves (#11050) ... * filer_pb: walk a re-delivered directory only once in TraverseBfs A directory handed back twice by a listing (a page-boundary race with concurrent renames, or a store whose ordering misbehaves) was enqueued twice; the second walk re-lists the same subtree and can keep the traversal from ever terminating. Claude-Session: https://claude.ai/code/session_01XH7iM88ZqWMEvsLB8tkWPQ * filer_pb: fail a directory listing whose pagination stops advancing A full page ending on the very name the cursor started from re-fetches the same page forever; a store whose listing order does not advance past the cursor turns any full-directory read into a silent infinite loop. Return an error naming the stuck cursor instead. Claude-Session: https://claude.ai/code/session_01XH7iM88ZqWMEvsLB8tkWPQ * shell: skip foreign-collection manifests in fs.mergeVolumes Every manifest chunk in the namespace was resolved, downloading its manifest needle, even when the merge plan only touches one collection. Sub-chunks live in the manifest's own collection, so a manifest on a volume outside the plan's collections cannot reference a source volume; skip it and spare a cluster-wide download pass that looks like a hang after the real moves finish. Claude-Session: https://claude.ai/code/session_01XH7iM88ZqWMEvsLB8tkWPQ	2026-08-31 10:23:35 -07:00
Chris Lu	23d424248d	sts: session duration no longer clamped to the web identity token exp (#11048) ... * sts: session duration no longer clamped to the web identity token exp The assumed-role session lifetime is governed by DurationSeconds and the configured tokenDuration/maxSessionLength, matching AWS. Clamping to the already-verified token's exp made short-lived id_tokens (GitLab issues ~2-minute ones) yield unusable sessions regardless of configuration. Claude-Session: https://claude.ai/code/session_01XH7iM88ZqWMEvsLB8tkWPQ * sts: cover session duration against short-lived web identity tokens The mock OIDC provider now carries the token exp through to the identity like the real provider, so the integration test would catch the clamp. Claude-Session: https://claude.ai/code/session_01XH7iM88ZqWMEvsLB8tkWPQ	2026-08-31 10:22:06 -07:00
Guang Jiong LouandChris Lu	f740210235	get volume topology info without volume details (#11036) ... * get volume topology info without volume details Signed-off-by: lou <alex1988@outlook.com> * master: rename VolumeListRequest.without_volumes to topology_only The field shapes the reply rather than selecting volumes, and it leaves out the ec shards too, which the old name denied. Match the message's *_only style and say what a master that predates the field does with it. Claude-Session: https://claude.ai/code/session_01QHnaNRgxnjzZsiz7WTFML5 * master: refuse topology_only combined with a volume selector A topology_only request that also names a collection or volume ids contradicts itself, and answering either half in silence surprises the caller. Answer InvalidArgument from both VolumeList and its stream, before the stream sends its header. Claude-Session: https://claude.ai/code/session_01QHnaNRgxnjzZsiz7WTFML5 --------- Signed-off-by: lou <alex1988@outlook.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-08-31 09:45:22 -07:00
Chris Lu	32df246a81	mq: fix idle-cleanup shard deadlock that permanently wedges the broker's topic map (#11051) ... mq: remove emptied topics after the cleanup iteration, not inside it cleanupIdlePartitions called manager.topics.Remove from inside manager.topics.IterCb. IterCb holds the shard's read lock while running the callback, and Remove takes the same shard's write lock, so removing an emptied topic self-deadlocked the cleanup goroutine. The pending writer then blocked every later reader of that shard, permanently hanging ListTopicsInMemory and, for shard-mates, TopicExistsInMemory. On the Kafka gateway this surfaced as flaky e2e consumer-group tests: one minute after any earlier topic went idle, the broker's first 'Removing empty topic' wedged the map, every gateway ListTopics/TopicExists RPC burned its full 5s timeout, Metadata could no longer finish inside kafka-go's 5s coordinator deadline, and consumer groups looped in PreparingRebalance until the test timed out. Collect the emptied topic keys during the iteration and remove them afterwards via RemoveCb, re-checking emptiness under the shard lock so a topic that just gained a partition is kept. Claude-Session: https://claude.ai/code/session_014yA6c8JQcY6MqPXCT13yYA	2026-08-31 09:44:34 -07:00
Chris Lu	d3b8030a69	master: shed assigns retryably until volume servers register capacity (#11032) ... An assign arriving before any volume server has heartbeated saw zero available space and failed outright with a plain error no client retries, so the first write to a fresh bucket answered 500 while the cluster was still starting. Distinguish a topology with no registered capacity from a genuinely full one: fail fast only when registered capacity is exhausted, and shed ResourceExhausted otherwise so the client's retry budget rides out the startup window. Claude-Session: https://claude.ai/code/session_018G9kWFgy8BaBAEkYV3YL9n	2026-08-30 11:08:53 -07:00
Chris Lu	9bafeb6139	ec: refuse to mount a 0-byte shard file when the index has entries (#11030) ... * ec: refuse to mount a 0-byte shard file when the index has entries The startup scan already skips (and eventually deletes) zero-sized shard files as residue of a failed copy, but the mount RPC path opens the file directly with no size check, so an explicit VolumeEcShardsMount over a truncated file registers a size-0 claim. A registered empty shard serves nothing while advertising ownership: with placement pinned to the owning disk, it would keep attracting re-copies to a file that was never valid. The one legitimate 0-byte shard is the empty volume's: encoding a volume with no live needles produces a 0-byte .ecx and 0-byte shards, and that mount must keep working (TestMountEcShards_EmptyEcxMountsSuccessfully). So the gate compares against the index: AddEcVolumeShard (Go) and EcVolume::add_shard (Rust) refuse a 0-byte shard file only when the volume's .ecx has entries. Go's AddEcVolumeShard grows an error return for this; the loader cleans up the refused shard and, when it just created the EcVolume, unregisters that too. The mount loop already collects non-ENOENT failures per disk and keeps scanning, so a sibling disk holding a real copy still wins. Regression tests in both trees: an empty shard beside an index with entries is refused and leaves nothing registered; an empty shard of an empty volume still mounts. Claude-Session: https://claude.ai/code/session_01AWpefvdi4U3HLng18x5CJ9 * ec: release the duplicate shard when a mount retry re-loads it Review follow-up: AddEcVolumeShard keeps the existing shard and reports added=false for a shard this disk already registered, but the loader discarded that result, so every retried LoadEcShard leaked the duplicate it had just opened — an fd and a mount-gauge increment per retry. Release both and return the existing volume. Regression test pins the gauge. Claude-Session: https://claude.ai/code/session_01AWpefvdi4U3HLng18x5CJ9 * ec: close the test DiskLocation instead of only its EC volumes Review follow-up: DiskLocation.Close() also stops the background goroutine NewDiskLocation starts; closeEcVolumes left it running for the rest of the test process. Both uses are this PR's own tests. Claude-Session: https://claude.ai/code/session_01AWpefvdi4U3HLng18x5CJ9 * rust: unregister the just-created EcVolume when its first mount is refused Review follow-up: when the first mount of a volume rejects its shard (e.g. the new 0-byte-beside-nonempty-index refusal), the Rust mount path had already inserted the EcVolume and propagated the error without removing it — a zero-shard registration advertising a mount that serves no data while pinning the .ecx/.ecj descriptors (and, since placement's mounted tier keys off it, steering shard placement at this disk). Remove it on the way out, exactly as the Go loader already does; a volume that already holds shards keeps them (the RPC's first-error-aborts contract). Regression test covers both. Claude-Session: https://claude.ai/code/session_01AWpefvdi4U3HLng18x5CJ9 * rust: skip already mounted shards on a mount retry Review follow-up: EcVolume::add_shard replaces self.shards[id] for a shard the volume already holds, and the mount loop then bumps the ec_shards gauge although the mounted count did not grow — gauge drift on every mount retry, and a serving fd swapped for no reason. Skip shard ids the volume already reports, mirroring Go's AddEcVolumeShard added=false handling. Regression test pins the gauge across a duplicate mount (unique collection label: the gauge is process-global and tests run in parallel). Claude-Session: https://claude.ai/code/session_01AWpefvdi4U3HLng18x5CJ9	2026-08-29 14:19:16 -07:00
Chris Lu	74b520113e	ec: pin auto-selected shard placement to the disk that already owns the shard (#11029) ... * ec: pin auto-selected shard placement to the disk that already owns the shard A multi-disk server legitimately mounts one EC volume on several disks, so FindEcShardTargetLocation's per-volume tiers tie at "mounted" and the free-shard-count tie-break decides — pointing at whichever disk is emptier, not at the disk that already holds the shard being placed. A re-copy of a shard the server already has (a retried ec.balance / ec.rebuild move) then lands on a sibling disk, and both disks register the same (volume, shard id): the shard is reported to the master from two disk ids, and which claimant serves reads or survives a later unmount/delete becomes an accident of Locations order. Add a tier above "mounted": a disk that already claims one of the shard ids being placed wins, ahead of the space filters too — re-copying in place needs no new shard slot, and a genuinely full disk should fail the write rather than silently split the claim. Applied to the Go selector and the VolumeEcShardsCopy auto-select (ReceiveFile refuses mounted EC volumes, so no claim can exist there) and mirrored in the Rust volume server. Claude-Session: https://claude.ai/code/session_01AWpefvdi4U3HLng18x5CJ9 * ec: refuse a copy batch whose shards are already owned by different disks Review follow-up: ownership-aware selection ranks a mixed-owner batch (shard 0 on disk A, shard 2 on disk B — the legitimate multi-disk spread) into one destination, so the copy would still duplicate the losing disk's claim. No production caller sends such a batch (balance moves one shard, rebuild and encode copy shards the target lacks), so fail closed: report every owning disk via Store.EcShardOwnerDisks and refuse the copy with an error naming them, telling the caller to split per shard or pass disk_id. Go and Rust, with unit tests for the owner-reporting contract. Claude-Session: https://claude.ai/code/session_01AWpefvdi4U3HLng18x5CJ9	2026-08-29 12:12:41 -07:00
Lorentz Kinde	83a929720a	fix: Move cluster.LiveLock.generation to first field fix 32bit alignment (#11028) ... On 32-bit systems the generation fields becomes unaligned. Not an issue on 64bit. Discovered on 32bit raspbian, validated by manual patch (in addition to test).	2026-08-29 11:38:16 -07:00
Chris Lu	88c873ecd4	ec: uniform shard block layout (#10932) ... * ec: uniform shard block layout An EC volume is striped as 1GiB blocks until less than one row remains, then 1MiB blocks, and consecutive blocks land on different shards. With ec.encode's -fullPercent 95 against the 30GiB default limit, ~30% of every volume sits in that 1MiB tail, so a 4MB filer chunk there is five stripes on five servers. New encodes now use one block per shard, sized ceil(datSize/dataShards) rounded up to 1MiB and recorded in the .vif (EcShardConfig.block_size, also carried by the .ecsum manifest). A needle now maps to one shard unless it is larger than the block or straddles a boundary. The chosen size equals the legacy layout's padded shard length for every input, so shard sizes, capacity math, and the shard-size credibility checks are unchanged; only the byte placement moved. Reads, decode, and scrub resolve the block sizes from the volume's .vif; absence keeps the legacy interpretation, so existing EC volumes read exactly as before. Rebuild is layout-agnostic. weed fix -ecx recovers the layout from the .vif, else the .ecsum sidecar, and with neither de-stripes under both candidate layouts and keeps the one that indexes more valid needles. Same change in the Rust volume server, which now also streams the encode in 256KB sub-batches like Go instead of allocating whole blocks, and computes the large-row count as shardSize/largeBlock to match Go on exact multiples. On a 26MB fixture both encoders produce byte-identical shards, and a Go-written .vif parses in Rust with the block size intact. * ec: resolve the rust ecx rebuild through the recorded layout The Rust rebuild path regenerated a lost .ecx by scanning the logical .dat through a hand-rolled pure-1MiB striping, which was already wrong for legacy volumes with large-block rows and is wrong for any uniform volume with a block past 1MiB. Route the scan through locate_data with the .vif-recorded block size, the same mapping the read path uses. Also seed the new tests' random data instead of the deprecated global math/rand.Read. * ec: fail the Rust ecx rebuild on any shard read error A read error mid-scan published the entries collected so far as a successful .ecx, and read_at's byte count was ignored so a legal short read passed as complete — a truncated or failing shard could produce a silently incomplete recovery index. Exact-read semantics in read_from_data_shards, error propagation in the needle walk, and a truncated-shard regression test. * ec: fail the mount on an unreadable or malformed vif Both servers silently fell back to the legacy layout when an existing .vif could not be read or parsed. Every new encode records a positive uniform block size there, so the fallback mounted the same shards with legacy offset math and could return wrong data. Absent stays legal (legacy volumes predate the sidecar), and a zero-byte stub still reads as absent (Go's MaybeLoadVolumeInfo convention, now mirrored in Rust); a present-but-unreadable or malformed .vif fails the mount instead. * ec: bound the reconstruct fan-out of one needle's intervals A degraded interval fans out a read to every reachable shard location, each with a buffer the size of the interval. Reading a needle's intervals in parallel multiplied that by the interval concurrency: a needle spanning 8 blocks could hold 8 x MaxShardCount remote reads and buffers at once, where the sequential version peaked at MaxShardCount. Give each needle a single reconstruct budget its intervals share, held for the buffer's lifetime, so separate reads stay independent but one read cannot multiply its own fan-out. * ec: drop the duplicated shard-size formula calculateExpectedShardSize reimplemented the padding rule that UniformBlockSize already owns — TestUniformBlockSizeMatchesLegacyShardSize asserts the two agree for every input — so a change to the rule would have had to be made in both. Defer to the helper, keeping the historic answer for an empty .dat. * ec: resolve the shard block layout from whatever records it Four places still answered the layout question by inference when a record of it was available, or accepted an answer that was not one: - A mount with no .vif defaulted to the legacy layout; the bitrot sidecar records the same config at encode time, so take it when present, as weed fix -ecx already does. The vif itself is now parsed once per mount rather than twice. - The Rust ecx rebuild derived its row count from the padded shard extent, which under the legacy layout reads a shard that is an exact large-block multiple as one row too many. Pass the encode-time .dat size from the .vif and keep the extent as the fallback. - weed fix -ecx read the block size outside the EC-config guard (collapsing the unknown sentinel into a definitive legacy), only wrote the recovered layout back when the .vif was absent rather than unusable, and broke a scan tie by candidate order instead of the documented reach. - The uniform layout tripped writeDatFile's large-block ambiguity guard, which cannot apply when the large and small blocks are the same size. * ec: give the index-recovery tests a parseable vif The fixtures wrote the literal bytes "volinfo" as the source .vif and the recovery copies it verbatim, so the receiving server then mounted the volume from a .vif it could not parse. That used to pass by silently defaulting to the legacy layout; a mount now refuses a vif it cannot read, which is what the tests were exercising all along without meaning to. * ec: validate the layout a vif records, not just its syntax Review follow-ups on the mount-strictness change: - A .vif can parse and still record a block size no encoder could have produced (negative, or not a whole number of small blocks). Both servers took it and mapped every read through it. ValidateBlockSize / the Rust mirror now refuse the mount, the same way an unparseable vif does; 0 stays valid as the legacy two-tier layout. - The bitrot-sidecar fallback accepted parity_shards == 0 and summed the counts in their own width, so values near the ceiling wrapped past the MaxShardCount bound. Require both counts and sum in a wider type. - weed fix -ecx treated a config with only DataShards > 0 as usable, so a half-written .vif suppressed the recovery paths AND survived the rewrite. Require a complete, in-range config before trusting it. - Returning the vif-load error left the .ecx and .ecj descriptors open; repeated mount attempts on malformed metadata could exhaust them. * ec: refuse to act on a layout the metadata does not establish - The worker encode only logged a failed .vif write and skipped it in the distribution set, and treated the .ecsum write as best-effort. A worker whose disk filled after the much larger shards landed could still distribute, mount, verify shard inventory, and delete the source replicas — leaving holders with shards whose geometry nothing records. Both writes and both inclusions are encode success conditions now. - A generation-matching .ecsum that disagreed with the .vif geometry only disabled checksums in Go, and in Rust was not compared at all, so protection stayed On while reads used the other layout. Both files record the layout their generation was encoded with, so a disagreement now fails the mount. * ec: reject an invalid recorded block size in weed fix -ecx A .vif with valid shard counts but a negative or unaligned block size was marked usable: a positive invalid value pinned the scan to a geometry that de-stripes to garbage, and a negative one ran the dual scan but left the invalid .vif in place afterwards. Validate it with the same rule the mount applies, and when it fails leave the layout unknown so the scan recovers it and the file is rewritten. * ec: validate the sidecar layout weed fix -ecx recovers from The .ecsum fallback was taken on DataShards > 0 alone, so a CRC-valid sidecar carrying the wrong generation, an incomplete ratio, or an unaligned block size would pin the reconstruction to one incorrect uniform-layout candidate instead of letting the dual scan decide. Require generation 0, a complete in-range ratio, and a valid block size; anything less leaves the layout unknown, which is the answer that still recovers by scanning. * ec: let only a genuinely absent sidecar choose the legacy layout With no .vif the bitrot sidecar is the only record of a volume's layout, and the mount fallback read a failed load, an unusable config, or a sidecar stamped for another generation as "assume legacy". A uniform generation-0 volume could therefore mount with legacy or another generation's geometry and answer reads with the wrong bytes. Present-but-unusable now fails the mount; only actual absence keeps the legacy defaults. Shared as EcShardConfigFromSidecar so every caller reads the sidecar the same way. * ec: treat a recorded-but-impossible layout as corruption, not as legacy - A .vif whose ecShardConfig is PRESENT but records an impossible ratio was answered with the default 10+4 and the legacy block layout, in both languages. That reads a uniform volume's shards at the wrong offsets and returns the wrong bytes. Only an entirely absent config still means "this predates the record"; a present one that cannot be true fails the mount. - The shard-count bound summed two uint32 counts as int, which wraps on a 32-bit build: 0x7fffffff + 0x7fffffff lands at -2 and slips under MaxShardCount. ValidEcShardCounts sums in uint64, and every EC call site that checked a recorded ratio now goes through it. * ec: rebuild on the geometry the sidecar records, and flag it when it disagrees The rebuild RPC passes BackgroundECContext, so RebuildEcFiles resolves the layout itself — and it resolved a missing or invalid .vif to the default 10+4 with the legacy block size. Two consequences: a 12+4 volume was reconstructed through a 10+4 matrix, which produces wrong bytes and never regenerates shards 14-15; and the chosen geometry then contradicted a valid uniform sidecar, which loadRebuildSidecar reported as BitrotOff — silently skipping the input and regenerated-shard checksum checks precisely when the volume had already lost its metadata. The layout now resolves from the bitrot sidecar (found across the server's disks, not just beside the base name) before falling back to the defaults, and a present-but-impossible ratio fails instead of being replaced. A sidecar that contradicts the chosen geometry is BitrotInvalid, which the existing unsafeIgnoreSidecar override still lets an operator push past. * ec: let the Rust rebuild read metadata off a sibling disk read_ec_shard_config searches only the location the rebuild writes into, so a volume whose .vif or generation-0 .ecsum sits on another of the server's disks resolved to the default 10+4 with the legacy block layout — the Rust half of the geometry-guessing the Go rebuild just stopped doing. It then reconstructs a custom-ratio or uniform volume through the wrong Reed-Solomon matrix and de-striping geometry. The rebuild now looks for the .vif in its own location and then each sibling, falls back to the generation-0 sidecar wherever that lives, and only defaults when neither exists anywhere. The encode-time .dat size the ecx rebuild needs is resolved the same way. * ec: resolve a rebuild's vif from every directory that may hold it RebuildEcFiles probed only <data-base>.vif. The caller knows the selected location's index directory and the sibling locations, but passed neither for metadata: additionalDirs carried shard directories only, and were searched for shards and the checksum sidecar. A split -dir/-dir.idx layout, or a disk holding only shards, therefore resolved a pre-sidecar custom-ratio volume to 10+4 and reconstructed through the wrong matrix — never regenerating shards 14-15. The caller now hands over the index and sibling directories, and the resolver probes the vif across all of them, matching what the Rust resolver already does for both the vif and the sidecar. * ec: make every rebuild consumer agree on the layout it resolved - The post-rebuild bitrot backfill re-derived the geometry from this directory's .vif alone and dropped the block size entirely, so a rebuild that resolved its layout from a sibling, the sidecar, or a uniform vif wrote a manifest describing a DIFFERENT layout — one later mounts reject, or that covers only the default shard count. The layout is resolved once now, through an exported ResolveRebuildECContext, and the rebuild and the backfill share that answer. - The Rust rebuild collected only each location's data directory, so a sibling's INDEX directory — where a split -dir/-dir.idx layout keeps .ecx/.ecj/.vif — was never probed, and a custom-ratio volume still resolved to 10+4 with the legacy layout. Both directories of every location are carried now, deduped against the rebuild's own. - A shard delivery can bring the checksum manifest with it, but the receive path only writes the file: a server that already had the volume mounted kept its resolved protection state (off) until a remount. The mount RPC re-resolves it once the shards it describes have been added. * ec: cover the rebuild's directory search with tests Reviewers flagged the sibling index directory twice, and the fix that closed it had no test of its own: the assembly sat inline in the rebuild handler, reachable only through a gRPC call against a populated store. Lifting it into rebuildSearchDirs / select_rebuild_location makes the rule assertable — a sibling contributes BOTH its data and its index directory, a shared index directory is listed once, and the rebuild's own data directory never repeats. Writing the Rust cases surfaced that the two implementations do not agree on where the rebuild's own index directory belongs, and both are right: Go's resolver takes a single directory list, so that directory has to be inside it, while Rust's takes the rebuild's data and index directories as their own arguments and would search them twice. The tests now state which contract each side is holding to, so neither drifts into the other's shape. Pure refactor otherwise; no behaviour change. * ec: search the index directory for the layout sidecar The Rust resolver looked for the generation-0 .ecsum in the rebuild's data directory and the sibling list, but not in the rebuild's own index directory — while the .vif lookup directly above it did, and Go's findBitrotSidecar has always checked both bases. On a split -dir/-dir.idx location that directory is where the metadata lives, and callers leave it out of the sibling list precisely because it is passed here separately, so nothing searched it. With no .vif anywhere the sidecar is the only surviving record of the layout. Missing it resolved a 12+4 uniform volume to 10+4 with the legacy striping — the test added here fails with (10, 4, 0) against the old code — and the rebuild then reconstructs through the wrong matrix and writes .ecx offsets that no reader can follow. * ec: let the rebuild see its own index directory The Rust rebuild takes a single flat directory list — the shape Go's RebuildEcFiles uses — so it cannot be handed the rebuild location's index directory separately the way the layout resolvers are, and the handler was passing the sibling list, which deliberately omits exactly that directory. On a split -dir/-dir.idx location that is where .ecx and .vif live, so the shard and index lookups could not see them. Go has always carried that directory in additionalDirs; this lines the two call sites up. * ec: let a config-free vif fall through to the layout sidecar A .vif that carries no ecShardConfig answers nothing about the layout, so it is no more informative than an absent one — but both trees treated its mere existence as the end of the search. Go went straight to the 10+4 legacy defaults without consulting the sidecar at all; Rust returned whatever ec_shard_config_from could make of a single directory. A 12+4 uniform volume with a legacy config-free vif therefore resolved as 10+4 legacy, and every read landed at the wrong shard offset. The sidecar lookup was also single-directory on both sides, while a split -dir/-dir.idx layout keeps .vif and .ecsum with the INDEX. Go's findBitrotSidecar has always taken both bases; the callers here passed only the data base, and the Rust bitrot resolver derived its path from the data base alone. Rust's layout resolver now takes a candidate directory list — data, index, then any siblings — and searches all of it, which also removes the early return that made the vif's presence decisive. load_vif_info_across_dirs reported `dir` even when load_vif_info had found the vif in `dir_idx`. Nothing reads that field today, so this changes no behaviour; it stops the next caller that resolves the rest of the volume's metadata against the answer from being sent to a disk holding none of it. Absence stays legal throughout: a volume with neither record is genuinely legacy. Present-but-unusable still fails the mount, now in the config-free-vif branch too. * ec: activate a delivered sidecar on every per-disk runtime A vid mounts as one EcVolume per disk, each with its own resolved protection state, but the post-delivery reload used the first-match lookup and so touched exactly one of them. The siblings kept reporting no protection until a remount — and since shard distribution deduplicates the metadata files onto the first target disk for a node, the runtime that got the .ecsum is not necessarily the one the lookup returns. Iterate every runtime instead, via a new FindAllEcVolumes and its Rust mut equivalent. Combined with each runtime now resolving its sidecar against its index directory as well as its data directory, a server sharing one -dir.idx across its disks activates all of them from the single delivered copy. The Rust volume server had no post-mount reload at all; it gets one here, matching Go. * ec: resolve the delivered sidecar across every EC metadata directory Reloading every per-disk runtime, added last round, did not by itself make the delivered manifest reachable. Startup mirroring copies .ecx/.ecj/.vif to every shard-bearing disk so each mounts self-contained, but deliberately not .ecsum, and a repair delivers exactly one copy. Each runtime was resolving against its own two directories, so every sibling of the disk that received the file kept reporting no protection however often it reloaded. Resolve one authoritative copy across every EC metadata directory instead of duplicating the file. Mirroring .ecsum would have to keep pace with a file that is rewritten as shards are repaired, and would not help the reported case at all: the delivery happens at runtime, and mirroring only runs at startup. The regression test pins both halves — a reload restricted to the volume's own directories still finds nothing, and the same reload given the server's metadata directories turns protection on. * ec: ask every directory before writing a TOFU baseline After a rebuild the opportunistic backfill asks whether this volume already has a checksum manifest, and answered from the data base alone. A split -dir/-dir.idx layout keeps the sidecar with the index, and a multi-disk server may keep it on a sibling, so an existing manifest read as absent. The consequence is worse than a missed read. On a false "no" the backfill writes a fresh sidecar at the data base from whatever the shards say right now — and the data base is the first candidate every resolver checks, so that TOFU baseline shadows the real manifest rather than sitting beside it. A shard that was silently corrupt gets blessed, and the record that would have caught it stops being consulted. FindBitrotSidecar exports the search the package already used internally, so the question is asked of the data base, the index base and the sibling disks — the same candidates the rebuild resolves its layout from. * ec: refuse a shard block size no encoder could have produced weed fix -ecx derived one from the raw shard extent, so a truncated or partially copied shard wrote a .vif that NewEcVolume then permanently refuses — the volume the tool was run to rescue could never mount again. An extent that is not a whole number of small blocks cannot have come from a uniform encode, so it is no longer offered as a candidate, and nothing unvalidated reaches the .vif. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 * ec: derive the .vif's dat size and block size from one measurement VolumeEcShardsGenerate stat'ed the .dat before the encode while WriteEcFiles stat'ed it again to size the blocks. A write landing between the two produced a .vif whose own two fields describe different files. WriteEcFiles now leaves both on the context, and fills a placeholder context in place so the caller can read them back. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 * ec: keep the source volume until every holder serves its shard layout The uniform layout rides in a .vif field older volume servers never knew: they discard it, mount the shards as legacy and return wrong bytes with nothing erroring, and the shard files are the same length either way so no other check notices. The upgrade order lived only in the release note. VolumeEcShardsInfo now reports the block size the holder actually serves, in both the Go and Rust servers, and the pre-delete verification refuses to drop the source unless every reachable holder echoes the one the shards were encoded with — while a rollback still exists. A server that predates the field answers 0, which is the negative answer. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 * ec: drop the rebuild's dead block-size parameters generateMissingEcFiles never reads largeBlockSize/smallBlockSize — Reed-Solomon reconstruction is layout-agnostic — so passing the legacy constants only advertised a layout the rebuild does not use. Also move UniformBlockSize's doc off ValidateBlockSize. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 * ec: warn about EC defaults only when the mount used them The "vif file not found, using defaults" warning fired even after the bitrot sidecar supplied a non-default layout, sending anyone triaging wrong bytes after the legacy layout the volume never mounted on. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 * ec: stat the distributed bitrot sidecar once The strict check re-stat'ed the file immediately before the stat that already gates inclusion, and a failed sidecar write now fails the encode outright, so the first could only fire on a deletion between the two lines. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 * ec: say what the reconstruct budget actually bounds A shard's buffer stays in bufs until its interval reconstructs, which is after the read that filled it released its permit, so the semaphore bounds round trips in flight and not retained bytes. Peak memory is the intervals reconstructing at once times the shards each reaches times the interval size. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7 * test: let the fake volume server report its delivered EC layout The pre-delete verification now asks each holder which shard block layout it serves, and a fake that always answered "unset" looked exactly like a volume server too old to know the field. Distribution ships the .vif to every holder alongside its shards, so read the layout back out of it as a real holder does. Claude-Session: https://claude.ai/code/session_011FRRoNKBiGbH58rs2AQyA7	2026-08-28 20:46:59 -07:00
Guang Jiong LouandChris Lu	93666c90e9	filter by volume ids (#10983) ... * filter by volume ids * master: carry the volume ids VolumeList asks about in one repeated field One id and a list of them ask the same question, so field 2 holds the list rather than standing beside a second field that supersedes it. Claude-Session: https://claude.ai/code/session_011qAmAdhrYvnzGkw7A9N4mP --------- Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-08-28 20:43:29 -07:00
Chris LuandLisandro Pin	fcc2ea61d3	ec: scrub a volume through its parity data (#11006) ... * Introduce a new `READS` scrub mode. `READS` performs a full volume scrub but, unlike `FULL`, it will attempt to reconstruct data for missing/damaged shard intervals from other shards in the cluster when necessary. The goal of this check is to ensure that EC volume contents _are readable by Seaweed_ even on a degraded storage state, by exercising parity data which is not read in `FULL` mode. This is useful not only to validate data is user-readable, but also to detect potential parity shard issues which may be difficult to pinpoint otherwise - particularly for older volumes lacking sidecar data, and hence unaffected by `CHECKSUM` scrubs. For regular volumes, this operation is equivalent to `FULL`. Example: ``` > ec.shard.unmount --volumeId=1 --shardId=0,3,11 --delete --apply Live shard topology for volume ID 1 (14 shards): 0@10.200.18.89:9001 1@10.200.18.89:9002 2@10.200.18.89:9003 3@10.200.18.89:9004 4@10.200.18.89:9005 5@10.200.18.89:9006 6@10.200.18.89:9007 7@10.200.18.89:9008 8@10.200.18.89:9009 9@10.200.18.89:9013 10@10.200.18.89:9010 11@10.200.18.89:9011 12@10.200.18.89:9012 13@10.200.18.89:9020 Will unmount + delete 3 shard(s): 0@10.200.18.89:9001 3@10.200.18.89:9004 11@10.200.18.89:9011 Unmounting shard 0@10.200.18.89:9001 for volume ID 1... Deleting shard 0@10.200.18.89:9001 for volume ID 1... Unmounting shard 3@10.200.18.89:9004 for volume ID 1... Deleting shard 3@10.200.18.89:9004 for volume ID 1... Unmounting shard 11@10.200.18.89:9011 for volume ID 1... Deleting shard 11@10.200.18.89:9011 for volume ID 1... All done! > ec.scrub --volumeId=1 --node=10.200.18.89:9002 --mode=full using FULL mode Scrubbing 10.200.18.89:9002 (1/1)... Scrubbed 6 EC files and 1 volumes on 1 nodes Got scrub failures on 1 EC volumes and 1 EC shards :( Affected volumes: 10.200.18.89:9002:1 Affected shards: 10.200.18.89:9002:1:0 > ec.scrub --volumeId=1 --node=10.200.18.89:9002 --mode=reads using READS mode Scrubbing 10.200.18.89:9002 (1/1)... Scrubbed 6 EC files and 1 volumes on 1 nodes ``` * ec: report the shards a READS scrub had to rebuild A READS scrub that recovers an interval was recording nothing, so a volume missing three shards came back clean and nobody repaired it. The unreadable shard is now recorded before the rebuild is attempted: READS reports the same broken shards as FULL and differs only in whether the needles themselves failed, which is the signal worth having - shards are gone, data is still there. forceDeletedNeedlesCheck now applies to READS as well, in the shell and in the RPC guard: it runs the same needle walk as FULL. Regenerated the proto instead of hand-editing it, so the pancis typo (which protoc-gen-go-grpc emits into eight other files here) and the header whitespace stay as generated. Mirrors into the Rust volume server, which also now honors force_deleted_needles_check rather than hardcoding it off. Claude-Session: https://claude.ai/code/session_014yMNebkUjSbx9sfUCWJJtq * ec: answer a deleted needle from a READS rebuild as deleted #11020 gave the Rust recovery a deleted flag alongside its bytes, and it answers a deleted needle with no bytes at all. The READS scrub appended that empty answer, which does not compile against the new signature and, once it did, would leave the needle short and report the size mismatch as damage. Zero-fill the interval instead, the way the direct read beside it already does: the assembled needle then reaches read_bytes as the delete-state mismatch the walk already tolerates. Go takes the same branch off the flag its recovery returns, rather than discarding it. Claude-Session: https://claude.ai/code/session_014yMNebkUjSbx9sfUCWJJtq --------- Co-authored-by: Lisandro Pin <lisandro.pin@proton.ch>	2026-08-28 16:42:34 -07:00
Chris Lu	ba5b14b457	master, filer, s3api: bound the collection deletes that strand a caller (#11026) ... * master: bound each volume server DeleteCollection, and finish the fan-out A collection delete fanned out to every volume server holding it with context.Background(), so a server that accepted the connection and then went quiet held the whole delete open with nothing to end it. Each RPC is bounded now, on the same budget allocateVolumeTimeout gives the other master-to-volume-server admin RPC. The volume server runs the delete to completion regardless of the request context, so giving up costs the confirmation and not the deletion. The walk itself is the caller's, not a per-server one: - It outlives the caller. A cancelled request must not abandon a destructive fan-out part-done, with volumes left behind and no request still running to come back for them. - It no longer stops at the first server that refuses, which left the collection on every server after it in the list. The first failure is still what is reported, and the collection stays in the topology so a later delete comes back for the rest. - It sends one RPC per server rather than one per replica. ListVolumeServers reports a node once for every replica it holds, while DeleteCollection removes the whole collection from the server it reaches, so a collection with thousands of volumes repeated the same whole-collection delete thousands of times over. Both passes run too. Returning after a failed normal pass left the collection's EC shards in place with nothing left to retry them. Claude-Session: https://claude.ai/code/session_01EnB1fbryyKc2LetRZxQPTP * master: delete the EC shards behind /col/delete too The HTTP handler carried its own copy of the volume-server walk and only ever ran the normal pass, so a collection deleted through it kept its EC shards. It shares the gRPC path now, which also gets it the bounded RPCs and the one-per-server fan-out. Claude-Session: https://claude.ai/code/session_01EnB1fbryyKc2LetRZxQPTP * filer: bound the collection delete a bucket delete leaves behind Deleting a bucket entry deletes its collection afterwards, deliberately detached from the request so a client that hangs up cannot strand the bucket's volumes. Detached meant unbounded, though: with the master down or mid-election the wait for a leader has nothing to end it, so the handler parks, and the client retrying behind it parks another. It keeps outliving the request and now carries a deadline of its own. The budget bounds the wait, not the work: the master keeps deleting on its own fan-out once asked, so giving up costs the confirmation. Claude-Session: https://claude.ai/code/session_01EnB1fbryyKc2LetRZxQPTP * s3api: bound the collection RPCs a bucket creation and deletion issue Neither carried a deadline, so a transient failure anywhere down the chain held the S3 request open until the client gave up on it. Both budgets are taken outside the filer failover walk, so one budget covers the whole walk rather than granting each filer a fresh one. The walk itself stops when that budget is spent, and stops without blaming anyone: the caller's own expiry is not evidence against the filer that was answering, and the next filer has no time left to answer in either. Recorded as a filer failure, a slow master upstream would flag every filer in the walk, and the three failures that open the circuit take unrelated object reads down with them. Claude-Session: https://claude.ai/code/session_01EnB1fbryyKc2LetRZxQPTP * s3api: a failed collection listing no longer fails a bucket creation PutBucket lists collections to notice a leftover one it is about to reuse. The result feeds a warning and nothing else -- s3a.exists is what decides whether the bucket already exists -- yet a transient failure of that listing returned 500 and refused the creation. It is advisory now, so a failure is logged and the creation continues, exactly as it does when the listing returns false. Claude-Session: https://claude.ai/code/session_01EnB1fbryyKc2LetRZxQPTP	2026-08-28 16:32:30 -07:00
Chris Lu	7dc3835b02	s3: an abort answered mid-part no longer leaves the upload completable (#11025) ... * s3: reject a part whose upload was aborted while its body was in flight The upload-exists check runs before the part body is read. An abort answered during the read deletes the upload directory, and the part write that follows re-creates it, so the aborted upload is listed nowhere yet completes. Re-check after the write: only createMultipartUpload stamps the destination key on .uploads/<id>, so a directory without it is one the part write resurrected. Drop it along with the part and answer NoSuchUpload. Claude-Session: https://claude.ai/code/session_01ByZ49KQdUtHhmjG6SpNczT * s3: reject a copied part whose upload was aborted mid-copy UploadPartCopy has the same window as UploadPart: the upload-exists check runs before the bytes are copied, and the part write that follows re-creates the directory an abort removed. Both the re-encryption and the raw-copy path re-check before answering. Claude-Session: https://claude.ai/code/session_01ByZ49KQdUtHhmjG6SpNczT * s3: do not complete an upload whose directory holds no upload record A .uploads/<id> directory that a part write created rather than createMultipartUpload carries no destination key, no owner and no encryption settings. Completing one turned stray parts into an object; answer NoSuchUpload instead. Claude-Session: https://claude.ai/code/session_01ByZ49KQdUtHhmjG6SpNczT * s3: log the part left behind when the resurrected directory survives abortMultipartUpload can fail to remove what the part write re-created. The client still hears NoSuchUpload, since the upload is gone either way and a retry would only write another part, but the leftover is worth a line. Claude-Session: https://claude.ai/code/session_01ByZ49KQdUtHhmjG6SpNczT	2026-08-28 16:12:09 -07:00
Chris Lu	c858e01a09	ec: split the shard-interval recovery into a gather and a rebuild (#11005) ... * ec: split the shard-interval recovery into a gather and a rebuild Recovering an interval is now one function doing the local seeding, the waved peer fetch, the shard accounting and the Reed-Solomon rebuild, under a memory budget. Splitting the gather from the rebuild makes the rebuild a plain function over a set of intervals, which is testable on its own and reusable by the parity checks a full scrub wants. The rebuild refuses a parity target, and the caller checks that before the gather so a doomed target costs no fan-out. ReconstructData rebuilds data shards only, so asking it for a parity shard returned no error and left the slot nil, and the caller copied that out as a successful read of zeroes. Only data shard ids reach here today, so this is a guard, not a live fix. Claude-Session: https://claude.ai/code/session_014yMNebkUjSbx9sfUCWJJtq * ec: rebuild only the EC shard the read asked for ReconstructData rebuilds every missing data shard. The gather stops as soon as DataShards intervals are in hand, so on a distributed volume it routinely finishes holding parity where data is missing -- and each of those data shards is then rebuilt into an interval-sized buffer, decoded, and never read. Ask for the one shard the read needs. The budget covers it now too: DataShards gathered plus the one the rebuild allocates. It never covered the rebuild's output, and with ReconstructData that output was up to ParityShards buffers. The required mask is Total() long rather than DataShards. reedsolomon documents both lengths, but its presence scan walks every shard and indexes the short mask past its end, so the documented short form panics whenever a parity shard is absent - which here it usually is. Claude-Session: https://claude.ai/code/session_014yMNebkUjSbx9sfUCWJJtq	2026-08-28 15:55:12 -07:00
Chris Lu	cd5013f116	Re-check an EC shard map a failed read has disproved (#11023) ... * Re-check an EC shard map a failed read has disproved A read that fails against a cached location drops that shard from the map, which leaves it one short of complete -- and a map one short is trusted for seven more minutes. So a moment's trouble between volume servers cost minutes in which every read of that shard skipped the direct fetch and paid for a Reed-Solomon recovery instead, at DataShards times the memory and the peer load. Mark the map when a read disproves it, and re-check a marked map on the same eleven-second footing as one that never had enough shards to begin with. The mark clears on refresh, so it buys one prompt re-check rather than a master lookup per read. The tiers move into a helper; they were three overlapping conditions in one expression, and the reading of them was not obvious. Rust keeps the entry rather than dropping it -- a dead peer fails fast on the next attempt, and it was the freshness window, not the entry, hiding a shard that had moved. Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN * Invalidate the location of an EC shard whose own read failed Recovery fans out to the other shards, so the one whose direct read just failed is the only location nothing ever invalidates: a shard that moved to another server was reconstructed on every read until the map's own window expired, up to thirty-seven minutes for a map still complete. Mark the map there too. The entry stays -- a moved shard's old holder fails fast, and the next refresh is seconds away. Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN * Consume the stale mark before the lookup, not after A read that fails while the master is answering has disproved the very map that answer is about to install, and clearing the mark on the refresh's return swallowed it. Clear it where it is acted on instead. A lookup that then fails loses the mark, which costs nothing: the refresh time is only advanced on success, so the next read looks up regardless. Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN * Judge the shard map and consume its mark in one critical section Reading the mark and clearing it were two separate acquisitions, so a mark raised between them was cleared by a refresh that had not seen it. In Go that gap was a few instructions; in Rust the mark was read when the read first snapshotted the volume and cleared at the decision point, with the local interval reads in between. Take both under one hold. Rust needs a mutex rather than an atomic to do it, and no longer carries the mark through the snapshot. Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN * Put the stale mark back when the lookup does not answer for it Consuming the mark up front assumed the lookup would supersede it. A lookup that fails, or comes back with fewer than DataShards holders, supersedes nothing: the map is unchanged, its refresh time unadvanced, and with the mark gone the map a read had disproved is trusted for its full window again on the strength of a lookup that never landed. Put the mark back on both branches. Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN	2026-08-28 15:43:05 -07:00
Chris Lu	624deaf3a4	mount: implement fallocate (#11021) ... * mount: implement fallocate instead of reporting it unsupported Fallocate answered ENOSYS, so the kernel marked the mount as having no fallocate and returned EOPNOTSUPP. glibc then fell back to its emulation, which preads a byte from every block already inside the file to see if it is allocated; on a write-only descriptor that pread is EBADF, and posix_fallocate returned it. Volume space is assigned when a write is flushed, so nothing can be reserved up front: a range inside the file is answered OK untouched, and one past the end grows the file the way a truncate would. A mode we cannot honor is refused with ENOTSUP, not ENOSYS, so the kernel keeps sending the ones we do. Claude-Session: https://claude.ai/code/session_01XG6sAAdkqTwWffqK5h4rH9 * mount: let a fallocate that allocates nothing past the quota and worm guards A range already inside the file, and any FALLOC_FL_KEEP_SIZE request, reserve no space and rewrite no entry, but the preflight refused them with ENOSPC on a full mount and EPERM on a worm-enforced file. Decide the no-op first and guard only the growth. Claude-Session: https://claude.ai/code/session_01XG6sAAdkqTwWffqK5h4rH9 * mount: charge a fallocate growth to the uncommitted byte counter Write charges the counter by how much the file grew, so the writes that fill a range fallocate already extended charge nothing and the real-time quota check never sees that data — only the periodic filer refresh does. Count the growth where it happens. Claude-Session: https://claude.ai/code/session_01XG6sAAdkqTwWffqK5h4rH9 * mount: charge a truncate-up growth to the uncommitted byte counter Same gap Fallocate had: Write charges the counter by how much the file grew, so the writes that fill a range ftruncate already extended charge nothing and the real-time quota check never sees that data. Count the growth where it happens; a shrink still leaves the counter alone, since it is only ever raised and then reset by the periodic filer refresh. Claude-Session: https://claude.ai/code/session_01XG6sAAdkqTwWffqK5h4rH9	2026-08-28 14:56:50 -07:00
Chris Lu	7bb0a1c127	s3: replay a delete whose reply the transport dropped (#11022) ... * s3: stop retrying a delete the filer refused for a non-empty folder The filer looked and the children are there, so the answer will not change. retryFilerOp spent six attempts and up to 3.1s of backoff on it before the caller could act on the condition it was already holding. Claude-Session: https://claude.ai/code/session_01XqaJrwgXQ5GSUpyzRbe5nD * s3: thread the request context through the unversioned delete path doDeleteEntry issued every DeleteEntry on context.Background(), so an S3 client that hung up left the gateway working on its behalf, out of reach of both cancellation and the per-request retry allowance that DeleteMultipleObjectsHandler installs. Claude-Session: https://claude.ai/code/session_01XqaJrwgXQ5GSUpyzRbe5nD * s3: treat a cancelled filer RPC as terminal, not transient isRetryableFilerErr matched context.Canceled and DeadlineExceeded by sentinel, which only holds while the error is still local. Once it has crossed gRPC it is a status, so an abandoned request was retried six times on behalf of a caller that had already gone. Claude-Session: https://claude.ai/code/session_01XqaJrwgXQ5GSUpyzRbe5nD * s3: replay a delete whose reply the transport dropped A delete is idempotent at the filer, which answers an entry that is already gone with an empty resp.Error, so a reply lost in transit can be reissued rather than surfaced. Surfaced, it becomes a 500 on the bucket delete, which boto3 resends and is then answered NoSuchBucket, or a per-key InternalError inside the 200 of a multi-object delete, which no SDK retries at all. The replay runs through retryFilerOp, so it draws on the allowance the request already installs rather than paying a backoff per key, and stops for a caller that has gone. rm and rmObject re-enter WithFilerClient per attempt, so each one walks the failover list again on a connection the failed attempt had invalidated; the multi-object loop holds one client for the batch, so there the replay reuses it. Classification stays structural. The filer reports its own refusals in resp.Error, which carries no status and has the deleted path - and, for a recursive delete, the children it stopped on - formatted into it, so no key name can steer the decision either way. rm and rmObject now take the caller's context. Cleanup and rollback paths pass context.Background() deliberately: they have to run whether or not the caller is still waiting. Claude-Session: https://claude.ai/code/session_01XqaJrwgXQ5GSUpyzRbe5nD * s3: share one retry allowance across multipart completion cleanup The unused-entry loop deletes once per entry, and each delete now retries, so a filer that stays unavailable held the response for 3.1s per entry after the object was already committed. Claude-Session: https://claude.ai/code/session_01XqaJrwgXQ5GSUpyzRbe5nD	2026-08-28 14:30:21 -07:00
Chris Lu	af6f69740c	Read metadata log chunks the way the mount reads every other chunk (#11018) ... * Replay metadata log chunks the way the mount reads every other chunk The subscription's log-chunk replay built its own lookup, which always resolves volume server addresses. A mount started with -volumeServerAccess=filerProxy cannot reach those, so every fresh subscription failed on the previous minute's persisted segment and resubscribed a second later, forever. Take the lookup from the caller instead; the mount hands over the one it uses for file reads, which also keeps publicUrl and the bounded location cache in play. Claude-Session: https://claude.ai/code/session_01NGqrxYj7cHUpSrL249n3Z6 * Keep a log chunk read failure off the filer connection A metadata subscriber reads persisted log chunks over HTTP from volume servers and hands whatever went wrong back as the subscription's error. "connection refused" from a volume server then matched the transport patterns that decide a gRPC channel is dead, so every failed replay closed the shared filer ClientConn and cancelled the assign and upload RPCs riding on it with "the client connection is closing". Mark those read failures so they are judged for what they are. Claude-Session: https://claude.ai/code/session_01NGqrxYj7cHUpSrL249n3Z6	2026-08-28 14:15:39 -07:00
Chris Lu	95248f7492	Bound the memory an EC shard recovery holds (#11020) ... * Reconstruct an EC shard from the shards already on this server recoverOneRemoteEcShardInterval only ever fanned out to the cached shard locations, so a server holding shards of the volume still fetched them over gRPC from itself -- and when the peers were unreachable it could not reconstruct at all, even holding the whole volume on local disk. Seed the Reed-Solomon buffers from the locally mounted shards first; each one is a peer round trip, and an interval-sized buffer, the fan-out no longer needs. Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN * Fetch only the EC shards reconstruction still needs The recovery fan-out read every surviving shard, so a 10+4 volume pulled 13 interval-sized buffers to feed Reed-Solomon 10 -- a third more memory held, and a third more load asked of peers that were, by definition, already having trouble. Fetch what is missing, and widen only when some of those reads fail. A shard reporting the needle deleted ends the walk: the rest would only answer the same. Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN * Bound the bytes EC recovery holds in flight Recovery is the one read path that multiplies the served bytes: it holds an interval-sized buffer per shard until Reed-Solomon runs, and a peer that is slow to fail keeps them all alive for the whole gRPC timeout. Nothing bounded how many of those fan-outs ran at once, so a transient problem between volume servers turned every read into a DataShards-fold allocation and the server died of it -- 64 concurrent 4MB intervals pin 3.6GB, and that is a small burst. Charge each recovery against a process-wide budget, so a burst queues on the semaphore instead of on the heap. Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN * Answer a deleted EC needle as deleted, not as a failed recovery A holder reporting the needle deleted is authoritative: deletes are never invented and never undone. Recovery already collected that flag, then dropped it on the branch where too few shards came back -- so a read of a deleted needle that had to recover surfaced as "cannot recover shard", and the volume server answered 500 where it owed a 404. Carry the flag out of the shortfall, and let it decide ahead of the error it came with. Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN * Check the encode run of a locally seeded EC shard in Rust The Rust recovery seeded Reed-Solomon straight from the mounted shards, without the encode-run check the remote reads and Go's readLocalEcShardInterval both apply. A volume remounted from a newer encode between the read's snapshot and its recovery would have fed mixed-generation bytes into the reconstruction. Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN * Say what the recovery budget actually guarantees Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN * Seed Rust EC recovery from shards on every local disk find_ec_volume returns the first disk's EcVolume, so a reconciled volume whose shards are split across data dirs had the siblings ignored and could report "cannot recover" while holding enough shards locally. Resolve each shard together with the disk that owns it, the way Go's recovery already does, and check that owner's encode run. Claude-Session: https://claude.ai/code/session_01SM5ARdPvFcnvGWNpBPgNRN	2026-08-28 14:14:40 -07:00