mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-15 00:09:00 +01:00
29d00ae7940407f941ef6ba89a92b3d1e24563ce
11
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
6840371774 |
sftp: keep a file's existing bytes when it is opened for write without truncate (#11723)
* sftp: keep a file's existing bytes when it is opened for write without truncate * sftp: preserve sparse gaps and refuse unreadable entries when copying existing content The copy used a single chunk-stream read, which ends at the first sparse gap: the tail past a hole was dropped, and a remote-only entry with no local chunks copied zero bytes, so closing the handle uploaded an empty file. Read through PrepareStreamContent, which fills unstored regions with zeros, and refuse entries that carry no local bytes at all. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: report a failed destination write from a completed chunk fetch retriedStreamFetchChunkData broke out of the retry loop on a clean HTTP end before checking writeErr, so a destination write that failed mid- stream returned a short write and nil error. Check writeErr first. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * sftp: cover remote and all-holes sizes when copying existing content Take the declared size from RemoteEntry when it exceeds FileSize, pad inline content to the declared size, and treat a chunkless local file as all holes rather than unreadable. A remote-only entry still refuses the copy, now checked with a zero FileSize as well. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * sftp: size the existing-content copy the way filer.FileSize does A locally shortened file still names its older, larger remote object; taking RemoteSize unconditionally padded the copy back out to it. filer.FileSize only honors the remote size when the remote is newer, and also covers the chunk-extent size. A remote-only entry still refuses the copy whenever it points at remote bytes. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
b2eefb8a73 |
filer: fail the request, not the process, when chunks cannot be resolved (#11685)
* filer: fail the request, not the process, when chunks cannot be resolved CompactFileChunks and ViewFromChunks discarded the error from NonOverlappingVisibleIntervals and iterated the nil interval list it returns on failure. A CreateEntry or UpdateEntry whose context was cancelled therefore panicked in SeparateGarbageChunks and took the whole filer down, losing the metadata log it had not yet flushed. Both functions now return the error. cleanupChunks fails the request with it, the mount flush paths return it, the replication sinks and the S3 SSE range reader return it, and a ChunkStreamReader that cannot build its view fails every read with it and reports it as its SourceError instead of reading as an empty file. Fixes #11682 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * filer: propagate chunk resolution failures in scan paths NonOverlappingVisibleIntervals can fail without a cancelled context; the query engine and log-store callers discarded the error and dereferenced nil intervals. * query: propagate chunk resolution errors instead of skipping files When a manifest cannot be resolved, ReadParquetStatistics, countLiveLogRowsExcludingParquetSources, and computeLiveLogMinMax logged a warning and skipped the failed file, so queries returned understated counts and wrong min/max values. Surface the errors instead: the fast-path aggregation collector now returns a DataSourceError so the engine falls back to a full scan, and the row-count helpers return the error rather than a partial total. --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris LuandDevin
|
e3fadc6e04 |
filer: keep the proxy-JWT test from hanging on a failed fetch (#11664)
The handler sends the header non-blocking and the test stops on fetch errors instead of waiting on a channel that may never be fed. Generated with [Devin](https://devin.ai) Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
4d1f49c638 |
filer.sync: sign proxied chunk I/O from the per-side security file (#11645)
* filer.sync: sign proxied chunk I/O from the per-side security file The -a.security / -b.security files were used for gRPC TLS and the HTTPS client but not for jwt.filer_signing, so filer-proxied chunk reads and writes carried a token signed with the process-wide key and failed authorization whenever the two clusters' keys differ. LoadFilerJwtFromFile returns a FilerJwtProvider for each side's file, which FilerSource and FilerSink now accept for proxied chunk reads and writes. With no keys in the file or no flag, both fall back to the process-wide jwt.filer_signing configuration as before. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * replication: use the side filer read key for manifest downloads and fall back per access level Manifest chunk resolution still signed proxied downloads with the process-wide read key, so a source filer requiring its own key 401'd on manifest-bearing files. A side security file that set only one access level also produced empty tokens for the other instead of inheriting the process-wide key, and the side file loader ignored the WEED_ environment overrides the filer itself honors. ResolveChunkManifest/ResolveOneChunkManifest keep their signatures; FilerJwt-aware variants thread the provider down to fetchWholeChunk, which prefers it on proxy URLs. The side loader now applies the same environment precedence and falls back to the process-wide signer per missing access level. * security: verify the configured filer token lifetimes * security: reject negative filer token lifetimes A negative expires_after_seconds reached GenJwtForFilerServer and produced a token with no expiration claim. Also synchronize the Authorization-header capture in the proxy test and restore the prior viper key on cleanup. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
8a68337256 |
filer: pack SSE chunks into manifests (#11175)
* filer: pack SSE chunks into manifests * s3: resolve encrypted manifests before reads * s3: scope encrypted manifest resolution to ranges |
||
|
Chris Lu
|
fe98520358 |
read: try a replica that stopped answering last, and relearn its volume's locations (#11130)
* http: try a volume server that failed to answer last A cached location list is shuffled on every read, so once a replica dies half the reads keep dialing it first and pay a connect failure or timeout before the healthy replica answers. Remember, per host, when a request got no answer at all and order such hosts last for the next half minute. Once that passes, one read probes the host in its usual place while the others keep it last until the probe settles, so a black-holed server costs one stalled read per interval instead of one per read. Nothing is ever skipped: a host that failed is still tried when the others fail too. Any response, including an error status, counts as reachable. Claude-Session: https://claude.ai/code/session_011NYXuzGttwrTMsfLYvmQFs * filer: refresh a chunk's locations after one of them fails A mount's location cache is only relearned when every cached location fails. When one replica dies and the other still answers, every read succeeds and the dead replica stays in the cache, and in the shuffled order it keeps being dialed first long after the master has dropped it. When a read fails on one location and a later one answers, call the refresh hook so the cached entry is dropped and looked up again. The read that already paid for the failure returns its data; the reads after it start from the locations the master knows now. Claude-Session: https://claude.ai/code/session_011NYXuzGttwrTMsfLYvmQFs * http: claim the probe for every expired host, and try it first The claim was only checked for the first url, so with two replicas whose marks expired together the second was probed by every read at once. Claim each expired host on its own and put the reads that won a claim ahead of the reachable hosts, so a probe is always a real attempt and a lost claim always means the host is tried last. Claude-Session: https://claude.ai/code/session_011NYXuzGttwrTMsfLYvmQFs * filer: refresh a chunk's locations in the streaming read path too The streaming loop had no refresh hook, so a manifest or streamed chunk that failed on one cached location and was served by another kept the stale entry until every location failed. Give it the same hook as the buffered loop, built by one refreshUrls function shared by the reader cache and the stream callers. Claude-Session: https://claude.ai/code/session_011NYXuzGttwrTMsfLYvmQFs * http: probe at most one expired host per read Claiming every expired host in one ordering left all but the first claim without an attempt, since a read stops at its first answer, and a host that had come back waited another interval for nothing. Claim only the first expired host a read sees and leave the rest last and unclaimed, so each following read probes one of them. Claude-Session: https://claude.ai/code/session_011NYXuzGttwrTMsfLYvmQFs * test: start the live server before releasing the dead server's port Closing the dead server first let the live server come up on the same port, in which case the dead location answers and the partial failure under test never happens. Claude-Session: https://claude.ai/code/session_011NYXuzGttwrTMsfLYvmQFs |
||
|
Chris Luandbruce-zzz
|
0f05957bc4 |
filer: self-heal chunk manifest reads when volume locations go stale (#11107)
* filer: self-heal fetchWholeChunk on stale volume locations Upstream #10156/#10800 wired cache invalidation into the buffer-based read paths, but manifest resolution still goes through fetchWholeChunk, which returns the raw error on failure. When cached volume locations are stale (volume tiered to remote storage, server rolled), resolving a large multipart file fails permanently even though other locations are healthy. Thread the ChunkGroup's cacheInvalidator through ResolveChunkManifest / ResolveOneChunkManifest / fetchWholeChunk, and on failure invalidate, re-lookup and retry once via the existing retryFetchWithFreshLocations helper. The streaming bytesBuffer is reset before the retry so partial bytes from the failed attempt cannot corrupt the manifest proto.Unmarshal. Non-mount callers pass nil and keep their semantics. * filer: move the manifest self-heal tests in with the other manifest tests Also make the stale server stream a prefix and then abort mid-body, which is what actually leaves partial bytes in the buffer: an HTTP error status returns before ReadUrlAsStream ever calls the writer, so a 500 never exercised the Reset the tests claimed to cover. Claude-Session: https://claude.ai/code/session_01FK3oGC5ZVeJYvNBWgb9JUD * filer: keep the cached volume locations when a manifest read is cancelled A cancelled or timed-out read says nothing about where the volume lives, so dropping the location and going back to the master only costs the next reader a round trip. PrepareStreamContentWithThrottler already guards its self-heal this way. The guard also goes inside retryFetchWithFreshLocations, since the caller can be cancelled between its own check and the invalidation, and that covers the reader cache and prefetch paths too. fetchWholeChunk returns the context error rather than the stream failure it provoked, and ResolveOneChunkManifest wraps with %w so errors.Is still sees it. That matters even where no invalidator is passed: volume.fsck resolves manifests with nil and tells its own abort from a corrupt manifest that way, so the cancellation check sits ahead of the nil-invalidator return. Claude-Session: https://claude.ai/code/session_01FK3oGC5ZVeJYvNBWgb9JUD * filer: self-heal manifest reads on the filer and s3 paths too Every caller that already holds the location cache backing its lookup function can hand it over: the filer's read, copy and deletion paths and the log cache have the MasterClient right there, and s3api has the FilerClient. MinusChunks takes one for the same reason, since the deletion path resolves manifests through it. Only the shell tools and the replication sinks, whose lookup functions cache privately with nothing to invalidate, keep passing nil. Claude-Session: https://claude.ai/code/session_01FK3oGC5ZVeJYvNBWgb9JUD --------- Co-authored-by: bruce-zzz <bruce.zou@hhy-data.com> |
||
|
Chris Lu
|
00a2e22478 |
fix(mount): remove fid pool to stop master over-allocating volumes (#9111)
* fix(mount): remove fid pool to stop master over-allocating volumes
The writeback-cache fid pool pre-allocated file IDs with
ExpectedDataSize = ChunkSizeLimit (typically 8+ MB). The master's
PickForWrite charges count * expectedDataSize against the volume's
effectiveSize, so a full pool refill could charge hundreds of MB
against a single volume before any bytes were actually written.
That tripped RecordAssign's hard-limit path and eagerly removed
volumes from writable, causing the master to grow new volumes
even when the real data being written was tiny.
Drop the pool entirely. Every chunk upload goes through
UploadWithRetry -> AssignVolume with no ExpectedDataSize hint,
letting the master fall back to the 1 MB default estimate. The
mount->filer grpc connection is already cached in pb.WithGrpcClient
(non-streaming mode), so per-chunk AssignVolume is a unary RPC
over an existing HTTP/2 stream, not a full dial. Path-based
filer.conf storage rules now apply to mount chunk assigns again,
which the pool had to skip.
Also remove the now-unused operation.UploadWithAssignFunc and its
AssignFunc type.
* fix(upload): populate ExpectedDataSize from actual chunk bytes
UploadWithRetry already buffers the full chunk into `data` before
calling AssignVolume, so the real size is known. Previously the
assign request went out with ExpectedDataSize=0, making the master
fall back to the 1 MB DefaultNeedleSizeEstimate per fid — same
over-reservation symptom the pool had, just smaller per call.
Stamp ExpectedDataSize = len(data) before the assign RPC when the
caller hasn't already set it. This covers mount chunk uploads,
filer_copy, filersink, mq/logstore, broker_write, gateway_upload,
and nfs — all the UploadWithRetry paths.
* fix(assign): pass real ExpectedDataSize at every assign call site
After removing the mount fid pool, per-chunk AssignVolume calls went
out with ExpectedDataSize=0, making the master fall back to its 1 MB
DefaultNeedleSizeEstimate. That's still an over-estimate for small
writes. Thread the real payload size through every remaining assign
site so RecordAssign charges effectiveSize accurately and stops
prematurely marking volumes full.
- filer: assignNewFileInfo now takes expectedDataSize and stamps it
on both primary and alternate VolumeAssignRequests. Callers pass:
- SSE data-to-chunk: len(data)
- copy manifest save: len(data)
- streamCopyChunk: srcChunk.Size
- TUS sub-chunk: bytes read
- saveAsChunk (autochunk/manifestize): 0 (small, size unknown
until the reader is drained; master uses 1 MB default)
- filer gRPC remote fetch-and-write: ExpectedDataSize = chunkSize
after the adaptive chunkSize is computed.
- ChunkedUploadOption.AssignFunc gains an expectedDataSize parameter;
upload_chunked.go passes the buffered dataSize at the call site.
S3 PUT assignFunc stamps it on the AssignVolumeRequest.
- S3 copy: assignNewVolume / prepareChunkCopy take expectedDataSize;
all seven call sites pass the source chunk's Size.
- operation.SubmitFiles / FilePart.Upload: derive per-fid size from
FileSize (average for batched requests, real per-chunk size for
sequential chunk assigns).
- benchmark: pass fileSize.
- filer append-to-file: pass len(data).
* fix(assign): thread size through SaveDataAsChunkFunctionType
The saveAsChunk path (autochunk, filer_copy, webdav, mount) ran
AssignVolume before the reader was drained, so it had to pass
ExpectedDataSize=0 and fall back to the master's 1 MB default.
Add an expectedDataSize parameter to SaveDataAsChunkFunctionType.
- mergeIntoManifest already has the serialized manifest bytes, so
it passes uint64(len(data)) directly.
- Mount's saveDataAsChunk ignores the parameter because it uses
UploadWithRetry, which already stamps len(data) on the assign
after reading the payload.
- webdav and filer_copy saveDataAsChunk follow the same UploadWithRetry
path and also ignore the hint.
- Filer's saveAsChunk (used for manifestize) plumbs the value to
assignNewFileInfo so manifest-chunk assigns get a real size.
Callers of saveFunc-as-value (weedfs_file_sync, dirty_pages_chunked)
pass the chunk size they're about to upload.
|
||
|
Chris Lu
|
213b6c3107 |
fix(mount): count manifest sizes in merge condition to prevent accumulation (#9101)
* fix(mount): count manifest sizes in merge condition to prevent manifest accumulation shouldMergeChunks only counted non-manifest chunk sizes toward the bloat threshold, so overlapping manifests accumulated undetected across flush cycles. During sustained random writes (e.g. fio), each metadata flush compacts non-manifest chunks and may group them into a new manifest via MaybeManifestize, while carrying forward all existing manifests. Since the merge condition only checked non-manifest totals against 2x file size, the growing pile of redundant manifests never triggered a merge. In a real workload: 4 GB file, 25 manifests each covering ~4 GB (107 GB manifest data on volume servers), but shouldMergeChunks saw only 4.2 GB of non-manifest chunks vs the 8.6 GB threshold -- no merge. Fix: include manifest coverage sizes in totalChunkSize. This correctly detects the 111 GB total vs 8.6 GB threshold and triggers the merge, which re-reads the file as clean chunks and lets the filer's MinusChunks (which resolves manifests) garbage-collect all redundant sub-chunks. * test(mount,filer): add manifest chunk correctness tests Filer-level tests (filechunk_manifest_test.go): - TestManifestRoundTripPreservesChunks: create -> serialize -> deserialize preserves fileId, offset, size, and timestamp for all sub-chunks - TestCompactResolvedOverlappingManifests: two manifests covering the same range, older sub-chunks correctly identified as garbage after compaction - TestDoMinusChunksWithResolvedManifests: old manifest sub-chunks detected as garbage when compared against new clean chunks (mirrors filer cleanup) - TestManifestizeSmallBatchWithRemainder: batch=3 with 7 chunks produces 2 manifests + 1 remainder, all data recoverable after resolve - TestCompactMultipleOverlappingManifestGenerations: 5 generations of full-file manifests, compaction keeps only the newest generation - TestManifestBloatDetection: with N overlapping manifests, merge triggers at N >= 2 (stored > 2x file size) Mount-level test (weedfs_file_sync_test.go): - TestFlushCycleManifestAccumulation: simulates the flushMetadataToFiler pipeline over multiple cycles with a small manifest batch (5 chunks). Verifies merge triggers at cycle 2 when accumulated manifests exceed the 2x threshold. Uses SeparateManifestChunks + CompactFileChunks + shouldMergeChunks to mirror the real code path. |
||
|
chrislu
|
26dbc6c905 | move to https://github.com/seaweedfs/seaweedfs | ||
|
Chris Lu
|
eb7929a971 | rename filer2 to filer |