mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-19 02:15:02 +01:00
c7d8b106d50f72da1ab10df8aee3cd2fa080caa4
7
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
a3b2fd722d |
mount: report opens for reading with TouchAccessTime (-accessReport) (#12721)
* mount: report opens for reading with TouchAccessTime (-accessReport) An open answered from the metadata cache never reaches the filer, so a filer that tracks access cannot see that a cached file is still being read. With -accessReport, the mount sends one TouchAccessTime per open for reading, with the open time, whether the open was answered from the cache or not. Write-only opens are not reported. Reports go out in the background and never hold up the open: a report that does not fit in the queue is dropped, and drops and failures are logged once a minute. A filer that answers Unimplemented pauses reporting for 10 minutes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: address review of the access report - Drop reports still queued when reporting pauses; the filer that answered Unimplemented would answer them the same way. - Bound each filer attempt to 2s, so a filer that accepts the connection but never answers leaves time for the next one, and keep using the filer that answered. - Fail over only when a filer cannot be reached. Any answer, Unimplemented included, ends the walk. - Report an O_CREAT open of an existing file with read access, which comes through Create instead of Open. A new file has nothing to read and is not reported. - Stop the reporter on unmount: stop ends the workers and cancels the report in flight. - Say in the flag help that it has no effect unless the filer implements TouchAccessTime. - Tests no longer wait a fixed time to see that nothing else was sent; they look at the queue of a reporter that is not started. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: fail an access report over only when the filer did not answer A status the filer sent ends the walk even when its message contains a word the transient classifier matches, so an Unimplemented answer still pauses reporting and a rejected request is not resent elsewhere. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: keep a filer selection made while an access report failed over A report that reached a later filer moves the mount there only if the selection it started from is still current, so a slower report cannot undo another call's failover. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: share an access report's time with the filers not yet tried Each attempt is capped by an equal share of the report time left, so a row of stalled filers cannot use up the report before a healthy one later in the list is tried. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: report the path an open for reading came through A create that opens an existing hard link reports the name it was given rather than the inode's first known name. Open reads the inode's path from the inode table rather than the handle's saved path, which other handle operations may rewrite concurrently. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: check the access report stop and failover tests without races The post-stop report is checked on a reporter without workers, so no worker can take an accepted report before the check, and the failover test asserts that the stalled filer was tried first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * pb: keep the shared channel when the peer answers with its own status gRPC never raises InvalidArgument, NotFound, AlreadyExists, FailedPrecondition, OutOfRange or DataLoss itself, so such a status is the peer's answer. It fell through to the string matching, and a message naming a path like /transport closed the cached channel and cancelled every other call on it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: say which filer selection an access report keeps Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: bound the in-flight call in the rejected-report channel test The call held in flight went through touchAccessTime, whose 2 s attempt cap could expire on a slow runner and fail the test on a healthy channel. It now uses callFiler with the test deadline. The wait for it to reach the filer also ends if it returns early or the deadline passes, and the blocked handler is released on every exit path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: report each file at most once per -accessReport.interval Every read open used to send a TouchAccessTime, adding a filer metadata call per open. A path reported within the interval (default 1h) is now skipped; the last report times live in a bounded LRU. 0 keeps reporting every open. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: report an open again once the file was modified since its last report Within -accessReport.interval a read open was skipped even after a write, so the filer kept an atime older than the mtime until the interval ran out. relatime records that first read after a write; do the same by comparing the entry's mtime with the last report. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: keep the report time apart from the mtime it saw Recording max(now, mtime) as the last report let a file dated a day ahead suppress reports for a day plus the interval. Record when the report was sent and the mtime the entry had, and treat any change of that mtime as a modification, so neither clock can stretch or shorten the interval. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: MapleWhiteZZZ <18653045+MapleWhiteZZZ@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
|
50b388771a |
s3: stop one abandoned request from cancelling every concurrent upload (#10948)
* grpc: a non-cancellable context is no evidence of a stale channel shouldInvalidateConnection only invalidates on Canceled/DeadlineExceeded while the context handed to WithGrpcClient is still live, so that an RPC timing out on its own does not close the shared cached ClientConn and cancel every other in-flight RPC on it. context.Background()/TODO never expire, so Err() stays nil forever and that guard always answered "invalidate" - and Background is what almost every caller passes, the S3 gateway included. One S3 request whose RPC rode an abandoned HTTP request context therefore closed the shared filer connection, and every multipart part in flight died with "the client connection is closing", surfacing to the client as 400 InvalidRequest. Only a cancellable context bounds an RPC attempt, so require one before reading it. A genuinely stale channel (a peer restart behind a stable L4 endpoint) surfaces as Unavailable, which invalidates on its own branch. * grpc: a bystander of a connection teardown is not a stale-channel witness gRPC raises ErrClientConnClosing locally, before an RPC reaches the wire, when this process has already closed the ClientConn. Every caller that touches a channel during another goroutine's teardown gets it, so reading it as a stale-channel signal lets one teardown re-arm itself across the whole herd of callers it just cancelled. The cached-connection version check keeps those callers from closing a replacement channel, but the streaming path invalidates by address alone and has no such guard. * grpc: end a stream without dropping the peer connection under it A streaming caller gets its own ClientConn, but on any error it also drops the cached non-streaming ClientConn every request handler shares with that peer, to recover a peer restart hidden behind a stable L4 endpoint. Any error includes the ordinary ones: a metadata subscription that reached its stop point, a follow callback that refused an event, a caller that gave up. The S3 gateway follows filer metadata on such a stream and reconnects forever, so each ordinary end of it cancelled every S3 request in flight against the filer. Drop the shared channel only for errors that say the peer went away, which is what invalidation is for. * test: close the connections the cascade tests leave cached Each test swaps in a fresh connection cache and restores the previous one, dropping its own entries without closing them, so the ClientConn's transport and reconnect goroutines outlive the fake filer they dialed. * grpc: say why ErrClientConnClosing's deprecation notice does not apply It points at codes.Canceled, which is the code this function exists to disambiguate. Only the message distinguishes a teardown a caller merely walked into, so the sentinel stays. |
||
|
|
f643893891 |
fix(master): shed assign load when volume growth is already in flight (#10121)
Under a herd of concurrent assigns with no writable volume, Assign spun PickForWrite for the full 10s timeout, pinning a goroutine per request and starving the master of the cycles it needs to process growth and answer heartbeats. When growth is the relevant remedy and already in flight, stop spinning: if free space exists, shed with a fast retryable error so clients back off and retry once growth lands; if the cluster is out of space, fail fast with the real out-of-space error instead of masking it as retryable. The gRPC shed uses ResourceExhausted, not Unavailable: operation.Assign retries it, but the client connection layer doesn't treat it as a dead channel, so a per-request shed across a herd doesn't tear down the shared master connection and cancel every other in-flight assign. The HTTP dirAssignHandler sheds with 503 + Retry-After. |
||
|
|
2386fa550a |
grpc: don't tear down the shared master connection on a caller's own timeout (#9775)
A Canceled/DeadlineExceeded from the caller's per-request context was treated like a dead channel: it closed the shared cached ClientConn and cancelled every other in-flight RPC on it with "the client connection is closing". Under a burst of concurrent chunk assigns (e.g. a large S3 multipart upload) one slow assign hitting its 10s attempt timeout could poison the connection for all the rest, cascading into a flood of 500s. Thread the caller's context into shouldInvalidateConnection and only invalidate on Canceled/DeadlineExceeded while that context is still live, which isolates the genuine stale-channel signal (a peer restart behind a k8s Service VIP). To carry the context, add a ctx parameter to the existing WithGrpcClient, WithMasterClient, and WithMasterServerClient; the master assign and volume-lookup paths pass their per-attempt context and every other caller passes context.Background(). |
||
|
|
a64483885c |
fix(pb): skip Unix-socket gRPC registration on Windows (#9430) (#9434)
The same-host gRPC fast path registered /tmp/...sock paths and dialed
them with net.Listen("unix", ...) / net.Dial("unix", ...). Those paths
are POSIX-only, so on Windows the listener failed at startup and every
local gRPC call routed through the registered port lost its transport.
Gate RegisterLocalGrpcSocket to return early on Windows. ServeGrpcOnLocalSocket
and resolveLocalGrpcSocket already short-circuit when no socket is registered,
so all same-host RPCs fall back to TCP without touching any callsite.
|
||
|
|
135af25b55 |
fix(grpc): require host match before routing dials to local Unix socket (#9254) (#9257)
* fix(grpc): require host match before routing dials to local Unix socket (#9254) resolveLocalGrpcSocket keyed the Unix-socket hijack on port alone, so a remote peer reusing a local gRPC port (e.g. a standalone `weed volume` defaulting port.grpc=17334 against a `weed server` whose in-process volume socket is also on 17334) had its inbound RPCs silently rerouted to the local socket. In a cross-DC replication=100 cluster this surfaced as persistent volume-grow failure: both AllocateVolume RPCs landed on the local volume server, the second returned "Volume Id N already exists!", the grow rolled back, and S3 PUTs to the new collection returned 500. Track a per-port set of host strings that count as "this machine" and require the dial host to be in that set before redirecting. Loopback aliases (localhost, 127.0.0.1, ::1, "") are always included so same-process dials via loopback still take the socket fast path. * test(grpc): cover empty-host bare-port dial in local-socket regression test The empty alias is registered explicitly so SplitHostPort outputs like ":17334" (which can occur when a caller dials a bare port) take the local-socket fast path. Add a case so that path is exercised. |
||
|
|
da2e90aefd |
fix(mount): sanitize non-UTF-8 filenames; keep marshal errors per-request (#9207)
* fix(mount): sanitize non-UTF-8 filenames; keep marshal errors per-request (#9139) A single file with invalid-UTF-8 bytes in its name (e.g. a GNOME Trash "partial" like \x10\x98=\\\x8a\x7f.trashinfo.9a51454f.partial) made every FUSE-initiated filer RPC fail with: rpc error: code = Internal desc = grpc: error while marshaling: string field contains invalid UTF-8 and then produced an avalanche of "connection is closing" errors on unrelated LookupEntry / ReadDirAll / UpdateEntry calls, causing the volume-server QPS dips reported in #9139. Root cause is twofold: 1. Proto3 `string` fields require valid UTF-8, but the FUSE kernel passes raw name bytes. Create/Mknod/Mkdir/Unlink/Rmdir/Rename/Lookup/Link/ Symlink all forwarded those bytes directly into CreateEntryRequest.Name, DeleteEntryRequest.Name, StreamRenameEntryRequest.{Old,New}Name and Entry.Name. saveDataAsChunk also copied the FullPath into AssignVolumeRequest.Path unchecked. 2. When the marshal failed, shouldInvalidateConnection treated the resulting codes.Internal as a connection problem and dropped the shared cached ClientConn — canceling every other in-flight RPC on it. Fix: - Add sanitizeFuseName (strings.ToValidUTF8 with '?' replacement, matching util.FullPath.DirAndName) and make checkName return the sanitized name. Apply at every FUSE entry point that passes a name to the filer RPC, including Unlink/Rmdir (which did not previously call checkName) and both oldName/newName in Rename. Add a backstop scrub for AssignVolumeRequest.Path so async flush paths cannot reintroduce invalid bytes from a pre-sanitization cached FullPath. - In weed/pb.shouldInvalidateConnection, detect client-side marshal errors via the gRPC library's "error while marshaling" prefix and return false: the connection is healthy, only the request is bad. Refs: https://github.com/seaweedfs/seaweedfs/issues/9139#issuecomment-4301184231 * fix(mount,util): use '_' for invalid-UTF-8 replacement (URL-safe) Sanitized filenames flow downstream into HTTP URLs (volume-server uploads, filer HTTP API, S3/WebDAV gateways). '?' is the URL query-string delimiter and would split the path the first time the name lands in one, so swap every invalid-UTF-8 replacement to '_'. This covers the two pre-existing sites in weed/util/fullpath.go as well, keeping all paths sanitized the same way. * refactor(pb): detect client-side marshal errors via errors.As, not substring Replace the raw `strings.Contains(err.Error(), ...)` check with a type-based carve-out: use errors.As against the `GRPCStatus() *Status` interface to pull the original Status out of any fmt.Errorf("...: %w") wrapping, then match the library-owned "grpc:" prefix on that Status's Message. Why not errors.Is against a proto-level sentinel: gRPC's encode() collapses the inner proto error with "%v" (stringification) before wrapping it in a Status, so the original error type does not survive into the caller. The Status itself is the structural signal that does survive. Why not status.FromError: when the caller wraps the Status error with fmt.Errorf("...: %w", ...), status.FromError rewrites Status.Message with the full err.Error() of the outermost wrapper, which defeats a prefix check on the library-owned message. errors.As gives us the original Status whose Message is still verbatim from the gRPC library. A new test asserts that a plain errors.New("grpc: error while marshaling: …") — i.e. the same text attached to something that is NOT a gRPC status — does not short-circuit invalidation, so we never silently keep a cached connection alive based on a coincidental substring match. * refactor(util): centralize UTF-8 sanitization; add FullPath.Sanitized Addresses review feedback on PR #9207. Nitpick: every invalid-UTF-8 replacement across the codebase (DirAndName, Name, mount.sanitizeFuseName, the weedfs_write.go backstop) now goes through a single util.SanitizeUTF8Name helper, so the replacement char ('_' — URL-safe) is chosen in one place. Outside-diff: three proto fields took raw FullPath strings that could break marshaling if an entry ever carried invalid UTF-8 (CreateEntryRequest.Directory in Mkdir, DeleteEntryRequest.Directory in Unlink, AssignVolumeRequest.Path in command_fs_merge_volumes). The reviewer's suggested fix — using DirAndName() — would have silently changed Directory from parent to grandparent, because DirAndName sanitizes only the trailing component. Added FullPath.Sanitized(), which scrubs every component, and applied it at the three sites. Exposure is narrow in practice (FUSE-boundary sanitization and the gRPC-side isClientSideMarshalError carve-out already cover the #9139 cascade), but the defense-in-depth is cheap and consistent with the existing AssignVolume backstop. New tests in weed/util/fullpath_test.go document: - SanitizeUTF8Name: valid UTF-8 passes through unchanged; invalid bytes become '_' (not '?', which is URL-special). - FullPath.Sanitized: scrubs bytes in any component, not just the last. - FullPath.DirAndName: dir remains raw on purpose — callers needing a clean full path must use Sanitized(). The test pins this behavior so it is not accidentally "fixed" in a way that changes the (dir, name) semantics callers depend on. |