* pb: add multipart concurrency fields to RemoteConf and tier move requests
RemoteConf gains upload_concurrency/download_concurrency (0 = client
default); VolumeTierMoveDatToRemote/FromRemote requests gain a
concurrency field (0 = backend default).
* remote storage: honor RemoteConf upload/download concurrency in s3 and azure clients
s3 client: ReadFile passes conf download_concurrency to the downloader,
WriteFile uses upload_concurrency for the uploader; previously
hard-coded 1 upload / 5 download parts. 0 keeps defaults. Same for
azure client.
* storage: plumb concurrency through backend interface and tier upload/download
BackendStorage.CopyFile/DownloadFile take a concurrency hint (<=0 =
backend configured default); s3 backend reads
upload_concurrency/download_concurrency from scaffold config with
parseConcurrency fallback, rclone updated to the new signature. Tier
move gRPC handlers forward the request concurrency to the backend.
* shell: -upload_concurrency/-download_concurrency for remote.configure, -concurrent for volume.tier
remote.configure exposes upload/download concurrency persisted into
RemoteConf; volume.tier move/evict commands forward -concurrent to the
tier move requests. Documented in master-cloud.toml scaffold.
* test: cover concurrency propagation in remote tier integration test
* remote.configure: merge existing config on partial update
Load the stored RemoteConf before saving so a partial update (e.g. only
-upload_concurrency) preserves credentials, endpoints, and type instead
of replacing them with new-config defaults. Only treat a confirmed
ErrNotFound as a new configuration; propagate all other load errors so a
transient filer failure does not overwrite stored settings.
On a type transition, reset backend-specific fields to the destination
type's new-config defaults rather than inheriting the old backend's
empty values. Bound configured concurrency to a sane maximum.
* remote storage: honor configured download concurrency in S3 and Azure
ReadFileWithConcurrency now resolves a zero request override against the
client's configured download_concurrency (new downloadConcurrency()
helpers), so the remote-mount/cache read path honors
RemoteConf.DownloadConcurrency instead of the hard-coded default.
Azure also clamps the resolved value to math.MaxUint16 regardless of
whether the fallback was used, preventing uint16 wraparound when a
configured value exceeds 65535.
* shell: rename -concurrent to -concurrency and validate tier transfer bounds
Rename the -concurrent flag to -concurrency across volume.tier.upload,
volume.tier.download, and volume.tier.compact to match the proto field and
RemoteConf field names. Add validateTierConcurrency to reject values that
would wrap int32 or exceed a 1024 cap before constructing the request.
* server: clamp tier move concurrency in gRPC handlers
Add clampTierConcurrency to both VolumeTierMoveDatToRemote and
VolumeTierMoveDatFromRemote handlers so a direct gRPC caller cannot spawn
an unbounded number of network workers.
* trim verbose comments added with concurrency feature
Remove redundant doc comments on the backend interface, rclone backend,
s3_backend parseConcurrency, and test helpers that restated the obvious.
* remote.configure: apply type defaults before re-parse so explicit flags win
applyTypeDefaults ran after the second flag parse, overwriting explicit
destination flags (e.g. -s3.region=eu-west-1) with new-config defaults.
Move the type-transition default reset before the re-parse so user-supplied
flags override the destination defaults.
* remote.configure: only treat explicit -type as a type transition
The first parse defaults -type to s3, so a concurrency-only update on an
existing non-S3 config captured requestedType=s3 and wrongly triggered a
type transition, resetting the stored backend to S3. Use fs.Visit to
detect whether -type was explicitly supplied; an omitted -type keeps the
stored backend.
---------
Co-authored-by: Jack Meredith <9480542+jackusm@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* volume: gate the admin RPCs that only shell and workers call
checkGrpcAdminAuth covered 19 of the 48 VolumeServer RPCs, so an operator who
sets -whiteList expecting it to cover the gRPC surface gets partial coverage.
Extend it to ten that mutate state and are only ever called by the shell or a
worker: SetState, VolumeCopy, the EC generate/rebuild/copy/unmount/to-volume
pair, both tier moves, and VolumeTailReceiver. That is safe because the same
callers already reach gated RPCs today -- VolumeMarkReadonly, VacuumVolume*,
VolumeEcShardsDelete, VolumeDelete -- so a whitelist deployment already lists
those hosts. Nothing here is on a master or peer path, which is what made the
earlier fail-closed gate break multi-host clusters.
The split is by caller rather than by blast radius: the guard matches a peer IP
against the whitelist, and a whitelist holds masters, shell hosts and workers,
not every peer volume server. Gating a call one volume server makes to another
would break replication, EC and tiering, so those stay open.
Two test fakes embedded a nil grpc.ServerStream and only implemented Send;
they now implement Context, which the streaming RPCs read to authorize.
* volume: fail the build when a gRPC method skips the admin gate
The admin gate is an opt-in list in a 48-method service, which is how it
drifted down to covering 19 of them: nothing tied adding an RPC to deciding
whether it needed the gate.
Parse volume_server.proto, walk the AST of every *VolumeServer method, and
require each RPC to either call checkGrpcAdminAuth or appear in
ungatedVolumeServerRPCs with the reason it stays open. A stale entry naming an
RPC that no longer exists fails too, so the list can't quietly stop exempting
anything.
The exemptions are the cluster-internal calls -- replica sync, EC shard
distribution, vacuum reads, backup, tailing -- plus the read-only and liveness
RPCs. Closing the cluster-internal ones needs a peer identity rather than an
IP whitelist; recording them here makes that a visible decision instead of an
omission.
The AST walk also corrects the count: a line-window scan credits
VacuumVolumeCheck and VolumeServerStatus with a neighbouring function's guard.
* fix(tier): preserve volume data modification time
* fix(tier): best-effort restore of data mtime on download
A failed Chtimes should not abort an otherwise complete tier-down; warn
and continue, matching the EC copy path.
* fix(tier): preserve volume data mtime in rust volume server
Mirror the Go fix: store the source .dat mtime on upload instead of the
upload time, and restore it on the downloaded .dat. Without this a
tiered-then-restored volume loads last_modified_ts_seconds from the
upload/download time, extending its TTL across a restart or remount.
* fix(tier): read source mtime via DiskFile.GetStat()
GetStat() is nil-safe when the backend is closed concurrently and skips a
redundant stat syscall; its cached modTime is the on-disk mtime a reload
reads, since every .dat write or Chtimes is followed by a DiskFile (re)open.
* fix(tier): surface mtime-restore failures on rust tier-down
set_file_mtime now returns io::Result; the tier-down path warns on a
failed restore instead of dropping it silently, so a wrong local .dat
mtime (and the TTL drift it causes) is observable. Matches the Go
download. The EC copy path keeps its best-effort silence.
* Boostrap persistent state for volume servers.
This PR implements logic load/save persistent state information for storages
associated with volume servers, and reporting state changes back to masters
via heartbeat messages.
More work ensues!
See https://github.com/seaweedfs/seaweedfs/issues/7977 for details.
* Block RPC operations writing to volume servers when maintenance mode is on.
