Github-Mirror/seaweedfs
1
0 Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-10-06 14:31:57 +02:00
Code Issues Packages Projects Releases Wiki Activity
Files
483dd4b12e9940d368dee3a585d0ebfb5aba3dc7
seaweedfs/weed/command/filer.go
T
Khris RichardsonClaude Opus 5.5Chris Lu
 30069f3e45 iam: manage OIDC providers and roles over the filer IAM gRPC service (#11523) 
* s3/iam: manage roles through the IAM API, with an opt-in persistent role store

Roles could only come from the IAM config file: the S3 server pinned the
role store to memory and the embedded IAM API had no role actions, so a
role could not be created, retrusted or revoked without editing the file
and restarting every gateway.

Role store
- Read the `roleStore` key (the IAMConfig field already existed). With an
  IAM config file the default stays memory; with none it is the filer, as
  for OIDC providers, so zero-config clusters keep runtime-created roles.
- Roles from the IAM config file never go into a persistent role store,
  which outlives the file and may be shared by S3 servers with different
  files. They are served from memory beneath the store, as OIDC providers
  are: a stored role of the same name takes precedence, and deleting it
  restores the file's. A config-file role cannot be changed or deleted
  through the API (UnmodifiableEntity), and removing one from the file
  removes it at the next start. An in-memory store holds them as records,
  as before. They have no creation time, so CreateDate is omitted rather
  than reporting when this server started. SetRoleStore installs a store
  the same way, so a store set after startup keeps the config-file roles,
  as SetOIDCProviderStore does for providers.
- Watch /etc/iam/roles and drop the cached role definitions on change. The
  cached filer store otherwise serves a peer's stale role for up to its 5m
  TTL, which keeps a revoked trust policy in force on the other gateways.
- Role stores wrap ErrRoleNotFound for a missing role; the filer store
  used to report any failed lookup as "role not found". CreateRole proceeds
  only on a confirmed absence, so an unreadable store cannot let it write
  over an existing role.

IAM actions
- CreateRole, GetRole, ListRoles, DeleteRole, UpdateAssumeRolePolicy,
  AttachRolePolicy, DetachRolePolicy, ListAttachedRolePolicies. The reads
  are allowed in read-only mode.
- A role defined in the config file is reloaded from it at every start, so
  changing or deleting it through the API is refused (UnmodifiableEntity)
  rather than silently reverted.
- DeleteRole with policies attached is refused (DeleteConflict), as on AWS.
- Role names follow AWS's rules ([\w+=,.@-]{1,64}); a role is stored as
  <name>.json in the filer, so this also keeps a name from leaving the role
  store's directory. At most 10 managed policies per role (AWS's default
  quota; MaxManagedPoliciesPerUser is 10 too), LimitExceeded beyond.
- DeletePolicy is refused (DeleteConflict) while a role attaches the
  policy, as it already is for users and groups: roles attach policies by
  name, so a policy created later under the deleted one's name would
  otherwise take effect on the role.
- Role paths other than "/" and role tags are not stored, so they are
  refused rather than dropped.

Role IDs and sessions
- Roles get a unique RoleId when first stored (random, AWS AROA form),
  kept across updates; a config-file role gets a stable ID derived from its
  name, since it is created again at every start.
- Sessions issued through AssumeRoleWithWebIdentity, AssumeRoleWithCredentials
  and AssumeRole carry the role's ID (claim "rid"), and a request under a role
  whose current ID differs is denied. Resolving a session's policies by role
  name let a session outlive its role: once a role was deleted, a role later
  created under the same name — with a different trust policy and different
  policies — revived every unexpired session of the old one with the new
  role's permissions. Sessions issued before this change carry no ID and are
  unaffected until they expire.

Integration test (test/s3/iam, run with `make start-services`):
TestWebIdentityWithProviderAndRoleManagedThroughIAMAPI configures an OIDC
provider, a managed policy and a role entirely through the IAM API against a
JWKS served by the test, then checks the trusted subject gets credentials
scoped to the attached policy; another subject, a token signed by another
key, an unsigned token and a token for another audience are refused; and UpdateAssumeRolePolicy moves the
trust at once.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* iam: manage OIDC providers and roles over the filer IAM gRPC service

The filer's SeaweedIdentityAccessManagement service covers users, access
keys, policies and service accounts, but not the OIDC providers and roles
that STS web-identity federation needs. A controller that already manages
IAM over this service (seaweedfs-operator's S3OIDCProvider) has no
transport for them; its swadmin client returns ErrOIDCNotWired and names
this as the recommended fix.

- PutOIDCProvider / GetOIDCProvider / DeleteOIDCProvider / ListOIDCProviders
  and PutRole / GetRole / DeleteRole / ListRoles.
- They write the filer-backed stores at their default paths, which S3
  servers read when configured with a filer-typed "oidcProviderStore" and
  "roleStore"; the S3 servers' /etc/iam subscription applies changes
  without a restart.
- Put is an upsert, so a controller can reconcile to it. Deleting a
  provider or role that does not exist returns NotFound, as DeleteUser does
  for a user; clients treat that as already deleted. The provider's account
  ID travels in the request, since the filer does not know the STS
  accountId.
- PutRole applies the IAM API's rules: AWS role names, at most 10 managed
  policies.
- An S3 server serves the roles and providers of its own IAM config file
  ahead of the store, so a stored entry with the same name has no effect
  on that server.
- PutRole keeps a replaced role's RoleId and gives a role created anew a
  fresh one, so sessions of a deleted role do not carry over to a later role
  of the same name.
- DeletePolicy returns FailedPrecondition while a role attaches the policy
  (see the IAM API's DeleteConflict in the previous change). DeletePolicy on
  this service still does not check user attachments, which predates this.
- PutOIDCProvider requires an https issuer (http only for a loopback host):
  STS fetches the issuer's signing keys from it, so over plain HTTP anyone
  on the network path could substitute their own.
- The OIDC provider and role RPCs refuse to run on an unauthenticated
  service (FailedPrecondition until jwt.filer_signing.key is set). Users and
  policies keep the service's opt-in auth, but these grant STS access
  outright: otherwise anyone who can reach the port could register an issuer
  they control, create a role trusting it, and exchange a token for S3
  credentials. The filer's unauthenticated notice becomes a warning that says
  so.
- A store that cannot be read is Unavailable, never "not found", so a Put
  never writes over an entry it could not see.
- Validation is shared with the IAM API through PrepareRoleDefinition and
  PrepareOIDCProviderRecord.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* s3/iam: bind every role session to its role, and change roles atomically

Review follow-ups.

Session binding
- The role-ID check ran only when a session carried no policy names, and
  AssumeRole embeds the role's attached policies, so those sessions kept
  their permissions after the role was deleted or recreated. The check
  now runs for every session carrying a role ID, before policy selection.
- A named role that cannot be resolved at issuance gets no session,
  instead of one with no role ID (which nothing binds).
- A config-file role's ID is derived from its name and trust policy, not
  the name alone: a different role put in the file under the same name
  gets a new ID, while an unchanged role keeps its sessions across restarts.

Role writes
- RoleStore gains UpdateRole, a read-modify-write that lands only if the
  role is unchanged since the read, and otherwise re-reads and retries. The
  filer store uses the filer's write conditions (IF_NOT_EXISTS for a new
  role, IF_ENTRY_EQUAL otherwise). CreateRole, UpdateAssumeRolePolicy and
  Attach/DetachRolePolicy all go through it, so two gateways no longer
  overwrite each other's changes, a change racing a delete no longer
  writes the role back, and of two concurrent creates one gets
  EntityAlreadyExists.
- The filer store's ListRoles pages past 1,000 entries and fails on a
  broken stream instead of returning what arrived, so DeletePolicy's
  attachment check sees every role. ListRoles skips a role deleted between
  listing and reading it.
- CreateRole validates first; a failed write is ServiceFailure, not
  InvalidInput. Any Tags.* parameter is refused, not only the first key.
- ExecuteAction's skipPersist covers the S3ApiConfiguration only; the
  comment now says so. Role and OIDC provider actions write their own stores.

Each fix has a test that fails without it. Against a real filer with two
gateways, concurrent AttachRolePolicy calls lost 1-4 of 8 attachments per
run before this change and none after.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* iam: PutRole changes roles atomically and checks its ARN; https issuers' keys stay on https

Review follow-ups on top of the role-store changes.

- PutRole goes through RoleStore.UpdateRole, so the decision to keep an
  existing role's ID or mint a new one is made against the role as it is
  when written. A PutRole racing a DeleteRole can no longer write the
  deleted role back with its old ID, which would revive its sessions. A
  failed store read or write is Unavailable.
- PutRole refuses a role_arn that does not name the role: STS resolves a
  role by the name in the ARN it is given.
- PutOIDCProvider requires an https issuer, but discovery could still name
  a plain-http jwks_uri, and a key fetch could be redirected to http. For
  an https issuer, a non-https jwks_uri from discovery is refused (the
  issuer's own /.well-known/jwks.json is used instead), and the client
  that fetches discovery and keys refuses any https-to-http redirect. An
  operator-set jwksUri is left as configured.

Each has a test that fails without its guard.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* s3/iam: one role snapshot per decision; DeleteRole is atomic; watch a custom role store path

Review follow-ups.

- Authorization evaluates the policies of the role definition the session's
  binding was checked against, instead of reading the role again: a role
  replaced in between cannot lend a session its policies.
- AssumeRole and AssumeRoleWithLDAPIdentity issue the session from the
  definition whose trust admits the caller (IAMManager.ResolveRoleForPrincipal),
  and take its ID, duration cap and embedded policies from that same
  definition. A role replaced after the caller's trust check by one that does
  not trust the caller now yields AccessDenied, not a session bound to the
  replacement.
- A RoleUpdate that returns nil deletes the role, on the same condition as a
  write: the filer store deletes with ObjectTransaction on IF_ENTRY_EQUAL,
  routed and locked like the conditional CreateEntry. DeleteRole decides
  against the role it deletes, so a policy attached meanwhile on another
  server is a DeleteConflict, and a delete never removes a role written
  after its check.
- S3 servers watch the role store's configured basePath, not only
  /etc/iam/roles, so a custom path also drops peers' cached roles on change.

Each has a test that fails without it. Live against a real filer: DeleteRole
refuses while a policy is attached and removes the entry once detached; all
test/s3/iam CI stages pass.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* s3/iam: state which roles DeletePolicy's attachment check can see

RolesAttachingPolicy sees the stored roles and this server's config-file
roles. A role defined only in another server's IAM config file is invisible
to it, so a config-file role that attaches a managed policy is protected
only on the servers whose file defines it. The doc comment now says so and
how to avoid it: keep such roles in every server's file, or attach only
config-file policies to config-file roles.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* iam: note that a role store set after startup is not watched for peer changes

S3 servers build their metadata watch list once, at startup, from the role
store installed then. SetRoleStore's doc now says that a filer-backed store
installed later with a different basePath is not watched, so peers' changes
to it reach this server's cached roles only when the cache expires.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* iam: DeleteRole deletes only the role it saw; issuer URLs are bare

Review follow-ups.

- The filer IAM service's DeleteRole looked the role up, then deleted by
  name, so a PutRole landing in between had its new definition deleted. It
  now deletes through RoleStore.UpdateRole, conditional on the entry it
  read. If the role was replaced meanwhile, it returns Aborted rather than
  deleting the replacement, and the caller decides again.
- PutOIDCProvider refuses an issuer URL with userinfo, a query or a
  fragment. The provider's ARN comes from host and path alone, while STS
  matches a token's iss claim against the stored URL exactly, so such a
  provider shared the bare issuer's ARN and matched no token. A loopback
  "localhost" is now matched without regard to case.

Both have tests that fail without them.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* iam: write OIDC providers atomically over the filer IAM gRPC service

PutOIDCProvider read the record, then stored unconditionally; a racing
DeleteOIDCProvider left the put's stale read merged into the rewritten
record. DeleteOIDCProvider read, then deleted unconditionally; a racing
PutOIDCProvider's newer record could be removed instead. These are the
races the role RPCs closed with UpdateRole.

OIDCProviderStore gains UpdateProvider with the same contract: memory
under its lock, filer as a conditional write (IF_ENTRY_EQUAL /
IF_NOT_EXISTS) or conditional delete retrying a changed entry.
PutOIDCProvider merges the fields the request cannot carry against the
record as it is written; DeleteOIDCProvider aborts rather than delete a
record replaced meanwhile.

isRoleWriteConflict is renamed isEntryWriteConflict — the conditional-
write check is shared by both stores now.

* iam: guard PutRole against a nil credential manager, fix its doc comment

PutRole read attached policies through s.credentialManager without the
nil check its sibling handlers make, so a server built without one
panicked on a PutRole naming a policy. It now fails the call as
FailedPrecondition like the others.

The doc comment also had the store/static precedence backwards: a stored
role shadows a same-named config-file role (as the overlay serves it),
not the other way around.

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-10-03 08:36:19 +08:00

663 lines
31 KiB
Go
Raw Blame History

package command
import (
"context"
"crypto/tls"
"crypto/x509"
"fmt"
"net"
"net/http"
"os"
"runtime"
"sort"
"strings"
"sync"
"time"
"github.com/spf13/viper"
"google.golang.org/grpc/reflection"
"github.com/seaweedfs/seaweedfs/weed/credential"
_ "github.com/seaweedfs/seaweedfs/weed/credential/filer_etc"
_ "github.com/seaweedfs/seaweedfs/weed/credential/memory"
_ "github.com/seaweedfs/seaweedfs/weed/credential/postgres"
"github.com/seaweedfs/seaweedfs/weed/filer"
"github.com/seaweedfs/seaweedfs/weed/glog"
"github.com/seaweedfs/seaweedfs/weed/iam/integration"
"github.com/seaweedfs/seaweedfs/weed/pb"
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
"github.com/seaweedfs/seaweedfs/weed/pb/iam_pb"
"github.com/seaweedfs/seaweedfs/weed/security"
weed_server "github.com/seaweedfs/seaweedfs/weed/server"
stats_collect "github.com/seaweedfs/seaweedfs/weed/stats"
"github.com/seaweedfs/seaweedfs/weed/util"
"github.com/seaweedfs/seaweedfs/weed/util/grace"
"github.com/seaweedfs/seaweedfs/weed/util/version"
)
var (
f FilerOptions
filerStartS3 *bool
filerS3Options S3Options
filerStartWebDav *bool
filerWebDavOptions WebDavOption
filerStartIam *bool
filerIamOptions IamOptions
filerStartSftp *bool
filerSftpOptions SftpOptions
)
// allowUntrustedRemoteEndpointsUsage documents the flag shared by the filer and
// S3 gateway, whose remote-mount read paths dial the mounted endpoint directly.
const allowUntrustedRemoteEndpointsUsage = "if true, a read of a remote-only entry accepts arbitrary remote S3 endpoints including loopback / link-local hosts. Default rejects internal / metadata endpoints."
type FilerOptions struct {
masters *pb.ServerDiscovery
mastersString *string
ip *string
bindIp *string
port *int
portGrpc *int
publicPort *int
filerGroup *string
collection *string
defaultReplicaPlacement *string
disableDirListing *bool
maxMB *int
dirListingLimit *int
dataCenter *string
rack *string
enableNotification *bool
disableHttp *bool
cipher *bool
metricsHttpPort *int
metricsHttpIp *string
saveToFilerLimit *int
defaultLevelDbDirectory *string
concurrentUploadLimitMB *int
concurrentFileUploadLimit *int
debug *bool
debugPort *int
localSocket *string
showUIDirectoryDelete *bool
downloadMaxMBps *int
diskType *string
allowedOrigins *string
exposeDirectoryData *bool
tusBasePath *string
tusMaxSizeMB *int
tusSessionExpiry *time.Duration
s3ConfigFile *string // optional path to static S3 identity config
allowUntrustedRemoteEndpoints *bool
remoteCacheEvictThreshold *float64
// shutdownCtx, when non-nil, tells startFiler to gracefully shut down its
// HTTP/gRPC servers once the ctx is cancelled. Used by integration tests
// and by weed mini; nil for standalone weed filer.
shutdownCtx context.Context
// gracefulStopTimeout caps how long startFiler waits for gRPC graceful
// stop before forcing the server to stop. Zero means the default of 15s.
gracefulStopTimeout time.Duration
}
func init() {
cmdFiler.Run = runFiler // break init cycle
f.mastersString = cmdFiler.Flag.String("master", "localhost:9333", "comma-separated master servers or a single DNS SRV record of at least 1 master server, prepended with dnssrv+")
f.filerGroup = cmdFiler.Flag.String("filerGroup", "", "share metadata with other filers in the same filerGroup")
f.collection = cmdFiler.Flag.String("collection", "", "all data will be stored in this default collection")
f.ip = cmdFiler.Flag.String("ip", util.DetectedHostAddress(), "filer server http listen ip address")
f.bindIp = cmdFiler.Flag.String("ip.bind", "", "ip address to bind to. If empty, default to same as -ip option.")
f.port = cmdFiler.Flag.Int("port", 8888, "filer server http listen port")
f.portGrpc = cmdFiler.Flag.Int("port.grpc", 0, "filer server grpc listen port")
f.publicPort = cmdFiler.Flag.Int("port.readonly", 0, "readonly port opened to public")
f.defaultReplicaPlacement = cmdFiler.Flag.String("defaultReplicaPlacement", "", "default replication type. If not specified, use master setting.")
f.disableDirListing = cmdFiler.Flag.Bool("disableDirListing", false, "turn off directory listing")
f.maxMB = cmdFiler.Flag.Int("maxMB", 4, "split files larger than the limit")
f.dirListingLimit = cmdFiler.Flag.Int("dirListLimit", 100000, "limit sub dir listing size")
f.dataCenter = cmdFiler.Flag.String("dataCenter", "", "prefer to read and write to volumes in this data center")
f.rack = cmdFiler.Flag.String("rack", "", "prefer to write to volumes in this rack")
f.disableHttp = cmdFiler.Flag.Bool("disableHttp", false, "disable http request, only gRpc operations are allowed")
f.cipher = cmdFiler.Flag.Bool("encryptVolumeData", false, "encrypt data on volume servers")
f.metricsHttpPort = cmdFiler.Flag.Int("metricsPort", 0, "Prometheus metrics listen port")
f.metricsHttpIp = cmdFiler.Flag.String("metricsIp", "", "metrics listen ip. If empty, default to same as -ip.bind option.")
f.saveToFilerLimit = cmdFiler.Flag.Int("saveToFilerLimit", 0, "files smaller than this limit will be saved in filer store")
f.defaultLevelDbDirectory = cmdFiler.Flag.String("defaultStoreDir", ".", "if filer.toml is empty, use an embedded filer store in the directory")
f.concurrentUploadLimitMB = cmdFiler.Flag.Int("concurrentUploadLimitMB", 0, "limit total concurrent upload size, 0 means unlimited")
f.concurrentFileUploadLimit = cmdFiler.Flag.Int("concurrentFileUploadLimit", 0, "limit number of concurrent file uploads, 0 means unlimited")
f.debug = cmdFiler.Flag.Bool("debug", false, "serves runtime profiling data, e.g., http://localhost:<debug.port>/debug/pprof/goroutine?debug=2")
f.debugPort = cmdFiler.Flag.Int("debug.port", 6060, "http port for debugging")
f.localSocket = cmdFiler.Flag.String("localSocket", "", "default to /tmp/seaweedfs-filer-<port>.sock")
f.showUIDirectoryDelete = cmdFiler.Flag.Bool("ui.deleteDir", true, "enable filer UI show delete directory button")
f.downloadMaxMBps = cmdFiler.Flag.Int("downloadMaxMBps", 0, "download max speed for each download request, in MB per second")
f.diskType = cmdFiler.Flag.String("disk", "", "[hdd|ssd|<tag>] hard drive or solid state drive or any tag")
f.allowedOrigins = cmdFiler.Flag.String("allowedOrigins", "*", "comma separated list of allowed origins")
f.exposeDirectoryData = cmdFiler.Flag.Bool("exposeDirectoryData", true, "whether to return directory metadata and content in Filer UI")
f.tusBasePath = cmdFiler.Flag.String("tusBasePath", "/.tus", "TUS resumable upload endpoint base path (e.g., /.tus)")
f.tusMaxSizeMB = cmdFiler.Flag.Int("tusMaxSizeMB", 5*1024, "maximum TUS upload size in MB")
f.tusSessionExpiry = cmdFiler.Flag.Duration("tusSessionExpiry", 24*time.Hour, "incomplete TUS upload sessions are cleaned up after this duration, e.g. \"48h\", \"7h30m\"")
f.allowUntrustedRemoteEndpoints = cmdFiler.Flag.Bool("allowUntrustedRemoteEndpoints", false, allowUntrustedRemoteEndpointsUsage)
f.remoteCacheEvictThreshold = cmdFiler.Flag.Float64("remoteCacheEvictThreshold", 0.9, "evict remote-cached objects (oldest first) when any volume disk exceeds this usage fraction; 0 disables")
// start s3 on filer
filerStartS3 = cmdFiler.Flag.Bool("s3", false, "whether to start S3 gateway")
filerS3Options.port = cmdFiler.Flag.Int("s3.port", 8333, "s3 server http listen port")
filerS3Options.portHttps = cmdFiler.Flag.Int("s3.port.https", 0, "s3 server https listen port")
filerS3Options.portGrpc = cmdFiler.Flag.Int("s3.port.grpc", 0, "s3 server grpc listen port")
filerS3Options.domainName = cmdFiler.Flag.String("s3.domainName", "", "suffix of the host name in comma separated list, {bucket}.{domainName}")
filerS3Options.allowedOrigins = cmdFiler.Flag.String("s3.allowedOrigins", "*", "comma separated list of allowed origins")
filerS3Options.dataCenter = cmdFiler.Flag.String("s3.dataCenter", "", "prefer to read and write to volumes in this data center")
filerS3Options.tlsPrivateKey = cmdFiler.Flag.String("s3.key.file", "", "path to the TLS private key file")
filerS3Options.tlsCertificate = cmdFiler.Flag.String("s3.cert.file", "", "path to the TLS certificate file")
filerS3Options.config = cmdFiler.Flag.String("s3.config", "", "path to the config file")
filerS3Options.iamConfig = cmdFiler.Flag.String("s3.iam.config", "", "path to the advanced IAM config file")
filerS3Options.auditLogConfig = cmdFiler.Flag.String("s3.auditLogConfig", "", "path to the audit log config file")
filerS3Options.metricsHttpPort = cmdFiler.Flag.Int("s3.metricsPort", 0, "Prometheus metrics listen port")
filerS3Options.metricsHttpIp = cmdFiler.Flag.String("s3.metricsIp", "", "metrics listen ip. If empty, default to same as -s3.ip.bind option.")
cmdFiler.Flag.Bool("s3.allowEmptyFolder", true, "deprecated, ignored. Empty folder cleanup is now automatic.")
filerS3Options.allowDeleteBucketNotEmpty = cmdFiler.Flag.Bool("s3.allowDeleteBucketNotEmpty", true, "allow recursive deleting all entries along with bucket")
filerS3Options.autoCreateBucket = cmdFiler.Flag.Bool("s3.autoCreateBucket", true, "create the bucket on upload if it does not exist, for admin identities only")
filerS3Options.localSocket = cmdFiler.Flag.String("s3.localSocket", "", "default to /tmp/seaweedfs-s3-<port>.sock")
filerS3Options.tlsCACertificate = cmdFiler.Flag.String("s3.cacert.file", "", "path to the TLS CA certificate file")
filerS3Options.tlsVerifyClientCert = cmdFiler.Flag.Bool("s3.tlsVerifyClientCert", false, "whether to verify the client's certificate")
filerS3Options.bindIp = cmdFiler.Flag.String("s3.ip.bind", "", "ip address to bind to. If empty, default to same as -ip.bind option.")
filerS3Options.idleTimeout = cmdFiler.Flag.Int("s3.idleTimeout", 120, "connection idle seconds")
filerS3Options.concurrentUploadLimitMB = cmdFiler.Flag.Int("s3.concurrentUploadLimitMB", 0, "limit total concurrent upload size for S3, 0 means unlimited")
filerS3Options.concurrentFileUploadLimit = cmdFiler.Flag.Int("s3.concurrentFileUploadLimit", 0, "limit number of concurrent file uploads for S3, 0 means unlimited")
filerS3Options.enableIam = cmdFiler.Flag.Bool("s3.iam", true, "enable embedded IAM API on the same S3 port")
filerS3Options.cipher = cmdFiler.Flag.Bool("s3.encryptVolumeData", false, "encrypt data on volume servers for S3 uploads")
filerS3Options.iamReadOnly = cmdFiler.Flag.Bool("s3.iam.readOnly", true, "disable IAM write operations on this server")
filerS3Options.portIceberg = cmdFiler.Flag.Int("s3.port.iceberg", 8181, "Iceberg REST Catalog server listen port (0 to disable)")
filerS3Options.portLance = cmdFiler.Flag.Int("s3.port.lance", 9101, "Lance Namespace server listen port (0 to disable)")
filerS3Options.externalUrl = cmdFiler.Flag.String("s3.externalUrl", "", "the external URL clients use to connect (e.g. https://api.example.com:9000). Advertised to Iceberg and Lance clients, and tried first when verifying S3 signatures behind a reverse proxy. Falls back to S3_EXTERNAL_URL env var.")
filerS3Options.defaultFileMode = cmdFiler.Flag.String("s3.defaultFileMode", "", "default file mode for S3 uploaded objects, e.g. 0660, 0644, 0666")
filerS3Options.cacheSizeMB = cmdFiler.Flag.Int64("s3.cacheCapacityMB", 0, "in-memory chunk cache capacity in MB for S3 GETs shared across requests (0 disables)")
filerS3Options.readerCacheSizeMB = cmdFiler.Flag.Int64("s3.readerCacheSizeMB", 0, "memory budget in MiB for downloaded and in-flight reader buffers across all S3 GETs (0 means unlimited)")
filerS3Options.allowUntrustedRemoteEndpoints = cmdFiler.Flag.Bool("s3.allowUntrustedRemoteEndpoints", false, allowUntrustedRemoteEndpointsUsage)
// start webdav on filer
filerStartWebDav = cmdFiler.Flag.Bool("webdav", false, "whether to start webdav gateway")
filerWebDavOptions.port = cmdFiler.Flag.Int("webdav.port", 7333, "webdav server http listen port")
filerWebDavOptions.collection = cmdFiler.Flag.String("webdav.collection", "", "collection to create the files")
filerWebDavOptions.replication = cmdFiler.Flag.String("webdav.replication", "", "replication to create the files")
filerWebDavOptions.disk = cmdFiler.Flag.String("webdav.disk", "", "[hdd|ssd|<tag>] hard drive or solid state drive or any tag")
filerWebDavOptions.tlsPrivateKey = cmdFiler.Flag.String("webdav.key.file", "", "path to the TLS private key file")
filerWebDavOptions.tlsCertificate = cmdFiler.Flag.String("webdav.cert.file", "", "path to the TLS certificate file")
filerWebDavOptions.cacheDir = cmdFiler.Flag.String("webdav.cacheDir", os.TempDir(), "local cache directory for file chunks")
filerWebDavOptions.cacheSizeMB = cmdFiler.Flag.Int64("webdav.cacheCapacityMB", 0, "local cache capacity in MB")
filerWebDavOptions.maxMB = cmdFiler.Flag.Int("webdav.maxMB", 4, "split files larger than the limit")
filerWebDavOptions.filerRootPath = cmdFiler.Flag.String("webdav.filer.path", "/", "use this remote path from filer server")
// start iam on filer
filerStartIam = cmdFiler.Flag.Bool("iam", false, "whether to start IAM service")
filerIamOptions.ip = cmdFiler.Flag.String("iam.ip", *f.ip, "iam server http listen ip address")
filerIamOptions.port = cmdFiler.Flag.Int("iam.port", 8111, "iam server http listen port")
filerStartSftp = cmdFiler.Flag.Bool("sftp", false, "whether to start the SFTP server")
filerSftpOptions.port = cmdFiler.Flag.Int("sftp.port", 2022, "SFTP server listen port")
filerSftpOptions.sshPrivateKey = cmdFiler.Flag.String("sftp.sshPrivateKey", "", "path to the SSH private key file for host authentication")
filerSftpOptions.hostKeysFolder = cmdFiler.Flag.String("sftp.hostKeysFolder", "", "path to folder containing SSH private key files for host authentication")
filerSftpOptions.authMethods = cmdFiler.Flag.String("sftp.authMethods", "password,publickey", "comma-separated list of allowed auth methods: password, publickey, certificate")
filerSftpOptions.maxAuthTries = cmdFiler.Flag.Int("sftp.maxAuthTries", 6, "maximum number of authentication attempts per connection")
filerSftpOptions.bannerMessage = cmdFiler.Flag.String("sftp.bannerMessage", "SeaweedFS SFTP Server - Unauthorized access is prohibited", "message displayed before authentication")
filerSftpOptions.loginGraceTime = cmdFiler.Flag.Duration("sftp.loginGraceTime", 2*time.Minute, "timeout for authentication")
filerSftpOptions.clientAliveInterval = cmdFiler.Flag.Duration("sftp.clientAliveInterval", 5*time.Second, "interval for sending keep-alive messages")
filerSftpOptions.clientAliveCountMax = cmdFiler.Flag.Int("sftp.clientAliveCountMax", 3, "maximum number of missed keep-alive messages before disconnecting")
filerSftpOptions.userStoreFile = cmdFiler.Flag.String("sftp.userStoreFile", "", "path to JSON file containing user credentials and permissions")
filerSftpOptions.trustedUserCAKeysFile = cmdFiler.Flag.String("sftp.trustedUserCAKeysFile", "", "path to a file with trusted user CA public keys (OpenSSH authorized_keys format); required when 'certificate' is in -sftp.authMethods")
filerSftpOptions.dataCenter = cmdFiler.Flag.String("sftp.dataCenter", "", "prefer to read and write to volumes in this data center")
filerSftpOptions.bindIp = cmdFiler.Flag.String("sftp.ip.bind", "", "ip address to bind to. If empty, default to same as -ip.bind option.")
filerSftpOptions.localSocket = cmdFiler.Flag.String("sftp.localSocket", "", "default to /tmp/seaweedfs-sftp-<port>.sock")
}
func filerLongDesc() string {
desc := `start a file server which accepts REST operation for any files.
//create or overwrite the file, the directories /path/to will be automatically created
POST /path/to/file
//get the file content
GET /path/to/file
//create or overwrite the file, the filename in the multipart request will be used
POST /path/to/
//return a json format subdirectory and files listing
GET /path/to/
The configuration file "filer.toml" is read from ".", "$HOME/.seaweedfs/", "/usr/local/etc/seaweedfs/", or "/etc/seaweedfs/", in that order.
If the "filer.toml" is not found, an embedded filer store will be created under "-defaultStoreDir".
The example filer.toml configuration file can be generated by "weed scaffold -config=filer"
Supported Filer Stores:
`
storeNames := make([]string, len(filer.Stores))
for i, store := range filer.Stores {
storeNames[i] = "\t" + store.GetName()
}
sort.Strings(storeNames)
storeList := strings.Join(storeNames, "\n")
return desc + storeList
}
var cmdFiler = &Command{
UsageLine: "filer -port=8888 -master=<ip:port>[,<ip:port>]*",
Short: "start a file server that points to a master server, or a list of master servers",
Long: filerLongDesc(),
}
func runFiler(cmd *Command, args []string) bool {
if *f.debug {
go http.ListenAndServe(fmt.Sprintf(":%d", *f.debugPort), nil)
}
*f.defaultLevelDbDirectory = util.ResolvePath(*f.defaultLevelDbDirectory)
filerS3Options.resolvePaths()
filerWebDavOptions.resolvePaths()
filerSftpOptions.resolvePaths()
util.LoadSecurityConfiguration()
// Share the S3 static identity config file with the filer regardless of
// whether the embedded S3 gateway runs on this node: the IAM gRPC service
// the admin UI and weed shell talk to is wired up unconditionally, and it
// needs the same identities the S3 server would load from -s3.config.
f.s3ConfigFile = filerS3Options.config
switch {
case *f.metricsHttpIp != "":
// noting to do, use f.metricsHttpIp
case *f.bindIp != "":
*f.metricsHttpIp = *f.bindIp
case *f.ip != "":
*f.metricsHttpIp = *f.ip
}
go stats_collect.StartMetricsServer(*f.metricsHttpIp, *f.metricsHttpPort)
filerAddress := pb.NewServerAddress(*f.ip, *f.port, *f.portGrpc).String()
startDelay := time.Duration(2)
if *filerStartS3 {
filerS3Options.filer = &filerAddress
filerS3Options.ip = f.ip
if *filerS3Options.bindIp == "" {
filerS3Options.bindIp = f.bindIp
}
filerS3Options.localFilerSocket = f.localSocket
if *f.dataCenter != "" && *filerS3Options.dataCenter == "" {
filerS3Options.dataCenter = f.dataCenter
}
// Set S3 metrics IP based on bind IP if not explicitly set
if *filerS3Options.metricsHttpIp == "" {
*filerS3Options.metricsHttpIp = *filerS3Options.bindIp
}
go func(delay time.Duration) {
time.Sleep(delay * time.Second)
filerS3Options.startS3Server()
}(startDelay)
startDelay++
}
if *filerStartWebDav {
filerWebDavOptions.filer = &filerAddress
filerWebDavOptions.ipBind = f.bindIp
if *filerWebDavOptions.disk == "" {
filerWebDavOptions.disk = f.diskType
}
go func(delay time.Duration) {
time.Sleep(delay * time.Second)
filerWebDavOptions.startWebDav()
}(startDelay)
startDelay++
}
if *filerStartIam {
filerIamOptions.filer = &filerAddress
filerIamOptions.masters = f.mastersString
go func(delay time.Duration) {
time.Sleep(delay * time.Second)
filerIamOptions.startIamServer()
}(startDelay)
startDelay++
}
if *filerStartSftp {
filerSftpOptions.filer = &filerAddress
if *filerSftpOptions.bindIp == "" {
filerSftpOptions.bindIp = f.bindIp
}
if *f.dataCenter != "" && *filerSftpOptions.dataCenter == "" {
filerSftpOptions.dataCenter = f.dataCenter
}
go func(delay time.Duration) {
time.Sleep(delay * time.Second)
filerSftpOptions.startSftpServer()
}(startDelay)
}
f.masters = pb.ServerAddresses(*f.mastersString).ToServiceDiscovery()
f.startFiler()
return true
}
func (fo *FilerOptions) startFiler() {
defaultMux := http.NewServeMux()
publicVolumeMux := defaultMux
if *fo.publicPort != 0 {
publicVolumeMux = http.NewServeMux()
}
if *fo.portGrpc == 0 {
*fo.portGrpc = 10000 + *fo.port
}
if *fo.bindIp == "" {
*fo.bindIp = *fo.ip
}
util.SetOutboundLocalIP(*fo.bindIp)
if *fo.allowedOrigins == "" {
*fo.allowedOrigins = "*"
}
defaultLevelDbDirectory := *fo.defaultLevelDbDirectory + "/filerldb2"
filerAddress := pb.NewServerAddress(*fo.ip, *fo.port, *fo.portGrpc)
// Initialize credential manager for IAM gRPC service
var credentialManager *credential.CredentialManager
var err error
credentialManager, err = credential.NewCredentialManagerWithDefaults("")
if err != nil {
glog.Warningf("Failed to initialize credential manager: %v", err)
} else {
glog.V(0).Infof("Initialized credential manager: %s", credentialManager.GetStoreName())
}
// Load static S3 identities from config file if specified
if fo.s3ConfigFile != nil && *fo.s3ConfigFile != "" {
if credentialManager != nil {
if err := credentialManager.LoadS3ConfigFile(*fo.s3ConfigFile); err != nil {
glog.Warningf("Failed to load S3 config file for static identities: %v", err)
}
}
}
fs, nfs_err := weed_server.NewFilerServer(defaultMux, publicVolumeMux, &weed_server.FilerOption{
Masters: fo.masters,
FilerGroup: *fo.filerGroup,
Collection: *fo.collection,
DefaultReplication: *fo.defaultReplicaPlacement,
DisableDirListing: *fo.disableDirListing,
MaxMB: *fo.maxMB,
DirListingLimit: *fo.dirListingLimit,
DataCenter: *fo.dataCenter,
Rack: *fo.rack,
DefaultLevelDbDir: defaultLevelDbDirectory,
DisableHttp: *fo.disableHttp,
Host: filerAddress,
Cipher: *fo.cipher,
SaveToFilerLimit: int64(*fo.saveToFilerLimit),
ConcurrentUploadLimit: int64(*fo.concurrentUploadLimitMB) * 1024 * 1024,
ConcurrentFileUploadLimit: int64(*fo.concurrentFileUploadLimit),
ShowUIDirectoryDelete: *fo.showUIDirectoryDelete,
DownloadMaxBytesPs: int64(*fo.downloadMaxMBps) * 1024 * 1024,
DiskType: *fo.diskType,
AllowedOrigins: strings.Split(*fo.allowedOrigins, ","),
ExposeDirectoryData: *fo.exposeDirectoryData,
TusBasePath: *fo.tusBasePath,
TusMaxSize: int64(*fo.tusMaxSizeMB) * 1024 * 1024,
TusSessionExpiry: *fo.tusSessionExpiry,
CredentialManager: credentialManager,
AllowUntrustedRemoteEndpoints: *fo.allowUntrustedRemoteEndpoints,
RemoteCacheEvictThreshold: *fo.remoteCacheEvictThreshold,
})
if nfs_err != nil {
glog.Fatalf("Filer startup error: %v", nfs_err)
}
// Serve "//" and ".." paths at their cleaned form instead of letting the mux
// redirect: its Location is double-escaped, turning non-ASCII names into
// percent-encoded directory names when a client follows it (#11125).
defaultHandler := weed_server.CleanPathHandler(defaultMux)
var httpServers []*http.Server
if *fo.publicPort != 0 {
publicListeningAddress := util.JoinHostPort(*fo.bindIp, *fo.publicPort)
glog.V(0).Infoln("Start Seaweed filer server", version.Version(), "public at", publicListeningAddress)
publicListener, localPublicListener, e := util.NewIpAndLocalListeners(*fo.bindIp, *fo.publicPort, 0)
if e != nil {
glog.Fatalf("Filer server public listener error on port %d:%v", *fo.publicPort, e)
}
publicHandler := weed_server.CleanPathHandler(publicVolumeMux)
publicServer := newHttpServer(publicHandler, nil)
httpServers = append(httpServers, publicServer)
go func() {
if e := publicServer.Serve(publicListener); e != nil && e != http.ErrServerClosed {
glog.Fatalf("Volume server fail to serve public: %v", e)
}
}()
if localPublicListener != nil {
localPublicServer := newHttpServer(publicHandler, nil)
httpServers = append(httpServers, localPublicServer)
go func() {
if e := localPublicServer.Serve(localPublicListener); e != nil && e != http.ErrServerClosed {
glog.Errorf("Volume server fail to serve public: %v", e)
}
}()
}
}
glog.V(0).Infof("Start Seaweed Filer %s at %s:%d", version.Version(), *fo.ip, *fo.port)
filerListener, filerLocalListener, e := util.NewIpAndLocalListeners(
*fo.bindIp, *fo.port,
time.Duration(10)*time.Second,
)
if e != nil {
glog.Fatalf("Filer listener error: %v", e)
}
// starting grpc server
grpcPort := *fo.portGrpc
grpcL, grpcLocalL, err := util.NewIpAndLocalListeners(*fo.bindIp, grpcPort, 0)
if err != nil {
glog.Fatalf("failed to listen on grpc port %d: %v", grpcPort, err)
}
grpcS := pb.NewGrpcServer(security.LoadServerTLS(util.GetViper(), "grpc.filer"))
filer_pb.RegisterSeaweedFilerServer(grpcS, fs)
// Register the IAM gRPC service. Auth is opt-in: when
// jwt.filer_signing.key is configured the service requires a Bearer token
// signed with that key; otherwise it runs unauthenticated, matching the
// rest of the filer's gRPC surface. Operators who expose the filer gRPC
// port beyond a trusted network should set jwt.filer_signing.key on both
// the filer and the admin server.
if credentialManager != nil {
adminSigningKey := security.SigningKey(util.GetViper().GetString("jwt.filer_signing.key"))
iamGrpcServer := weed_server.NewIamGrpcServer(credentialManager, adminSigningKey)
// The OIDC provider and role RPCs write where S3 servers configured with
// filer-typed "oidcProviderStore" and "roleStore" read: this filer, at
// the stores' default base paths.
selfAddress := func() string { return string(filerAddress) }
if roleStore, err := integration.NewFilerRoleStore(nil, selfAddress); err != nil {
glog.Warningf("IAM gRPC: role RPCs disabled: %v", err)
} else {
iamGrpcServer.SetSTSStores(integration.NewFilerOIDCProviderStore(nil, selfAddress), roleStore)
}
iam_pb.RegisterSeaweedIdentityAccessManagementServer(grpcS, iamGrpcServer)
if len(adminSigningKey) == 0 {
glog.Warning("IAM gRPC service on filer is UNAUTHENTICATED: anyone who can reach this port can create users and policies, and its OIDC provider and role RPCs are refused; set jwt.filer_signing.key in security.toml to require an admin Bearer token")
} else {
glog.V(0).Info("Registered IAM gRPC service on filer (admin Bearer token required)")
}
}
reflection.Register(grpcS)
if grpcLocalL != nil {
go grpcS.Serve(grpcLocalL)
}
go grpcS.Serve(grpcL)
pb.ServeGrpcOnLocalSocket(grpcS, grpcPort)
// Helper to gracefully stop the gRPC server, waiting for active RPCs.
gracefulTimeout := fo.gracefulStopTimeout
if gracefulTimeout <= 0 {
gracefulTimeout = 15 * time.Second
}
stopGrpcServer := func() {
glog.V(0).Infof("Gracefully stopping gRPC server")
stopped := make(chan struct{})
go func() {
grpcS.GracefulStop()
close(stopped)
}()
select {
case <-stopped:
glog.V(0).Infof("gRPC server stopped gracefully")
case <-time.After(gracefulTimeout):
glog.V(0).Infof("gRPC server graceful stop timed out after %s, forcing stop", gracefulTimeout)
grpcS.Stop()
}
}
var socketServer *http.Server
if runtime.GOOS != "windows" {
localSocket := *fo.localSocket
if localSocket == "" {
localSocket = fmt.Sprintf("/tmp/seaweedfs-filer-%d.sock", *fo.port)
}
if err := os.Remove(localSocket); err != nil && !os.IsNotExist(err) {
glog.Fatalf("Failed to remove %s, error: %s", localSocket, err.Error())
}
filerSocketListener, err := net.Listen("unix", localSocket)
if err != nil {
glog.Fatalf("Failed to listen on %s: %v", localSocket, err)
}
socketServer = newHttpServer(defaultHandler, nil)
httpServers = append(httpServers, socketServer)
go socketServer.Serve(filerSocketListener)
}
if viper.GetString("https.filer.key") != "" {
certFile := viper.GetString("https.filer.cert")
keyFile := viper.GetString("https.filer.key")
caCertFile := viper.GetString("https.filer.ca")
disbaleTlsVerifyClientCert := viper.GetBool("https.filer.disable_tls_verify_client_cert")
getCert, certProvider, err := security.NewReloadingServerCertificate(certFile, keyFile)
if err != nil {
glog.Fatalf("Filer failed to load HTTPS certificate: %v", err)
}
defer certProvider.Close()
caCertPool := x509.NewCertPool()
if caCertFile != "" {
caCertFile, err := os.ReadFile(caCertFile)
if err != nil {
glog.Fatalf("error reading CA certificate: %v", err)
}
caCertPool.AppendCertsFromPEM(caCertFile)
}
clientAuth := tls.NoClientCert
if !disbaleTlsVerifyClientCert {
clientAuth = tls.RequireAndVerifyClientCert
}
tlsConfig := &tls.Config{
GetCertificate: getCert,
ClientAuth: clientAuth,
ClientCAs: caCertPool,
}
err = security.FixTlsConfig(util.GetViper(), tlsConfig)
if err != nil {
glog.Fatalf("Filer failed to fix TLS config: %v", err)
}
var localTLSServer *http.Server
if filerLocalListener != nil {
localTLSServer = newHttpServer(defaultHandler, tlsConfig)
httpServers = append(httpServers, localTLSServer)
go func() {
if err := localTLSServer.ServeTLS(filerLocalListener, "", ""); err != nil {
glog.Errorf("Filer Fail to serve: %v", err)
}
}()
}
httpS := newHttpServer(defaultHandler, tlsConfig)
httpServers = append(httpServers, httpS)
shutdown := newFilerShutdown(stopGrpcServer, fs.Shutdown, httpServers...)
grace.OnInterrupt(shutdown)
if fo.shutdownCtx != nil {
go func() {
<-fo.shutdownCtx.Done()
shutdown()
}()
}
if err := httpS.ServeTLS(filerListener, "", ""); err != nil && err != http.ErrServerClosed {
glog.Fatalf("Filer Fail to serve: %v", err)
}
// Serve returns when listeners close, before active requests finish.
// Join the same shutdown sequence instead of closing the filer here.
shutdown()
} else {
var localHTTPServer *http.Server
if filerLocalListener != nil {
localHTTPServer = newHttpServer(defaultHandler, nil)
httpServers = append(httpServers, localHTTPServer)
go func() {
if err := localHTTPServer.Serve(filerLocalListener); err != nil {
glog.Errorf("Filer Fail to serve: %v", err)
}
}()
}
httpS := newHttpServer(defaultHandler, nil)
httpServers = append(httpServers, httpS)
shutdown := newFilerShutdown(stopGrpcServer, fs.Shutdown, httpServers...)
grace.OnInterrupt(shutdown)
if fo.shutdownCtx != nil {
go func() {
<-fo.shutdownCtx.Done()
shutdown()
}()
}
if err := httpS.Serve(filerListener); err != nil && err != http.ErrServerClosed {
glog.Fatalf("Filer Fail to serve: %v", err)
}
// Serve returns when listeners close, before active requests finish.
// Join the same shutdown sequence instead of closing the filer here.
shutdown()
}
}
// newFilerShutdown joins shutdown callers while gRPC and HTTP drain concurrently.
func newFilerShutdown(stopGrpc, shutdownFiler func(), httpServers ...*http.Server) func() {
return sync.OnceFunc(func() {
shutdownCtx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
defer cancel()
var drained sync.WaitGroup
drained.Add(1)
go func() {
defer drained.Done()
stopGrpc()
}()
for _, server := range httpServers {
drained.Add(1)
go func() {
defer drained.Done()
if err := server.Shutdown(shutdownCtx); err != nil {
glog.Warningf("filer HTTP shutdown: %v", err)
}
}()
}
drained.Wait()
shutdownFiler()
})
}
Reference in new issue View Git Blame Copy Permalink