diff --git a/NFS-Server.md b/NFS-Server.md deleted file mode 100644 index a8582b3..0000000 --- a/NFS-Server.md +++ /dev/null @@ -1,366 +0,0 @@ -# NFS Server (experimental) - -SeaweedFS ships an experimental NFSv3 server that exports a filer subtree -over the NFS wire protocol. Unlike running `weed mount` and re-exporting -the FUSE mount with an external NFS server, the NFS frontend talks to the -SeaweedFS cluster directly: metadata RPCs go to the filer, and chunk -reads and writes go straight to volume servers (or through `publicUrl` / -`filerProxy` routing if you configure it). Filehandles are deterministic -and use the same filer-assigned inode that backs the `weed mount` data -path, so an NFS head shares object identity with any FUSE mount or -other filer client. - -The code lives under [`weed/server/nfs/`](https://github.com/seaweedfs/seaweedfs/tree/master/weed/server/nfs). -Status: experimental, safe for evaluation and development. Do **not** -use it for production workloads that depend on advisory file locking — -shared NLM / NFSv4 lock state is not implemented yet (see -[Limitations](#limitations)). - -```mermaid -flowchart LR - Client["NFS client
(Linux / macOS)"] - NFS["weed nfs"] - Filer["Filer"] - Volumes["Volume servers"] - - Client -- "NFSv3 RPC" --> NFS - NFS -- "metadata" --> Filer - NFS -- "chunk read / write" --> Volumes - Filer -. "AssignVolume" .-> Volumes -``` - -Metadata RPCs go to the filer; chunk bytes flow directly between the NFS head and volume servers. - -## Overview - -The NFS server provides: - -- **NFSv3 wire protocol** served directly from SeaweedFS -- **Deterministic filehandles** that survive filer restarts and - multi-head failover by encoding the export ID, the filer-assigned - inode, and a per-inode generation -- **Filer-backed metadata operations** — lookup, readdir, create, - mkdir, rename, remove, symlink, readlink, and the hardlink RPC -- **Direct volume-server data path** for chunk reads and writes, - with optional `publicUrl` or `filerProxy` routing -- **Streaming writes** — each NFS WRITE RPC issues one - `AssignVolume` + one volume-server chunk upload + one filer - `UpdateEntry`, identical to how `weed filer` HTTP uploads and the - S3 gateway persist object data. The NFS server does not buffer - whole files in memory, so there is no per-file write cap. -- **Inline fast path for small files** — a write whose post-write - size still fits in the 4 MiB inline budget and that does not touch - an existing chunked file is persisted directly in the filer entry, - skipping the volume-server round-trip (matches the filer HTTP - handler's `SaveToFilerLimit` shortcut) -- **Export access controls** via CIDR or hostname allowlist -- **Metadata subscription-based cache invalidation** for each NFS head -- **Read-only export mode** for safe serving of an existing subtree - -## Quick Start - -### Starting the NFS Server - -```bash -weed nfs \ - -filer=localhost:8888 \ - -port=2049 \ - -filer.path=/nfs -``` - -`weed nfs` is a standalone binary today. The `filer.path` subtree must -already exist in the filer namespace — the server does not create the -export root on its own. Create the directory first (for example, by -uploading a placeholder file under it with the filer HTTP API) if it -does not already exist. - -### Mounting from a Client - -The `weed nfs` server does not register with the system portmapper -(rpcbind on port 111) by default. You have two options: bypass portmap -on the client side, or enable the built-in portmap responder on the -server side with `-portmap.bind`. - -#### Client-side bypass (server default) - -Pass `port=` and `mountport=` so the kernel client skips its portmap -discovery step. - -Linux: - -```bash -sudo mount -t nfs \ - -o nfsvers=3,nolock,port=2049,mountport=2049,proto=tcp,mountproto=tcp \ - 127.0.0.1:/nfs /mnt/seaweedfs-nfs -``` - -macOS: - -```bash -sudo mount -t nfs \ - -o nfsvers=3,tcp,port=2049,mountport=2049,noresvport \ - 127.0.0.1:/nfs /Volumes/seaweedfs-nfs -``` - -#### Server-side portmap responder - -Start `weed nfs` with `-portmap.bind=`; the server then answers -PORTMAP v2 queries on port 111 (TCP + UDP) for its own NFS v3 and -MOUNT v3 services. Clients can then mount with the plain form: - -```bash -sudo mount -t nfs -o nfsvers=3,nolock 127.0.0.1:/nfs /mnt/seaweedfs-nfs -``` - -Binding port 111 requires root or `CAP_NET_BIND_SERVICE` and must not -collide with an existing `rpcbind` on the host. - -### Mount Path Resolution - -The MOUNT3 dirpath the client sends is matched against the configured -`-filer.path`: - -- **Exact match** (`:`) — mounts at the export root. -- **Subdirectory of the export** (`:/`) — the - server resolves `` in the filer and mounts the client directly - inside it. The returned filehandle encodes the subdirectory's inode, - so the client lands at `` without having to `cd` into it. One - `weed nfs` instance can therefore expose any subtree of its export - just by varying the client mount string — useful for NFS-CSI setups - where the subdirectory is encoded in `PV.spec.nfs.path`. Missing - entries return `NFS3ERR_NOENT`; non-directory entries return - `NFS3ERR_NOTDIR`. -- **Anything else** (outside the export, empty, garbage) — falls back to - the export root and logs an INFO line. This mirrors rclone's - `serve nfs` behavior so operator typos still mount somewhere - sensible rather than producing a transport-dependent split between - "empty mount" and "mount failure". - -Example with `weed nfs -filer.path=/buckets`: - -```bash -# resolves to /buckets: -mount -t nfs :/buckets /mnt -# resolves to /buckets/data (subexport mount): -mount -t nfs :/buckets/data /mnt -# fails with NFS3ERR_NOENT (under-export, missing entry): -mount -t nfs :/buckets/no-such-thing /mnt -# falls back to /buckets, with INFO log noting the typo: -mount -t nfs :/wrong/path /mnt -``` - -File-handle scoping is unchanged: handles minted by any of these mounts -still cannot escape the configured export, regardless of what the -client asked for at MOUNT time. - -## Configuration - -### Command-Line Options - -| Flag | Default | Description | -|------|---------|-------------| -| `-filer` | `localhost:8888` | Filer server address (ip:port) | -| `-ip.bind` | `127.0.0.1` | IP address to bind to. Defaults to loopback so the experimental server is not reachable from other hosts unless you override it. | -| `-port` | `2049` | NFS server listen port | -| `-filer.path` | (required) | Remote filer path to export. No default — operators must opt in to exporting a namespace subtree. | -| `-readOnly` | `false` | Export the filer path as read-only | -| `-allowedClients` | (empty) | Comma-separated client IPs, hostnames, or CIDR blocks allowed to connect | -| `-volumeServerAccess` | `direct` | How the NFS server reaches volume servers: `direct`, `publicUrl`, or `filerProxy` | -| `-portmap.bind` | (empty) | If set, bind a built-in PORTMAP v2 responder on `:111` (TCP + UDP) so plain `mount -t nfs` works without client-side portmap bypass. Requires root or `CAP_NET_BIND_SERVICE`; must not conflict with a running `rpcbind`. | - -### Safer Defaults - -Because export ACLs and full POSIX lock support are not yet available, -the server ships with defensive defaults: - -- `-ip.bind` defaults to `127.0.0.1`. You must set it explicitly (for - example `-ip.bind=0.0.0.0` or a specific interface IP) to expose the - experimental server to the network. -- `-filer.path` has no default. `weed nfs` refuses to start without it, - to prevent an accidental full-namespace export. Setting - `-filer.path=/` still works, but the server logs a warning because - exporting the entire filer namespace with no ACLs in place is almost - never what you want. With `-filer.path=/`, clients can pick any - subtree at mount time via [subexport mounts](#mount-path-resolution), - but you must constrain access via `-allowedClients` or `-ip.bind` - before doing so. - -### Client Allowlist - -```bash -weed nfs \ - -filer=localhost:8888 \ - -filer.path=/nfs \ - -allowedClients=10.0.0.0/8,203.0.113.42,nfs-client.example.com -``` - -Entries may be: - -- an IPv4 or IPv6 literal — matched against the client's remote - address at connection time -- a CIDR block — matched against the client's remote address -- a hostname — resolved to IPs at startup and those IPs are then - matched. Resolution happens once; the server does not re-resolve - later. - -An unmatched client gets a TCP-level close before the MOUNT RPC is -served. - -### Read-Only Export - -```bash -weed nfs -filer=localhost:8888 -filer.path=/nfs -readOnly -``` - -In read-only mode every mutating RPC returns `NFS3ERR_ROFS`. The -server still uses the deterministic filehandle format and metadata -subscription invalidation, so it is safe to run a read-only replica -next to a read-write head. - -### Volume Server Access Modes - -The NFS server reads chunks directly from volume servers by default -(`-volumeServerAccess=direct`), matching the `weed mount` default. Use -`publicUrl` to route through the volume servers' public URLs, or -`filerProxy` to route chunk reads through the filer host when volume -servers are not reachable from the NFS host. - -## Architecture - -### Deterministic Filehandles - -Every filer entry in this PR now carries a filer-assigned inode -(Snowflake-style, unique per filer and stable across restarts). The -NFS server encodes its filehandles as: - -``` -exportID || filehandleKind || inode || generation -``` - -- `exportID` is a CRC32 of the export root path, so a handle minted by - one export cannot resolve inside another. -- `filehandleKind` distinguishes files, directories, and symlinks. -- `inode` is the filer-assigned inode, so the handle stays valid - across rename / move as long as the entry exists. -- `generation` is the inode secondary index generation counter. - Handles from a deleted-and-recreated path decode as stale and the - server returns `NFS3ERR_STALE` instead of handing the client the - wrong object. - -### Inode Secondary Index - -The filer writes an inode secondary index record next to every entry -it creates. The record maps inode -> current paths (multi-path for -hardlinks) and carries a generation counter that bumps on -delete/recreate. Handle resolution in the NFS server goes through this -index so it never has to walk the namespace. - -Secondary index updates are best-effort: if the KV write fails, the -primary store mutation is still visible and the index is rebuilt on -the next write to the same inode. - -### Metadata Subscription - -Each NFS head subscribes to filer metadata change events via the -filer's `SubscribeMetadata` stream starting from the export root. -Chunk entries cached in the shared reader cache get invalidated as -soon as the filer confirms a rename or delete, so a write on one head -becomes visible to a reader on another head without waiting for a -cache TTL. - -### Write Path - -The write path is fully streaming — no whole-file buffer in server -memory, no per-file ceiling. - -- `OpenFile` for write does **not** read the existing file contents - from the filer. If `O_TRUNC` is set on a non-empty file, the server - issues a single `UpdateEntry` to drop chunks + inline content; a - fresh create + `O_TRUNC` is a zero-RPC no-op. -- Every `Write` RPC either: - - **Fast path, inline**: the post-write size still fits within - `maxInlineWriteSize` (4 MiB) and the file has no existing chunks - — the bytes are merged into `entry.Content` directly via one - filer `UpdateEntry`. No volume-server round-trip. - - **Streaming path**: the server calls `AssignVolume` on the filer, - uploads the caller's bytes to the assigned volume server as a - single chunk at the current file offset, and atomically appends - the resulting chunk descriptor to the filer entry's chunk list - via `UpdateEntry`. Any pre-existing `entry.Content` is migrated - to a chunk at offset 0 in the same `UpdateEntry` so the chunk - list becomes the authoritative representation. -- `Close` is a no-op — everything is flushed synchronously inside - `Write` / `Truncate`. -- `Truncate` shrinks or grows the file by mutating the filer entry - directly (drop chunks past the new size, clip inline content, - update `FileSize`). It does not touch volume servers. - -Because there is no whole-file buffer, the NFS server's per-open -memory footprint is bounded by a single WRITE RPC's payload -(typically `wsize` — Linux defaults to 1 MiB). The volume-server -chunk layout matches what `weed filer` HTTP uploads and the S3 -gateway produce, so normal SeaweedFS GC, compaction, and replication -apply without special cases. - -## Limitations - -- **Experimental.** The on-wire protocol and filehandle codec are - stable inside a single server binary, but the server has not yet - been stress-tested as a production NFS endpoint. -- **Shared locks are not implemented.** The upstream - `github.com/willscott/go-nfs` backend exposes only local billy - `File.Lock` / `Unlock` hooks. It does not implement the NLM or - NFSv4 lock state RPCs, so cluster-wide advisory locks, lock - reclaim, and grace periods are not available. The server - advertises no lock capability and returns `ErrNotSupported` for any - lock RPC. Workloads that rely on `flock(2)` / `fcntl(2)` advisory - locks over NFS should not use this server until lock support lands — - resolving this requires either patching go-nfs to implement NLM / - NFSv4 lock state or migrating to a different NFSv4 stack. -- **One chunk per NFS WRITE RPC.** Each NFS WRITE RPC is persisted - as its own volume-server chunk rather than being coalesced with - adjacent writes into larger chunks. A Linux client that defaults to - a 1 MiB `wsize` and uploads a 100 MiB file therefore produces ~100 - small chunks. The chunks resolve correctly via the filer's chunk - view layer and can be compacted later by the volume-server GC, but - larger chunk batching would cut the filer UpdateEntry load. A - future refactor will route NFS writes through the filer's standard - chunking logic (typically 4–8 MiB per chunk) to match what a - direct `weed filer` HTTP upload produces. -- **Portmap is opt-in.** The server does not register with the system - `rpcbind`. By default, clients must pass `port=` and `mountport=` - explicitly. Set `-portmap.bind=` to enable the built-in PORTMAP - v2 responder on port 111 so plain `mount -t nfs` works (see - [Mounting from a Client](#mounting-from-a-client)). - -## Testing - -End-to-end integration tests live in -[`test/nfs`](https://github.com/seaweedfs/seaweedfs/tree/master/test/nfs). -Each test boots its own master + volume + filer + nfs subprocess -cluster on loopback and drives it through the NFSv3 RPC protocol via -`github.com/willscott/go-nfs-client`. No kernel NFS mount or -privileged port is required, which means the suite runs in CI on -GitHub Actions directly without host network configuration. - -```bash -cd test/nfs -make test -``` - -Covered scenarios include read/write round-trip, directory operations, -nested mkdir, rename content preservation, overwrite with explicit -truncate, large (3 MiB) binary files, arbitrary byte payloads, empty -files, symlinks, readdir ordering, remove-missing error propagation, -FSINFO limit sanity, sequential append writes, and readdir after -remove. - -## See Also - -- [[FUSE Mount]] — the original filesystem-in-userspace mount that - shares the same filer-backed data path -- [[WebDAV]] — a similar gateway over a different protocol -- [[SFTP Server]] — SSH/SFTP gateway -- [[Filer Commands and Operations]] — filer operations reference -- [NFS server source](https://github.com/seaweedfs/seaweedfs/tree/master/weed/server/nfs) - and [integration tests](https://github.com/seaweedfs/seaweedfs/tree/master/test/nfs) diff --git a/_Sidebar.md b/_Sidebar.md index efa3603..c655eca 100644 --- a/_Sidebar.md +++ b/_Sidebar.md @@ -69,8 +69,6 @@ ### [[SFTP Server]] -### [[NFS Server]] - ### Cloud Drive * [[Cloud Drive Benefits]] * [[Cloud Drive Architecture]]