diff --git a/NFS-Server.md b/NFS-Server.md new file mode 100644 index 0000000..ccc988f --- /dev/null +++ b/NFS-Server.md @@ -0,0 +1,236 @@ +# NFS Server (experimental) + +SeaweedFS ships an experimental NFSv3 server that exports a filer subtree +over the NFS wire protocol. Unlike running `weed mount` and re-exporting +the FUSE mount with an external NFS server, the NFS frontend talks to the +filer directly. Filehandles are deterministic, metadata operations go +through filer RPCs, and chunk reads go to volume servers using the same +direct-access rules as the FUSE mount. + +The code lives under [`weed/server/nfs/`](https://github.com/seaweedfs/seaweedfs/tree/master/weed/server/nfs). +Status: experimental, safe for evaluation and development, not yet +recommended for production workloads that need shared locking or +whole-file rewrites larger than 64 MiB. + +## Overview + +The NFS server provides: + +- **NFSv3 wire protocol** served directly from SeaweedFS +- **Deterministic filehandles** that survive filer restarts and + multi-head failover by encoding the export ID, the filer-assigned + inode, and a per-inode generation +- **Filer-backed metadata operations** — lookup, readdir, create, + mkdir, rename, remove, symlink, readlink, and the hardlink RPC +- **Direct volume-server data path** for chunk reads (with optional + `publicUrl` or `filerProxy` routing) +- **Buffered large-file writes** up to 64 MiB backed by + `AssignVolume` and volume-server chunk uploads +- **Export access controls** via CIDR or hostname allowlist +- **Metadata subscription-based cache invalidation** for each NFS head +- **Read-only export mode** for safe serving of an existing subtree + +## Quick Start + +### Starting the NFS Server + +```bash +weed nfs \ + -filer=localhost:8888 \ + -port=2049 \ + -filer.path=/nfs +``` + +`weed nfs` is a standalone binary today. The `filer.path` subtree must +already exist in the filer namespace — the server does not create the +export root on its own. Create the directory first (for example, by +uploading a placeholder file under it with the filer HTTP API) if it +does not already exist. + +### Mounting from a Client + +Linux: + +```bash +sudo mount -t nfs -o nfsvers=3,proto=tcp,port=2049,mountport=2049 \ + 127.0.0.1:/nfs /mnt/seaweedfs-nfs +``` + +macOS: + +```bash +sudo mount -t nfs \ + -o nfsvers=3,tcp,port=2049,mountport=2049,noresvport \ + 127.0.0.1:/nfs /Volumes/seaweedfs-nfs +``` + +Both `port=` and `mountport=` have to be set because the NFS server +does not register with the system portmapper — the kernel client would +otherwise look on port 111 for the MOUNT program. + +## Configuration + +### Command-Line Options + +| Flag | Default | Description | +|------|---------|-------------| +| `-filer` | `localhost:8888` | Filer server address (ip:port) | +| `-ip.bind` | `127.0.0.1` | IP address to bind to. Defaults to loopback so the experimental server is not reachable from other hosts unless you override it. | +| `-port` | `2049` | NFS server listen port | +| `-filer.path` | (required) | Remote filer path to export. No default — operators must opt in to exporting a namespace subtree. | +| `-readOnly` | `false` | Export the filer path as read-only | +| `-allowedClients` | (empty) | Comma-separated client IPs, hostnames, or CIDR blocks allowed to connect | +| `-volumeServerAccess` | `direct` | How the NFS server reaches volume servers: `direct`, `publicUrl`, or `filerProxy` | + +### Safer Defaults + +Because export ACLs and full POSIX lock support are not yet available, +the server ships with defensive defaults: + +- `-ip.bind` defaults to `127.0.0.1`. You must set it explicitly (for + example `-ip.bind=0.0.0.0` or a specific interface IP) to expose the + experimental server to the network. +- `-filer.path` has no default. `weed nfs` refuses to start without it, + to prevent an accidental full-namespace export. Setting + `-filer.path=/` still works, but the server logs a warning because + exporting the entire filer namespace with no ACLs in place is almost + never what you want. + +### Client Allowlist + +```bash +weed nfs \ + -filer=localhost:8888 \ + -filer.path=/nfs \ + -allowedClients=10.0.0.0/8,203.0.113.42,nfs-client.example.com +``` + +Entries may be: + +- an IPv4 or IPv6 literal — matched against the client's remote + address at connection time +- a CIDR block — matched against the client's remote address +- a hostname — resolved to IPs at startup and those IPs are then + matched. Resolution happens once; the server does not re-resolve + later. + +An unmatched client gets a TCP-level close before the MOUNT RPC is +served. + +### Read-Only Export + +```bash +weed nfs -filer=localhost:8888 -filer.path=/nfs -readOnly +``` + +In read-only mode every mutating RPC returns `NFS3ERR_ROFS`. The +server still uses the deterministic filehandle format and metadata +subscription invalidation, so it is safe to run a read-only replica +next to a read-write head. + +### Volume Server Access Modes + +The NFS server reads chunks directly from volume servers by default +(`-volumeServerAccess=direct`), matching the `weed mount` default. Use +`publicUrl` to route through the volume servers' public URLs, or +`filerProxy` to route chunk reads through the filer host when volume +servers are not reachable from the NFS host. + +## Architecture + +### Deterministic Filehandles + +Every filer entry in this PR now carries a filer-assigned inode +(Snowflake-style, unique per filer and stable across restarts). The +NFS server encodes its filehandles as: + +``` +exportID || filehandleKind || inode || generation +``` + +- `exportID` is a CRC32 of the export root path, so a handle minted by + one export cannot resolve inside another. +- `filehandleKind` distinguishes files, directories, and symlinks. +- `inode` is the filer-assigned inode, so the handle stays valid + across rename / move as long as the entry exists. +- `generation` is the inode secondary index generation counter. + Handles from a deleted-and-recreated path decode as stale and the + server returns `NFS3ERR_STALE` instead of handing the client the + wrong object. + +### Inode Secondary Index + +The filer writes an inode secondary index record next to every entry +it creates. The record maps inode -> current paths (multi-path for +hardlinks) and carries a generation counter that bumps on +delete/recreate. Handle resolution in the NFS server goes through this +index so it never has to walk the namespace. + +Secondary index updates are best-effort: if the KV write fails, the +primary store mutation is still visible and the index is rebuilt on +the next write to the same inode. + +### Metadata Subscription + +Each NFS head subscribes to filer metadata change events via the +filer's `SubscribeMetadata` stream starting from the export root. +Chunk entries cached in the shared reader cache get invalidated as +soon as the filer confirms a rename or delete, so a write on one head +becomes visible to a reader on another head without waiting for a +cache TTL. + +## Limitations + +- **Experimental.** The on-wire protocol and filehandle codec are + stable inside a single server binary, but the server has not yet + been stress-tested as a production NFS endpoint. +- **Shared locks are not implemented.** The upstream + `github.com/willscott/go-nfs` backend exposes only local billy + `File.Lock` / `Unlock` hooks. It does not implement the NLM or + NFSv4 lock state RPCs, so cluster-wide locks, lock reclaim, and + grace periods are not available yet. The server advertises no + lock capability and returns `ErrNotSupported` for NLM requests. +- **Write size cap of 64 MiB per file.** Writes to an existing file + are buffered in the NFS server memory up to `maxBufferedWriteSize` + (64 MiB). Larger whole-file rewrites return `NFS3ERR_NOTSUPP`. A + streaming WRITE path is tracked as follow-up work. +- **Single-chunk persistence for buffered writes.** Buffered writes + larger than `maxInlineWriteSize` (4 MiB) are flushed as a single + chunk rather than split on SeaweedFS chunk boundaries. A future + refactor will route writes through the filer's standard chunking + logic. +- **No portmapper registration.** Clients must pass `port=` and + `mountport=` explicitly because the server does not register with + the system rpcbind. + +## Testing + +End-to-end integration tests live in +[`test/nfs`](https://github.com/seaweedfs/seaweedfs/tree/master/test/nfs). +Each test boots its own master + volume + filer + nfs subprocess +cluster on loopback and drives it through the NFSv3 RPC protocol via +`github.com/willscott/go-nfs-client`. No kernel NFS mount or +privileged port is required, which means the suite runs in CI on +GitHub Actions directly without host network configuration. + +```bash +cd test/nfs +make test +``` + +Covered scenarios include read/write round-trip, directory operations, +nested mkdir, rename content preservation, overwrite with explicit +truncate, large (3 MiB) binary files, arbitrary byte payloads, empty +files, symlinks, readdir ordering, remove-missing error propagation, +FSINFO limit sanity, sequential append writes, and readdir after +remove. + +## See Also + +- [[FUSE Mount]] — the original filesystem-in-userspace mount that + shares the same filer-backed data path +- [[WebDAV]] — a similar gateway over a different protocol +- [[SFTP Server]] — SSH/SFTP gateway +- [[Filer Commands and Operations]] — filer operations reference +- [NFS design notes](https://github.com/seaweedfs/seaweedfs/blob/master/weed/server/nfs/DESIGN.md) and + [development plan](https://github.com/seaweedfs/seaweedfs/blob/master/weed/server/nfs/DEV_PLAN.md) in the repo diff --git a/_Sidebar.md b/_Sidebar.md index c859c40..2c34905 100644 --- a/_Sidebar.md +++ b/_Sidebar.md @@ -64,6 +64,8 @@ ### [[SFTP Server]] +### [[NFS Server]] + ### Cloud Drive * [[Cloud Drive Benefits]] * [[Cloud Drive Architecture]]