Github-Mirror/seaweedfs
1
0 Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-10-24 15:00:43 +02:00
Code Issues Packages Projects Releases Wiki Activity

remove nfs

Chris Lu committed 2026-05-28 14:00:11 -07:00
1 parent 5c0e4ae91c
commit acdbc7c572
2 files changed
-368
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

NFS-Server.md
-366
@@ -1,366 +0,0 @@
# NFS Server (experimental)
SeaweedFS ships an experimental NFSv3 server that exports a filer subtree
over the NFS wire protocol. Unlike running `weed mount` and re-exporting
the FUSE mount with an external NFS server, the NFS frontend talks to the
SeaweedFS cluster directly: metadata RPCs go to the filer, and chunk
reads and writes go straight to volume servers (or through `publicUrl` /
`filerProxy` routing if you configure it). Filehandles are deterministic
and use the same filer-assigned inode that backs the `weed mount` data
path, so an NFS head shares object identity with any FUSE mount or
other filer client.
The code lives under [`weed/server/nfs/`](https://github.com/seaweedfs/seaweedfs/tree/master/weed/server/nfs).
Status: experimental, safe for evaluation and development. Do **not**
use it for production workloads that depend on advisory file locking —
shared NLM / NFSv4 lock state is not implemented yet (see
[Limitations](#limitations)).
```mermaid
flowchart LR
Client["NFS client<br/>(Linux / macOS)"]
NFS["weed nfs"]
Filer["Filer"]
Volumes["Volume servers"]
Client -- "NFSv3 RPC" --> NFS
NFS -- "metadata" --> Filer
NFS -- "chunk read / write" --> Volumes
Filer -. "AssignVolume" .-> Volumes
```
Metadata RPCs go to the filer; chunk bytes flow directly between the NFS head and volume servers.
## Overview
The NFS server provides:
- **NFSv3 wire protocol** served directly from SeaweedFS
- **Deterministic filehandles** that survive filer restarts and
multi-head failover by encoding the export ID, the filer-assigned
inode, and a per-inode generation
- **Filer-backed metadata operations** — lookup, readdir, create,
mkdir, rename, remove, symlink, readlink, and the hardlink RPC
- **Direct volume-server data path** for chunk reads and writes,
with optional `publicUrl` or `filerProxy` routing
- **Streaming writes** — each NFS WRITE RPC issues one
`AssignVolume` + one volume-server chunk upload + one filer
`UpdateEntry`, identical to how `weed filer` HTTP uploads and the
S3 gateway persist object data. The NFS server does not buffer
whole files in memory, so there is no per-file write cap.
- **Inline fast path for small files** — a write whose post-write
size still fits in the 4 MiB inline budget and that does not touch
an existing chunked file is persisted directly in the filer entry,
skipping the volume-server round-trip (matches the filer HTTP
handler's `SaveToFilerLimit` shortcut)
- **Export access controls** via CIDR or hostname allowlist
- **Metadata subscription-based cache invalidation** for each NFS head
- **Read-only export mode** for safe serving of an existing subtree
## Quick Start
### Starting the NFS Server
```bash
weed nfs \
-filer=localhost:8888 \
-port=2049 \
-filer.path=/nfs
```
`weed nfs` is a standalone binary today. The `filer.path` subtree must
already exist in the filer namespace — the server does not create the
export root on its own. Create the directory first (for example, by
uploading a placeholder file under it with the filer HTTP API) if it
does not already exist.
### Mounting from a Client
The `weed nfs` server does not register with the system portmapper
(rpcbind on port 111) by default. You have two options: bypass portmap
on the client side, or enable the built-in portmap responder on the
server side with `-portmap.bind`.
#### Client-side bypass (server default)
Pass `port=` and `mountport=` so the kernel client skips its portmap
discovery step.
Linux:
```bash
sudo mount -t nfs \
-o nfsvers=3,nolock,port=2049,mountport=2049,proto=tcp,mountproto=tcp \
127.0.0.1:/nfs /mnt/seaweedfs-nfs
```
macOS:
```bash
sudo mount -t nfs \
-o nfsvers=3,tcp,port=2049,mountport=2049,noresvport \
127.0.0.1:/nfs /Volumes/seaweedfs-nfs
```
#### Server-side portmap responder
Start `weed nfs` with `-portmap.bind=<ip>`; the server then answers
PORTMAP v2 queries on port 111 (TCP + UDP) for its own NFS v3 and
MOUNT v3 services. Clients can then mount with the plain form:
```bash
sudo mount -t nfs -o nfsvers=3,nolock 127.0.0.1:/nfs /mnt/seaweedfs-nfs
```
Binding port 111 requires root or `CAP_NET_BIND_SERVICE` and must not
collide with an existing `rpcbind` on the host.
### Mount Path Resolution
The MOUNT3 dirpath the client sends is matched against the configured
`-filer.path`:
- **Exact match** (`<host>:<filer.path>`) — mounts at the export root.
- **Subdirectory of the export** (`<host>:<filer.path>/<sub>`) — the
server resolves `<sub>` in the filer and mounts the client directly
inside it. The returned filehandle encodes the subdirectory's inode,
so the client lands at `<sub>` without having to `cd` into it. One
`weed nfs` instance can therefore expose any subtree of its export
just by varying the client mount string — useful for NFS-CSI setups
where the subdirectory is encoded in `PV.spec.nfs.path`. Missing
entries return `NFS3ERR_NOENT`; non-directory entries return
`NFS3ERR_NOTDIR`.
- **Anything else** (outside the export, empty, garbage) — falls back to
the export root and logs an INFO line. This mirrors rclone's
`serve nfs` behavior so operator typos still mount somewhere
sensible rather than producing a transport-dependent split between
"empty mount" and "mount failure".
Example with `weed nfs -filer.path=/buckets`:
```bash
# resolves to /buckets:
mount -t nfs <host>:/buckets /mnt
# resolves to /buckets/data (subexport mount):
mount -t nfs <host>:/buckets/data /mnt
# fails with NFS3ERR_NOENT (under-export, missing entry):
mount -t nfs <host>:/buckets/no-such-thing /mnt
# falls back to /buckets, with INFO log noting the typo:
mount -t nfs <host>:/wrong/path /mnt
```
File-handle scoping is unchanged: handles minted by any of these mounts
still cannot escape the configured export, regardless of what the
client asked for at MOUNT time.
## Configuration
### Command-Line Options
| Flag | Default | Description |
|------|---------|-------------|
| `-filer` | `localhost:8888` | Filer server address (ip:port) |
| `-ip.bind` | `127.0.0.1` | IP address to bind to. Defaults to loopback so the experimental server is not reachable from other hosts unless you override it. |
| `-port` | `2049` | NFS server listen port |
| `-filer.path` | (required) | Remote filer path to export. No default — operators must opt in to exporting a namespace subtree. |
| `-readOnly` | `false` | Export the filer path as read-only |
| `-allowedClients` | (empty) | Comma-separated client IPs, hostnames, or CIDR blocks allowed to connect |
| `-volumeServerAccess` | `direct` | How the NFS server reaches volume servers: `direct`, `publicUrl`, or `filerProxy` |
| `-portmap.bind` | (empty) | If set, bind a built-in PORTMAP v2 responder on `<ip>:111` (TCP + UDP) so plain `mount -t nfs` works without client-side portmap bypass. Requires root or `CAP_NET_BIND_SERVICE`; must not conflict with a running `rpcbind`. |
### Safer Defaults
Because export ACLs and full POSIX lock support are not yet available,
the server ships with defensive defaults:
- `-ip.bind` defaults to `127.0.0.1`. You must set it explicitly (for
example `-ip.bind=0.0.0.0` or a specific interface IP) to expose the
experimental server to the network.
- `-filer.path` has no default. `weed nfs` refuses to start without it,
to prevent an accidental full-namespace export. Setting
`-filer.path=/` still works, but the server logs a warning because
exporting the entire filer namespace with no ACLs in place is almost
never what you want. With `-filer.path=/`, clients can pick any
subtree at mount time via [subexport mounts](#mount-path-resolution),
but you must constrain access via `-allowedClients` or `-ip.bind`
before doing so.
### Client Allowlist
```bash
weed nfs \
-filer=localhost:8888 \
-filer.path=/nfs \
-allowedClients=10.0.0.0/8,203.0.113.42,nfs-client.example.com
```
Entries may be:
- an IPv4 or IPv6 literal — matched against the client's remote
address at connection time
- a CIDR block — matched against the client's remote address
- a hostname — resolved to IPs at startup and those IPs are then
matched. Resolution happens once; the server does not re-resolve
later.
An unmatched client gets a TCP-level close before the MOUNT RPC is
served.
### Read-Only Export
```bash
weed nfs -filer=localhost:8888 -filer.path=/nfs -readOnly
```
In read-only mode every mutating RPC returns `NFS3ERR_ROFS`. The
server still uses the deterministic filehandle format and metadata
subscription invalidation, so it is safe to run a read-only replica
next to a read-write head.
### Volume Server Access Modes
The NFS server reads chunks directly from volume servers by default
(`-volumeServerAccess=direct`), matching the `weed mount` default. Use
`publicUrl` to route through the volume servers' public URLs, or
`filerProxy` to route chunk reads through the filer host when volume
servers are not reachable from the NFS host.
## Architecture
### Deterministic Filehandles
Every filer entry in this PR now carries a filer-assigned inode
(Snowflake-style, unique per filer and stable across restarts). The
NFS server encodes its filehandles as:
```
exportID || filehandleKind || inode || generation
```
- `exportID` is a CRC32 of the export root path, so a handle minted by
one export cannot resolve inside another.
- `filehandleKind` distinguishes files, directories, and symlinks.
- `inode` is the filer-assigned inode, so the handle stays valid
across rename / move as long as the entry exists.
- `generation` is the inode secondary index generation counter.
Handles from a deleted-and-recreated path decode as stale and the
server returns `NFS3ERR_STALE` instead of handing the client the
wrong object.
### Inode Secondary Index
The filer writes an inode secondary index record next to every entry
it creates. The record maps inode -> current paths (multi-path for
hardlinks) and carries a generation counter that bumps on
delete/recreate. Handle resolution in the NFS server goes through this
index so it never has to walk the namespace.
Secondary index updates are best-effort: if the KV write fails, the
primary store mutation is still visible and the index is rebuilt on
the next write to the same inode.
### Metadata Subscription
Each NFS head subscribes to filer metadata change events via the
filer's `SubscribeMetadata` stream starting from the export root.
Chunk entries cached in the shared reader cache get invalidated as
soon as the filer confirms a rename or delete, so a write on one head
becomes visible to a reader on another head without waiting for a
cache TTL.
### Write Path
The write path is fully streaming — no whole-file buffer in server
memory, no per-file ceiling.
- `OpenFile` for write does **not** read the existing file contents
from the filer. If `O_TRUNC` is set on a non-empty file, the server
issues a single `UpdateEntry` to drop chunks + inline content; a
fresh create + `O_TRUNC` is a zero-RPC no-op.
- Every `Write` RPC either:
- **Fast path, inline**: the post-write size still fits within
`maxInlineWriteSize` (4 MiB) and the file has no existing chunks
— the bytes are merged into `entry.Content` directly via one
filer `UpdateEntry`. No volume-server round-trip.
- **Streaming path**: the server calls `AssignVolume` on the filer,
uploads the caller's bytes to the assigned volume server as a
single chunk at the current file offset, and atomically appends
the resulting chunk descriptor to the filer entry's chunk list
via `UpdateEntry`. Any pre-existing `entry.Content` is migrated
to a chunk at offset 0 in the same `UpdateEntry` so the chunk
list becomes the authoritative representation.
- `Close` is a no-op — everything is flushed synchronously inside
`Write` / `Truncate`.
- `Truncate` shrinks or grows the file by mutating the filer entry
directly (drop chunks past the new size, clip inline content,
update `FileSize`). It does not touch volume servers.
Because there is no whole-file buffer, the NFS server's per-open
memory footprint is bounded by a single WRITE RPC's payload
(typically `wsize` — Linux defaults to 1 MiB). The volume-server
chunk layout matches what `weed filer` HTTP uploads and the S3
gateway produce, so normal SeaweedFS GC, compaction, and replication
apply without special cases.
## Limitations
- **Experimental.** The on-wire protocol and filehandle codec are
stable inside a single server binary, but the server has not yet
been stress-tested as a production NFS endpoint.
- **Shared locks are not implemented.** The upstream
`github.com/willscott/go-nfs` backend exposes only local billy
`File.Lock` / `Unlock` hooks. It does not implement the NLM or
NFSv4 lock state RPCs, so cluster-wide advisory locks, lock
reclaim, and grace periods are not available. The server
advertises no lock capability and returns `ErrNotSupported` for any
lock RPC. Workloads that rely on `flock(2)` / `fcntl(2)` advisory
locks over NFS should not use this server until lock support lands —
resolving this requires either patching go-nfs to implement NLM /
NFSv4 lock state or migrating to a different NFSv4 stack.
- **One chunk per NFS WRITE RPC.** Each NFS WRITE RPC is persisted
as its own volume-server chunk rather than being coalesced with
adjacent writes into larger chunks. A Linux client that defaults to
a 1 MiB `wsize` and uploads a 100 MiB file therefore produces ~100
small chunks. The chunks resolve correctly via the filer's chunk
view layer and can be compacted later by the volume-server GC, but
larger chunk batching would cut the filer UpdateEntry load. A
future refactor will route NFS writes through the filer's standard
chunking logic (typically 4–8 MiB per chunk) to match what a
direct `weed filer` HTTP upload produces.
- **Portmap is opt-in.** The server does not register with the system
`rpcbind`. By default, clients must pass `port=` and `mountport=`
explicitly. Set `-portmap.bind=<ip>` to enable the built-in PORTMAP
v2 responder on port 111 so plain `mount -t nfs` works (see
[Mounting from a Client](#mounting-from-a-client)).
## Testing
End-to-end integration tests live in
[`test/nfs`](https://github.com/seaweedfs/seaweedfs/tree/master/test/nfs).
Each test boots its own master + volume + filer + nfs subprocess
cluster on loopback and drives it through the NFSv3 RPC protocol via
`github.com/willscott/go-nfs-client`. No kernel NFS mount or
privileged port is required, which means the suite runs in CI on
GitHub Actions directly without host network configuration.
```bash
cd test/nfs
make test
```
Covered scenarios include read/write round-trip, directory operations,
nested mkdir, rename content preservation, overwrite with explicit
truncate, large (3 MiB) binary files, arbitrary byte payloads, empty
files, symlinks, readdir ordering, remove-missing error propagation,
FSINFO limit sanity, sequential append writes, and readdir after
remove.
## See Also
- [[FUSE Mount]] — the original filesystem-in-userspace mount that
shares the same filer-backed data path
- [[WebDAV]] — a similar gateway over a different protocol
- [[SFTP Server]] — SSH/SFTP gateway
- [[Filer Commands and Operations]] — filer operations reference
- [NFS server source](https://github.com/seaweedfs/seaweedfs/tree/master/weed/server/nfs)
and [integration tests](https://github.com/seaweedfs/seaweedfs/tree/master/test/nfs)
_Sidebar.md
-2
@@ -69,8 +69,6 @@
### [[SFTP Server]]
### [[NFS Server]]
### Cloud Drive
* [[Cloud Drive Benefits]]
* [[Cloud Drive Architecture]]