mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-28 16:00:37 +01:00
Chris Lu
a5d0e4a735
* test(mount): add Samba over FUSE integration test Export a SeaweedFS FUSE mount over SMB with smbd and drive it with smbclient: file round-trips, directories, rename, large-file chunking, recursive upload, cross-protocol consistency, and deletes. A second -dlm mount adds locking coverage: POSIX fcntl byte-range locks, distributed-lock write coordination, and concurrent writers. The two cross-mount handoff checks currently fail and pin a known limitation - the distributed lock is released on FUSE Release, which the kernel can delay under contention. Runs locally via test/samba/run.sh or in Docker via the compose file; wired into CI as samba-integration.yml. * fix(cluster): release distributed lock without racing the renewal goroutine Stop() closed the cancel channel, slept 10ms, then unlocked using renewToken. A renewal in flight during that window rotates the token on the server, so the unlock may be sent with a stale token, fail with a mismatch, and leave the lock to linger until its TTL expires - stalling other mounts waiting to write the same file. Wait for the renewal goroutine to exit before unlocking. The channel close also makes the renewToken read happen-after the last renewal. * fix(cluster): poll for distributed lock acquisition without exponential backoff A mount waiting to write a file held by another mount acquired through util.RetryUntil, whose backoff grows to several seconds. Once the holder released, the waiter could sleep that long before retrying, stretching the cross-mount handoff past client timeouts. Poll at the steady ~1s cadence AttemptToLock already enforces instead. * test(mount): tighten Samba harness and mark the DLM handoff checks xfail Run the workflow for weed/cluster changes, fail fast when the filer or smbd port never opens, and fold the recursive mput result into its own assertion so it cannot false-pass. Mark the two cross-mount handoff checks expected-fail: they pin the remaining DLM liveness bug (the lock is freed only on the delayed FUSE Release) without failing CI, and turn the suite red if the handoff is ever fixed. * fix(cluster): keep a wedged renewal shutdown from sending a stale unlock If the renewal goroutine is stuck in a slow RPC, Stop() fell through to unlock anyway once it timed out waiting. A late renewal can rotate renewToken, so that unlock races it, is rejected on a stale token, and leaves the lock lingering until its TTL regardless. On the timeout path, skip the unlock and let the TTL expire the lock instead. * fix(cluster): wake the long-lived lock renewal loop promptly on Stop StartLongLivedLock's renewal loop slept uninterruptibly between attempts, up to 5*renewInterval (2.5*lockTTL) while unlocked. Stop() waits only lockTTL+2s for the goroutine to exit, so a Stop() during that backoff would time out before the goroutine woke and closed renewalDone, breaking the shutdown synchronization. Sleep on a timer with a select on cancelCh so the loop exits immediately.
121 lines
5.5 KiB
Markdown
121 lines
5.5 KiB
Markdown
# Samba on FUSE integration test
|
|
|
|
Exports a SeaweedFS FUSE mount over SMB with Samba's `smbd` and drives it with
|
|
`smbclient`, verifying that SMB file operations work correctly on top of the
|
|
mount and that data stays consistent across both protocols.
|
|
|
|
## What it checks
|
|
|
|
The functional battery in `smb_tests.sh` covers:
|
|
|
|
- connecting to the share and listing the root
|
|
- 1 MiB upload/download round-trip with content verification
|
|
- subdirectory creation and writes into it
|
|
- file rename
|
|
- 64 MiB upload/download (exercises SeaweedFS chunk splitting)
|
|
- recursive upload of a directory tree
|
|
- cross-protocol consistency: files written over SMB appear on the FUSE mount
|
|
with identical content, and files written directly on the FUSE mount are
|
|
readable over SMB
|
|
- deleting files and directory trees
|
|
|
|
The locking / concurrency battery in `lock_tests.sh` covers the harder cases a
|
|
network-filesystem backend has to get right:
|
|
|
|
- **POSIX `fcntl` byte-range locking** on the FUSE mount: a held exclusive lock
|
|
denies a conflicting lock, allows a non-overlapping range, and is reacquirable
|
|
after release (exercises the mount's `SetLk`/`GetLk`)
|
|
- **Distributed locking** (`-dlm`): a file held open for writing on one mount
|
|
blocks a writer on a second mount until it is released
|
|
- **Distributed-lock integrity**: concurrent writers to the same file from two
|
|
mounts leave exactly one intact payload, never a torn mix
|
|
- **Concurrency**: parallel writers to distinct files all succeed
|
|
|
|
Both FUSE mounts are started with `-dlm` (distributed lock manager). The second
|
|
mount (`/mnt/seaweedfs2`) exists only to contend with the smbd-backed mount in
|
|
the distributed-locking tests; both see the same filer path, so `.../share` is
|
|
the same data on each.
|
|
|
|
> Note on DLM semantics: `-dlm` coordinates *write access* (one mount writes a
|
|
> file at a time) and guarantees writes are not torn. It does not guarantee
|
|
> which concurrent writer wins or instant cross-mount read convergence — the
|
|
> holder's buffered data is flushed on close, asynchronously to lock release.
|
|
|
|
### Known issue: DLM handoff stalls under same-file contention
|
|
|
|
The two handoff checks in test 2 (`blocked SMB write succeeds after the other
|
|
mount releases` and `post-release content is the SMB writer's payload`) are
|
|
marked **expected-fail** (xfail) — they pin a remaining DLM liveness bug without
|
|
failing CI. If the handoff is fixed they flip to `[XPASS]` and turn the suite
|
|
red, a reminder to promote them to hard assertions.
|
|
|
|
When two mounts contend for the *same* file, the lock handoff does not complete
|
|
in a reasonable time because the holder releases the distributed lock only on
|
|
the FUSE `Release` op, which the kernel delays by tens of seconds after
|
|
`close()` (vs ~12 ms uncontended). The waiting writer's client gives up before
|
|
the lock frees. This is a **liveness/latency** problem, not data corruption —
|
|
the lock stays over-conservative, so no torn writes occur.
|
|
|
|
Two contributing causes have been fixed in the lock client (`weed/cluster/lock_client.go`):
|
|
|
|
- the waiter no longer polls with `util.RetryUntil`'s growing backoff; it polls
|
|
at a steady cadence so a freed lock is picked up promptly, and
|
|
- `Stop()` no longer races the renewal goroutine, which previously could send a
|
|
stale unlock token and leave the lock lingering as "owned" at the filer.
|
|
|
|
The remaining cause — the holder-side release waiting on FUSE `Release` — needs
|
|
the lock released promptly on flush/close (with care for the multi-fd case), and
|
|
is left as a follow-up.
|
|
|
|
## Layout
|
|
|
|
| File | Purpose |
|
|
| --- | --- |
|
|
| `smb_tests.sh` | SMB functional battery. Shared by both runners. |
|
|
| `lock_tests.sh` | SMB locking / concurrency battery. Shared by both runners. |
|
|
| `smb.conf.template` | Samba config; placeholders are filled in at run time. |
|
|
| `run.sh` | Local runner: `weed mini` + two `-dlm` mounts + `smbd` + both batteries, all as the current user on unprivileged ports. |
|
|
| `entrypoint.sh` | Container entrypoint: starts two `-dlm` FUSE mounts and runs `smbd`. |
|
|
| `run_inside_container.sh` | Runs both batteries inside the container against the local `smbd`. |
|
|
| `Dockerfile` | Adds Samba to the `chrislusf/seaweedfs:e2e` image. |
|
|
| `docker-compose.yml` | master + volume + filer + samba services. |
|
|
|
|
## Running locally
|
|
|
|
Requirements: `weed` on `$PATH`, `fusermount3`, and Samba's `smbd` /
|
|
`smbclient` / `smbpasswd` (Debian/Ubuntu: `apt-get install samba smbclient`).
|
|
|
|
```sh
|
|
test/samba/run.sh
|
|
```
|
|
|
|
No `sudo` is needed: `smbd` runs as the current user on port 4450 and all state
|
|
lives under a temp work dir that is cleaned up on exit.
|
|
|
|
## Running with Docker
|
|
|
|
Mirrors the CI job. Requires `/dev/fuse` and `SYS_ADMIN` (provided in the
|
|
compose file).
|
|
|
|
```sh
|
|
# build the base e2e image first (from the repo's docker/ dir)
|
|
docker compose -f test/samba/docker-compose.yml up --wait
|
|
docker compose -f test/samba/docker-compose.yml exec -T samba /run_inside_container.sh
|
|
docker compose -f test/samba/docker-compose.yml down -v
|
|
```
|
|
|
|
## CI
|
|
|
|
`.github/workflows/samba-integration.yml` runs on changes to `weed/mount/**`,
|
|
`weed/filer/**`, or `test/samba/**`. It builds the e2e image, builds the Samba
|
|
harness image on top, brings up the cluster, runs the battery, and uploads
|
|
server logs as artifacts.
|
|
|
|
## Notes
|
|
|
|
- The share disables Samba's DOS-attribute / xattr mapping and oplocks. The
|
|
SeaweedFS FUSE mount does not implement that surface, and leaving it on
|
|
produces `NT_STATUS_NOT_SUPPORTED` errors unrelated to data integrity.
|
|
- The share path is a subdirectory of the mount (`.../share`) so the runner can
|
|
verify SMB-side operations directly on the FUSE side.
|