6 Commits

Include renames

Author	SHA1	Message	Date
Chris LuandDevin	7c980774e7	FUSE mount: add posix compliance reports (#7183) ... * fuse integration: add POSIX compliance test suite Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: add posix-compliance workflow Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fuse integration: skip non-executable weed path candidates * fuse integration: size readv buffer to the written segment * fuse integration: reference posix_Makefile explicitly in recursive make calls GNU make does not propagate -f through MAKEFLAGS, so bare $(MAKE) re-read the default Makefile in this directory. * fuse integration: fail posix test targets when go test fails Every suite pipes go test through tee under /bin/sh, so the recipe status was tee's and test failures stayed green. Run recipes under bash -o pipefail and emit the color escapes with printf since bash echo does not expand them. * fuse integration: run only benchmarks in benchmark-posix go test -bench=. without -run also executes the whole suite, mixing compliance output into the benchmark log. * fuse integration: emit single-number test counts in reports grep -c prints 0 and exits nonzero when nothing matches, so '|| echo 0' produced two lines that broke the expr arithmetic and JSON fields. awk prints exactly one count either way. * fuse integration: skip the sendfile test on macOS macOS sendfile(2) only accepts a socket destination; the test copies into a regular file and always gets ENOTSOCK. * fuse integration: honor TestConfig and own only self-started resources NewFuseTestFramework dropped its config argument and Cleanup read a fresh DefaultTestConfig, so SkipCleanup never did anything. Cleanup also skipped teardown whenever TEST_SKIP_CLUSTER_SETUP was set, even when TEST_MOUNT_POINT was empty and Setup had started the cluster itself. The framework now keeps the config it was built with, runs external-mount tests inside a per-run work dir under the mount, and only skips teardown when an external mount was actually used. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fuse integration: pass the debug flag before the subcommand -v is a weed global flag; appended after master/volume/filer/mount it lands on the subcommand flagset, which rejects it and the process dies at flag parsing whenever EnableDebug is on. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fuse integration: require a real mount before running tests Setup creates the mount point before weed mount starts, so statting and listing it succeeded on the plain directory and tests ran against local disk whenever the mount was slow or died. waitForMount now waits until the mount point's device id differs from its parent's. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fuse integration: resolve the weed binary to an absolute path The started processes run with cmd.Dir at the framework's temp dir, so a relative candidate like ../weed no longer resolves. Prefer PATH, then probe the usual build outputs and return absolute paths only. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fuse integration: build the external test work dir with MkdirTemp The suite staged cloned tools and generated scripts under a predictable /tmp path another user could pre-create and swap mid-run. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fuse integration: use the pjdfstest tree setup-pjdfstest installs The test gated on a pjdfstest binary in PATH but then proved a suite it cloned itself, so a tree installed by make setup-pjdfstest was never found. Look for the built suite under external_tools first, then clone into the work dir as before. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: require a real mount before posix tests The wait loop fell back to listing the mount point, which passes on the directory created before weed mount runs — a dead or slow mount meant the suite exercised local disk. Only mountpoint counts now, and a dead mount process fails fast instead of timing out. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: collect failure logs before cleanup deletes them The always-on cleanup step removed /tmp/seaweedfs before the failure-only debug step could copy the service logs, so failed runs uploaded nothing. Run cleanup after log collection. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fuse integration: point MOUNT_POINT at TEST_MOUNT_POINT Targets that work against MOUNT_POINT (pjdfstest, nfstest, fio configs) ignored TEST_MOUNT_POINT and ran against the local-disk default while the go tests used the external mount. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fuse integration: skip the permission-denied check as root Root bypasses mode-bit checks, so writing the read-only file succeeded and the assertion failed in root-run environments like containers. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fuse integration: align the posix docs with the suite The doc listed a docker-test-posix target and make variables that do not exist, and ran go test against single files that cannot compile alone. The roadmap also still claimed the platform-specific tests this suite now implements were all skipped. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fuse integration: honor SkipCleanup in the mini-based framework too Same gap as the fuse-package framework: Cleanup read a fresh DefaultTestConfig, so a test's SkipCleanup setting was ignored. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fuse integration: normalize config in Setup and clean the work dir too Setup(nil) panicked on config.EnableDebug and external-mount Cleanup left the temp dir behind; use the stored config for the start* calls and remove tempDir alongside the external work dir. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: point external posix suites at the workflow FUSE mount * fuse integration: create the fio target directory before running * fuse integration: use unix.O_DIRECT instead of a hard-coded x86 value 0x4000 is O_DIRECT only on x86 and amd64; on arm64 it is O_DIRECTORY. The test module already depends on golang.org/x/sys. * fuse integration: run autoreconf and configure before building pjdfstest A fresh pjdfstest clone is autotools-based and has no Makefile, so the Go-side setup fallback always failed and the suite silently skipped. * ci: require external posix tools only when setup installed them setup-external-tools always exits 0 because each sub-target tolerates failure, so POSIX_REQUIRE_EXTERNAL_TOOLS was set even with no tools installed and the suite then hard-failed on the missing suites. * ci: fail the posix workflow when the mount point is not a FUSE mount The file write/read check passes on the plain directory created before weed mount, so a dead mount let the whole suite grade the local disk. * fuse integration: skip when the host has no FUSE device node Setup failed the test outright on machines without /dev/fuse instead of skipping; cover every suite entry point by skipping inside Setup. * fuse integration: drop the chmod 666 /dev/fuse troubleshooting tip World-writable device access is a security risk; the fuse group membership on the line below is the safe fix. * fuse integration: require go 1.21+ in check-go, not 1.20+ * fuse integration: refuse external suites on an unmounted path * fuse integration: verify TEST_MOUNT_POINT is a real mount * fuse integration: clean TEST_MOUNT_POINT before the parent-device check filepath.Dir of a trailing-slash path returns the mount itself, so a mounted TEST_MOUNT_POINT ending in / was refused. * fuse integration: detect WinFsp mounts on Windows in mountedAt A directory mount is a reparse point and a drive mount is a volume root; a plain directory is now refused instead of accepted. * fuse integration: strip a trailing slash off MOUNT_POINT in require-mounted The mount-table grep fallback on hosts without mountpoint(8) would not match a slash-suffixed entry. * fuse integration: require an external mount to be a FUSE filesystem A device-ID boundary alone does not prove FUSE: a tmpfs mount would pass mountedAt and report its results as FUSE compliance. Check the filesystem type (statfs magic on linux, fstypename on the BSDs); platforms without a check keep the mount and write verification. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-22 14:37:03 -07:00
Chris LuandCopilot	9434d3733d	mount: async flush on close() when writebackCache is enabled (#8727) ... * mount: async flush on close() when writebackCache is enabled When -writebackCache is enabled, defer data upload and metadata flush from Flush() (triggered by close()) to a background goroutine in Release(). This allows processes like rsync that write many small files to proceed to the next file immediately instead of blocking on two network round-trips (volume upload + filer metadata) per file. Fixes #8718 * mount: add retry with backoff for async metadata flush The metadata flush in completeAsyncFlush now retries up to 3 times with exponential backoff (1s, 2s, 4s) on transient gRPC errors. Since the chunk data is already safely on volume servers at this point, only the filer metadata reference needs persisting — retrying is both safe and effective. Data flush (FlushData) is not retried externally because UploadWithRetry already handles transient HTTP/gRPC errors internally; if it still fails, the chunk memory has been freed. * test: add integration tests for writebackCache async flush Add comprehensive FUSE integration tests for the writebackCache async flush feature (issue #8718): - Basic operations: write/read, sequential files, large files, empty files, overwrites - Fsync correctness: fsync forces synchronous flush even in writeback mode, immediate read-after-fsync - Concurrent small files: multi-worker parallel writes (rsync-like workload), multi-directory, rapid create/close - Data integrity: append after close, partial writes, file size correctness, binary data preservation - Performance comparison: writeback vs synchronous flush throughput - Stress test: 16 workers x 100 files with content verification - Mixed concurrent operations: reads, writes, creates running together Also fix pre-existing test infrastructure issues: - Rename framework.go to framework_test.go (fixes Go package conflict) - Fix undefined totalSize variable in concurrent_operations_test.go * ci: update fuse-integration workflow to run full test suite The workflow previously only ran placeholder tests (simple_test.go, working_demo_test.go) in a temp directory due to a Go module conflict. Now that framework.go is renamed to framework_test.go, the full test suite compiles and runs correctly from test/fuse_integration/. Changes: - Run go test directly in test/fuse_integration/ (no temp dir copy) - Install weed binary to /usr/local/bin for test framework discovery - Configure /etc/fuse.conf with user_allow_other for FUSE mounts - Install fuse3 for modern FUSE support - Stream test output to log file for artifact upload * mount: fix three P1 races in async flush P1-1: Reopen overwrites data still flushing in background ReleaseByHandle removes the old handle from fhMap before the deferred flush finishes. A reopen of the same inode during that window would build from stale filer metadata, overwriting the async flush. Fix: Track in-flight async flushes per inode via pendingAsyncFlush map. AcquireHandle now calls waitForPendingAsyncFlush(inode) to block until any pending flush completes before reading filer metadata. P1-2: Deferred flush races rename and unlink after close completeAsyncFlush captured the path once at entry, but rename or unlink after close() could cause metadata to be written under the wrong name or recreate a deleted file. Fix: Re-resolve path from inode via GetPath right before metadata flush. GetPath returns the current path (reflecting renames) or ENOENT (if unlinked), in which case we skip the metadata flush. P1-3: SIGINT/SIGTERM bypasses the async-flush drain grace.OnInterrupt runs hooks then calls os.Exit(0), so WaitForAsyncFlush after server.Serve() never executes on signal. Fix: Add WaitForAsyncFlush (with 10s timeout) to the WFS interrupt handler, before cache cleanup. The timeout prevents hanging on Ctrl-C when the filer is unreachable. * mount: fix P1 races — draining handle stays in fhMap P1-1: Reopen TOCTOU The gap between ReleaseByHandle removing from fhMap and submitAsyncFlush registering in pendingAsyncFlush allowed a concurrent AcquireHandle to slip through with stale metadata. Fix: Hold pendingAsyncFlushMu across both the counter decrement (ReleaseByHandle) and the pending registration. The handle is registered as pending before the lock is released, so waitForPendingAsyncFlush always sees it. P1-2: Rename/unlink can't find draining handle ReleaseByHandle deleted from fhMap immediately. Rename's FindFileHandle(inode) at line 251 could not find the handle to update entry.Name. Unlink could not coordinate either. Fix: When asyncFlushPending is true, ReleaseByHandle/ReleaseByInode leave the handle in fhMap (counter=0 but maps intact). The handle stays visible to FindFileHandle so rename can update entry.Name. completeAsyncFlush re-resolves the path from the inode (GetPath) right before metadata flush for correctness after rename/unlink. After drain, RemoveFileHandle cleans up the maps. Double-return prevention: ReleaseByHandle/ReleaseByInode return nil if counter is already <= 0, so Forget after Release doesn't start a second drain goroutine. P1-3: SIGINT deletes swap files under running goroutines After the 10s timeout, os.RemoveAll deleted the write cache dir (containing swap files) while FlushData goroutines were still reading from them. Fix: Increase timeout to 30s. If timeout expires, skip write cache dir removal so in-flight goroutines can finish reading swap files. The OS (or next mount) cleans them up. Read cache is always removed. * mount: never skip metadata flush when Forget drops inode mapping Forget removes the inode→path mapping when the kernel's lookup count reaches zero, but this does NOT mean the file was unlinked — it only means the kernel evicted its cache entry. completeAsyncFlush was treating GetPath failure as "file unlinked" and skipping the metadata flush, which orphaned the just-uploaded chunks for live files. Fix: Save dir and name at doFlush defer time. In completeAsyncFlush, try GetPath first to pick up renames; if the mapping is gone, fall back to the saved dir/name. Always attempt the metadata flush — the filer is the authority on whether the file exists, not the local inode cache. * mount: distinguish Forget from Unlink in async flush path fallback The saved-path fallback (from the previous fix) always flushed metadata when GetPath failed, which recreated files that were explicitly unlinked after close(). The same stale fallback could recreate the pre-rename path if Forget dropped the inode mapping after a rename. Root cause: GetPath failure has two meanings: 1. Forget — kernel evicted the cache entry (file still exists) 2. Unlink — file was explicitly deleted (should not recreate) Fix (three coordinated changes): Unlink (weedfs_file_mkrm.go): Before RemovePath, look up the inode and find any draining handle via FindFileHandle. Set fh.isDeleted = true so the async flush knows the file was explicitly removed. Rename (weedfs_rename.go): When renaming a file with a draining handle, update asyncFlushDir/asyncFlushName to the post-rename location. This keeps the saved-path fallback current so Forget after rename doesn't flush to the old (pre-rename) path. completeAsyncFlush (weedfs_async_flush.go): Check fh.isDeleted first — if true, skip metadata flush (file was unlinked, chunks become orphans for volume.fsck). Otherwise, try GetPath for the current path (renames); fall back to saved path if Forget dropped the mapping (file is live, just evicted from kernel cache). * test/ci: address PR review nitpicks concurrent_operations_test.go: - Restore precise totalSize assertion instead of info.Size() > 0 writeback_cache_test.go: - Check rand.Read errors in all 3 locations (lines 310, 512, 757) - Check os.MkdirAll error in stress test (line 752) - Remove dead verifyErrors variable (line 332) - Replace both time.Sleep(5s) with polling via waitForFileContent to avoid flaky tests under CI load (lines 638, 700) fuse-integration.yml: - Add set -o pipefail so go test failures propagate through tee * ci: fix fuse3/fuse package conflict on ubuntu-22.04 runner fuse3 is pre-installed on ubuntu-22.04 runners and conflicts with the legacy fuse package. Only install libfuse3-dev for the headers. * mount/page_writer: remove debug println statements Remove leftover debug println("read new data1/2") from ReadDataAt in MemChunk and SwapFileChunk. * test: fix findWeedBinary matching source directory instead of binary findWeedBinary() matched ../../weed (the source directory) via os.Stat before checking PATH, then tried to exec a directory which fails with "permission denied" on the CI runner. Fix: Check PATH first (reliable in CI where the binary is installed to /usr/local/bin). For relative paths, verify the candidate is a regular file (!info.IsDir()). Add ../../weed/weed as a candidate for in-tree builds. * test: fix framework — dynamic ports, output capture, data dirs The integration test framework was failing in CI because: 1. All tests used hardcoded ports (19333/18080/18888), so sequential tests could conflict when prior processes hadn't fully released their ports yet. 2. Data subdirectories (data/master, data/volume) were not created before starting processes. 3. Master was started with -peers=none which is not a valid address. 4. Process stdout/stderr was not captured, making failures opaque ("service not ready within timeout" with no diagnostics). 5. The unmount fallback used 'umount' instead of 'fusermount -u'. 6. The mount used -cacheSizeMB (nonexistent) instead of -cacheCapacityMB and was missing -allowOthers=false for unprivileged CI runners. Fixes: - Dynamic port allocation via freePort() (net.Listen ":0") - Explicit gRPC ports via -port.grpc to avoid default port conflicts - Create data/master and data/volume directories in Setup() - Remove invalid -peers=none and -raftBootstrap flags - Capture process output to logDir/*.log via startProcess() helper - dumpLog() prints tail of log file on service startup failure - Use fusermount3/fusermount -u for unmount - Fix mount flag names (-cacheCapacityMB, -allowOthers=false) * test: remove explicit -port.grpc flags from test framework SeaweedFS convention: gRPC port = HTTP port + 10000. Volume and filer discover the master gRPC port by this convention. Setting explicit -port.grpc on master/volume/filer broke inter-service communication because the volume server computed master gRPC as HTTP+10000 but the actual gRPC was on a different port. Remove all -port.grpc flags and let the default convention work. Dynamic HTTP ports already ensure uniqueness; the derived gRPC ports (HTTP+10000) will also be unique. --------- Co-authored-by: Copilot <copilot@github.com>	2026-03-22 15:24:08 -07:00
promalertandChris Lu	9012069bd7	chore: execute goimports to format the code (#7983) ... * chore: execute goimports to format the code Signed-off-by: promalert <promalert@outlook.com> * goimports -w . --------- Signed-off-by: promalert <promalert@outlook.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-01-07 13:06:08 -08:00
Chris Lu	d5f21fd8ba	fix: add missing backslash for volume extraArgs in helm chart (#7676) ... Fixes #7467 The -mserver argument line in volume-statefulset.yaml was missing a trailing backslash, which prevented extraArgs from being passed to the weed volume process. Also: - Extracted master server list generation logic into shared helper templates in _helpers.tpl for better maintainability - Updated all occurrences of deprecated -mserver flag to -master across docker-compose files, test files, and documentation	2025-12-08 23:21:02 -08:00
chrislu	aef5121c36	faster master startup	2025-11-18 12:06:56 -08:00
Chris LuandCopilot	9982f91b4c	Add more fuse tests (#6992) ... * add more tests * move to new package * add github action * Update fuse-integration.yml * Update fuse-integration.yml * Update test/fuse_integration/README.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update test/fuse_integration/README.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update test/fuse_integration/framework.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update test/fuse_integration/README.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update test/fuse_integration/README.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix * Update test/fuse_integration/concurrent_operations_test.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	2025-07-16 12:43:08 -07:00