Mini auto-sizes a few large volume slots, but the master pre-grows 7
volumes per new collection. Under a filer group each S3 bucket is its
own collection, so the first buckets claimed every slot and later
writes failed to assign a volume. Cap mini's volume_growth copy counts
to 1.
TopicExists and getTopicConfFromCache negative-cached a topic for the full
30s TTL whenever a filer lookup failed for any reason, including timeouts.
A topic created earlier then looked gone until the TTL expired, and the
metadata auto-create path couldn't heal it (CreateTopic rejects an
already-persisted conf), so producers saw UNKNOWN_TOPIC_OR_PARTITION.
Only negative-cache on a definitive ErrNotFound; let transient errors fall
through and retry against the filer.
Random allocation could pick 33646 = admin.port (23646) + GrpcPortOffset.
weed mini reserves that as Admin's gRPC port even when the test only
overrides Master/Filer/S3/Iceberg, so the explicit Filer flag failed
with "reserved for gRPC calculation" and TestRisingWaveIcebergCatalog
flaked. Pre-seed the reserved set with every mini default HTTP port
plus its +10000 offset so a random pick (or its own gRPC offset) cannot
land on a service the caller left at its default.
The maintenance scanner tries to plan EC destinations for every
eligible volume, so clusters that can't place EC logged a warning per
volume every cycle. The min-node gate already skips clusters with fewer
nodes than parity shards; demote the rest to V(2).
thrift v0.23.0 uses math.MaxUint32 as an untyped int constant in
lib/go/thrift/framed_transport.go:206, which overflows int on 32-bit
targets (openbsd/arm, linux/arm, freebsd/arm, netbsd/arm) and breaks
the release binary builds.
Dispatch Tick / Cursor Checkpoint Tick / Engine Refresh / Bootstrap Re-walk
are internal terms — operators tuning the form had to read the descriptions
to guess what each field meant. Renames the visible labels and the section
blurb; underlying field names are unchanged so stored configs still load.
archive.ubuntu.com from GitHub-hosted runners has been Ign:/retrying for
~60s per package, eating the Start SeaweedFS step's 10-min budget before
apt-get install finishes. The host already uses azure.archive.ubuntu.com;
do the same inside Dockerfile.e2e and drop the Retries=5 amplifier.
Also rotate /tmp/.buildx-cache-new over /tmp/.buildx-cache so the apt
layer actually survives across runs, and bump the step to 15 minutes
as a safety margin.
The mini cluster's admin/plugin worker keeps creating files under
admin/plugin/job_types/ for ~1s after subtests finish, while the
previous Stop() only cancelled an unobserved context and slept 500ms.
t.TempDir()'s registered RemoveAll then raced the worker and
intermittently failed with "directory not empty" (CI run 25352039081).
Manage the data dir manually so it is removed only after the mini
goroutine has exited, and wire MiniClusterCtx so cancel actually drains
master/volume/filer/admin/s3/webdav.
The onRetry hook in TestOffsetManagement/ConsumerGroupResumption only fires
after defer reader.Close(), so every dump shows the post-LeaveGroup Empty
state — useless for diagnosing why the second consumer hangs in the join
cycle. Add an onTick callback fired every 1.5s while the reader is still
joined so we can see PreparingRebalance / CompletingRebalance churn,
leader, and member assignments during the 20s attempt window.
Replace the old "server -s3" form with a parallel docker run that
uses the same env vars (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY,
S3_BUCKET) as the binary weed mini quick start. Drop the explicit
"mini" subcommand since it is the default CMD in the image.
Remove the "Quick Start with Single Binary" section which duplicated
the binary-download instructions and pushed users toward "weed server".
Fold the go install alternative and the volume-scaling tip into the
weed mini section so there is one canonical Quick Start path.
Surface the table-buckets / Iceberg REST Catalog story in the
introduction and add a Data Lakehouse Features section so readers
landing on the README see that SeaweedFS replaces both the object
store and the metastore in an Iceberg stack.
Without this, when "weed mini" fails to bind its master port within the
45s startup timeout the only diagnostic is the timeout message itself,
making flakes impossible to root-cause. The sibling catalog, catalog_dremio,
and catalog_trino packages already wire stdout/stderr through; bring
catalog_spark in line.
The metadata-action used type=ref,event=tag, which produces no tag on
workflow_dispatch, causing build-push to fail with
"tag is needed when pushing to registry". Add a release_tag input and
build the tag from a RELEASE_TAG env, mirroring container_release_unified.yml.
The WORM suite creates one bucket per test, each backed by ~3 reserved
volumes on the data node. With ~30 tests and the default `weed mini`
volume cap, the data node runs out of slots midway through the run and
every PutObject after that fails with InternalError.
Hook a sweep of every test-prefix bucket into the create helpers so a
panicked or interrupted prior test cannot leak buckets into the next.
The 45m suite alarm fires on TestWritebackCacheStressSmallFiles with no
output from the test, since t.Logf is buffered until the test completes
and the alarm panic skips that flush. Add streaming stderr progress, an
explicit verify-phase budget that t.Fatalf's with a goroutine dump on
overrun, and per-retry/per-failure logging so the next hang shows which
file(s) the mount could not read back.
COMPLIANCE-mode retention leaves objects that BypassGovernanceRetention cannot
clear, so the test's DeleteBucket keeps returning BucketNotEmpty and the
underlying SeaweedFS collection (with its 7 reserved volumes) leaks. After a
few leaks on the single-node `weed mini` server, the master logs "Not enough
data nodes found" and every subsequent PutObject 500s, timing the suite out.
Call the master's /col/delete admin endpoint from deleteBucket so the
collection's volumes are reclaimed even when S3-level cleanup is blocked.
Two more spots that broke under a subdirectory deployment:
- plugin.templ pluginRequest() called fetch(url) with relative API
paths from 14+ callers; wrap once inside the helper so they all
honor window.__BASE_PATH__.
- plugin_lane.templ generated <a href="/plugin/configuration?job=...">
with an absolute path; wrap with basePath() so the link stays
inside the deployment prefix.
Follow-up to a6adf530c.
Plugin lane page fetches API endpoints with raw absolute URLs, breaking
deployments under a subdirectory. Wrap the fetch URL with basePath() so
window.__BASE_PATH__ is honored, matching other admin pages.
Addresses https://github.com/seaweedfs/seaweedfs/issues/9240
PR #9230 attempt 1 hit an intermittent
TestConcurrentFileOperations/ConcurrentReadWrite failure where stat
returned ENOENT for a path all writers had just succeeded against, and
the captured mount.log carried no signal about which layer dropped the
entry because the relevant lookup logged at V(4).
Two diagnostic-only changes (no behavior change on the happy path):
- weed/mount/weedfs.go: in lookupEntry, when filer GetEntry returns
ErrNotFound for a path whose inode is still tracked locally with no
in-flight create or flush, log Warningf with inode + dirtyHandle +
pendingFlush + localCache + dirCached. This surfaces layer-by-layer
state at the moment of the suspicious ENOENT.
- test/fuse_integration/framework_test.go: on AssertFileExists failure,
dump five 100ms-spaced stat retries, a parent ReadDir, and a direct
O_RDONLY open before failing. Triangulates kernel dentry caching vs
mount lookup vs filer state.
TestPosixFileLocking/ConcurrentLockContention failed in CI (run
24857323067) with ENOENT when re-opening the file after all 8 workers
had successfully written and closed. The 20s openWithRetry budget was
exhausted, pointing at a real but unproven metaCache/parent-cache
coherence issue in the mount under bursts of concurrent Release.
Test: hold the initial fd open for the whole subtest; use it for the
post-workers Sync() and the verification read. Workers still exercise
the concurrent-flock invariant and per-record write correctness; the
re-open path is no longer load-bearing. On Eventually failure, dump
ReadDir of the parent, Stat, and a fresh O_RDONLY open so a future
recurrence has state to debug from. Drop the darwin-only ENOENT
t.Skip branches that hid this same flake.
Mount: in weedfs.lookupEntry, when returning ENOENT from the
"parent cached but child missing" branch, log at Warningf instead of
V(4) when the kernel is still tracking this path's inode. That
combination is the smoking-gun signal for cache drift and is rare
enough in normal use not to spam the log.
The memory credential store backs LoadConfiguration with a map, so the
identity order is not stable across a save/load round trip. Indexing
Identities[1] intermittently pointed at the owner identity and produced
a spurious credential leak.
Marp-based markdown deck walking through the three-layer production
topology (masters, volumes, filers + DB, S3 gateways, admin + workers)
plus an erasure-coding note. Uses the object-store-layout diagram on the
overview slide. Makefile renders PDF/HTML/PPTX via marp-cli.
TestPosixFileLocking/FcntlReleaseOnClose was flaky because the
subprocess spawned by startLockHolder occasionally saw ENOENT when
opening a file the parent had just created on the FUSE mount. Retry on
ENOENT (matching the existing openWithRetry pattern used in
testConcurrentLockContention) so the subprocess waits for the mount's
dentry state to propagate before reporting the lock acquire as failed.
AllocateMiniPorts(1) reserved masterPort and masterPort+GrpcPortOffset
by holding listeners open, but closed them on return. The subsequent
AllocatePorts call bound 127.0.0.1:0, so the OS could immediately reuse
the just-released mini gRPC port as a volume port — causing the volume
server to fail at bind time with "address already in use".
Introduce AllocatePortSet(miniCount, regularCount) that holds every
listener open until the full set is chosen, and route the five volume
test cluster builders through it.
createTableBucket ran `weed shell` via exec.Command with no deadline.
When the shell's first command retries on a transient master connection
blip, the trailing `exit` on stdin never gets processed and the
subprocess blocks until the outer 20m `go test` timeout fires — the
surfacing symptom is a flaky 20m panic with no diagnostic output.
Wrap the invocation in exec.CommandContext with a 30s timeout, matching
the existing pattern in test/s3tables/catalog_risingwave/setup_test.go.
The random host port allocated by MustFreeMiniPorts was released before
docker run bound it, occasionally losing the race to another process and
failing with "address already in use". The sidecar already reaches
RisingWave via shared netns (--network container:...), so the host -p
mapping and the corresponding WaitForPort check were unused.
ceph/s3-tests pins lxml without an upper bound. When pip picks a release
whose prebuilt wheel isn't published for Python 3.9 on the runner, it
falls back to sdist and fails without libxml2-dev / libxslt1-dev.
freePort allocated in [20000, 55535], which overlaps the Linux ephemeral
range (32768-60999). The kernel could reuse the chosen port for an
outbound connection between the test closing its listener and weed mini
re-checking availability, causing:
Port allocation failed: port N for Filer (specified by flag
filer.port) is not available on 0.0.0.0 and cannot be used
Narrow the range to [20000, 32000] to stay below the ephemeral floor,
and pass -ip.bind=127.0.0.1 so mini's pre-check runs on the same IP the
test actually reserved the port on.
Multiple master pushes within the same minute produced identical BUILD_TIME
values, causing concurrent workflow runs to race on identically-named release
assets. Upload retries hit 422 already_exists and failed the build. Adding a
concurrency group with cancel-in-progress ensures only the latest dev build
runs at a time, which is fine since only the latest dev artifacts matter.
Trigger the helm release workflow automatically on tag pushes so each
software release also publishes the chart to gh-pages and the OCI
registry at ghcr.io/seaweedfs. workflow_dispatch is kept as a manual
fallback.
Refs #6296
Both workflows trigger on push to master and race to delete assets
from the same dev release. When one deletes assets the other is also
trying to delete, the "Not Found" error fails the cleanup job and
skips all downstream build jobs.
Add continue-on-error to both cleanup steps since the error is
harmless — build steps already use overwrite: true.
Multiple Rust tests were racing on the shared global S3TierRegistry by
calling clear(), which wiped entries registered by concurrently running
tests. Use test-specific backend IDs and targeted remove() instead of
clear() so tests no longer interfere with each other.
- Increase volume_size_limit in preallocate test from 1KB to 100MB so
disk-free fluctuations between get_disk_stats calls cannot make the
integer-division results equal.
- Add readiness synchronization to both spawn_fake_s3_server helpers so
the test thread waits until axum is about to serve before proceeding.
- Fix test_remote_vif_load_blocks_writes_but_allows_delete: register a
dummy S3 backend with a test-specific ID so the volume can load its
remote .vif without racing with other tests on the global registry.
When cross-compiling aws-lc-sys for aarch64-unknown-linux-musl using
aarch64-linux-gnu-gcc, glibc's _FORTIFY_SOURCE generates calls to
__memcpy_chk, __fprintf_chk etc. which don't exist in musl, causing
linker errors. Disable it via CFLAGS_aarch64_unknown_linux_musl.
reqwest's default features include native-tls which depends on
openssl-sys, causing builds to fail on musl targets where OpenSSL
headers are not available. Since we already use rustls-tls, disable
default features to eliminate the openssl-sys dependency entirely.
Both container_latest.yml and container_dev.yml use Dockerfile.go_build
which expects weed-volume-prebuilt/ with pre-compiled Rust binaries, but
neither workflow produced them, causing COPY failures during docker build.
Add build-rust-binaries jobs that natively cross-compile for amd64 and
arm64, then download and place the artifacts in the Docker build context.
Also fix the trivy-scan local build path in container_latest.yml.
SocketAddr::parse() only accepts numeric IPs, so binding the gRPC
server to "localhost:18833" panicked. Use tokio::net::lookup_host()
to resolve hostnames before passing to tonic's serve_with_shutdown.
Cross-compile Rust volume server natively for amd64/arm64 using musl
targets in a separate job, then inject pre-built binaries into the
Docker build. This replaces the ~5-hour QEMU-emulated cargo build
with ~15 minutes of native cross-compilation.
The Dockerfile falls back to building from source when no pre-built
binary is found, preserving local build compatibility.
Add sync.RWMutex to ChunkTransferStatus and lock around all field
mutations in fetchAndWrite. ActiveTransfers now returns value copies
under RLock so callers get immutable snapshots.
When the sync watermark is not advancing, print each in-progress chunk
transfer with its file path, bytes received so far, and current status
(downloading, uploading, or waiting with backoff duration). This helps
diagnose which files are blocking progress during replication.
Closes#8542
The test port allocation had a TOCTOU race where GetFreePort() would
open a listener, grab the port number, then immediately close it.
When called repeatedly, the OS could recycle a just-released port,
causing two services (e.g. Filer and S3) to be assigned the same port.
Replace per-call GetFreePort() with batch AllocatePorts() that holds
all listeners open until every port is obtained, matching the pattern
already used in test/volume_server/framework/cluster.go.
- concurrent_operations_test: Add retry loop for transient I/O errors
on file close during ConcurrentDirectoryOperations
- git_operations_test: Wait for pushed objects to become visible through
FUSE mount before cloning in Phase 3
The Rust weed-volume binary requires libgcc_s.so.1 for stack unwinding
(_Unwind_* symbols). Without it, the binary fails to load in the Alpine
container with "Error loading shared library libgcc_s.so.1".
The pprof crate uses Unix-only APIs (nix, libc::pthread_t,
libc::siginfo_t, etc.) that don't exist on Windows. Move it to
[target.'cfg(unix)'.dependencies] and gate all profiling/debug
module usage with #[cfg(unix)].
After git reset --hard on a FUSE mount, the kernel dcache can
transiently show the directory then drop it moments later. Add a
1-second stabilisation delay and re-verification in
resetToCommitWithRecovery and tryPullFromCommit so that recovery
retries if the entry vanishes in that window.
- GET /api/plugin/lanes returns all lanes with status and job types
- GET /api/plugin/workers?lane=X filters workers by lane
- GET /api/plugin/scheduler-states?lane=X filters job types by lane
- GET /api/plugin/scheduler-status?lane=X returns lane-scoped status
- GET /plugin/lanes/{lane}/workers renders per-lane worker page
- SchedulerJobTypeState now includes a "lane" field
The lane worker pages show scheduler status, job type configuration,
and connected workers scoped to a single lane, with links back to
the main plugin overview.
Split the single plugin scheduler loop into independent per-lane
goroutines so that volume management, iceberg compaction, and lifecycle
operations never block each other.
Each lane has its own:
- Goroutine (laneSchedulerLoop)
- Wake channel for immediate scheduling
- Admin lock scope (e.g. "plugin scheduler:default")
- Configurable idle sleep duration
- Loop state tracking
Three lanes are defined:
- default: vacuum, volume_balance, ec_balance, erasure_coding, admin_script
- iceberg: iceberg_maintenance
- lifecycle: s3_lifecycle (new, handler coming in a later commit)
Job types are mapped to lanes via a hardcoded map with LaneDefault as
the fallback. The SchedulerJobTypeState and SchedulerStatus types now
include a Lane field for API consumers.
Add upload error counter labeled by HTTP status code, matching Go
commit 5fa5507. Code "0" indicates a transport error (no HTTP
response received). Incremented on replication failures in
do_replicated_request.
Chris Lu