Commit Graph
100 Commits
Author SHA1 Message Date
Chris Lu 52d91008ac ci: bump hosted runners to ubuntu-24.04
GitHub is retiring ubuntu-22.04; move all 94 runs-on pins across 40 workflows to ubuntu-24.04. rust_binaries_release: the arm64 cross step was jammy-era — noble keeps apt sources in deb822 ubuntu.sources (the sources.list edit was a silent no-op that left amd64 repos open to arm64 and 404'd) and ports suites must say noble. Pin deb822 stanzas to amd64 and add noble ports entries.
2026-10-24 23:07:28 -07:00
Chris Lu ab559bca76 storage: hash index-metric bloom filter with xxhash
The reverse index walk dedupes seen NeedleIds with a BoomFilters bloom
filter. Its default kernel is FNV-1, whose final step only XORs the last
input byte into the hash's low bits, so keys sharing the high 7 bytes of
a big-endian NeedleId land on correlated probe positions and false
positives arrive in bursts of hundreds instead of the ~0.001 target
rate. Each false positive hides a live-latest key and inflates the
rebuilt deletion counters, which is what made
TestFastLoadingNeedleMapMetrics flake past its +256 bound in CI
(observed DeletedCount drift of 267).

Swap the kernel for xxhash, which fully diffuses the key bytes; the
observed false-positive count in 300 local runs of the test drops to
max 2 (avg 0.23).
2026-10-19 01:47:25 -07:00
Chris Lu 10ce19cb17 test: poll for TopologyId convergence in multi-master failover tests
A restarted goraft master wipes its raft state and only learns the
TopologyId after the leader re-admits it and replays the entry. That
trails HTTP readiness by several seconds, so the one-shot check in
assertTopologyIdConsistent raced the propagation (CI run 39199033687:
TopologyId applied on the follower ~2s after the test gave up).
2026-10-19 00:55:41 -07:00
Chris Lu 1155e65fc5 admin: regenerate the gzipped static mirror
admin.js changed in 161310d83 without rerunning go generate, leaving
static_gz/js/admin.js.gz stale and TestStaticGzMirror failing on master.
2026-10-17 10:49:51 -07:00
Chris Lu 44ba070d83 helm: document admin.allowInsecureBind and drop stale comment
- README and values.yaml now describe the allowInsecureBind escape
  hatch alongside the non-loopback bind guard
- remove a PR reference from the CI test comment
2026-09-20 12:28:53 -07:00
Chris Lu b77c42ff32 test: use BaseEndpoint for the checksum test client 2026-09-20 01:15:11 -07:00
Chris Lu 26fc90187e s3: accept x-amz-checksum-mode from the query string, case-insensitively
Presigned HeadObject/GetObject requests hoist x-amz-checksum-mode into the
signed query string, so a strict header-only check would withhold stored
checksums on presigned reads that AWS honors.
2026-09-20 01:15:11 -07:00
Chris Lu ef463fe1af s3: read complete-request checksum values from headers or query
Presigned CompleteMultipartUpload requests hoist x-amz-checksum-type and
the full-object checksum header into the signed query string, so a
header-only lookup would skip BadDigest validation for them.
2026-09-20 01:15:11 -07:00
Chris Lu 17ad5a1419 s3: accept FULL_OBJECT checksums without per-part checksums at complete
COMPOSITE uploads must still carry every part checksum in the complete
request, but FULL_OBJECT uploads may instead supply the whole-object
checksum in an x-amz-checksum-* request header. Compare that header
against the computed object checksum and return BadDigest on mismatch,
matching AWS.
2026-09-20 01:15:11 -07:00
Chris Lu b03419ee92 s3: reject UploadPart checksum algorithms conflicting with the upload
An UploadPart that explicitly selects a different checksum algorithm than
the one declared at CreateMultipartUpload would store a checksum
CompleteMultipartUpload could never accept. Reject the conflict up front
with InvalidRequest, matching AWS.
2026-09-20 01:15:11 -07:00
Chris Lu f849b7c823 s3: validate per-part checksums in CompleteMultipartUpload
Parse the Checksum* elements of each completed part and enforce what AWS
does for uploads created with x-amz-checksum-algorithm: every part must
carry a checksum in the complete request (InvalidRequest when missing,
BadDigest when it differs from the stored part checksum), and an
x-amz-checksum-type header must match the upload resolved checksum type
(BadDigest). Add the issue-11401 reproduction as a regression test.
2026-09-20 01:15:11 -07:00
Chris Lu f15b980976 s3: UploadPart inherits the checksum algorithm of its multipart upload
AWS computes a checksum for every part of an upload created with
x-amz-checksum-algorithm, even when the part request carries no checksum
headers. Mirror that: when the part request specifies no algorithm, apply
the one stored on the upload entry so the part entry keeps a checksum
CompleteMultipartUpload can fold into the object checksum.
2026-09-20 01:15:11 -07:00
Chris Lu e35b418693 readme: quick start commands that work on a laptop
-dir=/data fails on macOS, where the root filesystem is read-only, and
on any Linux box without root; -dir=./data is created on the spot.
go install of the weed package is refused because go.mod carries
replace directives, so the install script is the shortcut instead.

Claude-Session: https://claude.ai/code/session_014apMEkkquAtAYp89paTkAT
2026-09-03 00:14:50 -07:00
Chris Lu f877b99c90 readme: fix the quick start examples
The Helm values put filer metadata on a claim through filer.data, which
is what the chart reads; enablePVC was rendering a hostPath. Claims use
the cluster default storage class instead of local-path. The AWS CLI
test carries its own credentials, the compose download includes the
Prometheus config the compose file mounts, and the disk-read claim is
per blob, since large files are chunked.

Claude-Session: https://claude.ai/code/session_014apMEkkquAtAYp89paTkAT
2026-09-02 23:55:16 -07:00
Chris Lu 87bead0ab2 readme: drop the streaming clause from the data warehouse bullet
Claude-Session: https://claude.ai/code/session_014apMEkkquAtAYp89paTkAT
2026-09-02 23:42:47 -07:00
Chris Lu 144a6c68d0 readme: keep the two objectives at the top
Claude-Session: https://claude.ai/code/session_014apMEkkquAtAYp89paTkAT
2026-09-02 23:33:44 -07:00
Chris Lu 3b8931c2f6 admin: address review feedback on the maintenance scanner fix 2026-08-23 00:26:38 -07:00
Chris Lu 8d8a25b1cf s3api: remove the duplicated listing retry helpers left by overlapping merges 2026-08-23 00:26:02 -07:00
Chris Lu 9d8acbd244 Create icon.svg 2026-08-17 16:11:27 -07:00
Chris Lu eb6d8ebd5f remove unused files 2026-07-18 23:11:07 -07:00
Chris Lu 65f2f1488a iam: test that groups and roles claims reach request-time policy evaluation
Drives the full path: AssumeRoleWithWebIdentity embeds the claims in the
session JWT, AuthenticateJWT restores them, and AuthorizeAction evaluates
ForAnyValue:StringEquals on jwt:groups / jwt:roles per bucket. The mock
OIDC provider now carries token claims through and surfaces roles,
matching the real provider.
2026-07-08 01:53:37 -07:00
Chris Lu 218f23fdf0 ci: bump wiki java client versions after publish 2026-07-07 23:41:58 -07:00
Chris Lu 56fec510e4 ci: make java release version-bump push rebase-safe 2026-07-07 23:36:44 -07:00
Chris Lu 94078b6e3f ci: default java release dry_run to false 2026-07-07 23:32:15 -07:00
Chris Lu cc9549def8 ci: add java client release workflow 2026-07-07 23:26:01 -07:00
Chris Lu 60d9c7ef4c java: bump maven-gpg-plugin to 3.2.4 for headless signing 2026-07-07 23:26:01 -07:00
Chris Lu 98e49d2d42 s3: read bucket owner and crtime from the entry-free bucket config
The owner index read the raw filer entry off BucketConfig, which no
longer carries one. Cache Crtime alongside IdentityId, and pass the
owner id rather than the entry through the ListBuckets visibility
checks so the scan and granted-bucket paths share one implementation.
2026-07-02 21:27:58 -07:00
Chris Lu 57ffef8543 fix(admin): skip task state files with no task data on load
An empty or truncated tasks/*.pb file unmarshals into a TaskStateFile
with a nil Task, and protobufToMaintenanceTask dereferenced it
immediately, panicking the whole admin process on startup. Guard the
nil case so the loader logs a warning and skips the bad file.
2026-06-26 17:36:42 -07:00
Chris Lu c01cea8786 docker release: run all platform jobs in one wave, cache rocksdb compile
Drop max-parallel so the 13 per-platform builds run together instead of two
waves of 8 (rocksdb was queuing behind the cap and starting ~8 min late).

Keep cache-to mode=max for rocksdb: its RocksDB static_lib compile is
sha-independent, so it caches across releases and stops being the ~16-min
long-pole that gates the merge fan-in. go-build variants stay mode=min.
2026-06-25 01:13:32 -07:00
Chris Lu d65ed3b557 add release version-bump workflow 2026-06-24 22:08:06 -07:00
Chris Lu eb93976166 4.35 2026-06-22 00:24:50 -07:00
Chris Lu 443b5b1184 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2026-06-17 13:45:47 -07:00
Chris Lu 0c343e76eb admin: don't log normal 2xx/3xx HTTP requests (incl. 304 cache hits) 2026-06-17 11:34:38 -07:00
Chris Lu d6da0e0e13 ci: only run heavy workflows when related paths change
Add path filters to workflows that fired on every PR/push regardless
of the diff: CodeQL, go build, the e2e/EC/vacuum/TLS/plugin-worker
integration suites, the Kafka and Postgres gateways, the S3 suites
(Ceph s3tests, s3-go, s3-tables, proxy-signature, https, example,
filer-group), TUS, and the dev binary/container builds. Each scopes
to its subsystem under weed/, its test dir, go.mod/go.sum, and the
workflow file, so docs-, helm-, terraform-, rust- or java-only
changes no longer trigger a full compile-and-test fleet.
2026-06-16 18:38:28 -07:00
Chris Lu 33df4fe2c4 storage: nil-safe ReplicaPlacement.String()
Guard a nil receiver like TTL.String() already does, so formatting a
zero-value VolumeInfo can't depend on fmt's panic recovery.
2026-06-16 10:42:43 -07:00
Chris Lu c6cf5a5bd7 4.34 2026-06-14 22:41:44 -07:00
Chris Lu 55010be19b 4.33 2026-06-11 00:52:31 -07:00
Chris Lu ed470dccb1 mini: grow volumes one at a time
Mini auto-sizes a few large volume slots, but the master pre-grows 7
volumes per new collection. Under a filer group each S3 bucket is its
own collection, so the first buckets claimed every slot and later
writes failed to assign a volume. Cap mini's volume_growth copy counts
to 1.
2026-06-08 14:51:40 -07:00
Chris Lu 5a4ff2a122 fix(mq): don't cache topic non-existence on transient filer errors
TopicExists and getTopicConfFromCache negative-cached a topic for the full
30s TTL whenever a filer lookup failed for any reason, including timeouts.
A topic created earlier then looked gone until the TTL expired, and the
metadata auto-create path couldn't heal it (CreateTopic rejects an
already-persisted conf), so producers saw UNKNOWN_TOPIC_OR_PARTITION.

Only negative-cache on a definitive ErrNotFound; let transient errors fall
through and retry against the filer.
2026-06-08 12:04:48 -07:00
Chris Lu 78da9572ae 4.32 2026-06-07 23:37:57 -07:00
Chris Lu 2a46d457ac 4.31 2026-06-01 23:32:04 -07:00
Chris Lu 34be9170f0 4.30 2026-05-30 10:52:32 -07:00
Chris Lu 5834c834e3 Refine enterprise edition feature blurb in version output and docs 2026-05-30 09:29:06 -07:00
Chris Lu ba9e74d8a7 docs: add zyner as a gold sponsor 2026-05-29 12:40:58 -07:00
Chris Lu 4f17c6661a test: keep AllocateMiniPorts off weed mini default ports
Random allocation could pick 33646 = admin.port (23646) + GrpcPortOffset.
weed mini reserves that as Admin's gRPC port even when the test only
overrides Master/Filer/S3/Iceberg, so the explicit Filer flag failed
with "reserved for gRPC calculation" and TestRisingWaveIcebergCatalog
flaked. Pre-seed the reserved set with every mini default HTTP port
plus its +10000 offset so a random pick (or its own gRPC offset) cannot
land on a service the caller left at its default.
2026-05-26 16:48:46 -07:00
Chris Lu 1355c7a102 4.29 2026-05-25 22:41:25 -07:00
Chris Lu adfd731bb8 4.28 2026-05-21 17:16:32 -07:00
Chris Lu 0accff0e4a fix(ec): log EC destination planning failures at v=2
The maintenance scanner tries to plan EC destinations for every
eligible volume, so clusters that can't place EC logged a warning per
volume every cycle. The min-node gate already skips clusters with fewer
nodes than parity shards; demote the rest to V(2).
2026-05-21 10:35:34 -07:00
Chris Lu d82b3a8d6a refactor(s3): drop unused source path in copy ETag check
ETagEntry derives the tag from chunks/Md5/remote-etag, never the entry path,
so the conditional-copy check no longer builds a bogus FullPath.
2026-05-21 09:51:50 -07:00
Chris Lu 868849392c 4.27 2026-05-20 00:25:16 -07:00
Chris Lu 01b3e4a71c template 2026-05-17 23:12:04 -07:00
Chris Lu 136eb1b7c8 4.26 2026-05-17 21:05:25 -07:00
Chris Lu 7acba59a5c 4.25 2026-05-14 12:16:26 -07:00
Chris Lu e56a3ee4a2 ci(s3-lifecycle): split into per-test matrix jobs
Each test now runs against a fresh `weed mini`, so per-collection TTL
volume budget no longer leaks across tests and exhausts the pool.
2026-05-14 11:47:24 -07:00
Chris Lu 4bac9985b4 fix(build): pin apache/thrift to v0.22.0 for 32-bit GOARCH
thrift v0.23.0 uses math.MaxUint32 as an untyped int constant in
lib/go/thrift/framed_transport.go:206, which overflows int on 32-bit
targets (openbsd/arm, linux/arm, freebsd/arm, netbsd/arm) and breaks
the release binary builds.
2026-05-13 20:56:32 -07:00
Chris Lu 1dfea8a502 4.24 2026-05-13 19:56:22 -07:00
Chris Lu b456628a7a ui(s3_lifecycle): plain-English labels for cadence fields
Dispatch Tick / Cursor Checkpoint Tick / Engine Refresh / Bootstrap Re-walk
are internal terms — operators tuning the form had to read the descriptions
to guess what each field meant. Renames the visible labels and the section
blurb; underlying field names are unchanged so stored configs still load.
2026-05-10 23:46:12 -07:00
Chris Lu ec83a87d68 perf(s3/lifecycle): defer pool Put on ShardID hasher
defer guarantees the hasher returns to the pool even if h.Write or
h.Sum panic, preventing pool leak under unexpected failure modes.
2026-05-07 17:08:50 -07:00
Chris Lu 22ebe9feb0 ci(e2e): switch FUSE Mount build to Azure Ubuntu mirror, persist buildx cache
archive.ubuntu.com from GitHub-hosted runners has been Ign:/retrying for
~60s per package, eating the Start SeaweedFS step's 10-min budget before
apt-get install finishes. The host already uses azure.archive.ubuntu.com;
do the same inside Dockerfile.e2e and drop the Retries=5 amplifier.

Also rotate /tmp/.buildx-cache-new over /tmp/.buildx-cache so the apt
layer actually survives across runs, and bump the step to 15 minutes
as a safety margin.
2026-05-05 00:22:59 -07:00
Chris Lu 6d95a5592a test(s3/policy): stop racing t.TempDir cleanup against mini shutdown
The mini cluster's admin/plugin worker keeps creating files under
admin/plugin/job_types/ for ~1s after subtests finish, while the
previous Stop() only cancelled an unobserved context and slept 500ms.
t.TempDir()'s registered RemoveAll then raced the worker and
intermittently failed with "directory not empty" (CI run 25352039081).

Manage the data dir manually so it is removed only after the mini
goroutine has exited, and wire MiniClusterCtx so cancel actually drains
master/volume/filer/admin/s3/webdav.
2026-05-04 20:16:53 -07:00
Chris Lu 6aa353716a test(kafka): snapshot consumer-group state mid-attempt for resumption flake
The onRetry hook in TestOffsetManagement/ConsumerGroupResumption only fires
after defer reader.Close(), so every dump shows the post-LeaveGroup Empty
state — useless for diagnosing why the second consumer hangs in the join
cycle. Add an onTick callback fired every 1.5s while the reader is still
joined so we can see PreparingRebalance / CompletingRebalance churn,
leader, and member assignments during the 20s attempt window.
2026-05-04 15:29:33 -07:00
Chris Lu 90b706984e docs(readme): align Docker quick start with weed mini defaults
Replace the old "server -s3" form with a parallel docker run that
uses the same env vars (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY,
S3_BUCKET) as the binary weed mini quick start. Drop the explicit
"mini" subcommand since it is the default CMD in the image.
2026-05-04 00:09:13 -07:00
Chris Lu 7f1ac8cf1a docs(readme): dedup Quick Start, drop "weed server" section
Remove the "Quick Start with Single Binary" section which duplicated
the binary-download instructions and pushed users toward "weed server".
Fold the go install alternative and the volume-scaling tip into the
weed mini section so there is one canonical Quick Start path.
2026-05-03 23:56:24 -07:00
Chris Lu 4e22d9533b docs(readme): highlight built-in Iceberg catalog and lakehouse use case
Surface the table-buckets / Iceberg REST Catalog story in the
introduction and add a Data Lakehouse Features section so readers
landing on the README see that SeaweedFS replaces both the object
store and the metastore in an Iceberg stack.
2026-05-03 23:46:08 -07:00
Chris Lu 73fc9e3833 4.23 2026-05-03 23:15:34 -07:00
Chris Lu 87bb0a4115 test(s3tables): capture weed mini stdout/stderr in catalog_spark tests
Without this, when "weed mini" fails to bind its master port within the
45s startup timeout the only diagnostic is the timeout message itself,
making flakes impossible to root-cause. The sibling catalog, catalog_dremio,
and catalog_trino packages already wire stdout/stderr through; bring
catalog_spark in line.
2026-05-03 21:39:42 -07:00
Chris Lu 9b624a73fe ci: provide a Docker tag for foundationdb release container on workflow_dispatch
The metadata-action used type=ref,event=tag, which produces no tag on
workflow_dispatch, causing build-push to fail with
"tag is needed when pushing to registry". Add a release_tag input and
build the tag from a RELEASE_TAG env, mirroring container_release_unified.yml.
2026-04-29 23:15:33 -07:00
Chris Lu 14cd426cf9 templ 2026-04-29 16:08:38 -07:00
Chris Lu 363d5caa85 test(s3-retention): purge stale buckets before each create to avoid volume exhaustion
The WORM suite creates one bucket per test, each backed by ~3 reserved
volumes on the data node. With ~30 tests and the default `weed mini`
volume cap, the data node runs out of slots midway through the run and
every PutObject after that fails with InternalError.

Hook a sweep of every test-prefix bucket into the create helpers so a
panicked or interrupted prior test cannot leak buckets into the next.
2026-04-27 22:14:20 -07:00
Chris Lu fe50da4934 test(fuse): stream verify-phase diagnostics from writeback stress test
The 45m suite alarm fires on TestWritebackCacheStressSmallFiles with no
output from the test, since t.Logf is buffered until the test completes
and the alarm panic skips that flush. Add streaming stderr progress, an
explicit verify-phase budget that t.Fatalf's with a goroutine dump on
overrun, and per-retry/per-failure logging so the next hang shows which
file(s) the mount could not read back.
2026-04-27 19:44:26 -07:00
Chris Lu ac3a756dae test(s3-retention): force-drop collection after deleteBucket to free volumes
COMPLIANCE-mode retention leaves objects that BypassGovernanceRetention cannot
clear, so the test's DeleteBucket keeps returning BucketNotEmpty and the
underlying SeaweedFS collection (with its 7 reserved volumes) leaks. After a
few leaks on the single-node `weed mini` server, the master logs "Not enough
data nodes found" and every subsequent PutObject 500s, timing the suite out.

Call the master's /col/delete admin endpoint from deleteBucket so the
collection's volumes are reclaimed even when S3-level cleanup is blocked.
2026-04-27 12:12:36 -07:00
Chris Lu fa492a9eed fix(admin): wrap plugin URLs with basePath for subdir deployments
Two more spots that broke under a subdirectory deployment:

- plugin.templ pluginRequest() called fetch(url) with relative API
  paths from 14+ callers; wrap once inside the helper so they all
  honor window.__BASE_PATH__.
- plugin_lane.templ generated <a href="/plugin/configuration?job=...">
  with an absolute path; wrap with basePath() so the link stays
  inside the deployment prefix.

Follow-up to a6adf530c.
2026-04-27 09:04:12 -07:00
Chris Lu 3ea489d013 fix(admin): wrap plugin lane fetch URL with basePath
Plugin lane page fetches API endpoints with raw absolute URLs, breaking
deployments under a subdirectory. Wrap the fetch URL with basePath() so
window.__BASE_PATH__ is honored, matching other admin pages.

Addresses https://github.com/seaweedfs/seaweedfs/issues/9240
2026-04-27 09:00:29 -07:00
Chris Lu 0b3cc8d121 4.22 2026-04-26 21:06:39 -07:00
Chris Lu 6cbcdf488c chore(mount,fuse-test): diagnostics for FUSE ConcurrentReadWrite ENOENT flake
PR #9230 attempt 1 hit an intermittent
TestConcurrentFileOperations/ConcurrentReadWrite failure where stat
returned ENOENT for a path all writers had just succeeded against, and
the captured mount.log carried no signal about which layer dropped the
entry because the relevant lookup logged at V(4).

Two diagnostic-only changes (no behavior change on the happy path):

- weed/mount/weedfs.go: in lookupEntry, when filer GetEntry returns
  ErrNotFound for a path whose inode is still tracked locally with no
  in-flight create or flush, log Warningf with inode + dirtyHandle +
  pendingFlush + localCache + dirCached. This surfaces layer-by-layer
  state at the moment of the suspicious ENOENT.

- test/fuse_integration/framework_test.go: on AssertFileExists failure,
  dump five 100ms-spaced stat retries, a parent ReadDir, and a direct
  O_RDONLY open before failing. Triangulates kernel dentry caching vs
  mount lookup vs filer state.
2026-04-26 16:57:37 -07:00
Chris Lu a14cbc176b debug(kafka): add restart flake diagnostics 2026-04-24 15:02:07 -07:00
Chris Lu a0be40e070 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2026-04-23 16:25:12 -07:00
Chris Lu b94ad82472 fix(test): stabilize ConcurrentLockContention; warn on coherence drift
TestPosixFileLocking/ConcurrentLockContention failed in CI (run
24857323067) with ENOENT when re-opening the file after all 8 workers
had successfully written and closed. The 20s openWithRetry budget was
exhausted, pointing at a real but unproven metaCache/parent-cache
coherence issue in the mount under bursts of concurrent Release.

Test: hold the initial fd open for the whole subtest; use it for the
post-workers Sync() and the verification read. Workers still exercise
the concurrent-flock invariant and per-record write correctness; the
re-open path is no longer load-bearing. On Eventually failure, dump
ReadDir of the parent, Stat, and a fresh O_RDONLY open so a future
recurrence has state to debug from. Drop the darwin-only ENOENT
t.Skip branches that hid this same flake.

Mount: in weedfs.lookupEntry, when returning ENOENT from the
"parent cached but child missing" branch, log at Warningf instead of
V(4) when the kernel is still tracking this path's inode. That
combination is the smoking-gun signal for cache drift and is rare
enough in normal use not to spam the log.
2026-04-23 15:57:35 -07:00
Chris Lu 036191c78a Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2026-04-23 11:09:59 -07:00
Chris Lu 34b236acfa test(s3api): look up NewUser by name in CreateAccessKey collision test
The memory credential store backs LoadConfiguration with a map, so the
identity order is not stable across a save/load round trip. Indexing
Identities[1] intermittently pointed at the owner identity and produced
a spurious credential leak.
2026-04-23 11:09:17 -07:00
Chris Lu ae93f87a46 adjust logo 2026-04-23 10:05:51 -07:00
Chris Lu 6e950e0e7e docs(note): add production-setup slide deck
Marp-based markdown deck walking through the three-layer production
topology (masters, volumes, filers + DB, S3 gateways, admin + workers)
plus an erasure-coding note. Uses the object-store-layout diagram on the
overview slide. Makefile renders PDF/HTML/PPTX via marp-cli.
2026-04-23 02:36:58 -07:00
Chris Lu cb882ced46 fix(test): retry ENOENT in fcntl lock subprocess helper
TestPosixFileLocking/FcntlReleaseOnClose was flaky because the
subprocess spawned by startLockHolder occasionally saw ENOENT when
opening a file the parent had just created on the FUSE mount. Retry on
ENOENT (matching the existing openWithRetry pattern used in
testConcurrentLockContention) so the subprocess waits for the mount's
dentry state to propagate before reporting the lock acquire as failed.
2026-04-22 10:33:21 -07:00
Chris Lu be9996962d fix(test): avoid port collision between master gRPC and volume ports
AllocateMiniPorts(1) reserved masterPort and masterPort+GrpcPortOffset
by holding listeners open, but closed them on return. The subsequent
AllocatePorts call bound 127.0.0.1:0, so the OS could immediately reuse
the just-released mini gRPC port as a volume port — causing the volume
server to fail at bind time with "address already in use".

Introduce AllocatePortSet(miniCount, regularCount) that holds every
listener open until the full set is chosen, and route the five volume
test cluster builders through it.
2026-04-21 23:33:57 -07:00
Chris Lu 96e5fea08e test(catalog_spark): bound weed shell invocation with 30s timeout
createTableBucket ran `weed shell` via exec.Command with no deadline.
When the shell's first command retries on a transient master connection
blip, the trailing `exit` on stdin never gets processed and the
subprocess blocks until the outer 20m `go test` timeout fires — the
surfacing symptom is a flaky 20m panic with no diagnostic output.

Wrap the invocation in exec.CommandContext with a 30s timeout, matching
the existing pattern in test/s3tables/catalog_risingwave/setup_test.go.
2026-04-21 23:27:10 -07:00
Chris Lu 8857dbfb74 fix(test): drop host port mapping from risingwave catalog test to kill TOCTOU flake
The random host port allocated by MustFreeMiniPorts was released before
docker run bound it, occasionally losing the race to another process and
failing with "address already in use". The sidecar already reaches
RisingWave via shared netns (--network container:...), so the host -p
mapping and the corresponding WaitForPort check were unused.
2026-04-19 23:26:45 -07:00
Chris Lu 3ff92f797d 4.21 2026-04-19 14:38:29 -07:00
Chris Lu 6832b9945b ci(s3tests): install libxml2/libxslt dev headers before pip install
ceph/s3-tests pins lxml without an upper bound. When pip picks a release
whose prebuilt wheel isn't published for Python 3.9 on the runner, it
falls back to sdist and fails without libxml2-dev / libxslt1-dev.
2026-04-18 01:06:39 -07:00
Chris Lu 32cbed9658 fix(fuse-tests): avoid ephemeral port reuse racing weed mini bind
freePort allocated in [20000, 55535], which overlaps the Linux ephemeral
range (32768-60999). The kernel could reuse the chosen port for an
outbound connection between the test closing its listener and weed mini
re-checking availability, causing:

  Port allocation failed: port N for Filer (specified by flag
  filer.port) is not available on 0.0.0.0 and cannot be used

Narrow the range to [20000, 32000] to stay below the ephemeral floor,
and pass -ip.bind=127.0.0.1 so mini's pre-check runs on the same IP the
test actually reserved the port on.
2026-04-17 13:47:17 -07:00
Chris Lu 664ae64646 ci(binaries_dev): serialize concurrent runs to prevent asset name collisions
Multiple master pushes within the same minute produced identical BUILD_TIME
values, causing concurrent workflow runs to race on identically-named release
assets. Upload retries hit 422 already_exists and failed the build. Adding a
concurrency group with cancel-in-progress ensures only the latest dev build
runs at a time, which is fine since only the latest dev artifacts matter.
2026-04-16 18:09:57 -07:00
Chris Lu 108e6b0d5f Delete scheduled_tasks.lock 2026-04-16 16:16:25 -07:00
Chris Lu 4bcbe9ded3 ci(helm): publish chart on tag push
Trigger the helm release workflow automatically on tag pushes so each
software release also publishes the chart to gh-pages and the OCI
registry at ghcr.io/seaweedfs. workflow_dispatch is kept as a manual
fallback.

Refs #6296
2026-04-14 02:20:06 -07:00
Chris Lu 50f25bb5cd 4.20 2026-04-13 13:25:13 -07:00
Chris Lu 512912cbb8 Update plugin_templ.go 2026-04-13 13:10:03 -07:00
Chris Lu e648c76bcf go fmt 2026-04-10 17:31:14 -07:00
Chris Lu c249eb5a8b reduce masterClient log verbosity for shell startup
Move bootstraps, gRPC stream established, and leader redirect logs
from V(0) to V(1) to keep weed shell output clean.
2026-04-08 21:28:50 -07:00
Chris Lu ac12a735c7 ci: fix dev build cleanup race between Go and Rust workflows
Both workflows trigger on push to master and race to delete assets
from the same dev release. When one deletes assets the other is also
trying to delete, the "Not Found" error fails the cleanup job and
skips all downstream build jobs.

Add continue-on-error to both cleanup steps since the error is
harmless — build steps already use overwrite: true.
2026-04-08 00:11:41 -07:00
Chris Lu 3d17bab544 fix(seaweed-volume): eliminate global S3 tier registry races in tests
Multiple Rust tests were racing on the shared global S3TierRegistry by
calling clear(), which wiped entries registered by concurrently running
tests.  Use test-specific backend IDs and targeted remove() instead of
clear() so tests no longer interfere with each other.
2026-04-07 23:11:55 -07:00
Chris Lu 0220b67115 fix(seaweed-volume): fix flaky Rust unit tests
- Increase volume_size_limit in preallocate test from 1KB to 100MB so
  disk-free fluctuations between get_disk_stats calls cannot make the
  integer-division results equal.
- Add readiness synchronization to both spawn_fake_s3_server helpers so
  the test thread waits until axum is about to serve before proceeding.
- Fix test_remote_vif_load_blocks_writes_but_allows_delete: register a
  dummy S3 backend with a test-specific ID so the volume can load its
  remote .vif without racing with other tests on the global registry.
2026-04-07 22:11:31 -07:00
Chris Lu 0bdf9b0683 4.19 2026-04-07 19:21:35 -07:00