Compare commits

..
Author SHA1 Message Date
Chris Lu d9d2dec9ac Merge remote-tracking branch 'origin/master' into pr11709-merge
# Conflicts:
#	weed/command/filer_remote_sync_dir.go
#	weed/command/filer_remote_sync_dir_test.go
2026-10-12 03:14:35 -07:00
Alex Russell 16110fd387 remote.sync: release a pending source only after the destination is safe
e6d6438 released the source a renamed entry's old version read before
the destination was uploaded. A cached destination that still carried
the pending marker then had nothing behind it: had its upload failed,
remote.uncache could drop its chunks and leave it reading a deleted
source. Release the source only after the upload is stamped, and only
when the entry no longer reads through a marker.
2026-10-12 07:00:56 +02:00
Alex Russell e6d6438cad remote.sync: release a pending source when the renamed entry is cached
A rename copied right away left its pending source to remote.sync,
which released it only while the destination was still remote-only.
Cached before the sync got there, the destination took the other path
and the source stayed on the remote for good. Release it there too,
with the same checks.
2026-10-12 06:51:23 +02:00
Alex Russell d9cd505a96 remote.sync: release a pending rename's source, the filer leaves it
The filer deleted a pending entry's source after copying it on a later
rename, but that rename locks only its own paths: another rename can
copy onto the source key before its entry is visible, and the cleanup
then deleted that copy. And deleting a pending entry never released its
source at all.

The filer now leaves marked sources alone. remote.sync releases one
when the pending entry is deleted, or when a later rename was copied
right away, and only while the live object is still the one the
entry's stamp describes and no remote-only entry holds the path: a
copy that landed there since has a different ETag and is kept.
2026-10-12 06:39:48 +02:00
Alex Russell 81e220e979 filer, remote.sync: follow a pending rename's source across renames
When a remote-only entry is renamed over another remote-only file and
the deferred copy fails, it reads the source key through its marker.
Renamed again the same way, the marker keeps naming that source, but:

- remote.sync completed each rename from the event's old key. For the
  second one that is the first target's old object, which it copied
  over the destination before clearing the marker. It now copies from
  the source the entry's marker names.
- For the first rename it found no entry at the new path and deleted
  the old key, which the later entry still reads. When the event's new
  entry carried a marker and the path is gone, it now leaves the key to
  the entry that moved on.
- When the filer copies a pending entry immediately on a later rename,
  it now also releases the source the copy read, after the commit and
  with the same checks as the old key.
2026-10-12 06:21:57 +02:00
Alex Russell 4e63394302 filer: tighten the pending rename source marker
Follow-ups to reading a pending remote rename copy from its source key:

- Only the filer sets the marker. The HTTP upload and tagging handlers
  store Seaweed-* request headers as extended attributes, so they skip
  this one, and a marker that leads out of the entry's own mount is
  ignored, so it cannot point reads at another remote.
- A pending entry renamed again copies from the marker's source, not
  from its own key, which still holds the old target's bytes. A further
  deferred rename keeps naming the original source; an immediate copy
  clears the marker.
- When remote.sync finds the destination already holding the content,
  it stamps the entry (clearing the marker) before deleting the source,
  and deletes the source only once the entry no longer reads it.
- A read that resolved the source just before the copy finished and
  the source went away retries with the entry as it is now.
2026-10-12 06:07:52 +02:00
Chris LuandDevin 0a59ec11d6 remote.sync: delete only the object the event's stamp describes
An object at a deleted key is not always the one the delete meant: a
rename copy can land before its filer entry is visible, and a rewrite
can upload over it. When the event recorded no remote stamp, or the
live stat cannot confirm the recorded etag, the delete now keeps the
object instead of treating it as a late upload.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 05:42:55 +02:00
Chris LuandDevin 1c003d0e52 filer: stamp the rename event with the deferred copy's result
A remote-only rename over a remote-only target queued the new entry's
event with the source's stamp before the deferred copy ran. Peers
replaying it saved a stamp the copy had already superseded, and the
sync serving such a peer rejected the destination as a match and tried
to copy the deleted source. The event now carries the copy's result
and drops the pending-source marker once it lands.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 05:38:35 +02:00
Chris LuandDevin 68109213a4 filer: read a pending remote rename copy from the source key
A remote-only rename over a remote-only target commits before its copy
lands, so reads on the new path served the target's old object until the
copy stamped the entry — and indefinitely when the copy failed without
remote.sync running. The committed entry now records the renamed-from
path, and remote-only reads resolve that object's key while the stamp
is pending; the copy's stamp and the sync's completion stamp both clear
the marker.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-12 05:37:55 +02:00
Alex Russell bc6f528d97 remote.sync: keep a changed object even with no entry at its path
Deleting a changed object when no entry holds the path also deletes a
copy a directory rename has written but not yet made visible, while a
delete of that path waits in the sync; the rename then removes the old
key and the content is gone. Keep any object whose ETag changed since
the delete, as before. An upload that finishes after the delete leaks
its object instead, which is the cheaper failure.
2026-10-12 05:36:15 +02:00
Alex Russell a76ed8dd7a filer: carry a deferred copy's stamp in the rename event; delete orphaned uploads
- The deferred copy stamps the entry with a plain UpdateEntry, which
  publishes no event, so the queued rename event still carried the old
  key's stamp to peer filers and subscribers. Give it the copy's
  RemoteEntry before it is sent. The two notify loops share the code now.
- remote.sync kept any object whose ETag changed since the delete. With
  no entry at the path that is an upload which finished after the
  delete, and keeping it leaves it on the remote for good. Look up the
  path first and keep a changed object only when an entry holds it.
2026-10-12 05:29:26 +02:00
Alex Russell 5882fb4af0 filer: finish a rename's remote work outside the committed transaction
The SQL stores keep the transaction in the context and refuse every
call through it once it has committed. The work that runs after
CommitTransaction used that context, so on postgres2 the deferred
old-key delete could not look up the old path and kept the object, and
finishRemoteCopies could not read or stamp the entry. Both failed with
"sql: transaction has already been committed or rolled back" on every
copied rename; remote.sync cleaned up after them.

Keep the context from before the transaction for the work after the
commit. The rename test store gets a strict mode that fails calls
through a committed transaction the way database/sql does.
2026-10-12 05:11:50 +02:00
Chris LuandDevin 29f9d0b805 remote.sync: match the delete event's etag before deleting the object
A rename can copy another entry's content onto a key between the filer's
check and the remote delete, and the delete then removes the copy. The
delete event carries the object's etag; delete only when the live object
still matches it.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-11 19:35:58 -07:00
Chris LuandDevin 912b027e6f filer: copy over a remote-only rename target only after the commit
Copying the source object onto the target's key before the transaction
committed destroyed the target's only content, which a rolled-back
commit cannot restore. When the target is remote-only the copy now runs
after the commit: the new entry keeps the old key's stamp until the copy
lands and re-stamps it, the target's object is kept meanwhile, and a
failed copy keeps the old object so remote.sync can finish it.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-11 19:35:58 -07:00
Alex Russell eabffd9294 filer: keep a renamed entry's old object when the path lookup fails
The deferred delete must only run once the lookup shows no remote-only
entry at the old path. A failed lookup shows nothing, and an extra
object loses nothing.
2026-10-10 21:15:21 +02:00
Alex Russell 590624908b filer: keep an old key a remote-only entry has retaken by cleanup time
A directory rename does not lock its children's paths, so between the
commit and the deferred delete of a copied child's old key another
rename can put a remote-only entry there, whose content is then only in
that object. Check the store before deleting.
2026-10-10 21:05:12 +02:00
Alex Russell d45d8750e8 filer: do not sweep the old prefix of a directory rename that copied
Removing the old directory on the remote deletes everything under its
prefix. Run after the commit, that could also take an object another
rename copied in under the same path meanwhile. When a child was copied,
leave the prefix alone and delete only the copied children's old keys.
2026-10-10 20:25:49 +02:00
Alex Russell 937d9af4a5 filer: delete a copied rename's old object only after the commit
Review follow-ups:

- The old key of a copied remote-only entry, and the old directory of a
  rename that copied one, are now deleted once the rename has committed.
  A rename that fails after the copy leaves the source entry and its
  object in place, plus an extra object at the new key.
- Every part of a multipart copy is conditioned on the ETag the copy
  started from, so an object overwritten mid-copy fails the copy
  instead of mixing versions.
- Both copy paths keep the source's server-side encryption settings.
- filer.remote.sync no longer deletes a key that a remote-only entry
  holds by the time the delete event is processed: a file deleted and
  then replaced by a remote-only rename has its content only there.
2026-10-10 20:05:34 +02:00
Alex Russell d5cff0a38f filer: copy a remote-only entry on the remote when renaming it
Renaming a file under a remote.mount whose content exists only on the
remote (never cached, or remote.uncache'd) lost it: the new entry kept
the old RemoteEntry, deleting the old entry deleted the old object, and
with no chunks nothing ever wrote the new key. filer.remote.sync could
not finish the rename either, since the old object was already gone.

moveSelfEntry now copies such an entry's object to the new key before
it touches anything, and the new entry gets the RemoteEntry of the
copy. The copy is a new optional RemoteStorageObjectCopier interface,
implemented for S3 with CopyObject up to 5 GB and UploadPartCopy above
that; the multipart copy carries the source's metadata and, where the
remote supports tagging, its tags. When an existing target is
overwritten, its delete keeps the remote object (the copy replaced it)
and its event is marked metadata-only so the sync leaves it alone too.

A remote-only rename that leaves its mount, or whose remote cannot
copy, is refused instead of deleting the only copy. Empty entries and
entries with local data rename as before.

UpdateFileMetadata builds its copy source with the same helper, which
escapes each key segment and keeps the slashes.
2026-10-10 19:39:17 +02:00
1953 changed files with 21460 additions and 254244 deletions

No files matched your search

+1 -2
View File
@@ -20,5 +20,4 @@ ignore-regex = \b[a-z]+[A-Z]\w*\b|\b[A-Z][a-z]+[A-Z]\w*\b
# unknwon: GitHub username / Go module path (`github.com/unknwon/goconfig`)
# atleast: CLI mode literal string in test/benchmark/fuse_db/bin/sqlite_verify.py
# sme: local variable for a *streamMutateError in mount tests
# nin: AWS S3 Vectors "$nin" filter operator literal (API-defined, cannot rename)
ignore-words-list = visibles,fo,te,ser,bject,unparseable,keep-alives,tread,anc,ue,auther,thirdparty,unknwon,atleast,sme,nin
ignore-words-list = visibles,fo,te,ser,bject,unparseable,keep-alives,tread,anc,ue,auther,thirdparty,unknwon,atleast,sme
@@ -1,11 +0,0 @@
name: Clean up stale SeaweedFS test mounts
description: Detach disconnected FUSE test mounts left by earlier jobs on a shared runner.
runs:
using: composite
steps:
- name: Unmount disconnected SeaweedFS test mounts
if: runner.os == 'Linux' && runner.environment == 'self-hosted'
shell: bash
env:
CLEANUP_SCRIPT: ${{ github.action_path }}/cleanup.py
run: python3 "$CLEANUP_SCRIPT"
@@ -1,89 +0,0 @@
#!/usr/bin/env python3
"""Detach disconnected SeaweedFS integration-test mounts without touching live ones."""
import errno
import os
from pathlib import Path
import re
import shutil
import subprocess
import sys
import tempfile
def test_mounts(mountinfo):
"""Read the kernel table directly: traversing a stale mount may block."""
roots = {tempfile.gettempdir(), "/tmp", "/var/tmp"}
for line in mountinfo.splitlines():
fields, filesystem = line.split(" - ", 1)
if filesystem.split()[0].split(".")[0] != "fuse":
continue
fields = fields.split()
# mountinfo escapes whitespace and backslashes with octal sequences.
target = re.sub(r"\\([0-7]{3})", lambda m: chr(int(m[1], 8)), fields[4])
path = Path(target)
# Test mounts are a mount* leaf under a seaweedfs_fuse_* scratch dir
# (seaweedfs_fuse_test_*/mount, seaweedfs_fuse_failover_*/mount0, ...)
# or the posix-compliance workflow's fixed /tmp/seaweedfs/mount.
if (not path.name.startswith("mount")
or str(path.parent.parent) not in roots
or not (path.parent.name.startswith("seaweedfs_fuse")
or path.parent.name == "seaweedfs")):
continue
yield fields[0], target
def mounted(mount_id, target):
return (mount_id, target) in test_mounts(Path("/proc/self/mountinfo").read_text())
def cleanup():
failed = False
for mount_id, target in test_mounts(Path("/proc/self/mountinfo").read_text()):
try:
# A separate, bounded probe prevents a wedged FUSE daemon from
# hanging cleanup. Only ENOTCONN proves the daemon disconnected;
# a timeout or another error is not enough to detach a live mount.
probe = subprocess.run(
[sys.executable, __file__, "--probe", target], timeout=5,
capture_output=True, check=False,
)
except subprocess.TimeoutExpired:
print(f"::warning::FUSE probe timed out; leaving {target} mounted")
continue
if probe.returncode != errno.ENOTCONN or not mounted(mount_id, target):
continue
print(f"Unmounting disconnected FUSE test mount: {target}", flush=True)
# Lazy unmount detaches a dead mount even when an old process still
# holds it open. Prefer the unprivileged FUSE helper; sudo is fallback.
for command in (
["fusermount3", "-uz", "--", target],
["fusermount", "-uz", "--", target],
["sudo", "-n", "umount", "-l", "--", target],
):
try:
subprocess.run(command, timeout=10, check=False)
except (OSError, subprocess.TimeoutExpired) as error:
print(f"{command[0]}: {error}", file=sys.stderr)
if not mounted(mount_id, target):
break
else:
print(f"::error::Could not unmount disconnected FUSE test mount: {target}")
failed = True
continue
# The detached mount's parent still holds the crashed run's data and
# logs; test_mounts only yields targets under a SeaweedFS test dir in
# a temp root, so the parent is safe to reclaim here.
shutil.rmtree(Path(target).parent, ignore_errors=True)
return int(failed)
if __name__ == "__main__":
if len(sys.argv) == 3 and sys.argv[1] == "--probe":
try:
os.stat(sys.argv[2])
except OSError as error:
sys.exit(error.errno)
sys.exit(0)
sys.exit(cleanup())
@@ -1,137 +0,0 @@
import errno
import importlib.util
from pathlib import Path
import subprocess
import unittest
from unittest.mock import patch
spec = importlib.util.spec_from_file_location("cleanup", Path(__file__).with_name("cleanup.py"))
cleanup = importlib.util.module_from_spec(spec)
spec.loader.exec_module(cleanup)
TARGET = "/tmp/seaweedfs_fuse_test_123/mount"
MOUNTINFO = f"42 1 0:9 / {TARGET} rw - fuse.seaweedfs seaweedfs rw\n"
class CleanupTests(unittest.TestCase):
def run_cleanup(self, results, mounted=True):
with patch.object(cleanup.Path, "read_text", return_value=MOUNTINFO), \
patch.object(cleanup, "mounted", side_effect=mounted if isinstance(mounted, list) else None,
return_value=mounted) as check, \
patch.object(cleanup.shutil, "rmtree") as rmtree, \
patch.object(cleanup.subprocess, "run", side_effect=results) as run:
status = cleanup.cleanup()
return status, run, check, rmtree
def result(self, status):
return subprocess.CompletedProcess([], status)
def test_only_fuse_integration_test_mounts_are_candidates(self):
rows = MOUNTINFO
rows += MOUNTINFO.replace("fuse.seaweedfs", "ext4")
rows += MOUNTINFO.replace(TARGET, "/mnt/production")
rows += MOUNTINFO.replace(TARGET, "/tmp/seaweedfs_fuse_test_123/data")
rows += MOUNTINFO.replace(TARGET, "/home/user/seaweedfs_fuse_test_123/mount")
rows += MOUNTINFO.replace(TARGET, "/tmp/seaweedfs/data")
self.assertEqual(list(cleanup.test_mounts(rows)), [("42", TARGET)])
def test_all_test_mount_layouts_are_candidates(self):
targets = (
"/tmp/seaweedfs_fuse_test_123/mount",
"/tmp/seaweedfs_fuse_failover_456/mount0",
"/tmp/seaweedfs_fuse_failover_456/mount1",
"/var/tmp/seaweedfs_fuse_p2p_test_789/mount2",
"/tmp/seaweedfs/mount",
)
rows = "".join(
MOUNTINFO.replace("42", str(40 + i)).replace(TARGET, target)
for i, target in enumerate(targets)
)
self.assertEqual(
list(cleanup.test_mounts(rows)),
[(str(40 + i), target) for i, target in enumerate(targets)],
)
def test_mountinfo_paths_are_unescaped_once(self):
target = "/tmp/seaweedfs_fuse_test_space and\\slash/mount"
encoded = target.replace("\\", r"\134").replace(" ", r"\040")
rows = MOUNTINFO.replace(TARGET, encoded)
self.assertEqual(list(cleanup.test_mounts(rows)), [("42", target)])
def test_no_mounts_is_a_noop(self):
with patch.object(cleanup.Path, "read_text", return_value=""), \
patch.object(cleanup.subprocess, "run") as run:
self.assertEqual(cleanup.cleanup(), 0)
run.assert_not_called()
def test_healthy_mount_is_untouched(self):
status, run, _, _ = self.run_cleanup([self.result(0)])
self.assertEqual(status, 0)
self.assertEqual(run.call_count, 1)
def test_other_probe_errors_are_untouched(self):
for error in (errno.EACCES, errno.ENOENT, errno.EIO):
with self.subTest(error=error):
status, run, _, _ = self.run_cleanup([self.result(error)])
self.assertEqual(status, 0)
self.assertEqual(run.call_count, 1)
def test_probe_timeout_does_not_detach_a_possibly_live_mount(self):
status, run, _, _ = self.run_cleanup([subprocess.TimeoutExpired("probe", 5)])
self.assertEqual(status, 0)
self.assertEqual(run.call_count, 1)
def test_disconnected_mount_is_lazily_unmounted(self):
status, run, _, _ = self.run_cleanup([self.result(errno.ENOTCONN), self.result(0)], [True, False])
self.assertEqual(status, 0)
self.assertEqual(run.call_args_list[1].args[0], ["fusermount3", "-uz", "--", TARGET])
self.assertEqual(run.call_args_list[0].kwargs["timeout"], 5)
self.assertEqual(run.call_args_list[1].kwargs["timeout"], 10)
def test_replaced_or_removed_mount_is_untouched(self):
status, run, _, _ = self.run_cleanup([self.result(errno.ENOTCONN)], False)
self.assertEqual(status, 0)
self.assertEqual(run.call_count, 1)
def test_mount_identity_is_checked_without_stat(self):
with patch.object(cleanup.Path, "read_text", return_value=MOUNTINFO):
self.assertTrue(cleanup.mounted("42", TARGET))
self.assertFalse(cleanup.mounted("43", TARGET))
def test_missing_or_timed_out_helper_uses_fallback(self):
for error in (FileNotFoundError(), subprocess.TimeoutExpired("fusermount3", 10)):
with self.subTest(error=error):
status, run, _, _ = self.run_cleanup(
[self.result(errno.ENOTCONN), error, self.result(0)], [True, True, False])
self.assertEqual(status, 0)
self.assertEqual(run.call_args.args[0], ["fusermount", "-uz", "--", TARGET])
def test_failed_helpers_fall_back_to_sudo(self):
status, run, _, _ = self.run_cleanup(
[self.result(errno.ENOTCONN), self.result(1), self.result(1), self.result(0)],
[True, True, True, False])
self.assertEqual(status, 0)
self.assertEqual(run.call_args.args[0], ["sudo", "-n", "umount", "-l", "--", TARGET])
def test_cleanup_fails_when_stale_mount_remains(self):
status, run, _, rmtree = self.run_cleanup(
[self.result(errno.ENOTCONN), self.result(1), self.result(1), self.result(1)])
self.assertEqual(status, 1)
self.assertEqual(run.call_count, 4)
rmtree.assert_not_called()
def test_unmounted_test_dir_is_reclaimed(self):
status, _, _, rmtree = self.run_cleanup(
[self.result(errno.ENOTCONN), self.result(0)], [True, False])
self.assertEqual(status, 0)
rmtree.assert_called_once_with(Path(TARGET).parent, ignore_errors=True)
def test_probe_process_reports_errno(self):
for target, expected in ((str(Path(__file__)), 0), (str(Path(__file__) / "absent"), errno.ENOTDIR)):
with self.subTest(target=target):
result = subprocess.run([cleanup.sys.executable, cleanup.__file__, "--probe", target], check=False)
self.assertEqual(result.returncode, expected)
if __name__ == "__main__":
unittest.main()
@@ -23,14 +23,8 @@ runs:
bin=$(command -v fusermount3 || true)
if [ -z "$bin" ]; then
# Self-hosted runners are not guaranteed to carry fuse3.
sudo apt-get update
sudo apt-get install -y fuse3
bin=$(command -v fusermount3 || true)
if [ -z "$bin" ]; then
echo "no fusermount3 in PATH after installing fuse3" >&2
exit 1
fi
echo "no fusermount3 in PATH" >&2
exit 1
fi
if can_mount "$bin"; then
ls -l "$bin"
@@ -1,57 +0,0 @@
name: Set up a musl Rust target
description: Install the stable toolchain for a static Linux musl target, with the C toolchain and the link shims aws-lc-sys needs.
inputs:
target:
description: x86_64-unknown-linux-musl or aarch64-unknown-linux-musl
required: true
runs:
using: composite
steps:
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
with:
targets: ${{ inputs.target }}
- name: Install musl tools (amd64)
if: inputs.target == 'x86_64-unknown-linux-musl'
shell: bash
run: |
sudo apt-get install -y musl-tools
# C deps (aws-lc-sys) must compile against musl headers, not the host
# gcc's glibc headers — glibc's C23 redirects (__isoc23_sscanf,
# __isoc23_strtol) don't exist in musl and fail the link.
echo "CC_x86_64_unknown_linux_musl=musl-gcc" >> "$GITHUB_ENV"
- name: Install cross-compilation tools (arm64)
if: inputs.target == 'aarch64-unknown-linux-musl'
shell: bash
run: |
sudo apt-get install -y gcc-aarch64-linux-gnu
echo "CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_LINKER=aarch64-linux-gnu-gcc" >> "$GITHUB_ENV"
# Disable glibc fortify source — its __memcpy_chk etc. symbols don't exist in musl
echo "CFLAGS_aarch64_unknown_linux_musl=-U_FORTIFY_SOURCE" >> "$GITHUB_ENV"
# aws-lc-sys compiles against glibc headers, which redirect
# sscanf/strtol to C23 __isoc23_* symbols musl does not provide.
# Link a tiny forwarding shim so those references resolve to the
# musl implementations. Prototypes are declared manually: including
# glibc headers here would re-redirect the calls back to __isoc23_*.
cat > /tmp/isoc23-shim.c <<'SHIM'
#include <stdarg.h>
extern int vsscanf(const char *s, const char *fmt, va_list ap);
extern long strtol(const char *nptr, char **endptr, int base);
int __isoc23_sscanf(const char *s, const char *fmt, ...) {
va_list ap;
va_start(ap, fmt);
int r = vsscanf(s, fmt, ap);
va_end(ap);
return r;
}
long __isoc23_strtol(const char *nptr, char **endptr, int base) {
return strtol(nptr, endptr, base);
}
SHIM
aarch64-linux-gnu-gcc -O2 -c /tmp/isoc23-shim.c -o /tmp/isoc23-shim.o
# link-args land after libc on the link line; rescan musl libc so
# the shim's own vsscanf/strtol references resolve, and libgcc for
# the fp128 builtins musl's vfscanf/floatscan pull in on aarch64.
echo "CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_RUSTFLAGS=-Clink-arg=/tmp/isoc23-shim.o -Clink-arg=-lc -Clink-arg=-lgcc" >> "$GITHUB_ENV"
+1 -1
View File
@@ -15,7 +15,7 @@ runs:
using: composite
steps:
- name: Install cosign
uses: sigstore/cosign-installer@v4.1.3
uses: sigstore/cosign-installer@v4.1.2
- name: Sign
shell: bash
+1 -25
View File
@@ -9,8 +9,6 @@ logs a warning rather than under-reporting.
import json
import os
import sys
import time
import urllib.error
import urllib.request
from datetime import datetime
@@ -24,28 +22,6 @@ REPO = "seaweedfs/seaweedfs"
TOKEN = os.environ["GITHUB_TOKEN"]
OUT = os.environ.get("OUT", "note/star_history.svg")
PAGE_CAP = 400 # GitHub's hard limit on stargazer pagination
REQUEST_ATTEMPTS = 5
RETRYABLE_STATUS_CODES = {429, 500, 502, 503, 504}
def open_url(req):
for attempt in range(REQUEST_ATTEMPTS):
try:
return urllib.request.urlopen(req, timeout=30)
except urllib.error.HTTPError as error:
if error.code not in RETRYABLE_STATUS_CODES or attempt == REQUEST_ATTEMPTS - 1:
raise
retry_after = error.headers.get("Retry-After")
try:
delay = min(float(retry_after), 60) if retry_after else 2**attempt
except ValueError:
delay = 2**attempt
error.close()
except urllib.error.URLError:
if attempt == REQUEST_ATTEMPTS - 1:
raise
delay = 2**attempt
time.sleep(delay)
def fetch_stargazers():
@@ -62,7 +38,7 @@ def fetch_stargazers():
"User-Agent": "seaweedfs-star-history",
},
)
with open_url(req) as resp:
with urllib.request.urlopen(req) as resp:
batch = json.load(resp)
if not batch:
break
-75
View File
@@ -1,75 +0,0 @@
import importlib.util
import os
from pathlib import Path
import sys
import types
import unittest
from unittest.mock import patch
from urllib.error import HTTPError
def load_star_history():
matplotlib = types.ModuleType("matplotlib")
matplotlib.use = lambda backend: None
pyplot = types.ModuleType("matplotlib.pyplot")
dates = types.ModuleType("matplotlib.dates")
dates.AutoDateLocator = object
dates.DateFormatter = object
modules = {
"matplotlib": matplotlib,
"matplotlib.pyplot": pyplot,
"matplotlib.dates": dates,
}
with patch.dict(sys.modules, modules), patch.dict(os.environ, {"GITHUB_TOKEN": "test"}):
path = Path(__file__).with_name("star_history.py")
spec = importlib.util.spec_from_file_location("star_history", path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
star_history = load_star_history()
class StarHistoryTests(unittest.TestCase):
def http_error(self, code):
return HTTPError("https://api.github.com", code, "error", {}, None)
@patch.object(star_history.time, "sleep")
@patch.object(star_history.urllib.request, "urlopen")
def test_retries_server_error(self, urlopen, sleep):
response = object()
urlopen.side_effect = [self.http_error(500), response]
result = star_history.open_url(object())
self.assertIs(result, response)
self.assertEqual(urlopen.call_count, 2)
sleep.assert_called_once_with(1)
@patch.object(star_history.time, "sleep")
@patch.object(star_history.urllib.request, "urlopen")
def test_does_not_retry_non_transient_http_error(self, urlopen, sleep):
error = self.http_error(404)
urlopen.side_effect = error
with self.assertRaises(HTTPError):
star_history.open_url(object())
urlopen.assert_called_once()
sleep.assert_not_called()
@patch.object(star_history.time, "sleep")
@patch.object(star_history.urllib.request, "urlopen")
def test_stops_after_retry_limit(self, urlopen, sleep):
urlopen.side_effect = self.http_error(500)
with self.assertRaises(HTTPError):
star_history.open_url(object())
self.assertEqual(urlopen.call_count, star_history.REQUEST_ATTEMPTS)
self.assertEqual(sleep.call_count, star_history.REQUEST_ATTEMPTS - 1)
if __name__ == "__main__":
unittest.main()
-30
View File
@@ -1,30 +0,0 @@
name: "actions: unit tests"
on:
push:
branches: [ master ]
paths:
- '.github/actions/**'
- '.github/workflows/action-tests.yml'
pull_request:
branches: [ master ]
paths:
- '.github/actions/**'
- '.github/workflows/action-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
jobs:
cleanup-stale-fuse-mounts:
runs-on: ubuntu-latest
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Run cleanup action unit tests
run: python3 .github/actions/cleanup-stale-fuse-mounts/test_cleanup.py
@@ -1,37 +0,0 @@
name: "Cancel runs of closed PRs"
# Merging or closing a PR leaves its pull_request runs going: concurrency
# groups cancel a run only when a newer push arrives. pull_request_target gives
# this job a token that can cancel runs, fork PRs included. It checks out and
# runs no PR code.
on:
pull_request_target:
types: [closed]
permissions:
actions: write
pull-requests: read
jobs:
cancel:
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: Cancel the PR's queued and running pull_request runs
env:
GH_TOKEN: ${{ github.token }}
GH_REPO: ${{ github.repository }}
PR: ${{ github.event.pull_request.number }}
HEAD_REF: ${{ github.event.pull_request.head.ref }}
run: |
set -euo pipefail
# Match the PR's own commits: forks reuse branch names such as master.
shas=$(gh api --paginate "repos/$GH_REPO/pulls/$PR/commits" --jq '.[].sha')
for status in queued in_progress waiting pending requested; do
gh run list --branch "$HEAD_REF" --event pull_request --status "$status" \
--limit 200 --json databaseId,headSha --jq '.[] | "\(.databaseId) \(.headSha)"'
done | while read -r id sha; do
grep -qx "$sha" <<<"$shas" || continue
echo "cancel run $id at $sha"
gh run cancel "$id" || true
done
+5 -11
View File
@@ -9,8 +9,8 @@ on:
- '.github/workflows/codeql.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
group: ${{ github.head_ref }}/codeql
cancel-in-progress: true
jobs:
CodeQL-Build:
@@ -25,15 +25,9 @@ jobs:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@v4.39.3
uses: github/codeql-action/init@v4.38.2
# Override language selection by uncommenting this and choosing your languages
with:
languages: go
@@ -41,7 +35,7 @@ jobs:
# Autobuild attempts to build any compiled languages (C/C++, C#, or Java).
# If this step fails, then you should remove it and run the build manually (see below).
- name: Autobuild
uses: github/codeql-action/autobuild@v4.39.3
uses: github/codeql-action/autobuild@v4.38.2
# ℹ️ Command-line programs to run using the OS shell.
# 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun
@@ -55,4 +49,4 @@ jobs:
# make release
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v4.39.3
uses: github/codeql-action/analyze@v4.38.2
-4
View File
@@ -8,10 +8,6 @@ on:
pull_request:
branches: [master]
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
+20 -36
View File
@@ -8,16 +8,11 @@ on:
- 'seaweed-volume/**'
- 'seaweed-common/**'
- 'seaweed-worker/**'
- 'seaweed-vector/**'
- 'docker/**'
- 'go.mod'
- 'go.sum'
- '.github/workflows/container_dev.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -25,7 +20,7 @@ jobs:
# ── Pre-build the Rust binaries natively ────────────────────────────
build-rust-binaries:
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
strategy:
matrix:
include:
@@ -33,17 +28,29 @@ jobs:
arch: amd64
- target: aarch64-unknown-linux-musl
arch: arm64
cross: true
steps:
- name: Checkout
uses: actions/checkout@v7
- name: Set up the musl target
uses: ./.github/actions/rust-musl-target
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
with:
target: ${{ matrix.target }}
targets: ${{ matrix.target }}
- name: Install musl tools (amd64)
if: ${{ !matrix.cross }}
run: sudo apt-get install -y musl-tools
- name: Install cross-compilation tools (arm64)
if: matrix.cross
run: |
sudo apt-get install -y gcc-aarch64-linux-gnu
echo "CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_LINKER=aarch64-linux-gnu-gcc" >> "$GITHUB_ENV"
# Disable glibc fortify source — its __memcpy_chk etc. symbols don't exist in musl
echo "CFLAGS_aarch64_unknown_linux_musl=-U_FORTIFY_SOURCE" >> "$GITHUB_ENV"
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
@@ -51,8 +58,7 @@ jobs:
~/.cargo/git
seaweed-volume/target
seaweed-worker/target/${{ matrix.target }}/release
seaweed-vector/target/${{ matrix.target }}/release
key: rust-docker-dev-${{ matrix.target }}-${{ hashFiles('seaweed-volume/Cargo.lock', 'seaweed-worker/Cargo.lock', 'seaweed-vector/Cargo.lock') }}
key: rust-docker-dev-${{ matrix.target }}-${{ hashFiles('seaweed-volume/Cargo.lock', 'seaweed-worker/Cargo.lock') }}
restore-keys: |
rust-docker-dev-${{ matrix.target }}-
@@ -85,12 +91,6 @@ jobs:
cargo build --release -p weed-lance-worker --target ${{ matrix.target }}
cp target/${{ matrix.target }}/release/weed-worker ../weed-worker-${{ matrix.arch }}
- name: Build the vector engine
run: |
cd seaweed-vector
cargo build --release -p weed-vector-server --target ${{ matrix.target }}
cp target/${{ matrix.target }}/release/weed-vector ../weed-vector-${{ matrix.arch }}
- name: Upload artifacts
uses: actions/upload-artifact@v7
with:
@@ -98,7 +98,6 @@ jobs:
path: |
weed-volume-normal-${{ matrix.arch }}
weed-worker-${{ matrix.arch }}
weed-vector-${{ matrix.arch }}
build-dev-containers:
needs: [build-rust-binaries]
@@ -136,17 +135,8 @@ jobs:
echo "Placed pre-built Rust worker for ${arch}"
fi
done
mkdir -p docker/weed-vector-prebuilt
for arch in amd64 arm64; do
src="./rust-bins/weed-vector-${arch}"
if [ -f "$src" ]; then
cp "$src" "docker/weed-vector-prebuilt/weed-vector-${arch}"
echo "Placed pre-built vector engine for ${arch}"
fi
done
ls -la docker/weed-volume-prebuilt/
ls -la docker/weed-worker-prebuilt/
ls -la docker/weed-vector-prebuilt/
- name: Docker meta
id: docker_meta
@@ -180,14 +170,14 @@ jobs:
- name: Login to Docker Hub
if: github.event_name != 'pull_request'
uses: docker/login-action@v4.8.0
uses: docker/login-action@v4.6.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Login to GHCR
if: github.event_name != 'pull_request'
uses: docker/login-action@v4.8.0
uses: docker/login-action@v4.6.0
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
@@ -203,12 +193,6 @@ jobs:
platforms: linux/amd64, linux/arm64
tags: ${{ steps.docker_meta.outputs.tags }}
labels: ${{ steps.docker_meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max
# Key the checkout layer on the pushed sha so cached layers can't
# serve stale code; the Dockerfile fetch fallback picks it up.
build-args: |
BRANCH=${{ github.sha }}
- name: Sign
if: github.event_name != 'pull_request'
@@ -24,10 +24,6 @@ on:
required: false
default: ''
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.event.inputs.image_tag || format('foundationdb_{0}_seaweedfs_{1}', github.event.inputs.fdb_version || '7.4.5', github.event.inputs.seaweedfs_ref || github.ref_name) }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -68,7 +64,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: go.mod
cache: ${{ runner.environment == 'github-hosted' }}
- name: Run FoundationDB tagged tests
env:
@@ -141,7 +136,7 @@ jobs:
- name: Login to Docker Hub
if: github.event_name != 'pull_request'
uses: docker/login-action@v4.8.0
uses: docker/login-action@v4.6.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
+26 -55
View File
@@ -30,11 +30,6 @@ on:
type: boolean
default: false
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.event.inputs.image_tag || github.run_id }}
cancel-in-progress: false
queue: max
permissions:
contents: read
security-events: write
@@ -66,7 +61,7 @@ jobs:
# ── Pre-build the Rust binaries natively ────────────────────────────
build-rust-binaries:
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
strategy:
matrix:
include:
@@ -74,21 +69,31 @@ jobs:
arch: amd64
- target: aarch64-unknown-linux-musl
arch: arm64
cross: true
steps:
- name: Checkout
uses: actions/checkout@v7
with:
ref: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.source_ref || github.ref }}
# The checkout above is the source ref, which may predate these
# actions; take them from master, the ref this fallback runs from.
- name: Set up the musl target
uses: seaweedfs/seaweedfs/.github/actions/rust-musl-target@master
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
with:
target: ${{ matrix.target }}
targets: ${{ matrix.target }}
- name: Install musl tools (amd64)
if: ${{ !matrix.cross }}
run: sudo apt-get install -y musl-tools
- name: Install cross-compilation tools (arm64)
if: matrix.cross
run: |
sudo apt-get install -y gcc-aarch64-linux-gnu
echo "CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_LINKER=aarch64-linux-gnu-gcc" >> "$GITHUB_ENV"
# Disable glibc fortify source — its __memcpy_chk etc. symbols don't exist in musl
echo "CFLAGS_aarch64_unknown_linux_musl=-U_FORTIFY_SOURCE" >> "$GITHUB_ENV"
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
@@ -96,8 +101,7 @@ jobs:
~/.cargo/git
seaweed-volume/target
seaweed-worker/target/${{ matrix.target }}/release
seaweed-vector/target/${{ matrix.target }}/release
key: rust-docker-${{ matrix.target }}-${{ hashFiles('seaweed-volume/Cargo.lock', 'seaweed-worker/Cargo.lock', 'seaweed-vector/Cargo.lock') }}
key: rust-docker-${{ matrix.target }}-${{ hashFiles('seaweed-volume/Cargo.lock', 'seaweed-worker/Cargo.lock') }}
restore-keys: |
rust-docker-${{ matrix.target }}-
@@ -138,17 +142,6 @@ jobs:
cargo build --release -p weed-lance-worker --target ${{ matrix.target }}
cp target/${{ matrix.target }}/release/weed-worker ../weed-worker-${{ matrix.arch }}
- name: Build the vector engine
run: |
# An older source_ref predates the engine; its image goes without one.
if [ ! -f seaweed-vector/crates/server/Cargo.toml ]; then
echo "No vector engine in this source ref"
exit 0
fi
cd seaweed-vector
cargo build --release -p weed-vector-server --target ${{ matrix.target }}
cp target/${{ matrix.target }}/release/weed-vector ../weed-vector-${{ matrix.arch }}
- name: Upload artifacts
uses: actions/upload-artifact@v7
with:
@@ -157,7 +150,6 @@ jobs:
weed-volume-large-disk-${{ matrix.arch }}
weed-volume-normal-${{ matrix.arch }}
weed-worker-${{ matrix.arch }}
weed-vector-${{ matrix.arch }}
build:
needs: [setup, build-rust-binaries]
@@ -172,13 +164,10 @@ jobs:
uses: actions/checkout@v7
with:
ref: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.source_ref || github.ref }}
- name: Clean up stale FUSE mounts
uses: seaweedfs/seaweedfs/.github/actions/cleanup-stale-fuse-mounts@master
- name: Free Disk Space
run: |
echo "Available disk space before cleanup:"
timeout 15 df -h || true
df -h
# Remove pre-installed tools
sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /opt/hostedtoolcache/CodeQL
# Clean package managers
@@ -190,7 +179,7 @@ jobs:
[ -d ~/.cache/go-build ] && rm -rf ~/.cache/go-build || true
[ -d /go/pkg ] && rm -rf /go/pkg || true
echo "Available disk space after cleanup:"
timeout 15 df -h || true
df -h
- name: Configure variant
id: config
@@ -230,17 +219,8 @@ jobs:
echo "Placed pre-built Rust worker for ${arch}"
fi
done
mkdir -p docker/weed-vector-prebuilt
for arch in amd64 arm64; do
src="./rust-bins/weed-vector-${arch}"
if [ -f "$src" ]; then
cp "$src" "docker/weed-vector-prebuilt/weed-vector-${arch}"
echo "Placed pre-built vector engine for ${arch}"
fi
done
ls -la docker/weed-volume-prebuilt/
ls -la docker/weed-worker-prebuilt/
ls -la docker/weed-vector-prebuilt/
- name: Docker meta
id: docker_meta
@@ -270,13 +250,13 @@ jobs:
buildkitd-config: /tmp/buildkitd.toml
- name: Login to Docker Hub
if: needs.setup.outputs.publish == 'true'
uses: docker/login-action@v4.8.0
uses: docker/login-action@v4.6.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Login to GHCR
if: needs.setup.outputs.publish == 'true'
uses: docker/login-action@v4.8.0
uses: docker/login-action@v4.6.0
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
@@ -324,7 +304,7 @@ jobs:
fi
- name: Login to GHCR
if: needs.setup.outputs.publish == 'true'
uses: docker/login-action@v4.8.0
uses: docker/login-action@v4.6.0
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
@@ -364,17 +344,8 @@ jobs:
echo "Placed pre-built Rust worker for ${arch}"
fi
done
mkdir -p docker/weed-vector-prebuilt
for arch in amd64 arm64; do
src="./rust-bins/weed-vector-${arch}"
if [ -f "$src" ]; then
cp "$src" "docker/weed-vector-prebuilt/weed-vector-${arch}"
echo "Placed pre-built vector engine for ${arch}"
fi
done
ls -la docker/weed-volume-prebuilt/
ls -la docker/weed-worker-prebuilt/
ls -la docker/weed-vector-prebuilt/
- name: Create BuildKit config for local scan build
if: needs.setup.outputs.publish != 'true'
run: |
@@ -434,7 +405,7 @@ jobs:
output: trivy-results.sarif
exit-code: '0'
- name: Upload Trivy scan results to GitHub Security
uses: github/codeql-action/upload-sarif@v4.39.3
uses: github/codeql-action/upload-sarif@v4.38.2
if: always()
with:
sarif_file: trivy-results.sarif
@@ -501,12 +472,12 @@ jobs:
ghcr.io/chrislusf/seaweedfs
tags: type=raw,value=${{ github.event_name == 'workflow_dispatch' && github.event.inputs.image_tag || 'latest' }},suffix=${{ steps.config.outputs.tag_suffix }}
- name: Login to Docker Hub
uses: docker/login-action@v4.8.0
uses: docker/login-action@v4.6.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Login to GHCR
uses: docker/login-action@v4.8.0
uses: docker/login-action@v4.6.0
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
@@ -11,11 +11,6 @@ on:
required: true
default: ''
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || format('refs/tags/{0}', github.event.inputs.release_tag) || github.run_id }}
cancel-in-progress: false
queue: max
permissions:
contents: read
@@ -58,7 +53,7 @@ jobs:
-
name: Login to Docker Hub
if: github.event_name != 'pull_request'
uses: docker/login-action@v4.8.0
uses: docker/login-action@v4.6.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
+34 -89
View File
@@ -25,7 +25,7 @@ on:
rocksdb_version:
description: 'RocksDB git tag to use when variant=rocksdb'
required: false
default: 'v11.1.2'
default: 'v10.10.1'
permissions:
contents: read
@@ -43,11 +43,11 @@ concurrency:
jobs:
# ── Pre-build the Rust binaries natively ────────────────────────────
# The volume server, the Rust maintenance worker and the vector engine,
# cross-compiled for amd64 and arm64 without QEMU, turning a 5-hour emulated
# cargo build into ~15 minutes of native compilation.
# The volume server and the Rust maintenance worker, cross-compiled for
# amd64 and arm64 without QEMU, turning a 5-hour emulated cargo build into
# ~15 minutes of native compilation.
build-rust-binaries:
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
strategy:
matrix:
include:
@@ -55,17 +55,29 @@ jobs:
arch: amd64
- target: aarch64-unknown-linux-musl
arch: arm64
cross: true
steps:
- name: Checkout
uses: actions/checkout@v7
- name: Set up the musl target
uses: ./.github/actions/rust-musl-target
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
with:
target: ${{ matrix.target }}
targets: ${{ matrix.target }}
- name: Install musl tools (amd64)
if: ${{ !matrix.cross }}
run: sudo apt-get install -y musl-tools
- name: Install cross-compilation tools (arm64)
if: matrix.cross
run: |
sudo apt-get install -y gcc-aarch64-linux-gnu
echo "CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_LINKER=aarch64-linux-gnu-gcc" >> "$GITHUB_ENV"
# Disable glibc fortify source — its __memcpy_chk etc. symbols don't exist in musl
echo "CFLAGS_aarch64_unknown_linux_musl=-U_FORTIFY_SOURCE" >> "$GITHUB_ENV"
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
@@ -73,8 +85,7 @@ jobs:
~/.cargo/git
seaweed-volume/target
seaweed-worker/target/${{ matrix.target }}/release
seaweed-vector/target/${{ matrix.target }}/release
key: rust-docker-${{ matrix.target }}-${{ hashFiles('seaweed-volume/Cargo.lock', 'seaweed-worker/Cargo.lock', 'seaweed-vector/Cargo.lock') }}
key: rust-docker-${{ matrix.target }}-${{ hashFiles('seaweed-volume/Cargo.lock', 'seaweed-worker/Cargo.lock') }}
restore-keys: |
rust-docker-${{ matrix.target }}-
@@ -115,12 +126,6 @@ jobs:
cargo build --release -p weed-lance-worker --target ${{ matrix.target }}
cp target/${{ matrix.target }}/release/weed-worker ../weed-worker-${{ matrix.arch }}
- name: Build the vector engine
run: |
cd seaweed-vector
cargo build --release -p weed-vector-server --target ${{ matrix.target }}
cp target/${{ matrix.target }}/release/weed-vector ../weed-vector-${{ matrix.arch }}
- name: Upload artifacts
uses: actions/upload-artifact@v7
with:
@@ -129,7 +134,6 @@ jobs:
weed-volume-large-disk-${{ matrix.arch }}
weed-volume-normal-${{ matrix.arch }}
weed-worker-${{ matrix.arch }}
weed-vector-${{ matrix.arch }}
# One job per (variant, platform) on a native runner, pushed by digest;
# the merge job stitches the digests into one multi-arch tag.
@@ -175,10 +179,6 @@ jobs:
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Download pre-built Rust binaries
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
uses: actions/download-artifact@v8
@@ -206,17 +206,8 @@ jobs:
echo "Placed pre-built Rust worker for ${arch}"
fi
done
mkdir -p docker/weed-vector-prebuilt
for arch in amd64 arm64; do
src="./rust-bins/weed-vector-${arch}"
if [ -f "$src" ]; then
cp "$src" "docker/weed-vector-prebuilt/weed-vector-${arch}"
echo "Placed pre-built vector engine for ${arch}"
fi
done
ls -la docker/weed-volume-prebuilt/
ls -la docker/weed-worker-prebuilt/
ls -la docker/weed-vector-prebuilt/
- name: Free Disk Space
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
@@ -227,7 +218,7 @@ jobs:
sudo docker system prune -af --volumes
[ -d ~/.cache/go-build ] && rm -rf ~/.cache/go-build || true
[ -d /go/pkg ] && rm -rf /go/pkg || true
timeout 15 df -h || true
df -h
- name: Docker meta
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
@@ -262,7 +253,7 @@ jobs:
- name: Login to GHCR
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
uses: docker/login-action@v4.8.0
uses: docker/login-action@v4.6.0
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
@@ -279,11 +270,8 @@ jobs:
file: ${{ matrix.dockerfile }}
platforms: ${{ matrix.platform }}
labels: ${{ steps.docker_meta.outputs.labels }}
# mode=max keeps resolvedDependencies complete and records the
# builder identity; attestations ride inside each pushed index and
# survive the imagetools merge into the multi-arch manifest.
provenance: mode=max
sbom: true
# Flat single-platform manifest so imagetools create assembles cleanly.
provenance: false
outputs: type=image,name=${{ env.IMAGE }},push-by-digest=true,name-canonical=true,push=true
cache-from: type=gha,scope=${{ matrix.variant }}-${{ matrix.arch }}
# max only for rocksdb: its RocksDB compile is sha-independent and worth
@@ -293,7 +281,7 @@ jobs:
${{ matrix.build_args }}
BUILDKIT_INLINE_CACHE=1
BRANCH=${{ github.sha }}
${{ matrix.variant == 'rocksdb' && format('ROCKSDB_VERSION={0}', github.event.inputs.rocksdb_version || 'v11.1.2') || '' }}
${{ matrix.variant == 'rocksdb' && format('ROCKSDB_VERSION={0}', github.event.inputs.rocksdb_version || 'v10.10.1') || '' }}
- name: Export digest
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
@@ -319,8 +307,6 @@ jobs:
permissions:
contents: read
id-token: write
attestations: write
packages: write
strategy:
fail-fast: false
matrix:
@@ -352,7 +338,7 @@ jobs:
- name: Login to GHCR
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
uses: docker/login-action@v4.8.0
uses: docker/login-action@v4.6.0
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
@@ -374,7 +360,7 @@ jobs:
- name: Login to Docker Hub
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
uses: docker/login-action@v4.8.0
uses: docker/login-action@v4.6.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
@@ -428,45 +414,6 @@ jobs:
${{ env.IMAGE }}@${{ steps.manifest.outputs.digest }}
chrislusf/seaweedfs@${{ steps.manifest.outputs.digest }}
# Attest the merged multi-arch digest itself, so digest-pinning consumers
# can verify provenance against the exact index they pull. Both steps
# reuse the GHCR login above and push as OCI referrers.
- name: Attest build provenance for ${{ matrix.variant }}
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
uses: actions/attest-build-provenance@v4
with:
subject-name: ${{ env.IMAGE }}
subject-digest: ${{ steps.manifest.outputs.digest }}
push-to-registry: true
# SBOMs are generated and attested per platform manifest: an index-wide
# scan only describes syft's default platform, while the manifests it
# merges differ (some ship the real binary, others placeholders), so the
# SBOM has to be bound to the platform digest a consumer actually pulls.
- name: Install syft
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
uses: anchore/sbom-action/download-syft@v0
- name: Attest per-platform SBOMs for ${{ matrix.variant }}
if: github.event_name != 'workflow_dispatch' || github.event.inputs.variant == 'all' || github.event.inputs.variant == matrix.variant
working-directory: /tmp/digests
run: |
set -euo pipefail
for f in *; do
# Each saved digest is an OCI index (image + attestation manifests);
# scan and attest the image manifest a same-platform pull resolves.
ref="${{ env.IMAGE }}@sha256:${f}"
image=$(crane manifest "$ref" | jq -r '
[.manifests[]? | select((.annotations["vnd.docker.reference.type"] // "") != "attestation-manifest"
and (.platform.os // "unknown") != "unknown") | .digest][0] // empty')
ref="${{ env.IMAGE }}@${image:-sha256:${f}}"
syft "registry:${ref}" -o "spdx-json=/tmp/sbom-${f}.spdx.json"
cosign attest --yes \
--predicate "/tmp/sbom-${f}.spdx.json" \
--type spdxjson \
"$ref"
done
# Report-only trivy scan: uploads fixable HIGH/CRITICAL findings to GitHub
# Security for visibility, but never blocks the release. Releases (including
# `latest`) ship regardless — vulnerabilities are tracked, not gated, since
@@ -487,7 +434,7 @@ jobs:
variant: large_disk
steps:
- name: Login to GHCR
uses: docker/login-action@v4.8.0
uses: docker/login-action@v4.6.0
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
@@ -505,9 +452,7 @@ jobs:
scanners: vuln
vuln-type: os,library
severity: HIGH,CRITICAL
# Keep unfixed findings in the report so digest-pinning consumers see
# the complete picture, not just what a patch could resolve.
ignore-unfixed: false
ignore-unfixed: true
limit-severities-for-sarif: true
format: sarif
output: trivy-results.sarif
@@ -515,7 +460,7 @@ jobs:
- name: Upload Trivy scan results to GitHub Security
if: always()
uses: github/codeql-action/upload-sarif@v4.39.3
uses: github/codeql-action/upload-sarif@v4.38.2
with:
sarif_file: trivy-results.sarif
category: trivy-${{ matrix.variant }}
@@ -539,13 +484,13 @@ jobs:
latest_tag: latest_large_disk
steps:
- name: Login to Docker Hub
uses: docker/login-action@v4.8.0
uses: docker/login-action@v4.6.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Login to GHCR
uses: docker/login-action@v4.8.0
uses: docker/login-action@v4.6.0
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
@@ -4,9 +4,9 @@ on:
workflow_dispatch:
inputs:
rocksdb_version:
description: 'RocksDB git tag or branch to build (e.g. v11.1.2)'
description: 'RocksDB git tag or branch to build (e.g. v10.10.1)'
required: true
default: 'v11.1.2'
default: 'v10.10.1'
seaweedfs_ref:
description: 'SeaweedFS git tag, branch, or commit to build'
required: true
@@ -16,11 +16,6 @@ on:
required: false
default: ''
concurrency:
group: ${{ github.workflow }}-${{ github.event.inputs.image_tag || format('rocksdb_{0}_seaweedfs_{1}', github.event.inputs.rocksdb_version, github.event.inputs.seaweedfs_ref) }}
cancel-in-progress: false
queue: max
permissions:
contents: read
@@ -90,13 +85,13 @@ jobs:
echo "seaweedfs_ref=$seaweed" >> "$GITHUB_OUTPUT"
- name: Set up QEMU
uses: docker/setup-qemu-action@99012661954931238ded8c8b007157a8430204e1 # v4.4.0
uses: docker/setup-qemu-action@99012661954931238ded8c8b007157a8430204e1 # v1
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0
uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v1
- name: Login to Docker Hub
uses: docker/login-action@8aa5831fb8d48d8f76594c7ddde61ae58019f550 # v4.8.0
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v1
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
-1
View File
@@ -27,7 +27,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'telemetry/server/go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Build Telemetry Server
if: github.event_name == 'workflow_dispatch' && inputs.deploy
-4
View File
@@ -1,10 +1,6 @@
name: 'Dependency Review'
on: [pull_request]
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
+3 -8
View File
@@ -19,8 +19,8 @@ on:
- '.github/workflows/e2e.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
group: ${{ github.head_ref }}/e2e
cancel-in-progress: true
permissions:
contents: read
@@ -32,7 +32,7 @@ defaults:
jobs:
e2e:
name: FUSE Mount
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 30
steps:
- name: Check out code into the Go module directory
@@ -42,7 +42,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Configure Docker Hub mirror
run: |
@@ -53,7 +52,6 @@ jobs:
uses: docker/setup-buildx-action@v4
- name: Cache Docker layers
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: /tmp/.buildx-cache
@@ -61,9 +59,6 @@ jobs:
restore-keys: |
${{ runner.os }}-buildx-e2e-
- name: Test APT mirror fallback
run: python3 test_apt_install.py
- name: Install dependencies
run: |
sudo rm -f /etc/apt/sources.list.d/azure-cli.list /etc/apt/sources.list.d/microsoft-prod.list
+1 -6
View File
@@ -18,23 +18,18 @@ on:
- 'go.sum'
- '.github/workflows/ec-integration-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
jobs:
ec-integration-tests:
name: EC Integration Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 30
steps:
- name: Set up Go 1.x
uses: actions/setup-go@v7
with:
cache: ${{ runner.environment == 'github-hosted' }}
go-version: ^1.26
id: go
+3 -4
View File
@@ -19,8 +19,8 @@ on:
- '.github/actions/fix-fusermount-setuid/**'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
group: ${{ github.head_ref || github.ref }}/fuse-dlm-integration
cancel-in-progress: true
permissions:
contents: read
@@ -28,7 +28,7 @@ permissions:
jobs:
fuse-dlm-integration:
name: FUSE DLM Integration Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 30
steps:
@@ -39,7 +39,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Configure FUSE
run: |
+3 -7
View File
@@ -24,8 +24,8 @@ on:
- '.github/actions/fix-fusermount-setuid/**'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
group: ${{ github.head_ref || github.ref }}/fuse-failover
cancel-in-progress: true
permissions:
contents: read
@@ -33,7 +33,7 @@ permissions:
jobs:
fuse-failover:
name: FUSE Volume Server Failover
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 40
steps:
@@ -42,14 +42,10 @@ jobs:
with:
persist-credentials: false
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Configure FUSE
run: |
+3 -4
View File
@@ -17,8 +17,8 @@ on:
- '.github/actions/fix-fusermount-setuid/**'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
group: ${{ github.head_ref }}/fuse-integration
cancel-in-progress: true
permissions:
contents: read
@@ -26,7 +26,7 @@ permissions:
jobs:
fuse-integration:
name: FUSE Integration Testing
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 50
steps:
@@ -37,7 +37,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Configure FUSE
run: |
+3 -7
View File
@@ -25,8 +25,8 @@ on:
- '.github/actions/fix-fusermount-setuid/**'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
group: ${{ github.head_ref || github.ref }}/fuse-p2p-integration
cancel-in-progress: true
permissions:
contents: read
@@ -34,21 +34,17 @@ permissions:
jobs:
fuse-p2p-integration:
name: FUSE P2P Peer Chunk Sharing
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 20
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Configure FUSE
run: |
+2 -9
View File
@@ -17,8 +17,8 @@ on:
- '.github/workflows/go.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
group: ${{ github.head_ref }}/go
cancel-in-progress: true
permissions:
contents: read
@@ -35,7 +35,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Get dependencies
run: |
cd weed; go get -v -t -d ./...
@@ -58,7 +57,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Go Vet linux/386 (type-checks code and tests for 32-bit int overflows)
run: |
GOOS=linux GOARCH=386 go vet ./... 2>&1 | grep -v "MessageState contains sync.Mutex" | grep -v "IdentityAccessManagement contains sync.RWMutex" | tee vet-32bit-output.txt
@@ -74,7 +72,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Build
run: cd weed; go build -tags "elastic gocdk sqlite ydb tarantool tikv rclone" -v .
@@ -97,7 +94,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
# Nothing else on a PR compiles these, so per-OS syscall constants creep
# back into shared files unnoticed and only a release build catches it.
- name: Cross-compile
@@ -132,7 +128,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Test
env:
RUN_REDIS_TESTS: "1"
@@ -148,7 +143,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
# 386 test binaries run natively on the amd64 runner. This catches what vet
# can't: unaligned 64-bit atomics and arithmetic that wraps at runtime.
# -short skips the e2e suites already covered on amd64.
@@ -165,7 +159,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
# weed/kms/azure is excluded from every default build, so without this
# nothing compiles it and tag-gated code silently rots.
- name: Build and test with azurekms
+3 -442
View File
@@ -8,10 +8,6 @@ on:
branches: [ master ]
paths: ['k8s/**', '.github/workflows/helm_ci.yml', '.github/scripts/helm_jwt_keys.py']
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -37,20 +33,13 @@ jobs:
with:
version: v3.18.4
- uses: actions/setup-go@v7
with:
go-version-file: go.mod
- name: Test vector chart resources
run: go test ./k8s/charts/seaweedfs/tests -count=1
- uses: actions/setup-python@v7
with:
python-version: '3.10'
check-latest: true
- name: Set up chart-testing
uses: helm/chart-testing-action@v2.9.0
uses: helm/chart-testing-action@v2.8.0
- name: Run chart-testing (list-changed)
id: list-changed
@@ -104,51 +93,8 @@ jobs:
grep -A 4 "^ strategy:" /tmp/s3-rollout.yaml | grep -q "maxUnavailable: 0"
grep -A 4 "^ strategy:" /tmp/s3-rollout.yaml | grep -q "type: RollingUpdate"
grep -A 5 "^ lifecycle:" /tmp/s3-rollout.yaml | grep -q -- "- sleep"
test "$(grep -c -- '-preStopSeconds' /tmp/s3-rollout-defaults.yaml)" -eq 0
echo "S3 rollout settings render correctly"
echo "=== Testing S3 preStopSeconds ==="
if helm template test $CHART_DIR --show-only templates/s3/s3-deployment.yaml \
--set s3.enabled=true --set s3.preStopSeconds=20 > /tmp/s3-prestop-short.yaml 2>&1; then
echo "FAIL: s3.preStopSeconds=20 rendered with the default 10s grace period"; exit 1
fi
grep -q "must be at least 35" /tmp/s3-prestop-short.yaml
helm template test $CHART_DIR --show-only templates/s3/s3-deployment.yaml \
--set s3.enabled=true --set s3.preStopSeconds=20 \
--set s3.terminationGracePeriodSeconds=35 > /tmp/s3-prestop.yaml
grep -q -- "-preStopSeconds=20 " /tmp/s3-prestop.yaml
grep -q "terminationGracePeriodSeconds: 35$" /tmp/s3-prestop.yaml
helm template test $CHART_DIR --show-only templates/filer/filer-statefulset.yaml \
--set filer.s3.enabled=true > /tmp/filer-s3-prestop-defaults.yaml
test "$(grep -c -- '-s3.preStopSeconds' /tmp/filer-s3-prestop-defaults.yaml)" -eq 0
grep -q "terminationGracePeriodSeconds: 60$" /tmp/filer-s3-prestop-defaults.yaml
helm template test $CHART_DIR --show-only templates/filer/filer-statefulset.yaml \
--set filer.s3.enabled=true --set filer.s3.preStopSeconds=20 > /tmp/filer-s3-prestop.yaml
grep -q -- "-s3.preStopSeconds=20 " /tmp/filer-s3-prestop.yaml
grep -q "terminationGracePeriodSeconds: 80$" /tmp/filer-s3-prestop.yaml
helm template test $CHART_DIR --show-only templates/all-in-one/all-in-one-deployment.yaml \
--set allInOne.enabled=true --set allInOne.s3.enabled=true > /tmp/aio-s3-prestop-defaults.yaml
test "$(grep -c -- '-s3.preStopSeconds' /tmp/aio-s3-prestop-defaults.yaml)" -eq 0
grep -q "terminationGracePeriodSeconds: 60$" /tmp/aio-s3-prestop-defaults.yaml
helm template test $CHART_DIR --show-only templates/all-in-one/all-in-one-deployment.yaml \
--set allInOne.enabled=true --set allInOne.s3.enabled=true \
--set s3.preStopSeconds=20 > /tmp/aio-s3-prestop.yaml
grep -q -- "-s3.preStopSeconds=20 " /tmp/aio-s3-prestop.yaml
grep -q "terminationGracePeriodSeconds: 80$" /tmp/aio-s3-prestop.yaml
helm template test $CHART_DIR --show-only templates/all-in-one/all-in-one-deployment.yaml \
--set allInOne.enabled=true --set allInOne.s3.enabled=true \
--set s3.preStopSeconds=20 --set allInOne.s3.preStopSeconds=0 > /tmp/aio-s3-prestop-off.yaml
test "$(grep -c -- '-s3.preStopSeconds' /tmp/aio-s3-prestop-off.yaml)" -eq 0
grep -q "terminationGracePeriodSeconds: 60$" /tmp/aio-s3-prestop-off.yaml
helm template test $CHART_DIR --show-only templates/all-in-one/all-in-one-deployment.yaml \
--set allInOne.enabled=true --set allInOne.s3.enabled=true \
--set s3.preStopSeconds=20 --set allInOne.s3.preStopSeconds=25 > /tmp/aio-s3-prestop-own.yaml
grep -q -- "-s3.preStopSeconds=25 " /tmp/aio-s3-prestop-own.yaml
grep -q "terminationGracePeriodSeconds: 85$" /tmp/aio-s3-prestop-own.yaml
echo "S3 preStopSeconds renders correctly"
echo "=== Testing the S3 config checksum is stable across renders ==="
# s3-secret.yaml keeps its generated keys with lookup. Hashing the rendered
# Secret drew new keys for the checksum whenever lookup found nothing (on
@@ -259,40 +205,6 @@ jobs:
grep -q "security-config" /tmp/security.yaml
echo "Security configuration renders correctly"
echo "=== Testing volume server HTTPS ==="
awk '/^ *\[https.client\]$/{f=1;next} /^ *\[/{f=0} f' /tmp/security.yaml | grep -q 'enabled = false'
helm template test $CHART_DIR \
--set global.seaweedfs.enableSecurity=true \
--set global.seaweedfs.volumeHttps.enabled=true \
--set volumes.hot.replicas=1 \
> /tmp/volume-https.yaml
awk '/^ *\[https.client\]$/{f=1;next} /^ *\[/{f=0} f' /tmp/volume-https.yaml > /tmp/volume-https-client.txt
grep -q 'enabled = true' /tmp/volume-https-client.txt
grep -q 'cert = "/usr/local/share/ca-certificates/client/tls.crt"' /tmp/volume-https-client.txt
grep -q 'ca = "/usr/local/share/ca-certificates/ca/tls.crt"' /tmp/volume-https-client.txt
awk '/^ *\[https.volume\]$/{f=1;next} /^ *\[/{f=0} f' /tmp/volume-https.yaml \
| grep -q 'cert = "/usr/local/share/ca-certificates/volume/tls.crt"'
grep -q "'\*.test-seaweedfs-volume.default'" /tmp/volume-https.yaml
grep -q "'\*.test-seaweedfs-volume-hot.default'" /tmp/volume-https.yaml
test "$(grep -c 'scheme: HTTPS' /tmp/volume-https.yaml)" -eq 4
test "$(grep -c 'checksum/volume-https:' /tmp/volume-https.yaml)" -eq 4
if grep -q 'checksum/volume-https' /tmp/security.yaml; then
echo "FAIL: the volume HTTPS rollout annotation should only render while the option is on"
exit 1
fi
if helm template test $CHART_DIR --set global.seaweedfs.volumeHttps.enabled=true > /dev/null 2>/tmp/volume-https-nosec.err; then
echo "FAIL: volumeHttps without enableSecurity should fail to render"
exit 1
fi
grep -q "requires global.seaweedfs.enableSecurity" /tmp/volume-https-nosec.err
if helm template test $CHART_DIR --set global.seaweedfs.enableSecurity=true \
--set global.seaweedfs.volumeHttps.enabled=true --set allInOne.enabled=true > /dev/null 2>/tmp/volume-https-aio.err; then
echo "FAIL: volumeHttps with allInOne should fail to render"
exit 1
fi
grep -q "not supported with allInOne" /tmp/volume-https-aio.err
echo "Volume server HTTPS renders [https.volume], [https.client], certificate names, HTTPS probes and rollout annotations"
echo "=== Testing secure bucket-creation hook certificate mounts ==="
helm template test $CHART_DIR \
--show-only templates/shared/post-install-bucket-hook.yaml \
@@ -644,7 +556,7 @@ jobs:
print("cluster alias: hook waits on the renamed names and the chart's addresses")
# Keys kept in a Secret are referenced, not inlined, so the hook
# must render valueFrom for map-valued extraEnvironmentVars.
# must render valueFrom for non-string extraEnvironmentVars.
out = render(dict(jwt, **{
"global.seaweedfs.extraEnvironmentVars.WEED_JWT_FILER_SIGNING_KEY.secretKeyRef.name": "signing-keys",
"global.seaweedfs.extraEnvironmentVars.WEED_JWT_FILER_SIGNING_KEY.secretKeyRef.key": "filerWrite",
@@ -683,22 +595,6 @@ jobs:
--set master.data.storageClass=standard > /tmp/pvc.yaml
grep -q "PersistentVolumeClaim" /tmp/pvc.yaml
echo "PVC configuration renders correctly"
echo "=== Testing existingClaim volumes are mounted ==="
# The filer's data and logs and the master's logs declared an existingClaim
# volume but mounted only hostPath, persistentVolumeClaim and emptyDir, so
# the filer kept its metadata store on the container's own filesystem.
helm template test $CHART_DIR --show-only templates/filer/filer-statefulset.yaml \
--set filer.data.type=existingClaim --set filer.data.claimName=filer-data \
--set filer.logs.type=existingClaim --set filer.logs.claimName=filer-logs > /tmp/filer-existing-claim.yaml
grep -A 1 -- '- name: data-filer$' /tmp/filer-existing-claim.yaml | grep -q 'mountPath: /data$'
grep -A 1 -- '- name: seaweedfs-filer-log-volume$' /tmp/filer-existing-claim.yaml | grep -q 'mountPath: "/logs/"$'
grep -q -- '-logdir=/logs' /tmp/filer-existing-claim.yaml
helm template test $CHART_DIR --show-only templates/master/master-statefulset.yaml \
--set master.logs.type=existingClaim --set master.logs.claimName=master-logs > /tmp/master-existing-claim.yaml
grep -A 1 -- '- name: seaweedfs-master-log-volume$' /tmp/master-existing-claim.yaml | grep -q 'mountPath: "/logs/"$'
grep -q -- '-logdir=/logs' /tmp/master-existing-claim.yaml
echo "existingClaim volumes are mounted"
echo "=== Testing with custom replicas ==="
helm template test $CHART_DIR \
@@ -736,47 +632,12 @@ jobs:
--set filer.secretExtraEnvironmentVars.WEED_MYSQL_PASSWORD.secretKeyRef.name=db \
--set filer.secretExtraEnvironmentVars.WEED_MYSQL_PASSWORD.secretKeyRef.key=password > /tmp/filer-mysql-on-secret.yaml
grep -A 4 -- "- name: WEED_MYSQL_PASSWORD$" /tmp/filer-mysql-on-secret.yaml | grep -q "name: db"
# It replaces the db secret's entry rather than adding a second one, which
# Helm 4's server-side apply rejects as a duplicate.
test "$(grep -c -- "- name: WEED_MYSQL_PASSWORD$" /tmp/filer-mysql-on-secret.yaml)" -eq 1
test "$(grep -c -- "- name: WEED_MYSQL_USERNAME$" /tmp/filer-mysql-on-secret.yaml)" -eq 1
helm template test $CHART_DIR \
--set-string filer.extraEnvironmentVars.WEED_MYSQL_ENABLED=true \
--set-string filer.extraEnvironmentVars.WEED_MYSQL_USERNAME=app > /tmp/filer-mysql-on-plain.yaml
test "$(grep -c -- "- name: WEED_MYSQL_USERNAME$" /tmp/filer-mysql-on-plain.yaml)" -eq 1
grep -A 1 -- "- name: WEED_MYSQL_USERNAME$" /tmp/filer-mysql-on-plain.yaml | grep -q 'value: "app"'
# Credentials render ahead of the merged vars so a $(...) reference resolves.
helm template test $CHART_DIR \
--set-string filer.extraEnvironmentVars.WEED_MYSQL_ENABLED=true \
--set-string filer.extraEnvironmentVars.WEED_MYSQL_USERNAME=app \
--set-string 'filer.extraEnvironmentVars.WEED_MYSQL_DSN=$(WEED_MYSQL_USERNAME)@tcp(h)/db' > /tmp/filer-mysql-dsn.yaml
test "$(grep -n -- "- name: WEED_MYSQL_USERNAME$" /tmp/filer-mysql-dsn.yaml | cut -d: -f1)" -lt "$(grep -n -- "- name: WEED_MYSQL_DSN$" /tmp/filer-mysql-dsn.yaml | cut -d: -f1)"
# Vars a credential's own $(...) references point at render before it.
helm template test $CHART_DIR \
--set-string filer.extraEnvironmentVars.WEED_MYSQL_ENABLED=true \
--set-string filer.extraEnvironmentVars.DB_USER=app \
--set-string 'filer.extraEnvironmentVars.WEED_MYSQL_USERNAME=$(DB_USER)' > /tmp/filer-mysql-ref.yaml
test "$(grep -n -- "- name: DB_USER$" /tmp/filer-mysql-ref.yaml | cut -d: -f1)" -lt "$(grep -n -- "- name: WEED_MYSQL_USERNAME$" /tmp/filer-mysql-ref.yaml | cut -d: -f1)"
helm template test $CHART_DIR \
--set-string filer.extraEnvironmentVars.WEED_MYSQL_ENABLED=true \
--set filer.secretExtraEnvironmentVars.DB_USER.secretKeyRef.name=db \
--set filer.secretExtraEnvironmentVars.DB_USER.secretKeyRef.key=user \
--set-string 'filer.extraEnvironmentVars.WEED_MYSQL_USERNAME=$(DB_USER)' > /tmp/filer-mysql-ref-secret.yaml
test "$(grep -c -- "- name: DB_USER$" /tmp/filer-mysql-ref-secret.yaml)" -eq 1
test "$(grep -n -- "- name: DB_USER$" /tmp/filer-mysql-ref-secret.yaml | cut -d: -f1)" -lt "$(grep -n -- "- name: WEED_MYSQL_USERNAME$" /tmp/filer-mysql-ref-secret.yaml | cut -d: -f1)"
# A flag the chart cannot read counts as selected, not as off.
helm template test $CHART_DIR \
--set filer.secretExtraEnvironmentVars.WEED_MYSQL_ENABLED.secretKeyRef.name=store \
--set filer.secretExtraEnvironmentVars.WEED_MYSQL_ENABLED.secretKeyRef.key=enabled > /tmp/filer-mysql-secret.yaml
grep -q "name: test-seaweedfs-db-secret" /tmp/filer-mysql-secret.yaml
grep -q "name: WEED_MYSQL_HOSTNAME" /tmp/filer-mysql-secret.yaml
# A secret-backed key also replaces the matching plain entry, not adds one.
test "$(grep -c -- "- name: WEED_MYSQL_ENABLED$" /tmp/filer-mysql-secret.yaml)" -eq 1
helm template test $CHART_DIR \
--set-string filer.extraEnvironmentVars.WEED_MYSQL_ENABLED=true \
--set filer.secretExtraEnvironmentVars.WEED_MYSQL_HOSTNAME.secretKeyRef.name=db \
--set filer.secretExtraEnvironmentVars.WEED_MYSQL_HOSTNAME.secretKeyRef.key=host > /tmp/filer-mysql-hostname-secret.yaml
test "$(grep -c -- "- name: WEED_MYSQL_HOSTNAME$" /tmp/filer-mysql-hostname-secret.yaml)" -eq 1
helm template test $CHART_DIR \
--set-string filer.extraEnvironmentVars.WEED_MYSQL2_HOSTNAME=other > /tmp/filer-mysql2.yaml
grep -q "name: WEED_MYSQL2_HOSTNAME" /tmp/filer-mysql2.yaml
@@ -1359,43 +1220,6 @@ jobs:
print("all-in-one env: shared key renders once, component value wins")
PYEOF
echo "=== Testing extraEnvironmentVars: numbers and booleans render as values ==="
# Only a string rendered as value; anything else went to valueFrom, so an
# unquoted 3306 or true in a values file (or --set) became "valueFrom: 3306"
# and the API server rejected the workload. A map still renders valueFrom,
# and a null still renders an empty variable. --set-json decodes numbers as
# float64, as values files do, which quote would print as 1e+06.
helm template test $CHART_DIR \
--set-json global.seaweedfs.extraEnvironmentVars.WEED_GLOBAL_INT=1000000 \
--set master.extraEnvironmentVars.WEED_MASTER_VOLUME_GROWTH_COPY_1=7 \
--set master.extraEnvironmentVars.WEED_EMPTY=null \
--set filer.extraEnvironmentVars.WEED_MYSQL_ENABLED=true \
--set filer.extraEnvironmentVars.WEED_MYSQL_PORT=3306 \
--set filer.extraEnvironmentVars.WEED_REF.secretKeyRef.name=s \
--set filer.extraEnvironmentVars.WEED_REF.secretKeyRef.key=k > /tmp/env-scalars.yaml
python3 - /tmp/env-scalars.yaml <<'PYEOF'
import sys, yaml
docs = [d for d in yaml.safe_load_all(open(sys.argv[1])) if d]
def env(suffix):
sts = next(d for d in docs if d.get("kind") == "StatefulSet"
and d["metadata"]["name"].endswith(suffix))
return {e["name"]: e for e in sts["spec"]["template"]["spec"]["containers"][0]["env"]}
master, filer = env("-master"), env("-filer")
want = [(master, "WEED_MASTER_VOLUME_GROWTH_COPY_1", "7"), (master, "WEED_GLOBAL_INT", "1000000"),
(filer, "WEED_MYSQL_ENABLED", "true"), (filer, "WEED_MYSQL_PORT", "3306"),
(filer, "WEED_GLOBAL_INT", "1000000")]
failed = [f"{name}: {e.get(name)}" for e, name, value in want
if e.get(name) != {"name": name, "value": value}]
if master.get("WEED_EMPTY") != {"name": "WEED_EMPTY"}:
failed.append(f"WEED_EMPTY: {master.get('WEED_EMPTY')}")
if filer.get("WEED_REF", {}).get("valueFrom") != {"secretKeyRef": {"name": "s", "key": "k"}}:
failed.append(f"WEED_REF: {filer.get('WEED_REF')}")
if failed:
print("FAIL: extraEnvironmentVars rendered wrong:", *failed, sep="\n ", file=sys.stderr)
sys.exit(1)
print("extraEnvironmentVars: numbers and booleans render as values, a map as valueFrom, a null as empty")
PYEOF
echo "=== Testing bucket versioning: YAML bool false suspends like string \"false\" ==="
# bool false used to be a silent no-op while string "false" suspended.
BOOL_FALSE=$(helm template test $CHART_DIR \
@@ -1405,67 +1229,6 @@ jobs:
echo "$BOOL_FALSE" | grep -q -- '-status Suspended' || { echo "FAIL: bool false versioning did not Suspend the bucket"; exit 1; }
echo "Bucket versioning: YAML bool false suspends consistently with string \"false\""
echo "=== Testing bucket ttl, replication and disk in one fs.configure ==="
FS_CONFIGURE=$(helm template test $CHART_DIR \
--set s3.enabled=true \
--set s3.createBuckets[0].name=cfgbucket \
--set s3.createBuckets[0].ttl=7d \
--set-string s3.createBuckets[0].replication=001 \
--set s3.createBuckets[0].disk=ssd | grep 'fs.configure -locationPrefix=/buckets/cfgbucket/' || true)
echo "$FS_CONFIGURE" | grep -q -- '-ttl=7d -replication=001 -disk=ssd -apply' || { echo "FAIL: bucket fs.configure missing ttl/replication/disk: $FS_CONFIGURE"; exit 1; }
echo "Bucket fs.configure: ttl, replication and disk render together"
FS_CONFIGURE=$(helm template test $CHART_DIR \
--set s3.enabled=true \
--set s3.createBuckets[0].name=replbucket \
--set-string s3.createBuckets[0].replication=002 | grep 'fs.configure -locationPrefix=/buckets/replbucket/' || true)
echo "$FS_CONFIGURE" | grep -q -- '/buckets/replbucket/ -replication=002 -apply' || { echo "FAIL: replication-only bucket did not render fs.configure: $FS_CONFIGURE"; exit 1; }
echo "Bucket fs.configure: replication without ttl"
FS_CONFIGURE=$(helm template test $CHART_DIR \
--set s3.enabled=true \
--set s3.createBuckets[0].name=diskbucket \
--set s3.createBuckets[0].disk=ssd | grep 'fs.configure -locationPrefix=/buckets/diskbucket/' || true)
echo "$FS_CONFIGURE" | grep -q -- '/buckets/diskbucket/ -disk=ssd -apply' || { echo "FAIL: disk-only bucket did not render fs.configure: $FS_CONFIGURE"; exit 1; }
echo "Bucket fs.configure: disk without ttl"
if helm template test $CHART_DIR \
--set s3.enabled=true \
--set s3.createBuckets[0].name=numbucket \
--set s3.createBuckets[0].replication=0 >/dev/null 2>&1; then
echo "FAIL: a numeric replication should fail to render"; exit 1
fi
echo "Bucket fs.configure: numeric replication is rejected"
if helm template test $CHART_DIR \
--set s3.enabled=true \
--set s3.createBuckets[0].name=badbucket \
--set-string s3.createBuckets[0].replication=00O >/dev/null 2>&1; then
echo "FAIL: a non-digit replication should fail to render"; exit 1
fi
echo "Bucket fs.configure: non-digit replication is rejected"
if helm template test $CHART_DIR \
--set s3.enabled=true \
--set s3.createBuckets[0].name=bigbucket \
--set-string s3.createBuckets[0].replication=256 >/dev/null 2>&1; then
echo "FAIL: a replication above 255 should fail to render"; exit 1
fi
FS_CONFIGURE=$(helm template test $CHART_DIR \
--set s3.enabled=true \
--set s3.createBuckets[0].name=maxbucket \
--set-string s3.createBuckets[0].replication=255 | grep 'fs.configure -locationPrefix=/buckets/maxbucket/' || true)
echo "$FS_CONFIGURE" | grep -q -- '/buckets/maxbucket/ -replication=255 -apply' || { echo "FAIL: replication 255 should render: $FS_CONFIGURE"; exit 1; }
echo "Bucket fs.configure: replication is limited to 000-255"
FS_CONFIGURE=$(helm template test $CHART_DIR \
--set s3.enabled=true \
--set s3.createBuckets[0].name=nullbucket \
--set s3.createBuckets[0].ttl=7d \
--set s3.createBuckets[0].replication=null | grep 'fs.configure -locationPrefix=/buckets/nullbucket/' || true)
echo "$FS_CONFIGURE" | grep -q -- '/buckets/nullbucket/ -ttl=7d -apply' || { echo "FAIL: null replication should be treated as unset: $FS_CONFIGURE"; exit 1; }
echo "Bucket fs.configure: null replication is unset"
echo ""
echo "=== Testing hook Job labels ==="
# The hook Jobs were the only pods in the chart without the standard
@@ -2079,176 +1842,6 @@ jobs:
PYEOF
echo "License + master persistence tests passed"
echo ""
echo "=== Testing commonLabels ==="
python3 - "$CHART_DIR" <<'PYEOF'
import copy, subprocess, sys, yaml
chart = sys.argv[1]
class NoDuplicates(yaml.SafeLoader):
pass
def mapping(loader, node, deep=False):
keys = [loader.construct_object(k, deep=deep) for k, _ in node.value]
dups = {k for k in keys if keys.count(k) > 1}
if dups:
raise ValueError(f"duplicate keys {dups} at line {node.start_mark.line + 1}")
return yaml.SafeLoader.construct_mapping(loader, node, deep)
NoDuplicates.add_constructor(yaml.resolver.BaseResolver.DEFAULT_MAPPING_TAG, mapping)
BASE = [
"--set", "global.seaweedfs.enableSecurity=true",
"--set", "global.seaweedfs.monitoring.enabled=true",
"--set", "global.seaweedfs.monitoring.additionalLabels.release=prometheus",
"--set", "master.data.type=persistentVolumeClaim",
"--set", "master.ingress.enabled=true",
"--set", "master.ingress.labels.team=ingress",
"--set", "filer.data.type=persistentVolumeClaim",
"--set", "filer.ingresses.http.enabled=true",
"--set", "s3.enabled=true", "--set", "s3.enableAuth=true",
"--set", "s3.httpRoute.enabled=true",
"--set", "sftp.enabled=true", "--set", "worker.enabled=true",
"--set", "admin.enabled=true", "--set", "admin.secret.adminPassword=ci",
"--set", "cosi.enabled=true", "--set", "networkPolicy.enabled=true",
"--set", "volumes.dc2.replicas=1",
# render the security ConfigMap without its generated jwt key
"--set", "global.seaweedfs.enableSecurity=true",
"--set", "global.seaweedfs.securityConfig.jwtSigning.volumeWrite=false",
]
LABELS = "/tmp/common-labels-values.yaml"
with open(LABELS, "w") as f:
yaml.safe_dump({"commonLabels": {
"example.com/team": "storage",
"example.com/release": "{{ .Release.Name }}",
"team": "common", # also set by master.ingress.labels
"release": "common", # also set by monitoring.additionalLabels
"app.kubernetes.io/component": "hijack",
"app.kubernetes.io/instance": "hijack",
}}, f)
EXPECTED = {
"example.com/team": "storage",
"example.com/release": "test",
"team": "common",
"release": "common",
}
def render(extra):
out = subprocess.check_output(["helm", "template", "test", chart] + BASE + extra, text=True)
docs = {}
for d in yaml.load_all(out, Loader=NoDuplicates):
if not d:
continue
# generated credentials differ between renders
if d["kind"] == "Secret" and d["metadata"]["name"].endswith(
("-s3-secret", "-sftp-secret", "-sftp-ssh-secret")
):
d.pop("data", None)
d.pop("stringData", None)
docs[(d["kind"], d["metadata"]["name"])] = d
return docs
plain, labelled = render([]), render(["-f", LABELS])
failed = []
if set(plain) != set(labelled):
failed.append(f"commonLabels changed the set of objects: {set(plain) ^ set(labelled)}")
for key, d in labelled.items():
if key not in plain:
continue
before = plain[key]["metadata"].get("labels") or {}
labels = d["metadata"].get("labels") or {}
for k, v in EXPECTED.items():
if k in before:
continue
if labels.get(k) != v:
failed.append(f"{key}: {k}={labels.get(k)!r}, want {v!r}")
# labels the chart or an object-specific value sets are never overridden
for k, v in before.items():
if labels.get(k) != v:
failed.append(f"{key}: {k} changed from {v!r} to {labels.get(k)!r}")
# nothing else changes: selectors, pod templates, volumeClaimTemplates, checksums
stripped = copy.deepcopy(d)
rest = {k: v for k, v in labels.items() if k in before}
if rest or "labels" in plain[key]["metadata"]:
stripped["metadata"]["labels"] = rest
else:
stripped["metadata"].pop("labels", None)
if stripped != plain[key]:
failed.append(f"{key}: commonLabels changed more than metadata.labels")
if failed:
print("\nFAIL:", file=sys.stderr)
for f in failed:
print(f" - {f}", file=sys.stderr)
sys.exit(1)
print(f"{len(labelled)} objects carry commonLabels and are otherwise unchanged")
PYEOF
# an empty value renders exactly like the default
helm template test $CHART_DIR --set-string 'commonLabels=' > /tmp/common-labels-empty.yaml
cmp /tmp/default.yaml /tmp/common-labels-empty.yaml
# the whole value may be a template rendering to a map, e.g. a parent chart's labels
echo "commonLabels: 'example.com/owner: {{ .Release.Name }}'" > /tmp/common-labels-string-values.yaml
helm template test $CHART_DIR --show-only templates/master/master-service.yaml \
-f /tmp/common-labels-string-values.yaml > /tmp/common-labels-string.yaml
grep -q "^ example.com/owner: test$" /tmp/common-labels-string.yaml
if helm template test $CHART_DIR --set-json 'commonLabels=["a"]' > /dev/null 2>&1; then
echo "a commonLabels that is not a map must fail the render"; exit 1
fi
echo "commonLabels tests passed"
echo ""
echo "=== Testing namespaceOverride ==="
NS_ARGS=(
--set s3.enabled=true --set s3.enableAuth=true
--set 's3.createBuckets[0].name=data'
--set sftp.enabled=true --set sftp.enableAuth=true
--set admin.enabled=true --set admin.secret.adminPassword=ci
--set worker.enabled=true
--set cosi.enabled=true --set cosi.bucketClassName=seaweedfs
--set global.seaweedfs.enableSecurity=true
--set global.seaweedfs.monitoring.enabled=true
--set networkPolicy.enabled=true
--set volumes.extra.replicas=1
)
helm template test $CHART_DIR --namespace release-ns "${NS_ARGS[@]}" \
--set namespaceOverride=other-ns > /tmp/ns-override.yaml
helm template test $CHART_DIR --namespace release-ns \
--set allInOne.enabled=true --set allInOne.s3.enabled=true \
--set master.enabled=false --set volume.enabled=false --set filer.enabled=false \
--set 'allInOne.s3.createBuckets[0].name=data' \
--set global.seaweedfs.enableSecurity=true \
--set namespaceOverride=other-ns > /tmp/ns-override-aio.yaml
grep -q 'kind: Job' /tmp/ns-override-aio.yaml
python3 - /tmp/ns-override.yaml /tmp/ns-override-aio.yaml <<'PYEOF'
import sys, yaml
CLUSTER_SCOPED = {"ClusterRole", "ClusterRoleBinding", "BucketClass", "BucketAccessClass"}
failed = []
for path in sys.argv[1:]:
text = open(path).read()
if "release-ns" in text:
failed.append(f"{path}: release namespace still rendered")
for d in yaml.safe_load_all(text):
if not d:
continue
kind, md = d["kind"], d["metadata"]
if kind in CLUSTER_SCOPED:
for s in d.get("subjects") or []:
if s.get("namespace") != "other-ns":
failed.append(f"{path}: {kind}/{md['name']} subject in {s.get('namespace')!r}")
elif md.get("namespace") != "other-ns":
failed.append(f"{path}: {kind}/{md['name']} in {md.get('namespace')!r}")
if failed:
print("\n".join(failed), file=sys.stderr)
sys.exit(1)
PYEOF
grep -q -- '-filer=test-seaweedfs-filer-client.other-ns:' /tmp/ns-override.yaml
grep -q 'test-seaweedfs-master-0.test-seaweedfs-master.other-ns:' /tmp/ns-override.yaml
grep -qF "'*.other-ns.svc.cluster.local'" /tmp/ns-override.yaml
# Setting it to the release namespace changes nothing.
helm template test $CHART_DIR --namespace release-ns > /tmp/ns-default.yaml
helm template test $CHART_DIR --namespace release-ns \
--set namespaceOverride=release-ns > /tmp/ns-same.yaml
diff /tmp/ns-default.yaml /tmp/ns-same.yaml
echo "namespaceOverride moves every namespaced object and address"
echo "All template rendering tests passed!"
- name: Resolve an image tag that is published
@@ -2276,7 +1869,7 @@ jobs:
echo "IMAGE_TAG=$TAG" >> $GITHUB_ENV
- name: Create kind cluster
uses: helm/kind-action@v1.15.2
uses: helm/kind-action@v1.15.0
- name: Verify volume resize hook security contexts
run: |
@@ -2400,38 +1993,6 @@ jobs:
kubectl delete namespace "$NS"
echo "SFTP host key lifecycle tests passed"
- name: Verify the filer db secret survives uninstall and install
run: |
set -e
CHART_DIR="k8s/charts/seaweedfs"
NS="filer-db-secret"
SECRET="db-seaweedfs-db-secret"
DB_ARGS="--set image.tag=$IMAGE_TAG --set-string filer.extraEnvironmentVars.WEED_MYSQL_ENABLED=true --set master.enabled=false --set volume.enabled=false"
kubectl create namespace "$NS"
echo "=== install generates the db credentials ==="
helm install db $CHART_DIR -n "$NS" $DB_ARGS
[ "$(kubectl get secret "$SECRET" -n "$NS" -o jsonpath='{.data.user}' | base64 -d)" = YourSWUser ] \
|| { echo "FAIL: install did not create the default user"; exit 1; }
[ "$(kubectl get secret "$SECRET" -n "$NS" -o jsonpath='{.data.password}' | base64 -d | wc -c)" -eq 32 ] \
|| { echo "FAIL: install did not generate a 32-character password"; exit 1; }
echo "=== credentials in the kept Secret survive uninstall and install ==="
PASSWORD='p"a\ss: #x {{ .Values }}'
kubectl patch secret "$SECRET" -n "$NS" --type merge \
-p "$(jq -n --arg p "$PASSWORD" '{stringData: {user: "seaweed_app", password: $p}}')"
helm uninstall db -n "$NS"
kubectl get secret "$SECRET" -n "$NS" >/dev/null || { echo "FAIL: uninstall removed the db secret"; exit 1; }
helm install db $CHART_DIR -n "$NS" $DB_ARGS
[ "$(kubectl get secret "$SECRET" -n "$NS" -o jsonpath='{.data.user}' | base64 -d)" = seaweed_app ] \
|| { echo "FAIL: reinstall reset the db user"; exit 1; }
[ "$(kubectl get secret "$SECRET" -n "$NS" -o jsonpath='{.data.password}' | base64 -d)" = "$PASSWORD" ] \
|| { echo "FAIL: reinstall replaced the db password"; exit 1; }
helm uninstall db -n "$NS"
kubectl delete namespace "$NS"
echo "Filer db secret reinstall test passed"
- name: Verify JWT signing key persistence across upgrades
run: |
# chart-testing puts its pip-less venv first on PATH; use setup-python.
+1 -7
View File
@@ -14,10 +14,6 @@ on:
- 'weed/**'
- '.github/workflows/java_integration_tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
jobs:
test:
name: Java Integration Tests
@@ -35,7 +31,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Set up Java
@@ -43,7 +38,7 @@ jobs:
with:
java-version: ${{ matrix.java }}
distribution: 'temurin'
cache: ${{ runner.environment == 'github-hosted' && 'maven' || '' }}
cache: 'maven'
- name: Build SeaweedFS
run: |
@@ -60,7 +55,6 @@ jobs:
# Start SeaweedFS with optimized settings for CI
# Include S3 API for s3copier integration tests
weed server -dir="$WEED_DATA_DIR" \
-ip=127.0.0.1 -ip.bind=127.0.0.1 \
-master.raftHashicorp \
-master.electionTimeout=1s \
-master.volumeSizeLimitMB=100 \
+1 -1
View File
@@ -77,7 +77,7 @@ jobs:
with:
java-version: '17'
distribution: 'temurin'
cache: ${{ runner.environment == 'github-hosted' && 'maven' || '' }}
cache: 'maven'
server-id: central
server-username: MAVEN_CENTRAL_USERNAME
server-password: MAVEN_CENTRAL_PASSWORD
+1 -5
View File
@@ -12,10 +12,6 @@ on:
- 'other/java/**'
- '.github/workflows/java_unit_tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
jobs:
test:
name: Java Unit Tests
@@ -34,7 +30,7 @@ jobs:
with:
java-version: ${{ matrix.java }}
distribution: 'temurin'
cache: ${{ runner.environment == 'github-hosted' && 'maven' || '' }}
cache: 'maven'
- name: Build and Install SeaweedFS Client
working-directory: other/java/client
+3 -3
View File
@@ -24,8 +24,8 @@ on:
workflow_dispatch: # Allow manual trigger
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
group: ${{ github.head_ref }}/kafka-quicktest
cancel-in-progress: true
permissions:
contents: read
@@ -43,7 +43,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version: ^1.26
cache: ${{ runner.environment == 'github-hosted' }}
cache: true
cache-dependency-path: |
**/go.sum
id: go
+6 -9
View File
@@ -23,8 +23,8 @@ on:
- '.github/workflows/kafka-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
group: ${{ github.head_ref }}/kafka-tests
cancel-in-progress: true
# Force different runners for better isolation
env:
@@ -53,7 +53,6 @@ jobs:
- name: Set up Go 1.x
uses: actions/setup-go@v7
with:
cache: ${{ runner.environment == 'github-hosted' }}
go-version: ^1.26
id: go
@@ -99,7 +98,6 @@ jobs:
- name: Set up Go 1.x
uses: actions/setup-go@v7
with:
cache: ${{ runner.environment == 'github-hosted' }}
go-version: ^1.26
id: go
@@ -151,7 +149,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version: ^1.26
cache: ${{ runner.environment == 'github-hosted' }}
cache: true
cache-dependency-path: |
**/go.sum
id: go
@@ -330,7 +328,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version: ^1.26
cache: ${{ runner.environment == 'github-hosted' }}
cache: true
cache-dependency-path: |
**/go.sum
id: go
@@ -492,7 +490,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version: ^1.26
cache: ${{ runner.environment == 'github-hosted' }}
cache: true
cache-dependency-path: |
**/go.sum
id: go
@@ -650,7 +648,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version: ^1.26
cache: ${{ runner.environment == 'github-hosted' }}
cache: true
cache-dependency-path: |
**/go.sum
id: go
@@ -807,7 +805,6 @@ jobs:
- name: Set up Go 1.x
uses: actions/setup-go@v7
with:
cache: ${{ runner.environment == 'github-hosted' }}
go-version: ^1.26
id: go
+4 -6
View File
@@ -19,8 +19,8 @@ on:
- 'test/kms/**'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
group: ${{ github.head_ref || github.ref }}-kms-tests
cancel-in-progress: true
permissions:
contents: read
@@ -32,7 +32,7 @@ defaults:
jobs:
kms-provider-tests:
name: KMS Provider Integration Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 20
steps:
@@ -43,7 +43,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -78,7 +77,7 @@ jobs:
s3-kms-e2e-tests:
name: S3 KMS End-to-End Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 25
steps:
@@ -89,7 +88,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -21,8 +21,8 @@ on:
- '.github/workflows/master-cold-start-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
group: ${{ github.head_ref || github.ref }}/master-cold-start-tests
cancel-in-progress: true
permissions:
contents: read
@@ -30,7 +30,7 @@ permissions:
jobs:
master-cold-start-tests:
name: Master Cold Start Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 10
steps:
- name: Check out code
@@ -42,7 +42,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Build weed binary
run: |
@@ -25,8 +25,8 @@ on:
- '.github/workflows/metadata-subscribe-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
group: ${{ github.head_ref }}/metadata-subscribe-tests
cancel-in-progress: true
permissions:
contents: read
@@ -37,7 +37,7 @@ env:
jobs:
metadata-subscribe-integration:
name: Metadata Subscribe Integration Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 20
steps:
@@ -48,7 +48,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Build SeaweedFS
run: |
+2 -4
View File
@@ -10,8 +10,8 @@ on:
branches: [ 'winfsp-bench**' ]
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
cancel-in-progress: true
permissions:
contents: read
@@ -32,7 +32,6 @@ jobs:
- uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Install WinFsp and rclone
run: choco install winfsp rclone -y --no-progress
@@ -151,7 +150,6 @@ jobs:
- uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Repair the fusermount3 setuid bit
uses: ./.github/actions/fix-fusermount-setuid
@@ -24,8 +24,8 @@ on:
- '.github/workflows/mount-windows-conformance.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
cancel-in-progress: true
permissions:
contents: read
@@ -49,7 +49,6 @@ jobs:
- uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
# cgofuse loads winfsp-x64.dll at run time, so WinFsp is needed here but
# not to build.
+2 -3
View File
@@ -24,8 +24,8 @@ on:
- '.github/workflows/mount-windows.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
cancel-in-progress: true
permissions:
contents: read
@@ -49,7 +49,6 @@ jobs:
- uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
# cgofuse loads winfsp-x64.dll at run time, so WinFsp is needed here but
# not to build.
+3 -4
View File
@@ -21,8 +21,8 @@ on:
- '.github/workflows/multi-master-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
group: ${{ github.head_ref || github.ref }}/multi-master-tests
cancel-in-progress: true
permissions:
contents: read
@@ -30,7 +30,7 @@ permissions:
jobs:
multi-master-failover-tests:
name: Multi-Master Failover Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 10
steps:
- name: Check out code
@@ -40,7 +40,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Install SeaweedFS
run: |
+4 -7
View File
@@ -45,8 +45,8 @@ on:
type: string
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
group: ${{ github.head_ref || github.ref }}/performance
cancel-in-progress: true
permissions:
contents: read
@@ -67,7 +67,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Build weed
run: go build -o weed_bin ./weed
@@ -176,14 +175,13 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Install Rust toolchain
if: matrix.impl == 'rust'
uses: dtolnay/rust-toolchain@stable
- name: Cache cargo registry and target
if: matrix.impl == 'rust' && runner.environment == 'github-hosted'
if: matrix.impl == 'rust'
uses: actions/cache@v6
with:
path: |
@@ -329,14 +327,13 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Install Rust toolchain
if: matrix.impl == 'rust'
uses: dtolnay/rust-toolchain@stable
- name: Cache cargo registry and target
if: matrix.impl == 'rust' && runner.environment == 'github-hosted'
if: matrix.impl == 'rust'
uses: actions/cache@v6
with:
path: |
+2 -10
View File
@@ -27,7 +27,7 @@ permissions:
jobs:
pjdfstest:
name: pjdfstest
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 60
steps:
@@ -38,23 +38,15 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Start local Docker registry
run: |
if ! docker start registry 2>/dev/null; then
docker rm -f registry >/dev/null 2>&1 || true
docker run -d --restart=always -p 5000:5000 --name registry registry:2
fi
run: docker run -d --restart=always -p 5000:5000 --name registry registry:2
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4
with:
driver-opts: network=host
- name: Set up Docker Compose
uses: docker/setup-compose-action@v2
- name: Build weed race binary
run: |
cd docker
+1 -6
View File
@@ -18,17 +18,13 @@ on:
- 'go.sum'
- '.github/workflows/plugin-workers.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
jobs:
plugin-worker:
name: "Plugin Worker: ${{ matrix.worker }}"
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 10
strategy:
fail-fast: false
@@ -45,7 +41,6 @@ jobs:
- name: Set up Go 1.x
uses: actions/setup-go@v7
with:
cache: ${{ runner.environment == 'github-hosted' }}
go-version: ^1.26
id: go
-791
View File
@@ -1,791 +0,0 @@
name: POSIX Compliance Tests
on:
push:
branches: [ main, master, develop ]
paths:
- 'weed/mount/**'
- 'weed/command/mount*'
- 'weed/command/fuse*'
- 'test/fuse_integration/**'
pull_request:
branches: [ main, master, develop ]
paths:
- 'weed/mount/**'
- 'weed/command/mount*'
- 'weed/command/fuse*'
- 'test/fuse_integration/**'
workflow_dispatch:
inputs:
test_type:
description: 'Type of POSIX tests to run (default: full for comprehensive testing)'
required: true
default: 'full'
type: choice
options:
- critical
- basic
- extended
- full
enable_external_tests:
description: 'Run external test suites (slower)'
required: false
default: false
type: boolean
permissions:
contents: read
pull-requests: write
issues: write
env:
GO_VERSION: '1.26'
TIMEOUT: '45m'
jobs:
posix-compliance-ubuntu:
runs-on: ubuntu-latest
timeout-minutes: 60
strategy:
matrix:
fuse-version: ['2.9', '3.0']
fail-fast: false
steps:
- name: Checkout code
uses: actions/checkout@v7
with:
fetch-depth: 1
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go ${{ env.GO_VERSION }}
uses: actions/setup-go@v7
with:
go-version: ${{ env.GO_VERSION }}
cache: ${{ runner.environment == 'github-hosted' }}
cache-dependency-path: |
go.sum
test/fuse_integration/go.sum
- name: Install system dependencies
run: |
sudo apt-get update
sudo apt-get install -y \
attr \
acl \
build-essential \
git \
python3-pip
# Install FUSE version specific packages
if [ "${{ matrix.fuse-version }}" = "2.9" ]; then
echo "Installing FUSE 2.9..."
sudo apt-get install -y fuse libfuse-dev
elif [ "${{ matrix.fuse-version }}" = "3.0" ]; then
echo "Installing FUSE 3.0..."
sudo apt-get install -y fuse3 libfuse3-dev
# Also install fuse2 for compatibility if needed
sudo apt-get install -y fuse libfuse-dev
fi
- name: Set up user permissions for FUSE
run: |
# Create fuse group if it doesn't exist
sudo groupadd -f fuse
# Add user to fuse group
sudo usermod -a -G fuse $USER
# Set permissions on FUSE device
sudo chmod 666 /dev/fuse
# Allow non-root mounts to pass -o allow_other (weed -allowOthers)
echo 'user_allow_other' | sudo tee -a /etc/fuse.conf
sudo chmod 644 /etc/fuse.conf
# Ensure fuse module is loaded
sudo modprobe fuse || true
# Verify setup
echo "FUSE setup verification:"
ls -la /dev/fuse
groups $USER | grep fuse || echo "User not in fuse group yet (will be effective after login)"
lsmod | grep fuse || echo "FUSE module not loaded"
- name: Install external test tools
if: ${{ github.event.inputs.enable_external_tests == 'true' }}
run: |
# Install nfstest for POSIX API verification
pip3 install --user nfstest
# Install FIO for performance testing
sudo apt-get install -y fio
# Install additional test utilities
sudo apt-get install -y stress-ng
- name: Build SeaweedFS
run: |
make
# Verify binary exists and is executable (it's installed to GOPATH/bin)
weed version
# Make weed binary available in PATH (it's already there from go install)
sudo cp $(which weed) /usr/local/bin/weed
which weed
weed version
- name: Set up SeaweedFS cluster
run: |
start_and_wait() {
local name=$1
local url=$2
local pidfile=$3
shift 3
local cmd="$@"
echo "Starting $name..."
$cmd > "/tmp/seaweedfs/$name.log" 2>&1 &
local pid=$!
echo $pid > "$pidfile"
echo "Waiting for $name to start..."
for i in {1..30}; do
if curl -sf "$url" > /dev/null 2>&1; then
echo "$name is ready"
return 0
fi
if [ $i -eq 30 ]; then
echo "$name failed to start"
cat "/tmp/seaweedfs/$name.log"
exit 1
fi
sleep 2
done
}
mkdir -p /tmp/seaweedfs/{master,volume,filer,mount}
start_and_wait "master" "http://127.0.0.1:9333/cluster/status" "/tmp/seaweedfs/master.pid" \
"weed master -ip=127.0.0.1 -port=9333 -mdir=/tmp/seaweedfs/master -raftBootstrap=true"
start_and_wait "volume" "http://127.0.0.1:8080/status" "/tmp/seaweedfs/volume.pid" \
"weed volume -mserver=127.0.0.1:9333 -ip=127.0.0.1 -port=8080 -dir=/tmp/seaweedfs/volume -max=100"
start_and_wait "filer" "http://127.0.0.1:8888/" "/tmp/seaweedfs/filer.pid" \
"weed filer -master=127.0.0.1:9333 -ip=127.0.0.1 -port=8888"
echo "SeaweedFS cluster status:"
curl -s http://127.0.0.1:9333/cluster/status || true
- name: Set up FUSE mount
run: |
# Create mount point
MOUNT_POINT="/tmp/seaweedfs/mount"
mkdir -p $MOUNT_POINT
echo "Mounting SeaweedFS FUSE filesystem..."
# Mount SeaweedFS FUSE filesystem in background
weed mount \
-filer=127.0.0.1:8888 \
-dir=$MOUNT_POINT \
-filer.path=/ \
-dirAutoCreate=true \
-allowOthers=true \
-nonempty=true \
> /tmp/seaweedfs/mount.log 2>&1 &
MOUNT_PID=$!
echo $MOUNT_PID > /tmp/seaweedfs/mount.pid
# Wait for mount to be ready. The mount point directory already
# exists, so only a real mount counts — a directory listing would
# pass on the local disk even if weed mount died.
echo "Waiting for FUSE mount to be ready..."
for i in {1..30}; do
if mountpoint -q $MOUNT_POINT 2>/dev/null; then
echo "FUSE mount is ready"
break
fi
if ! kill -0 $MOUNT_PID 2>/dev/null; then
echo "FUSE mount process exited"
cat /tmp/seaweedfs/mount.log
exit 1
fi
if [ $i -eq 30 ]; then
echo "FUSE mount failed to be ready"
cat /tmp/seaweedfs/mount.log
echo "Mount status:"
mount | grep fuse || echo "No FUSE mounts found"
ps aux | grep weed || echo "No weed processes found"
exit 1
fi
sleep 2
done
# Verify mount is working
echo "Testing FUSE mount functionality..."
echo "test" > $MOUNT_POINT/test_file.txt
if [ "$(cat $MOUNT_POINT/test_file.txt)" = "test" ]; then
echo "FUSE mount is working correctly"
rm $MOUNT_POINT/test_file.txt
else
echo "FUSE mount is not working properly"
exit 1
fi
# Export mount point for tests
echo "SEAWEEDFS_MOUNT_POINT=$MOUNT_POINT" >> $GITHUB_ENV
- name: Set up test environment
run: |
cd test/fuse_integration
# Initialize Go module for tests
if [ ! -f go.mod ]; then
go mod init seaweedfs-posix-tests
go mod tidy
fi
# Create reports directory
mkdir -p reports
# Set up external tools if requested
# Note: External tools setup is now optional and won't fail the build
if [ "${{ github.event.inputs.enable_external_tests }}" = "true" ]; then
echo "Setting up external POSIX testing tools..."
make -f posix_Makefile setup-external-tools || true
# setup-external-tools exits 0 even when a sub-target failed, so
# require the tools only when they are actually installed
if [ -d external_tools/pjdfstest ] && command -v nfstest_posix >/dev/null 2>&1; then
export POSIX_REQUIRE_EXTERNAL_TOOLS="true"
echo "POSIX_REQUIRE_EXTERNAL_TOOLS=true" >> $GITHUB_ENV
echo "✅ External tools setup completed successfully"
else
echo "⚠️ External tools setup incomplete - continuing with basic tests only"
echo "External tools will be skipped but tests will continue"
echo "This is expected behavior when build tools are not available"
fi
fi
# Verify SeaweedFS cluster is accessible
echo "Verifying SeaweedFS cluster accessibility..."
curl -sf http://127.0.0.1:9333/cluster/status
curl -sf http://127.0.0.1:8080/status
curl -sf http://127.0.0.1:8888/
# Verify mount point
echo "Mount point: $SEAWEEDFS_MOUNT_POINT"
ls -la $SEAWEEDFS_MOUNT_POINT
# Verify mount is actually working by testing file operations
echo "Verifying SeaweedFS mount functionality..."
TEST_FILE="$SEAWEEDFS_MOUNT_POINT/.mount_verification_test"
echo "SeaweedFS mount test" > "$TEST_FILE"
if [ "$(cat "$TEST_FILE")" = "SeaweedFS mount test" ]; then
echo "✅ Mount verification successful"
rm -f "$TEST_FILE"
else
echo "❌ Mount verification failed - mount may not be working properly"
exit 1
fi
# Check if this is actually a FUSE mount — the plain directory created
# above passes the file test, so without this the suite can run
# against the local disk and report a pass it never earned
if mount | grep -q "$SEAWEEDFS_MOUNT_POINT"; then
echo "✅ FUSE mount detected:"
mount | grep "$SEAWEEDFS_MOUNT_POINT"
else
echo "❌ Mount point not found in mount table - refusing to test a local directory"
exit 1
fi
- name: Run POSIX compliance tests
id: posix-tests
env:
SEAWEEDFS_MOUNT_POINT: ${{ env.SEAWEEDFS_MOUNT_POINT }}
POSIX_REQUIRE_EXTERNAL_TOOLS: ${{ env.POSIX_REQUIRE_EXTERNAL_TOOLS }}
run: |
cd test/fuse_integration
# Determine which tests to run
TEST_TYPE="${{ github.event.inputs.test_type }}"
if [ -z "$TEST_TYPE" ]; then
TEST_TYPE="full"
fi
echo "Running POSIX tests: $TEST_TYPE"
if [ "$TEST_TYPE" = "full" ]; then
echo "🚀 Running COMPREHENSIVE POSIX test suite (including external test suites)"
else
echo "Using mount point: $SEAWEEDFS_MOUNT_POINT"
fi
# Set test configuration to use our mounted filesystem
export TEST_MOUNT_POINT="$SEAWEEDFS_MOUNT_POINT"
export TEST_SKIP_CLUSTER_SETUP="true"
# Debug: Show environment variables
echo "🔍 Test Environment:"
echo " TEST_MOUNT_POINT=$TEST_MOUNT_POINT"
echo " TEST_SKIP_CLUSTER_SETUP=$TEST_SKIP_CLUSTER_SETUP"
echo " SEAWEEDFS_MOUNT_POINT=$SEAWEEDFS_MOUNT_POINT"
case "$TEST_TYPE" in
"critical")
echo "Running critical POSIX tests (basic compliance only)..."
make -f posix_Makefile test-posix-critical
;;
"basic")
echo "Running basic POSIX tests (core file operations)..."
make -f posix_Makefile test-posix-basic
;;
"extended")
echo "Running extended POSIX tests (including advanced features)..."
make -f posix_Makefile test-posix-extended
;;
"full")
echo "🚀 Running COMPREHENSIVE POSIX test suite (all tests including external)..."
echo " This includes: basic, extended, and external test suites"
echo " External suites: pjdfstest, nfstest_posix, and custom stress tests"
make -f posix_Makefile test-posix-full
;;
*)
echo "Unknown test type: $TEST_TYPE, defaulting to full suite"
echo "🚀 Running full POSIX test suite (all tests including external)..."
make -f posix_Makefile test-posix-full
;;
esac
- name: Run external test suites
if: ${{ github.event.inputs.enable_external_tests == 'true' }}
continue-on-error: true
run: |
cd test/fuse_integration
# Point the external suites at the same mount as the Go tests;
# without these the Makefile falls back to a local-disk path.
export TEST_MOUNT_POINT="$SEAWEEDFS_MOUNT_POINT"
export TEST_SKIP_CLUSTER_SETUP="true"
# Run external tests (may fail on some systems)
make -f posix_Makefile test-nfstest-posix || echo "nfstest failed or not available"
make -f posix_Makefile test-fio-posix || echo "FIO tests failed or not available"
- name: Generate compliance report
if: always()
run: |
cd test/fuse_integration
make -f posix_Makefile generate-report
- name: Collect system information for debugging
if: failure()
run: |
cd test/fuse_integration
mkdir -p reports/debug
echo "System Information" > reports/debug/system_info.txt
uname -a >> reports/debug/system_info.txt
echo "" >> reports/debug/system_info.txt
echo "Mount Information" >> reports/debug/system_info.txt
mount >> reports/debug/system_info.txt
echo "" >> reports/debug/system_info.txt
echo "Disk Space" >> reports/debug/system_info.txt
timeout 15 df -h >> reports/debug/system_info.txt || true
echo "" >> reports/debug/system_info.txt
echo "Memory Information" >> reports/debug/system_info.txt
free -h >> reports/debug/system_info.txt
echo "" >> reports/debug/system_info.txt
echo "FUSE Information" >> reports/debug/system_info.txt
ls -la /dev/fuse >> reports/debug/system_info.txt
lsmod | grep fuse >> reports/debug/system_info.txt
echo "" >> reports/debug/system_info.txt
echo "SeaweedFS Version" >> reports/debug/system_info.txt
weed version >> reports/debug/system_info.txt
echo "" >> reports/debug/system_info.txt
echo "Go Version" >> reports/debug/system_info.txt
go version >> reports/debug/system_info.txt
echo "" >> reports/debug/system_info.txt
echo "Running Processes" >> reports/debug/system_info.txt
ps aux | grep -E "(weed|fuse)" >> reports/debug/system_info.txt
echo "" >> reports/debug/system_info.txt
# Collect SeaweedFS service logs
echo "=== SeaweedFS Service Logs ===" >> reports/debug/system_info.txt
if [ -f /tmp/seaweedfs/master.log ]; then
echo "Master Log:" >> reports/debug/system_info.txt
tail -50 /tmp/seaweedfs/master.log >> reports/debug/system_info.txt
echo "" >> reports/debug/system_info.txt
fi
if [ -f /tmp/seaweedfs/volume.log ]; then
echo "Volume Log:" >> reports/debug/system_info.txt
tail -50 /tmp/seaweedfs/volume.log >> reports/debug/system_info.txt
echo "" >> reports/debug/system_info.txt
fi
if [ -f /tmp/seaweedfs/filer.log ]; then
echo "Filer Log:" >> reports/debug/system_info.txt
tail -50 /tmp/seaweedfs/filer.log >> reports/debug/system_info.txt
echo "" >> reports/debug/system_info.txt
fi
if [ -f /tmp/seaweedfs/mount.log ]; then
echo "Mount Log:" >> reports/debug/system_info.txt
tail -50 /tmp/seaweedfs/mount.log >> reports/debug/system_info.txt
echo "" >> reports/debug/system_info.txt
fi
# Copy full logs as separate files
if [ -f /tmp/seaweedfs/master.log ]; then
cp /tmp/seaweedfs/master.log reports/debug/master.log
fi
if [ -f /tmp/seaweedfs/volume.log ]; then
cp /tmp/seaweedfs/volume.log reports/debug/volume.log
fi
if [ -f /tmp/seaweedfs/filer.log ]; then
cp /tmp/seaweedfs/filer.log reports/debug/filer.log
fi
if [ -f /tmp/seaweedfs/mount.log ]; then
cp /tmp/seaweedfs/mount.log reports/debug/mount.log
fi
- name: Cleanup SeaweedFS cluster and FUSE mount
if: always()
run: |
stop_process() {
local name=$1
local pidfile=$2
if [ -f "$pidfile" ]; then
local pid=$(cat "$pidfile")
if kill -0 $pid 2>/dev/null; then
echo "Stopping $name process (PID: $pid)..."
kill -TERM $pid || true
sleep 2
kill -KILL $pid 2>/dev/null || true
fi
fi
}
echo "Cleaning up SeaweedFS cluster and FUSE mount..."
MOUNT_POINT="/tmp/seaweedfs/mount"
if mountpoint -q $MOUNT_POINT 2>/dev/null; then
echo "Unmounting FUSE filesystem..."
fusermount -u $MOUNT_POINT || umount $MOUNT_POINT || true
fi
stop_process "mount" "/tmp/seaweedfs/mount.pid"
stop_process "filer" "/tmp/seaweedfs/filer.pid"
stop_process "volume" "/tmp/seaweedfs/volume.pid"
stop_process "master" "/tmp/seaweedfs/master.pid"
pkill -f "weed " || true
fusermount -u $MOUNT_POINT 2>/dev/null || true
umount $MOUNT_POINT 2>/dev/null || true
rm -rf /tmp/seaweedfs || true
echo "Cleanup completed"
- name: Upload test results
uses: actions/upload-artifact@v7
if: always()
with:
name: posix-test-results-ubuntu-fuse${{ matrix.fuse-version }}
path: |
test/fuse_integration/reports/
test/fuse_integration/*.log
retention-days: 30
- name: Upload test coverage
uses: actions/upload-artifact@v7
if: always() && (github.event.inputs.test_type == 'full' || github.event.inputs.test_type == 'extended')
with:
name: posix-coverage-ubuntu
path: test/fuse_integration/reports/posix_coverage.html
retention-days: 14
- name: Comment PR with results
if: github.event_name == 'pull_request' && always()
uses: actions/github-script@v9
with:
script: |
const fs = require('fs');
const path = require('path');
const reportPath = 'test/fuse_integration/reports/posix_compliance_summary.txt';
if (fs.existsSync(reportPath)) {
const report = fs.readFileSync(reportPath, 'utf8');
const comment = `## POSIX Compliance Test Results
**Go Version:** ${{ env.GO_VERSION }}
**FUSE Version:** ${{ matrix.fuse-version }}
**Test Type:** ${{ github.event.inputs.test_type || 'full' }}
<details>
<summary>Test Summary</summary>
\`\`\`
${report}
\`\`\`
</details>
📊 Full results available in [test artifacts](https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }})
`;
try {
await github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: comment
});
} catch (err) {
console.log('Could not post PR comment: ' + err.message);
}
}
posix-compliance-macos:
runs-on: macos-latest
timeout-minutes: 60
steps:
- name: Checkout code
uses: actions/checkout@v7
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version: ${{ env.GO_VERSION }}
cache: ${{ runner.environment == 'github-hosted' }}
- name: Install macFUSE
run: |
# Install macFUSE
brew install macfuse
# Note: macFUSE may require system extension approval on macOS
# This step may need manual intervention in some cases
- name: Build SeaweedFS
run: |
make
# Verify binary exists and is executable (it's installed to GOPATH/bin)
weed version
- name: Run critical POSIX tests (macOS)
continue-on-error: true # macOS FUSE can be more restrictive
run: |
cd test/fuse_integration
if [ ! -f go.mod ]; then
go mod init seaweedfs-posix-tests
go mod tidy
fi
mkdir -p reports
# Run basic tests only on macOS due to FUSE limitations
make -f posix_Makefile test-posix-critical
- name: Upload macOS test results
uses: actions/upload-artifact@v7
if: always()
with:
name: posix-test-results-macos
path: |
test/fuse_integration/reports/
test/fuse_integration/*.log
retention-days: 30
security-analysis:
runs-on: ubuntu-latest
if: github.event_name == 'pull_request'
steps:
- name: Checkout code
uses: actions/checkout@v7
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version: ${{ env.GO_VERSION }}
cache: ${{ runner.environment == 'github-hosted' }}
- name: Install security analysis tools
run: |
echo "Installing staticcheck for security analysis..."
go install honnef.co/go/tools/cmd/staticcheck@latest
- name: Run security analysis on FUSE code
run: |
# Analyze mount and FUSE-related code for security and quality issues
echo "Running staticcheck security analysis..."
# Run staticcheck on FUSE-related code
echo "Analyzing FUSE mount code..."
staticcheck ./weed/mount/... > staticcheck-report.txt 2>&1 || true
staticcheck ./weed/command/mount* ./weed/command/fuse* >> staticcheck-report.txt 2>&1 || true
# Convert to JSON format for consistency
echo "{" > gosec-report.json
echo " \"tool\": \"staticcheck\"," >> gosec-report.json
echo " \"timestamp\": \"$(date -Iseconds)\"," >> gosec-report.json
echo " \"issues\": [" >> gosec-report.json
if [ -s staticcheck-report.txt ]; then
echo " \"$(cat staticcheck-report.txt | head -20 | sed 's/"/\\"/g' | tr '\n' ' ')\"" >> gosec-report.json
fi
echo " ]," >> gosec-report.json
echo " \"stats\": {" >> gosec-report.json
echo " \"files_analyzed\": $(find ./weed/mount ./weed/command -name '*.go' | wc -l)," >> gosec-report.json
echo " \"issues_found\": $(wc -l < staticcheck-report.txt 2>/dev/null || echo 0)" >> gosec-report.json
echo " }" >> gosec-report.json
echo "}" >> gosec-report.json
echo "Security analysis completed"
echo "Issues found: $(wc -l < staticcheck-report.txt 2>/dev/null || echo 0)"
- name: Upload security analysis results
uses: actions/upload-artifact@v7
if: always()
with:
name: security-analysis-results
path: gosec-report.json
retention-days: 30
performance-baseline:
runs-on: ubuntu-latest
if: github.event.inputs.test_type == 'full' || github.event_name == 'schedule'
steps:
- name: Checkout code
uses: actions/checkout@v7
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version: ${{ env.GO_VERSION }}
cache: ${{ runner.environment == 'github-hosted' }}
- name: Install dependencies
run: |
sudo apt-get update
sudo apt-get install -y fuse libfuse-dev fio
sudo usermod -a -G fuse $USER
sudo chmod 666 /dev/fuse
# The posix suites mount with -allowOthers; allow_other needs this
echo 'user_allow_other' | sudo tee -a /etc/fuse.conf
- name: Build SeaweedFS
run: make
- name: Run performance baseline tests
run: |
cd test/fuse_integration
if [ ! -f go.mod ]; then
go mod init seaweedfs-posix-tests
go mod tidy
fi
mkdir -p reports
# Run performance benchmarks
make -f posix_Makefile benchmark-posix
# This job has no FUSE mount, so the fio suite can only skip; keep the
# run green but honest about producing no mounted-FS results
make -f posix_Makefile test-fio-posix || echo "FIO skipped - no mounted filesystem in this job"
- name: Store performance baseline
uses: actions/upload-artifact@v7
with:
name: performance-baseline-results
path: |
test/fuse_integration/reports/posix_benchmark_results.log
test/fuse_integration/reports/fio_*_results.log
retention-days: 90
# Schedule regular compliance checks
scheduled-compliance-check:
runs-on: ubuntu-latest
if: github.event_name == 'schedule'
steps:
- name: Checkout code
uses: actions/checkout@v7
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version: ${{ env.GO_VERSION }}
cache: ${{ runner.environment == 'github-hosted' }}
- name: Install dependencies
run: |
sudo apt-get update
sudo apt-get install -y fuse libfuse-dev
pip3 install --user nfstest
# The posix suites mount with -allowOthers; allow_other needs this
echo 'user_allow_other' | sudo tee -a /etc/fuse.conf
- name: Build SeaweedFS
run: make
- name: Run comprehensive compliance check
run: |
cd test/fuse_integration
make -f posix_Makefile test-posix-full
make -f posix_Makefile generate-report
- name: Create compliance issue if tests fail
if: failure()
uses: actions/github-script@v9
with:
script: |
const issue = await github.rest.issues.create({
owner: context.repo.owner,
repo: context.repo.repo,
title: 'POSIX Compliance Check Failed - ' + new Date().toISOString().split('T')[0],
body: `## POSIX Compliance Check Failure
The scheduled POSIX compliance check has failed. This may indicate:
- Regression in FUSE mount functionality
- Changes affecting POSIX compatibility
- Infrastructure issues with the test environment
**Action Required:** Review the test results and investigate any failures.
**Test Run:** [Workflow Run](https://github.com/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId})
**Date:** ${new Date().toISOString()}
---
This issue was automatically created by the POSIX compliance workflow.
`,
labels: ['bug', 'posix-compliance', 'automated-issue']
});
console.log('Created issue:', issue.data.number);
# Schedule this workflow to run weekly
# Uncomment the following lines to enable scheduled runs:
# schedule:
# - cron: '0 2 * * 1' # Every Monday at 2 AM UTC
+2 -4
View File
@@ -23,8 +23,8 @@ on:
- '.github/workflows/postgres-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
group: ${{ github.head_ref }}/postgres-tests
cancel-in-progress: true
permissions:
contents: read
@@ -41,7 +41,6 @@ jobs:
- name: Set up Go 1.x
uses: actions/setup-go@v7
with:
cache: ${{ runner.environment == 'github-hosted' }}
go-version: ^1.26
id: go
@@ -57,7 +56,6 @@ jobs:
uses: docker/setup-buildx-action@v4
- name: Cache Docker layers
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: /tmp/.buildx-cache
-135
View File
@@ -1,135 +0,0 @@
name: "rust: vector desktop platforms"
on:
pull_request:
paths:
- 'seaweed-vector/**'
- 'seaweed-worker/**'
- 'seaweed-common/**'
- 'weed/pb/**'
- '.github/workflows/rust-vector-platforms.yml'
- 'install.sh'
- 'test/installer/**'
push:
tags: ['*']
workflow_dispatch:
permissions:
contents: read
concurrency:
group: vector-platforms-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
jobs:
vector:
permissions:
contents: read
strategy:
fail-fast: false
matrix:
include:
- runner: macos-15-intel
suffix: darwin_amd64
protoc: macos-x86_64
- runner: macos-latest
suffix: darwin_arm64
protoc: macos-aarch_64
- runner: windows-latest
suffix: windows_amd64
protoc: win32
runs-on: ${{ matrix.runner }}
timeout-minutes: 120
env:
CARGO_BUILD_JOBS: '2'
CARGO_PROFILE_TEST_DEBUG: '0'
SEAWEEDFS_COMMIT: ${{ github.sha }}
defaults:
run:
shell: bash
steps:
- uses: actions/checkout@v7
with:
persist-credentials: false
- uses: dtolnay/rust-toolchain@stable
- uses: actions/setup-python@v7
with:
python-version: '3.13'
- uses: actions/cache/restore@v6
id: cargo-cache
if: ${{ !startsWith(github.ref, 'refs/tags/') }}
with:
path: |
~/.cargo/registry
~/.cargo/git
seaweed-vector/target
key: vector-${{ matrix.suffix }}-${{ hashFiles('seaweed-vector/Cargo.lock') }}
restore-keys: vector-${{ matrix.suffix }}-
- name: Fetch dependencies
run: cargo fetch --locked --manifest-path seaweed-vector/Cargo.toml
- name: Select vendored protoc
env:
PROTOC_PACKAGE: protoc-bin-vendored-${{ matrix.protoc }}
run: |
python - <<'PY'
import os, pathlib, tomllib
lock=tomllib.loads(pathlib.Path('seaweed-vector/Cargo.lock').read_text())
name=os.environ['PROTOC_PACKAGE']
version=next(p['version'] for p in lock['package'] if p['name']==name)
cargo=pathlib.Path(os.environ.get('CARGO_HOME',pathlib.Path.home()/'.cargo'))
executable='protoc.exe' if os.name=='nt' else 'protoc'
path=next(cargo.glob(f'registry/src/*/{name}-{version}/bin/{executable}'))
with open(os.environ['GITHUB_ENV'],'a') as out: out.write(f'PROTOC={path}\n')
PY
- name: Build vector engine
id: cargo-build
run: cargo build --locked --release --manifest-path seaweed-vector/Cargo.toml -p weed-vector-server
- name: Test vector crates
run: cargo test --locked --release --manifest-path seaweed-vector/Cargo.toml --workspace
- name: Test installer
run: |
if [ "$RUNNER_OS" = "Windows" ]; then
BASH_BINARY="$(cygpath -w "$BASH")"
export BASH_BINARY
fi
python -m unittest discover -s test/installer
- name: Save compiled dependencies
if: ${{ always() && !startsWith(github.ref, 'refs/tags/') && steps.cargo-build.outcome == 'success' && steps.cargo-cache.outputs.cache-hit != 'true' }}
uses: actions/cache/save@v6
with:
path: |
~/.cargo/registry
~/.cargo/git
seaweed-vector/target
key: vector-${{ matrix.suffix }}-${{ hashFiles('seaweed-vector/Cargo.lock') }}
- name: Package engine
env:
ASSET_SUFFIX: ${{ matrix.suffix }}
run: |
python - <<'PY'
import hashlib, os, pathlib, tarfile, zipfile
suffix=os.environ['ASSET_SUFFIX']
binary='weed-vector.exe' if os.name=='nt' else 'weed-vector'
source=pathlib.Path('seaweed-vector/target/release')/binary
asset=f'weed-vector_{suffix}'+('.zip' if os.name=='nt' else '.tar.gz')
if os.name=='nt':
with zipfile.ZipFile(asset,'w',zipfile.ZIP_DEFLATED) as archive: archive.write(source,binary)
else:
with tarfile.open(asset,'w:gz') as archive: archive.add(source,arcname=binary)
digest=hashlib.md5(pathlib.Path(asset).read_bytes()).hexdigest()
pathlib.Path(asset+'.md5').write_text(f'{digest} {asset}\n')
PY
- uses: actions/upload-artifact@v7
with:
name: vector-${{ matrix.suffix }}
path: weed-vector_${{ matrix.suffix }}.*
release:
if: startsWith(github.ref, 'refs/tags/')
needs: vector
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- uses: actions/download-artifact@v8
with:
pattern: vector-*
merge-multiple: true
- uses: softprops/action-gh-release@v3
with:
files: weed-vector_*
-181
View File
@@ -1,181 +0,0 @@
name: "Rust Vector Tests"
on:
pull_request:
branches: [ master ]
paths:
- 'seaweed-vector/**'
- 'seaweed-volume/**'
- 'seaweed-worker/**'
- 'seaweed-common/**'
- 'weed/pb/plugin.proto'
- 'weed/pb/vector.proto'
- 'weed/pb/vector_pb/**'
- 'weed/s3api/**'
- 'weed/command/**'
- 'test/s3vectors/**'
- '.github/workflows/rust-vector-tests.yml'
push:
branches: [ master, main ]
paths:
- 'seaweed-vector/**'
- 'seaweed-volume/**'
- 'seaweed-worker/**'
- 'seaweed-common/**'
- 'weed/pb/plugin.proto'
- 'weed/pb/vector.proto'
- 'weed/pb/vector_pb/**'
- 'weed/s3api/**'
- 'weed/command/**'
- 'test/s3vectors/**'
- '.github/workflows/rust-vector-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
jobs:
rust-vector-build:
name: Rust Vector Build and Unit Tests
runs-on: ubuntu-24.04
timeout-minutes: 45
steps:
- name: Checkout code
uses: actions/checkout@v7
with:
persist-credentials: false
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
- name: Fingerprint build toolchain
id: toolchain
run: echo "fingerprint=$(getconf GNU_LIBC_VERSION | tr ' ' '-')-rustc-$(rustc -V | awk '{print $2}')" >> "$GITHUB_OUTPUT"
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
~/.cargo/registry
~/.cargo/git
seaweed-vector/target/release
key: rust-vector-${{ steps.toolchain.outputs.fingerprint }}-${{ hashFiles('seaweed-vector/Cargo.lock') }}
restore-keys: |
rust-vector-${{ steps.toolchain.outputs.fingerprint }}-
- name: Build the vector crates
run: cd seaweed-vector && cargo build --release
- name: Run unit tests
run: cd seaweed-vector && cargo test --release --workspace
vectors-e2e:
name: S3 Vectors Integration (${{ matrix.volume }} volume${{ matrix.hardened && ', hardened' || '' }}${{ matrix.worker && ', worker' || '' }})
runs-on: ubuntu-24.04
timeout-minutes: 45
strategy:
fail-fast: false
matrix:
volume: [go, rust]
# Hardened: volume servers behind mutual TLS, every gRPC hop (master,
# volume, filer, gateway, engine) under mutual TLS, read JWTs
# required, and every chunk encrypted by the gateway.
hardened: ["", "1"]
worker: ["", "1"]
# Worker: the admin server and the Lance maintenance worker join the
# cluster, for the tests that build vector indexes through a job.
env:
CARGO_PROFILE_DEV_DEBUG: "0"
CARGO_BUILD_JOBS: "2"
CARGO_TARGET_DIR: ${{ github.workspace }}/seaweed-vector/target
VOLUME_SERVER_IMPL: ${{ matrix.volume }}
VECTOR_QUERY_MODE: single
VECTOR_FILER_GROUP: vectors-ci
VECTOR_TLS: ${{ matrix.hardened }}
VECTOR_READ_JWT: ${{ matrix.hardened }}
VECTOR_ENCRYPT_VOLUMES: ${{ matrix.hardened }}
VECTOR_WORKER: ${{ matrix.worker }}
steps:
- uses: actions/checkout@v7
with:
persist-credentials: false
- uses: actions/setup-go@v7
with:
go-version-file: go.mod
cache: ${{ runner.environment == 'github-hosted' }}
- uses: dtolnay/rust-toolchain@stable
- name: Fingerprint build toolchain
id: toolchain
run: echo "fingerprint=$(getconf GNU_LIBC_VERSION | tr ' ' '-')-rustc-$(rustc -V | awk '{print $2}')" >> "$GITHUB_OUTPUT"
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
~/.cargo/registry
~/.cargo/git
seaweed-vector/target/debug
key: vectors-e2e-${{ runner.os }}-${{ runner.arch }}-${{ steps.toolchain.outputs.fingerprint }}-${{ hashFiles('seaweed-vector/Cargo.lock', 'seaweed-volume/Cargo.lock') }}
restore-keys: |
vectors-e2e-${{ runner.os }}-${{ runner.arch }}-${{ steps.toolchain.outputs.fingerprint }}-
- name: Test Go vector gateway
run: |
go test ./weed/s3api/s3vectors -count=1
go test ./weed/command -run 'Vector|Vectors' -count=1
# The worker workspace has no protoc of its own for lance's build
# scripts; use the vendored one its lock names, as its own workflow does.
- name: Use the vendored protoc for the worker
if: matrix.worker == '1'
run: |
cd seaweed-worker
cargo fetch --locked
version=$(awk '/^name = "protoc-bin-vendored-linux-x86_64"$/{found=1; next} found && /^version = /{gsub(/"/,"",$3); print $3; exit}' Cargo.lock)
test -n "$version" || { echo "protoc-bin-vendored-linux-x86_64 is not in Cargo.lock" >&2; exit 1; }
protoc=$(find ~/.cargo/registry/src -path "*protoc-bin-vendored-linux-x86_64-$version/bin/protoc" | head -1)
test -x "$protoc" || { echo "no vendored protoc $version in the registry" >&2; exit 1; }
echo "PROTOC=$protoc" >> "$GITHUB_ENV"
- name: Build integration binaries
working-directory: test/s3vectors
# dev profile: correctness suite, no release codegen needed. weed-volume
# (the heaviest crate) only builds on the rust leg.
run: |
set -e
make ./weed ./weed-vector CARGO_PROFILE=dev
if [ "${{ matrix.volume }}" = "rust" ]; then make ./weed-volume CARGO_PROFILE=dev; fi
if [ "${{ matrix.worker }}" = "1" ]; then make ./weed-vector-cli ./weed-worker CARGO_PROFILE=dev; fi
- uses: actions/setup-python@v7
with:
python-version: "3.12"
- name: Install test dependencies
run: python -m pip install -r test/s3vectors/requirements.txt
- name: Run the S3 Vectors integration matrix
working-directory: test/s3vectors
run: |
./run_test.sh --basetemp="$RUNNER_TEMP/vector-tests" --junitxml="$RUNNER_TEMP/vector-results.xml"
python - <<'PYTHON'
import os
from pathlib import Path
import xml.etree.ElementTree as ET
cases = ET.parse(Path(os.environ["RUNNER_TEMP"]) / "vector-results.xml").findall(".//testcase")
assert len(cases) >= 17, f"expected at least 17 cases, got {len(cases)}"
assert not any(case.find("skipped") is not None for case in cases), "integration cases were skipped"
volume = os.environ["VOLUME_SERVER_IMPL"]
assert all(case.get("name", "").endswith(f"[{volume}-single]") for case in cases), volume
PYTHON
- name: Upload integration diagnostics
if: always()
uses: actions/upload-artifact@v7
with:
name: vectors-e2e-diagnostics-${{ matrix.volume }}${{ matrix.hardened && '-hardened' || '' }}${{ matrix.worker && '-worker' || '' }}
path: |
${{ runner.temp }}/vector-tests
${{ runner.temp }}/vector-results.xml
if-no-files-found: ignore
retention-days: 7
+63 -85
View File
@@ -21,8 +21,8 @@ on:
- '.github/workflows/rust-volume-server-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
group: ${{ github.workflow }}-${{ github.head_ref || github.ref }}
cancel-in-progress: true
permissions:
contents: read
@@ -52,68 +52,9 @@ jobs:
- 'seaweed-volume/**'
- '.github/workflows/rust-volume-server-tests.yml'
build-test-binaries:
name: Build Test Binaries
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
- name: Checkout code
uses: actions/checkout@v7
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
# cargo tracks its own inputs but not the runner's C toolchain, so a cached
# target/ can carry C objects built against a different glibc than we link against.
- name: Fingerprint build toolchain
id: toolchain
run: echo "fingerprint=$(getconf GNU_LIBC_VERSION | tr ' ' '-')-rustc-$(rustc -V | awk '{print $2}')" >> "$GITHUB_OUTPUT"
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
~/.cargo/registry
~/.cargo/git
seaweed-volume/target
key: rust-${{ steps.toolchain.outputs.fingerprint }}-${{ hashFiles('seaweed-volume/Cargo.lock', 'seaweed-common/Cargo.lock') }}
restore-keys: |
rust-${{ steps.toolchain.outputs.fingerprint }}-
- name: Build Go weed binary
run: |
cd weed
go build -tags 5BytesOffset -o weed .
chmod +x weed
./weed version
- name: Build Rust volume binary
run: cd seaweed-volume && cargo build --release
- name: Package binaries
run: tar cf rust-test-binaries.tar weed/weed seaweed-volume/target/release/weed-volume
- name: Upload test binaries
uses: actions/upload-artifact@v7
with:
# archive layout mirrors the workspace: weed/weed and
# seaweed-volume/target/release/weed-volume
name: rust-test-binaries
if-no-files-found: error
retention-days: 1
path: rust-test-binaries.tar
rust-unit-tests:
name: Rust Unit Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 15
steps:
@@ -130,7 +71,6 @@ jobs:
run: echo "fingerprint=$(getconf GNU_LIBC_VERSION | tr ' ' '-')-rustc-$(rustc -V | awk '{print $2}')" >> "$GITHUB_OUTPUT"
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
@@ -168,6 +108,9 @@ jobs:
- name: Run Rust unit tests
run: cd seaweed-volume && cargo test
- name: Run Rust unit tests (redb experimental cursor)
run: cd seaweed-volume && cargo test --features redb-experimental-cursor --lib storage::needle_map
rust-unit-tests-windows:
name: Rust Unit Tests (Windows)
runs-on: windows-latest
@@ -191,7 +134,6 @@ jobs:
run: echo "fingerprint=windows-rustc-$(rustc -V | awk '{print $2}')" >> "$GITHUB_OUTPUT"
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
@@ -205,11 +147,13 @@ jobs:
- name: Run Rust unit tests
run: cd seaweed-volume && cargo test
- name: Run Rust unit tests (redb experimental cursor)
run: cd seaweed-volume && cargo test --features redb-experimental-cursor --lib storage::needle_map
rust-integration-tests:
name: Rust Integration Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 30
needs: [build-test-binaries]
steps:
- name: Checkout code
@@ -219,18 +163,36 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Download test binaries
uses: actions/download-artifact@v8
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
# cargo tracks its own inputs but not the runner's C toolchain, so a cached
# target/ can carry C objects built against a different glibc than we link against.
- name: Fingerprint build toolchain
id: toolchain
run: echo "fingerprint=$(getconf GNU_LIBC_VERSION | tr ' ' '-')-rustc-$(rustc -V | awk '{print $2}')" >> "$GITHUB_OUTPUT"
- name: Cache cargo registry and target
uses: actions/cache@v6
with:
name: rust-test-binaries
path: |
~/.cargo/registry
~/.cargo/git
seaweed-volume/target
key: rust-${{ steps.toolchain.outputs.fingerprint }}-${{ hashFiles('seaweed-volume/Cargo.lock', 'seaweed-common/Cargo.lock') }}
restore-keys: |
rust-${{ steps.toolchain.outputs.fingerprint }}-
- name: Restore binaries
- name: Build Go weed binary
run: |
tar xf rust-test-binaries.tar
chmod +x weed/weed seaweed-volume/target/release/weed-volume
./weed/weed version
cd weed
go build -tags 5BytesOffset -o weed .
chmod +x weed
./weed version
- name: Build Rust volume binary
run: cd seaweed-volume && cargo build --release
- name: Run integration tests
env:
@@ -264,9 +226,8 @@ jobs:
rust-volume-go-tests:
name: Go Tests with Rust Volume (${{ matrix.test-type }} - Shard ${{ matrix.shard }})
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 45
needs: [build-test-binaries]
env:
# Keep in step with the length of matrix.shard below.
SHARD_COUNT: 3
@@ -284,19 +245,36 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Download test binaries
uses: actions/download-artifact@v8
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
# cargo tracks its own inputs but not the runner's C toolchain, so a cached
# target/ can carry C objects built against a different glibc than we link against.
- name: Fingerprint build toolchain
id: toolchain
run: echo "fingerprint=$(getconf GNU_LIBC_VERSION | tr ' ' '-')-rustc-$(rustc -V | awk '{print $2}')" >> "$GITHUB_OUTPUT"
- name: Cache cargo registry and target
uses: actions/cache@v6
with:
name: rust-test-binaries
path: |
~/.cargo/registry
~/.cargo/git
seaweed-volume/target
key: rust-${{ steps.toolchain.outputs.fingerprint }}-${{ hashFiles('seaweed-volume/Cargo.lock', 'seaweed-common/Cargo.lock') }}
restore-keys: |
rust-${{ steps.toolchain.outputs.fingerprint }}-
- name: Restore binaries
- name: Build Go weed binary
run: |
tar xf rust-test-binaries.tar
chmod +x weed/weed seaweed-volume/target/release/weed-volume
./weed/weed version
./seaweed-volume/target/release/weed-volume --help >/dev/null || true
cd weed
go build -tags 5BytesOffset -o weed .
chmod +x weed
./weed version
- name: Build Rust volume binary
run: cd seaweed-volume && cargo build --release
# Dealing the listed tests out one by one keeps the shards even. Bucketing
# them by first letter did not: names cluster, so ^Test[I-S] drew 50 of
+3 -4
View File
@@ -17,8 +17,8 @@ on:
- '.github/workflows/rust-worker-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
group: ${{ github.workflow }}-${{ github.head_ref || github.ref }}
cancel-in-progress: true
permissions:
contents: read
@@ -26,7 +26,7 @@ permissions:
jobs:
rust-worker-build:
name: Rust Plugin Worker Build and Unit Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 45
steps:
@@ -45,7 +45,6 @@ jobs:
run: echo "fingerprint=$(getconf GNU_LIBC_VERSION | tr ' ' '-')-rustc-$(rustc -V | awk '{print $2}')" >> "$GITHUB_OUTPUT"
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
+1 -3
View File
@@ -32,7 +32,7 @@ jobs:
permissions:
contents: write
needs: cleanup
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
strategy:
matrix:
include:
@@ -46,7 +46,6 @@ jobs:
uses: dtolnay/rust-toolchain@stable
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
@@ -115,7 +114,6 @@ jobs:
targets: ${{ matrix.target }}
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
+39 -97
View File
@@ -15,27 +15,39 @@ jobs:
build-rust-volume-linux:
permissions:
contents: write
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
strategy:
matrix:
include:
# Static musl, as the container images ship: a glibc build only
# starts on distributions at least as new as the runner's.
- target: x86_64-unknown-linux-musl
- target: x86_64-unknown-linux-gnu
asset_suffix: linux_amd64
- target: aarch64-unknown-linux-musl
- target: aarch64-unknown-linux-gnu
asset_suffix: linux_arm64
cross: true
steps:
- uses: actions/checkout@v7
- name: Set up the musl target
uses: ./.github/actions/rust-musl-target
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
with:
target: ${{ matrix.target }}
targets: ${{ matrix.target }}
- name: Install cross-compilation tools
if: matrix.cross
run: |
sudo dpkg --add-architecture arm64
sudo sed -i 's/^deb /deb [arch=amd64] /' /etc/apt/sources.list
echo "deb [arch=arm64] http://ports.ubuntu.com/ jammy main restricted universe multiverse" | sudo tee /etc/apt/sources.list.d/arm64.list
echo "deb [arch=arm64] http://ports.ubuntu.com/ jammy-updates main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list.d/arm64.list
sudo apt-get update
sudo apt-get install -y gcc-aarch64-linux-gnu libssl-dev:arm64
echo "CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER=aarch64-linux-gnu-gcc" >> "$GITHUB_ENV"
echo "OPENSSL_DIR=/usr" >> "$GITHUB_ENV"
echo "OPENSSL_INCLUDE_DIR=/usr/include" >> "$GITHUB_ENV"
echo "OPENSSL_LIB_DIR=/usr/lib/aarch64-linux-gnu" >> "$GITHUB_ENV"
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
@@ -107,16 +119,15 @@ jobs:
build-rust-worker-linux:
permissions:
contents: write
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
strategy:
matrix:
include:
# Static musl, as the container images ship: a glibc build only
# starts on distributions at least as new as the runner's.
- target: x86_64-unknown-linux-musl
- target: x86_64-unknown-linux-gnu
asset_suffix: linux_amd64
- target: aarch64-unknown-linux-musl
- target: aarch64-unknown-linux-gnu
asset_suffix: linux_arm64
cross: true
steps:
- uses: actions/checkout@v7
@@ -125,13 +136,23 @@ jobs:
# should not find another one sitting in the checkout's git config.
persist-credentials: false
- name: Set up the musl target
uses: ./.github/actions/rust-musl-target
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
with:
target: ${{ matrix.target }}
targets: ${{ matrix.target }}
- name: Install cross-compilation tools
if: matrix.cross
run: |
sudo dpkg --add-architecture arm64
sudo sed -i 's/^deb /deb [arch=amd64] /' /etc/apt/sources.list
echo "deb [arch=arm64] http://ports.ubuntu.com/ jammy main restricted universe multiverse" | sudo tee /etc/apt/sources.list.d/arm64.list
echo "deb [arch=arm64] http://ports.ubuntu.com/ jammy-updates main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list.d/arm64.list
sudo apt-get update
sudo apt-get install -y gcc-aarch64-linux-gnu
echo "CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER=aarch64-linux-gnu-gcc" >> "$GITHUB_ENV"
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
@@ -188,83 +209,6 @@ jobs:
weed-worker_${{ matrix.asset_suffix }}.tar.gz
weed-worker_${{ matrix.asset_suffix }}.tar.gz.md5
# The vector engine behind the S3 Vectors API: Linux only, like the worker,
# whose lance dependency tree it shares.
build-rust-vector-linux:
permissions:
contents: write
runs-on: ubuntu-24.04
strategy:
matrix:
include:
- target: x86_64-unknown-linux-musl
asset_suffix: linux_amd64
- target: aarch64-unknown-linux-musl
asset_suffix: linux_arm64
steps:
- uses: actions/checkout@v7
with:
persist-credentials: false
- name: Set up the musl target
uses: ./.github/actions/rust-musl-target
with:
target: ${{ matrix.target }}
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
~/.cargo/registry
~/.cargo/git
seaweed-vector/target/${{ matrix.target }}/release
key: rust-vector-release-${{ matrix.target }}-${{ hashFiles('seaweed-vector/Cargo.lock') }}
restore-keys: |
rust-vector-release-${{ matrix.target }}-
- name: Use the vendored protoc
run: |
cd seaweed-vector
cargo fetch
version=$(awk '/^name = "protoc-bin-vendored-linux-x86_64"$/{found=1; next} found && /^version = /{gsub(/"/,"",$3); print $3; exit}' Cargo.lock)
test -n "$version" || { echo "protoc-bin-vendored-linux-x86_64 is not in Cargo.lock" >&2; exit 1; }
protoc=$(find ~/.cargo/registry/src -path "*protoc-bin-vendored-linux-x86_64-$version/bin/protoc" | head -1)
test -x "$protoc" || { echo "no vendored protoc $version in the registry" >&2; exit 1; }
echo "PROTOC=$protoc" >> "$GITHUB_ENV"
- name: Build the vector engine
run: |
cd seaweed-vector
cargo build --release -p weed-vector-server --target ${{ matrix.target }}
- name: Package binary
run: |
cp seaweed-vector/target/${{ matrix.target }}/release/weed-vector weed-vector
tar czf weed-vector_${{ matrix.asset_suffix }}.tar.gz weed-vector
rm weed-vector
md5sum weed-vector_${{ matrix.asset_suffix }}.tar.gz > weed-vector_${{ matrix.asset_suffix }}.tar.gz.md5
- name: Upload release assets
if: startsWith(github.ref, 'refs/tags/')
uses: softprops/action-gh-release@v3
with:
files: |
weed-vector_${{ matrix.asset_suffix }}.tar.gz
weed-vector_${{ matrix.asset_suffix }}.tar.gz.md5
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Upload artifacts
if: ${{ !startsWith(github.ref, 'refs/tags/') }}
uses: actions/upload-artifact@v7
with:
name: rust-vector-${{ matrix.asset_suffix }}
path: |
weed-vector_${{ matrix.asset_suffix }}.tar.gz
weed-vector_${{ matrix.asset_suffix }}.tar.gz.md5
build-rust-volume-darwin:
permissions:
contents: write
@@ -286,7 +230,6 @@ jobs:
targets: ${{ matrix.target }}
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
@@ -362,7 +305,6 @@ jobs:
uses: dtolnay/rust-toolchain@stable
- name: Cache cargo registry and target
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: |
+3 -4
View File
@@ -21,8 +21,8 @@ on:
- '.github/workflows/s3-etag-acl-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
group: ${{ github.head_ref || github.ref }}/s3-etag-acl-tests
cancel-in-progress: true
permissions:
contents: read
@@ -30,7 +30,7 @@ permissions:
jobs:
s3-etag-acl-tests:
name: S3 ETag + ACL Integration Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 15
steps:
- name: Check out code
@@ -40,7 +40,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Install SeaweedFS
run: |
@@ -13,8 +13,8 @@ on:
- '.github/workflows/s3-example-integration-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
group: ${{ github.head_ref }}/s3-integration-tests
cancel-in-progress: true
permissions:
contents: read
@@ -22,7 +22,7 @@ permissions:
jobs:
tests:
name: S3 Integration Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 30
steps:
@@ -33,7 +33,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Build SeaweedFS
+4 -8
View File
@@ -12,8 +12,8 @@ on:
- '.github/workflows/s3-filer-group-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
group: ${{ github.head_ref }}/s3-filer-group-tests
cancel-in-progress: true
permissions:
contents: read
@@ -25,21 +25,17 @@ defaults:
jobs:
s3-filer-group-tests:
name: S3 Filer Group Integration Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 20
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -54,7 +50,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
timeout 15 df -h || true
df -h
echo "=== Starting Filer Group Tests ==="
# Run tests with automatic server management
+17 -42
View File
@@ -4,7 +4,6 @@ on:
pull_request:
paths:
- 'weed/s3api/**'
- 'weed/remote_storage/**'
- 'weed/filer/**'
- 'weed/server/**'
- 'test/s3/**'
@@ -13,8 +12,8 @@ on:
- '.github/workflows/s3-go-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
group: ${{ github.head_ref }}/s3-go-tests
cancel-in-progress: true
permissions:
contents: read
@@ -26,7 +25,7 @@ defaults:
jobs:
s3-versioning-tests:
name: S3 Versioning Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 30
strategy:
matrix:
@@ -36,14 +35,10 @@ jobs:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -58,7 +53,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
timeout 15 df -h || true
df -h
echo "=== Starting Tests ==="
# Run tests with automatic server management
@@ -97,7 +92,7 @@ jobs:
s3-versioning-compatibility:
name: S3 Versioning Compatibility Test
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 20
steps:
@@ -108,7 +103,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -146,7 +140,7 @@ jobs:
s3-cors-compatibility:
name: S3 CORS Compatibility Test
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 20
steps:
@@ -157,7 +151,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -195,7 +188,7 @@ jobs:
s3-retention-tests:
name: S3 Retention Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 30
strategy:
matrix:
@@ -205,14 +198,10 @@ jobs:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -227,7 +216,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
timeout 15 df -h || true
df -h
echo "=== Starting Tests ==="
# Run tests with automatic server management
@@ -266,7 +255,7 @@ jobs:
s3-lifecycle-tests:
name: S3 Lifecycle Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 10
strategy:
fail-fast: false
@@ -294,7 +283,6 @@ jobs:
- TestLifecycleSuspendedVersioningExpiration
- TestLifecycleTagFilter
- TestLifecycleVersionedBucketCreatesDeleteMarker
- TestLifecycleVersionedBucketExpiresOnSecondRun
steps:
- name: Check out code
@@ -304,7 +292,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -339,21 +326,17 @@ jobs:
s3-checksum-tests:
name: S3 Checksum Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 20
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -368,7 +351,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
timeout 15 df -h || true
df -h
echo "=== Starting Tests ==="
make test-with-server
@@ -398,7 +381,7 @@ jobs:
s3-cors-tests:
name: S3 CORS Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 30
strategy:
matrix:
@@ -408,14 +391,10 @@ jobs:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -430,7 +409,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
timeout 15 df -h || true
df -h
echo "=== Starting Tests ==="
# Run tests with automatic server management
@@ -469,7 +448,7 @@ jobs:
s3-retention-worm:
name: S3 Retention WORM Integration Test
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 20
steps:
@@ -480,7 +459,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -519,7 +497,7 @@ jobs:
s3-versioning-stress:
name: S3 Versioning Stress Test
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 35
steps:
@@ -530,7 +508,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -588,7 +565,7 @@ jobs:
s3-tagging-tests:
# CI job for S3 object tagging tests
name: S3 Tagging Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 20
steps:
@@ -599,7 +576,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -644,7 +620,7 @@ jobs:
s3-remote-cache-tests:
name: S3 Remote Cache Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 20
steps:
@@ -655,7 +631,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
+7 -14
View File
@@ -20,8 +20,8 @@ on:
- '.github/workflows/s3-iam-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
group: ${{ github.head_ref }}/s3-iam-tests
cancel-in-progress: true
permissions:
contents: read
@@ -34,7 +34,7 @@ jobs:
# Unit tests for IAM components
iam-unit-tests:
name: IAM Unit Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 15
steps:
@@ -45,7 +45,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Get dependencies
@@ -81,7 +80,7 @@ jobs:
# S3 IAM integration tests with SeaweedFS services
s3-iam-integration-tests:
name: S3 IAM Integration Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 25
strategy:
matrix:
@@ -91,14 +90,10 @@ jobs:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -114,7 +109,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
timeout 15 df -h || true
df -h
echo "=== Starting S3 IAM Integration Tests (${{ matrix.test-type }}) ==="
# Set WEED_BINARY to use the installed version
@@ -202,7 +197,7 @@ jobs:
# Distributed IAM tests
s3-iam-distributed-tests:
name: S3 IAM Distributed Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 25
steps:
@@ -213,7 +208,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -263,7 +257,7 @@ jobs:
# Performance and stress tests
s3-iam-performance-tests:
name: S3 IAM Performance Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 30
steps:
@@ -274,7 +268,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
+4 -8
View File
@@ -16,8 +16,8 @@ on:
- '.github/workflows/s3-keycloak-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
group: ${{ github.head_ref }}/s3-keycloak-tests
cancel-in-progress: true
permissions:
contents: read
@@ -30,21 +30,17 @@ jobs:
# Dedicated job for Keycloak integration tests
s3-keycloak-integration-tests:
name: S3 Keycloak Integration Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 30
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -60,7 +56,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
timeout 15 df -h || true
df -h
echo "=== Starting S3 Keycloak Integration Tests ==="
# Set WEED_BINARY to use the installed version
@@ -1,74 +0,0 @@
name: "S3 listing study tests"
on:
push:
branches: [ master ]
paths:
- 'weed/s3api/**'
- 'weed/filer/**'
- 'weed/server/**'
- 'test/s3/listing_study/**'
- '.github/workflows/s3-listing-study-tests.yml'
pull_request:
branches: [ master ]
paths:
- 'weed/s3api/**'
- 'weed/filer/**'
- 'weed/server/**'
- 'test/s3/listing_study/**'
- '.github/workflows/s3-listing-study-tests.yml'
concurrency:
group: ${{ github.event.pull_request.number || github.ref }}/s3-listing-study-tests
cancel-in-progress: true
permissions:
contents: read
jobs:
listing-study:
name: s3-listing-study tools and strategies
runs-on: ubuntu-24.04
timeout-minutes: 40
defaults:
run:
working-directory: test/s3/listing_study
steps:
- name: Check out code
uses: actions/checkout@v7
with:
persist-credentials: false
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Cache pinned listing tools
uses: actions/cache@v6
with:
path: test/s3/listing_study/.work/tools
key: listing-study-tools-${{ runner.os }}-${{ hashFiles('test/s3/listing_study/Makefile', 'test/s3/listing_study/install_tools.py') }}
- name: Build weed and install the study's pinned tools
run: make build-weed tools
- name: Run the listing study against SeaweedFS
timeout-minutes: 15
run: make test
- name: Show logs on failure
if: failure()
run: |
tail -200 .work/weed.log || true
- name: Upload results
if: always()
uses: actions/upload-artifact@v7
with:
name: s3-listing-study-results
path: |
test/s3/listing_study/.work/run/results.json
test/s3/listing_study/.work/weed.log
retention-days: 14
@@ -6,15 +6,14 @@ on:
- 'weed/s3api/**'
- 'test/s3/delete/**'
- 'test/s3/distributed_lock/**'
- 'test/s3/sqlite_cbs/**'
- 'test/s3/versioning/**'
- 'test/volume_server/framework/**'
- 'docker/compose/s3.json'
- '.github/workflows/s3-mutation-regression-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
group: ${{ github.head_ref }}/s3-mutation-regression-tests
cancel-in-progress: true
permissions:
contents: read
@@ -22,7 +21,7 @@ permissions:
jobs:
s3-versioning-regressions:
name: S3 Versioning Regression Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 25
steps:
@@ -33,7 +32,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Run S3 versioning regression tests
timeout-minutes: 20
@@ -76,7 +74,7 @@ jobs:
s3-delete-regressions:
name: S3 Delete Regression Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 20
steps:
@@ -87,7 +85,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Run S3 delete regression tests
timeout-minutes: 15
@@ -115,7 +112,7 @@ jobs:
s3-distributed-lock-regressions:
name: S3 Distributed Lock Regression Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 30
steps:
@@ -126,7 +123,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Build SeaweedFS
run: |
@@ -150,41 +146,3 @@ jobs:
name: s3-distributed-lock-regression-logs
path: test/s3/distributed_lock/tmp/seaweedfs_s3_distributed_lock_*
retention-days: 3
s3-sqlite-cbs-regressions:
name: S3 SQLite CBS Regression Tests
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Build SeaweedFS
run: |
go build -o weed/weed -buildvcs=false ./weed
- name: Run sqlite cbs regressions
timeout-minutes: 15
env:
TMPDIR: ${{ github.workspace }}/test/s3/sqlite_cbs/tmp
S3_SQLITE_CBS_KEEP_LOGS: "1"
WEED_BINARY: ${{ github.workspace }}/weed/weed
run: |
set -x
mkdir -p "$TMPDIR"
go test -v -count=1 -timeout=12m ./test/s3/sqlite_cbs
- name: Upload sqlite cbs logs on failure
if: failure()
uses: actions/upload-artifact@v7
with:
name: s3-sqlite-cbs-regression-logs
path: test/s3/sqlite_cbs/tmp/seaweedfs_sqlite_cbs_*
retention-days: 3
+4 -8
View File
@@ -23,10 +23,6 @@ env:
S3_ENDPOINT_URL: http://localhost:8333
BUCKET_NAME: test-parquet-bucket
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
jobs:
parquet-integration-tests:
name: PyArrow Parquet Tests (Python ${{ matrix.python-version }})
@@ -36,7 +32,7 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: ['3.10', '3.11', '3.12']
python-version: ['3.9', '3.11', '3.12']
steps:
- name: Checkout code
@@ -46,13 +42,13 @@ jobs:
uses: actions/setup-go@v7
with:
go-version: ^1.26
cache: ${{ runner.environment == 'github-hosted' }}
cache: true
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v7
with:
python-version: ${{ matrix.python-version }}
cache: ${{ runner.environment == 'github-hosted' && 'pip' || '' }}
cache: 'pip'
cache-dependency-path: 'test/s3/parquet/requirements.txt'
- name: Install system dependencies
@@ -154,7 +150,7 @@ jobs:
uses: actions/setup-go@v7
with:
go-version: ^1.26
cache: ${{ runner.environment == 'github-hosted' }}
cache: true
- name: Run Go unit tests
run: |
+9 -20
View File
@@ -24,8 +24,8 @@ on:
- '.github/workflows/s3-policy-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
group: ${{ github.head_ref }}/s3-policy-tests
cancel-in-progress: true
permissions:
contents: read
@@ -38,7 +38,7 @@ jobs:
# Unit tests for policy components
policy-unit-tests:
name: S3 Policy Unit Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 15
steps:
@@ -49,7 +49,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Get dependencies
@@ -82,21 +81,17 @@ jobs:
# S3 Policy Variables Integration Tests
s3-policy-variables-tests:
name: S3 Policy Variables Integration Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 25
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -111,7 +106,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
timeout 15 df -h || true
df -h
echo "=== Starting S3 Policy Variables Integration Tests ==="
@@ -192,7 +187,7 @@ jobs:
# S3 Policy Enforcement Integration Tests
s3-policy-enforcement-tests:
name: S3 Policy Enforcement Integration Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 30
strategy:
matrix:
@@ -202,14 +197,10 @@ jobs:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -224,7 +215,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
timeout 15 df -h || true
df -h
echo "=== Starting S3 Policy Enforcement Tests (${{ matrix.test-case }}) ==="
@@ -317,7 +308,7 @@ jobs:
# Trusted Proxy Detection Tests
trusted-proxy-tests:
name: Trusted Proxy Detection Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 20
steps:
@@ -328,7 +319,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -405,7 +395,7 @@ jobs:
# S3 Policy Shell Integration Tests
s3-policy-shell-tests:
name: S3 Policy Shell Integration Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 15
steps:
@@ -416,7 +406,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -21,8 +21,8 @@ on:
- '.github/workflows/s3-proxy-signature-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
group: ${{ github.head_ref || github.ref }}/s3-proxy-signature-tests
cancel-in-progress: true
permissions:
contents: read
@@ -30,7 +30,7 @@ permissions:
jobs:
proxy-signature-tests:
name: S3 Proxy Signature Verification Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 15
steps:
- name: Check out code into the Go module directory
@@ -40,7 +40,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -15,8 +15,8 @@ on:
- '.github/workflows/s3-sdk-v2-routing-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
group: ${{ github.head_ref || github.ref }}/s3-sdk-v2-routing-tests
cancel-in-progress: true
permissions:
contents: read
@@ -24,7 +24,7 @@ permissions:
jobs:
s3-sdk-v2-routing-tests:
name: S3 SDK V2 Routing Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 10
steps:
- name: Check out code
@@ -34,7 +34,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Install SeaweedFS
run: |
+2 -13
View File
@@ -46,7 +46,7 @@ permissions:
jobs:
snowflake-s3compat-tests:
name: Snowflake S3Compat API tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 30
env:
WORK_DIR: /tmp/seaweedfs-snowflake-tests
@@ -60,7 +60,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Set up Java
@@ -68,15 +67,7 @@ jobs:
with:
java-version: '17'
distribution: 'temurin'
- name: Cache suite Maven dependencies
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: ~/.m2/repository
key: snowflake-s3compat-m2-${{ hashFiles('test/s3/snowflake/run.sh') }}
restore-keys: |
snowflake-s3compat-m2-
cache: 'maven'
- name: Install SeaweedFS
run: |
@@ -86,8 +77,6 @@ jobs:
- name: Run Snowflake S3Compat API tests
timeout-minutes: 20
env:
MAVEN_ARGS: -ntp -s ${{ github.workspace }}/test/s3/snowflake/maven-settings.xml
run: |
# Starts weed server, creates the buckets/objects the suite needs,
# clones the upstream suite, and runs mvn -Dtest=S3CompatApiTest.
+4 -8
View File
@@ -11,8 +11,8 @@ on:
workflow_dispatch:
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
group: ${{ github.head_ref }}/s3-spark-tests
cancel-in-progress: true
permissions:
contents: read
@@ -20,21 +20,17 @@ permissions:
jobs:
s3-spark-issue-repro-tests:
name: S3 Spark Issue Reproduction Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 45
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -60,7 +56,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
timeout 15 df -h || true
df -h
echo "=== Starting S3 Spark Integration Tests ==="
go test -v -timeout 30m . 2>&1 | tee test-output.log || {
+10 -20
View File
@@ -23,8 +23,8 @@ on:
- 'test/s3/sse/**'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
group: ${{ github.head_ref }}/s3-sse-tests
cancel-in-progress: true
permissions:
contents: read
@@ -36,7 +36,7 @@ defaults:
jobs:
s3-sse-integration-tests:
name: S3 SSE Integration Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 30
strategy:
matrix:
@@ -46,14 +46,10 @@ jobs:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -68,7 +64,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
timeout 15 df -h || true
df -h
echo "=== Starting SSE Tests ==="
# Run tests with automatic server management
@@ -111,7 +107,7 @@ jobs:
s3-sse-compatibility:
name: S3 SSE Compatibility Test
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 20
steps:
@@ -122,7 +118,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -160,7 +155,7 @@ jobs:
s3-sse-metadata-persistence:
name: S3 SSE Metadata Persistence Test
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 20
steps:
@@ -171,7 +166,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -210,7 +204,7 @@ jobs:
s3-sse-copy-operations:
name: S3 SSE Copy Operations Test
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 25
steps:
@@ -221,7 +215,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -260,7 +253,7 @@ jobs:
s3-sse-multipart:
name: S3 SSE Multipart Upload Test
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 25
steps:
@@ -271,7 +264,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -310,7 +302,7 @@ jobs:
s3-sse-performance:
name: S3 SSE Performance Test
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 35
# Only run performance tests on master branch pushes to avoid overloading PR testing
if: github.event_name == 'push' && (github.ref == 'refs/heads/master' || github.ref == 'refs/heads/main')
@@ -323,7 +315,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -361,7 +352,7 @@ jobs:
s3-volume-encryption:
name: S3 Volume Encryption Test
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 20
steps:
@@ -372,7 +363,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
+36 -116
View File
@@ -12,10 +12,6 @@ on:
- 'go.sum'
- '.github/workflows/s3-tables-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -23,21 +19,17 @@ permissions:
jobs:
s3-tables-tests:
name: S3 Tables Integration Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 30
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Run go mod tidy
@@ -56,8 +48,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
# A stale FUSE mount can hang or fail df; keep diagnostics best effort.
timeout 15 df -h || true
df -h
echo "=== Starting S3 Tables Tests ==="
# Run S3 Tables integration tests
@@ -88,21 +79,17 @@ jobs:
iceberg-catalog-tests:
name: Iceberg Catalog Integration Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 30
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -122,7 +109,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
timeout 15 df -h || true
df -h
echo "=== Starting Iceberg Catalog Tests ==="
# Run Iceberg catalog integration tests using Makefile (handles build)
@@ -153,21 +140,17 @@ jobs:
trino-iceberg-catalog-tests:
name: Trino Iceberg Catalog Integration Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 30
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -196,7 +179,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
timeout 15 df -h || true
df -h
echo "=== Starting Trino Iceberg Catalog Tests ==="
# Run Trino + Iceberg catalog integration tests
@@ -227,21 +210,17 @@ jobs:
dremio-iceberg-catalog-tests:
name: Dremio Iceberg Catalog Integration Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 30
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -271,7 +250,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
timeout 15 df -h || true
df -h
docker info
echo "=== Starting Dremio Iceberg Catalog Tests ==="
@@ -304,21 +283,17 @@ jobs:
doris-iceberg-catalog-tests:
name: Doris Iceberg Catalog Integration Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 35
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -348,7 +323,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
timeout 15 df -h || true
df -h
docker info
echo "=== Starting Doris Iceberg Catalog Tests ==="
@@ -381,7 +356,7 @@ jobs:
clickhouse-iceberg-catalog-tests:
name: ClickHouse Iceberg Catalog Integration Tests (${{ matrix.tag }})
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 30
strategy:
fail-fast: false
@@ -398,14 +373,10 @@ jobs:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -437,7 +408,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
timeout 15 df -h || true
df -h
docker info
echo "=== Starting ClickHouse Iceberg Catalog Tests ==="
@@ -470,7 +441,7 @@ jobs:
olake-iceberg-catalog-tests:
name: OLake Iceberg Catalog Integration Tests (${{ matrix.tag }})
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 30
strategy:
fail-fast: false
@@ -489,14 +460,10 @@ jobs:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -529,7 +496,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
timeout 15 df -h || true
df -h
docker info
echo "=== Starting OLake Iceberg Catalog Tests ==="
@@ -585,21 +552,17 @@ jobs:
polaris-integration-tests:
name: Polaris Integration Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 30
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Run go mod tidy
@@ -627,7 +590,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
timeout 15 df -h || true
df -h
echo "=== Starting Polaris Tests ==="
go test -v -timeout 20m ./test/s3tables/polaris 2>&1 | tee test/s3tables/polaris/test-output.log || {
@@ -657,21 +620,17 @@ jobs:
spark-iceberg-catalog-tests:
name: Spark Iceberg Catalog Integration Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 30
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -684,12 +643,6 @@ jobs:
pull() { for i in 1 2 3; do docker pull "$1" && return 0; sleep 15; done; return 1; }
pull apache/spark:3.5.1
- name: Cache Iceberg jars
uses: actions/cache@v6
with:
path: ~/.cache/seaweedfs-test-jars
key: iceberg-jars-${{ hashFiles('test/s3tables/catalog_spark/setup_test.go') }}
- name: Run go mod tidy
run: go mod tidy
@@ -706,7 +659,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
timeout 15 df -h || true
df -h
echo "=== Starting Spark Iceberg Catalog Tests ==="
# Run Spark + Iceberg catalog integration tests
@@ -737,21 +690,17 @@ jobs:
risingwave-catalog-tests:
name: RisingWave Catalog Integration Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 30
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -781,7 +730,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
timeout 15 df -h || true
df -h
echo "=== Starting RisingWave Catalog Tests ==="
# Run RisingWave catalog integration tests
@@ -812,21 +761,17 @@ jobs:
sts-integration-tests:
name: STS Integration Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 30
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -840,7 +785,6 @@ jobs:
- name: Restore python:3 image cache
id: python-image
if: runner.environment == 'github-hosted'
uses: actions/cache@v6
with:
path: /tmp/python3-image.tar
@@ -880,7 +824,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
timeout 15 df -h || true
df -h
echo "=== Starting STS Integration Tests ==="
# Run STS integration tests
@@ -911,21 +855,17 @@ jobs:
lakekeeper-integration-tests:
name: Lakekeeper Integration Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 30
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Run go mod tidy
@@ -944,7 +884,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
timeout 15 df -h || true
df -h
echo "=== Starting Lakekeeper Integration Tests ==="
# Run Lakekeeper integration tests
@@ -975,21 +915,17 @@ jobs:
unity-catalog-integration-tests:
name: Unity Catalog Integration Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 30
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -1019,7 +955,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
timeout 15 df -h || true
df -h
docker info
echo "=== Starting Unity Catalog Tests ==="
@@ -1052,21 +988,17 @@ jobs:
lancedb-namespace-tests:
name: LanceDB Namespace Integration Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 30
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -1095,7 +1027,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
timeout 15 df -h || true
df -h
docker info
echo "=== Starting LanceDB Namespace Tests ==="
@@ -1126,7 +1058,7 @@ jobs:
table-lifecycle-tests:
name: Table Lifecycle Integration Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 40
steps:
@@ -1137,7 +1069,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -1195,7 +1126,7 @@ jobs:
duckdb-lance-tests:
name: DuckDB Lance Integration Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 30
steps:
@@ -1206,14 +1137,10 @@ jobs:
# so do not leave a token in the checkout for it to pick up.
persist-credentials: false
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -1243,7 +1170,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
timeout 15 df -h || true
df -h
docker info
echo "=== Starting DuckDB Lance Tests ==="
@@ -1274,7 +1201,7 @@ jobs:
spark-lance-namespace-tests:
name: Spark Lance Namespace Integration Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 40
steps:
@@ -1285,14 +1212,10 @@ jobs:
# so do not leave a token in the checkout for it to pick up.
persist-credentials: false
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Configure Docker Hub mirror
@@ -1321,7 +1244,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
timeout 15 df -h || true
df -h
docker info
echo "=== Starting Spark Lance Namespace Tests ==="
@@ -1352,7 +1275,7 @@ jobs:
s3-tables-build-verification:
name: S3 Tables Build Verification
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 15
steps:
@@ -1363,7 +1286,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Verify S3 Tables Package Builds
@@ -1418,7 +1340,7 @@ jobs:
s3-tables-fmt-check:
name: S3 Tables Format Check
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 10
steps:
@@ -1429,7 +1351,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Check Go Format
@@ -1470,7 +1391,7 @@ jobs:
s3-tables-vet:
name: S3 Tables Go Vet Check
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 10
steps:
@@ -1481,7 +1402,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Run Go Vet
@@ -1,57 +0,0 @@
name: "S3 Tigris Compatibility Tests"
on:
push:
branches: [ master ]
paths:
- 'weed/s3api/**'
- 'weed/server/filer_server_handlers_*.go'
- 'test/s3/tigris_compat/**'
- '.github/workflows/s3-tigris-compat-tests.yml'
pull_request:
branches: [ master ]
paths:
- 'weed/s3api/**'
- 'weed/server/filer_server_handlers_*.go'
- 'test/s3/tigris_compat/**'
- '.github/workflows/s3-tigris-compat-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
jobs:
s3-tigris-compat-tests:
name: s3-api-compat-tests (Tigris)
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Run s3-api-compat-tests
working-directory: test/s3/tigris_compat
run: make test
- name: Show server log on failure
if: failure()
working-directory: test/s3/tigris_compat
run: tail -200 weed-test.log || true
- name: Upload report
if: always()
uses: actions/upload-artifact@v7
with:
name: s3-tigris-compat-results
path: test/s3/tigris_compat/results.json
if-no-files-found: ignore
retention-days: 7
+29 -48
View File
@@ -25,8 +25,8 @@ on:
- '.github/workflows/s3tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
group: ${{ github.head_ref }}/s3tests
cancel-in-progress: true
permissions:
contents: read
@@ -34,7 +34,7 @@ permissions:
jobs:
basic-s3-tests:
name: Basic S3 tests (KV store)
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 15
steps:
- name: Check out code into the Go module directory
@@ -44,13 +44,12 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Set up Python
uses: actions/setup-python@v7
with:
python-version: '3.10'
python-version: '3.9'
- name: Clone s3-tests
run: |
@@ -58,9 +57,9 @@ jobs:
cd s3-tests
sudo apt-get update -qq
sudo apt-get install -y -qq libxml2-dev libxslt1-dev zlib1g-dev
python3 -m pip install -r requirements.txt
python3 -m pip install tox
python3 -m pip install -e .
pip install -r requirements.txt
pip install tox
pip install -e .
- name: Fix S3 tests bucket creation conflicts
run: |
@@ -380,21 +379,14 @@ jobs:
s3tests/functional/test_s3.py::test_copy_object_ifnonematch_good \
s3tests/functional/test_s3.py::test_lifecycle_set \
s3tests/functional/test_s3.py::test_lifecycle_get \
s3tests/functional/test_s3.py::test_lifecycle_set_filter
# The expiration tests race the lifecycle worker's meta-log replay;
# under the suite's write flood a shard can trail an object's PUT
# event past the test's fixed sleep window. One retry absorbs it.
lifecycle_expiration() {
tox -- s3tests/functional/test_s3.py::test_lifecycle_expiration \
s3tests/functional/test_s3.py::test_lifecyclev2_expiration
}
lifecycle_expiration || { echo "lifecycle expiration tests flaked; retry 1"; lifecycle_expiration; } || { echo "lifecycle expiration tests flaked; retry 2"; lifecycle_expiration; }
s3tests/functional/test_s3.py::test_lifecycle_set_filter \
s3tests/functional/test_s3.py::test_lifecycle_expiration \
s3tests/functional/test_s3.py::test_lifecyclev2_expiration
# cleanup() trap handles worker/server kill + data dir wipe.
versioning-tests:
name: S3 Versioning & Object Lock tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 15
steps:
- name: Check out code into the Go module directory
@@ -404,13 +396,12 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Set up Python
uses: actions/setup-python@v7
with:
python-version: '3.10'
python-version: '3.9'
- name: Clone s3-tests
run: |
@@ -418,9 +409,9 @@ jobs:
cd s3-tests
sudo apt-get update -qq
sudo apt-get install -y -qq libxml2-dev libxslt1-dev zlib1g-dev
python3 -m pip install -r requirements.txt
python3 -m pip install tox
python3 -m pip install -e .
pip install -r requirements.txt
pip install tox
pip install -e .
- name: Fix S3 tests bucket creation conflicts
run: |
@@ -565,7 +556,7 @@ jobs:
cors-tests:
name: S3 CORS tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 10
steps:
- name: Check out code into the Go module directory
@@ -575,13 +566,12 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Set up Python
uses: actions/setup-python@v7
with:
python-version: '3.10'
python-version: '3.9'
- name: Clone s3-tests
run: |
@@ -589,9 +579,9 @@ jobs:
cd s3-tests
sudo apt-get update -qq
sudo apt-get install -y -qq libxml2-dev libxslt1-dev zlib1g-dev
python3 -m pip install -r requirements.txt
python3 -m pip install tox
python3 -m pip install -e .
pip install -r requirements.txt
pip install tox
pip install -e .
- name: Run S3 CORS tests
timeout-minutes: 10
@@ -691,7 +681,7 @@ jobs:
copy-tests:
name: SeaweedFS Custom S3 Copy tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 10
steps:
- name: Check out code into the Go module directory
@@ -701,7 +691,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Run SeaweedFS Custom S3 Copy tests
@@ -794,7 +783,7 @@ jobs:
sql-store-tests:
name: Basic S3 tests (SQL store)
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 15
steps:
- name: Check out code into the Go module directory
@@ -804,13 +793,12 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Set up Python
uses: actions/setup-python@v7
with:
python-version: '3.10'
python-version: '3.9'
- name: Clone s3-tests
run: |
@@ -818,9 +806,9 @@ jobs:
cd s3-tests
sudo apt-get update -qq
sudo apt-get install -y -qq libxml2-dev libxslt1-dev zlib1g-dev
python3 -m pip install -r requirements.txt
python3 -m pip install tox
python3 -m pip install -e .
pip install -r requirements.txt
pip install tox
pip install -e .
- name: Run Ceph S3 tests with SQL store
timeout-minutes: 15
@@ -1257,16 +1245,9 @@ jobs:
s3tests/functional/test_s3.py::test_copy_object_ifnonematch_good \
s3tests/functional/test_s3.py::test_lifecycle_set \
s3tests/functional/test_s3.py::test_lifecycle_get \
s3tests/functional/test_s3.py::test_lifecycle_set_filter
# The expiration tests race the lifecycle worker's meta-log replay;
# under the suite's write flood a shard can trail an object's PUT
# event past the test's fixed sleep window. One retry absorbs it.
lifecycle_expiration() {
tox -- s3tests/functional/test_s3.py::test_lifecycle_expiration \
s3tests/functional/test_s3.py::test_lifecyclev2_expiration
}
lifecycle_expiration || { echo "lifecycle expiration tests flaked; retry 1"; lifecycle_expiration; } || { echo "lifecycle expiration tests flaked; retry 2"; lifecycle_expiration; }
s3tests/functional/test_s3.py::test_lifecycle_set_filter \
s3tests/functional/test_s3.py::test_lifecycle_expiration \
s3tests/functional/test_s3.py::test_lifecyclev2_expiration
# cleanup() trap handles worker/server kill + data dir wipe.
+2 -10
View File
@@ -29,7 +29,7 @@ permissions:
jobs:
samba-integration:
name: samba-integration
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 45
steps:
@@ -40,23 +40,15 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Start local Docker registry
run: |
if ! docker start registry 2>/dev/null; then
docker rm -f registry >/dev/null 2>&1 || true
docker run -d --restart=always -p 5000:5000 --name registry registry:2
fi
run: docker run -d --restart=always -p 5000:5000 --name registry registry:2
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4
with:
driver-opts: network=host
- name: Set up Docker Compose
uses: docker/setup-compose-action@v2
- name: Build weed race binary
run: |
cd docker
+3 -4
View File
@@ -17,8 +17,8 @@ on:
- '.github/workflows/sftp-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
group: ${{ github.head_ref }}/sftp-tests
cancel-in-progress: true
permissions:
contents: read
@@ -29,7 +29,7 @@ env:
jobs:
sftp-integration:
name: SFTP Integration Testing
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 20
steps:
@@ -40,7 +40,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Install dependencies
run: |
@@ -15,10 +15,6 @@ on:
- '.github/workflows/spark-integration-tests.yml'
workflow_dispatch:
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
checks: write
@@ -42,13 +38,12 @@ jobs:
with:
java-version: '11'
distribution: 'temurin'
cache: ${{ runner.environment == 'github-hosted' && 'maven' || '' }}
cache: maven
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Build SeaweedFS binary
run: |
+6 -5
View File
@@ -10,11 +10,12 @@ permissions:
contents: write
concurrency:
# Only one chart regeneration per branch at a time so overlapping runs never
# conflict on note/star_history.svg during rebase. Scoped by ref so a manual
# run on another branch can't cancel the daily master update.
# Only one chart regeneration per branch at a time; a newer run on the same
# branch cancels an in-flight one so overlapping runs never conflict on
# note/star_history.svg during rebase. Scoped by ref so a manual run on
# another branch can't cancel the daily master update.
group: star-history-${{ github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
cancel-in-progress: true
jobs:
render:
@@ -32,7 +33,7 @@ jobs:
uses: actions/setup-python@v7
with:
python-version: "3.x"
cache: ${{ runner.environment == 'github-hosted' && 'pip' || '' }}
cache: pip
cache-dependency-path: .github/scripts/star_history.py
- name: Install matplotlib
@@ -6,22 +6,14 @@ on:
paths:
- 'telemetry/**'
- 'weed/telemetry/**'
- 'go.mod'
- 'go.sum'
- '.github/workflows/telemetry-integration.yml'
pull_request:
branches: [ master ]
paths:
- 'telemetry/**'
- 'weed/telemetry/**'
- 'go.mod'
- 'go.sum'
- '.github/workflows/telemetry-integration.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -38,7 +30,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Build telemetry server
run: cd telemetry/server && go build -o telemetry-server .
-5
View File
@@ -8,10 +8,6 @@ on:
branches: [ master ]
paths: ['terraform/**', '.github/workflows/terraform_ci.yml']
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
@@ -66,7 +62,6 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: go.mod
cache: ${{ runner.environment == 'github-hosted' }}
- name: Build weed
run: go build -o "$RUNNER_TEMP/weed" ./weed
@@ -27,10 +27,6 @@ defaults:
run:
working-directory: weed
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
jobs:
awscli-tests:
runs-on: ubuntu-latest
@@ -41,7 +37,6 @@ jobs:
- uses: actions/setup-go@v7
with:
go-version: ^1.26
cache: ${{ runner.environment == 'github-hosted' }}
- name: Build SeaweedFS
run: |
@@ -91,11 +86,7 @@ jobs:
set -e
dd if=/dev/urandom of=generated bs=1M count=32
aws --no-verify-ssl s3 cp --no-progress generated s3://bucket/test-multipart
for i in 1 2 3 4 5; do
aws --no-verify-ssl s3 cp --no-progress s3://bucket/test-multipart downloaded && break
if [ "$i" -eq 5 ]; then exit 1; fi
sleep 2
done
aws --no-verify-ssl s3 cp --no-progress s3://bucket/test-multipart downloaded
diff -q generated downloaded
rm -f generated downloaded
+1 -6
View File
@@ -18,23 +18,18 @@ on:
- 'go.sum'
- '.github/workflows/tls-rotation-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
jobs:
tls-rotation-tests:
name: TLS Rotation Integration Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 10
steps:
- name: Set up Go 1.x
uses: actions/setup-go@v7
with:
cache: ${{ runner.environment == 'github-hosted' }}
go-version: ^1.26
id: go
+4 -8
View File
@@ -11,8 +11,8 @@ on:
- '.github/workflows/tus-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
group: ${{ github.head_ref || github.ref }}/tus-tests
cancel-in-progress: true
permissions:
contents: read
@@ -24,21 +24,17 @@ defaults:
jobs:
tus-integration-tests:
name: TUS Protocol Integration Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 20
steps:
- name: Check out code
uses: actions/checkout@v7
- name: Clean up stale FUSE mounts
uses: ./.github/actions/cleanup-stale-fuse-mounts
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Install SeaweedFS
@@ -53,7 +49,7 @@ jobs:
echo "=== System Information ==="
uname -a
free -h
timeout 15 df -h || true
df -h
echo "=== Starting TUS Tests ==="
# Run tests with automatic server management
@@ -18,23 +18,18 @@ on:
- 'go.sum'
- '.github/workflows/vacuum-integration-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
permissions:
contents: read
jobs:
vacuum-integration-tests:
name: Vacuum Integration Tests
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 15
steps:
- name: Set up Go 1.x
uses: actions/setup-go@v7
with:
cache: ${{ runner.environment == 'github-hosted' }}
go-version: ^1.26
id: go
@@ -21,8 +21,8 @@ on:
- '.github/workflows/volume-server-integration-tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
group: ${{ github.head_ref || github.ref }}/volume-server-integration-tests
cancel-in-progress: true
permissions:
contents: read
@@ -33,44 +33,10 @@ env:
SHARD_COUNT: 3
jobs:
build-weed:
name: Build SeaweedFS Binary
runs-on: ubuntu-24.04
timeout-minutes: 15
steps:
- name: Checkout code
uses: actions/checkout@v7
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Build SeaweedFS binary
run: |
cd weed
go build -o weed .
chmod +x weed
./weed version
- name: Package binary
run: tar cf weed-binary.tar weed/weed
- name: Upload binary
uses: actions/upload-artifact@v7
with:
name: weed-binary
if-no-files-found: error
retention-days: 1
path: weed-binary.tar
volume-server-integration-tests:
name: Volume Server Integration Tests (${{ matrix.test-type }} - Shard ${{ matrix.shard }})
runs-on: ubuntu-24.04
runs-on: ubuntu-22.04
timeout-minutes: 45
needs: [build-weed]
strategy:
fail-fast: false
matrix:
@@ -85,18 +51,13 @@ jobs:
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
- name: Download binary
uses: actions/download-artifact@v8
with:
name: weed-binary
- name: Restore binary
- name: Build SeaweedFS binary
run: |
tar xf weed-binary.tar
chmod +x weed/weed
./weed/weed version
cd weed
go build -o weed .
chmod +x weed
./weed version
# Dealing the listed tests out one by one keeps the shards even. Bucketing
# them by first letter did not: names cluster, so ^Test[I-S] drew 50 of
-7
View File
@@ -144,10 +144,3 @@ weed_bin
telemetry/server/telemetry-server
.aider*
/seaweed-volume/docs
__pycache__/
/test/s3vectors/weed
/test/s3vectors/weed-vector
/test/s3vectors/weed-vector-cli
/test/s3vectors/weed-worker
/test/s3vectors/weed-volume
.pytest_cache/
+4 -45
View File
@@ -17,9 +17,7 @@ SeaweedFS is a simple and highly scalable distributed file system. There are two
1. to store billions of files!
2. to serve the files fast!
One SeaweedFS cluster serves an S3 object store, a POSIX file system, a lakehouse with S3 Tables and a built-in Iceberg REST Catalog, and [vector search with S3 Vector Buckets](#vector-search-with-s3-vector-buckets), all on shared storage. Each blob is one disk read away, capacity grows by starting another volume server, and cloud storage can be cached or tiered transparently. Blob reads and writes have O(1) complexity and can run at the full speed supported by the underlying hardware.
![SeaweedFS feature overview: S3 objects, POSIX files, table buckets with an Iceberg REST Catalog and Lance namespace, and S3 Vector Buckets share one cluster of masters, filers, and volume servers, with cloud caching, tiering, and cross-cluster replication.](note/seaweedfs-features.svg)
One `weed` binary serves an S3 object store, a POSIX file system, and a lakehouse with S3 Tables, all over the same data. Each blob is one disk read away, capacity grows by starting another volume server, and cloud storage can be cached or tiered transparently. Both read and write operations have O(1) complexity and can run at the full speed supported by the underlying hardware.
- [Download Binaries for different platforms](https://github.com/seaweedfs/seaweedfs/releases/latest)
- [Wiki Documentation](https://github.com/seaweedfs/seaweedfs/wiki)
@@ -42,7 +40,6 @@ Table of Contents
* [Scalable](#scalable)
* [The most complete S3 API](#the-most-complete-s3-api)
* [A data warehouse with S3 Tables](#a-data-warehouse-with-s3-tables)
* [Vector search with S3 Vector Buckets](#vector-search-with-s3-vector-buckets)
* [A fast cache for cloud storage](#a-fast-cache-for-cloud-storage)
* [Active-active replication and more](#active-active-replication-and-more)
* [Architecture](#architecture)
@@ -67,8 +64,6 @@ Download the latest binary from the [releases](https://github.com/seaweedfs/seaw
curl -fsSL https://raw.githubusercontent.com/seaweedfs/seaweedfs/master/install.sh | bash
```
For SeaweedFS Enterprise binaries, pass `--enterprise` to the script: `... | bash -s -- --enterprise`.
Then start a ready-to-use S3 object store:
```bash
@@ -209,13 +204,12 @@ On one laptop, [`weed benchmark`][Benchmarks] writes 1KB files at 15,700 per sec
## The most complete S3 API ##
The S3 gateway implements the object, bucket, S3 Tables, S3 Vectors, IAM, and STS APIs on one endpoint, so the AWS SDKs and CLI, rclone, restic, Spark, and Trino work unchanged. S3 Vectors uses the `weed-vector` service described [below](#vector-search-with-s3-vector-buckets).
The S3 gateway implements the object, bucket, S3 Tables, IAM, and STS APIs on one endpoint, so the AWS SDKs and CLI, rclone, restic, Spark, and Trino work unchanged.
| API | Operations |
| --- | --- |
| S3 bucket and object | 73 |
| S3 Tables | 36 |
| S3 Vectors | 19 |
| IAM | 39 |
| STS | 5 |
@@ -229,49 +223,15 @@ The full operation list is in [Amazon S3 API][AmazonS3API], and [Supported APIs
## A data warehouse with S3 Tables ##
SeaweedFS is a lakehouse in one system. [S3 Table Buckets][S3TableBucket] hold Apache Iceberg tables by default, or [Lance][LanceCatalog] tables for vectors and multimodal data, and the built-in [Iceberg REST Catalog][IcebergCatalog] and Lance namespace serve them directly. The S3 Tables API and Iceberg REST Catalog share the same namespaces, table metadata, and permissions. There is no Hive Metastore, Glue, or separate catalog service to deploy, secure, and back up.
SeaweedFS is a lakehouse in one system. [S3 Table Buckets][S3TableBucket] hold Apache Iceberg tables by default, or [Lance][LanceCatalog] tables for vectors and multimodal data, and the built-in [Iceberg REST Catalog][IcebergCatalog] and Lance namespace serve them directly. There is no Hive Metastore, Glue, or separate catalog service to deploy, secure, and back up.
* Query engines operate on the same tables at the same time: [Spark][SparkIceberg], [Trino][TrinoIceberg], [Dremio][DremioIceberg], [DuckDB][DuckDBIceberg], [Apache Doris][DorisIceberg], [RisingWave][RisingWaveIceberg], ClickHouse, and [LanceDB][LanceDB]. Catalog commits are atomic compare-and-swap, so concurrent writers are safe. [Lakekeeper][Lakekeeper] can front the same storage with STS-vended credentials. [OLake Go][OLake] is a high-performance, open-source EL (Extract–Load) platform that bridges operational databases and open lakehouse storage - writing Apache Iceberg tables or Parquet files on Amazon S3.
* Query engines operate on the same tables at the same time: [Spark][SparkIceberg], [Trino][TrinoIceberg], [Dremio][DremioIceberg], [DuckDB][DuckDBIceberg], [Apache Doris][DorisIceberg], [RisingWave][RisingWaveIceberg], ClickHouse, and [LanceDB][LanceDB]. Catalog commits are atomic compare-and-swap, so concurrent writers are safe. [Lakekeeper][Lakekeeper] can front the same storage with STS-vended credentials.
* [Automated table maintenance][IcebergMaintenance]: compaction, snapshot expiration, orphan file removal, and manifest rewriting, configured per bucket or table through the S3 Tables maintenance APIs, and the same for [Lance][LanceMaintenance].
* IAM at the bucket, namespace, and table level with standard bucket policies, see [S3 Tables Security][S3TablesSecurity].
* A [Hadoop compatible file system][Hadoop] for Spark, Flink, and HBase.
`S3_TABLE_BUCKET=warehouse ./weed mini -dir=./data` brings the whole stack up on a laptop.
## Vector search with S3 Vector Buckets ##
Keep embeddings alongside your objects and tables in the same SeaweedFS cluster. S3 Vector Buckets expose the AWS S3 Vectors API for semantic search, retrieval-augmented generation (RAG), and recommendations, using Lance tables on SeaweedFS storage.
* Create vector buckets and indexes; insert, retrieve, list, and delete vectors with the AWS SDKs or CLI pointed at your S3 endpoint.
* Find the nearest vectors with cosine or Euclidean distance, filter by metadata, and return distances and metadata with the results.
* Control access with `s3vectors:*` IAM actions and vector bucket policies; tag buckets and indexes for organization and access control.
Install the separate vector engine; the default installer installs only `weed`:
```bash
curl -fsSL https://raw.githubusercontent.com/seaweedfs/seaweedfs/master/install.sh | bash -s -- --component vector
```
Release 4.53 provides vector binaries for Linux amd64 and arm64. If your release has no binary for your platform, build from [source](seaweed-vector). From a directory where you want a new checkout:
```bash
git clone https://github.com/seaweedfs/seaweedfs.git
cd seaweedfs
(cd seaweed-vector && cargo build --release -p weed-vector-server)
```
If you already have a checkout, start at its root and skip the clone and `cd seaweedfs` steps.
For Windows source builds, set `PROTOC` to a native `protoc.exe` before running Cargo; the [desktop build workflow](.github/workflows/rust-vector-platforms.yml) shows how to select the vendored executable.
Run the installed `weed-vector` binary alongside the local cluster started with `weed mini` above (or use `seaweed-vector/target/release/weed-vector` after building from source):
```bash
weed-vector --masters=localhost:9333
```
It registers with the master and the S3 gateway discovers it automatically. An AWS S3 Vectors client uses the same S3 endpoint at http://localhost:8333. The vector engine stores its data through the cluster's filers and volume servers, so embeddings share the storage layer's replication and erasure coding. See the [AWS SDK examples](test/s3vectors/s3vectors_test.py) for bucket, index, and query operations.
## A fast cache for cloud storage ##
[Cloud Drive][CloudDrive] mounts a bucket from S3, Google Cloud Storage, Azure, Backblaze B2, Wasabi, Storj, or any S3-compatible store into SeaweedFS and serves it at local speed:
@@ -527,7 +487,6 @@ Your support will be really appreciated by me and other supporters!
[RisingWaveIceberg]: https://github.com/seaweedfs/seaweedfs/wiki/RisingWave-Iceberg-Integration
[LanceDB]: https://github.com/seaweedfs/seaweedfs/wiki/LanceDB-Integration
[Lakekeeper]: https://github.com/seaweedfs/seaweedfs/wiki/Lakekeeper-Iceberg-Integration
[OLake]: https://github.com/seaweedfs/seaweedfs/wiki/OLake-Go-Iceberg-Integration
[IcebergMaintenance]: https://github.com/seaweedfs/seaweedfs/wiki/Iceberg-Table-Maintenance
[LanceMaintenance]: https://github.com/seaweedfs/seaweedfs/wiki/Lance-Maintenance-Worker
[S3TablesSecurity]: https://github.com/seaweedfs/seaweedfs/wiki/S3-Tables-Security
-4
View File
@@ -271,10 +271,6 @@ curl "http://localhost:9333/vol/vacuum?garbageThreshold=0.4"
Vacuuming makes a volume read-only, copies live needles to a new volume, and
swaps it in.
The master runs one vacuum at a time. A request that arrives while the
automatic vacuum or another request is running does nothing and gets
`409 Conflict` with `{"error":"another vacuum is running, try again later"}`.
### Delete a collection
```bash
-1
View File
@@ -169,7 +169,6 @@ seaweed-volume/
- `--memprofile`
- `--compactionMBps` (default 0)
- `--maintenanceMBps` (default 0)
- `--compactionMaxParallelCommits` (default 0)
- `--fileSizeLimitMB` (default 256)
- `--concurrentUploadLimitMB` (default 0)
- `--concurrentDownloadLimitMB` (default 0)
+1 -1
View File
@@ -543,7 +543,7 @@ a language-agnostic gRPC stream that external maintenance workers connect on: th
opens `WorkerStream`, sends `WorkerHello` with the job types it can `detect` and `execute`,
answers `RequestConfigSchema` with a `JobTypeDescriptor`, replies to `RunDetectionRequest`
with `JobProposal`s and to `ExecuteJobRequest` with `JobProgressUpdate`s and `JobCompleted`.
`weed worker -admin=host:12535` is the Go reference implementation of exactly that contract,
`weed worker -admin=host:23646` is the Go reference implementation of exactly that contract,
from outside the admin process.
Nothing in it is Go-specific, and the Rust toolchain is already in the tree.
+1 -12
View File
@@ -33,7 +33,6 @@ ARG TARGETARCH
ARG TAGS
COPY weed-volume-prebuilt/ /prebuilt/
COPY weed-worker-prebuilt/ /prebuilt-worker/
COPY weed-vector-prebuilt/ /prebuilt-vector/
COPY --from=builder /go/src/github.com/seaweedfs/seaweedfs/seaweed-volume /build/seaweed-volume
# seaweed-common is a path dependency of seaweed-volume that lives beside it,
# so the source build below needs it in the same relative position.
@@ -66,19 +65,11 @@ RUN if [ -f "/prebuilt-worker/weed-worker-${TARGETARCH}" ]; then \
echo "No pre-built Rust worker for ${TARGETARCH}" && \
touch /weed-worker; \
fi
# The vector engine carries the same lance tree, so it is pre-built or absent too.
RUN if [ -f "/prebuilt-vector/weed-vector-${TARGETARCH}" ]; then \
echo "Using pre-built vector engine for ${TARGETARCH}" && \
cp "/prebuilt-vector/weed-vector-${TARGETARCH}" /weed-vector; \
else \
echo "No pre-built vector engine for ${TARGETARCH}" && \
touch /weed-vector; \
fi
# Pre-built binaries arrive via GitHub Actions artifacts, which drop the
# executable bit, so the copied file is 0644 and exec fails with "Permission
# denied". Restore it (no-op for the empty placeholders, which stay size 0).
RUN chmod 0755 /weed-volume /weed-worker /weed-vector
RUN chmod 0755 /weed-volume /weed-worker
FROM alpine AS final
LABEL author="Chris Lu"
@@ -87,8 +78,6 @@ COPY --from=builder /go/bin/weed /usr/bin/
COPY --from=rust_builder /weed-volume /usr/bin/weed-volume
# Same for the Rust maintenance worker, which serves Lance table buckets
COPY --from=rust_builder /weed-worker /usr/bin/weed-worker
# And the vector engine behind the S3 Vectors API
COPY --from=rust_builder /weed-vector /usr/bin/weed-vector
RUN mkdir -p /etc/seaweedfs
COPY --from=builder /go/src/github.com/seaweedfs/seaweedfs/docker/filer.toml /etc/seaweedfs/filer.toml
COPY --from=builder /go/src/github.com/seaweedfs/seaweedfs/docker/entrypoint.sh /entrypoint.sh
+1 -1
View File
@@ -3,7 +3,7 @@ FROM golang:1.26 AS builder
RUN apt-get update
RUN apt-get install -y build-essential libsnappy-dev zlib1g-dev libbz2-dev libgflags-dev liblz4-dev libzstd-dev
ARG ROCKSDB_VERSION=v11.1.2
ARG ROCKSDB_VERSION=v10.10.1
ENV ROCKSDB_VERSION=${ROCKSDB_VERSION}
# build RocksDB
+1 -1
View File
@@ -3,7 +3,7 @@ FROM golang:1.26 AS builder
RUN apt-get update
RUN apt-get install -y build-essential libsnappy-dev zlib1g-dev libbz2-dev libgflags-dev liblz4-dev libzstd-dev
ARG ROCKSDB_VERSION=v11.1.2
ARG ROCKSDB_VERSION=v10.10.1
ENV ROCKSDB_VERSION=${ROCKSDB_VERSION}
# build RocksDB
@@ -21,7 +21,7 @@ The testing environment uses **official SeaweedFS commands** and includes:
- **1 Master Server** (port 9333) - Coordinates the cluster with 50MB volume size limit
- **6 Volume Servers** (ports 8080-8085) - Distributed across 2 data centers and 3 racks for diversity
- **1 Filer** (port 8888) - Provides file system interface
- **1 Admin Server** (port 12535) - Detects volumes needing EC and manages workers using official `admin` command
- **1 Admin Server** (port 23646) - Detects volumes needing EC and manages workers using official `admin` command
- **3 EC Workers** - Execute erasure coding tasks using official `worker` command with task-specific working directories
- **1 Load Generator** - Continuously writes and deletes files using SeaweedFS shell commands
- **1 Monitor** - Tracks cluster health and EC progress using shell scripts
@@ -47,7 +47,7 @@ This provides:
- Docker and Docker Compose installed
- GNU Make installed
- At least 4GB RAM available for containers
- Ports 8080-8085, 8888, 9333, 12535 available
- Ports 8080-8085, 8888, 9333, 23646 available
### Start the Environment
@@ -127,7 +127,7 @@ Run `make help` to see all available targets:
|---------|-----|-------------|
| Master UI | http://localhost:9333 | Cluster status and topology |
| Filer | http://localhost:8888 | File operations |
| Admin Server | http://localhost:12535/ | Task management |
| Admin Server | http://localhost:23646/ | Task management |
| Monitor | http://localhost:9999/status | Complete cluster monitoring |
| Volume Servers | http://localhost:8080-8085/status | Individual volume server stats |
+15 -15
View File
@@ -26,7 +26,7 @@ build-and-restart: build ## Build with latest changes and restart services
@echo "2️⃣ Recreating workers to reconnect..."
@docker-compose -f $(COMPOSE_FILE) up -d worker1 worker2 worker3
@echo "✅ All services recreated with latest changes!"
@echo "🌐 Admin UI: http://localhost:12535/"
@echo "🌐 Admin UI: http://localhost:23646/"
@echo "💡 Workers will reconnect to the new admin server"
restart-workers: ## Restart all workers to reconnect to admin server
@@ -64,7 +64,7 @@ start: ## Start the complete SeaweedFS cluster with admin and workers
@echo "✅ Cluster started!"
@echo ""
@echo "📊 Access points:"
@echo " • Admin UI: http://localhost:12535/"
@echo " • Admin UI: http://localhost:23646/"
@echo " • Master UI: http://localhost:9333/"
@echo " • Filer: http://localhost:8888/"
@echo ""
@@ -107,7 +107,7 @@ start-staged: ## Start services in proper order with delays
@echo "✅ All services started!"
@echo ""
@echo "📊 Access points:"
@echo " • Admin UI: http://localhost:12535/"
@echo " • Admin UI: http://localhost:23646/"
@echo " • Master UI: http://localhost:9333/"
@echo " • Filer: http://localhost:8888/"
@echo ""
@@ -136,7 +136,7 @@ status: ## Check the status of all services
@echo "Master:"
@curl -s http://localhost:9333/cluster/status | jq '.IsLeader' 2>/dev/null || echo " ❌ Master not ready"
@echo "Admin:"
@curl -s http://localhost:12535/ | grep -q "Admin" && echo " ✅ Admin ready" || echo " ❌ Admin not ready"
@curl -s http://localhost:23646/ | grep -q "Admin" && echo " ✅ Admin ready" || echo " ❌ Admin not ready"
logs: ## Show logs from all services
@echo "📜 Following logs from all services..."
@@ -157,7 +157,7 @@ master-logs: ## Show logs from master server
admin-ui: ## Open admin UI in browser (macOS)
@echo "🌐 Opening admin UI in browser..."
@open http://localhost:12535/ || echo "💡 Manually open: http://localhost:12535/"
@open http://localhost:23646/ || echo "💡 Manually open: http://localhost:23646/"
test: ## Run integration test to verify task assignment and completion
@echo "🧪 Running Admin-Worker Integration Test"
@@ -166,11 +166,11 @@ test: ## Run integration test to verify task assignment and completion
@echo "1️⃣ Checking cluster health..."
@sleep 5
@curl -s http://localhost:9333/cluster/status | jq '.IsLeader' > /dev/null && echo "✅ Master healthy" || echo "❌ Master not ready"
@curl -s http://localhost:12535/ | grep -q "Admin" && echo "✅ Admin healthy" || echo "❌ Admin not ready"
@curl -s http://localhost:23646/ | grep -q "Admin" && echo "✅ Admin healthy" || echo "❌ Admin not ready"
@echo ""
@echo "2️⃣ Checking worker registration..."
@sleep 10
@echo "💡 Check admin UI for connected workers: http://localhost:12535/"
@echo "💡 Check admin UI for connected workers: http://localhost:23646/"
@echo ""
@echo "3️⃣ Generating load to trigger EC tasks..."
@echo "📝 Creating test files to fill volumes..."
@@ -193,13 +193,13 @@ test: ## Run integration test to verify task assignment and completion
@echo " • EC shards being distributed"
@echo ""
@echo "✅ Integration test setup complete!"
@echo "📊 Monitor progress at: http://localhost:12535/"
@echo "📊 Monitor progress at: http://localhost:23646/"
quick-test: ## Quick verification that core services are running
@echo "⚡ Quick Health Check"
@echo "===================="
@echo "Master: $$(curl -s http://localhost:9333/cluster/status | jq -r '.IsLeader // "not ready"')"
@echo "Admin: $$(curl -s http://localhost:12535/ | grep -q "Admin" && echo "ready" || echo "not ready")"
@echo "Admin: $$(curl -s http://localhost:23646/ | grep -q "Admin" && echo "ready" || echo "not ready")"
@echo "Workers: $$(docker-compose -f $(COMPOSE_FILE) ps worker1 worker2 worker3 | grep -c Up) running"
validate: ## Validate integration test configuration
@@ -223,7 +223,7 @@ demo: start ## Start cluster and run demonstration
@echo " • Load generator (creates files continuously)"
@echo ""
@echo "📊 Watch the process:"
@echo " 1. Visit: http://localhost:12535/"
@echo " 1. Visit: http://localhost:23646/"
@echo " 2. Observe workers connecting"
@echo " 3. Watch tasks being created and assigned"
@echo " 4. See tasks progress from pending → completed"
@@ -244,13 +244,13 @@ vacuum-test: ## Create test data with garbage and verify vacuum detection
@echo ""
@echo "1️⃣ Checking cluster health..."
@curl -s http://localhost:9333/cluster/status | jq '.IsLeader' > /dev/null && echo "✅ Master ready" || (echo "❌ Master not ready. Run 'make start' first." && exit 1)
@curl -s http://localhost:12535/ | grep -q "Admin" && echo "✅ Admin ready" || (echo "❌ Admin not ready. Run 'make start' first." && exit 1)
@curl -s http://localhost:23646/ | grep -q "Admin" && echo "✅ Admin ready" || (echo "❌ Admin not ready. Run 'make start' first." && exit 1)
@echo ""
@echo "2️⃣ Creating test data with garbage..."
@docker-compose -f $(COMPOSE_FILE) exec vacuum-tester go run create_vacuum_test_data.go -files=25 -delete=0.5 -size=200
@echo ""
@echo "3️⃣ Configuration Instructions:"
@echo " Visit: http://localhost:12535/maintenance/config/vacuum"
@echo " Visit: http://localhost:23646/maintenance/config/vacuum"
@echo " Set for testing:"
@echo " • Enable Vacuum Tasks: ✅ Checked"
@echo " • Garbage Threshold: 0.20 (20%)"
@@ -258,7 +258,7 @@ vacuum-test: ## Create test data with garbage and verify vacuum detection
@echo " • Min Volume Age: [0] [Minutes]"
@echo " • Max Concurrent: 2"
@echo ""
@echo "4️⃣ Monitor vacuum tasks at: http://localhost:12535/maintenance"
@echo "4️⃣ Monitor vacuum tasks at: http://localhost:23646/maintenance"
@echo ""
@echo "💡 Use 'make vacuum-status' to check volume garbage ratios"
@@ -335,8 +335,8 @@ vacuum-help: ## Show vacuum testing help and examples
@echo " make vacuum-demo # Full guided demonstration"
@echo ""
@echo "Configuration:"
@echo " Visit: http://localhost:12535/maintenance/config/vacuum"
@echo " Monitor: http://localhost:12535/maintenance"
@echo " Visit: http://localhost:23646/maintenance/config/vacuum"
@echo " Monitor: http://localhost:23646/maintenance"
@echo ""
@echo "Custom Parameters:"
@echo " make vacuum-data FILES=50 DELETE=0.8 SIZE=200"
@@ -243,7 +243,7 @@ func printTestingInstructions() {
fmt.Println("🧪 Testing Instructions:")
fmt.Println()
fmt.Println("1. Configure Vacuum for Testing:")
fmt.Println(" Visit: http://localhost:12535/maintenance/config/vacuum")
fmt.Println(" Visit: http://localhost:23646/maintenance/config/vacuum")
fmt.Println(" Set:")
fmt.Printf(" - Garbage Percentage Threshold: 20 (20%% - lower than default 30)\n")
fmt.Printf(" - Scan Interval: [30] [Seconds] (faster than default)\n")
@@ -253,7 +253,7 @@ func printTestingInstructions() {
fmt.Println()
fmt.Println("2. Monitor Vacuum Tasks:")
fmt.Println(" Visit: http://localhost:12535/maintenance")
fmt.Println(" Visit: http://localhost:23646/maintenance")
fmt.Println(" Watch for vacuum tasks to appear in the queue")
fmt.Println()
@@ -7,7 +7,7 @@ echo ""
# Check if SeaweedFS is running
echo "📋 Checking SeaweedFS status..."
MASTER_URL="${MASTER_HOST:-master:9333}"
ADMIN_URL="${ADMIN_HOST:-admin:12535}"
ADMIN_URL="${ADMIN_HOST:-admin:23646}"
if ! curl -s http://$MASTER_URL/cluster/status > /dev/null; then
echo "❌ SeaweedFS master not running at $MASTER_URL"
@@ -42,7 +42,7 @@ echo ""
# Phase 3: Configure vacuum
echo "⚙️ Phase 3: Vacuum configuration instructions..."
echo " 1. Visit: http://localhost:12535/maintenance/config/vacuum"
echo " 1. Visit: http://localhost:23646/maintenance/config/vacuum"
echo " 2. Set these values for testing:"
echo " - Enable Vacuum Tasks: ✅ Checked"
echo " - Garbage Threshold: 0.30"
@@ -57,7 +57,7 @@ echo ""
# Phase 4: Monitor tasks
echo "🎯 Phase 4: Monitoring vacuum tasks..."
echo " Visit: http://localhost:12535/maintenance"
echo " Visit: http://localhost:23646/maintenance"
echo " You should see vacuum tasks appear within 30 seconds"
echo ""
@@ -93,9 +93,9 @@ echo ""
echo "🎉 Demo Complete!"
echo ""
echo "🔍 Things to check:"
echo " 1. Maintenance Queue: http://localhost:12535/maintenance"
echo " 1. Maintenance Queue: http://localhost:23646/maintenance"
echo " 2. Volume Status: http://localhost:9333/vol/status"
echo " 3. Admin Dashboard: http://localhost:12535"
echo " 3. Admin Dashboard: http://localhost:23646"
echo ""
echo "💡 Next Steps:"
echo " - Try different garbage thresholds (0.10, 0.50, 0.80)"
@@ -114,9 +114,9 @@ services:
admin:
image: chrislusf/seaweedfs:local
ports:
- "12535:12535" # HTTP admin interface (default port)
- "22535:22535" # gRPC worker communication (12535 + 10000)
command: "-v=2 admin -port=12535 -masters=master:9333 -dataDir=/data"
- "23646:23646" # HTTP admin interface (default port)
- "33646:33646" # gRPC worker communication (23646 + 10000)
command: "-v=2 admin -port=23646 -masters=master:9333 -dataDir=/data"
depends_on:
- master
- filer
@@ -127,7 +127,7 @@ services:
worker1:
image: chrislusf/seaweedfs:local
command: "-v=2 worker -admin=admin:12535 -capabilities=erasure_coding,vacuum -maxConcurrent=2"
command: "-v=2 worker -admin=admin:23646 -capabilities=erasure_coding,vacuum -maxConcurrent=2"
depends_on:
- admin
volumes:
@@ -139,7 +139,7 @@ services:
worker2:
image: chrislusf/seaweedfs:local
command: "-v=2 worker -admin=admin:12535 -capabilities=erasure_coding,vacuum -maxConcurrent=2"
command: "-v=2 worker -admin=admin:23646 -capabilities=erasure_coding,vacuum -maxConcurrent=2"
depends_on:
- admin
volumes:
@@ -151,7 +151,7 @@ services:
worker3:
image: chrislusf/seaweedfs:local
command: "-v=2 worker -admin=admin:12535 -capabilities=erasure_coding,vacuum -maxConcurrent=2"
command: "-v=2 worker -admin=admin:23646 -capabilities=erasure_coding,vacuum -maxConcurrent=2"
depends_on:
- admin
volumes:
@@ -199,7 +199,7 @@ services:
curl -s http://master:9333/cluster/status | jq '.IsLeader, .Peers' || echo 'Master not ready';
echo;
echo 'Admin status:';
curl -s http://admin:12535/ | grep -o 'Admin.*Interface' || echo 'Admin not ready';
curl -s http://admin:23646/ | grep -o 'Admin.*Interface' || echo 'Admin not ready';
echo;
echo 'Volume count by server:';
curl -s http://master:9333/vol/status | jq '.Volumes | length' || echo 'Volumes not ready';
@@ -237,4 +237,4 @@ services:
- seaweed_net
environment:
- MASTER_HOST=master:9333
- ADMIN_HOST=admin:12535
- ADMIN_HOST=admin:23646
+2 -2
View File
@@ -23,7 +23,7 @@ else
fi
echo -e "${BLUE}2. Checking if required ports are available...${NC}"
for port in 9333 8080 8081 8082 8083 8084 8085 8888 12535; do
for port in 9333 8080 8081 8082 8083 8084 8085 8888 23646; do
if lsof -i :$port > /dev/null 2>&1; then
echo -e "${YELLOW}⚠️ Port $port is in use${NC}"
else
@@ -41,7 +41,7 @@ else
fi
echo -e "${BLUE}4. Verifying admin server configuration...${NC}"
if docker-compose -f docker-compose-ec-test.yml config | grep -q "admin:12535"; then
if docker-compose -f docker-compose-ec-test.yml config | grep -q "admin:23646"; then
echo -e "${GREEN}✅ Admin server port configuration is correct${NC}"
else
echo -e "${RED}❌ Admin server port configuration is incorrect${NC}"
+10 -8
View File
@@ -3,15 +3,17 @@
# unreachable. See docker/Dockerfile.e2e.
set -e
# Every rewrite starts from the pristine list, so a mirror that just failed does
# not become the pattern the next rewrite has to match.
[ -f /etc/apt/sources.list.orig ] || cp /etc/apt/sources.list /etc/apt/sources.list.orig
for mirror in azure.archive.ubuntu.com archive.ubuntu.com; do
for apt_source in /etc/apt/sources.list /etc/apt/sources.list.d/*.list /etc/apt/sources.list.d/*.sources; do
[ -f "$apt_source" ] || continue
[ -f "$apt_source.orig" ] || cp "$apt_source" "$apt_source.orig"
sed -E "s|(https?://)[a-z0-9.]*archive\.ubuntu\.com/ubuntu|\1$mirror/ubuntu|g; s|(https?://)security\.ubuntu\.com/ubuntu|\1$mirror/ubuntu|g" \
"$apt_source.orig" > "$apt_source"
done
if timeout --kill-after=5s 120s apt-get -o Acquire::Retries=1 -o Acquire::http::Timeout=15 -o Acquire::https::Timeout=15 -o APT::Update::Error-Mode=any update && \
DEBIAN_FRONTEND=noninteractive timeout --kill-after=5s 180s apt-get -o Acquire::Retries=1 -o Acquire::http::Timeout=15 -o Acquire::https::Timeout=15 install -y \
# Any archive host, so this works whether the pristine list came from the
# base image (archive.ubuntu.com) or a CI runner (azure.archive.ubuntu.com).
sed "s|http://[a-z0-9.]*archive\.ubuntu\.com/ubuntu|http://$mirror/ubuntu|g; s|http://security\.ubuntu\.com/ubuntu|http://$mirror/ubuntu|g" \
/etc/apt/sources.list.orig > /etc/apt/sources.list
if apt-get -o Acquire::Retries=3 -o Acquire::http::Timeout=15 update && \
DEBIAN_FRONTEND=noninteractive apt-get -o Acquire::Retries=3 -o Acquire::http::Timeout=15 install -y \
--no-install-recommends --no-install-suggests "$@"; then
apt-get clean
rm -rf /var/lib/apt/lists/*
+8 -21
View File
@@ -64,7 +64,7 @@ case "$1" in
'master')
ARGS="-mdir=/data -volumeSizeLimitMB=1024"
shift
exec /usr/bin/weed -logtostderr=true master $ARGS "$@"
exec /usr/bin/weed -logtostderr=true master $ARGS $@
;;
'volume')
@@ -73,7 +73,7 @@ case "$1" in
ARGS="-dir=/data"
fi
shift
exec /usr/bin/weed -logtostderr=true volume $ARGS "$@"
exec /usr/bin/weed -logtostderr=true volume $ARGS $@
;;
'volume-rust')
@@ -87,7 +87,7 @@ case "$1" in
echo "Use 'volume' for the Go volume server instead." >&2
exit 1
fi
exec /usr/bin/weed-volume $ARGS "$@"
exec /usr/bin/weed-volume $ARGS $@
;;
'worker-rust')
@@ -100,26 +100,13 @@ case "$1" in
exec /usr/bin/weed-worker "$@"
;;
'vector')
shift
if [ ! -s /usr/bin/weed-vector ]; then
echo "Error: the vector engine is not available on this platform ($(uname -m))." >&2
exit 1
fi
# The gateway dials the engine from another container, so not loopback.
if [ -z "$WEED_VECTOR_LISTEN" ] && ! isArgPassed "-listen" "$@"; then
set -- --listen 0.0.0.0:8090 "$@"
fi
exec /usr/bin/weed-vector "$@"
;;
'server')
ARGS="-dir=/data -volume.max=0 -master.volumeSizeLimitMB=1024"
if isArgPassed "-volume.max" "$@"; then
ARGS="-dir=/data -master.volumeSizeLimitMB=1024"
fi
shift
exec /usr/bin/weed -logtostderr=true server $ARGS "$@"
exec /usr/bin/weed -logtostderr=true server $ARGS $@
;;
'mini')
@@ -128,19 +115,19 @@ case "$1" in
ARGS=""
fi
shift
exec /usr/bin/weed -logtostderr=true mini $ARGS "$@"
exec /usr/bin/weed -logtostderr=true mini $ARGS $@
;;
'filer')
ARGS=""
shift
exec /usr/bin/weed -logtostderr=true filer $ARGS "$@"
exec /usr/bin/weed -logtostderr=true filer $ARGS $@
;;
's3')
ARGS="-domainName=$S3_DOMAIN_NAME -key.file=$S3_KEY_FILE -cert.file=$S3_CERT_FILE"
shift
exec /usr/bin/weed -logtostderr=true s3 $ARGS "$@"
exec /usr/bin/weed -logtostderr=true s3 $ARGS $@
;;
'shell')
@@ -150,6 +137,6 @@ case "$1" in
;;
*)
exec /usr/bin/weed "$@"
exec /usr/bin/weed $@
;;
esac
-39
View File
@@ -1,39 +0,0 @@
# Vector buckets locally
Run on Linux containers (amd64 or arm64):
```sh
docker compose up -d
export AWS_ACCESS_KEY_ID=vector-demo
export AWS_SECRET_ACCESS_KEY=vector-demo-secret
export AWS_DEFAULT_REGION=us-east-1
aws --endpoint-url http://localhost:8333 s3vectors create-index \
--vector-bucket-name embeddings --index-name images \
--data-type float32 --dimension 3 --distance-metric cosine
aws --endpoint-url http://localhost:8333 s3vectors put-vectors \
--vector-bucket-name embeddings --index-name images \
--vectors '[{"key":"first","data":{"float32":[1,0,0]}}]'
aws --endpoint-url http://localhost:8333 s3vectors query-vectors \
--vector-bucket-name embeddings --index-name images \
--query-vector '{"float32":[1,0,0]}' --top-k 1
```
The engine registers with the master. Wait for that registration before creating
an index. The Lance worker builds and refreshes ANN indexes as data grows;
queries can scan data before an ANN index is built. The demo's admin UI is at
http://localhost:12535 (user `admin`, password `vector-demo`). Demo credentials
and plaintext cluster connections are for this local example. Production
settings are in the Helm/operator TLS and authentication documentation.
`docker compose down` preserves the named data volume. Add `-v` to discard it.
Override `SEAWEEDFS_IMAGE` to test a newer image. With metrics-enabled engines,
append `--metrics-port=9329 --metrics-ip=0.0.0.0` to the vector command and scrape
`vector:9329/metrics` inside the Compose network.
A native development setup can use `weed mini -vector -vectorBucket=embeddings`
with `weed-vector` beside `weed` or on PATH. `-vector.binary` selects an explicit
binary, and `-vector.metricsPort=9329` enables HTTP readiness and metrics on a
compatible engine. The child inherits the cluster's configuration directory and
stops with mini. Run `weed-worker` separately for ANN maintenance. `weed shell`
provides `s3vectors -operation <Operation> -file request.json` for the same API
operations using trusted cluster credentials.
-34
View File
@@ -1,34 +0,0 @@
services:
seaweedfs:
image: ${SEAWEEDFS_IMAGE:-chrislusf/seaweedfs:4.53}
command: mini -dir=/data -ip=seaweedfs -ip.bind=0.0.0.0 -admin.password=vector-demo -admin.worker.ip=0.0.0.0 -tableBucket=embeddings:LANCE
environment: &environment
WEED_JWT_FILER_SIGNING_KEY: vector-demo-cluster-key
AWS_ACCESS_KEY_ID: vector-demo
AWS_SECRET_ACCESS_KEY: vector-demo-secret
ports:
- "127.0.0.1:8333:8333"
- "127.0.0.1:12535:12535"
volumes:
- data:/data
healthcheck:
test: [CMD-SHELL, "wget -q -O /dev/null http://127.0.0.1:9333/cluster/status"]
interval: 5s
timeout: 3s
retries: 30
vector:
image: ${SEAWEEDFS_IMAGE:-chrislusf/seaweedfs:4.53}
command: vector --listen=vector:8090 --masters=seaweedfs:9333
environment: *environment
depends_on:
seaweedfs:
condition: service_healthy
worker:
image: ${SEAWEEDFS_IMAGE:-chrislusf/seaweedfs:4.53}
command: worker-rust --admin=seaweedfs:12535 --masters=seaweedfs:9333 --namespace=http://seaweedfs:9101
environment: *environment
depends_on:
seaweedfs:
condition: service_healthy
volumes:
data:
+1 -1
View File
@@ -45,7 +45,7 @@ services:
admin:
image: chrislusf/seaweedfs:dev # use a remote dev image
ports:
- 12535:12535
- 23646:23646
command: 'admin -master=master:9333'
depends_on:
- master
Loaded 100 of 1953 files, more files were not shown because too many files have changed in this diff. Show more