mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-19 10:54:08 +01:00
* test/s3: run tigrisdata-community s3-api-compat-tests in CI Runs the suite, pinned to a commit, against weed mini through a small provider patch, and fails on any test that is not in expected_failures.txt or on a listed test that starts passing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3/tigris_compat: prepare the suite again when the patch changes The checkout was reused whenever it sat at the pinned commit, so an edited seaweedfs-provider.patch, or a checkout whose patch failed to apply, kept being built as it was. Record the commit and the patch's hash once the patch applies, and prepare the checkout again when either differs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3/tigris_compat: keep the pid file of a server already running start-server removed weed-server.pid before it found the port taken, so a second start lost the pid of the server holding it and stop-server could no longer stop that one. Remove the file only once the port is free, and have test-with-server stop only a server it started. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3/tigris_compat: bound the S3 port probes A listener that accepts but never answers held curl, and with it the readiness loop and the cleanup after it, for good. Give each probe two seconds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: accept PutBucketAccelerateConfiguration with Status=Suspended GetBucketAccelerateConfiguration always reports Suspended, but the PUT answered 501 for every request. Suspended is now accepted, Enabled stays NotImplemented, and the accelerate subresource resolves to s3:GetAccelerateConfiguration / s3:PutAccelerateConfiguration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3api: accept an empty PutBucketNotificationConfiguration GetBucketNotificationConfiguration always reports an empty configuration, but the PUT answered 501, including for the empty configuration clients send to clear notifications. An empty one is now accepted; a topic, queue, Lambda or EventBridge destination stays NotImplemented. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3api: reject accelerate and notification bodies with the wrong root Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test/s3/tigris_compat: expect a listed test to end with its listed status A listed test passed the check with any non-pass status, so a setup error that skipped the call under test (RestoreObject's PutObject, say) went unnoticed. Entries now expect fail unless they name another status; only GetBucketAccelerateConfiguration, whose setup PUT is unsupported, expects error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3/tigris_compat: refuse to start while the recorded server still runs A second start-server before the first server answered on the S3 port passed the port probe and dropped the pid file of the live server, so stop-server could no longer stop it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3/tigris_compat: recognize the test server by its command line start-server and stop-server took any process named weed at the recorded pid for the test server, so a weed process that reused the pid would be reported as running and killed by stop-server. Match the command line start-server launches instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3/tigris_compat: recognize the test server by its start time A weed mini started from another checkout has the same command line, so a pid it reused still matched. Record the process start time next to the pid at launch and require both before treating the process as the test server. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3/tigris_compat: PutObjectTagging_VersionId passes now Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3/tigris_compat: keep the server records until the server exits stop-server waited a fixed two seconds after SIGTERM and then removed weed-server.pid and weed-server.started even if weed was still shutting down, so a later stop-server could not find it and clean could delete data it still used. Wait up to 30 seconds for the recorded server to exit and keep the records, failing, while it still runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test/s3/tigris_compat: start the server on an empty data directory A data directory left by an earlier run on other ports made the master freeze volume growth (topology id mismatch), and every PutObject failed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
132 lines
4.7 KiB
Makefile
132 lines
4.7 KiB
Makefile
# Runs https://github.com/tigrisdata-community/s3-api-compat-tests against a
|
|
# local SeaweedFS S3 gateway and compares the report with expected_failures.txt.
|
|
|
|
.PHONY: all test help build-weed build-suite start-server stop-server run-suite test-with-server logs clean
|
|
|
|
SUITE_REPO := https://github.com/tigrisdata-community/s3-api-compat-tests.git
|
|
SUITE_COMMIT := e846275e489ec30b324344d462e698252c3a1f67
|
|
SUITE_DIR := s3-api-compat-tests
|
|
SUITE_STAMP := $(SUITE_DIR)/.seaweedfs-prepared
|
|
|
|
WEED_BINARY := ../../../weed/weed_binary
|
|
S3_PORT ?= 8335
|
|
MASTER_PORT ?= 9335
|
|
VOLUME_PORT ?= 8085
|
|
FILER_PORT ?= 8889
|
|
WEBDAV_PORT ?= 7335
|
|
ADMIN_PORT ?= 23647
|
|
SERVER_DIR := ./test-volume-data
|
|
S3_CONFIG := ./test_s3.json
|
|
# Succeeds while weed-server.pid names the server start-server launched: the
|
|
# process at that pid must have the start time recorded with it, so a process
|
|
# that reused the pid is left alone.
|
|
OWN_SERVER_RUNNING = [ -f weed-server.pid ] && [ -s weed-server.started ] && \
|
|
[ "$$(ps -p "$$(cat weed-server.pid)" -o lstart= 2>/dev/null)" = "$$(cat weed-server.started)" ]
|
|
|
|
.DEFAULT_GOAL := help
|
|
|
|
all: test
|
|
|
|
test: test-with-server
|
|
|
|
help:
|
|
@echo "s3-api-compat-tests (Tigris) against SeaweedFS"
|
|
@echo ""
|
|
@echo " build-weed - Build the SeaweedFS binary"
|
|
@echo " build-suite - Fetch the suite at $(SUITE_COMMIT) and build it"
|
|
@echo " start-server - Start weed mini with S3 on port $(S3_PORT)"
|
|
@echo " run-suite - Run the suite and check expected_failures.txt"
|
|
@echo " stop-server - Stop the test server"
|
|
@echo " test-with-server - Start server, run suite, stop server"
|
|
@echo " logs - Show server logs"
|
|
@echo " clean - Remove test artifacts"
|
|
|
|
build-weed:
|
|
@cd ../../../weed && go build -o weed_binary .
|
|
|
|
build-suite:
|
|
@want="$(SUITE_COMMIT) $$(git hash-object seaweedfs-provider.patch)"; \
|
|
if [ "$$(cat $(SUITE_STAMP) 2>/dev/null)" != "$$want" ]; then \
|
|
rm -rf $(SUITE_DIR) && mkdir -p $(SUITE_DIR) && \
|
|
git -C $(SUITE_DIR) init -q && \
|
|
git -C $(SUITE_DIR) fetch -q --depth 1 $(SUITE_REPO) $(SUITE_COMMIT) && \
|
|
git -C $(SUITE_DIR) checkout -q FETCH_HEAD && \
|
|
git -C $(SUITE_DIR) apply ../seaweedfs-provider.patch && \
|
|
echo "$$want" > $(SUITE_STAMP); \
|
|
fi
|
|
@cd $(SUITE_DIR) && go build -o ../compat-runner .
|
|
|
|
start-server: build-weed
|
|
@if curl -sS --max-time 2 -o /dev/null http://127.0.0.1:$(S3_PORT)/ >/dev/null 2>&1; then \
|
|
echo "port $(S3_PORT) is already served by another process; refusing to test against it"; \
|
|
exit 1; \
|
|
fi
|
|
@if $(OWN_SERVER_RUNNING); then \
|
|
echo "the server this Makefile started (pid $$(cat weed-server.pid)) is still running; run make stop-server first"; \
|
|
exit 1; \
|
|
fi
|
|
@rm -rf weed-server.pid weed-server.started $(SERVER_DIR)
|
|
@mkdir -p $(SERVER_DIR)
|
|
@$(WEED_BINARY) mini \
|
|
-dir=$(SERVER_DIR) \
|
|
-ip=127.0.0.1 \
|
|
-ip.bind=127.0.0.1 \
|
|
-master.port=$(MASTER_PORT) \
|
|
-volume.port=$(VOLUME_PORT) \
|
|
-filer.port=$(FILER_PORT) \
|
|
-webdav.port=$(WEBDAV_PORT) \
|
|
-admin.port=$(ADMIN_PORT) \
|
|
-admin.port.legacy=0 \
|
|
-s3.port=$(S3_PORT) \
|
|
-s3.config=$(S3_CONFIG) \
|
|
-master.peers=none \
|
|
> weed-test.log 2>&1 & echo $$! > weed-server.pid; \
|
|
ps -p $$! -o lstart= > weed-server.started
|
|
@for i in $$(seq 1 60); do \
|
|
kill -0 $$(cat weed-server.pid) 2>/dev/null || { \
|
|
echo "SeaweedFS server exited during startup"; \
|
|
cat weed-test.log; \
|
|
exit 1; \
|
|
}; \
|
|
if curl -sS --max-time 2 -o /dev/null http://127.0.0.1:$(S3_PORT)/ >/dev/null 2>&1; then \
|
|
echo "SeaweedFS S3 server is ready on port $(S3_PORT)"; \
|
|
exit 0; \
|
|
fi; \
|
|
sleep 1; \
|
|
done; \
|
|
echo "SeaweedFS S3 server failed to start"; \
|
|
cat weed-test.log; \
|
|
$(MAKE) --no-print-directory stop-server; \
|
|
exit 1
|
|
|
|
stop-server:
|
|
@if [ -f weed-server.pid ]; then \
|
|
if $(OWN_SERVER_RUNNING); then \
|
|
kill $$(cat weed-server.pid) 2>/dev/null || true; \
|
|
for i in $$(seq 1 30); do $(OWN_SERVER_RUNNING) || break; sleep 1; done; \
|
|
if $(OWN_SERVER_RUNNING); then \
|
|
echo "the server this Makefile started (pid $$(cat weed-server.pid)) is still running after 30s; keeping weed-server.pid"; \
|
|
exit 1; \
|
|
fi; \
|
|
fi; \
|
|
rm -f weed-server.pid weed-server.started; \
|
|
fi
|
|
|
|
run-suite: build-suite
|
|
SEAWEEDFS_S3_ENDPOINT=http://127.0.0.1:$(S3_PORT) \
|
|
SEAWEEDFS_S3_ACCESS_KEY_ID=admin \
|
|
SEAWEEDFS_S3_SECRET_ACCESS_KEY=admin \
|
|
./compat-runner --providers=seaweedfs --output=results.json
|
|
python3 check_results.py results.json expected_failures.txt
|
|
|
|
test-with-server:
|
|
@$(MAKE) start-server || exit $$?; \
|
|
$(MAKE) run-suite; TEST_RESULT=$$?; \
|
|
$(MAKE) stop-server; exit $$TEST_RESULT
|
|
|
|
logs:
|
|
@test -f weed-test.log && tail -100 weed-test.log || echo "No log file found"
|
|
|
|
clean: stop-server
|
|
@rm -rf $(SERVER_DIR) $(SUITE_DIR) compat-runner results.json weed-test.log weed-server.pid weed-server.started
|