Files
seaweedfs/.github/workflows/java_integration_tests.yml
T
Chris LuandDevin 3e5aa8c3b3 topology: mark replica quarantined before recording the pair (#11983)
* topology: serialize quarantine records with their layout marks

A quarantine publishes a (node, vid) record and marks the replica read-only at the layout level; release and heartbeat flag updates must observe that pair as one state. Store-then-mark let a concurrent release delete the record before the mark landed, stranding the surviving replica read-only; mark-then-store let a heartbeat clear the mark before the record was visible, returning a conflicting replica to the writable list.

quarantineLock now covers every record<->mark pair: flagDuplicateVolumeId's quarantine, releaseResolvedQuarantine's undo, the isQuarantined-gated marks on registration, and the quarantine-folded read-only flag writes on heartbeat paths. Remote server pushes stay outside the lock; a push that lands after the record was released is undone, and a failed release push re-adds the record for a later retry.

The release test now waits for in-flight async flags to settle instead of asserting on a single read.

* topology: fold quarantine into RegisterVolume's flag rebuild

RegisterVolume rewrites every replica's location read-only flag from what
its node last reported, and reported flags deliberately exclude quarantine.
Any registration of a shared vid -- a reconnected replica, a new copy, the
self-heal and delta-heartbeat re-register paths -- therefore cleared the
quarantine mark on every replica in the location, while the gate afterwards
only re-marked the registering node. Two call sites had no gate at all.

RegisterVolume now takes an isQuarantined predicate and evaluates it under
accessLock for every replica it touches, so the mark is never dropped in
the first place: a release unmarks after the rebuild, and a quarantine
mark re-applies on the next rebuild, with no window between.

* topology: keep quarantine when a vacuum commit re-adds a replica

SetVolumeAvailable re-adds the replica into the location list; a re-added
slot starts with a cleared read-only flag, so a quarantined replica came
back flag-free and the next ensureCorrectWritables could return a
conflicting volume to the writable list. Pass the quarantine state in and
set the flag back under the same accessLock, with the caller holding
quarantineLock so release cannot interleave between the check and the mark.

* topology: merge quarantine records instead of overwriting them

enforceQuarantine wrote back the state copy it loaded before its remote
push, so a concurrent writer's enforced=true was lost: the server stayed
read-only while the record said unenforced, and a later release skipped
the writable push it owed. The write-back now reloads the record under
quarantineLock and merges -- enforced wins from either side, reported is
taken from the newer pre-enforcement observation only while no push has
landed yet.

All three compensating writable pushes had the same hole in reverse: a
failed un-quarantine left the server read-only with no record to retry
from, and release's restore blindly overwrote any record a fresh flag
pass had created in between. They now go through restoreQuarantineRecord,
which adds the record only when absent and otherwise sets enforced on
the current one, preserving its reported flag.

* topology: hold quarantineLock over registration's quarantine check-and-mark

RegisterVolume reads the quarantine record and marks the replica
read-only under accessLock, but the pair ran without quarantineLock.
A release could push writable, watch the record get re-read, and then
delete it: the replica stayed read-only with no record left to clear,
and unchanged writable heartbeats skip SetReplicaReadOnlyFlag so the
mark never repaired. Holding quarantineLock around the check-and-mark
in all three production callers serializes it against the release
clear-and-delete.

* topology: serialize remote quarantine pushes behind a record recheck

releaseResolvedQuarantine deletes the record under quarantineLock and
sends the writable request after unlock. A conflict flagged while that
request is in flight creates a fresh record and marks the replica
read-only again, and the older writable request can still land last,
leaving the server writable under an active quarantine. The reverse
flap strands a stale read-only mark the same way.

Route every QuarantineVolumeFn call through pushQuarantine, which
holds quarantinePushLock across the remote call and re-checks the
record under quarantineLock right before sending: a read-only push
requires the pair to exist, a writable push requires it to be gone.
A skipped push means the decision already moved on, so the final
server state always follows the newest record.

* topology: serialize quarantine pushes per replica, not globally

A single quarantinePushLock held across the remote mark call let one slow
volume server (up to the 10s client timeout) delay pushes for unrelated
replicas. Keying the lock by (node, vid) keeps pushes to one replica
ordered in record-decision order while pushes to others run in parallel.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* topology: restore the layout read-only mark with a released record

A failed writable push re-armed the quarantine record but left the
replica marked writable in the layout, so PickForWrite could route to a
replica the server still holds read-only.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* topology: wait out async duplicate checks in the release-retry test

TestFailedReleasePushRestoresQuarantineMark asserts layout marks around a
release pass, but the two RegisterVolumeLayout calls each start a
background flagDuplicateVolumeId that can still be mid-flight. A late
mark could land after the final release and fail the writable assertion
even though the code behaved correctly. Track the async passes in a
WaitGroup and drain them before the explicit flag call so the rest of
the test is deterministic.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* topology: keep quarantine when a replica's layout key is unreadable

* topology: keep a failed writable push's restore ordered under the push lock

* topology: retry the quarantine release on every duplicate check

* ci: isolate Java and S3 test servers on persistent runners

* ci: drop the duplicated -ip flags in the lifecycle test server

The bind flags were already on the command; the isolation pass added a
second pair.

* topology: retry a failed quarantine writable push on the next report

* topology: retry pending quarantine releases off the leader pulse

A failed writable push re-arms the quarantine record with a retry that only
enforceQuarantine fires, and that runs only for volumes a heartbeat carries.
Once the server has reported the replica read-only it drops out of the delta
reports, and restoreQuarantineRecord keeps the stored flag matching the
server's so the digests agree and no full list is requested - the pending
release then never retries and the idle volume stays read-only forever. Run
the same release pass from the leader's pulse loop so a pending retry no
longer waits on a report.

* topology: exercise the pulse-driven quarantine release retry

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 07:00:45 -07:00

200 lines
6.4 KiB
YAML

name: Java Client Integration Tests
on:
push:
branches: [ master ]
paths:
- 'other/java/**'
- 'weed/**'
- '.github/workflows/java_integration_tests.yml'
pull_request:
branches: [ master ]
paths:
- 'other/java/**'
- 'weed/**'
- '.github/workflows/java_integration_tests.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || (github.event_name == 'push' && github.ref) || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || (github.event_name == 'push' && !startsWith(github.ref, 'refs/tags/')) }}
jobs:
test:
name: Java Integration Tests
runs-on: ubuntu-latest
strategy:
matrix:
java: ['11', '17']
steps:
- name: Checkout code
uses: actions/checkout@v7
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
id: go
- name: Set up Java
uses: actions/setup-java@v6
with:
java-version: ${{ matrix.java }}
distribution: 'temurin'
cache: ${{ runner.environment == 'github-hosted' && 'maven' || '' }}
- name: Build SeaweedFS
run: |
cd weed
go install -buildvcs=false
weed version
- name: Start SeaweedFS Server
run: |
# Create clean data directory
export WEED_DATA_DIR="/tmp/seaweedfs-java-tests-$(date +%s)"
mkdir -p "$WEED_DATA_DIR"
# Start SeaweedFS with optimized settings for CI
# Include S3 API for s3copier integration tests
weed server -dir="$WEED_DATA_DIR" \
-ip=127.0.0.1 -ip.bind=127.0.0.1 \
-master.raftHashicorp \
-master.electionTimeout=1s \
-master.volumeSizeLimitMB=100 \
-volume.max=100 \
-volume.preStopSeconds=1 \
-master.peers=none \
-filer -filer.maxMB=64 \
-s3 -s3.port=8333 \
-s3.config="$GITHUB_WORKSPACE/docker/compose/s3.json" \
-s3.allowDeleteBucketNotEmpty=true \
-master.port=9333 \
-volume.port=8080 \
-filer.port=8888 \
-metricsPort=9324 > seaweedfs.log 2>&1 &
SERVER_PID=$!
echo "SERVER_PID=$SERVER_PID" >> $GITHUB_ENV
echo "WEED_DATA_DIR=$WEED_DATA_DIR" >> $GITHUB_ENV
echo "SeaweedFS server started with PID: $SERVER_PID"
- name: Wait for SeaweedFS Components
run: |
echo "Waiting for SeaweedFS components to start..."
# Wait for master
for i in {1..30}; do
if curl -s http://localhost:9333/cluster/status > /dev/null 2>&1; then
echo "✓ Master server is ready"
break
fi
echo "Waiting for master server... ($i/30)"
sleep 2
done
# Wait for volume
for i in {1..30}; do
if curl -s http://localhost:8080/status > /dev/null 2>&1; then
echo "✓ Volume server is ready"
break
fi
echo "Waiting for volume server... ($i/30)"
sleep 2
done
# Wait for filer
for i in {1..30}; do
if curl -s http://localhost:8888/ > /dev/null 2>&1; then
echo "✓ Filer is ready"
break
fi
echo "Waiting for filer... ($i/30)"
sleep 2
done
# Wait for S3 API
for i in {1..30}; do
if curl -s http://localhost:8333/healthz > /dev/null 2>&1; then
echo "✓ S3 API is ready"
break
fi
echo "Waiting for S3 API... ($i/30)"
sleep 2
done
echo "✓ All SeaweedFS components are ready!"
# Display cluster status
echo "Cluster status:"
curl -s http://localhost:9333/cluster/status | head -20
- name: Build and Install SeaweedFS Client
working-directory: other/java/client
run: |
mvn clean install -DskipTests -Dmaven.javadoc.skip=true -Dgpg.skip=true
- name: Run Client Unit Tests
working-directory: other/java/client
run: |
mvn test -Dtest=SeaweedReadTest,SeaweedCipherTest
- name: Run Client Integration Tests
working-directory: other/java/client
env:
SEAWEEDFS_TEST_ENABLED: true
run: |
mvn test -Dtest=*IntegrationTest
- name: Run HDFS3 Configuration Tests
working-directory: other/java/hdfs3
run: |
mvn test -Dtest=SeaweedFileSystemConfigTest -Dmaven.javadoc.skip=true -Dgpg.skip=true
- name: Run S3 ETag Validation Tests (Issue #7768)
working-directory: other/java/s3copier
env:
S3_ENDPOINT: http://127.0.0.1:8333
S3_ACCESS_KEY: some_access_key1
S3_SECRET_KEY: some_secret_key1
run: |
echo "Running S3 ETag validation tests against $S3_ENDPOINT"
mvn test -Dtest=ETagValidationTest \
-DS3_ENDPOINT=$S3_ENDPOINT \
-DS3_ACCESS_KEY=$S3_ACCESS_KEY \
-DS3_SECRET_KEY=$S3_SECRET_KEY \
-Dmaven.javadoc.skip=true -Dgpg.skip=true
- name: Display logs on failure
if: failure()
run: |
echo "=== SeaweedFS Server Log ==="
tail -100 seaweedfs.log || echo "No server log"
echo ""
echo "=== Cluster Status ==="
curl -s http://localhost:9333/cluster/status || echo "Cannot reach cluster"
echo ""
echo "=== Process Status ==="
ps aux | grep weed || echo "No weed processes"
- name: Cleanup
if: always()
run: |
# Stop server using stored PID
if [ -n "$SERVER_PID" ]; then
echo "Stopping SeaweedFS server (PID: $SERVER_PID)"
kill -9 $SERVER_PID 2>/dev/null || true
fi
# Fallback: kill any remaining weed processes
pkill -f "weed server" || true
# Clean up data directory
if [ -n "$WEED_DATA_DIR" ]; then
echo "Cleaning up data directory: $WEED_DATA_DIR"
rm -rf "$WEED_DATA_DIR" || true
fi