Root cause: With -max=0 (unlimited volumes), volumes are created on-demand,
but no volumes existed when tests started, causing first write to fail.
Solution:
- Explicitly trigger volume growth via /vol/grow API
- Create 3 volumes with replication=000 before running tests
- Verify volumes exist before proceeding
- Fail early with clear message if volumes can't be created
Changes:
- POST to http://localhost:9333/vol/grow?replication=000&count=3
- Wait up to 10 seconds for volumes to appear
- Show volume count and layout status
- Exit with error if no volumes after 10 attempts
- Applied to both spark-tests and spark-example jobs
This ensures writable volumes exist before Spark tries to write data.
- Add 'weed shell' execution to run 'volume.list' on failure
- Shows which volumes exist, their status, and available space
- Add cluster status JSON output for detailed topology view
- Helps diagnose volume allocation issues and full volumes
- Added to both spark-tests and spark-example jobs
- Diagnostic runs only when tests fail (if: failure())
- Add 'docker compose down -v' before starting services to clean up stale volumes
- Prevents accumulation of data/buckets from previous test runs
- Add volume registration verification after service startup
- Check that volume server has registered with master and volumes are available
- Helps diagnose 'No writable volumes' errors
- Shows volume count and waits up to 30 seconds for volumes to be created
- Both spark-tests and spark-example jobs updated with same improvements
- Add -Dcentral.publishing.skip=true to all Maven builds
- Central publishing plugin is only needed for Maven Central releases
- Prevents plugin resolution errors during CI builds
- Complements existing -Dgpg.skip=true flag
- Add container status (docker compose ps -a) on startup failure
- Add detailed logs for all three services (master, volume, filer)
- Add container inspection to verify binary exists
- Add debugging info for spark-example job
- Helps diagnose startup failures before containers are torn down
- Add ls -la to show build-artifacts/docker/ contents
- Add file command to verify binary type
- Add --no-cache to docker compose build to prevent stale cache issues
- Ensures fresh build with current binary
- Download artifacts to 'build-artifacts' directory instead of '.'
- Prevents checkout from overwriting downloaded files
- Explicitly copy weed binary from build-artifacts to docker/ directory
- Update Maven artifact restoration to use new path
- Add step to chmod +x the weed binary after downloading artifacts
- Artifacts lose executable permissions during upload/download
- Prevents 'Permission denied' errors when Docker tries to run the binary
- Add step to restore Maven artifacts from download to ~/.m2/repository
- Restructure artifact upload to use consistent directory layout
- Remove obsolete 'version' field from docker-compose.yml to eliminate warnings
- Ensures SeaweedFS Java dependencies are available during test execution
1. Add explicit permissions (least privilege):
- contents: read
- checks: write (for test reports)
- pull-requests: write (for PR comments)
2. Extract duplicate build steps into shared 'build-deps' job:
- Eliminates duplication between spark-tests and spark-example
- Build artifacts are uploaded and reused by dependent jobs
- Reduces CI time and ensures consistency
3. Fix spark-example service startup verification:
- Match robust approach from spark-tests job
- Add explicit timeout and failure handling
- Verify all services (master, volume, filer)
- Include diagnostic logging on failure
- Prevents silent failures and obscure errors
These changes improve maintainability, security, and reliability
of the Spark integration test workflow.
chrislu