mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-13 17:48:29 +02:00
Add comprehensive Docker Compose setup for Kafka integration tests
MAJOR ENHANCEMENT: Complete Docker-based integration testing infrastructure ## New Docker Compose Infrastructure: - docker-compose.yml: Complete multi-service setup with health checks - Apache Kafka + Zookeeper - Confluent Schema Registry - SeaweedFS full stack (Master, Volume, Filer, MQ Broker, MQ Agent) - Kafka Gateway service - Test setup and utility services ## Docker Services: - Dockerfile.kafka-gateway: Custom Kafka Gateway container - Dockerfile.test-setup: Schema registration and test data setup - kafka-gateway-start.sh: Service startup script with dependency waiting - wait-for-services.sh: Comprehensive service readiness verification ## Test Setup Utility: - cmd/setup/main.go: Automated schema registration utility - Registers User, UserEvent, and LogEntry Avro schemas - Handles service discovery and health checking ## Integration Tests: - docker_integration_test.go: Comprehensive Docker-based integration tests - Kafka connectivity and topic operations - Schema Registry integration - Kafka Gateway functionality - Sarama and kafka-go client compatibility - Cross-client message compatibility - Performance benchmarking ## Build and Test Infrastructure: - Makefile: 30+ targets for development and testing - setup, test-unit, test-integration, test-e2e - Performance testing and benchmarking - Individual service management - Debugging and monitoring tools - CI/CD integration targets ## Documentation: - README.md: Comprehensive documentation - Architecture overview and service descriptions - Quick start guide and development workflow - Troubleshooting and performance tuning - CI/CD integration examples ## Key Features: ✅ Complete service orchestration with health checks ✅ Automated schema registration and test data setup ✅ Multi-client compatibility testing (Sarama, kafka-go) ✅ Performance benchmarking and monitoring ✅ Development-friendly debugging tools ✅ CI/CD ready with proper cleanup ✅ Comprehensive documentation and examples ## Usage: make setup-schemas # Start all services and register schemas make test-e2e # Run end-to-end integration tests make clean # Clean up environment This provides a production-ready testing infrastructure that ensures Kafka Gateway compatibility with real Kafka ecosystems and validates schema registry integration in realistic deployment scenarios.
This commit is contained in:
chrislu committed
1 parent
e70421bb81
commit
00a672d12e
9 files changed
+1739
No files matched your search
@@ -0,0 +1,355 @@
|
||||
# Kafka Integration Testing with Docker Compose
|
||||
|
||||
This directory contains comprehensive integration tests for SeaweedFS Kafka Gateway using Docker Compose to set up all required dependencies.
|
||||
|
||||
## Overview
|
||||
|
||||
The Docker Compose setup provides:
|
||||
- **Apache Kafka** with Zookeeper
|
||||
- **Confluent Schema Registry** for schema management
|
||||
- **SeaweedFS** complete stack (Master, Volume, Filer, MQ Broker, MQ Agent)
|
||||
- **Kafka Gateway** that bridges Kafka protocol to SeaweedFS MQ
|
||||
- **Test utilities** for schema registration and data setup
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Docker and Docker Compose
|
||||
- Go 1.21+
|
||||
- Make (optional, for convenience)
|
||||
|
||||
### Basic Usage
|
||||
|
||||
1. **Start the environment:**
|
||||
```bash
|
||||
make setup-schemas
|
||||
```
|
||||
This starts all services and registers test schemas.
|
||||
|
||||
2. **Run integration tests:**
|
||||
```bash
|
||||
make test-integration
|
||||
```
|
||||
|
||||
3. **Run end-to-end tests:**
|
||||
```bash
|
||||
make test-e2e
|
||||
```
|
||||
|
||||
4. **Clean up:**
|
||||
```bash
|
||||
make clean
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐
|
||||
│ Kafka Client │───▶│ Kafka Gateway │───▶│ SeaweedFS MQ │
|
||||
│ (Sarama/ │ │ (Port 9093) │ │ (Broker/Agent)│
|
||||
│ kafka-go) │ │ │ │ │
|
||||
└─────────────────┘ └──────────────────┘ └─────────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────┐
|
||||
│ Schema Registry │
|
||||
│ (Port 8081) │
|
||||
└──────────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────┐
|
||||
│ Native Kafka │
|
||||
│ (Port 9092) │
|
||||
└──────────────────┘
|
||||
```
|
||||
|
||||
## Services
|
||||
|
||||
### Core Services
|
||||
|
||||
| Service | Port | Description |
|
||||
|---------|------|-------------|
|
||||
| Zookeeper | 2181 | Kafka coordination |
|
||||
| Kafka | 9092 | Native Kafka broker |
|
||||
| Schema Registry | 8081 | Confluent Schema Registry |
|
||||
| SeaweedFS Master | 9333 | SeaweedFS master server |
|
||||
| SeaweedFS Volume | 8080 | SeaweedFS volume server |
|
||||
| SeaweedFS Filer | 8888 | SeaweedFS filer server |
|
||||
| SeaweedFS MQ Broker | 17777 | SeaweedFS message queue broker |
|
||||
| SeaweedFS MQ Agent | 16777 | SeaweedFS message queue agent |
|
||||
| Kafka Gateway | 9093 | Kafka protocol gateway to SeaweedFS |
|
||||
|
||||
### Test Services
|
||||
|
||||
| Service | Description |
|
||||
|---------|-------------|
|
||||
| test-setup | Registers schemas and sets up test data |
|
||||
| kafka-producer | Creates topics and produces test messages |
|
||||
| kafka-consumer | Consumes messages for verification |
|
||||
|
||||
## Available Tests
|
||||
|
||||
### Unit Tests
|
||||
```bash
|
||||
make test-unit
|
||||
```
|
||||
Tests individual Kafka components without external dependencies.
|
||||
|
||||
### Integration Tests
|
||||
```bash
|
||||
make test-integration
|
||||
```
|
||||
Tests with real Kafka, Schema Registry, and SeaweedFS services.
|
||||
|
||||
### Schema Tests
|
||||
```bash
|
||||
make test-schema
|
||||
```
|
||||
Tests schema registry integration and schema evolution.
|
||||
|
||||
### End-to-End Tests
|
||||
```bash
|
||||
make test-e2e
|
||||
```
|
||||
Complete workflow tests with all services running.
|
||||
|
||||
### Performance Tests
|
||||
```bash
|
||||
make test-performance
|
||||
```
|
||||
Benchmarks and performance measurements.
|
||||
|
||||
### Client-Specific Tests
|
||||
```bash
|
||||
make test-sarama # IBM Sarama client tests
|
||||
make test-kafka-go # Segmentio kafka-go client tests
|
||||
```
|
||||
|
||||
## Development Workflow
|
||||
|
||||
### Start Individual Components
|
||||
|
||||
```bash
|
||||
# Start only Kafka ecosystem
|
||||
make dev-kafka
|
||||
|
||||
# Start only SeaweedFS
|
||||
make dev-seaweedfs
|
||||
|
||||
# Start Kafka Gateway
|
||||
make dev-gateway
|
||||
```
|
||||
|
||||
### Debugging
|
||||
|
||||
```bash
|
||||
# Show all service logs
|
||||
make logs
|
||||
|
||||
# Show specific service logs
|
||||
make logs-kafka
|
||||
make logs-schema-registry
|
||||
make logs-seaweedfs
|
||||
make logs-gateway
|
||||
|
||||
# Check service status
|
||||
make status
|
||||
|
||||
# Debug environment
|
||||
make debug
|
||||
```
|
||||
|
||||
### Interactive Testing
|
||||
|
||||
```bash
|
||||
# List Kafka topics
|
||||
make topics
|
||||
|
||||
# Create a topic
|
||||
make create-topic TOPIC=my-test-topic
|
||||
|
||||
# Produce messages interactively
|
||||
make produce TOPIC=my-test-topic
|
||||
|
||||
# Consume messages
|
||||
make consume TOPIC=my-test-topic
|
||||
|
||||
# Open shell in containers
|
||||
make shell-kafka
|
||||
make shell-gateway
|
||||
```
|
||||
|
||||
## Test Configuration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `KAFKA_BOOTSTRAP_SERVERS` | `localhost:9092` | Kafka broker addresses |
|
||||
| `KAFKA_GATEWAY_URL` | `localhost:9093` | Kafka Gateway address |
|
||||
| `SCHEMA_REGISTRY_URL` | `http://localhost:8081` | Schema Registry URL |
|
||||
| `TEST_TIMEOUT` | `10m` | Test timeout duration |
|
||||
|
||||
### Running Tests with Custom Configuration
|
||||
|
||||
```bash
|
||||
KAFKA_BOOTSTRAP_SERVERS=localhost:9092 \
|
||||
KAFKA_GATEWAY_URL=localhost:9093 \
|
||||
SCHEMA_REGISTRY_URL=http://localhost:8081 \
|
||||
go test -v ./test/kafka/ -run Integration
|
||||
```
|
||||
|
||||
## Test Schemas
|
||||
|
||||
The setup automatically registers these test schemas:
|
||||
|
||||
### User Schema
|
||||
```json
|
||||
{
|
||||
"type": "record",
|
||||
"name": "User",
|
||||
"fields": [
|
||||
{"name": "id", "type": "int"},
|
||||
{"name": "name", "type": "string"},
|
||||
{"name": "email", "type": ["null", "string"], "default": null}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### User Event Schema
|
||||
```json
|
||||
{
|
||||
"type": "record",
|
||||
"name": "UserEvent",
|
||||
"fields": [
|
||||
{"name": "userId", "type": "int"},
|
||||
{"name": "eventType", "type": "string"},
|
||||
{"name": "timestamp", "type": "long"},
|
||||
{"name": "data", "type": ["null", "string"], "default": null}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Log Entry Schema
|
||||
```json
|
||||
{
|
||||
"type": "record",
|
||||
"name": "LogEntry",
|
||||
"fields": [
|
||||
{"name": "level", "type": "string"},
|
||||
{"name": "message", "type": "string"},
|
||||
{"name": "timestamp", "type": "long"},
|
||||
{"name": "service", "type": "string"},
|
||||
{"name": "metadata", "type": {"type": "map", "values": "string"}}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **Services not starting:**
|
||||
```bash
|
||||
make status
|
||||
make debug
|
||||
```
|
||||
|
||||
2. **Port conflicts:**
|
||||
- Check if ports 2181, 8081, 9092, 9093, etc. are available
|
||||
- Modify `docker-compose.yml` to use different ports if needed
|
||||
|
||||
3. **Schema Registry connection issues:**
|
||||
```bash
|
||||
curl http://localhost:8081/subjects
|
||||
make logs-schema-registry
|
||||
```
|
||||
|
||||
4. **Kafka Gateway not responding:**
|
||||
```bash
|
||||
nc -z localhost 9093
|
||||
make logs-gateway
|
||||
```
|
||||
|
||||
5. **Test timeouts:**
|
||||
- Increase `TEST_TIMEOUT` environment variable
|
||||
- Check service health with `make status`
|
||||
|
||||
### Performance Tuning
|
||||
|
||||
For better performance in testing:
|
||||
|
||||
1. **Increase Docker resources:**
|
||||
- Memory: 4GB+
|
||||
- CPU: 2+ cores
|
||||
|
||||
2. **Adjust Kafka settings:**
|
||||
- Modify `docker-compose.yml` Kafka environment variables
|
||||
- Tune partition counts and replication factors
|
||||
|
||||
3. **SeaweedFS optimization:**
|
||||
- Adjust volume size limits
|
||||
- Configure appropriate replication settings
|
||||
|
||||
## CI/CD Integration
|
||||
|
||||
### GitHub Actions Example
|
||||
|
||||
```yaml
|
||||
name: Kafka Integration Tests
|
||||
|
||||
on: [push, pull_request]
|
||||
|
||||
jobs:
|
||||
kafka-integration:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
- uses: actions/setup-go@v3
|
||||
with:
|
||||
go-version: '1.21'
|
||||
- name: Run Kafka Integration Tests
|
||||
run: |
|
||||
cd test/kafka
|
||||
make ci-test
|
||||
```
|
||||
|
||||
### Local CI Testing
|
||||
|
||||
```bash
|
||||
# Run full CI test suite
|
||||
make ci-test
|
||||
|
||||
# Run end-to-end CI tests
|
||||
make ci-e2e
|
||||
```
|
||||
|
||||
## Contributing
|
||||
|
||||
When adding new tests:
|
||||
|
||||
1. **Follow naming conventions:**
|
||||
- Integration tests: `TestDockerIntegration_*`
|
||||
- Unit tests: `Test*_Unit`
|
||||
- Performance tests: `TestDockerIntegration_Performance`
|
||||
|
||||
2. **Use environment variables:**
|
||||
- Check for required environment variables
|
||||
- Skip tests gracefully if dependencies unavailable
|
||||
|
||||
3. **Clean up resources:**
|
||||
- Use unique topic names with timestamps
|
||||
- Clean up test data after tests
|
||||
|
||||
4. **Add documentation:**
|
||||
- Update this README for new test categories
|
||||
- Document any new environment variables or configuration
|
||||
|
||||
## References
|
||||
|
||||
- [Apache Kafka Documentation](https://kafka.apache.org/documentation/)
|
||||
- [Confluent Schema Registry](https://docs.confluent.io/platform/current/schema-registry/index.html)
|
||||
- [SeaweedFS Documentation](https://github.com/seaweedfs/seaweedfs/wiki)
|
||||
- [IBM Sarama Kafka Client](https://github.com/IBM/sarama)
|
||||
- [Segmentio kafka-go Client](https://github.com/segmentio/kafka-go)
|
||||
Reference in new issue
Block a user