seaweedfs/test/s3tables/catalog_dremio/IMPLEMENTATION.md

Dremio Integration Test Implementation

Overview

This implementation adds comprehensive integration testing for Dremio with SeaweedFS's Iceberg REST Catalog, following the same patterns as existing Spark and Trino integration tests.

Files Created

Test Files

1. dremio_catalog_test.go (13.2 KB)

   • Core test environment setup and initialization
   • TestDremioIcebergCatalog: Basic catalog connectivity and schema operations
   • TestDremioTableOperations: Table creation, insertion, and querying
   • TestEnvironment struct: Manages SeaweedFS mini instance and Dremio container
   • Helper functions for:
     • Starting SeaweedFS with all necessary services
     • Writing Dremio configuration files
     • Starting and managing Dremio Docker containers
     • Waiting for service readiness
     • Executing Dremio SQL commands
     • Creating S3 table buckets

2. dremio_crud_operations_test.go (4.5 KB)

   • TestSchemaCRUD: Schema creation, listing, and deletion
   • TestTableCRUD: Table creation, listing, insertion, and deletion
   • TestDataInsertAndQuery: Complex queries with WHERE, GROUP BY, and aggregations
   • setupDremioTest: Common test environment initialization helper

3. dremio_deterministic_location_test.go (3.7 KB)

   • TestDeterministicTableLocation: Explicit table location specification
   • TestMultiLevelNamespace: Multi-level namespace support (e.g., "analytics.daily")
   • Verifies correct S3 path handling for namespaces

4. README.md (3.9 KB)

   • Comprehensive documentation
   • Prerequisites and setup instructions
   • Test file descriptions
   • How to run tests locally
   • Test scenario explanations
   • Troubleshooting guide
   • CI/CD integration details

5. IMPLEMENTATION.md (this file)

   • Implementation details and design decisions

CI/CD Integration

Updated .github/workflows/s3-tables-tests.yml:

• Added new job: dremio-iceberg-catalog-tests
• Pre-pulls latest Dremio Docker image
• Runs with 30-minute timeout (25 minutes for tests)
• Uploads test artifacts on failure
• Integrated alongside existing Trino, Spark, and other catalog tests

Design Decisions

Architecture

1. Test Environment Pattern

   • Follows the Trino integration test pattern (vs Spark's testcontainers approach)
   • Uses raw Docker commands for container management
   • Allocates random ports to avoid conflicts
   • Manages SeaweedFS process directly using exec.Cmd

2. SQL Execution

   • Uses Dremio REST API (/api/v3/sql endpoint)
   • Executes via docker exec with curl
   • Parses output using jq for JSON extraction
   • Graceful handling of format variations

3. Configuration

   • Dremio config file format supports REST catalog integration
   • AWS SigV4 authentication (matching Trino pattern)
   • S3 path-style access enabled for compatibility
   • Warehouse bucket: iceberg-tables

4. Service Readiness

   • Iceberg REST API: HTTP health check on /v1/config
   • Dremio: Curl-based ping check on /api/v2/ping
   • Timeout: 120 seconds for Dremio (longer than Trino due to startup time)

Key Features

1. Complete CRUD Testing

   • Schema creation, listing, deletion
   • Table creation with explicit locations
   • Data insertion with bulk operations
   • Data querying with WHERE, GROUP BY, aggregations

2. Namespace Support

   • Single-level namespaces (basic case)
   • Multi-level namespaces with dot-separation
   • Deterministic table locations

3. Error Handling

   • Graceful cleanup of resources
   • Temporary directory management
   • Docker container cleanup on test completion

4. Logging

   • Comprehensive debug output
   • Service startup verification
   • SQL command logging
   • Port allocation diagnostics

Implementation Notes

Differences from Trino Tests

1. Container Management

   • Similar approach: raw Docker commands (not testcontainers)
   • Uses host.docker.internal for networking (same as Trino)

2. Configuration Format

   • Dremio uses JSON-based config (vs Trino .properties files)
   • Simpler configuration for single catalog setup

3. SQL Endpoint

   • Dremio: REST API with JSON request/response
   • Trino: CLI-based execution via docker exec

4. Startup Time

   • Dremio: ~120 seconds (longer initialization)
   • Trino: ~60 seconds

Potential Enhancements

1. Additional Test Scenarios

   • Parquet file format verification
   • ORC file format support
   • Schema evolution testing
   • Partition projection testing
   • Time-travel queries (Iceberg-specific)

2. Performance Testing

   • Large table operations
   • Concurrent access patterns
   • Query performance benchmarks

3. Error Scenarios

   • Invalid table creation
   • Concurrent modifications
   • Network failure handling

4. Advanced Features

   • Metastore security testing
   • Audit log verification
   • Snapshot browsing
   • Manifest file handling

Testing the Implementation

Local Testing

cd test/s3tables/catalog_dremio

# Run all tests
go test -v -timeout 20m ./...

# Run specific test
go test -v -run TestDremioIcebergCatalog

# Run tests skipping integration tests
go test -short ./...

CI/CD Testing

Tests are automatically triggered on:

• Pull requests
• Changes to test files
• Changes to S3/catalog implementation

Troubleshooting

Common Issues

1. Port Allocation Failures

   • Solution: Close other services or reboot
   • The tests allocate 9 ports dynamically

2. Dremio Container Timeout

   • Dremio has longer startup time than Trino
   • Increase timeout if needed: 120 seconds default

3. Iceberg REST API Not Ready

   • Verify SeaweedFS started correctly
   • Check Iceberg port is listening: netstat -tlnp | grep :8080

4. SQL Execution Failures

   • Check Dremio container logs: docker logs <container>
   • Verify S3 credentials in configuration

Related Files

• .github/workflows/s3-tables-tests.yml - CI/CD pipeline
• test/s3tables/catalog_trino/ - Similar Trino tests (reference)
• test/s3tables/catalog_spark/ - Spark tests (reference)
• test/s3tables/catalog/ - Base Iceberg catalog tests
• test/testutil/ - Shared testing utilities

Version Compatibility

• Go: 1.18+
• Docker: Latest stable version
• Dremio: Latest (tested with dremio/dremio:latest)
• SeaweedFS: Current master branch
• Iceberg: Via REST catalog (version-agnostic)