0 Fork 0

mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-10-24 23:10:41 +02:00

Files

T

Chris Lu aeb5db1b13 tests: pin weed mini admin ports and disable unused aux services (#11997 )

* tests: pin weed mini admin ports and disable unused aux services

Every weed mini that does not configure them still binds the fixed
default ports for the admin server (12535 + derived 22535 gRPC), its
legacy compat listener (23646 + 33646), WebDAV (7333), Lance (9101) and
the Iceberg REST catalog (8181). Sequential or concurrent minis on one
host then collide on them: the port check warns and shifts, and the
admin's late bind can land on a stolen ephemeral port, adding noise and
flakiness to test logs (e.g. run 40017181958, job 120659267479).

Add testutil.MiniAdminArgs to pin the admin server to an allocated
mini-style port (HTTP + port+10000 gRPC, both below the Linux ephemeral
port floor since admin binds late) and disable the legacy compat
listener. Every test that spawns weed mini now also disables the aux
services it does not use: WebDAV, Lance and the Iceberg REST endpoint
get port=0 unless the test actually exercises them, in which case their
ports stay pinned as before.

* tests: take mini admin ports from the below-ephemeral range

* tests: reuse MiniAdminArgs for the remote-s3 fetch mini

* tests: fully disable webdav and co-allocate mini admin ports

- -webdav.port=0 only makes the service bind a random port; add
-webdav=false so minis that do not test WebDAV skip it entirely.
- Spark, RisingWave and s3 Spark launchers allocated the admin port in
a second MustFreeMiniPort call, which could pick a port already
handed out by the first batch. Fold 'Admin' into the same
MustFreeMiniPorts call.
- test/fuse_integration is a separate Go module and cannot import
testutil; allocate the admin ports with its local freePort helper
instead.

* test/fuse: derive the mini admin gRPC port from its HTTP port

Two independent freePort calls could return the same port, making admin.port equal admin.port.grpc (or collide with the filer port). Allocate once and derive the gRPC port with the +10000 convention.

* test/fuse: pin the mini admin port in the fuse_test framework copy

The fuse_test package framework spawns weed mini too; without a pinned admin port it still binds the fixed defaults 12535/23646 and collides with other minis.

* tests: drop the redundant -webdav.port=0 flag

-webdav=false alone disables the service; the port flag is dead config.

2026-10-23 15:38:09 -07:00

append_rows.py

s3tables: add ClickHouse iceberg catalog integration test (#10637 )

2026-08-07 22:39:50 -07:00

clickhouse_catalog_test.go

tests: pin weed mini admin ports and disable unused aux services (#11997 )

2026-10-23 15:38:09 -07:00

Dockerfile.writer

iceberg: repair non-compliant manifests at commit (#10641 )

2026-08-08 21:24:37 -07:00

read_rows.py

iceberg: repair non-compliant manifests at commit (#10641 )

2026-08-08 21:24:37 -07:00

README.md

clickhouse catalog test: cover latest ClickHouse and catalog-side CREATE TABLE (#10707 )

2026-08-10 20:04:33 -07:00

README.md

ClickHouse Iceberg Catalog Integration Test

This directory contains a ClickHouse integration smoke test for SeaweedFS's Iceberg REST Catalog implementation, using ClickHouse's DataLakeCatalog database engine.

What It Tests

TestClickHouseIcebergCatalog verifies the ClickHouse path end to end:

Starts a local SeaweedFS mini cluster with S3 Tables and Iceberg REST enabled.
Creates a SeaweedFS table bucket.
Creates an Iceberg namespace and an empty table through the SeaweedFS REST catalog OAuth flow.
Creates a second table and populates it with three rows by running a PyIceberg writer container (Dockerfile.writer + append_rows.py) before ClickHouse connects, so the snapshot is part of the catalog's first scan.
Starts the ClickHouse server container (clickhouse/clickhouse-server:25.8 by default, overridable via CLICKHOUSE_IMAGE) and waits for the HTTP interface.
Attaches the catalog with CREATE DATABASE ... ENGINE = DataLakeCatalog (catalog_type = 'rest'), authenticating to the catalog via the OAuth2 client-credentials flow (catalog_credential + oauth_server_uri) and to S3 via the engine's access/secret key arguments and storage_endpoint.
Runs subtests against the SeaweedFS-backed Iceberg tables:
- BasicSelect: ClickHouse is alive and answering SQL.
- DatabaseVisible: the DataLakeCatalog database exists.
- TableVisible: seeded tables appear as namespace.table entries in SHOW TABLES (ClickHouse flattens Iceberg namespaces into table names).
- DescribeTable: the Iceberg schema mapped to id Int64 and label Nullable(String). Failure here means ClickHouse could not parse the schema returned by the SeaweedFS catalog.
- CountEmptyTable: catalog-to-table resolution and a scan of an empty table.
- ReadWrittenDataCount and ReadWrittenDataValues: ClickHouse reads back the three PyIceberg-appended rows and the values match. This exercises the actual data path (parquet reads via S3), not just metadata.
- WriteReadBack: ClickHouse inserts rows with its experimental Iceberg write support using default settings, which produces manifests without avro field-ids, bucket-relative paths, and parquet without field ids. The SeaweedFS catalog repairs the manifests at commit time and stamps a default name mapping on the table, so PyIceberg (read_rows.py, a strict reader) must return the rows ClickHouse wrote.
- CreateTableViaCatalog: ClickHouse creates a table through the catalog (CREATE TABLE ... ENGINE = IcebergS3(...) with write_full_path_in_iceberg_metadata = 1), the test verifies it is registered in the REST catalog, inserts rows, reads them back, and has PyIceberg read them too. ClickHouse only issues the catalog createTable from 26.4, so the subtest skips on older servers.

Queries go through ClickHouse's HTTP interface (port 8123, mapped to a dynamically allocated host port), so the test needs no ClickHouse client driver. Tables are referenced as iceberg_catalog.`namespace.table` .

Running Locally

Build or install weed, then run:

cd test/s3tables/catalog_clickhouse
go test -v -timeout 20m .

The ClickHouse image defaults to clickhouse/clickhouse-server:25.8 and can be overridden:

CLICKHOUSE_IMAGE=clickhouse/clickhouse-server:latest go test -v -timeout 20m .

The test requires Docker. The GitHub Actions job runs on ubuntu-22.04 and executes the test for pull requests against both the pinned baseline image and latest, so new ClickHouse releases are covered as they ship.

Configuration

The test uses these fixed credentials for the local SeaweedFS IAM config:

S3 access key: AKIAIOSFODNN7EXAMPLE
S3 secret key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Region: us-west-2
Warehouse bucket: iceberg-tables

ClickHouse ports:

Only 8123 (HTTP interface) is mapped to a host port (allocated dynamically) so the Go test can issue queries from the test process.
The Iceberg REST endpoint and the S3 endpoint are reached from inside the ClickHouse container via host.docker.internal, matching the Doris, Trino, and Dremio test paths.

Troubleshooting

Ensure Docker is running: docker version
Ensure weed is built or available on PATH
DataLakeCatalog requires allow_experimental_database_iceberg = 1; the test passes it as a URL setting on the CREATE DATABASE request.
Container logs are printed in the failure message; you can also check docker logs <seaweed-clickhouse-...> while the test is running.