mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-24 23:10:41 +02:00
Chris Lu
aeb5db1b13
* tests: pin weed mini admin ports and disable unused aux services Every weed mini that does not configure them still binds the fixed default ports for the admin server (12535 + derived 22535 gRPC), its legacy compat listener (23646 + 33646), WebDAV (7333), Lance (9101) and the Iceberg REST catalog (8181). Sequential or concurrent minis on one host then collide on them: the port check warns and shifts, and the admin's late bind can land on a stolen ephemeral port, adding noise and flakiness to test logs (e.g. run 40017181958, job 120659267479). Add testutil.MiniAdminArgs to pin the admin server to an allocated mini-style port (HTTP + port+10000 gRPC, both below the Linux ephemeral port floor since admin binds late) and disable the legacy compat listener. Every test that spawns weed mini now also disables the aux services it does not use: WebDAV, Lance and the Iceberg REST endpoint get port=0 unless the test actually exercises them, in which case their ports stay pinned as before. * tests: take mini admin ports from the below-ephemeral range * tests: reuse MiniAdminArgs for the remote-s3 fetch mini * tests: fully disable webdav and co-allocate mini admin ports - -webdav.port=0 only makes the service bind a random port; add -webdav=false so minis that do not test WebDAV skip it entirely. - Spark, RisingWave and s3 Spark launchers allocated the admin port in a second MustFreeMiniPort call, which could pick a port already handed out by the first batch. Fold 'Admin' into the same MustFreeMiniPorts call. - test/fuse_integration is a separate Go module and cannot import testutil; allocate the admin ports with its local freePort helper instead. * test/fuse: derive the mini admin gRPC port from its HTTP port Two independent freePort calls could return the same port, making admin.port equal admin.port.grpc (or collide with the filer port). Allocate once and derive the gRPC port with the +10000 convention. * test/fuse: pin the mini admin port in the fuse_test framework copy The fuse_test package framework spawns weed mini too; without a pinned admin port it still binds the fixed defaults 12535/23646 and collides with other minis. * tests: drop the redundant -webdav.port=0 flag -webdav=false alone disables the service; the port flag is dead config.
Apache Doris Iceberg Catalog Integration Test
This directory contains an Apache Doris integration smoke test for SeaweedFS's Iceberg REST Catalog implementation.
What It Tests
TestDorisIcebergCatalog verifies the Doris path end to end:
- Starts a local SeaweedFS mini cluster with S3 Tables and Iceberg REST enabled.
- Creates a SeaweedFS table bucket.
- Creates an Iceberg namespace and an empty table through the SeaweedFS REST catalog OAuth flow.
- Creates a second table and populates it with three rows by running a
PyIceberg writer container (
Dockerfile.writer+append_rows.py) before Doris bootstraps, so the snapshot is part of Doris's first scan. - Starts
apache/doris:doris-all-in-one-2.1.0and waits for the FE MySQL query port and at least one alive BE. - Connects to Doris over the MySQL protocol and registers a Doris
EXTERNAL CATALOGoftype=icebergandiceberg.catalog.type=restthat points at the SeaweedFS REST endpoint, usingcredential = key:secretfor the OAuth2 client-credentials flow. - Runs subtests against the SeaweedFS-backed Iceberg tables:
BasicSelect: Doris is alive and answering SQL.CatalogVisible:SHOW CATALOGSlists the SeaweedFS-backed catalog.DatabaseVisible: the seeded namespace is exposed as a Doris database.TableVisible: the seeded table appears underSHOW TABLES FROM ....CountEmptyTable: catalog-to-table resolution and a scan of an empty table.ColumnProjection:SELECT id, labelsucceeds and the response columns areid, label. Failure here means Doris could not parse the schema returned by the SeaweedFS catalog.ReadWrittenDataCountandReadWrittenDataValues: Doris reads back the three PyIceberg-appended rows and the values match. This exercises the actual data path (parquet reads via S3), not just metadata.
The PyIceberg writer image is built on demand via Docker layer caching. The
first build pulls python:3.11-slim and pip-installs PyIceberg + PyArrow
(~1-2 min in CI); subsequent invocations are cheap.
Running Locally
Build or install weed, then run:
cd test/s3tables/catalog_doris
go test -v -timeout 25m .
The test requires Docker. The GitHub Actions job runs on ubuntu-22.04 and
executes the test for pull requests.
Configuration
The test uses these fixed credentials for the local SeaweedFS IAM config:
- S3 access key:
AKIAIOSFODNN7EXAMPLE - S3 secret key:
wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY - Region:
us-west-2 - Warehouse bucket:
iceberg-tables
Doris ports:
- The all-in-one container exposes
9030(FE MySQL/SQL) and8030(FE HTTP). Only9030is mapped to a host port (allocated dynamically) so the Go MySQL client can connect from the test process. - The Iceberg REST endpoint and the S3 endpoint are reached from inside the
Doris container via
host.docker.internal, matching the Trino and Dremio test paths.
Troubleshooting
- Ensure Docker is running:
docker version - Ensure
weedis built or available onPATH - The Doris all-in-one image is large (~1 GB) and the FE+BE need ~30-60 seconds to register before queries succeed; the test waits up to 4 minutes for the query port and 60 seconds for at least one alive BE.
- If queries hang on metadata, run
REFRESH CATALOG iceberg_catalogfrom a Doris MySQL client and retry. - Container logs are printed in the failure message; you can also check
docker logs <seaweed-doris-...>while the test is running.