Chris Lu
aeb5db1b13
* tests: pin weed mini admin ports and disable unused aux services Every weed mini that does not configure them still binds the fixed default ports for the admin server (12535 + derived 22535 gRPC), its legacy compat listener (23646 + 33646), WebDAV (7333), Lance (9101) and the Iceberg REST catalog (8181). Sequential or concurrent minis on one host then collide on them: the port check warns and shifts, and the admin's late bind can land on a stolen ephemeral port, adding noise and flakiness to test logs (e.g. run 40017181958, job 120659267479). Add testutil.MiniAdminArgs to pin the admin server to an allocated mini-style port (HTTP + port+10000 gRPC, both below the Linux ephemeral port floor since admin binds late) and disable the legacy compat listener. Every test that spawns weed mini now also disables the aux services it does not use: WebDAV, Lance and the Iceberg REST endpoint get port=0 unless the test actually exercises them, in which case their ports stay pinned as before. * tests: take mini admin ports from the below-ephemeral range * tests: reuse MiniAdminArgs for the remote-s3 fetch mini * tests: fully disable webdav and co-allocate mini admin ports - -webdav.port=0 only makes the service bind a random port; add -webdav=false so minis that do not test WebDAV skip it entirely. - Spark, RisingWave and s3 Spark launchers allocated the admin port in a second MustFreeMiniPort call, which could pick a port already handed out by the first batch. Fold 'Admin' into the same MustFreeMiniPorts call. - test/fuse_integration is a separate Go module and cannot import testutil; allocate the admin ports with its local freePort helper instead. * test/fuse: derive the mini admin gRPC port from its HTTP port Two independent freePort calls could return the same port, making admin.port equal admin.port.grpc (or collide with the filer port). Allocate once and derive the gRPC port with the +10000 convention. * test/fuse: pin the mini admin port in the fuse_test framework copy The fuse_test package framework spawns weed mini too; without a pinned admin port it still binds the fixed defaults 12535/23646 and collides with other minis. * tests: drop the redundant -webdav.port=0 flag -webdav=false alone disables the service; the port flag is dead config.
Table Lifecycle Integration Tests
One table, all the way through: created in the catalog, filled by a real client, maintained, read again, dropped. Once for Iceberg and once for Lance. The Iceberg half always maintains through the worker; the Lance half maintains through the Rust worker or through the lance library, depending on what the environment has - see below.
Why this suite exists
#10853 was a compaction that rewrote every dictionary-encoded column onto a single value. It shipped. The maintenance tests we had were thorough about the bookkeeping - sequence numbers, added and deleted manifest entries, metadata versions, the manifest list - and every one of them passed, because not one of them opened the parquet file the worker had just written.
So the assertion here is the dull one nothing else was making: tally the table before maintenance, tally it again after, and require the two to be equal. The tally is a row count, the cardinality of each dictionary-encoded column, and an md5 over whole rows. The cardinalities name the failure that happened; the digest catches a rewrite that keeps every column's cardinality and hands the values to the wrong rows.
The same shape covers Lance, because the exposure is the same: a compaction that merges fragments can hand back a table that reads without complaint and answers wrongly.
What runs
TestIcebergTableLifecycle starts a weed mini cluster, declares an ICEBERG
table bucket, and runs two clients against it:
| Client | Why both |
|---|---|
| DuckDB | the client the bug was reported against, and the only one here that writes the deprecated PLAIN_DICTIONARY encoding - parquet-go normalizes it away on write, so a Go writer cannot produce it |
| PyIceberg | writes RLE_DICTIONARY, the modern spelling, so between the two the merge is checked against both dictionary encodings in the spec |
Between the write and the read, the test runs the worker's whole maintenance cycle in-process against the live filer: compact, expire snapshots, remove orphans, rewrite manifests. A compaction that merged nothing fails the test rather than passing it - otherwise the read afterwards is checking a file the worker never wrote.
TestLanceTableLifecycle does the same against a LANCE bucket: declare
through the namespace, write a fragment per append, maintain, read, drop.
Maintenance goes through the Rust worker's own handlers where cargo is
installed, and through the two lance calls those handlers wrap where it is not.
WEED_LANCE_MAINTENANCE=library|worker picks one instead of letting the test
guess; CI sets library, because a cold build of the lance crate costs more
than the layer it would be checking.
Both tests finish by dropping the table and checking the data actually left the filer, which is the half of a lifecycle a catalog test never reaches.
Running it
cd test/s3tables/lifecycle
(cd ../../../weed && go build .) # the harness runs this binary
go test -v -timeout 40m .
Skipped without Docker and in -short mode. The first run builds the two client
images and pulls duckdb/duckdb:latest; later runs reuse them. The DuckDB half
skips itself, rather than failing, on an image whose iceberg extension cannot
write through a REST catalog.
To watch it catch the bug it was written for, pin parquet-go back to the version that had it:
go mod edit -require=github.com/parquet-go/parquet-go@v0.30.1 && go mod tidy
go test -run TestIcebergTableLifecycle/DuckDB -v .
maintenance collapsed the category column: 7 distinct values -> 1
maintenance collapsed the value column: 13 distinct values -> 1
The PyIceberg half still passes there, which is the reason both clients are in this directory.