seaweedfs/test/s3tables/catalog_olake

OLake Iceberg Catalog Integration Test

An integration test for OLake against SeaweedFS's Iceberg REST Catalog, in the same shape as catalog_clickhouse.

Why OLake, given we already test other engines

Two things here are covered by nothing else in this directory.

It is a strict Java Iceberg client. OLake does not write Iceberg from Go — its Go process spawns a Java sidecar over gRPC and writes through the official Apache Iceberg library, because the Go library has no equality deletes and CDC needs them. So this test exercises the client class that weed/s3api/iceberg/metadata_compliance.go exists to serve: the one that fails with "Cannot parse missing long current-snapshot-id" when the catalog omits spec-required keys that iceberg-go strips via omitempty.

It produces equality deletes. OLake is a CDC tool. Its upsert path commits operation=overwrite with an equality-delete file, and a delete manifest alongside the data manifests. ClickHouse, StarRocks, Doris and DuckDB all only append.

What it asserts

TestOLakeIcebergCatalog runs six subtests against a weed mini cluster with a pre-created table bucket and a Postgres source:

Subtest	What a failure means
CheckDestination	olake check did not reach SUCCEEDED, or it passed without ever loading org.apache.iceberg.rest.RESTSessionCatalog — the second case means the destination was never actually contacted.
Discover	OLake could not enumerate the source table, or wrote no streams.json.
FullSyncAppendsRows	The sync read fewer than the three seeded rows, or committed no Iceberg snapshot.
StrictReaderSeesRows	PyIceberg could not read back what the Java writer committed, or the values differ. This is the data path, not just metadata.
UpsertProducesEqualityDelete	After an UPDATE and a re-sync, no snapshot recorded an overwrite carrying equality deletes, or the current snapshot has no delete manifest.
CompliantWriterNeedsNoRepair	The catalog rewrote a manifest the official Iceberg Java writer produced. That is a regression in the repair gate, not a problem with OLake.

The one thing this test deliberately does not check

It does not assert that a reader sees the updated row and no duplicate.

PyIceberg refuses to scan a table carrying equality deletes (apache/iceberg#6568) while reading its metadata perfectly well — so a rows-mode read after the upsert would raise, not pass. The alternative is an engine that applies equality deletes, which for StarRocks means a 3 GB image and 12 GB of RAM in CI.

The line drawn instead: recording the commit correctly is the catalog's contract; applying deletes on read is the query engine's. The metadata assertions cover our half.

This was verified once by hand outside CI, on 2026-09-23, with StarRocks 4.1.4 attached to the same catalog: after the upsert it read 3 rows / 3 distinct ids with id=1 showing the updated value and _op_type=u. If someone later wants that inside the gate, add a reader that supports equality deletes — do not "upgrade" this test to a PyIceberg rows-mode read after the upsert. It would not pass; and if PyIceberg ever starts silently skipping deletes instead of raising, it would pass by not looking.

What the config proves

Nothing in the destination config is SeaweedFS-specific:

{
  "catalog_type": "rest",
  "rest_catalog_url": "http://HOST:ICEBERG_PORT",
  "iceberg_s3_path": "s3://olake-tables",
  "s3_endpoint": "http://HOST:S3_PORT",
  "rest_auth_type": "oauth2",
  "oauth2_uri": "http://HOST:ICEBERG_PORT/v1/oauth/tokens",
  "credential": "ACCESS_KEY:SECRET_KEY"
}

catalog_type is the generic rest, auth is the standard OAuth2 client-credentials flow, and the test config sets s3_path_style explicitly; OLake also defaults to path style whenever s3_endpoint is non-empty, so either form works. As of OLake v0.10.1 this works with no change on either side.

Running it

go test ./test/s3tables/catalog_olake/ -run TestOLakeIcebergCatalog -v -timeout 25m

Needs Docker and a weed binary (at weed/weed under the repo root, or on PATH). Takes about 35 seconds. It skips rather than fails when Docker is absent, and SEAWEEDFS_SKIP_OLAKE_TESTS=1 skips it outright.

Overrides: OLAKE_IMAGE (default olakego/source-postgres:latest), POSTGRES_IMAGE (default postgres:16).

In CI

Runs as olake-iceberg-catalog-tests in .github/workflows/s3-tables-tests.yml, on a matrix of a pinned image plus latest — the same shape the ClickHouse job uses, and for the same reason: OLake's Iceberg writer is a Java sidecar whose library version moves independently of the Go release, so the latest leg is what catches drift in the client rather than in OLake itself.

The job asserts the suite actually ran — at least one top-level --- PASS and zero --- SKIP — rather than trusting a green exit. This suite skips itself when Docker is unavailable, and a skipped suite reporting success is how a gate quietly stops being one.