OLake Go Iceberg Integration
OLake Go syncs data from supported sources to Iceberg destinations. This example uses PostgreSQL as the source and SeaweedFS as the Iceberg REST catalog and S3-compatible storage.
Prerequisites
- OLake Go v0.12.3
Clone the OLake Go repository
Clone the OLake Go repository before starting SeaweedFS:
git clone https://github.com/datazip-inc/olake.git
cd olake
Run all ./build.sh commands below from the root of the olake repository. Keep the source,
destination, and streams configuration files in this directory.
Source and Destination Setup
OLake Go provides source-specific drivers and configurations for multiple connectors, including PostgreSQL, MongoDB, MySQL, Oracle, Kafka, DB2, S3, and Microsoft SQL Server (MSSQL). This walkthrough uses PostgreSQL.
Start SeaweedFS
docker volume create seaweed-data
docker run -d --name seaweedfs \
-e AWS_ACCESS_KEY_ID=admin -e AWS_SECRET_ACCESS_KEY=password \
-p 8333:8333 -p 8181:8181 -p 9333:9333 \
-v seaweed-data:/data \
chrislusf/seaweedfs:latest \
mini -dir=/data -tableBucket=warehouse
PostgreSQL Source
To try a local setup, start the PostgreSQL source with CDC enabled. Download it with:
curl -fsSL https://raw.githubusercontent.com/datazip-inc/olake-docs/master/docs/community/docker-compose.yml \
-o docker-compose.source.yml
To start the sample PostgreSQL database:
docker compose -f docker-compose.source.yml --profile postgres up -d
Once all the services are up, quickly check the sample data by running the below query:
docker exec -it primary_postgres psql -U main -d main -c "SELECT * FROM public.sample_data;"
Configure the PostgreSQL source in source.json:
{
"host": "localhost",
"port": 5431,
"database": "main",
"username": "main",
"password": "password",
"jdbc_url_params": {},
"ssl": {
"mode": "disable"
},
"update_method": {
"replication_slot": "postgres_slot",
"publication": "olake_publication",
"initial_wait_time": 120
},
"reader_batch_size": 1000,
"max_threads": 10
}
Iceberg Destination
Configure SeaweedFS as the Iceberg destination in destination.json:
{
"type": "ICEBERG",
"writer": {
"catalog_type": "seaweedfs",
"rest_catalog_url": "http://localhost:8181",
"rest_auth_type": "OAuth2",
"credential": "admin:password",
"oauth2_uri": "http://localhost:8181/v1/oauth/tokens",
"s3_endpoint": "http://localhost:8333",
"iceberg_s3_path": "s3://warehouse",
"aws_access_key": "admin",
"aws_secret_key": "password",
"aws_region": "us-east-1",
"iceberg_db": "olake_iceberg",
"catalog_name": "olake_iceberg"
}
}
Sync PostgreSQL to Iceberg
Discover
From the root of the olake repository, run discovery to connect to PostgreSQL and list the
available tables (streams):
./build.sh driver-postgres discover --config $(pwd)/source.json
Streams
Discovery generates streams.json, which contains the discovered tables, their schemas and
metadata, and a selected_streams section. Keep or add the tables you want to sync under
selected_streams.
To learn more about stream-level properties and how to configure them, see Stream Level Properties.
Sync
In streams.json, set each selected stream's sync_mode to choose how its data is synced:
| Sync mode | What it does |
|---|---|
full_refresh |
Copies the selected table's current contents on each run. |
cdc |
Performs an initial full load, then captures inserts, updates, and deletes using CDC. |
incremental |
Performs an initial full load, then reads new or changed rows using a cursor. It does not detect deletes. |
Run the sync command:
./build.sh driver-postgres sync --config $(pwd)/source.json --catalog $(pwd)/streams.json --destination $(pwd)/destination.json
On subsequent runs, pass the saved state file to continue syncing changes from the source:
./build.sh driver-postgres sync --config $(pwd)/source.json --catalog $(pwd)/streams.json --destination $(pwd)/destination.json --state $(pwd)/state.json
Query your data
Verify that your data is accessible using a query engine that supports Apache Iceberg tables, such as Apache Spark.
The namespace comes from the destination database name set in the job.
Here it is postgres_main_public, so OLake Go created that namespace and stored the synced tables in it.
List the tables in the postgres_main_public namespace:
The output shows the available table names; here, the table is sample_data.
Inspect the table's columns and data types:
The output lists each column and its Spark data type.
Check the Iceberg table's storage location in SeaweedFS:
The Location row shows where the table's Iceberg data is stored:
s3://warehouse/postgres_main_public/sample_data.
Read a few records from the table:
See also
For more about OLake Go's sources, sync strategies, and Iceberg ingestion options, explore the OLake documentation.