Updated OLake Go Iceberg Integration (markdown)

gauri-i committed 2026-10-21 10:38:05 +05:30
1 parent 917ece986e
commit 9174fa7aa7
1 file changed
+30 -5
+30 -5
@@ -7,6 +7,18 @@ as the source and SeaweedFS as the Iceberg REST catalog and S3-compatible storag
- OLake Go v0.12.3
## Clone the OLake Go repository
Clone the OLake Go repository before starting SeaweedFS:
```bash
git clone https://github.com/datazip-inc/olake.git
cd olake
```
Run all `./build.sh` commands below from the root of the `olake` repository. Keep the source,
destination, and streams configuration files in this directory.
## Source and Destination Setup
OLake Go provides source-specific drivers and configurations for multiple connectors, including [PostgreSQL](https://olake.io/docs/connectors/postgres/), [MongoDB](https://olake.io/docs/connectors/mongodb/), [MySQL](https://olake.io/docs/connectors/mysql/), [Oracle](https://olake.io/docs/connectors/oracle/), [Kafka](https://olake.io/docs/connectors/kafka/), [DB2](https://olake.io/docs/connectors/db2/), [S3](https://olake.io/docs/connectors/s3/), and [Microsoft SQL Server (MSSQL)](https://olake.io/docs/connectors/mssql/). This walkthrough uses PostgreSQL.
@@ -94,15 +106,24 @@ Configure SeaweedFS as the Iceberg destination in `destination.json`:
## Sync PostgreSQL to Iceberg
Run these commands from the directory containing the PostgreSQL driver executable and the JSON
configuration files. Discover the available PostgreSQL streams:
### Discover
From the root of the `olake` repository, run discovery to connect to PostgreSQL and list the
available tables (streams):
```bash
./build.sh driver-postgres discover --config $(pwd)/source.json
```
Discovery creates `streams.json`. Set the selected stream's `sync_mode` to the value for the
strategy you want:
### Streams
Discovery generates `streams.json`, which contains the discovered tables, their schemas and
metadata, and a `selected_streams` section. Keep or add the tables you want to sync under
`selected_streams`.
### Sync
In `streams.json`, set each selected stream's `sync_mode` to choose how its data is synced:
| Sync mode | What it does |
| --- | --- |
@@ -116,12 +137,16 @@ Run the sync command:
./build.sh driver-postgres sync --config $(pwd)/source.json --catalog $(pwd)/streams.json --destination $(pwd)/destination.json
```
After the full refresh, run the next sync with state flag added to only capture the changes in source:
On subsequent runs, pass the saved state file to continue syncing changes from the source:
```bash
./build.sh driver-postgres sync --config $(pwd)/source.json --catalog $(pwd)/streams.json --destination $(pwd)/destination.json --state $(pwd)/state.json
```
## Query your data
We used Apache Spark to query the data in Iceberg.
## See also
For more about OLake Go's sources, sync strategies, and Iceberg ingestion options, explore the