Updated OLake Iceberg Integration (markdown)

GAURI committed 2026-10-20 16:19:40 +05:30
1 parent 025d81eb50
commit 7ceb75c498
1 file changed
+20 -34
@@ -33,6 +33,17 @@ curl -fsSL https://raw.githubusercontent.com/datazip-inc/olake-docs/master/docs/
-o docker-compose.source.yml
```
To start the sample PostgreSQL database:
```bash
docker compose -f docker-compose.source.yml --profile postgres up -d
```
Once all the services are up, quickly check the sample data by running the below query:
```bash
docker exec -it primary_postgres psql -U main -d main -c "SELECT * FROM public.sample_data;"
```
Configure the PostgreSQL source in `source.json`:
@@ -81,19 +92,13 @@ Configure SeaweedFS as the Iceberg destination in `destination.json`:
}
```
From the OLake repository root, start the sample PostgreSQL database:
```bash
docker compose -f docker-compose.source.yml --profile postgres up -d
```
## Sync PostgreSQL to Iceberg
Run these commands from the directory containing the PostgreSQL driver executable and the JSON
configuration files. Discover the available PostgreSQL streams:
```bash
./olake-postgres discover --config source.json --destination destination.json
./build.sh driver-postgres discover --config $(pwd)/source.json
```
Discovery creates `streams.json`. Set the selected stream's `sync_mode` to the value for the
@@ -102,41 +107,22 @@ strategy you want:
| Sync mode | What it does |
| --- | --- |
| `full_refresh` | Copies the selected table's current contents on each run. |
| `cdc` | Performs an initial full load, then captures inserts, updates, and deletes from PostgreSQL using CDC. |
| `cdc` | Performs an initial full load, then captures inserts, updates, and deletes using CDC. |
| `incremental` | Performs an initial full load, then reads new or changed rows using a cursor. It does not detect deletes. |
For Incremental, configure a cursor field for the selected stream. A cursor such as `updated_at`
can track updates if the source changes that field.
Run the sync with the same command for any of these modes:
Run the sync command:
```bash
./olake-postgres sync \
--config source.json \
--catalog streams.json \
--destination destination.json \
--state state.json
./build.sh driver-postgres sync --config $(pwd)/source.json --catalog $(pwd)/streams.json --destination $(pwd)/destination.json
```
Keep `state.json` for later runs of the same sync. OLake also supports CDC Only (`strict_cdc`); see
[sync modes documentation](https://olake.io/docs/understanding/terminologies/olake/#2-sync-modes).
After the full refresh, run the next sync with state flag added to only capture the changes in source:
### Append and Upsert
For CDC and Incremental syncs, OLake's [Append and Upsert ingestion modes](https://olake.io/docs/understanding/terminologies/olake/#7-modes-to-ingest-data-upsert-vs-append) control how changes are written to Iceberg. Append keeps incoming records as new rows; Upsert applies changes by key to retain the latest row state and requires a deduplication key for each stream. Full Refresh always uses Append.
## Query the synced data
Open Spark and run:
```sql
%%sql
SELECT * FROM olake_iceberg.postgres_main_public.sample_data;
```bash
./build.sh driver-postgres sync --config $(pwd)/source.json --catalog $(pwd)/streams.json --destination $(pwd)/destination.json --state $(pwd)/state.json
```
To try inserts, updates, and deletes, follow OLake's [demo workflow](https://olake.io/docs/community/setting-up-a-dev-env/#5-demo-workflow) and check the synced changes.
## See also
For more about OLake's sources, sync strategies, and Iceberg ingestion options, explore the
[OLake documentation](https://olake.io/).
For more about OLake Go's sources, sync strategies, and Iceberg ingestion options, explore the
[OLake documentation](https://olake.io/docs/).