From c0c2de78e4be2400308492ce398f5e229b5e9e37 Mon Sep 17 00:00:00 2001
From: GAURI <gauriagarwal25@gmail.com>
Date: Tue, 20 Oct 2026 15:21:39 +0530
Subject: [PATCH] Created OLake Iceberg Integration (markdown)

---
 OLake-Iceberg-Integration.md | 143 +++++++++++++++++++++++++++++++++++
 1 file changed, 143 insertions(+)
 create mode 100644 OLake-Iceberg-Integration.md

diff --git a/OLake-Iceberg-Integration.md b/OLake-Iceberg-Integration.md
new file mode 100644
index 0000000..c4fd084
--- /dev/null
+++ b/OLake-Iceberg-Integration.md
@@ -0,0 +1,143 @@
+# OLake Iceberg Integration
+
+OLake Go syncs data from supported sources to Iceberg destinations. This example uses PostgreSQL
+as the source and SeaweedFS as the Iceberg REST catalog and S3-compatible storage.
+
+## Prerequisites
+
+- OLake Go v1.0
+- Docker available for running SeaweedFS and the sample PostgreSQL database
+
+## Source and Destination Setup
+
+OLake Go has source-specific drivers and configuration. The [OLake development environment guide](https://olake.io/docs/community/setting-up-a-dev-env/#1-source-and-destination-setup) includes examples for PostgreSQL, MySQL, MongoDB, Oracle, and other sources. This walkthrough uses PostgreSQL.
+
+### Start SeaweedFS
+
+```bash
+docker volume create seaweed-data
+
+docker run -d --name seaweedfs \
+  -e AWS_ACCESS_KEY_ID=admin -e AWS_SECRET_ACCESS_KEY=password \
+  -p 8333:8333 -p 8181:8181 -p 9333:9333 \
+  -v seaweed-data:/data \
+  chrislusf/seaweedfs:latest \
+  mini -dir=/data -tableBucket=warehouse
+```
+
+### PostgreSQL Source
+
+To try a local setup, start the PostgreSQL source with CDC enabled. Download it with:
+
+```bash
+curl -fsSL https://raw.githubusercontent.com/datazip-inc/olake-docs/master/docs/community/docker-compose.yml \
+  -o docker-compose.source.yml
+```
+
+
+Configure the PostgreSQL source in `source.json`:
+
+```json
+{
+  "host": "localhost",
+  "port": 5431,
+  "database": "main",
+  "username": "main",
+  "password": "password",
+  "jdbc_url_params": {},
+  "ssl": {
+    "mode": "disable"
+  },
+  "update_method": {
+    "replication_slot": "postgres_slot",
+    "publication": "olake_publication",
+    "initial_wait_time": 120
+  },
+  "reader_batch_size": 1000,
+  "max_threads": 10
+}
+```
+
+### Iceberg Destination
+
+Configure SeaweedFS as the Iceberg destination in `destination.json`:
+
+```json
+{
+  "type": "ICEBERG",
+  "writer": {
+    "catalog_type": "seaweedfs",
+    "rest_catalog_url": "http://localhost:8181",
+    "rest_auth_type": "OAuth2",
+    "credential": "admin:password",
+    "oauth2_uri": "http://localhost:8181/v1/oauth/tokens",
+    "s3_endpoint": "http://localhost:8333",
+    "iceberg_s3_path": "s3://warehouse",
+    "aws_access_key": "admin",
+    "aws_secret_key": "password",
+    "aws_region": "us-east-1",
+    "iceberg_db": "olake_iceberg",
+    "catalog_name": "olake_iceberg"
+  }
+}
+```
+
+From the OLake repository root, start the sample PostgreSQL database:
+
+```bash
+docker compose -f docker-compose.source.yml --profile postgres up -d
+```
+
+## Sync PostgreSQL to Iceberg
+
+Run these commands from the directory containing the PostgreSQL driver executable and the JSON
+configuration files. Discover the available PostgreSQL streams:
+
+```bash
+./olake-postgres discover --config source.json --destination destination.json
+```
+
+Discovery creates `streams.json`. Set the selected stream's `sync_mode` to the value for the
+strategy you want:
+
+| Sync mode | What it does |
+| --- | --- |
+| `full_refresh` | Copies the selected table's current contents on each run. |
+| `cdc` | Performs an initial full load, then captures inserts, updates, and deletes from PostgreSQL using CDC. |
+| `incremental` | Performs an initial full load, then reads new or changed rows using a cursor. It does not detect deletes. |
+
+For Incremental, configure a cursor field for the selected stream. A cursor such as `updated_at`
+can track updates if the source changes that field.
+
+Run the sync with the same command for any of these modes:
+
+```bash
+./olake-postgres sync \
+  --config source.json \
+  --catalog streams.json \
+  --destination destination.json \
+  --state state.json
+```
+
+Keep `state.json` for later runs of the same sync. OLake also supports CDC Only (`strict_cdc`); see
+[sync modes documentation](https://olake.io/docs/understanding/terminologies/olake/#2-sync-modes).
+
+### Append and Upsert
+
+For CDC and Incremental syncs, OLake's [Append and Upsert ingestion modes](https://olake.io/docs/understanding/terminologies/olake/#7-modes-to-ingest-data-upsert-vs-append) control how changes are written to Iceberg. Append keeps incoming records as new rows; Upsert applies changes by key to retain the latest row state and requires a deduplication key for each stream. Full Refresh always uses Append.
+
+## Query the synced data
+
+Open Spark and run:
+
+```sql
+%%sql
+SELECT * FROM olake_iceberg.postgres_main_public.sample_data;
+```
+
+To try inserts, updates, and deletes, follow OLake's [demo workflow](https://olake.io/docs/community/setting-up-a-dev-env/#5-demo-workflow) and check the synced changes.
+
+## See also
+
+For more about OLake's sources, sync strategies, and Iceberg ingestion options, explore the
+[OLake documentation](https://olake.io/).