diff --git a/OLake-Iceberg-Integration.md b/OLake-Go-Iceberg-Integration.md similarity index 66% rename from OLake-Iceberg-Integration.md rename to OLake-Go-Iceberg-Integration.md index 4d26808..8971c87 100644 --- a/OLake-Iceberg-Integration.md +++ b/OLake-Go-Iceberg-Integration.md @@ -33,6 +33,17 @@ curl -fsSL https://raw.githubusercontent.com/datazip-inc/olake-docs/master/docs/ -o docker-compose.source.yml ``` +To start the sample PostgreSQL database: + +```bash +docker compose -f docker-compose.source.yml --profile postgres up -d +``` + +Once all the services are up, quickly check the sample data by running the below query: + +```bash +docker exec -it primary_postgres psql -U main -d main -c "SELECT * FROM public.sample_data;" +``` Configure the PostgreSQL source in `source.json`: @@ -81,19 +92,13 @@ Configure SeaweedFS as the Iceberg destination in `destination.json`: } ``` -From the OLake repository root, start the sample PostgreSQL database: - -```bash -docker compose -f docker-compose.source.yml --profile postgres up -d -``` - ## Sync PostgreSQL to Iceberg Run these commands from the directory containing the PostgreSQL driver executable and the JSON configuration files. Discover the available PostgreSQL streams: ```bash -./olake-postgres discover --config source.json --destination destination.json +./build.sh driver-postgres discover --config $(pwd)/source.json ``` Discovery creates `streams.json`. Set the selected stream's `sync_mode` to the value for the @@ -102,41 +107,22 @@ strategy you want: | Sync mode | What it does | | --- | --- | | `full_refresh` | Copies the selected table's current contents on each run. | -| `cdc` | Performs an initial full load, then captures inserts, updates, and deletes from PostgreSQL using CDC. | +| `cdc` | Performs an initial full load, then captures inserts, updates, and deletes using CDC. | | `incremental` | Performs an initial full load, then reads new or changed rows using a cursor. It does not detect deletes. | -For Incremental, configure a cursor field for the selected stream. A cursor such as `updated_at` -can track updates if the source changes that field. - -Run the sync with the same command for any of these modes: +Run the sync command: ```bash -./olake-postgres sync \ - --config source.json \ - --catalog streams.json \ - --destination destination.json \ - --state state.json +./build.sh driver-postgres sync --config $(pwd)/source.json --catalog $(pwd)/streams.json --destination $(pwd)/destination.json ``` -Keep `state.json` for later runs of the same sync. OLake also supports CDC Only (`strict_cdc`); see -[sync modes documentation](https://olake.io/docs/understanding/terminologies/olake/#2-sync-modes). +After the full refresh, run the next sync with state flag added to only capture the changes in source: -### Append and Upsert - -For CDC and Incremental syncs, OLake's [Append and Upsert ingestion modes](https://olake.io/docs/understanding/terminologies/olake/#7-modes-to-ingest-data-upsert-vs-append) control how changes are written to Iceberg. Append keeps incoming records as new rows; Upsert applies changes by key to retain the latest row state and requires a deduplication key for each stream. Full Refresh always uses Append. - -## Query the synced data - -Open Spark and run: - -```sql -%%sql -SELECT * FROM olake_iceberg.postgres_main_public.sample_data; +```bash +./build.sh driver-postgres sync --config $(pwd)/source.json --catalog $(pwd)/streams.json --destination $(pwd)/destination.json --state $(pwd)/state.json ``` -To try inserts, updates, and deletes, follow OLake's [demo workflow](https://olake.io/docs/community/setting-up-a-dev-env/#5-demo-workflow) and check the synced changes. - ## See also -For more about OLake's sources, sync strategies, and Iceberg ingestion options, explore the -[OLake documentation](https://olake.io/). +For more about OLake Go's sources, sync strategies, and Iceberg ingestion options, explore the +[OLake documentation](https://olake.io/docs/).