mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-24 23:10:41 +02:00
Updated OLake Iceberg Integration (markdown)
1 parent
025d81eb50
commit
7ceb75c498
1 file changed
+20
-34
@@ -33,6 +33,17 @@ curl -fsSL https://raw.githubusercontent.com/datazip-inc/olake-docs/master/docs/
|
||||
-o docker-compose.source.yml
|
||||
```
|
||||
|
||||
To start the sample PostgreSQL database:
|
||||
|
||||
```bash
|
||||
docker compose -f docker-compose.source.yml --profile postgres up -d
|
||||
```
|
||||
|
||||
Once all the services are up, quickly check the sample data by running the below query:
|
||||
|
||||
```bash
|
||||
docker exec -it primary_postgres psql -U main -d main -c "SELECT * FROM public.sample_data;"
|
||||
```
|
||||
|
||||
Configure the PostgreSQL source in `source.json`:
|
||||
|
||||
@@ -81,19 +92,13 @@ Configure SeaweedFS as the Iceberg destination in `destination.json`:
|
||||
}
|
||||
```
|
||||
|
||||
From the OLake repository root, start the sample PostgreSQL database:
|
||||
|
||||
```bash
|
||||
docker compose -f docker-compose.source.yml --profile postgres up -d
|
||||
```
|
||||
|
||||
## Sync PostgreSQL to Iceberg
|
||||
|
||||
Run these commands from the directory containing the PostgreSQL driver executable and the JSON
|
||||
configuration files. Discover the available PostgreSQL streams:
|
||||
|
||||
```bash
|
||||
./olake-postgres discover --config source.json --destination destination.json
|
||||
./build.sh driver-postgres discover --config $(pwd)/source.json
|
||||
```
|
||||
|
||||
Discovery creates `streams.json`. Set the selected stream's `sync_mode` to the value for the
|
||||
@@ -102,41 +107,22 @@ strategy you want:
|
||||
| Sync mode | What it does |
|
||||
| --- | --- |
|
||||
| `full_refresh` | Copies the selected table's current contents on each run. |
|
||||
| `cdc` | Performs an initial full load, then captures inserts, updates, and deletes from PostgreSQL using CDC. |
|
||||
| `cdc` | Performs an initial full load, then captures inserts, updates, and deletes using CDC. |
|
||||
| `incremental` | Performs an initial full load, then reads new or changed rows using a cursor. It does not detect deletes. |
|
||||
|
||||
For Incremental, configure a cursor field for the selected stream. A cursor such as `updated_at`
|
||||
can track updates if the source changes that field.
|
||||
|
||||
Run the sync with the same command for any of these modes:
|
||||
Run the sync command:
|
||||
|
||||
```bash
|
||||
./olake-postgres sync \
|
||||
--config source.json \
|
||||
--catalog streams.json \
|
||||
--destination destination.json \
|
||||
--state state.json
|
||||
./build.sh driver-postgres sync --config $(pwd)/source.json --catalog $(pwd)/streams.json --destination $(pwd)/destination.json
|
||||
```
|
||||
|
||||
Keep `state.json` for later runs of the same sync. OLake also supports CDC Only (`strict_cdc`); see
|
||||
[sync modes documentation](https://olake.io/docs/understanding/terminologies/olake/#2-sync-modes).
|
||||
After the full refresh, run the next sync with state flag added to only capture the changes in source:
|
||||
|
||||
### Append and Upsert
|
||||
|
||||
For CDC and Incremental syncs, OLake's [Append and Upsert ingestion modes](https://olake.io/docs/understanding/terminologies/olake/#7-modes-to-ingest-data-upsert-vs-append) control how changes are written to Iceberg. Append keeps incoming records as new rows; Upsert applies changes by key to retain the latest row state and requires a deduplication key for each stream. Full Refresh always uses Append.
|
||||
|
||||
## Query the synced data
|
||||
|
||||
Open Spark and run:
|
||||
|
||||
```sql
|
||||
%%sql
|
||||
SELECT * FROM olake_iceberg.postgres_main_public.sample_data;
|
||||
```bash
|
||||
./build.sh driver-postgres sync --config $(pwd)/source.json --catalog $(pwd)/streams.json --destination $(pwd)/destination.json --state $(pwd)/state.json
|
||||
```
|
||||
|
||||
To try inserts, updates, and deletes, follow OLake's [demo workflow](https://olake.io/docs/community/setting-up-a-dev-env/#5-demo-workflow) and check the synced changes.
|
||||
|
||||
## See also
|
||||
|
||||
For more about OLake's sources, sync strategies, and Iceberg ingestion options, explore the
|
||||
[OLake documentation](https://olake.io/).
|
||||
For more about OLake Go's sources, sync strategies, and Iceberg ingestion options, explore the
|
||||
[OLake documentation](https://olake.io/docs/).
|
||||
Reference in new issue
Block a user