mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-26 15:00:48 +01:00
Updated OLake Go Iceberg Integration (markdown)
1 parent
917ece986e
commit
9174fa7aa7
1 file changed
+30
-5
@@ -7,6 +7,18 @@ as the source and SeaweedFS as the Iceberg REST catalog and S3-compatible storag
|
||||
|
||||
- OLake Go v0.12.3
|
||||
|
||||
## Clone the OLake Go repository
|
||||
|
||||
Clone the OLake Go repository before starting SeaweedFS:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/datazip-inc/olake.git
|
||||
cd olake
|
||||
```
|
||||
|
||||
Run all `./build.sh` commands below from the root of the `olake` repository. Keep the source,
|
||||
destination, and streams configuration files in this directory.
|
||||
|
||||
## Source and Destination Setup
|
||||
|
||||
OLake Go provides source-specific drivers and configurations for multiple connectors, including [PostgreSQL](https://olake.io/docs/connectors/postgres/), [MongoDB](https://olake.io/docs/connectors/mongodb/), [MySQL](https://olake.io/docs/connectors/mysql/), [Oracle](https://olake.io/docs/connectors/oracle/), [Kafka](https://olake.io/docs/connectors/kafka/), [DB2](https://olake.io/docs/connectors/db2/), [S3](https://olake.io/docs/connectors/s3/), and [Microsoft SQL Server (MSSQL)](https://olake.io/docs/connectors/mssql/). This walkthrough uses PostgreSQL.
|
||||
@@ -94,15 +106,24 @@ Configure SeaweedFS as the Iceberg destination in `destination.json`:
|
||||
|
||||
## Sync PostgreSQL to Iceberg
|
||||
|
||||
Run these commands from the directory containing the PostgreSQL driver executable and the JSON
|
||||
configuration files. Discover the available PostgreSQL streams:
|
||||
### Discover
|
||||
|
||||
From the root of the `olake` repository, run discovery to connect to PostgreSQL and list the
|
||||
available tables (streams):
|
||||
|
||||
```bash
|
||||
./build.sh driver-postgres discover --config $(pwd)/source.json
|
||||
```
|
||||
|
||||
Discovery creates `streams.json`. Set the selected stream's `sync_mode` to the value for the
|
||||
strategy you want:
|
||||
### Streams
|
||||
|
||||
Discovery generates `streams.json`, which contains the discovered tables, their schemas and
|
||||
metadata, and a `selected_streams` section. Keep or add the tables you want to sync under
|
||||
`selected_streams`.
|
||||
|
||||
### Sync
|
||||
|
||||
In `streams.json`, set each selected stream's `sync_mode` to choose how its data is synced:
|
||||
|
||||
| Sync mode | What it does |
|
||||
| --- | --- |
|
||||
@@ -116,12 +137,16 @@ Run the sync command:
|
||||
./build.sh driver-postgres sync --config $(pwd)/source.json --catalog $(pwd)/streams.json --destination $(pwd)/destination.json
|
||||
```
|
||||
|
||||
After the full refresh, run the next sync with state flag added to only capture the changes in source:
|
||||
On subsequent runs, pass the saved state file to continue syncing changes from the source:
|
||||
|
||||
```bash
|
||||
./build.sh driver-postgres sync --config $(pwd)/source.json --catalog $(pwd)/streams.json --destination $(pwd)/destination.json --state $(pwd)/state.json
|
||||
```
|
||||
|
||||
## Query your data
|
||||
|
||||
We used Apache Spark to query the data in Iceberg.
|
||||
|
||||
## See also
|
||||
|
||||
For more about OLake Go's sources, sync strategies, and Iceberg ingestion options, explore the
|
||||
|
||||
Reference in new issue
Block a user