mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-21 05:24:28 +02:00
Add OLake to README and Admin UI; fix catalog_olake path-style docs (#11858)
* README: mention OLake in the Iceberg lakehouse section * catalog_olake: fix path-style prose; admin UI: list OLake as a REST catalog client * admin UI: restore generated file import block, keep only the string change --------- Co-authored-by: GAURI <174253719+gauriiiiiiiiiiii@users.noreply.github.com>
This commit is contained in:
1 parent
2095c905ba
commit
da0d423bd9
5 files changed
+10
-8
No files matched your search
@@ -225,7 +225,7 @@ The full operation list is in [Amazon S3 API][AmazonS3API], and [Supported APIs
|
||||
|
||||
SeaweedFS is a lakehouse in one system. [S3 Table Buckets][S3TableBucket] hold Apache Iceberg tables by default, or [Lance][LanceCatalog] tables for vectors and multimodal data, and the built-in [Iceberg REST Catalog][IcebergCatalog] and Lance namespace serve them directly. There is no Hive Metastore, Glue, or separate catalog service to deploy, secure, and back up.
|
||||
|
||||
* Query engines operate on the same tables at the same time: [Spark][SparkIceberg], [Trino][TrinoIceberg], [Dremio][DremioIceberg], [DuckDB][DuckDBIceberg], [Apache Doris][DorisIceberg], [RisingWave][RisingWaveIceberg], ClickHouse, and [LanceDB][LanceDB]. Catalog commits are atomic compare-and-swap, so concurrent writers are safe. [Lakekeeper][Lakekeeper] can front the same storage with STS-vended credentials.
|
||||
* Query engines operate on the same tables at the same time: [Spark][SparkIceberg], [Trino][TrinoIceberg], [Dremio][DremioIceberg], [DuckDB][DuckDBIceberg], [Apache Doris][DorisIceberg], [RisingWave][RisingWaveIceberg], ClickHouse, and [LanceDB][LanceDB]. Catalog commits are atomic compare-and-swap, so concurrent writers are safe. [Lakekeeper][Lakekeeper] can front the same storage with STS-vended credentials. [OLake][OLake] replicates PostgreSQL, MySQL, and MongoDB into the same tables with change data capture.
|
||||
* [Automated table maintenance][IcebergMaintenance]: compaction, snapshot expiration, orphan file removal, and manifest rewriting, configured per bucket or table through the S3 Tables maintenance APIs, and the same for [Lance][LanceMaintenance].
|
||||
* IAM at the bucket, namespace, and table level with standard bucket policies, see [S3 Tables Security][S3TablesSecurity].
|
||||
* A [Hadoop compatible file system][Hadoop] for Spark, Flink, and HBase.
|
||||
@@ -487,6 +487,7 @@ Your support will be really appreciated by me and other supporters!
|
||||
[RisingWaveIceberg]: https://github.com/seaweedfs/seaweedfs/wiki/RisingWave-Iceberg-Integration
|
||||
[LanceDB]: https://github.com/seaweedfs/seaweedfs/wiki/LanceDB-Integration
|
||||
[Lakekeeper]: https://github.com/seaweedfs/seaweedfs/wiki/Lakekeeper-Iceberg-Integration
|
||||
[OLake]: https://github.com/seaweedfs/seaweedfs/wiki/OLake-Iceberg-Integration
|
||||
[IcebergMaintenance]: https://github.com/seaweedfs/seaweedfs/wiki/Iceberg-Table-Maintenance
|
||||
[LanceMaintenance]: https://github.com/seaweedfs/seaweedfs/wiki/Lance-Maintenance-Worker
|
||||
[S3TablesSecurity]: https://github.com/seaweedfs/seaweedfs/wiki/S3-Tables-Security
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
An integration test for [OLake](https://github.com/datazip-inc/olake) against
|
||||
SeaweedFS's Iceberg REST Catalog, in the same shape as `catalog_clickhouse`.
|
||||
|
||||
## Why OLake, given we already test five engines
|
||||
## Why OLake, given we already test other engines
|
||||
|
||||
Two things here are covered by nothing else in this directory.
|
||||
|
||||
@@ -73,8 +73,9 @@ Nothing in the destination config is SeaweedFS-specific:
|
||||
```
|
||||
|
||||
`catalog_type` is the generic `rest`, auth is the standard OAuth2
|
||||
client-credentials flow, and `s3_path_style` does not even need setting —
|
||||
OLake turns it on by itself whenever `s3_endpoint` is non-empty. As of
|
||||
client-credentials flow, and the test config sets `s3_path_style` explicitly;
|
||||
OLake also defaults to path style whenever `s3_endpoint` is non-empty, so
|
||||
either form works. As of
|
||||
OLake v0.10.1 this works with no change on either side.
|
||||
|
||||
## Running it
|
||||
|
||||
@@ -376,8 +376,8 @@ func (env *TestEnvironment) execSQL(t *testing.T, sqlText string) {
|
||||
// writeOLakeConfigs writes the source and destination configs. The destination
|
||||
// is the point of this test: nothing in it is SeaweedFS-specific. catalog_type
|
||||
// is the generic "rest", auth is the standard OAuth2 client-credentials flow,
|
||||
// and path-style access is not even set here because OLake turns it on by
|
||||
// itself whenever s3_endpoint is non-empty.
|
||||
// and path-style access is set explicitly here; OLake also turns it on by
|
||||
// itself whenever s3_endpoint is non-empty, so either form works.
|
||||
func (env *TestEnvironment) writeOLakeConfigs(t *testing.T) {
|
||||
t.Helper()
|
||||
|
||||
|
||||
@@ -342,7 +342,7 @@ dataset = lance.dataset(table.location, storage_options=table.storage_options)`
|
||||
}
|
||||
<label class="btn btn-outline-primary w-100 text-start p-3" for="s3tablesBucketFormatIceberg">
|
||||
<span class="d-block fw-semibold"><i class="bi bi-snow me-1"></i>Iceberg</span>
|
||||
<span class="d-block small text-muted mt-1">Served over the Iceberg REST catalog. Spark, Trino, PyIceberg, DuckDB.</span>
|
||||
<span class="d-block small text-muted mt-1">Served over the Iceberg REST catalog. Spark, Trino, PyIceberg, DuckDB, OLake Go.</span>
|
||||
</label>
|
||||
</div>
|
||||
<div class="col-6">
|
||||
|
||||
@@ -636,7 +636,7 @@ dataset = lance.dataset(table.location, storage_options=table.storage_options)`)
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
}
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 61, "<label class=\"btn btn-outline-primary w-100 text-start p-3\" for=\"s3tablesBucketFormatIceberg\"><span class=\"d-block fw-semibold\"><i class=\"bi bi-snow me-1\"></i>Iceberg</span> <span class=\"d-block small text-muted mt-1\">Served over the Iceberg REST catalog. Spark, Trino, PyIceberg, DuckDB.</span></label></div><div class=\"col-6\">")
|
||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 61, "<label class=\"btn btn-outline-primary w-100 text-start p-3\" for=\"s3tablesBucketFormatIceberg\"><span class=\"d-block fw-semibold\"><i class=\"bi bi-snow me-1\"></i>Iceberg</span> <span class=\"d-block small text-muted mt-1\">Served over the Iceberg REST catalog. Spark, Trino, PyIceberg, DuckDB, OLake Go.</span></label></div><div class=\"col-6\">")
|
||||
if templ_7745c5c3_Err != nil {
|
||||
return templ_7745c5c3_Err
|
||||
}
|
||||
|
||||
Reference in new issue
Block a user