verify and update integration examples against current versions (Hadoop 3.4.1, Spark 4.0, HBase 2.6, Trino 475)

2026-09-10 16:40:46 +02:00 · 2026-07-08 01:09:49 -07:00
parent 2229f31c59
commit 8bb90fe474
6 changed files with 455 additions and 281 deletions
@@ -4,43 +4,121 @@ However, the downside is that you need to add a SeaweedFS jar to classpath, and

 # HDFS Access SeaweedFS via S3 connector

-The S3a connector is already included in hadoop distributions. You can use it directly.
+The S3A connector (`hadoop-aws`) points at the SeaweedFS S3 gateway. It ships with Hadoop distributions, so no SeaweedFS jar is needed.

-Here is an example spark job pom.xml file, using hadoop version later than `3.3.1`: 
+Tested with Spark 4.0.3 (`spark-4.0.3-bin-hadoop3`, bundled Hadoop 3.4.1) on JDK 17. The `hadoop-aws` jar must match the Hadoop version bundled in Spark. For this build that is `hadoop-aws-3.4.1.jar` plus the AWS SDK v2 bundle `bundle-2.24.6.jar`, both found under `share/hadoop/tools/lib/` of a Hadoop 3.4.1 distribution.
+
+## Configuration
+
+Point S3A at the SeaweedFS S3 gateway (default port 8333):
+
+```
+fs.s3a.endpoint=http://localhost:8333
+fs.s3a.path.style.access=true
+fs.s3a.connection.ssl.enabled=false
+```
+
+Create the bucket before writing:
+
+```
+$ aws --endpoint-url http://localhost:8333 s3 mb s3://test
+```
+
+## Credentials
+
+A SeaweedFS S3 gateway started without an identity config allows anonymous access. Use the anonymous provider:
+
+```
+fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider
+```
+
+To require credentials, start the gateway with a static identity config, e.g. `s3.json`:
+
+```
+{
+  "identities": [
+    {
+      "name": "spark",
+      "credentials": [
+        { "accessKey": "sparkkey", "secretKey": "sparksecret" }
+      ],
+      "actions": ["Admin", "Read", "Write", "List", "Tagging"]
+    }
+  ]
+}
+```
+```
+$ weed server -s3 -s3.config=s3.json ...
+```
+Then use the simple provider with those keys:
+
+```
+fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
+fs.s3a.access.key=sparkkey
+fs.s3a.secret.key=sparksecret
+```
+
+## Example
+
+```
+$ bin/spark-shell \
+  --master 'local[2]' \
+  --jars /path/to/hadoop-aws-3.4.1.jar,/path/to/bundle-2.24.6.jar \
+  --conf spark.hadoop.fs.s3a.endpoint=http://localhost:8333 \
+  --conf spark.hadoop.fs.s3a.path.style.access=true \
+  --conf spark.hadoop.fs.s3a.connection.ssl.enabled=false \
+  --conf spark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider
+...
+scala> val df = Seq((10,"x"),(20,"y"),(30,"z")).toDF("id","label")
+scala> df.write.mode("overwrite").parquet("s3a://test/spark-s3a/data")
+scala> val back = spark.read.parquet("s3a://test/spark-s3a/data")
+scala> back.count()
+res: Long = 3
+scala> back.orderBy("id").show(false)
+---+-----+
+|id |label|
+---+-----+
+|10 |x    |
+|20 |y    |
+|30 |z    |
+---+-----+
+```
+
+The write lands in the bucket as `spark-s3a/data/_SUCCESS` plus the snappy parquet part files.
+
+## Packaged spark job
+
+Example `pom.xml` properties for a job compiled against Spark 4.0.3:

 ```
 <properties>
-    <maven.compiler.source>8</maven.compiler.source>
-    <maven.compiler.target>8</maven.compiler.target>
+    <maven.compiler.source>17</maven.compiler.source>
+    <maven.compiler.target>17</maven.compiler.target>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
-    <scala.version>2.12.11</scala.version>
-    <spark.version>3.1.2</spark.version>
-    <hadoop.version>3.3.1</hadoop.version>
+    <scala.version>2.13.16</scala.version>
+    <spark.version>4.0.3</spark.version>
+    <hadoop.version>3.4.1</hadoop.version>
    <spark.pom.scope>compile</spark.pom.scope>
 </properties>
-
 ```
-And add this in your code:
+And set the S3A configuration in your code:
 ```
 SparkSession spark = SparkSession.builder()
    .master("local[*]")
    .config("spark.eventLog.enabled", "false")
-    .config("spark.driver.memory", "1g")
-    .config("spark.executor.memory", "1g")
    .appName("SparkDemoFromS3")
    .getOrCreate();
-spark.sparkContext().hadoopConfiguration().set("fs.s3a.access.key", "admin");
-spark.sparkContext().hadoopConfiguration().set("fs.s3a.secret.key", "xx");
-spark.sparkContext().hadoopConfiguration().set("fs.s3a.endpoint", "ip:8333");
-spark.sparkContext().hadoopConfiguration().set("com.amazonaws.services.s3a.enableV4", "true");
-spark.sparkContext().hadoopConfiguration().set("fs.s3a.path.style.access", "true");
-spark.sparkContext().hadoopConfiguration().set("fs.s3a.connection.ssl.enabled", "false");
-spark.sparkContext().hadoopConfiguration().set("fs.s3a.multiobjectdelete.enable", "false");
-spark.sparkContext().hadoopConfiguration().set("fs.s3a.directory.marker.retention", "keep");
-spark.sparkContext().hadoopConfiguration().set("fs.s3a.change.detection.version.required", "false");
-spark.sparkContext().hadoopConfiguration().set("fs.s3a.change.detection.mode", "warn");
-RDD<String> rdd = spark.sparkContext().textFile("s3a://bk002/test1.txt", 1);
-System.out.println(rdd.count());
-rdd.saveAsTextFile("s3a://bk002/testcc/t2");
-
+Configuration conf = spark.sparkContext().hadoopConfiguration();
+conf.set("fs.s3a.endpoint", "http://localhost:8333");
+conf.set("fs.s3a.path.style.access", "true");
+conf.set("fs.s3a.connection.ssl.enabled", "false");
+// anonymous access
+conf.set("fs.s3a.aws.credentials.provider", "org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider");
+// or, with a static identity:
+// conf.set("fs.s3a.aws.credentials.provider", "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider");
+// conf.set("fs.s3a.access.key", "sparkkey");
+// conf.set("fs.s3a.secret.key", "sparksecret");
+Dataset<Row> df = spark.read().parquet("s3a://test/spark-s3a/data");
+System.out.println(df.count());
+df.write().mode("overwrite").parquet("s3a://test/testcc/t2");
 ```
@@ -1,10 +1,10 @@
 # Setup Hadoop Benchmark

-Here are my steps. First, checkout hadoop 2.10.0 binary, untar, and cd in to the hadoop directory.
+Here are my steps. First, checkout the hadoop 3.4.1 binary, untar, and cd in to the hadoop directory.
 ```
-wget http://apache.mirrors.hoobly.com/hadoop/common/hadoop-2.10.0/hadoop-2.10.0.tar.gz
-tar xvf hadoop-2.10.0.tar.gz
-cd hadoop-2.10.0
+wget https://downloads.apache.org/hadoop/common/hadoop-3.4.1/hadoop-3.4.1.tar.gz
+tar xvf hadoop-3.4.1.tar.gz
+cd hadoop-3.4.1
 ```

 Modify the file `./etc/hadoop/core-site.xml`
@@ -19,6 +19,10 @@ Modify the file `./etc/hadoop/core-site.xml`
        <name>fs.defaultFS</name>
        <value>seaweedfs://localhost:8888</value>
    </property>
+    <property>
+       <name>fs.AbstractFileSystem.seaweedfs.impl</name>
+       <value>seaweed.hdfs.SeaweedAbstractFileSystem</value>
+    </property>
 </configuration>
 ```

@@ -27,6 +31,7 @@ Then get the seaweedfs hadoop client jar.
 ```
 cd share/hadoop/common/lib/
 wget https://repo1.maven.org/maven2/com/seaweedfs/seaweedfs-hadoop3-client/4.38/seaweedfs-hadoop3-client-4.38.jar
+cd ../../../..
 ```

 # TestDFSIO Benchmark
@@ -39,18 +44,17 @@ The TestDFSIO benchmark is used for measuring I/O (read/write) performance.
 Start the TestDFSIO write tests:

 ```
-bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.10.0-tests.jar TestDFSIO -write -nrFiles 8 -size 32GB -bufferSize 8388608 -resFile /tmp/TestDFSIOwrite.txt
+bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.4.1-tests.jar TestDFSIO -write -nrFiles 4 -size 64MB -bufferSize 8388608 -resFile /tmp/TestDFSIOwrite.txt

 ...
-20/07/25 16:48:21 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read
-20/07/25 16:48:21 INFO fs.TestDFSIO:             Date & time: Sat Jul 25 16:48:21 PDT 2020
-20/07/25 16:48:21 INFO fs.TestDFSIO:         Number of files: 8
-20/07/25 16:48:21 INFO fs.TestDFSIO:  Total MBytes processed: 262144
-20/07/25 16:48:21 INFO fs.TestDFSIO:       Throughput mb/sec: 399.16
-20/07/25 16:48:21 INFO fs.TestDFSIO:  Average IO rate mb/sec: 399.34
-20/07/25 16:48:21 INFO fs.TestDFSIO:   IO rate std deviation: 8.56
-20/07/25 16:48:21 INFO fs.TestDFSIO:      Test exec time sec: 659.45
-20/07/25 16:48:21 INFO fs.TestDFSIO:
+INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
+INFO fs.TestDFSIO:             Date & time: Wed Jul 08 00:45:01 PDT 2026
+INFO fs.TestDFSIO:         Number of files: 4
+INFO fs.TestDFSIO:  Total MBytes processed: 256
+INFO fs.TestDFSIO:       Throughput mb/sec: 605.2
+INFO fs.TestDFSIO:  Average IO rate mb/sec: 633.27
+INFO fs.TestDFSIO:   IO rate std deviation: 130.3
+INFO fs.TestDFSIO:      Test exec time sec: 1.32
 ```

 ## TestDFSIO read tests
@@ -58,19 +62,24 @@ bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.10.0
 Start the TestDFSIO read tests:

 ```
-bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.10.0-tests.jar TestDFSIO -read -nrFiles 8 -size 32GB -bufferSize 8388608 -resFile /tmp/TestDFSIOwrite.txt
+bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.4.1-tests.jar TestDFSIO -read -nrFiles 4 -size 64MB -bufferSize 8388608 -resFile /tmp/TestDFSIOread.txt

 ...

-20/07/17 15:59:38 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read
-20/07/17 15:59:38 INFO fs.TestDFSIO:             Date & time: Fri Jul 17 15:59:38 PDT 2020
-20/07/17 15:59:38 INFO fs.TestDFSIO:         Number of files: 8
-20/07/17 15:59:38 INFO fs.TestDFSIO:  Total MBytes processed: 8192
-20/07/17 15:59:38 INFO fs.TestDFSIO:       Throughput mb/sec: 393.26
-20/07/17 15:59:38 INFO fs.TestDFSIO:  Average IO rate mb/sec: 393.72
-20/07/17 15:59:38 INFO fs.TestDFSIO:   IO rate std deviation: 13.33
-20/07/17 15:59:38 INFO fs.TestDFSIO:      Test exec time sec: 22.76
-20/07/17 15:59:38 INFO fs.TestDFSIO:
+INFO fs.TestDFSIO: ----- TestDFSIO ----- : read
+INFO fs.TestDFSIO:             Date & time: Wed Jul 08 00:45:10 PDT 2026
+INFO fs.TestDFSIO:         Number of files: 4
+INFO fs.TestDFSIO:  Total MBytes processed: 256
+INFO fs.TestDFSIO:       Throughput mb/sec: 1174.31
+INFO fs.TestDFSIO:  Average IO rate mb/sec: 1226.51
+INFO fs.TestDFSIO:   IO rate std deviation: 227.6
+INFO fs.TestDFSIO:      Test exec time sec: 1.28
+```
+
+Clean up the benchmark data when done:
+
+```
+bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.4.1-tests.jar TestDFSIO -clean
 ```

 # Independent Benchmarks
@@ -29,17 +29,17 @@ Or you can download the latest version from MavenCentral

 # Test SeaweedFS on Hadoop

-Suppose you are getting a new Hadoop installation. Here are the minimum steps to get SeaweedFS to run.
+Suppose you are getting a new Hadoop installation. Here are the minimum steps to get SeaweedFS to run. This was verified with Hadoop 3.4.1 on JDK 17.

-You would need to start a weed filer first, build the seaweedfs-hadoop3-client-4.38.jar, and do the following:
+You would need to start a weed filer first, build or download the seaweedfs-hadoop3-client-4.38.jar, and do the following:

 ```
 # optionally adjust hadoop memory allocation
 $ export HADOOP_CLIENT_OPTS="-Xmx4g"

 $ cd ${HADOOP_HOME}
-# create etc/hadoop/mapred-site.xml, just to satisfy hdfs dfs. skip this if the file already exists.
-$ echo "<configuration></configuration>" > etc/hadoop/mapred-site.xml
+# Hadoop 3.x already ships etc/hadoop/mapred-site.xml. Create it only if it is missing.
+$ [ -f etc/hadoop/mapred-site.xml ] || echo "<configuration></configuration>" > etc/hadoop/mapred-site.xml

 # on hadoop3
 $ bin/hdfs dfs -Dfs.defaultFS=seaweedfs://localhost:8888 \
@@ -50,6 +50,22 @@ $ bin/hdfs dfs -Dfs.defaultFS=seaweedfs://localhost:8888 \
 ```
 Both reads and writes are working fine.

+A MapReduce job works the same way, passing the client jar with `-libjars`. This teragen writes to SeaweedFS, and terasort reads it back and writes the sorted output, all on `seaweedfs://`:
+
+```
+$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.4.1.jar teragen \
+               -Dfs.defaultFS=seaweedfs://localhost:8888 \
+               -Dfs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem \
+               -libjars ./seaweedfs-hadoop3-client-4.38.jar \
+               100000 seaweedfs://localhost:8888/teragen-out
+
+$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.4.1.jar terasort \
+               -Dfs.defaultFS=seaweedfs://localhost:8888 \
+               -Dfs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem \
+               -libjars ./seaweedfs-hadoop3-client-4.38.jar \
+               seaweedfs://localhost:8888/teragen-out seaweedfs://localhost:8888/terasort-out
+```
+
 # Installation for Hadoop
 * Configure Hadoop to use SeaweedFS in `etc/hadoop/conf/core-site.xml`. `core-site.xml` resides on each node in the Hadoop cluster. You must add the same properties to each instance of `core-site.xml`. There are several properties to modify:
    1. `fs.seaweedfs.impl`: This property defines the Seaweed HCFS implementation classes that are contained in the SeaweedFS HDFS client JAR. It is required.
@@ -1,152 +1,182 @@
-# Installation for Presto
-The installation steps are divided into 2 steps:
-## install Hive Metastore
-### Follow instructions for installation of Hive Metastore
-* https://cwiki.apache.org/confluence/display/Hive/AdminManual+Metastore+Administration
+# Run Trino (Presto) on SeaweedFS

-### Configure Hive Metastore to support SeaweedFS
-1. Copy the seaweedfs-hadoop3-client-4.38.jar to hive lib directory,for example:
+Trino (the maintained successor to PrestoDB / PrestoSQL) queries data stored on
+SeaweedFS through its Hive connector. These steps replace the older PrestoDB 347
+instructions. The mechanism changed in current Trino: the `hive-hadoop2`
+connector was renamed to `hive`, and the Hadoop FileSystem path is now opt-in
+through `fs.hadoop.enabled`.
+
+Two storage paths work, both verified with Trino 475, Eclipse Temurin JDK 23,
+and `seaweedfs-hadoop3-client-4.38.jar`:
+
+- **A. HCFS** - the Hive connector talks to the filer directly through the
+  `seaweedfs-hadoop3-client` jar (`seaweedfs://` URIs).
+- **B. S3 gateway** - the Hive connector talks to SeaweedFS's S3 gateway with
+  Trino's built-in S3 filesystem (`s3://` URIs). Simpler: no extra jar, no
+  `core-site.xml`.
+
+JDK note: Trino 475 is the last release that runs on JDK 23. Trino 476+ needs
+JDK 24 and 479+ needs JDK 25. Use the JDK your Trino version requires; the config
+below is otherwise the same.
+
+## Metastore
+
+The Hive connector needs a metastore. This guide uses Trino's built-in file
+metastore (`hive.metastore=file`), which stores table metadata as files beside
+the data, so there is no separate service to run. For production, run a
+standalone Hive Metastore and set `hive.metastore=thrift` instead.
+
+## Trino base config
+
+`etc/node.properties`:
 ```
-cp seaweedfs-hadoop3-client-4.38.jar /opt/hadoop/share/hadoop/common/lib/
-cp seaweedfs-hadoop3-client-4.38.jar /opt/hive-metastore/lib/
+node.environment=test
+node.id=trino-1
+node.data-dir=/opt/trino/data
 ```
-2. Modify core-site.xml  
-modify core-site.xml to support SeaweedFS, 30888 is the filer port
+
+`etc/config.properties` (single node coordinator, 8080 is the Trino port):
+```
+coordinator=true
+node-scheduler.include-coordinator=true
+http-server.http.port=8080
+discovery.uri=http://localhost:8080
+```
+
+`etc/jvm.config` (the last two flags are required on JDK 23/24):
+```
+-server
+-Xmx3G
+-XX:InitialRAMPercentage=60
+-XX:MaxRAMPercentage=60
+-XX:+ExitOnOutOfMemoryError
+-XX:+HeapDumpOnOutOfMemoryError
+-XX:-OmitStackTraceInFastThrow
+-Djdk.attach.allowAttachSelf=true
+-Dfile.encoding=UTF-8
+--sun-misc-unsafe-memory-access=allow
+--enable-native-access=ALL-UNNAMED
+```
+
+## A. HCFS with seaweedfs-hadoop3-client
+
+1. Put the client jar on the Hive connector classpath:
+```
+cp seaweedfs-hadoop3-client-4.38.jar /opt/trino/plugin/hive/hdfs/
+```
+
+2. `etc/catalog/core-site.xml`:
 ```
 <configuration>
-	<property>
-	    <name>fs.defaultFS</name>
-	    <value>seaweedfs://10.0.100.51:30888</value>
-	</property>
-	<property>
-	    <name>fs.seaweedfs.impl</name>
-	    <value>seaweed.hdfs.SeaweedFileSystem</value>
-	</property>
-	<property>
-	   <name>fs.AbstractFileSystem.seaweedfs.impl</name>
-	   <value>seaweed.hdfs.SeaweedAbstractFileSystem</value>
-	</property>
-	<property>
-	   <name>fs.seaweed.buffer.size</name>
-	   <value>4194304</value>
-	</property>
-</configuration>
-```
-3. Modify hive-site.xml  
-modify hive-site.xml to support SeaweedFS, need to manually create the /presto/warehouse directory in Filer  
-metastore.thrift.port is the access port exposed by the Hive Metadata service itself
-```
-	<property>
-	    <name>metastore.warehouse.dir</name>
-	    <value>seaweedfs://10.0.100.51:30888/presto/warehouse</value>
-	</property>
-	<property>
-	    <name>metastore.thrift.port</name>
-	    <value>9850</value>
-	</property>
-```
-
-## install Presto
-Follow instructions for installation of Presto:
-* https://prestosql.io/docs/current/installation/deployment.html
-### Configure Presto to support SeaweedFS
-1. Copy the seaweedfs-hadoop3-client-4.38.jar to Presto directory,for example:
-```
-cp seaweedfs-hadoop3-client-4.38.jar /opt/presto-server-347/plugin/hive-hadoop3/
-```
-2. Modify core-site.xml
-
-modify /opt/presto-server-347/etc/catalog/core-site.xml to support SeaweedFS, 30888 is the filer port
-```
-<configuration>
-    <property>
-        <name>fs.defaultFS</name>
-        <value>seaweedfs://10.0.100.51:30888</value>
-    </property>
    <property>
        <name>fs.seaweedfs.impl</name>
        <value>seaweed.hdfs.SeaweedFileSystem</value>
    </property>
    <property>
-       <name>fs.AbstractFileSystem.seaweedfs.impl</name>
-       <value>seaweed.hdfs.SeaweedAbstractFileSystem</value>
+        <name>fs.AbstractFileSystem.seaweedfs.impl</name>
+        <value>seaweed.hdfs.SeaweedAbstractFileSystem</value>
    </property>
    <property>
-       <name>fs.seaweed.buffer.size</name>
-       <value>4194304</value>
+        <name>fs.seaweed.buffer.size</name>
+        <value>4194304</value>
    </property>
 </configuration>
-
-```
-3. Modify hive.properties  
-hive.metastore.uri is the service address of the previously deployed Hive Metastore  
-hive.config.resources points to the core-site.xml above
-```
-connector.name=hive-hadoop2
-hive.metastore.uri=thrift://10.0.100.51:9850
-hive.allow-drop-table=true
-hive.max-partitions-per-scan=1000000
-hive.compression-codec=NONE
-hive.config.resources=/opt/presto-server-347/etc/catalog/core-site.xml
-```
-4. Modify config.properties  
-The default port of presto is 8080  
-If you want to modify the default port of the Presto service, you can modify /opt/presto-server-347/etc/config.properties  
-Need to modify the ports of http-server.http.port and discovery.uri
-
-```
-coordinator=true
-node-scheduler.include-coordinator=true
-http-server.http.port=8080
-query.max-memory=200GB
-query.max-memory-per-node=8GB
-query.max-total-memory-per-node=10GB
-query.max-stage-count=200
-task.writer-count=4
-discovery-server.enabled=true
-discovery.uri=http://10.0.100.51:8080
 ```

-# Using Examples
-1. Connect to Presto and create a table boshen  
--server is the ip and port of the Presto service
+3. `etc/catalog/seaweedfs.properties` (8888 is the filer port):
 ```
-[root@cluster9 ~]# ./presto --server 10.0.100.51:8080--catalog hive --schema default
-presto:default> create table boshen(name varchar);
-CREATE TABLE
-presto:default> 
+connector.name=hive
+hive.metastore=file
+hive.metastore.catalog.dir=seaweedfs://localhost:8888/presto/warehouse
+fs.hadoop.enabled=true
+hive.config.resources=/opt/trino/etc/catalog/core-site.xml
 ```
-2. Query whether the boshen directory has been generated in 10.0.10.51:30888/presto/warehouse
+
+`fs.hadoop.enabled=true` is required: current Trino uses native object-storage
+filesystems by default and only loads a Hadoop FileSystem implementation
+(SeaweedFS's) when this is set. The file metastore creates
+`seaweedfs://localhost:8888/presto/warehouse` on demand, so no directory needs
+to be precreated.
+
+## B. S3 gateway with native S3
+
+1. Run SeaweedFS with the S3 gateway and one identity with credentials:
+```
+weed server -dir=/data -filer -s3 -s3.config=s3config.json
+```
+`s3config.json`:
 ```
-[root@cluster9 ~]# curl -H "Accept: application/json"  http://10.0.100.51:30888/presto/warehouse/?pretty=y
 {
-  "Path": "presto/warehouse",
-  "Entries": [
+  "identities": [
    {
-      "FullPath": "/presto/warehouse/boshen",
-      "Mtime": "2020-12-02T20:29:08+08:00",
-      "Crtime": "2020-12-02T20:29:08+08:00",
-      "Mode": 2147484159,
-      "Uid": 0,
-      "Gid": 0,
-      "Mime": "",
-      "Replication": "",
-      "Collection": "",
-      "TtlSec": 0,
-      "UserName": "root",
-      "GroupNames": [
-        "root"
+      "name": "trino",
+      "credentials": [
+        { "accessKey": "trinokey", "secretKey": "trinosecret" }
      ],
-      "SymlinkTarget": "",
-      "Md5": null,
-      "FileSize": 0,
-      "Extended": null,
-      "HardLinkId": null,
-      "HardLinkCounter": 0
+      "actions": ["Admin", "Read", "Write", "List", "Tagging"]
    }
-  ],
-  "Limit": 100,
-  "LastFileName": "boshen",
-  "ShouldDisplayLoadMore": false
+  ]
 }
 ```
+Create a bucket for the warehouse (8333 is the S3 gateway port):
+```
+AWS_ACCESS_KEY_ID=trinokey AWS_SECRET_ACCESS_KEY=trinosecret \
+  aws --endpoint-url http://localhost:8333 s3 mb s3://warehouse
+```
+
+2. `etc/catalog/seaweedfs.properties`:
+```
+connector.name=hive
+hive.metastore=file
+hive.metastore.catalog.dir=s3://warehouse/
+fs.native-s3.enabled=true
+s3.endpoint=http://localhost:8333
+s3.region=us-east-1
+s3.path-style-access=true
+s3.aws-access-key=trinokey
+s3.aws-secret-key=trinosecret
+```
+No jar and no `core-site.xml` are needed for this path.
+
+## Run a query
+
+Start Trino and connect with the Trino CLI:
+```
+bin/launcher start
+trino --server http://localhost:8080 --catalog seaweedfs
+```
+```
+trino> CREATE SCHEMA seaweedfs.demo;
+CREATE SCHEMA
+trino> CREATE TABLE seaweedfs.demo.boshen (name varchar);
+CREATE TABLE
+trino> INSERT INTO seaweedfs.demo.boshen VALUES ('alice'), ('bob'), ('carol');
+INSERT: 3 rows
+trino> SELECT * FROM seaweedfs.demo.boshen ORDER BY name;
+ name
+-------
+ alice
+ bob
+ carol
+(3 rows)
+```
+
+## Confirm the data is on SeaweedFS
+
+The table is stored as ORC under the metastore catalog directory.
+
+HCFS - list the table directory on the filer:
+```
+curl -H "Accept: application/json" \
+  http://localhost:8888/presto/warehouse/demo/boshen/
+```
+
+S3 - list the bucket:
+```
+AWS_ACCESS_KEY_ID=trinokey AWS_SECRET_ACCESS_KEY=trinosecret \
+  aws --endpoint-url http://localhost:8333 s3 ls s3://warehouse/demo/boshen/ --recursive
+```
+
+Either way you will see the ORC data file, named like
+`20260708_080447_00007_4q5f3_<uuid>`, next to the `.trinoSchema` metadata the
+file metastore wrote. The data file begins and ends with the `ORC` magic bytes.
@@ -1,30 +1,75 @@
 # Installation for HBase

-Two steps to run HBase on SeaweedFS
-1. Copy the seaweedfs-hadoop3-client-4.38.jar to `${HBASE_HOME}/lib`
-1. And add the following 2 properties in `${HBASE_HOME}/conf/hbase-site.xml`
+Verified with Apache HBase 2.6.2 (standalone) on JDK 17, using `seaweedfs-hadoop3-client-4.38.jar`.
+HBase talks to the SeaweedFS filer through the HCFS (`seaweedfs://`) scheme, so no S3 gateway is
+needed. HBase 2.6.x ships Hadoop 2.10 jars; the hadoop3 client works against them since it only
+adds the `seaweed.hdfs.*` classes and shades its own dependencies.
+
+Three steps to run HBase on SeaweedFS:
+
+1. Run HBase on JDK 8, 11, or 17. The `bin/hbase` launcher already adds the JDK 17 `--add-opens`
+   flags, so setting `JAVA_HOME` is enough:
+
+    ```
+    export JAVA_HOME=/path/to/jdk-17
+    ```
+
+2. Copy `seaweedfs-hadoop3-client-4.38.jar` to `${HBASE_HOME}/lib`.
+
+3. Set the following properties in `${HBASE_HOME}/conf/hbase-site.xml`. Point `hbase.rootdir` at
+   your filer (default filer port is `8888`) and give `hbase.tmp.dir` a local path:
+
+    ```
+    <configuration>
+      <property>
+        <name>hbase.cluster.distributed</name>
+        <value>false</value>
+      </property>
+      <property>
+        <name>hbase.rootdir</name>
+        <value>seaweedfs://localhost:8888/hbase</value>
+      </property>
+      <property>
+        <name>hbase.tmp.dir</name>
+        <value>/var/lib/hbase/tmp</value>
+      </property>
+      <property>
+        <name>fs.seaweedfs.impl</name>
+        <value>seaweed.hdfs.SeaweedFileSystem</value>
+      </property>
+      <property>
+        <name>fs.AbstractFileSystem.seaweedfs.impl</name>
+        <value>seaweed.hdfs.SeaweedAbstractFileSystem</value>
+      </property>
+      <property>
+        <name>hbase.unsafe.stream.capability.enforce</name>
+        <value>false</value>
+      </property>
+    </configuration>
+    ```
+
+    The SeaweedFS HCFS stream does not advertise `hflush`/`hsync`, so
+    `hbase.unsafe.stream.capability.enforce=false` is required or HBase refuses to start.
+
+Start HBase:

 ```
-<configuration>
-  <property>
-    <name>hbase.cluster.distributed</name>
-    <value>true</value>
-  </property>
-  <property>
-    <name>hbase.rootdir</name>
-    <value>seaweedfs://localhost:8888/hbase</value>
-  </property>
-  <property>
-    <name>fs.seaweedfs.impl</name>
-    <value>seaweed.hdfs.SeaweedFileSystem</value>
-  </property>
-  <property>
-    <name>hbase.unsafe.stream.capability.enforce</name>
-    <value>false</value>
-  </property>
-</configuration>
+${HBASE_HOME}/bin/start-hbase.sh
 ```

-Visit HBase Web UI at `http://<hostname>:16010` to confirm that HBase is running on SeaweedFS
+Visit the HBase Web UI at `http://<hostname>:16010` to confirm HBase is running on SeaweedFS
+(the "HBase Root Directory" reads `seaweedfs://...`).

 ![](HBaseOnSeaweedFS.png)
+
+Verify from `hbase shell`:
+
+```
+create 'swtest', 'cf'
+put 'swtest', 'row1', 'cf:name', 'seaweed'
+scan 'swtest'
+```
+
+The store files land under the filer, e.g. `weed shell` / `curl` on
+`seaweedfs://localhost:8888/hbase/data/default/swtest/...`. Tables and rows survive a
+`stop-hbase.sh` / `start-hbase.sh` restart.
@@ -3,6 +3,8 @@ Follow instructions on spark doc:
 * https://spark.apache.org/docs/latest/configuration.html#inheriting-hadoop-cluster-configuration
 * https://spark.apache.org/docs/latest/configuration.html#custom-hadoophive-configuration

+Tested with Spark 4.0.3 (`spark-4.0.3-bin-hadoop3`, Scala 2.13, bundled Hadoop 3.4.1) on JDK 17. Spark 4.0.x requires Java 17+. The same steps work with Spark 3.5.x on JDK 17.
+
 ## installation inheriting from Hadoop cluster configuration

 Inheriting from Hadoop cluster configuration should be the easiest way. 
@@ -17,102 +19,96 @@ Add the following to spark/conf/spark-defaults.conf on every node running Spark
 ```
 spark.driver.extraClassPath=/path/to/seaweedfs-hadoop3-client-4.38.jar
 spark.executor.extraClassPath=/path/to/seaweedfs-hadoop3-client-4.38.jar
+spark.hadoop.fs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem
+spark.hadoop.fs.AbstractFileSystem.seaweedfs.impl=seaweed.hdfs.SeaweedAbstractFileSystem
 ```
+`fs.AbstractFileSystem.seaweedfs.impl` is required on Hadoop 3.x.

 And modify the configuration at runtime:

 ```
-./bin/spark-submit \ 
-  --name "My app" \ 
-  --master local[4] \  
-  --conf spark.eventLog.enabled=false \ 
-  --conf "spark.executor.extraJavaOptions=-XX:+PrintGCDetails -XX:+PrintGCTimeStamps" \ 
-  --conf spark.hadoop.fs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem \ 
-  --conf spark.hadoop.fs.defaultFS=seaweedfs://localhost:8888 \ 
+./bin/spark-submit \
+  --name "My app" \
+  --master local[4] \
+  --conf spark.eventLog.enabled=false \
+  --conf spark.hadoop.fs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem \
+  --conf spark.hadoop.fs.AbstractFileSystem.seaweedfs.impl=seaweed.hdfs.SeaweedAbstractFileSystem \
+  --conf spark.hadoop.fs.defaultFS=seaweedfs://localhost:8888 \
+  --jars /path/to/seaweedfs-hadoop3-client-4.38.jar \
  myApp.jar
 ```

 # Example

-  1. change the spark-defaults.conf
+Put the client jar on the classpath and point Spark at the filer. Start a filer first (`weed filer`), then:

 ```
-spark.driver.extraClassPath=/Users/chris/go/src/github.com/seaweedfs/seaweedfs/other/java/hdfs3/target/seaweedfs-hadoop3-client-4.38.jar
-spark.executor.extraClassPath=/Users/chris/go/src/github.com/seaweedfs/seaweedfs/other/java/hdfs3/target/seaweedfs-hadoop3-client-4.38.jar
-spark.hadoop.fs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem
-```
-
-  2. create the spark history folder
-```
-$ curl -X POST http://192.168.2.3:8888/spark2-history/
-```
-  3. Run a spark job
-```
-$ bin/spark-submit --name spark-pi \
--class org.apache.spark.examples.SparkPi \
--conf spark.jars.ivy=/tmp/.ivy \
--conf spark.eventLog.enabled=true \
--conf spark.hadoop.fs.defaultFS=seaweedfs://192.168.2.3:8888 \
--conf spark.eventLog.dir=seaweedfs://192.168.2.3:8888/spark2-history/ \
-file:///usr/local/spark/examples/jars/spark-examples_2.12-3.0.0.jar
-
-```
-
-
-# A Full Example
-Here is my local example switching everything to SeaweedFS. In the `/usr/local/spark/conf/spark-defaults.conf` file, 
-
-  1. this is my local `/usr/local/spark/conf/spark-defaults.conf`
-```
-spark.eventLog.enabled=true
-spark.sql.hive.convertMetastoreOrc=true
-spark.yarn.queue=default
-spark.master=local
-spark.history.ui.port=18081
-spark.history.fs.cleaner.interval=7d
-spark.sql.statistics.fallBackToHdfs=true
-spark.yarn.historyServer.address=master:18081
-spark.sql.orc.filterPushdown=true
-spark.history.provider=org.apache.spark.deploy.history.FsHistoryProvider
-spark.history.fs.cleaner.maxAge=90d
-spark.sql.orc.impl=native
-spark.history.fs.cleaner.enabled=true
-
-spark.history.fs.logDirectory=seaweedfs://localhost:8888/spark2-history/
-spark.eventLog.dir=seaweedfs://localhost:8888/spark2-history/
-
-spark.driver.extraClassPath=/Users/chris/go/src/github.com/seaweedfs/seaweedfs/other/java/hdfs3/target/seaweedfs-hadoop3-client-4.38.jar
-spark.executor.extraClassPath=/Users/chris/go/src/github.com/seaweedfs/seaweedfs/other/java/hdfs3/target/seaweedfs-hadoop3-client-4.38.jar
-spark.hadoop.fs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem
-spark.hadoop.fs.defaultFS=seaweedfs://localhost:8888
-```
-  2. create the spark history folder
-```
-$ curl -X POST http://192.168.2.4:8888/spark2-history/
-```
-  3. Run a spark shell
-```
-$ bin/spark-shell
-20/10/18 14:11:44 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
-20/10/18 14:12:15 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
-Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
-Setting default log level to "WARN".
-To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
-Spark context Web UI available at http://192.168.2.4:4040
-Spark context available as 'sc' (master = local, app id = local-1603055539864).
-Spark session available as 'spark'.
+$ bin/spark-shell \
+  --master 'local[2]' \
+  --conf spark.driver.extraClassPath=/path/to/seaweedfs-hadoop3-client-4.38.jar \
+  --conf spark.executor.extraClassPath=/path/to/seaweedfs-hadoop3-client-4.38.jar \
+  --conf spark.hadoop.fs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem \
+  --conf spark.hadoop.fs.AbstractFileSystem.seaweedfs.impl=seaweed.hdfs.SeaweedAbstractFileSystem
+...
 Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
-   /___/ .__/\_,_/_/ /_/\_\   version 3.0.0
+   /___/ .__/\_,_/_/ /_/\_\   version 4.0.3
      /_/

-Using Scala version 2.12.10 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_202)
-Type in expressions to have them evaluated.
-Type :help for more information.
-
-scala> sc.textFile("/buckets/large/ttt.txt").count
-res0: Long = 9374
+Using Scala version 2.13.16 (OpenJDK 64-Bit Server VM, Java 17.0.19)

+scala> val df = Seq((1,"alice"),(2,"bob"),(3,"carol"),(4,"dave")).toDF("id","name")
+scala> df.write.mode("overwrite").parquet("seaweedfs://localhost:8888/spark-test/people")
+scala> val back = spark.read.parquet("seaweedfs://localhost:8888/spark-test/people")
+scala> back.count()
+res: Long = 4
+scala> back.orderBy("id").show(false)
+---+-----+
+|id |name |
+---+-----+
+|1  |alice|
+|2  |bob  |
+|3  |carol|
+|4  |dave |
+---+-----+
+```
+
+The write lands under `/spark-test/people/` on the filer as `_SUCCESS` plus the snappy parquet part files. The path prefix `spark-test` is created on write, no need to pre-create it.
+
+# A Full Example
+Here is a local example switching everything to SeaweedFS. In the `$SPARK_HOME/conf/spark-defaults.conf` file,
+
+  1. `spark-defaults.conf`
+```
+spark.master=local[*]
+
+spark.hadoop.fs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem
+spark.hadoop.fs.AbstractFileSystem.seaweedfs.impl=seaweed.hdfs.SeaweedAbstractFileSystem
+spark.hadoop.fs.defaultFS=seaweedfs://localhost:8888
+
+spark.driver.extraClassPath=/path/to/seaweedfs-hadoop3-client-4.38.jar
+spark.executor.extraClassPath=/path/to/seaweedfs-hadoop3-client-4.38.jar
+
+spark.eventLog.enabled=true
+spark.eventLog.dir=seaweedfs://localhost:8888/spark-history/
+spark.history.fs.logDirectory=seaweedfs://localhost:8888/spark-history/
+```
+  2. Run a spark shell. Event logs are written to SeaweedFS under `/spark-history/`.
+```
+$ bin/spark-shell
+...
+Welcome to
+      ____              __
+     / __/__  ___ _____/ /__
+    _\ \/ _ \/ _ `/ __/  '_/
+   /___/ .__/\_,_/_/ /_/\_\   version 4.0.3
+      /_/
+
+Using Scala version 2.13.16 (OpenJDK 64-Bit Server VM, Java 17.0.19)
+Spark session available as 'spark'.
+
+scala> sc.parallelize(1 to 100).sum
+res0: Double = 5050.0
 ```