diff --git a/HDFS-via-S3-connector.md b/HDFS-via-S3-connector.md
index 11aec0e..2446169 100644
--- a/HDFS-via-S3-connector.md
+++ b/HDFS-via-S3-connector.md
@@ -4,43 +4,121 @@ However, the downside is that you need to add a SeaweedFS jar to classpath, and
# HDFS Access SeaweedFS via S3 connector
-The S3a connector is already included in hadoop distributions. You can use it directly.
+The S3A connector (`hadoop-aws`) points at the SeaweedFS S3 gateway. It ships with Hadoop distributions, so no SeaweedFS jar is needed.
-Here is an example spark job pom.xml file, using hadoop version later than `3.3.1`:
+Tested with Spark 4.0.3 (`spark-4.0.3-bin-hadoop3`, bundled Hadoop 3.4.1) on JDK 17. The `hadoop-aws` jar must match the Hadoop version bundled in Spark. For this build that is `hadoop-aws-3.4.1.jar` plus the AWS SDK v2 bundle `bundle-2.24.6.jar`, both found under `share/hadoop/tools/lib/` of a Hadoop 3.4.1 distribution.
+
+## Configuration
+
+Point S3A at the SeaweedFS S3 gateway (default port 8333):
+
+```
+fs.s3a.endpoint=http://localhost:8333
+fs.s3a.path.style.access=true
+fs.s3a.connection.ssl.enabled=false
+```
+
+Create the bucket before writing:
+
+```
+$ aws --endpoint-url http://localhost:8333 s3 mb s3://test
+```
+
+## Credentials
+
+A SeaweedFS S3 gateway started without an identity config allows anonymous access. Use the anonymous provider:
+
+```
+fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider
+```
+
+To require credentials, start the gateway with a static identity config, e.g. `s3.json`:
+
+```
+{
+ "identities": [
+ {
+ "name": "spark",
+ "credentials": [
+ { "accessKey": "sparkkey", "secretKey": "sparksecret" }
+ ],
+ "actions": ["Admin", "Read", "Write", "List", "Tagging"]
+ }
+ ]
+}
+```
+```
+$ weed server -s3 -s3.config=s3.json ...
+```
+Then use the simple provider with those keys:
+
+```
+fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
+fs.s3a.access.key=sparkkey
+fs.s3a.secret.key=sparksecret
+```
+
+## Example
+
+```
+$ bin/spark-shell \
+ --master 'local[2]' \
+ --jars /path/to/hadoop-aws-3.4.1.jar,/path/to/bundle-2.24.6.jar \
+ --conf spark.hadoop.fs.s3a.endpoint=http://localhost:8333 \
+ --conf spark.hadoop.fs.s3a.path.style.access=true \
+ --conf spark.hadoop.fs.s3a.connection.ssl.enabled=false \
+ --conf spark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider
+...
+scala> val df = Seq((10,"x"),(20,"y"),(30,"z")).toDF("id","label")
+scala> df.write.mode("overwrite").parquet("s3a://test/spark-s3a/data")
+scala> val back = spark.read.parquet("s3a://test/spark-s3a/data")
+scala> back.count()
+res: Long = 3
+scala> back.orderBy("id").show(false)
++---+-----+
+|id |label|
++---+-----+
+|10 |x |
+|20 |y |
+|30 |z |
++---+-----+
+```
+
+The write lands in the bucket as `spark-s3a/data/_SUCCESS` plus the snappy parquet part files.
+
+## Packaged spark job
+
+Example `pom.xml` properties for a job compiled against Spark 4.0.3:
```
- 8
- 8
+ 17
+ 17
UTF-8
- 2.12.11
- 3.1.2
- 3.3.1
+ 2.13.16
+ 4.0.3
+ 3.4.1
compile
-
```
-And add this in your code:
+And set the S3A configuration in your code:
```
SparkSession spark = SparkSession.builder()
.master("local[*]")
.config("spark.eventLog.enabled", "false")
- .config("spark.driver.memory", "1g")
- .config("spark.executor.memory", "1g")
.appName("SparkDemoFromS3")
.getOrCreate();
-spark.sparkContext().hadoopConfiguration().set("fs.s3a.access.key", "admin");
-spark.sparkContext().hadoopConfiguration().set("fs.s3a.secret.key", "xx");
-spark.sparkContext().hadoopConfiguration().set("fs.s3a.endpoint", "ip:8333");
-spark.sparkContext().hadoopConfiguration().set("com.amazonaws.services.s3a.enableV4", "true");
-spark.sparkContext().hadoopConfiguration().set("fs.s3a.path.style.access", "true");
-spark.sparkContext().hadoopConfiguration().set("fs.s3a.connection.ssl.enabled", "false");
-spark.sparkContext().hadoopConfiguration().set("fs.s3a.multiobjectdelete.enable", "false");
-spark.sparkContext().hadoopConfiguration().set("fs.s3a.directory.marker.retention", "keep");
-spark.sparkContext().hadoopConfiguration().set("fs.s3a.change.detection.version.required", "false");
-spark.sparkContext().hadoopConfiguration().set("fs.s3a.change.detection.mode", "warn");
-RDD rdd = spark.sparkContext().textFile("s3a://bk002/test1.txt", 1);
-System.out.println(rdd.count());
-rdd.saveAsTextFile("s3a://bk002/testcc/t2");
-
+Configuration conf = spark.sparkContext().hadoopConfiguration();
+conf.set("fs.s3a.endpoint", "http://localhost:8333");
+conf.set("fs.s3a.path.style.access", "true");
+conf.set("fs.s3a.connection.ssl.enabled", "false");
+// anonymous access
+conf.set("fs.s3a.aws.credentials.provider", "org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider");
+// or, with a static identity:
+// conf.set("fs.s3a.aws.credentials.provider", "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider");
+// conf.set("fs.s3a.access.key", "sparkkey");
+// conf.set("fs.s3a.secret.key", "sparksecret");
+Dataset df = spark.read().parquet("s3a://test/spark-s3a/data");
+System.out.println(df.count());
+df.write().mode("overwrite").parquet("s3a://test/testcc/t2");
```
diff --git a/Hadoop-Benchmark.md b/Hadoop-Benchmark.md
index 5a37cc9..9c730cc 100644
--- a/Hadoop-Benchmark.md
+++ b/Hadoop-Benchmark.md
@@ -1,10 +1,10 @@
# Setup Hadoop Benchmark
-Here are my steps. First, checkout hadoop 2.10.0 binary, untar, and cd in to the hadoop directory.
+Here are my steps. First, checkout the hadoop 3.4.1 binary, untar, and cd in to the hadoop directory.
```
-wget http://apache.mirrors.hoobly.com/hadoop/common/hadoop-2.10.0/hadoop-2.10.0.tar.gz
-tar xvf hadoop-2.10.0.tar.gz
-cd hadoop-2.10.0
+wget https://downloads.apache.org/hadoop/common/hadoop-3.4.1/hadoop-3.4.1.tar.gz
+tar xvf hadoop-3.4.1.tar.gz
+cd hadoop-3.4.1
```
Modify the file `./etc/hadoop/core-site.xml`
@@ -19,6 +19,10 @@ Modify the file `./etc/hadoop/core-site.xml`
fs.defaultFS
seaweedfs://localhost:8888
+
+ fs.AbstractFileSystem.seaweedfs.impl
+ seaweed.hdfs.SeaweedAbstractFileSystem
+
```
@@ -27,6 +31,7 @@ Then get the seaweedfs hadoop client jar.
```
cd share/hadoop/common/lib/
wget https://repo1.maven.org/maven2/com/seaweedfs/seaweedfs-hadoop3-client/4.38/seaweedfs-hadoop3-client-4.38.jar
+cd ../../../..
```
# TestDFSIO Benchmark
@@ -39,18 +44,17 @@ The TestDFSIO benchmark is used for measuring I/O (read/write) performance.
Start the TestDFSIO write tests:
```
-bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.10.0-tests.jar TestDFSIO -write -nrFiles 8 -size 32GB -bufferSize 8388608 -resFile /tmp/TestDFSIOwrite.txt
+bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.4.1-tests.jar TestDFSIO -write -nrFiles 4 -size 64MB -bufferSize 8388608 -resFile /tmp/TestDFSIOwrite.txt
...
-20/07/25 16:48:21 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read
-20/07/25 16:48:21 INFO fs.TestDFSIO: Date & time: Sat Jul 25 16:48:21 PDT 2020
-20/07/25 16:48:21 INFO fs.TestDFSIO: Number of files: 8
-20/07/25 16:48:21 INFO fs.TestDFSIO: Total MBytes processed: 262144
-20/07/25 16:48:21 INFO fs.TestDFSIO: Throughput mb/sec: 399.16
-20/07/25 16:48:21 INFO fs.TestDFSIO: Average IO rate mb/sec: 399.34
-20/07/25 16:48:21 INFO fs.TestDFSIO: IO rate std deviation: 8.56
-20/07/25 16:48:21 INFO fs.TestDFSIO: Test exec time sec: 659.45
-20/07/25 16:48:21 INFO fs.TestDFSIO:
+INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
+INFO fs.TestDFSIO: Date & time: Wed Jul 08 00:45:01 PDT 2026
+INFO fs.TestDFSIO: Number of files: 4
+INFO fs.TestDFSIO: Total MBytes processed: 256
+INFO fs.TestDFSIO: Throughput mb/sec: 605.2
+INFO fs.TestDFSIO: Average IO rate mb/sec: 633.27
+INFO fs.TestDFSIO: IO rate std deviation: 130.3
+INFO fs.TestDFSIO: Test exec time sec: 1.32
```
## TestDFSIO read tests
@@ -58,19 +62,24 @@ bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.10.0
Start the TestDFSIO read tests:
```
-bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.10.0-tests.jar TestDFSIO -read -nrFiles 8 -size 32GB -bufferSize 8388608 -resFile /tmp/TestDFSIOwrite.txt
+bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.4.1-tests.jar TestDFSIO -read -nrFiles 4 -size 64MB -bufferSize 8388608 -resFile /tmp/TestDFSIOread.txt
...
-20/07/17 15:59:38 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read
-20/07/17 15:59:38 INFO fs.TestDFSIO: Date & time: Fri Jul 17 15:59:38 PDT 2020
-20/07/17 15:59:38 INFO fs.TestDFSIO: Number of files: 8
-20/07/17 15:59:38 INFO fs.TestDFSIO: Total MBytes processed: 8192
-20/07/17 15:59:38 INFO fs.TestDFSIO: Throughput mb/sec: 393.26
-20/07/17 15:59:38 INFO fs.TestDFSIO: Average IO rate mb/sec: 393.72
-20/07/17 15:59:38 INFO fs.TestDFSIO: IO rate std deviation: 13.33
-20/07/17 15:59:38 INFO fs.TestDFSIO: Test exec time sec: 22.76
-20/07/17 15:59:38 INFO fs.TestDFSIO:
+INFO fs.TestDFSIO: ----- TestDFSIO ----- : read
+INFO fs.TestDFSIO: Date & time: Wed Jul 08 00:45:10 PDT 2026
+INFO fs.TestDFSIO: Number of files: 4
+INFO fs.TestDFSIO: Total MBytes processed: 256
+INFO fs.TestDFSIO: Throughput mb/sec: 1174.31
+INFO fs.TestDFSIO: Average IO rate mb/sec: 1226.51
+INFO fs.TestDFSIO: IO rate std deviation: 227.6
+INFO fs.TestDFSIO: Test exec time sec: 1.28
+```
+
+Clean up the benchmark data when done:
+
+```
+bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.4.1-tests.jar TestDFSIO -clean
```
# Independent Benchmarks
diff --git a/Hadoop-Compatible-File-System.md b/Hadoop-Compatible-File-System.md
index ffacb41..5ed5867 100644
--- a/Hadoop-Compatible-File-System.md
+++ b/Hadoop-Compatible-File-System.md
@@ -29,17 +29,17 @@ Or you can download the latest version from MavenCentral
# Test SeaweedFS on Hadoop
-Suppose you are getting a new Hadoop installation. Here are the minimum steps to get SeaweedFS to run.
+Suppose you are getting a new Hadoop installation. Here are the minimum steps to get SeaweedFS to run. This was verified with Hadoop 3.4.1 on JDK 17.
-You would need to start a weed filer first, build the seaweedfs-hadoop3-client-4.38.jar, and do the following:
+You would need to start a weed filer first, build or download the seaweedfs-hadoop3-client-4.38.jar, and do the following:
```
# optionally adjust hadoop memory allocation
$ export HADOOP_CLIENT_OPTS="-Xmx4g"
$ cd ${HADOOP_HOME}
-# create etc/hadoop/mapred-site.xml, just to satisfy hdfs dfs. skip this if the file already exists.
-$ echo "" > etc/hadoop/mapred-site.xml
+# Hadoop 3.x already ships etc/hadoop/mapred-site.xml. Create it only if it is missing.
+$ [ -f etc/hadoop/mapred-site.xml ] || echo "" > etc/hadoop/mapred-site.xml
# on hadoop3
$ bin/hdfs dfs -Dfs.defaultFS=seaweedfs://localhost:8888 \
@@ -50,6 +50,22 @@ $ bin/hdfs dfs -Dfs.defaultFS=seaweedfs://localhost:8888 \
```
Both reads and writes are working fine.
+A MapReduce job works the same way, passing the client jar with `-libjars`. This teragen writes to SeaweedFS, and terasort reads it back and writes the sorted output, all on `seaweedfs://`:
+
+```
+$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.4.1.jar teragen \
+ -Dfs.defaultFS=seaweedfs://localhost:8888 \
+ -Dfs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem \
+ -libjars ./seaweedfs-hadoop3-client-4.38.jar \
+ 100000 seaweedfs://localhost:8888/teragen-out
+
+$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.4.1.jar terasort \
+ -Dfs.defaultFS=seaweedfs://localhost:8888 \
+ -Dfs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem \
+ -libjars ./seaweedfs-hadoop3-client-4.38.jar \
+ seaweedfs://localhost:8888/teragen-out seaweedfs://localhost:8888/terasort-out
+```
+
# Installation for Hadoop
* Configure Hadoop to use SeaweedFS in `etc/hadoop/conf/core-site.xml`. `core-site.xml` resides on each node in the Hadoop cluster. You must add the same properties to each instance of `core-site.xml`. There are several properties to modify:
1. `fs.seaweedfs.impl`: This property defines the Seaweed HCFS implementation classes that are contained in the SeaweedFS HDFS client JAR. It is required.
diff --git a/Run-Presto-on-SeaweedFS.md b/Run-Presto-on-SeaweedFS.md
index f3a1bbd..2890afb 100644
--- a/Run-Presto-on-SeaweedFS.md
+++ b/Run-Presto-on-SeaweedFS.md
@@ -1,152 +1,182 @@
-# Installation for Presto
-The installation steps are divided into 2 steps:
-## install Hive Metastore
-### Follow instructions for installation of Hive Metastore
-* https://cwiki.apache.org/confluence/display/Hive/AdminManual+Metastore+Administration
+# Run Trino (Presto) on SeaweedFS
-### Configure Hive Metastore to support SeaweedFS
-1. Copy the seaweedfs-hadoop3-client-4.38.jar to hive lib directory,for example:
+Trino (the maintained successor to PrestoDB / PrestoSQL) queries data stored on
+SeaweedFS through its Hive connector. These steps replace the older PrestoDB 347
+instructions. The mechanism changed in current Trino: the `hive-hadoop2`
+connector was renamed to `hive`, and the Hadoop FileSystem path is now opt-in
+through `fs.hadoop.enabled`.
+
+Two storage paths work, both verified with Trino 475, Eclipse Temurin JDK 23,
+and `seaweedfs-hadoop3-client-4.38.jar`:
+
+- **A. HCFS** - the Hive connector talks to the filer directly through the
+ `seaweedfs-hadoop3-client` jar (`seaweedfs://` URIs).
+- **B. S3 gateway** - the Hive connector talks to SeaweedFS's S3 gateway with
+ Trino's built-in S3 filesystem (`s3://` URIs). Simpler: no extra jar, no
+ `core-site.xml`.
+
+JDK note: Trino 475 is the last release that runs on JDK 23. Trino 476+ needs
+JDK 24 and 479+ needs JDK 25. Use the JDK your Trino version requires; the config
+below is otherwise the same.
+
+## Metastore
+
+The Hive connector needs a metastore. This guide uses Trino's built-in file
+metastore (`hive.metastore=file`), which stores table metadata as files beside
+the data, so there is no separate service to run. For production, run a
+standalone Hive Metastore and set `hive.metastore=thrift` instead.
+
+## Trino base config
+
+`etc/node.properties`:
```
-cp seaweedfs-hadoop3-client-4.38.jar /opt/hadoop/share/hadoop/common/lib/
-cp seaweedfs-hadoop3-client-4.38.jar /opt/hive-metastore/lib/
+node.environment=test
+node.id=trino-1
+node.data-dir=/opt/trino/data
```
-2. Modify core-site.xml
-modify core-site.xml to support SeaweedFS, 30888 is the filer port
+
+`etc/config.properties` (single node coordinator, 8080 is the Trino port):
+```
+coordinator=true
+node-scheduler.include-coordinator=true
+http-server.http.port=8080
+discovery.uri=http://localhost:8080
+```
+
+`etc/jvm.config` (the last two flags are required on JDK 23/24):
+```
+-server
+-Xmx3G
+-XX:InitialRAMPercentage=60
+-XX:MaxRAMPercentage=60
+-XX:+ExitOnOutOfMemoryError
+-XX:+HeapDumpOnOutOfMemoryError
+-XX:-OmitStackTraceInFastThrow
+-Djdk.attach.allowAttachSelf=true
+-Dfile.encoding=UTF-8
+--sun-misc-unsafe-memory-access=allow
+--enable-native-access=ALL-UNNAMED
+```
+
+## A. HCFS with seaweedfs-hadoop3-client
+
+1. Put the client jar on the Hive connector classpath:
+```
+cp seaweedfs-hadoop3-client-4.38.jar /opt/trino/plugin/hive/hdfs/
+```
+
+2. `etc/catalog/core-site.xml`:
```
-
- fs.defaultFS
- seaweedfs://10.0.100.51:30888
-
-
- fs.seaweedfs.impl
- seaweed.hdfs.SeaweedFileSystem
-
-
- fs.AbstractFileSystem.seaweedfs.impl
- seaweed.hdfs.SeaweedAbstractFileSystem
-
-
- fs.seaweed.buffer.size
- 4194304
-
-
-```
-3. Modify hive-site.xml
-modify hive-site.xml to support SeaweedFS, need to manually create the /presto/warehouse directory in Filer
-metastore.thrift.port is the access port exposed by the Hive Metadata service itself
-```
-
- metastore.warehouse.dir
- seaweedfs://10.0.100.51:30888/presto/warehouse
-
-
- metastore.thrift.port
- 9850
-
-```
-
-## install Presto
-Follow instructions for installation of Presto:
-* https://prestosql.io/docs/current/installation/deployment.html
-### Configure Presto to support SeaweedFS
-1. Copy the seaweedfs-hadoop3-client-4.38.jar to Presto directory,for example:
-```
-cp seaweedfs-hadoop3-client-4.38.jar /opt/presto-server-347/plugin/hive-hadoop3/
-```
-2. Modify core-site.xml
-
-modify /opt/presto-server-347/etc/catalog/core-site.xml to support SeaweedFS, 30888 is the filer port
-```
-
-
- fs.defaultFS
- seaweedfs://10.0.100.51:30888
-
fs.seaweedfs.impl
seaweed.hdfs.SeaweedFileSystem
- fs.AbstractFileSystem.seaweedfs.impl
- seaweed.hdfs.SeaweedAbstractFileSystem
+ fs.AbstractFileSystem.seaweedfs.impl
+ seaweed.hdfs.SeaweedAbstractFileSystem
- fs.seaweed.buffer.size
- 4194304
+ fs.seaweed.buffer.size
+ 4194304
-
-```
-3. Modify hive.properties
-hive.metastore.uri is the service address of the previously deployed Hive Metastore
-hive.config.resources points to the core-site.xml above
-```
-connector.name=hive-hadoop2
-hive.metastore.uri=thrift://10.0.100.51:9850
-hive.allow-drop-table=true
-hive.max-partitions-per-scan=1000000
-hive.compression-codec=NONE
-hive.config.resources=/opt/presto-server-347/etc/catalog/core-site.xml
-```
-4. Modify config.properties
-The default port of presto is 8080
-If you want to modify the default port of the Presto service, you can modify /opt/presto-server-347/etc/config.properties
-Need to modify the ports of http-server.http.port and discovery.uri
-
-```
-coordinator=true
-node-scheduler.include-coordinator=true
-http-server.http.port=8080
-query.max-memory=200GB
-query.max-memory-per-node=8GB
-query.max-total-memory-per-node=10GB
-query.max-stage-count=200
-task.writer-count=4
-discovery-server.enabled=true
-discovery.uri=http://10.0.100.51:8080
```
-# Using Examples
-1. Connect to Presto and create a table boshen
---server is the ip and port of the Presto service
+3. `etc/catalog/seaweedfs.properties` (8888 is the filer port):
```
-[root@cluster9 ~]# ./presto --server 10.0.100.51:8080--catalog hive --schema default
-presto:default> create table boshen(name varchar);
-CREATE TABLE
-presto:default>
+connector.name=hive
+hive.metastore=file
+hive.metastore.catalog.dir=seaweedfs://localhost:8888/presto/warehouse
+fs.hadoop.enabled=true
+hive.config.resources=/opt/trino/etc/catalog/core-site.xml
```
-2. Query whether the boshen directory has been generated in 10.0.10.51:30888/presto/warehouse
+
+`fs.hadoop.enabled=true` is required: current Trino uses native object-storage
+filesystems by default and only loads a Hadoop FileSystem implementation
+(SeaweedFS's) when this is set. The file metastore creates
+`seaweedfs://localhost:8888/presto/warehouse` on demand, so no directory needs
+to be precreated.
+
+## B. S3 gateway with native S3
+
+1. Run SeaweedFS with the S3 gateway and one identity with credentials:
+```
+weed server -dir=/data -filer -s3 -s3.config=s3config.json
+```
+`s3config.json`:
```
-[root@cluster9 ~]# curl -H "Accept: application/json" http://10.0.100.51:30888/presto/warehouse/?pretty=y
{
- "Path": "presto/warehouse",
- "Entries": [
+ "identities": [
{
- "FullPath": "/presto/warehouse/boshen",
- "Mtime": "2020-12-02T20:29:08+08:00",
- "Crtime": "2020-12-02T20:29:08+08:00",
- "Mode": 2147484159,
- "Uid": 0,
- "Gid": 0,
- "Mime": "",
- "Replication": "",
- "Collection": "",
- "TtlSec": 0,
- "UserName": "root",
- "GroupNames": [
- "root"
+ "name": "trino",
+ "credentials": [
+ { "accessKey": "trinokey", "secretKey": "trinosecret" }
],
- "SymlinkTarget": "",
- "Md5": null,
- "FileSize": 0,
- "Extended": null,
- "HardLinkId": null,
- "HardLinkCounter": 0
+ "actions": ["Admin", "Read", "Write", "List", "Tagging"]
}
- ],
- "Limit": 100,
- "LastFileName": "boshen",
- "ShouldDisplayLoadMore": false
+ ]
}
```
+Create a bucket for the warehouse (8333 is the S3 gateway port):
+```
+AWS_ACCESS_KEY_ID=trinokey AWS_SECRET_ACCESS_KEY=trinosecret \
+ aws --endpoint-url http://localhost:8333 s3 mb s3://warehouse
+```
+
+2. `etc/catalog/seaweedfs.properties`:
+```
+connector.name=hive
+hive.metastore=file
+hive.metastore.catalog.dir=s3://warehouse/
+fs.native-s3.enabled=true
+s3.endpoint=http://localhost:8333
+s3.region=us-east-1
+s3.path-style-access=true
+s3.aws-access-key=trinokey
+s3.aws-secret-key=trinosecret
+```
+No jar and no `core-site.xml` are needed for this path.
+
+## Run a query
+
+Start Trino and connect with the Trino CLI:
+```
+bin/launcher start
+trino --server http://localhost:8080 --catalog seaweedfs
+```
+```
+trino> CREATE SCHEMA seaweedfs.demo;
+CREATE SCHEMA
+trino> CREATE TABLE seaweedfs.demo.boshen (name varchar);
+CREATE TABLE
+trino> INSERT INTO seaweedfs.demo.boshen VALUES ('alice'), ('bob'), ('carol');
+INSERT: 3 rows
+trino> SELECT * FROM seaweedfs.demo.boshen ORDER BY name;
+ name
+-------
+ alice
+ bob
+ carol
+(3 rows)
+```
+
+## Confirm the data is on SeaweedFS
+
+The table is stored as ORC under the metastore catalog directory.
+
+HCFS - list the table directory on the filer:
+```
+curl -H "Accept: application/json" \
+ http://localhost:8888/presto/warehouse/demo/boshen/
+```
+
+S3 - list the bucket:
+```
+AWS_ACCESS_KEY_ID=trinokey AWS_SECRET_ACCESS_KEY=trinosecret \
+ aws --endpoint-url http://localhost:8333 s3 ls s3://warehouse/demo/boshen/ --recursive
+```
+
+Either way you will see the ORC data file, named like
+`20260708_080447_00007_4q5f3_`, next to the `.trinoSchema` metadata the
+file metastore wrote. The data file begins and ends with the `ORC` magic bytes.
diff --git a/run-HBase-on-SeaweedFS.md b/run-HBase-on-SeaweedFS.md
index d894f4a..d518aa5 100644
--- a/run-HBase-on-SeaweedFS.md
+++ b/run-HBase-on-SeaweedFS.md
@@ -1,30 +1,75 @@
# Installation for HBase
-Two steps to run HBase on SeaweedFS
-1. Copy the seaweedfs-hadoop3-client-4.38.jar to `${HBASE_HOME}/lib`
-1. And add the following 2 properties in `${HBASE_HOME}/conf/hbase-site.xml`
+Verified with Apache HBase 2.6.2 (standalone) on JDK 17, using `seaweedfs-hadoop3-client-4.38.jar`.
+HBase talks to the SeaweedFS filer through the HCFS (`seaweedfs://`) scheme, so no S3 gateway is
+needed. HBase 2.6.x ships Hadoop 2.10 jars; the hadoop3 client works against them since it only
+adds the `seaweed.hdfs.*` classes and shades its own dependencies.
+
+Three steps to run HBase on SeaweedFS:
+
+1. Run HBase on JDK 8, 11, or 17. The `bin/hbase` launcher already adds the JDK 17 `--add-opens`
+ flags, so setting `JAVA_HOME` is enough:
+
+ ```
+ export JAVA_HOME=/path/to/jdk-17
+ ```
+
+2. Copy `seaweedfs-hadoop3-client-4.38.jar` to `${HBASE_HOME}/lib`.
+
+3. Set the following properties in `${HBASE_HOME}/conf/hbase-site.xml`. Point `hbase.rootdir` at
+ your filer (default filer port is `8888`) and give `hbase.tmp.dir` a local path:
+
+ ```
+
+
+ hbase.cluster.distributed
+ false
+
+
+ hbase.rootdir
+ seaweedfs://localhost:8888/hbase
+
+
+ hbase.tmp.dir
+ /var/lib/hbase/tmp
+
+
+ fs.seaweedfs.impl
+ seaweed.hdfs.SeaweedFileSystem
+
+
+ fs.AbstractFileSystem.seaweedfs.impl
+ seaweed.hdfs.SeaweedAbstractFileSystem
+
+
+ hbase.unsafe.stream.capability.enforce
+ false
+
+
+ ```
+
+ The SeaweedFS HCFS stream does not advertise `hflush`/`hsync`, so
+ `hbase.unsafe.stream.capability.enforce=false` is required or HBase refuses to start.
+
+Start HBase:
```
-
-
- hbase.cluster.distributed
- true
-
-
- hbase.rootdir
- seaweedfs://localhost:8888/hbase
-
-
- fs.seaweedfs.impl
- seaweed.hdfs.SeaweedFileSystem
-
-
- hbase.unsafe.stream.capability.enforce
- false
-
-
+${HBASE_HOME}/bin/start-hbase.sh
```
-Visit HBase Web UI at `http://:16010` to confirm that HBase is running on SeaweedFS
+Visit the HBase Web UI at `http://:16010` to confirm HBase is running on SeaweedFS
+(the "HBase Root Directory" reads `seaweedfs://...`).

+
+Verify from `hbase shell`:
+
+```
+create 'swtest', 'cf'
+put 'swtest', 'row1', 'cf:name', 'seaweed'
+scan 'swtest'
+```
+
+The store files land under the filer, e.g. `weed shell` / `curl` on
+`seaweedfs://localhost:8888/hbase/data/default/swtest/...`. Tables and rows survive a
+`stop-hbase.sh` / `start-hbase.sh` restart.
diff --git a/run-Spark-on-SeaweedFS.md b/run-Spark-on-SeaweedFS.md
index 9c0f115..b735cf8 100644
--- a/run-Spark-on-SeaweedFS.md
+++ b/run-Spark-on-SeaweedFS.md
@@ -3,6 +3,8 @@ Follow instructions on spark doc:
* https://spark.apache.org/docs/latest/configuration.html#inheriting-hadoop-cluster-configuration
* https://spark.apache.org/docs/latest/configuration.html#custom-hadoophive-configuration
+Tested with Spark 4.0.3 (`spark-4.0.3-bin-hadoop3`, Scala 2.13, bundled Hadoop 3.4.1) on JDK 17. Spark 4.0.x requires Java 17+. The same steps work with Spark 3.5.x on JDK 17.
+
## installation inheriting from Hadoop cluster configuration
Inheriting from Hadoop cluster configuration should be the easiest way.
@@ -17,102 +19,96 @@ Add the following to spark/conf/spark-defaults.conf on every node running Spark
```
spark.driver.extraClassPath=/path/to/seaweedfs-hadoop3-client-4.38.jar
spark.executor.extraClassPath=/path/to/seaweedfs-hadoop3-client-4.38.jar
+spark.hadoop.fs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem
+spark.hadoop.fs.AbstractFileSystem.seaweedfs.impl=seaweed.hdfs.SeaweedAbstractFileSystem
```
+`fs.AbstractFileSystem.seaweedfs.impl` is required on Hadoop 3.x.
And modify the configuration at runtime:
```
-./bin/spark-submit \
- --name "My app" \
- --master local[4] \
- --conf spark.eventLog.enabled=false \
- --conf "spark.executor.extraJavaOptions=-XX:+PrintGCDetails -XX:+PrintGCTimeStamps" \
- --conf spark.hadoop.fs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem \
- --conf spark.hadoop.fs.defaultFS=seaweedfs://localhost:8888 \
+./bin/spark-submit \
+ --name "My app" \
+ --master local[4] \
+ --conf spark.eventLog.enabled=false \
+ --conf spark.hadoop.fs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem \
+ --conf spark.hadoop.fs.AbstractFileSystem.seaweedfs.impl=seaweed.hdfs.SeaweedAbstractFileSystem \
+ --conf spark.hadoop.fs.defaultFS=seaweedfs://localhost:8888 \
+ --jars /path/to/seaweedfs-hadoop3-client-4.38.jar \
myApp.jar
```
# Example
- 1. change the spark-defaults.conf
+Put the client jar on the classpath and point Spark at the filer. Start a filer first (`weed filer`), then:
```
-spark.driver.extraClassPath=/Users/chris/go/src/github.com/seaweedfs/seaweedfs/other/java/hdfs3/target/seaweedfs-hadoop3-client-4.38.jar
-spark.executor.extraClassPath=/Users/chris/go/src/github.com/seaweedfs/seaweedfs/other/java/hdfs3/target/seaweedfs-hadoop3-client-4.38.jar
-spark.hadoop.fs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem
-```
-
- 2. create the spark history folder
-```
-$ curl -X POST http://192.168.2.3:8888/spark2-history/
-```
- 3. Run a spark job
-```
-$ bin/spark-submit --name spark-pi \
---class org.apache.spark.examples.SparkPi \
---conf spark.jars.ivy=/tmp/.ivy \
---conf spark.eventLog.enabled=true \
---conf spark.hadoop.fs.defaultFS=seaweedfs://192.168.2.3:8888 \
---conf spark.eventLog.dir=seaweedfs://192.168.2.3:8888/spark2-history/ \
-file:///usr/local/spark/examples/jars/spark-examples_2.12-3.0.0.jar
-
-```
-
-
-# A Full Example
-Here is my local example switching everything to SeaweedFS. In the `/usr/local/spark/conf/spark-defaults.conf` file,
-
- 1. this is my local `/usr/local/spark/conf/spark-defaults.conf`
-```
-spark.eventLog.enabled=true
-spark.sql.hive.convertMetastoreOrc=true
-spark.yarn.queue=default
-spark.master=local
-spark.history.ui.port=18081
-spark.history.fs.cleaner.interval=7d
-spark.sql.statistics.fallBackToHdfs=true
-spark.yarn.historyServer.address=master:18081
-spark.sql.orc.filterPushdown=true
-spark.history.provider=org.apache.spark.deploy.history.FsHistoryProvider
-spark.history.fs.cleaner.maxAge=90d
-spark.sql.orc.impl=native
-spark.history.fs.cleaner.enabled=true
-
-spark.history.fs.logDirectory=seaweedfs://localhost:8888/spark2-history/
-spark.eventLog.dir=seaweedfs://localhost:8888/spark2-history/
-
-spark.driver.extraClassPath=/Users/chris/go/src/github.com/seaweedfs/seaweedfs/other/java/hdfs3/target/seaweedfs-hadoop3-client-4.38.jar
-spark.executor.extraClassPath=/Users/chris/go/src/github.com/seaweedfs/seaweedfs/other/java/hdfs3/target/seaweedfs-hadoop3-client-4.38.jar
-spark.hadoop.fs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem
-spark.hadoop.fs.defaultFS=seaweedfs://localhost:8888
-```
- 2. create the spark history folder
-```
-$ curl -X POST http://192.168.2.4:8888/spark2-history/
-```
- 3. Run a spark shell
-```
-$ bin/spark-shell
-20/10/18 14:11:44 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
-20/10/18 14:12:15 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
-Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
-Setting default log level to "WARN".
-To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
-Spark context Web UI available at http://192.168.2.4:4040
-Spark context available as 'sc' (master = local, app id = local-1603055539864).
-Spark session available as 'spark'.
+$ bin/spark-shell \
+ --master 'local[2]' \
+ --conf spark.driver.extraClassPath=/path/to/seaweedfs-hadoop3-client-4.38.jar \
+ --conf spark.executor.extraClassPath=/path/to/seaweedfs-hadoop3-client-4.38.jar \
+ --conf spark.hadoop.fs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem \
+ --conf spark.hadoop.fs.AbstractFileSystem.seaweedfs.impl=seaweed.hdfs.SeaweedAbstractFileSystem
+...
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
- /___/ .__/\_,_/_/ /_/\_\ version 3.0.0
+ /___/ .__/\_,_/_/ /_/\_\ version 4.0.3
/_/
-Using Scala version 2.12.10 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_202)
-Type in expressions to have them evaluated.
-Type :help for more information.
-
-scala> sc.textFile("/buckets/large/ttt.txt").count
-res0: Long = 9374
+Using Scala version 2.13.16 (OpenJDK 64-Bit Server VM, Java 17.0.19)
+scala> val df = Seq((1,"alice"),(2,"bob"),(3,"carol"),(4,"dave")).toDF("id","name")
+scala> df.write.mode("overwrite").parquet("seaweedfs://localhost:8888/spark-test/people")
+scala> val back = spark.read.parquet("seaweedfs://localhost:8888/spark-test/people")
+scala> back.count()
+res: Long = 4
+scala> back.orderBy("id").show(false)
++---+-----+
+|id |name |
++---+-----+
+|1 |alice|
+|2 |bob |
+|3 |carol|
+|4 |dave |
++---+-----+
+```
+
+The write lands under `/spark-test/people/` on the filer as `_SUCCESS` plus the snappy parquet part files. The path prefix `spark-test` is created on write, no need to pre-create it.
+
+# A Full Example
+Here is a local example switching everything to SeaweedFS. In the `$SPARK_HOME/conf/spark-defaults.conf` file,
+
+ 1. `spark-defaults.conf`
+```
+spark.master=local[*]
+
+spark.hadoop.fs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem
+spark.hadoop.fs.AbstractFileSystem.seaweedfs.impl=seaweed.hdfs.SeaweedAbstractFileSystem
+spark.hadoop.fs.defaultFS=seaweedfs://localhost:8888
+
+spark.driver.extraClassPath=/path/to/seaweedfs-hadoop3-client-4.38.jar
+spark.executor.extraClassPath=/path/to/seaweedfs-hadoop3-client-4.38.jar
+
+spark.eventLog.enabled=true
+spark.eventLog.dir=seaweedfs://localhost:8888/spark-history/
+spark.history.fs.logDirectory=seaweedfs://localhost:8888/spark-history/
+```
+ 2. Run a spark shell. Event logs are written to SeaweedFS under `/spark-history/`.
+```
+$ bin/spark-shell
+...
+Welcome to
+ ____ __
+ / __/__ ___ _____/ /__
+ _\ \/ _ \/ _ `/ __/ '_/
+ /___/ .__/\_,_/_/ /_/\_\ version 4.0.3
+ /_/
+
+Using Scala version 2.13.16 (OpenJDK 64-Bit Server VM, Java 17.0.19)
+Spark session available as 'spark'.
+
+scala> sc.parallelize(1 to 100).sum
+res0: Double = 5050.0
```