From 8bb90fe474bdaec36c174f93671ffebc207375fa Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Wed, 8 Jul 2026 01:09:49 -0700 Subject: [PATCH] verify and update integration examples against current versions (Hadoop 3.4.1, Spark 4.0, HBase 2.6, Trino 475) --- HDFS-via-S3-connector.md | 128 +++++++++++--- Hadoop-Benchmark.md | 57 ++++--- Hadoop-Compatible-File-System.md | 24 ++- Run-Presto-on-SeaweedFS.md | 284 +++++++++++++++++-------------- run-HBase-on-SeaweedFS.md | 89 +++++++--- run-Spark-on-SeaweedFS.md | 154 ++++++++--------- 6 files changed, 455 insertions(+), 281 deletions(-) diff --git a/HDFS-via-S3-connector.md b/HDFS-via-S3-connector.md index 11aec0e..2446169 100644 --- a/HDFS-via-S3-connector.md +++ b/HDFS-via-S3-connector.md @@ -4,43 +4,121 @@ However, the downside is that you need to add a SeaweedFS jar to classpath, and # HDFS Access SeaweedFS via S3 connector -The S3a connector is already included in hadoop distributions. You can use it directly. +The S3A connector (`hadoop-aws`) points at the SeaweedFS S3 gateway. It ships with Hadoop distributions, so no SeaweedFS jar is needed. -Here is an example spark job pom.xml file, using hadoop version later than `3.3.1`: +Tested with Spark 4.0.3 (`spark-4.0.3-bin-hadoop3`, bundled Hadoop 3.4.1) on JDK 17. The `hadoop-aws` jar must match the Hadoop version bundled in Spark. For this build that is `hadoop-aws-3.4.1.jar` plus the AWS SDK v2 bundle `bundle-2.24.6.jar`, both found under `share/hadoop/tools/lib/` of a Hadoop 3.4.1 distribution. + +## Configuration + +Point S3A at the SeaweedFS S3 gateway (default port 8333): + +``` +fs.s3a.endpoint=http://localhost:8333 +fs.s3a.path.style.access=true +fs.s3a.connection.ssl.enabled=false +``` + +Create the bucket before writing: + +``` +$ aws --endpoint-url http://localhost:8333 s3 mb s3://test +``` + +## Credentials + +A SeaweedFS S3 gateway started without an identity config allows anonymous access. Use the anonymous provider: + +``` +fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider +``` + +To require credentials, start the gateway with a static identity config, e.g. `s3.json`: + +``` +{ + "identities": [ + { + "name": "spark", + "credentials": [ + { "accessKey": "sparkkey", "secretKey": "sparksecret" } + ], + "actions": ["Admin", "Read", "Write", "List", "Tagging"] + } + ] +} +``` +``` +$ weed server -s3 -s3.config=s3.json ... +``` +Then use the simple provider with those keys: + +``` +fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider +fs.s3a.access.key=sparkkey +fs.s3a.secret.key=sparksecret +``` + +## Example + +``` +$ bin/spark-shell \ + --master 'local[2]' \ + --jars /path/to/hadoop-aws-3.4.1.jar,/path/to/bundle-2.24.6.jar \ + --conf spark.hadoop.fs.s3a.endpoint=http://localhost:8333 \ + --conf spark.hadoop.fs.s3a.path.style.access=true \ + --conf spark.hadoop.fs.s3a.connection.ssl.enabled=false \ + --conf spark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider +... +scala> val df = Seq((10,"x"),(20,"y"),(30,"z")).toDF("id","label") +scala> df.write.mode("overwrite").parquet("s3a://test/spark-s3a/data") +scala> val back = spark.read.parquet("s3a://test/spark-s3a/data") +scala> back.count() +res: Long = 3 +scala> back.orderBy("id").show(false) ++---+-----+ +|id |label| ++---+-----+ +|10 |x | +|20 |y | +|30 |z | ++---+-----+ +``` + +The write lands in the bucket as `spark-s3a/data/_SUCCESS` plus the snappy parquet part files. + +## Packaged spark job + +Example `pom.xml` properties for a job compiled against Spark 4.0.3: ``` - 8 - 8 + 17 + 17 UTF-8 - 2.12.11 - 3.1.2 - 3.3.1 + 2.13.16 + 4.0.3 + 3.4.1 compile - ``` -And add this in your code: +And set the S3A configuration in your code: ``` SparkSession spark = SparkSession.builder() .master("local[*]") .config("spark.eventLog.enabled", "false") - .config("spark.driver.memory", "1g") - .config("spark.executor.memory", "1g") .appName("SparkDemoFromS3") .getOrCreate(); -spark.sparkContext().hadoopConfiguration().set("fs.s3a.access.key", "admin"); -spark.sparkContext().hadoopConfiguration().set("fs.s3a.secret.key", "xx"); -spark.sparkContext().hadoopConfiguration().set("fs.s3a.endpoint", "ip:8333"); -spark.sparkContext().hadoopConfiguration().set("com.amazonaws.services.s3a.enableV4", "true"); -spark.sparkContext().hadoopConfiguration().set("fs.s3a.path.style.access", "true"); -spark.sparkContext().hadoopConfiguration().set("fs.s3a.connection.ssl.enabled", "false"); -spark.sparkContext().hadoopConfiguration().set("fs.s3a.multiobjectdelete.enable", "false"); -spark.sparkContext().hadoopConfiguration().set("fs.s3a.directory.marker.retention", "keep"); -spark.sparkContext().hadoopConfiguration().set("fs.s3a.change.detection.version.required", "false"); -spark.sparkContext().hadoopConfiguration().set("fs.s3a.change.detection.mode", "warn"); -RDD rdd = spark.sparkContext().textFile("s3a://bk002/test1.txt", 1); -System.out.println(rdd.count()); -rdd.saveAsTextFile("s3a://bk002/testcc/t2"); - +Configuration conf = spark.sparkContext().hadoopConfiguration(); +conf.set("fs.s3a.endpoint", "http://localhost:8333"); +conf.set("fs.s3a.path.style.access", "true"); +conf.set("fs.s3a.connection.ssl.enabled", "false"); +// anonymous access +conf.set("fs.s3a.aws.credentials.provider", "org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider"); +// or, with a static identity: +// conf.set("fs.s3a.aws.credentials.provider", "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider"); +// conf.set("fs.s3a.access.key", "sparkkey"); +// conf.set("fs.s3a.secret.key", "sparksecret"); +Dataset df = spark.read().parquet("s3a://test/spark-s3a/data"); +System.out.println(df.count()); +df.write().mode("overwrite").parquet("s3a://test/testcc/t2"); ``` diff --git a/Hadoop-Benchmark.md b/Hadoop-Benchmark.md index 5a37cc9..9c730cc 100644 --- a/Hadoop-Benchmark.md +++ b/Hadoop-Benchmark.md @@ -1,10 +1,10 @@ # Setup Hadoop Benchmark -Here are my steps. First, checkout hadoop 2.10.0 binary, untar, and cd in to the hadoop directory. +Here are my steps. First, checkout the hadoop 3.4.1 binary, untar, and cd in to the hadoop directory. ``` -wget http://apache.mirrors.hoobly.com/hadoop/common/hadoop-2.10.0/hadoop-2.10.0.tar.gz -tar xvf hadoop-2.10.0.tar.gz -cd hadoop-2.10.0 +wget https://downloads.apache.org/hadoop/common/hadoop-3.4.1/hadoop-3.4.1.tar.gz +tar xvf hadoop-3.4.1.tar.gz +cd hadoop-3.4.1 ``` Modify the file `./etc/hadoop/core-site.xml` @@ -19,6 +19,10 @@ Modify the file `./etc/hadoop/core-site.xml` fs.defaultFS seaweedfs://localhost:8888 + + fs.AbstractFileSystem.seaweedfs.impl + seaweed.hdfs.SeaweedAbstractFileSystem + ``` @@ -27,6 +31,7 @@ Then get the seaweedfs hadoop client jar. ``` cd share/hadoop/common/lib/ wget https://repo1.maven.org/maven2/com/seaweedfs/seaweedfs-hadoop3-client/4.38/seaweedfs-hadoop3-client-4.38.jar +cd ../../../.. ``` # TestDFSIO Benchmark @@ -39,18 +44,17 @@ The TestDFSIO benchmark is used for measuring I/O (read/write) performance. Start the TestDFSIO write tests: ``` -bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.10.0-tests.jar TestDFSIO -write -nrFiles 8 -size 32GB -bufferSize 8388608 -resFile /tmp/TestDFSIOwrite.txt +bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.4.1-tests.jar TestDFSIO -write -nrFiles 4 -size 64MB -bufferSize 8388608 -resFile /tmp/TestDFSIOwrite.txt ... -20/07/25 16:48:21 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read -20/07/25 16:48:21 INFO fs.TestDFSIO: Date & time: Sat Jul 25 16:48:21 PDT 2020 -20/07/25 16:48:21 INFO fs.TestDFSIO: Number of files: 8 -20/07/25 16:48:21 INFO fs.TestDFSIO: Total MBytes processed: 262144 -20/07/25 16:48:21 INFO fs.TestDFSIO: Throughput mb/sec: 399.16 -20/07/25 16:48:21 INFO fs.TestDFSIO: Average IO rate mb/sec: 399.34 -20/07/25 16:48:21 INFO fs.TestDFSIO: IO rate std deviation: 8.56 -20/07/25 16:48:21 INFO fs.TestDFSIO: Test exec time sec: 659.45 -20/07/25 16:48:21 INFO fs.TestDFSIO: +INFO fs.TestDFSIO: ----- TestDFSIO ----- : write +INFO fs.TestDFSIO: Date & time: Wed Jul 08 00:45:01 PDT 2026 +INFO fs.TestDFSIO: Number of files: 4 +INFO fs.TestDFSIO: Total MBytes processed: 256 +INFO fs.TestDFSIO: Throughput mb/sec: 605.2 +INFO fs.TestDFSIO: Average IO rate mb/sec: 633.27 +INFO fs.TestDFSIO: IO rate std deviation: 130.3 +INFO fs.TestDFSIO: Test exec time sec: 1.32 ``` ## TestDFSIO read tests @@ -58,19 +62,24 @@ bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.10.0 Start the TestDFSIO read tests: ``` -bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.10.0-tests.jar TestDFSIO -read -nrFiles 8 -size 32GB -bufferSize 8388608 -resFile /tmp/TestDFSIOwrite.txt +bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.4.1-tests.jar TestDFSIO -read -nrFiles 4 -size 64MB -bufferSize 8388608 -resFile /tmp/TestDFSIOread.txt ... -20/07/17 15:59:38 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read -20/07/17 15:59:38 INFO fs.TestDFSIO: Date & time: Fri Jul 17 15:59:38 PDT 2020 -20/07/17 15:59:38 INFO fs.TestDFSIO: Number of files: 8 -20/07/17 15:59:38 INFO fs.TestDFSIO: Total MBytes processed: 8192 -20/07/17 15:59:38 INFO fs.TestDFSIO: Throughput mb/sec: 393.26 -20/07/17 15:59:38 INFO fs.TestDFSIO: Average IO rate mb/sec: 393.72 -20/07/17 15:59:38 INFO fs.TestDFSIO: IO rate std deviation: 13.33 -20/07/17 15:59:38 INFO fs.TestDFSIO: Test exec time sec: 22.76 -20/07/17 15:59:38 INFO fs.TestDFSIO: +INFO fs.TestDFSIO: ----- TestDFSIO ----- : read +INFO fs.TestDFSIO: Date & time: Wed Jul 08 00:45:10 PDT 2026 +INFO fs.TestDFSIO: Number of files: 4 +INFO fs.TestDFSIO: Total MBytes processed: 256 +INFO fs.TestDFSIO: Throughput mb/sec: 1174.31 +INFO fs.TestDFSIO: Average IO rate mb/sec: 1226.51 +INFO fs.TestDFSIO: IO rate std deviation: 227.6 +INFO fs.TestDFSIO: Test exec time sec: 1.28 +``` + +Clean up the benchmark data when done: + +``` +bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.4.1-tests.jar TestDFSIO -clean ``` # Independent Benchmarks diff --git a/Hadoop-Compatible-File-System.md b/Hadoop-Compatible-File-System.md index ffacb41..5ed5867 100644 --- a/Hadoop-Compatible-File-System.md +++ b/Hadoop-Compatible-File-System.md @@ -29,17 +29,17 @@ Or you can download the latest version from MavenCentral # Test SeaweedFS on Hadoop -Suppose you are getting a new Hadoop installation. Here are the minimum steps to get SeaweedFS to run. +Suppose you are getting a new Hadoop installation. Here are the minimum steps to get SeaweedFS to run. This was verified with Hadoop 3.4.1 on JDK 17. -You would need to start a weed filer first, build the seaweedfs-hadoop3-client-4.38.jar, and do the following: +You would need to start a weed filer first, build or download the seaweedfs-hadoop3-client-4.38.jar, and do the following: ``` # optionally adjust hadoop memory allocation $ export HADOOP_CLIENT_OPTS="-Xmx4g" $ cd ${HADOOP_HOME} -# create etc/hadoop/mapred-site.xml, just to satisfy hdfs dfs. skip this if the file already exists. -$ echo "" > etc/hadoop/mapred-site.xml +# Hadoop 3.x already ships etc/hadoop/mapred-site.xml. Create it only if it is missing. +$ [ -f etc/hadoop/mapred-site.xml ] || echo "" > etc/hadoop/mapred-site.xml # on hadoop3 $ bin/hdfs dfs -Dfs.defaultFS=seaweedfs://localhost:8888 \ @@ -50,6 +50,22 @@ $ bin/hdfs dfs -Dfs.defaultFS=seaweedfs://localhost:8888 \ ``` Both reads and writes are working fine. +A MapReduce job works the same way, passing the client jar with `-libjars`. This teragen writes to SeaweedFS, and terasort reads it back and writes the sorted output, all on `seaweedfs://`: + +``` +$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.4.1.jar teragen \ + -Dfs.defaultFS=seaweedfs://localhost:8888 \ + -Dfs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem \ + -libjars ./seaweedfs-hadoop3-client-4.38.jar \ + 100000 seaweedfs://localhost:8888/teragen-out + +$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.4.1.jar terasort \ + -Dfs.defaultFS=seaweedfs://localhost:8888 \ + -Dfs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem \ + -libjars ./seaweedfs-hadoop3-client-4.38.jar \ + seaweedfs://localhost:8888/teragen-out seaweedfs://localhost:8888/terasort-out +``` + # Installation for Hadoop * Configure Hadoop to use SeaweedFS in `etc/hadoop/conf/core-site.xml`. `core-site.xml` resides on each node in the Hadoop cluster. You must add the same properties to each instance of `core-site.xml`. There are several properties to modify: 1. `fs.seaweedfs.impl`: This property defines the Seaweed HCFS implementation classes that are contained in the SeaweedFS HDFS client JAR. It is required. diff --git a/Run-Presto-on-SeaweedFS.md b/Run-Presto-on-SeaweedFS.md index f3a1bbd..2890afb 100644 --- a/Run-Presto-on-SeaweedFS.md +++ b/Run-Presto-on-SeaweedFS.md @@ -1,152 +1,182 @@ -# Installation for Presto -The installation steps are divided into 2 steps: -## install Hive Metastore -### Follow instructions for installation of Hive Metastore -* https://cwiki.apache.org/confluence/display/Hive/AdminManual+Metastore+Administration +# Run Trino (Presto) on SeaweedFS -### Configure Hive Metastore to support SeaweedFS -1. Copy the seaweedfs-hadoop3-client-4.38.jar to hive lib directory,for example: +Trino (the maintained successor to PrestoDB / PrestoSQL) queries data stored on +SeaweedFS through its Hive connector. These steps replace the older PrestoDB 347 +instructions. The mechanism changed in current Trino: the `hive-hadoop2` +connector was renamed to `hive`, and the Hadoop FileSystem path is now opt-in +through `fs.hadoop.enabled`. + +Two storage paths work, both verified with Trino 475, Eclipse Temurin JDK 23, +and `seaweedfs-hadoop3-client-4.38.jar`: + +- **A. HCFS** - the Hive connector talks to the filer directly through the + `seaweedfs-hadoop3-client` jar (`seaweedfs://` URIs). +- **B. S3 gateway** - the Hive connector talks to SeaweedFS's S3 gateway with + Trino's built-in S3 filesystem (`s3://` URIs). Simpler: no extra jar, no + `core-site.xml`. + +JDK note: Trino 475 is the last release that runs on JDK 23. Trino 476+ needs +JDK 24 and 479+ needs JDK 25. Use the JDK your Trino version requires; the config +below is otherwise the same. + +## Metastore + +The Hive connector needs a metastore. This guide uses Trino's built-in file +metastore (`hive.metastore=file`), which stores table metadata as files beside +the data, so there is no separate service to run. For production, run a +standalone Hive Metastore and set `hive.metastore=thrift` instead. + +## Trino base config + +`etc/node.properties`: ``` -cp seaweedfs-hadoop3-client-4.38.jar /opt/hadoop/share/hadoop/common/lib/ -cp seaweedfs-hadoop3-client-4.38.jar /opt/hive-metastore/lib/ +node.environment=test +node.id=trino-1 +node.data-dir=/opt/trino/data ``` -2. Modify core-site.xml -modify core-site.xml to support SeaweedFS, 30888 is the filer port + +`etc/config.properties` (single node coordinator, 8080 is the Trino port): +``` +coordinator=true +node-scheduler.include-coordinator=true +http-server.http.port=8080 +discovery.uri=http://localhost:8080 +``` + +`etc/jvm.config` (the last two flags are required on JDK 23/24): +``` +-server +-Xmx3G +-XX:InitialRAMPercentage=60 +-XX:MaxRAMPercentage=60 +-XX:+ExitOnOutOfMemoryError +-XX:+HeapDumpOnOutOfMemoryError +-XX:-OmitStackTraceInFastThrow +-Djdk.attach.allowAttachSelf=true +-Dfile.encoding=UTF-8 +--sun-misc-unsafe-memory-access=allow +--enable-native-access=ALL-UNNAMED +``` + +## A. HCFS with seaweedfs-hadoop3-client + +1. Put the client jar on the Hive connector classpath: +``` +cp seaweedfs-hadoop3-client-4.38.jar /opt/trino/plugin/hive/hdfs/ +``` + +2. `etc/catalog/core-site.xml`: ``` - - fs.defaultFS - seaweedfs://10.0.100.51:30888 - - - fs.seaweedfs.impl - seaweed.hdfs.SeaweedFileSystem - - - fs.AbstractFileSystem.seaweedfs.impl - seaweed.hdfs.SeaweedAbstractFileSystem - - - fs.seaweed.buffer.size - 4194304 - - -``` -3. Modify hive-site.xml -modify hive-site.xml to support SeaweedFS, need to manually create the /presto/warehouse directory in Filer -metastore.thrift.port is the access port exposed by the Hive Metadata service itself -``` - - metastore.warehouse.dir - seaweedfs://10.0.100.51:30888/presto/warehouse - - - metastore.thrift.port - 9850 - -``` - -## install Presto -Follow instructions for installation of Presto: -* https://prestosql.io/docs/current/installation/deployment.html -### Configure Presto to support SeaweedFS -1. Copy the seaweedfs-hadoop3-client-4.38.jar to Presto directory,for example: -``` -cp seaweedfs-hadoop3-client-4.38.jar /opt/presto-server-347/plugin/hive-hadoop3/ -``` -2. Modify core-site.xml - -modify /opt/presto-server-347/etc/catalog/core-site.xml to support SeaweedFS, 30888 is the filer port -``` - - - fs.defaultFS - seaweedfs://10.0.100.51:30888 - fs.seaweedfs.impl seaweed.hdfs.SeaweedFileSystem - fs.AbstractFileSystem.seaweedfs.impl - seaweed.hdfs.SeaweedAbstractFileSystem + fs.AbstractFileSystem.seaweedfs.impl + seaweed.hdfs.SeaweedAbstractFileSystem - fs.seaweed.buffer.size - 4194304 + fs.seaweed.buffer.size + 4194304 - -``` -3. Modify hive.properties -hive.metastore.uri is the service address of the previously deployed Hive Metastore -hive.config.resources points to the core-site.xml above -``` -connector.name=hive-hadoop2 -hive.metastore.uri=thrift://10.0.100.51:9850 -hive.allow-drop-table=true -hive.max-partitions-per-scan=1000000 -hive.compression-codec=NONE -hive.config.resources=/opt/presto-server-347/etc/catalog/core-site.xml -``` -4. Modify config.properties -The default port of presto is 8080 -If you want to modify the default port of the Presto service, you can modify /opt/presto-server-347/etc/config.properties -Need to modify the ports of http-server.http.port and discovery.uri - -``` -coordinator=true -node-scheduler.include-coordinator=true -http-server.http.port=8080 -query.max-memory=200GB -query.max-memory-per-node=8GB -query.max-total-memory-per-node=10GB -query.max-stage-count=200 -task.writer-count=4 -discovery-server.enabled=true -discovery.uri=http://10.0.100.51:8080 ``` -# Using Examples -1. Connect to Presto and create a table boshen ---server is the ip and port of the Presto service +3. `etc/catalog/seaweedfs.properties` (8888 is the filer port): ``` -[root@cluster9 ~]# ./presto --server 10.0.100.51:8080--catalog hive --schema default -presto:default> create table boshen(name varchar); -CREATE TABLE -presto:default> +connector.name=hive +hive.metastore=file +hive.metastore.catalog.dir=seaweedfs://localhost:8888/presto/warehouse +fs.hadoop.enabled=true +hive.config.resources=/opt/trino/etc/catalog/core-site.xml ``` -2. Query whether the boshen directory has been generated in 10.0.10.51:30888/presto/warehouse + +`fs.hadoop.enabled=true` is required: current Trino uses native object-storage +filesystems by default and only loads a Hadoop FileSystem implementation +(SeaweedFS's) when this is set. The file metastore creates +`seaweedfs://localhost:8888/presto/warehouse` on demand, so no directory needs +to be precreated. + +## B. S3 gateway with native S3 + +1. Run SeaweedFS with the S3 gateway and one identity with credentials: +``` +weed server -dir=/data -filer -s3 -s3.config=s3config.json +``` +`s3config.json`: ``` -[root@cluster9 ~]# curl -H "Accept: application/json" http://10.0.100.51:30888/presto/warehouse/?pretty=y { - "Path": "presto/warehouse", - "Entries": [ + "identities": [ { - "FullPath": "/presto/warehouse/boshen", - "Mtime": "2020-12-02T20:29:08+08:00", - "Crtime": "2020-12-02T20:29:08+08:00", - "Mode": 2147484159, - "Uid": 0, - "Gid": 0, - "Mime": "", - "Replication": "", - "Collection": "", - "TtlSec": 0, - "UserName": "root", - "GroupNames": [ - "root" + "name": "trino", + "credentials": [ + { "accessKey": "trinokey", "secretKey": "trinosecret" } ], - "SymlinkTarget": "", - "Md5": null, - "FileSize": 0, - "Extended": null, - "HardLinkId": null, - "HardLinkCounter": 0 + "actions": ["Admin", "Read", "Write", "List", "Tagging"] } - ], - "Limit": 100, - "LastFileName": "boshen", - "ShouldDisplayLoadMore": false + ] } ``` +Create a bucket for the warehouse (8333 is the S3 gateway port): +``` +AWS_ACCESS_KEY_ID=trinokey AWS_SECRET_ACCESS_KEY=trinosecret \ + aws --endpoint-url http://localhost:8333 s3 mb s3://warehouse +``` + +2. `etc/catalog/seaweedfs.properties`: +``` +connector.name=hive +hive.metastore=file +hive.metastore.catalog.dir=s3://warehouse/ +fs.native-s3.enabled=true +s3.endpoint=http://localhost:8333 +s3.region=us-east-1 +s3.path-style-access=true +s3.aws-access-key=trinokey +s3.aws-secret-key=trinosecret +``` +No jar and no `core-site.xml` are needed for this path. + +## Run a query + +Start Trino and connect with the Trino CLI: +``` +bin/launcher start +trino --server http://localhost:8080 --catalog seaweedfs +``` +``` +trino> CREATE SCHEMA seaweedfs.demo; +CREATE SCHEMA +trino> CREATE TABLE seaweedfs.demo.boshen (name varchar); +CREATE TABLE +trino> INSERT INTO seaweedfs.demo.boshen VALUES ('alice'), ('bob'), ('carol'); +INSERT: 3 rows +trino> SELECT * FROM seaweedfs.demo.boshen ORDER BY name; + name +------- + alice + bob + carol +(3 rows) +``` + +## Confirm the data is on SeaweedFS + +The table is stored as ORC under the metastore catalog directory. + +HCFS - list the table directory on the filer: +``` +curl -H "Accept: application/json" \ + http://localhost:8888/presto/warehouse/demo/boshen/ +``` + +S3 - list the bucket: +``` +AWS_ACCESS_KEY_ID=trinokey AWS_SECRET_ACCESS_KEY=trinosecret \ + aws --endpoint-url http://localhost:8333 s3 ls s3://warehouse/demo/boshen/ --recursive +``` + +Either way you will see the ORC data file, named like +`20260708_080447_00007_4q5f3_`, next to the `.trinoSchema` metadata the +file metastore wrote. The data file begins and ends with the `ORC` magic bytes. diff --git a/run-HBase-on-SeaweedFS.md b/run-HBase-on-SeaweedFS.md index d894f4a..d518aa5 100644 --- a/run-HBase-on-SeaweedFS.md +++ b/run-HBase-on-SeaweedFS.md @@ -1,30 +1,75 @@ # Installation for HBase -Two steps to run HBase on SeaweedFS -1. Copy the seaweedfs-hadoop3-client-4.38.jar to `${HBASE_HOME}/lib` -1. And add the following 2 properties in `${HBASE_HOME}/conf/hbase-site.xml` +Verified with Apache HBase 2.6.2 (standalone) on JDK 17, using `seaweedfs-hadoop3-client-4.38.jar`. +HBase talks to the SeaweedFS filer through the HCFS (`seaweedfs://`) scheme, so no S3 gateway is +needed. HBase 2.6.x ships Hadoop 2.10 jars; the hadoop3 client works against them since it only +adds the `seaweed.hdfs.*` classes and shades its own dependencies. + +Three steps to run HBase on SeaweedFS: + +1. Run HBase on JDK 8, 11, or 17. The `bin/hbase` launcher already adds the JDK 17 `--add-opens` + flags, so setting `JAVA_HOME` is enough: + + ``` + export JAVA_HOME=/path/to/jdk-17 + ``` + +2. Copy `seaweedfs-hadoop3-client-4.38.jar` to `${HBASE_HOME}/lib`. + +3. Set the following properties in `${HBASE_HOME}/conf/hbase-site.xml`. Point `hbase.rootdir` at + your filer (default filer port is `8888`) and give `hbase.tmp.dir` a local path: + + ``` + + + hbase.cluster.distributed + false + + + hbase.rootdir + seaweedfs://localhost:8888/hbase + + + hbase.tmp.dir + /var/lib/hbase/tmp + + + fs.seaweedfs.impl + seaweed.hdfs.SeaweedFileSystem + + + fs.AbstractFileSystem.seaweedfs.impl + seaweed.hdfs.SeaweedAbstractFileSystem + + + hbase.unsafe.stream.capability.enforce + false + + + ``` + + The SeaweedFS HCFS stream does not advertise `hflush`/`hsync`, so + `hbase.unsafe.stream.capability.enforce=false` is required or HBase refuses to start. + +Start HBase: ``` - - - hbase.cluster.distributed - true - - - hbase.rootdir - seaweedfs://localhost:8888/hbase - - - fs.seaweedfs.impl - seaweed.hdfs.SeaweedFileSystem - - - hbase.unsafe.stream.capability.enforce - false - - +${HBASE_HOME}/bin/start-hbase.sh ``` -Visit HBase Web UI at `http://:16010` to confirm that HBase is running on SeaweedFS +Visit the HBase Web UI at `http://:16010` to confirm HBase is running on SeaweedFS +(the "HBase Root Directory" reads `seaweedfs://...`). ![](HBaseOnSeaweedFS.png) + +Verify from `hbase shell`: + +``` +create 'swtest', 'cf' +put 'swtest', 'row1', 'cf:name', 'seaweed' +scan 'swtest' +``` + +The store files land under the filer, e.g. `weed shell` / `curl` on +`seaweedfs://localhost:8888/hbase/data/default/swtest/...`. Tables and rows survive a +`stop-hbase.sh` / `start-hbase.sh` restart. diff --git a/run-Spark-on-SeaweedFS.md b/run-Spark-on-SeaweedFS.md index 9c0f115..b735cf8 100644 --- a/run-Spark-on-SeaweedFS.md +++ b/run-Spark-on-SeaweedFS.md @@ -3,6 +3,8 @@ Follow instructions on spark doc: * https://spark.apache.org/docs/latest/configuration.html#inheriting-hadoop-cluster-configuration * https://spark.apache.org/docs/latest/configuration.html#custom-hadoophive-configuration +Tested with Spark 4.0.3 (`spark-4.0.3-bin-hadoop3`, Scala 2.13, bundled Hadoop 3.4.1) on JDK 17. Spark 4.0.x requires Java 17+. The same steps work with Spark 3.5.x on JDK 17. + ## installation inheriting from Hadoop cluster configuration Inheriting from Hadoop cluster configuration should be the easiest way. @@ -17,102 +19,96 @@ Add the following to spark/conf/spark-defaults.conf on every node running Spark ``` spark.driver.extraClassPath=/path/to/seaweedfs-hadoop3-client-4.38.jar spark.executor.extraClassPath=/path/to/seaweedfs-hadoop3-client-4.38.jar +spark.hadoop.fs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem +spark.hadoop.fs.AbstractFileSystem.seaweedfs.impl=seaweed.hdfs.SeaweedAbstractFileSystem ``` +`fs.AbstractFileSystem.seaweedfs.impl` is required on Hadoop 3.x. And modify the configuration at runtime: ``` -./bin/spark-submit \ - --name "My app" \ - --master local[4] \ - --conf spark.eventLog.enabled=false \ - --conf "spark.executor.extraJavaOptions=-XX:+PrintGCDetails -XX:+PrintGCTimeStamps" \ - --conf spark.hadoop.fs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem \ - --conf spark.hadoop.fs.defaultFS=seaweedfs://localhost:8888 \ +./bin/spark-submit \ + --name "My app" \ + --master local[4] \ + --conf spark.eventLog.enabled=false \ + --conf spark.hadoop.fs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem \ + --conf spark.hadoop.fs.AbstractFileSystem.seaweedfs.impl=seaweed.hdfs.SeaweedAbstractFileSystem \ + --conf spark.hadoop.fs.defaultFS=seaweedfs://localhost:8888 \ + --jars /path/to/seaweedfs-hadoop3-client-4.38.jar \ myApp.jar ``` # Example - 1. change the spark-defaults.conf +Put the client jar on the classpath and point Spark at the filer. Start a filer first (`weed filer`), then: ``` -spark.driver.extraClassPath=/Users/chris/go/src/github.com/seaweedfs/seaweedfs/other/java/hdfs3/target/seaweedfs-hadoop3-client-4.38.jar -spark.executor.extraClassPath=/Users/chris/go/src/github.com/seaweedfs/seaweedfs/other/java/hdfs3/target/seaweedfs-hadoop3-client-4.38.jar -spark.hadoop.fs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem -``` - - 2. create the spark history folder -``` -$ curl -X POST http://192.168.2.3:8888/spark2-history/ -``` - 3. Run a spark job -``` -$ bin/spark-submit --name spark-pi \ ---class org.apache.spark.examples.SparkPi \ ---conf spark.jars.ivy=/tmp/.ivy \ ---conf spark.eventLog.enabled=true \ ---conf spark.hadoop.fs.defaultFS=seaweedfs://192.168.2.3:8888 \ ---conf spark.eventLog.dir=seaweedfs://192.168.2.3:8888/spark2-history/ \ -file:///usr/local/spark/examples/jars/spark-examples_2.12-3.0.0.jar - -``` - - -# A Full Example -Here is my local example switching everything to SeaweedFS. In the `/usr/local/spark/conf/spark-defaults.conf` file, - - 1. this is my local `/usr/local/spark/conf/spark-defaults.conf` -``` -spark.eventLog.enabled=true -spark.sql.hive.convertMetastoreOrc=true -spark.yarn.queue=default -spark.master=local -spark.history.ui.port=18081 -spark.history.fs.cleaner.interval=7d -spark.sql.statistics.fallBackToHdfs=true -spark.yarn.historyServer.address=master:18081 -spark.sql.orc.filterPushdown=true -spark.history.provider=org.apache.spark.deploy.history.FsHistoryProvider -spark.history.fs.cleaner.maxAge=90d -spark.sql.orc.impl=native -spark.history.fs.cleaner.enabled=true - -spark.history.fs.logDirectory=seaweedfs://localhost:8888/spark2-history/ -spark.eventLog.dir=seaweedfs://localhost:8888/spark2-history/ - -spark.driver.extraClassPath=/Users/chris/go/src/github.com/seaweedfs/seaweedfs/other/java/hdfs3/target/seaweedfs-hadoop3-client-4.38.jar -spark.executor.extraClassPath=/Users/chris/go/src/github.com/seaweedfs/seaweedfs/other/java/hdfs3/target/seaweedfs-hadoop3-client-4.38.jar -spark.hadoop.fs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem -spark.hadoop.fs.defaultFS=seaweedfs://localhost:8888 -``` - 2. create the spark history folder -``` -$ curl -X POST http://192.168.2.4:8888/spark2-history/ -``` - 3. Run a spark shell -``` -$ bin/spark-shell -20/10/18 14:11:44 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address -20/10/18 14:12:15 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable -Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties -Setting default log level to "WARN". -To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). -Spark context Web UI available at http://192.168.2.4:4040 -Spark context available as 'sc' (master = local, app id = local-1603055539864). -Spark session available as 'spark'. +$ bin/spark-shell \ + --master 'local[2]' \ + --conf spark.driver.extraClassPath=/path/to/seaweedfs-hadoop3-client-4.38.jar \ + --conf spark.executor.extraClassPath=/path/to/seaweedfs-hadoop3-client-4.38.jar \ + --conf spark.hadoop.fs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem \ + --conf spark.hadoop.fs.AbstractFileSystem.seaweedfs.impl=seaweed.hdfs.SeaweedAbstractFileSystem +... Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ - /___/ .__/\_,_/_/ /_/\_\ version 3.0.0 + /___/ .__/\_,_/_/ /_/\_\ version 4.0.3 /_/ -Using Scala version 2.12.10 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_202) -Type in expressions to have them evaluated. -Type :help for more information. - -scala> sc.textFile("/buckets/large/ttt.txt").count -res0: Long = 9374 +Using Scala version 2.13.16 (OpenJDK 64-Bit Server VM, Java 17.0.19) +scala> val df = Seq((1,"alice"),(2,"bob"),(3,"carol"),(4,"dave")).toDF("id","name") +scala> df.write.mode("overwrite").parquet("seaweedfs://localhost:8888/spark-test/people") +scala> val back = spark.read.parquet("seaweedfs://localhost:8888/spark-test/people") +scala> back.count() +res: Long = 4 +scala> back.orderBy("id").show(false) ++---+-----+ +|id |name | ++---+-----+ +|1 |alice| +|2 |bob | +|3 |carol| +|4 |dave | ++---+-----+ +``` + +The write lands under `/spark-test/people/` on the filer as `_SUCCESS` plus the snappy parquet part files. The path prefix `spark-test` is created on write, no need to pre-create it. + +# A Full Example +Here is a local example switching everything to SeaweedFS. In the `$SPARK_HOME/conf/spark-defaults.conf` file, + + 1. `spark-defaults.conf` +``` +spark.master=local[*] + +spark.hadoop.fs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem +spark.hadoop.fs.AbstractFileSystem.seaweedfs.impl=seaweed.hdfs.SeaweedAbstractFileSystem +spark.hadoop.fs.defaultFS=seaweedfs://localhost:8888 + +spark.driver.extraClassPath=/path/to/seaweedfs-hadoop3-client-4.38.jar +spark.executor.extraClassPath=/path/to/seaweedfs-hadoop3-client-4.38.jar + +spark.eventLog.enabled=true +spark.eventLog.dir=seaweedfs://localhost:8888/spark-history/ +spark.history.fs.logDirectory=seaweedfs://localhost:8888/spark-history/ +``` + 2. Run a spark shell. Event logs are written to SeaweedFS under `/spark-history/`. +``` +$ bin/spark-shell +... +Welcome to + ____ __ + / __/__ ___ _____/ /__ + _\ \/ _ \/ _ `/ __/ '_/ + /___/ .__/\_,_/_/ /_/\_\ version 4.0.3 + /_/ + +Using Scala version 2.13.16 (OpenJDK 64-Bit Server VM, Java 17.0.19) +Spark session available as 'spark'. + +scala> sc.parallelize(1 to 100).sum +res0: Double = 5050.0 ```