diff --git a/Client-Libraries.md b/Client-Libraries.md index 828d44a..85bacea 100644 --- a/Client-Libraries.md +++ b/Client-Libraries.md @@ -41,4 +41,4 @@ SeaweedFS uses GRPC internally. You can use them too. Just check https://github. When developing HDFS compatible file system, which allows replacing HDFS with SeaweedFS, a Java implementation of the GRPC client API is developed. * Java GRPC client Source code: https://github.com/seaweedfs/seaweedfs/tree/master/other/java/client -* Java GRPC client Maven Repo: https://mvnrepository.com/artifact/com.github.chrislusf/seaweedfs-client +* Java GRPC client Maven Repo: https://mvnrepository.com/artifact/com.seaweedfs/seaweedfs-client diff --git a/Hadoop-Benchmark.md b/Hadoop-Benchmark.md index 7b0d5cd..d9ff646 100644 --- a/Hadoop-Benchmark.md +++ b/Hadoop-Benchmark.md @@ -26,7 +26,7 @@ Then get the seaweedfs hadoop client jar. ``` cd share/hadoop/common/lib/ -wget https://oss.sonatype.org/service/local/repositories/releases/content/com/github/chrislusf/seaweedfs-hadoop2-client/3.59/seaweedfs-hadoop2-client-3.59.jar +wget https://oss.sonatype.org/service/local/repositories/releases/content/com/seaweedfs/seaweedfs-hadoop3-client/4.00/seaweedfs-hadoop3-client-4.00.jar ``` # TestDFSIO Benchmark diff --git a/Hadoop-Compatible-File-System.md b/Hadoop-Compatible-File-System.md index 4fcf397..6b5a9c0 100644 --- a/Hadoop-Compatible-File-System.md +++ b/Hadoop-Compatible-File-System.md @@ -7,47 +7,31 @@ SeaweedFS excels on small files and has no issue to store large files. Now it is $cd $GOPATH/src/github.com/seaweedfs/seaweedfs/other/java/client $ mvn install -# build for hadoop2 -$cd $GOPATH/src/github.com/seaweedfs/seaweedfs/other/java/hdfs2 -$ mvn package -$ ls -al target/seaweedfs-hadoop2-client-3.59.jar - # build for hadoop3 $cd $GOPATH/src/github.com/seaweedfs/seaweedfs/other/java/hdfs3 $ mvn package -$ ls -al target/seaweedfs-hadoop3-client-3.59.jar +$ ls -al target/seaweedfs-hadoop3-client-4.00.jar ``` Maven ``` - com.github.chrislusf + com.seaweedfs seaweedfs-hadoop3-client - 3.59 - - -or - - - com.github.chrislusf - seaweedfs-hadoop2-client - 3.59 + 4.00 ``` Or you can download the latest version from MavenCentral -* https://mvnrepository.com/artifact/com.github.chrislusf/seaweedfs-hadoop2-client - * [seaweedfs-hadoop2-client-3.59.jar](https://oss.sonatype.org/service/local/repositories/releases/content/com/github/chrislusf/seaweedfs-hadoop2-client/3.59/seaweedfs-hadoop2-client-3.59.jar) -* https://mvnrepository.com/artifact/com.github.chrislusf/seaweedfs-hadoop3-client - * [seaweedfs-hadoop3-client-3.59.jar](https://oss.sonatype.org/service/local/repositories/releases/content/com/github/chrislusf/seaweedfs-hadoop3-client/3.59/seaweedfs-hadoop3-client-3.59.jar) +* https://mvnrepository.com/artifact/com.seaweedfs/seaweedfs-hadoop3-client + * [seaweedfs-hadoop3-client-4.00.jar](https://oss.sonatype.org/service/local/repositories/releases/content/com.seaweedfs/seaweedfs-hadoop3-client/4.00/seaweedfs-hadoop3-client-4.00.jar) # Test SeaweedFS on Hadoop Suppose you are getting a new Hadoop installation. Here are the minimum steps to get SeaweedFS to run. -You would need to start a weed filer first, build the seaweedfs-hadoop2-client-3.59.jar -or seaweedfs-hadoop3-client-3.59.jar, and do the following: +You would need to start a weed filer first, build the seaweedfs-hadoop3-client-4.00.jar, and do the following: ``` # optionally adjust hadoop memory allocation @@ -57,15 +41,10 @@ $ cd ${HADOOP_HOME} # create etc/hadoop/mapred-site.xml, just to satisfy hdfs dfs. skip this if the file already exists. $ echo "" > etc/hadoop/mapred-site.xml -# on hadoop2 +# on hadoop3 $ bin/hdfs dfs -Dfs.defaultFS=seaweedfs://localhost:8888 \ -Dfs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem \ - -libjars ./seaweedfs-hadoop2-client-3.59.jar \ - -ls / -# or on hadoop3 -$ bin/hdfs dfs -Dfs.defaultFS=seaweedfs://localhost:8888 \ - -Dfs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem \ - -libjars ./seaweedfs-hadoop3-client-3.59.jar \ + -libjars ./seaweedfs-hadoop3-client-4.00.jar \ -ls / ``` @@ -104,17 +83,15 @@ Both reads and writes are working fine. ``` -* Deploy the SeaweedFS HDFS client jar +# Deploy the SeaweedFS HDFS client jar ``` # Run the classpath command to get the list of directories in the classpath $ bin/hadoop classpath # Copy SeaweedFS HDFS client jar to one of the folders $ cd ${HADOOP_HOME} -# for hadoop2 -$ cp ./seaweedfs-hadoop2-client-3.59.jar share/hadoop/common/lib/ -# or for hadoop3 -$ cp ./seaweedfs-hadoop3-client-3.59.jar share/hadoop/common/lib/ +# for hadoop3 +$ cp ./seaweedfs-hadoop3-client-4.00.jar share/hadoop/common/lib/ ``` Now you can do this: diff --git a/Run-Presto-on-SeaweedFS.md b/Run-Presto-on-SeaweedFS.md index 7d200e2..ca18528 100644 --- a/Run-Presto-on-SeaweedFS.md +++ b/Run-Presto-on-SeaweedFS.md @@ -5,10 +5,10 @@ The installation steps are divided into 2 steps: * https://cwiki.apache.org/confluence/display/Hive/AdminManual+Metastore+Administration ### Configure Hive Metastore to support SeaweedFS -1. Copy the seaweedfs-hadoop2-client-3.59.jar to hive lib directory,for example: +1. Copy the seaweedfs-hadoop3-client-4.00.jar to hive lib directory,for example: ``` -cp seaweedfs-hadoop2-client-3.59.jar /opt/hadoop/share/hadoop/common/lib/ -cp seaweedfs-hadoop2-client-3.59.jar /opt/hive-metastore/lib/ +cp seaweedfs-hadoop3-client-4.00.jar /opt/hadoop/share/hadoop/common/lib/ +cp seaweedfs-hadoop3-client-4.00.jar /opt/hive-metastore/lib/ ``` 2. Modify core-site.xml modify core-site.xml to support SeaweedFS, 30888 is the filer port @@ -50,9 +50,9 @@ metastore.thrift.port is the access port exposed by the Hive Metadata service it Follow instructions for installation of Presto: * https://prestosql.io/docs/current/installation/deployment.html ### Configure Presto to support SeaweedFS -1. Copy the seaweedfs-hadoop2-client-3.59.jar to Presto directory,for example: +1. Copy the seaweedfs-hadoop3-client-4.00.jar to Presto directory,for example: ``` -cp seaweedfs-hadoop2-client-3.59.jar /opt/presto-server-347/plugin/hive-hadoop2/ +cp seaweedfs-hadoop3-client-4.00.jar /opt/presto-server-347/plugin/hive-hadoop3/ ``` 2. Modify core-site.xml diff --git a/SeaweedFS-Java-Client.md b/SeaweedFS-Java-Client.md index 6f31c1c..0323fbd 100644 --- a/SeaweedFS-Java-Client.md +++ b/SeaweedFS-Java-Client.md @@ -13,20 +13,20 @@ $ mvn install Gradle ```gradle -implementation 'com.github.chrislusf:seaweedfs-client:3.59' +implementation 'com.seaweedfs:seaweedfs-client:4.00' ``` Maven ```xml - com.github.chrislusf + com.seaweedfs seaweedfs-client - 3.59 + 4.00 ``` Or you can download the latest version from MavenCentral -https://mvnrepository.com/artifact/com.github.chrislusf/seaweedfs-client +https://mvnrepository.com/artifact/com.seaweedfs/seaweedfs-client ## Features diff --git a/run-HBase-on-SeaweedFS.md b/run-HBase-on-SeaweedFS.md index a97d5fa..a687005 100644 --- a/run-HBase-on-SeaweedFS.md +++ b/run-HBase-on-SeaweedFS.md @@ -1,7 +1,7 @@ # Installation for HBase Two steps to run HBase on SeaweedFS -1. Copy the seaweedfs-hadoop2-client-3.59.jar to `${HBASE_HOME}/lib` +1. Copy the seaweedfs-hadoop3-client-4.00.jar to `${HBASE_HOME}/lib` 1. And add the following 2 properties in `${HBASE_HOME}/conf/hbase-site.xml` ``` diff --git a/run-Spark-on-SeaweedFS.md b/run-Spark-on-SeaweedFS.md index 1bc6936..dbd7cbc 100644 --- a/run-Spark-on-SeaweedFS.md +++ b/run-Spark-on-SeaweedFS.md @@ -11,12 +11,12 @@ To make these files visible to Spark, set HADOOP_CONF_DIR in $SPARK_HOME/conf/sp ## installation not inheriting from Hadoop cluster configuration -Copy the seaweedfs-hadoop2-client-3.59.jar to all executor machines. +Copy the seaweedfs-hadoop3-client-4.00.jar to all executor machines. Add the following to spark/conf/spark-defaults.conf on every node running Spark ``` -spark.driver.extraClassPath=/path/to/seaweedfs-hadoop2-client-3.59.jar -spark.executor.extraClassPath=/path/to/seaweedfs-hadoop2-client-3.59.jar +spark.driver.extraClassPath=/path/to/seaweedfs-hadoop3-client-4.00.jar +spark.executor.extraClassPath=/path/to/seaweedfs-hadoop3-client-4.00.jar ``` And modify the configuration at runtime: @@ -37,8 +37,8 @@ And modify the configuration at runtime: 1. change the spark-defaults.conf ``` -spark.driver.extraClassPath=/Users/chris/go/src/github.com/seaweedfs/seaweedfs/other/java/hdfs2/target/seaweedfs-hadoop2-client-3.59.jar -spark.executor.extraClassPath=/Users/chris/go/src/github.com/seaweedfs/seaweedfs/other/java/hdfs2/target/seaweedfs-hadoop2-client-3.59.jar +spark.driver.extraClassPath=/Users/chris/go/src/github.com/seaweedfs/seaweedfs/other/java/hdfs3/target/seaweedfs-hadoop3-client-4.00.jar +spark.executor.extraClassPath=/Users/chris/go/src/github.com/seaweedfs/seaweedfs/other/java/hdfs3/target/seaweedfs-hadoop3-client-4.00.jar spark.hadoop.fs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem ``` @@ -81,8 +81,8 @@ spark.history.fs.cleaner.enabled=true spark.history.fs.logDirectory=seaweedfs://localhost:8888/spark2-history/ spark.eventLog.dir=seaweedfs://localhost:8888/spark2-history/ -spark.driver.extraClassPath=/Users/chris/go/src/github.com/seaweedfs/seaweedfs/other/java/hdfs2/target/seaweedfs-hadoop2-client-3.59.jar -spark.executor.extraClassPath=/Users/chris/go/src/github.com/seaweedfs/seaweedfs/other/java/hdfs2/target/seaweedfs-hadoop2-client-3.59.jar +spark.driver.extraClassPath=/Users/chris/go/src/github.com/seaweedfs/seaweedfs/other/java/hdfs3/target/seaweedfs-hadoop3-client-4.00.jar +spark.executor.extraClassPath=/Users/chris/go/src/github.com/seaweedfs/seaweedfs/other/java/hdfs3/target/seaweedfs-hadoop3-client-4.00.jar spark.hadoop.fs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem spark.hadoop.fs.defaultFS=seaweedfs://localhost:8888 ```