Updated Hadoop Compatible File System (markdown)

Chris Lu
2018-12-07 22:18:20 -08:00
parent 5c928ffff7
commit c5d9a219a2
+55 -1
@@ -28,4 +28,58 @@ $ bin/hdfs dfs -Dfs.defaultFS=seaweedfs://localhost:8888 \
-ls /
```
Both reads and writes are working fine.
Both reads and writes are working fine.
# Installation for Hadoop
* Configure Hadoop to use SeaweedFS in `etc/hadoop/conf/core-site.xml`. `core-site.xml` resides on each node in the Hadoop cluster. You must add the same properties to each instance of `core-site.xml`. There are several properties to modify:
1. `fs.seaweedfs.impl`: This property defines the Seaweed HCFS implementation classes that are contained in the SeaweedFS HDFS client JAR. It is required.
1. `fs.defaultFS`: This property defines the default file system URI to use. It is optional if a path always has prefix `seaweedfs://localhost:8888`.
```
<configuration>
<property>
<name>fs.seaweedfs.impl</name>
<value>seaweed.hdfs.SeaweedFileSystem</value>
</property>
<property>
<name>fs.defaultFS</name>
<value>seaweedfs://localhost:8888</value>
</property>
</configuration>
```
* Deploy the SeaweedFS HDFS client jar
```
# Run the classpath command to get the list of directories in the classpath
$ bin/hadoop classpath
# Copy SeaweedFS HDFS client jar to one of the folders
$ cd ${HADOOP_HOME}
$ cp ./seaweedfs-hadoop-client-1.0-SNAPSHOT.jar share/hadoop/common/lib/
```
Now you can do this:
```
$ cd ${HADOOP_HOME}
$ bin/hdfs dfs -ls /
```
# Installation for HBase
If HBase is used, create a folder and configure the HBase root directory in `etc/hbase/conf/hbase-site.xml`:
```
<configuration>
<property>
<name>hbase.rootdir</name>
<value>seaweedfs://localhost:8888/hbase</value>
</property>
</configuration>
```
## No native shared libraries
The SeaweedFS hadoop client is a pure java library. There are no native libraries to install if you already have Hadoop running.
This is different from many other HCFS options. If native shared libraries are needed, these libraries need to be install on all hadoop nodes. This is quite some task.
## Shaded Fat Jar
One of the headache with complicated Java systems is the jar runtime dependency problem, which is resolved by Go's build time dependency resolution. For this SeaweedFS hadoop client, the required jars are mostly shaded and packaged as one fat jar, so there are no extra jar files needed.