mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-11 16:57:45 +02:00
Updated Hadoop Compatible File System (markdown)
1 parent
12a5b40411
commit
d843b2da79
1 file changed
+27
@@ -77,8 +77,35 @@ Follow instructions on spark doc:
|
||||
* https://spark.apache.org/docs/latest/configuration.html#inheriting-hadoop-cluster-configuration
|
||||
* https://spark.apache.org/docs/latest/configuration.html#custom-hadoophive-configuration
|
||||
|
||||
## installation inheriting from Hadoop cluster configuration
|
||||
|
||||
Inheriting from Hadoop cluster configuration should be the easiest way.
|
||||
|
||||
To make these files visible to Spark, set HADOOP_CONF_DIR in $SPARK_HOME/conf/spark-env.sh to a location containing the configuration file `core-site.xml`, usually `/etc/hadoop/conf`
|
||||
|
||||
## installation not inheriting from Hadoop cluster configuration
|
||||
|
||||
Copy the seaweedfs-hadoop-client-x.x.x.jar to all executor machines.
|
||||
|
||||
Add the following to spark/conf/spark-defaults.conf on every node running Spark
|
||||
```
|
||||
spark.driver.extraClassPath /path/to/seaweedfs-hadoop-client-x.x.x.jar
|
||||
spark.executor.extraClassPath /path/to/seaweedfs-hadoop-client-x.x.x.jar
|
||||
```
|
||||
|
||||
And modify the configuration at runntime:
|
||||
|
||||
```
|
||||
./bin/spark-submit \
|
||||
--name "My app" \
|
||||
--master local[4] \
|
||||
--conf spark.eventLog.enabled=false \
|
||||
--conf "spark.executor.extraJavaOptions=-XX:+PrintGCDetails -XX:+PrintGCTimeStamps" \
|
||||
--conf spark.hadoop.fs.seaweedfs.impl=seaweed.hdfs.SeaweedFileSystem \
|
||||
--conf spark.hadoop.fs.defaultFS=seaweedfs://localhost:8888 \
|
||||
myApp.jar
|
||||
```
|
||||
|
||||
# Installation for HBase
|
||||
If HBase is used, create a folder and configure the HBase root directory in `etc/hbase/conf/hbase-site.xml`:
|
||||
|
||||
|
||||
Reference in new issue
Block a user