diff --git a/HDFS-via-S3-connector.md b/HDFS-via-S3-connector.md new file mode 100644 index 0000000..0155308 --- /dev/null +++ b/HDFS-via-S3-connector.md @@ -0,0 +1,46 @@ +Current recommended way for Hadoop to access SeaweedFS is via [SeaweedFS Hadoop Compatible File System](Hadoop-Compatible-File-System), which is the most efficient way with the client directly accessing filer for metadata and accessing volume servers for file content. + +However, the downside is that you need to add a SeaweedFS jar to classpath, and change some Hadoop settings. + +# HDFS Access SeaweedFS via S3 connector + +The S3a connector is already included in hadoop distributions. You can use it directly. + +Use + +``` + + 8 + 8 + UTF-8 + 2.12.11 + 3.1.2 + 3.3.1 + compile + + +``` +And add this in your code: +``` +SparkSession spark = SparkSession.builder() + .master("local[*]") + .config("spark.eventLog.enabled", "false") + .config("spark.driver.memory", "1g") + .config("spark.executor.memory", "1g") + .appName("SparkDemoFromS3") + .getOrCreate(); +spark.sparkContext().hadoopConfiguration().set("fs.s3a.access.key", "admin"); +spark.sparkContext().hadoopConfiguration().set("fs.s3a.secret.key", "xx"); +spark.sparkContext().hadoopConfiguration().set("fs.s3a.endpoint", "ip:8333"); +spark.sparkContext().hadoopConfiguration().set("com.amazonaws.services.s3a.enableV4", "true"); +spark.sparkContext().hadoopConfiguration().set("fs.s3a.path.style.access", "true"); +spark.sparkContext().hadoopConfiguration().set("fs.s3a.connection.ssl.enabled", "false"); +spark.sparkContext().hadoopConfiguration().set("fs.s3a.multiobjectdelete.enable", "false"); +spark.sparkContext().hadoopConfiguration().set("fs.s3a.directory.marker.retention", "keep"); +spark.sparkContext().hadoopConfiguration().set("fs.s3a.change.detection.version.required", "false"); +spark.sparkContext().hadoopConfiguration().set("fs.s3a.change.detection.mode", "warn"); +RDD rdd = spark.sparkContext().textFile("s3a://bk002/test1.txt", 1); +System.out.println(rdd.count()); +rdd.saveAsTextFile("s3a://bk002/testcc/t2"); + +```