55

Hadoop Benchmark

Chris Lu edited this page 2026-07-08 01:09:49 -07:00

Table of Contents

• Setup Hadoop Benchmark
• TestDFSIO Benchmark
• TestDFSIO write tests
• TestDFSIO read tests
• Independent Benchmarks
• Sugon (中科曙光)

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

Setup Hadoop Benchmark

Here are my steps. First, checkout the hadoop 3.4.1 binary, untar, and cd in to the hadoop directory.

wget https://downloads.apache.org/hadoop/common/hadoop-3.4.1/hadoop-3.4.1.tar.gz
tar xvf hadoop-3.4.1.tar.gz
cd hadoop-3.4.1

Modify the file ./etc/hadoop/core-site.xml

<configuration>
    <property>
        <name>fs.seaweedfs.impl</name>
        <value>seaweed.hdfs.SeaweedFileSystem</value>
    </property>
    <property>
        <name>fs.defaultFS</name>
        <value>seaweedfs://localhost:8888</value>
    </property>
    <property>
       <name>fs.AbstractFileSystem.seaweedfs.impl</name>
       <value>seaweed.hdfs.SeaweedAbstractFileSystem</value>
    </property>
</configuration>

Then get the seaweedfs hadoop client jar.

cd share/hadoop/common/lib/
wget https://repo1.maven.org/maven2/com/seaweedfs/seaweedfs-hadoop3-client/4.38/seaweedfs-hadoop3-client-4.38.jar
cd ../../../..

TestDFSIO Benchmark

The TestDFSIO benchmark is used for measuring I/O (read/write) performance.

However, the generated data to write is all zero. SeaweedFS automatically compress this kind of data. So this is not scientific for now.

TestDFSIO write tests

Start the TestDFSIO write tests:

bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.4.1-tests.jar TestDFSIO -write -nrFiles 4 -size 64MB -bufferSize 8388608 -resFile /tmp/TestDFSIOwrite.txt

...
INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
INFO fs.TestDFSIO:             Date & time: Wed Jul 08 00:45:01 PDT 2026
INFO fs.TestDFSIO:         Number of files: 4
INFO fs.TestDFSIO:  Total MBytes processed: 256
INFO fs.TestDFSIO:       Throughput mb/sec: 605.2
INFO fs.TestDFSIO:  Average IO rate mb/sec: 633.27
INFO fs.TestDFSIO:   IO rate std deviation: 130.3
INFO fs.TestDFSIO:      Test exec time sec: 1.32

TestDFSIO read tests

Start the TestDFSIO read tests:

bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.4.1-tests.jar TestDFSIO -read -nrFiles 4 -size 64MB -bufferSize 8388608 -resFile /tmp/TestDFSIOread.txt

...

INFO fs.TestDFSIO: ----- TestDFSIO ----- : read
INFO fs.TestDFSIO:             Date & time: Wed Jul 08 00:45:10 PDT 2026
INFO fs.TestDFSIO:         Number of files: 4
INFO fs.TestDFSIO:  Total MBytes processed: 256
INFO fs.TestDFSIO:       Throughput mb/sec: 1174.31
INFO fs.TestDFSIO:  Average IO rate mb/sec: 1226.51
INFO fs.TestDFSIO:   IO rate std deviation: 227.6
INFO fs.TestDFSIO:      Test exec time sec: 1.28

Clean up the benchmark data when done:

bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.4.1-tests.jar TestDFSIO -clean

Independent Benchmarks

Sugon (中科曙光)

To test the performance of HDFS and SeaweedFS, we did a comparison by running 4 common spark operators, such as count, group by, join and write, for group by and join , there is a count followed to act.

The basic configuration information of cluster is as follows:

• HDFS：

  • Node number: 25
  • Total disks: 36disk * 25node = 900disk
  • Disk capacity: 3.7T SATA
  • Total disk capacity: 3.19PB
  • Replication: 5

• SeaweedFS:

  • Node number: 6（3+3 rack）
  • Disk capacity: 3.7T SATA
  • Cluster max volume: 21500
  • Total disk capacity: 799TB
  • Replication policy: 010

Here are the details and results of our test. At the beginning of the test, we put our data to both HDFS and SeaweedFS. The amount of the data is 100 million records, and stored in 200 parquet files. The size of each parquet file is about 89 MB. We ran spark on yarn with 20 executors. In spark, we got two DataFrames by reading parquet from HDFS and HCFS separately, then executed count, group by and join by 100 times , and write by 10 times, on each DataFrame.

As for count, SeaweedFS's advantage is obvious. The average time of the DataFrame from HDFS is 4.05 seconds, while SeaweedFS is only 0.659. Following is the result:

Summary	HDFS	SeaweedFS
Count	100	100
Mean	4.050	0.659
Stddev	0.264	0.941
Min	3.678	0.392
Max	5.692	9.688

As for write, we wrote the DataFrame from HDFS to SeaweedFS, and wrote the DataFrame from HCFS to SeaweedFS. Following is the result:

Summary	HDFS	SeaweedFS
Count	10	10
Mean	234.279	232.078
Stddev	26.823	12.652
Min	216.931	214.349
Max	307.330	252.375

As for group by, following is the result:

Summary	HDFS	SeaweedFS
Count	100	100
Mean	14.121	12.515
Stddev	1.972	1.255
Min	12.879	11.322
Max	32.296	22.573

As for join, every DataFrame join with itself on one column. Following is the result:

Summary	HDFS	SeaweedFS
Count	100	100
Mean	25.684	23.897
Stddev	0.934	1.381
Min	24.006	22.275
Max	30.991	30.279

Introduction

• Quick Start with weed mini
• Simplest S3 Bucket and User Setup
• Components
• Blob Store Architecture
• Getting Started
• Production Setup
• A typical step‐by‐step example
• Benchmarks
• FAQ
• Applications

API

• Master Server API
• Volume Server API
• Filer Server API
• Client Libraries
• SeaweedFS Java Client

Configuration

• Replication
• Store file with a Time To Live
• Failover Master Server
• Erasure coding for warm storage
• EC Bitrot Detection
• Server Startup via Systemd
• Environment Variables

Filer

• Filer Setup
• Directories and Files
• File Operations Quick Reference
• Data Structure for Large Files
• Filer Data Encryption
• Filer Commands and Operations
• Filer JWT Use
• TUS Resumable Uploads

Filer Stores

• Filer Cassandra Setup
• Filer Redis Setup
• Super Large Directories
• Path-Specific Filer Store
• Choosing a Filer Store
• Customize Filer Store

Management

• Admin UI
• Admin UI OIDC
• Worker
• Plugin Worker Scheduling

Advanced Filer Configurations

• Migrate to Filer Store
• Add New Filer Store
• Filer Store Replication
• Filer Active Active cross cluster continuous synchronization
• Filer as a Key-Large-Value Store
• Path Specific Configuration
• Filer Change Data Capture
• Filer Operation Serialization

FUSE Mount

• Mount on Windows
• FIO benchmark
• fstab and systemd mount
• POSIX Compliance
• Distributed POSIX Locks
• P2P reading in weed mount
• Mount over the Internet

WebDAV

SFTP Server

Cloud Drive

• Cloud Drive Benefits
• Cloud Drive Architecture
• Configure Remote Storage
• Azure Blob Storage Authentication
• Mount Remote Storage
• Cache Remote Storage
• Cloud Drive Quick Setup
• Gateway to Remote Object Storage

AWS S3 API

• Amazon S3 API
• Supported APIs vs Minio
• S3 Lifecycle
  • S3 Lifecycle Recipes
  • S3 Lifecycle Operator Guide
  • S3 Lifecycle Monitoring
  • S3 Lifecycle Troubleshooting
  • S3 Lifecycle Architecture
• S3 Lifecycle vs Volume TTL
• S3 Conditional Operations
• S3 CORS
• S3 Object Lock and Retention
• S3 Object Versioning
• S3 RenameObject
• S3 API Benchmark
• S3 API FAQ
• S3 Bucket Quota
• S3 Rate Limiting
• S3 API Audit log
• S3 Nginx Proxy
• Docker Compose for S3

S3 Table Bucket

• S3 Table Bucket
• S3 Table Bucket Commands
• S3 Tables Security
• SeaweedFS Iceberg Catalog
• Iceberg REST Catalog API
• Iceberg Table Maintenance
• SeaweedFS Lance Catalog
• Lance Maintenance Worker

Iceberg Integrations

• Spark Iceberg Integration
• Trino Iceberg Integration
• Dremio Iceberg Integration
• DuckDB Iceberg Integration
• Doris Iceberg Integration
• RisingWave Iceberg Integration
• Lakekeeper Iceberg Integration

Lance Integrations

• LanceDB Integration
• Spark Lance Integration
• DuckDB Lance Integration

S3 Authentication & IAM

• S3 Configuration - Start Here
• S3 Credentials (-s3.config)
• OIDC Integration (-s3.iam.config)
• Kubernetes ServiceAccount Authentication (IRSA-style)
• S3 Policy Variables
• S3 Policy Conditions
• S3 Bucket Policies
• Amazon IAM API
• AWS IAM CLI
• weed shell - Shell IAM Commands

Server-Side Encryption

• Server-Side Encryption
• Server-Side Encryption SSE-KMS
• Server-Side Encryption SSE-C

S3 Client Tools

• AWS CLI with SeaweedFS
• s3cmd with SeaweedFS
• rclone with SeaweedFS
• restic with SeaweedFS
• nodejs with Seaweed S3

Machine Learning

• TensorFlow with SeaweedFS
• Distributing AI Model Files for Multi-GPU Loading

HDFS

• Hadoop Compatible File System
• run Spark on SeaweedFS
• run HBase on SeaweedFS
• Run Trino on SeaweedFS
• Hadoop Benchmark
• HDFS via S3 connector

Replication and Backup

• Async Replication to another Filer [Deprecated]
• Async Backup
• Async Filer Metadata Backup
• Async Replication to Cloud [Deprecated]
• Kubernetes Backups and Recovery with K8up

Metadata Change Events

• Filer Metadata Events
• Filer Notification Webhook

Messaging

• Structured Data Lake with SMQ and SQL
• Seaweed Message Queue
• SQL Queries on Message Queue
• SQL Quick Reference
• PostgreSQL-compatible Server weed db
• Pub-Sub to SMQ to SQL
• Kafka to Kafka Gateway to SMQ to SQL

Use Cases

• Use Cases
• Actual Users

Operations

• System Metrics
• weed shell
• Data Backup
• Deployment to Kubernetes and Minikube
• Helm Chart Recipes
• Deployment with seaweed-up
  • seaweed-up Command Reference
  • Cluster Plan Workflow
  • Cluster Plan Inventory Reference
  • Cluster Plan Day 2 Operations

Rust Volume Server

• Rust Volume Server

Advanced

• Large File Handling
• Optimization
• Optimization for Many Small Buckets
• Volume Management
• Tiered Storage
• Cloud Tier
• Cloud Monitoring
• Load Command Line Options from a file
• SRV Service Discovery
• Volume Files Structure

Security

• Security Overview
• Security Configuration
• Cryptography and FIPS Compliance
• Run Blob Storage on Public Internet

Misc Use Case Examples

• UrBackup with SeaweedFS
• Docker Image Registry with SeaweedFS
• SeaweedFS in Docker Swarm
• Words from SeaweedFS Users
• Independent Benchmarks
• Hardware