Github-Mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-09-12 17:40:43 +02:00
Code Issues Packages Projects Releases Wiki Activity
12,170 Commits 176 Branches 359 Tags
7b067d2e596fcfbe7b7ae38ca3ae020286548b27
Commit Graph
53 Commits
Author SHA1 Message Date
chrislu a3cf4eb843 debug: track stream lifecycle and total bytes written 
Added comprehensive logging to identify why Parquet files fail with
'EOFException: Still have: 78 bytes left'.

Key additions:
1. SeaweedHadoopOutputStream constructor logging with 🔧 marker
   - Shows when output streams are created
   - Logs path, position, bufferSize, replication

2. totalBytesWritten counter in SeaweedOutputStream
   - Tracks cumulative bytes written via write() calls
   - Helps identify if Parquet wrote 762 bytes but only 684 reached chunks

3. Enhanced close() logging with 🔒 and ✅ markers
   - Shows totalBytesWritten vs position vs buffer.position()
   - If totalBytesWritten=762 but position=684, write submission failed
   - If buffer.position()=78 at close, buffer wasn't flushed

Expected scenarios in next run:
A) Stream never created → No 🔧 log for .parquet files
B) Write failed → totalBytesWritten=762 but position=684
C) Buffer not flushed → buffer.position()=78 at close
D) All correct → totalBytesWritten=position=684, but Parquet expects 762

This will pinpoint whether the issue is in:
- Stream creation/lifecycle
- Write submission
- Buffer flushing
- Or Parquet's internal state
 2025-11-23 11:54:01 -08:00
chrislu c86177e063 add comments 2025-11-22 12:28:09 -08:00
chrislu a7f786ac92 NPE 2025-11-22 12:22:19 -08:00
chrislu c96448f3a5 more flexible replication configuration 2025-11-21 21:06:14 -08:00
dependabot[bot]andchrislu c14e513964 chore(deps): bump org.apache.hadoop:hadoop-common from 3.2.4 to 3.4.0 in /other/java/hdfs3 (#7512) 
* chore(deps): bump org.apache.hadoop:hadoop-common in /other/java/hdfs3

Bumps org.apache.hadoop:hadoop-common from 3.2.4 to 3.4.0.

---
updated-dependencies:
- dependency-name: org.apache.hadoop:hadoop-common
  dependency-version: 3.4.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* add java client unit tests

* Update dependency-reduced-pom.xml

* add java integration tests

* fix

* fix buffer

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: chrislu <chris.lu@gmail.com>
 2025-11-19 21:22:18 -08:00
orthoxerox d8cc269294 feature: added ssl support for HCFS (#6699) (#6775) 2025-05-09 09:09:13 -07:00
chrislu d003bb0166 java 3.13 2022-06-26 20:41:51 -07:00
Chris Lu e5fc35ed0c change server address from string to a type 2021-09-12 22:47:52 -07:00
Chris Lu ad36c7b0d7 refactoring: only expose FilerClient class 2021-02-08 02:28:45 -08:00
Chris Lu 9c1efdf11b HCFS: 1.5.9 2021-02-05 11:47:56 -08:00
Chris Lu 8f3a51f2b8 Java: 1.5.8 additional fixes 2021-02-05 10:42:20 -08:00
Chris Lu 6f4aab51f9 refactoring SeaweedInputStream 2021-01-30 06:16:02 -08:00
Chris Lu 043c2d7960 refactoring SeaweedOutputStream 2021-01-30 05:39:09 -08:00
Chris Lu 4d2855476c Hadoop: add BufferedByteBufferReadableInputStream 
fix https://github.com/chrislusf/seaweedfs/issues/1645
 2020-12-03 00:08:05 -08:00
Chris Lu 3857f9c840 Hadoop: switch to ByteBuffer 
fix https://github.com/chrislusf/seaweedfs/issues/1645
 2020-12-02 23:45:39 -08:00
Chris Lu a9efaa6385 HDFS: implement ByteBufferReadable 
fix https://github.com/chrislusf/seaweedfs/issues/1645
 2020-12-01 17:20:31 -08:00
Chris Lu f4abd01adf filer: cache small file to filer store 2020-11-30 04:34:04 -08:00
limd 4737df597d HCFS: 
1. add replication parameter
2. fix close sequence
 2020-11-17 19:05:57 +08:00
hilimd 546f1bcb90 Merge pull request #34 from chrislusf/master 
sync
 2020-11-05 12:02:47 +08:00
Chris Lu c709059b69 HCFS: add close() to SeaweedFileSystem.java 2020-11-03 00:46:00 -08:00
hilimd 843865f2ca Merge pull request #33 from chrislusf/master 
sync
 2020-10-26 22:01:50 +08:00
Chris Lu f375b93aef renaming 2020-10-25 15:32:43 -07:00
Chris Lu 596d476e3d HCFS: 1.5.2 2020-10-25 14:44:18 -07:00
hilimd 6e63e5eaf0 Merge pull request #29 from chrislusf/master 
sync
 2020-10-19 09:42:43 +08:00
Chris Lu 459de70a77 Hadoop: more accurate block size 2020-10-18 14:13:46 -07:00
hilimd b9a446839a Merge pull request #24 from chrislusf/master 
sync
 2020-10-10 16:20:26 +08:00
Chris Lu 912ef2bc53 Hadoop: remove unused variable bufferSize 2020-10-10 01:14:17 -07:00
limd ac162fc857 hdfs: Hadoop on SeaweedFS: create empty file 2020-10-10 13:07:27 +08:00
Chris Lu 4929d0634e Hadoop on SeaweedFS: create empty file 
fix https://github.com/chrislusf/seaweedfs/issues/1494
 2020-09-25 10:57:43 -07:00
limd 95bfec4931 hadoop: filesystem cannot create file 
issues: https://github.com/chrislusf/seaweedfs/issues/1494
 2020-09-26 01:38:48 +08:00
Chris Lu 5eee4983f3 1.4.7 hdfs configurable fs.seaweed.buffer.size 2020-09-16 17:18:18 -07:00
Chris Lu 13bfe5deef same logic for reading random access files from Go 2020-08-16 21:14:39 -07:00
Chris Lu 15dc0a704d Revert "add read ahead input stream" 
This reverts commit b3089dcc8e.
 2020-08-03 09:11:24 -07:00
Chris Lu b3089dcc8e add read ahead input stream 2020-08-03 09:06:09 -07:00
Chris Lu 6b41c5250b Hadoop file system: 1.4.3 
added buffered fs input stream
 2020-07-29 21:33:10 -07:00
Chris Lu 703057bff9 mirror changes from hdfs2 2020-07-29 20:15:43 -07:00
Chris Lu 6839f96c0c simplify 2020-07-22 22:52:49 -07:00
Chris Lu ae3e6d8244 remove changing buffer size 2020-07-20 20:27:19 -07:00
Chris Lu 1d724ab237 hdfs: support read write chunk manifest 2020-07-20 03:29:17 -07:00
Chris Lu f90d2c93c9 1.3.9 remove logs 2020-07-17 16:11:57 -07:00
Chris Lu 3abd74b1d7 1.3.8 2020-07-17 15:46:20 -07:00
Chris Lu 2629da2cb9 simplify inputstream 2020-07-16 10:54:29 -07:00
李明达 74456b3d5e 1、Add SeaweedFS implementation of Hadoop AbstractFileSystem. The implementation delegates to the existing SeaweedFS FileSystem and is only necessary for use with Hadoop 2.x/3.x. Configuration example in Hadoop core-site.xml file: 
<property>
    <name>fs.AbstractFileSystem.seaweedfs.impl</name>
    <value>seaweed.hdfs.SeaweedAbstractFileSystem</value>
 </property>

2、Fix hiveserver2 startup NullPointerException
 2020-07-16 16:47:09 +08:00
Chris Lu 7bca72deed reuse bytebuffer 2020-07-15 23:33:31 -07:00
Chris Lu 86c8137546 1.3.4 2020-07-15 13:49:06 -07:00
Chris Lu bc3be0bb37 Hadoop: 1.3.3 
improve memory efficiency
 2020-07-15 13:25:44 -07:00
Chris Lu 222f93e816 possibly fix concurrent access to entry object in SeaweedOutputStream 2020-06-17 14:59:43 -07:00
Chris Lu 8dfaaeabfd HCFS: 1.2.8 fix hbase related bugs 
1. SeaweedFileSystem.listStatus need to work with file also
2. SeaweedRead readChunkView has wrong len
 2020-05-24 17:07:34 -07:00
Chris Lu a999ed94d0 update hdfs client 2019-10-08 22:49:10 -07:00
Chris Lu ae53f63680 filer: recursive deletion optionally ignoring any errors 
fix https://github.com/chrislusf/seaweedfs/issues/1062
 2019-09-11 20:26:20 -07:00