fix(mount): remove fid pool to stop master over-allocating volumes (#9111)

* fix(mount): remove fid pool to stop master over-allocating volumes

The writeback-cache fid pool pre-allocated file IDs with
ExpectedDataSize = ChunkSizeLimit (typically 8+ MB). The master's
PickForWrite charges count * expectedDataSize against the volume's
effectiveSize, so a full pool refill could charge hundreds of MB
against a single volume before any bytes were actually written.
That tripped RecordAssign's hard-limit path and eagerly removed
volumes from writable, causing the master to grow new volumes
even when the real data being written was tiny.

Drop the pool entirely. Every chunk upload goes through
UploadWithRetry -> AssignVolume with no ExpectedDataSize hint,
letting the master fall back to the 1 MB default estimate. The
mount->filer grpc connection is already cached in pb.WithGrpcClient
(non-streaming mode), so per-chunk AssignVolume is a unary RPC
over an existing HTTP/2 stream, not a full dial. Path-based
filer.conf storage rules now apply to mount chunk assigns again,
which the pool had to skip.

Also remove the now-unused operation.UploadWithAssignFunc and its
AssignFunc type.

* fix(upload): populate ExpectedDataSize from actual chunk bytes

UploadWithRetry already buffers the full chunk into `data` before
calling AssignVolume, so the real size is known. Previously the
assign request went out with ExpectedDataSize=0, making the master
fall back to the 1 MB DefaultNeedleSizeEstimate per fid — same
over-reservation symptom the pool had, just smaller per call.

Stamp ExpectedDataSize = len(data) before the assign RPC when the
caller hasn't already set it. This covers mount chunk uploads,
filer_copy, filersink, mq/logstore, broker_write, gateway_upload,
and nfs — all the UploadWithRetry paths.

* fix(assign): pass real ExpectedDataSize at every assign call site

After removing the mount fid pool, per-chunk AssignVolume calls went
out with ExpectedDataSize=0, making the master fall back to its 1 MB
DefaultNeedleSizeEstimate. That's still an over-estimate for small
writes. Thread the real payload size through every remaining assign
site so RecordAssign charges effectiveSize accurately and stops
prematurely marking volumes full.

- filer: assignNewFileInfo now takes expectedDataSize and stamps it
  on both primary and alternate VolumeAssignRequests. Callers pass:
  - SSE data-to-chunk: len(data)
  - copy manifest save: len(data)
  - streamCopyChunk: srcChunk.Size
  - TUS sub-chunk: bytes read
  - saveAsChunk (autochunk/manifestize): 0 (small, size unknown
    until the reader is drained; master uses 1 MB default)
- filer gRPC remote fetch-and-write: ExpectedDataSize = chunkSize
  after the adaptive chunkSize is computed.
- ChunkedUploadOption.AssignFunc gains an expectedDataSize parameter;
  upload_chunked.go passes the buffered dataSize at the call site.
  S3 PUT assignFunc stamps it on the AssignVolumeRequest.
- S3 copy: assignNewVolume / prepareChunkCopy take expectedDataSize;
  all seven call sites pass the source chunk's Size.
- operation.SubmitFiles / FilePart.Upload: derive per-fid size from
  FileSize (average for batched requests, real per-chunk size for
  sequential chunk assigns).
- benchmark: pass fileSize.
- filer append-to-file: pass len(data).

* fix(assign): thread size through SaveDataAsChunkFunctionType

The saveAsChunk path (autochunk, filer_copy, webdav, mount) ran
AssignVolume before the reader was drained, so it had to pass
ExpectedDataSize=0 and fall back to the master's 1 MB default.

Add an expectedDataSize parameter to SaveDataAsChunkFunctionType.
- mergeIntoManifest already has the serialized manifest bytes, so
  it passes uint64(len(data)) directly.
- Mount's saveDataAsChunk ignores the parameter because it uses
  UploadWithRetry, which already stamps len(data) on the assign
  after reading the payload.
- webdav and filer_copy saveDataAsChunk follow the same UploadWithRetry
  path and also ignore the hint.
- Filer's saveAsChunk (used for manifestize) plumbs the value to
  assignNewFileInfo so manifest-chunk assigns get a real size.

Callers of saveFunc-as-value (weedfs_file_sync, dirty_pages_chunked)
pass the chunk size they're about to upload.

weed/mount/dirty_pages_chunked.go

@@ -72,7 +72,7 @@ func (pages *ChunkedDirtyPages) saveChunkedFileIntervalToStorage(reader io.Reade
 
 	fileFullPath := pages.fh.FullPath()
 	fileName := fileFullPath.Name()
-	chunk, err := pages.fh.wfs.saveDataAsChunk(fileFullPath)(reader, fileName, offset, modifiedTsNs)
+	chunk, err := pages.fh.wfs.saveDataAsChunk(fileFullPath)(reader, fileName, offset, modifiedTsNs, uint64(size))
 	if err != nil {
 		glog.V(0).Infof("%v saveToStorage [%d,%d): %v", fileFullPath, offset, offset+size, err)
 		pages.lastErr = err

weed/mount/fileids_pool.go

@@ -1,181 +0,0 @@
-package mount
-
-import (
-	"context"
-	"fmt"
-	"sync"
-	"time"
-
-	"github.com/seaweedfs/seaweedfs/weed/glog"
-	"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
-	"github.com/seaweedfs/seaweedfs/weed/security"
-)
-
-// FileIdEntry holds a pre-allocated file ID from the filer/master, ready for
-// immediate use by an upload worker without an AssignVolume round-trip.
-type FileIdEntry struct {
-	FileId string
-	Host   string // volume server address (already adjusted for access mode)
-	Auth   security.EncodedJwt
-	Time   time.Time
-}
-
-// FileIdPool pre-allocates file IDs in batches so that chunk uploads can grab
-// one instantly instead of blocking on an AssignVolume RPC per chunk.
-//
-// The pool is refilled in the background when it drops below a low-water mark.
-// All IDs are allocated with the mount's global (replication, collection, ttl,
-// diskType, dataCenter) parameters. Path-based storage rules (filer.conf) are
-// NOT applied to pooled IDs since the pool allocates ahead of any specific file
-// path. This is an intentional tradeoff for writeback cache performance.
-type FileIdPool struct {
-	wfs *WFS
-
-	mu       sync.Mutex
-	cond     *sync.Cond
-	entries  []FileIdEntry // available pre-allocated IDs
-	filling  bool          // true when a background refill is in progress
-
-	poolSize  int // target pool capacity
-	batchSize int // how many IDs to request per Assign RPC
-	lowWater  int // refill trigger threshold
-	maxAge    time.Duration
-}
-
-func NewFileIdPool(wfs *WFS) *FileIdPool {
-	concurrency := wfs.option.ConcurrentWriters
-	if concurrency <= 0 {
-		concurrency = 128 // match default async flush worker count
-	}
-	pool := &FileIdPool{
-		wfs:       wfs,
-		poolSize:  concurrency * 2,
-		batchSize: concurrency,
-		lowWater:  concurrency,
-		maxAge:    25 * time.Second, // conservative; JWT TTL is typically 30s+
-	}
-	pool.cond = sync.NewCond(&pool.mu)
-	return pool
-}
-
-// Get returns a pre-allocated file ID entry. If the pool is empty and a refill
-// is in progress, callers wait for it to complete rather than failing. Returns
-// an error only if the Assign RPC fails.
-func (p *FileIdPool) Get() (FileIdEntry, error) {
-	p.mu.Lock()
-	defer p.mu.Unlock()
-
-	for {
-		p.evictExpired()
-
-		if len(p.entries) > 0 {
-			entry := p.entries[0]
-			p.entries = p.entries[1:]
-			if len(p.entries) < p.lowWater && !p.filling {
-				p.filling = true
-				go p.doRefill()
-			}
-			return entry, nil
-		}
-
-		// Pool empty.
-		if p.filling {
-			// Wait for the in-flight refill to complete.
-			p.cond.Wait()
-			continue
-		}
-
-		// No refill in progress — start one synchronously.
-		p.filling = true
-		p.mu.Unlock()
-		entries, err := p.assignBatch(p.batchSize)
-		p.mu.Lock()
-		p.filling = false
-		p.cond.Broadcast()
-
-		if err != nil {
-			return FileIdEntry{}, fmt.Errorf("fileIdPool: %w", err)
-		}
-		p.entries = append(p.entries, entries...)
-		// Loop back to pop from entries.
-	}
-}
-
-func (p *FileIdPool) evictExpired() {
-	cutoff := time.Now().Add(-p.maxAge)
-	i := 0
-	for i < len(p.entries) && p.entries[i].Time.Before(cutoff) {
-		i++
-	}
-	if i > 0 {
-		p.entries = p.entries[i:]
-	}
-}
-
-// doRefill runs in a background goroutine to refill the pool.
-func (p *FileIdPool) doRefill() {
-	entries, err := p.assignBatch(p.batchSize)
-	if err != nil {
-		glog.V(1).Infof("fileIdPool refill: %v", err)
-	}
-
-	p.mu.Lock()
-	if err == nil {
-		p.entries = append(p.entries, entries...)
-	}
-	p.filling = false
-	p.cond.Broadcast()
-	p.mu.Unlock()
-}
-
-// assignBatch requests `count` file IDs from the filer using individual
-// Count=1 RPCs over a single gRPC connection. Each response includes a
-// per-fid JWT, so uploads work correctly when JWT security is enabled.
-//
-// We use individual requests instead of Count=N because the master generates
-// one JWT for the base file ID only (master_grpc_server_assign.go:158), and
-// the volume server validates that the JWT's Fid matches the upload's file ID
-// exactly (volume_server_handlers.go:367). Sequential IDs derived from a
-// Count=N response would fail this check.
-//
-// Note: the AssignVolumeRequest intentionally omits the Path field. Pooled IDs
-// use the mount's global storage parameters, not per-path rules from filer.conf
-// (detectStorageOption / MatchStorageRule). This is a writeback cache tradeoff.
-func (p *FileIdPool) assignBatch(count int) ([]FileIdEntry, error) {
-	var entries []FileIdEntry
-	err := p.wfs.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
-		now := time.Now()
-		req := &filer_pb.AssignVolumeRequest{
-			Count:            1,
-			Replication:      p.wfs.option.Replication,
-			Collection:       p.wfs.option.Collection,
-			TtlSec:           p.wfs.option.TtlSec,
-			DiskType:         string(p.wfs.option.DiskType),
-			DataCenter:       p.wfs.option.DataCenter,
-			ExpectedDataSize: uint64(p.wfs.option.ChunkSizeLimit),
-		}
-		for i := 0; i < count; i++ {
-			resp, assignErr := client.AssignVolume(context.Background(), req)
-			if assignErr != nil {
-				if len(entries) > 0 {
-					break // partial batch is fine
-				}
-				return assignErr
-			}
-			if resp.Error != "" {
-				if len(entries) > 0 {
-					break
-				}
-				return fmt.Errorf("assign: %s", resp.Error)
-			}
-			entries = append(entries, FileIdEntry{
-				FileId: resp.FileId,
-				Host:   p.wfs.AdjustedUrl(resp.Location),
-				Auth:   security.EncodedJwt(resp.Auth),
-				Time:   now,
-			})
-		}
-		return nil
-	})
-	return entries, err
-}

weed/mount/fileids_pool_test.go

@@ -1,280 +0,0 @@
-package mount
-
-import (
-	"context"
-	"fmt"
-	"sync"
-	"sync/atomic"
-	"testing"
-	"time"
-
-	"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
-	"github.com/seaweedfs/seaweedfs/weed/security"
-	"github.com/seaweedfs/seaweedfs/weed/storage/needle"
-	"google.golang.org/grpc"
-)
-
-// mockFilerClient simulates AssignVolume RPCs with configurable latency.
-type mockFilerClient struct {
-	latency   time.Duration
-	nextVolId uint32
-	nextKey   uint64
-	mu        sync.Mutex
-}
-
-func (m *mockFilerClient) AssignVolume(_ context.Context, req *filer_pb.AssignVolumeRequest, _ ...grpc.CallOption) (*filer_pb.AssignVolumeResponse, error) {
-	if m.latency > 0 {
-		time.Sleep(m.latency)
-	}
-	m.mu.Lock()
-	defer m.mu.Unlock()
-
-	count := int(req.Count)
-	if count <= 0 {
-		count = 1
-	}
-	vid := needle.VolumeId(m.nextVolId + 1)
-	key := m.nextKey + 1000 // start at a non-zero key for valid hex encoding
-	m.nextKey += uint64(count)
-	m.nextVolId++
-
-	fid := needle.NewFileId(vid, key, 0x12345678)
-	return &filer_pb.AssignVolumeResponse{
-		FileId: fid.String(),
-		Count:  int32(count),
-		Auth:   "test-jwt-token",
-		Location: &filer_pb.Location{
-			Url:       "127.0.0.1:8080",
-			PublicUrl: "127.0.0.1:8080",
-			GrpcPort:  18080,
-		},
-	}, nil
-}
-
-// TestFileIdPoolSequentialIds verifies that batch assignment generates
-// correct sequential file IDs from a single AssignVolume response.
-func TestFileIdPoolSequentialIds(t *testing.T) {
-	mock := &mockFilerClient{}
-	resp, err := mock.AssignVolume(context.Background(), &filer_pb.AssignVolumeRequest{Count: 5})
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	baseFid, parseErr := needle.ParseFileIdFromString(resp.FileId)
-	if parseErr != nil {
-		t.Fatal(parseErr)
-	}
-
-	baseKey := uint64(baseFid.Key)
-	for i := 0; i < int(resp.Count); i++ {
-		fid := needle.NewFileId(baseFid.VolumeId, baseKey+uint64(i), uint32(baseFid.Cookie))
-		t.Logf("ID %d: %s", i, fid.String())
-		parsed, err := needle.ParseFileIdFromString(fid.String())
-		if err != nil {
-			t.Fatalf("Failed to parse sequential ID %d: %v", i, err)
-		}
-		if uint64(parsed.Key) != baseKey+uint64(i) {
-			t.Fatalf("ID %d: expected key %d, got %d", i, baseKey+uint64(i), uint64(parsed.Key))
-		}
-	}
-}
-
-// TestFileIdPoolExpiry verifies that expired entries are evicted.
-func TestFileIdPoolExpiry(t *testing.T) {
-	pool := &FileIdPool{
-		maxAge: time.Second,
-	}
-	pool.cond = sync.NewCond(&pool.mu)
-	now := time.Now()
-	pool.entries = []FileIdEntry{
-		{FileId: "1,old", Time: now.Add(-2 * time.Second)},
-		{FileId: "2,old", Time: now.Add(-2 * time.Second)},
-		{FileId: "3,fresh", Time: now},
-	}
-	pool.evictExpired()
-	if len(pool.entries) != 1 {
-		t.Fatalf("expected 1 entry after eviction, got %d", len(pool.entries))
-	}
-	if pool.entries[0].FileId != "3,fresh" {
-		t.Fatalf("expected fresh entry, got %s", pool.entries[0].FileId)
-	}
-}
-
-// TestFileIdPoolGetWaitsForRefill verifies that concurrent Get() calls wait
-// for an in-flight refill instead of returning an error.
-func TestFileIdPoolGetWaitsForRefill(t *testing.T) {
-	pool := &FileIdPool{
-		poolSize:  10,
-		batchSize: 5,
-		lowWater:  3,
-		maxAge:    30 * time.Second,
-	}
-	pool.cond = sync.NewCond(&pool.mu)
-
-	// Simulate a slow refill in progress.
-	pool.filling = true
-	done := make(chan struct{})
-	go func() {
-		// After a short delay, deliver entries and signal.
-		time.Sleep(10 * time.Millisecond)
-		pool.mu.Lock()
-		now := time.Now()
-		for i := 0; i < 5; i++ {
-			pool.entries = append(pool.entries, FileIdEntry{
-				FileId: fmt.Sprintf("1,%x12345678", 1000+i),
-				Host:   "127.0.0.1:8080",
-				Auth:   "jwt",
-				Time:   now,
-			})
-		}
-		pool.filling = false
-		pool.cond.Broadcast()
-		pool.mu.Unlock()
-		close(done)
-	}()
-
-	// Get() should wait for the refill, not return an error.
-	entry, err := pool.Get()
-	if err != nil {
-		t.Fatalf("Get() returned error while refill in progress: %v", err)
-	}
-	if entry.FileId == "" {
-		t.Fatal("Get() returned empty entry")
-	}
-	<-done
-}
-
-// BenchmarkPoolGetVsDirectAssign measures the latency difference between
-// getting a file ID from the pool vs a direct (simulated) AssignVolume RPC.
-func BenchmarkPoolGetVsDirectAssign(b *testing.B) {
-	assignLatencies := []time.Duration{
-		0,
-		100 * time.Microsecond,
-		1 * time.Millisecond,
-		5 * time.Millisecond,
-	}
-
-	for _, latency := range assignLatencies {
-		name := "latency=" + latency.String()
-
-		b.Run("DirectAssign/"+name, func(b *testing.B) {
-			mock := &mockFilerClient{latency: latency}
-			b.ResetTimer()
-			for i := 0; i < b.N; i++ {
-				_, err := mock.AssignVolume(context.Background(), &filer_pb.AssignVolumeRequest{Count: 1})
-				if err != nil {
-					b.Fatal(err)
-				}
-			}
-		})
-
-		b.Run("PoolGet/"+name, func(b *testing.B) {
-			pool := &FileIdPool{
-				maxAge:   30 * time.Second,
-				lowWater: 1000000, // disable background refill
-			}
-			pool.cond = sync.NewCond(&pool.mu)
-			// Pre-fill with a fixed-size pool; refill under StopTimer when depleted.
-			const preload = 4096
-			refill := func() {
-				now := time.Now()
-				for i := 0; i < preload; i++ {
-					pool.entries = append(pool.entries, FileIdEntry{
-						FileId: fmt.Sprintf("1,%x12345678", i),
-						Host:   "127.0.0.1:8080",
-						Auth:   security.EncodedJwt("test-jwt"),
-						Time:   now,
-					})
-				}
-			}
-			refill()
-			b.ResetTimer()
-			for i := 0; i < b.N; i++ {
-				pool.mu.Lock()
-				if len(pool.entries) == 0 {
-					pool.mu.Unlock()
-					b.StopTimer()
-					refill()
-					b.StartTimer()
-					pool.mu.Lock()
-				}
-				_ = pool.entries[0]
-				pool.entries = pool.entries[1:]
-				pool.mu.Unlock()
-			}
-		})
-	}
-}
-
-// BenchmarkConcurrentPoolGet measures pool throughput under concurrent access.
-func BenchmarkConcurrentPoolGet(b *testing.B) {
-	for _, workers := range []int{1, 4, 16, 64} {
-		b.Run(fmt.Sprintf("workers=%d", workers), func(b *testing.B) {
-			pool := &FileIdPool{
-				maxAge:   30 * time.Second,
-				lowWater: 1000000,
-			}
-			pool.cond = sync.NewCond(&pool.mu)
-			// Pre-fill
-			now := time.Now()
-			total := b.N*workers + 1000
-			pool.entries = make([]FileIdEntry, total)
-			for i := range pool.entries {
-				pool.entries[i] = FileIdEntry{
-					FileId: fmt.Sprintf("1,%x12345678", i+1000),
-					Host:   "127.0.0.1:8080",
-					Auth:   security.EncodedJwt("test-jwt"),
-					Time:   now,
-				}
-			}
-
-			var ops atomic.Int64
-			b.ResetTimer()
-
-			var wg sync.WaitGroup
-			perWorker := b.N
-			for w := 0; w < workers; w++ {
-				wg.Add(1)
-				go func() {
-					defer wg.Done()
-					for i := 0; i < perWorker; i++ {
-						pool.mu.Lock()
-						if len(pool.entries) > 0 {
-							_ = pool.entries[0]
-							pool.entries = pool.entries[1:]
-							ops.Add(1)
-						}
-						pool.mu.Unlock()
-					}
-				}()
-			}
-			wg.Wait()
-			b.ReportMetric(float64(ops.Load())/b.Elapsed().Seconds(), "ids/sec")
-		})
-	}
-}
-
-// BenchmarkBatchAssign measures the cost of batch vs individual assign RPCs.
-func BenchmarkBatchAssign(b *testing.B) {
-	for _, batchSize := range []int{1, 8, 16, 32} {
-		b.Run(fmt.Sprintf("batch=%d", batchSize), func(b *testing.B) {
-			mock := &mockFilerClient{latency: 1 * time.Millisecond}
-			totalIds := 0
-			b.ResetTimer()
-			for i := 0; i < b.N; i++ {
-				resp, err := mock.AssignVolume(context.Background(), &filer_pb.AssignVolumeRequest{Count: int32(batchSize)})
-				if err != nil {
-					b.Fatal(err)
-				}
-
-				baseFid, _ := needle.ParseFileIdFromString(resp.FileId)
-				baseKey := uint64(baseFid.Key)
-				for j := 0; j < int(resp.Count); j++ {
-					_ = needle.NewFileId(baseFid.VolumeId, baseKey+uint64(j), uint32(baseFid.Cookie)).String()
-					totalIds++
-				}
-			}
-			b.ReportMetric(float64(totalIds)/b.Elapsed().Seconds(), "ids/sec")
-		})
-	}
-}

weed/mount/weedfs.go

@@ -142,7 +142,6 @@ type WFS struct {
 	dirHotWindow         time.Duration
 	dirHotThreshold      int
 	dirIdleEvict         time.Duration
-	fileIdPool           *FileIdPool
 
 	// openMtimeCache maps inode -> [mtime_sec, mtime_ns] from the last Open.
 	// Used to decide whether to set FOPEN_KEEP_CACHE on subsequent opens.
@@ -387,11 +386,6 @@ func (wfs *WFS) StartBackgroundTasks() error {
 	go wfs.loopEvictIdleDirCache()
 	go wfs.loopProactiveFlush()
 
-	if wfs.option.WritebackCache {
-		wfs.fileIdPool = NewFileIdPool(wfs)
-		glog.V(0).Infof("file ID pool enabled for writeback cache (batch=%d)", wfs.fileIdPool.batchSize)
-	}
-
 	return nil
 }
 

weed/mount/weedfs_file_sync.go

@@ -315,7 +315,7 @@ func (wfs *WFS) maybeMergeChunks(fileFullPath util.FullPath, compactedChunks []*
 	for {
 		n, readErr := io.ReadFull(reader, buf)
 		if n > 0 {
-			chunk, uploadErr := saveFunc(bytes.NewReader(buf[:n]), "", offset, 0)
+			chunk, uploadErr := saveFunc(bytes.NewReader(buf[:n]), "", offset, 0, uint64(n))
 			if uploadErr != nil {
 				return nil, uploadErr
 			}

weed/mount/weedfs_write.go

@@ -8,13 +8,12 @@ import (
 	"github.com/seaweedfs/seaweedfs/weed/glog"
 	"github.com/seaweedfs/seaweedfs/weed/operation"
 	"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
-	"github.com/seaweedfs/seaweedfs/weed/security"
 	"github.com/seaweedfs/seaweedfs/weed/util"
 )
 
 func (wfs *WFS) saveDataAsChunk(fullPath util.FullPath) filer.SaveDataAsChunkFunctionType {
 
-	return func(reader io.Reader, filename string, offset int64, tsNs int64) (chunk *filer_pb.FileChunk, err error) {
+	return func(reader io.Reader, filename string, offset int64, tsNs int64, _ uint64) (chunk *filer_pb.FileChunk, err error) {
 		uploader, err := operation.NewUploader()
 		if err != nil {
 			return
@@ -35,37 +34,19 @@ func (wfs *WFS) saveDataAsChunk(fullPath util.FullPath) filer.SaveDataAsChunkFun
 			return fileUrl
 		}
 
-		var fileId string
-		var uploadResult *operation.UploadResult
-		var data []byte
-
-		if wfs.fileIdPool != nil {
-			// Use pre-allocated file ID from pool — avoids AssignVolume RPC.
-			fileId, uploadResult, err, data = uploader.UploadWithAssignFunc(
-				func() (string, string, security.EncodedJwt, error) {
-					entry, getErr := wfs.fileIdPool.Get()
-					if getErr != nil {
-						return "", "", "", getErr
-					}
-					return entry.FileId, entry.Host, entry.Auth, nil
-				},
-				uploadOption, genFileUrlFn, reader,
-			)
-		} else {
-			fileId, uploadResult, err, data = uploader.UploadWithRetry(
-				wfs,
-				&filer_pb.AssignVolumeRequest{
-					Count:       1,
-					Replication: wfs.option.Replication,
-					Collection:  wfs.option.Collection,
-					TtlSec:     wfs.option.TtlSec,
-					DiskType:    string(wfs.option.DiskType),
-					DataCenter:  wfs.option.DataCenter,
-					Path:        string(fullPath),
-				},
-				uploadOption, genFileUrlFn, reader,
-			)
-		}
+		fileId, uploadResult, err, data := uploader.UploadWithRetry(
+			wfs,
+			&filer_pb.AssignVolumeRequest{
+				Count:       1,
+				Replication: wfs.option.Replication,
+				Collection:  wfs.option.Collection,
+				TtlSec:      wfs.option.TtlSec,
+				DiskType:    string(wfs.option.DiskType),
+				DataCenter:  wfs.option.DataCenter,
+				Path:        string(fullPath),
+			},
+			uploadOption, genFileUrlFn, reader,
+		)
 
 		if err != nil {
 			glog.V(0).Infof("upload data %v: %v", filename, err)