mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-19 10:54:08 +01:00
* mount: move an open file's read view to the chunks a merge wrote When a flush finds that a file's chunks take more than twice its size, maybeMergeChunks reads the file back and writes it again as new chunks. The commit that publishes them makes the filer delete the chunks they replaced. The flush put the new chunks into the handle's entry but left its read view on the old ones, so every descriptor that kept the file open went on reading the replaced chunks. Those read back until their volume is vacuumed. After that a read first retries the fetch of the missing needle for about 13 s; then refreshEntryForRead installs the filer's entry and the read is retried. The refresh skips a handle with writes it has not flushed, and such a read returns EIO instead. Files that are kept open and rewritten in place, such as PostgreSQL data files and SQLite databases, are the ones whose chunks bloat, so they are the ones that hit this. installRewrittenChunks puts the merge's chunks into the entry and the read view together, under the entry lock. Chunks attached while the merge ran were not part of it and stay. Flushes without a merge are unchanged. TestMergeMovesReadViewOffReplacedChunks covers the handle. TestReadOpenHandleAfterChunkMerge runs it on a real mount, with and without an unflushed write; for it, the fuse_integration framework starts mini on a free master port and exposes the address. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: drain uploads before a merge replaces the chunks Uploads attach their chunks without the handle lock, first to the entry and then to the read view, and writes that landed after doFlush drained the pages may still be uploading when the flush takes its lock. An upload that attached during the merge could keep a chunk older than data the merge already holds, or put a chunk the merge replaced back into the read view. Either outranks the merge's zero-timestamp chunk. When the chunks are bloated enough to merge, the flush now drains the pages again while its lock holds writes off, and takes the chunks again. FlushAll returns only after every upload has attached to both the entry and the read view, so nothing attaches during the merge. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: leave a pre-merge upload failure to the next flush The drain before a merge returned an upload failure through the metadata flush. doFlush then retried it as a filer error and answered EIO, where the page flush answers a full disk with ENOSPC. The drain is there only to wait for the uploads: a failure stays with the pages, and the next flush reports it as a write error, as it would without a merge. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: keep chunks attached while a flush installs its compacted list A write that lands after doFlush drained the pages can still be uploading when flushMetadataToFiler takes its chunk snapshot. Its upload attaches the chunk without the handle lock, and the plain flush then replaced the entry's chunks with the list compacted from the snapshot, dropping it from the handle and from every later commit. Keep chunks attached after the snapshot, as the merge path already does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: fail a merge's flush on a failed drain A failed upload found by the drain before a merge was only logged, and the flush went on to publish a size covering bytes that no chunk stores. It now fails the flush as a pageUploadError before anything is published; doFlush maps it with writeErrorToFuseStatus, as it does its own FlushData failure, and does not retry it as a filer error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: clone a flush's request entry under the entry lock The request snapshot cloned the live entry after GetEntry had released its read lock. Uploads attach chunks without the handle lock, so an AppendChunks that landed during the clone raced it on the Chunks field. Clone inside UpdateEntry, as content flush recovery does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: leave a merge to a later flush while uploads are pending Before a merge the flush drained the pages while it held the handle and publication locks. A writeback upload retries a transient storage error until it succeeds, so that drain could hold both locks for as long as storage stayed down, stalling reads, writes and stat on the handle. The flush now checks for pending uploads and, if any, publishes the compacted chunks and leaves the merge to a later flush. With none pending the lock still holds writes off, so nothing attaches during the merge, and a failed upload still fails the flush as a write error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: keep room for pending uploads when a flush merges onto a moved entry A flush that leaves the chunk merge to a later flush publishes while uploads are still pending, and writes that landed after the drain can be uploading on any flush. Merging a peer's inline content or a shrink into such a publication lets the upload attach its chunk next to the inline bytes, where reads never reach it. Tell createContentEntry when uploads are pending so the merge keeps room for them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: test chunk merges through the filer's chunk proxy, also under load The tests serve the filer's chunk proxy, and like the filer the fake deletes the chunks each commit replaces, so a read view still naming them fails: - TestFlushMergeThroughChunkProxy/Quiet: the flush merges, and reads through the open handle return the file from the merged chunk alone. - TestFlushMergeThroughChunkProxy/UploadInFlight: an older page is still uploading while a newer page of the same bytes has attached. The flush leaves the merge, reads return the newer page, and the next flush merges. - TestFlushMergeRacesWithWritesAndReads: four writers, three flushers and a reader on one handle, with uploads delayed at random. The writers pause now and then, so 14 to 18 flushes a run merge and the others leave the merge to a later one. Every read must succeed within 2 s, and at the end the handle and the filer must both hold exactly what was written. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: let a held upload go when UploadInFlight fails The subtest's proxy holds the older page's upload until the test releases it, and only the passing path did. A failure in between left the handler blocked, so the proxy's cleanup waited on it and the failure surfaced as a suite timeout. Release it from a cleanup as well, once, and bound the waits for the upload and the drains. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
791 lines
30 KiB
Go
791 lines
30 KiB
Go
package mount
|
|
|
|
import (
|
|
"bytes"
|
|
"context"
|
|
"errors"
|
|
"fmt"
|
|
"io"
|
|
"syscall"
|
|
"time"
|
|
|
|
"github.com/seaweedfs/go-fuse/v2/fuse"
|
|
"github.com/seaweedfs/seaweedfs/weed/filer"
|
|
"github.com/seaweedfs/seaweedfs/weed/glog"
|
|
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
|
|
"github.com/seaweedfs/seaweedfs/weed/util"
|
|
"google.golang.org/protobuf/proto"
|
|
)
|
|
|
|
// metadataFlushTimeout bounds the metadata commit of a close()/fsync flush
|
|
// (CreateEntry and its retries) so an overwhelmed filer cannot wedge the
|
|
// calling process forever. It is deliberately generous: a healthy CreateEntry
|
|
// completes in well under a second, so this only fires on a genuinely stuck
|
|
// filer, never on a normal flush. It is the flush's commitBudget, so it does
|
|
// not cover the uploads that come first. A variable so tests can shorten it.
|
|
var metadataFlushTimeout = 30 * time.Second
|
|
|
|
// commitBudget is the time one flush may spend in the filer calls of its
|
|
// metadata commit, added up over its attempts. Nothing else is charged to it:
|
|
// the data upload, the wait for the handle's locks, the merge of bloated
|
|
// chunks and the manifest upload can each outlast the whole budget. A deadline
|
|
// that also ran through that work was spent by it; CreateEntry then failed at
|
|
// once with "context deadline exceeded", and close() returned an error for
|
|
// data that was already stored. A nil budget sets no bound. The attempts run
|
|
// one after another, so it needs no lock.
|
|
type commitBudget struct {
|
|
left time.Duration
|
|
}
|
|
|
|
// begin returns ctx limited to what is left of the budget, for one filer call.
|
|
// The returned func ends the call and charges the budget the time it took.
|
|
func (b *commitBudget) begin(ctx context.Context) (context.Context, func()) {
|
|
if b == nil {
|
|
return ctx, func() {}
|
|
}
|
|
start := time.Now()
|
|
ctx, cancel := context.WithTimeout(ctx, b.left)
|
|
return ctx, func() {
|
|
cancel()
|
|
b.left -= time.Since(start)
|
|
}
|
|
}
|
|
|
|
// spent reports whether the budget has run out, so that another attempt could
|
|
// only fail.
|
|
func (b *commitBudget) spent() bool {
|
|
return b != nil && b.left <= 0
|
|
}
|
|
|
|
/**
|
|
* Flush method
|
|
*
|
|
* This is called on each close() of the opened file.
|
|
*
|
|
* Since file descriptors can be duplicated (dup, dup2, fork), for
|
|
* one open call there may be many flush calls.
|
|
*
|
|
* Filesystems shouldn't assume that flush will always be called
|
|
* after some writes, or that if will be called at all.
|
|
*
|
|
* fi->fh will contain the value set by the open method, or will
|
|
* be undefined if the open method didn't set any value.
|
|
*
|
|
* NOTE: the name of the method is misleading, since (unlike
|
|
* fsync) the filesystem is not forced to flush pending writes.
|
|
* One reason to flush data is if the filesystem wants to return
|
|
* write errors during close. However, such use is non-portable
|
|
* because POSIX does not require [close] to wait for delayed I/O to
|
|
* complete.
|
|
*
|
|
* If the filesystem supports file locking operations (setlk,
|
|
* getlk) it should remove all locks belonging to 'fi->owner'.
|
|
*
|
|
* If this request is answered with an error code of ENOSYS,
|
|
* this is treated as success and future calls to flush() will
|
|
* succeed automatically without being send to the filesystem
|
|
* process.
|
|
*
|
|
* Valid replies:
|
|
* fuse_reply_err
|
|
*
|
|
* @param req request handle
|
|
* @param ino the inode number
|
|
* @param fi file information
|
|
*
|
|
* [close]: http://pubs.opengroup.org/onlinepubs/9699919799/functions/close.html
|
|
*/
|
|
func (wfs *WFS) Flush(cancel <-chan struct{}, in *fuse.FlushIn) fuse.Status {
|
|
fh := wfs.GetHandle(FileHandleId(in.Fh))
|
|
if fh == nil {
|
|
// If handle is not found, it might have been already released
|
|
// This is not an error condition for FLUSH
|
|
if in.LockOwner != 0 && wfs.releasePosixOwner(in.NodeId, in.LockOwner) {
|
|
wfs.forgetLockOwner(in.NodeId, in.LockOwner)
|
|
}
|
|
return fuse.OK
|
|
}
|
|
|
|
// FlushIn.LockOwner is populated by some FUSE kernels even when the process
|
|
// did not hold byte-range locks. Only force the synchronous close path when
|
|
// this owner actually has POSIX locks to release; otherwise writebackCache
|
|
// would silently degrade to a blocking flush for ordinary close().
|
|
hasPosixLocks := wfs.hasPosixOwner(in.NodeId, in.LockOwner)
|
|
allowAsync := !hasPosixLocks
|
|
|
|
// Bound the metadata commit with a deadline instead of tying it to the FUSE
|
|
// cancel channel. A FUSE interrupt is not a process kill: Go's async
|
|
// preemption (SIGURG) makes a close() under load emit an interrupt on nearly
|
|
// every flush (see go-fuse RawFileSystem docs), so cancelling the in-flight
|
|
// metadata CreateEntry on that interrupt turned healthy concurrent close()s
|
|
// into EIO. The deadline still keeps close() from hanging forever against an
|
|
// overwhelmed filer without failing benign flushes. doFlush charges it only
|
|
// for the commit's filer calls, so a slow upload cannot use it up.
|
|
status := wfs.doFlush(context.Background(), fh, in.Uid, in.Gid, allowAsync, metadataFlushTimeout)
|
|
if in.LockOwner != 0 && wfs.releasePosixOwner(in.NodeId, in.LockOwner) {
|
|
wfs.forgetLockOwner(in.NodeId, in.LockOwner)
|
|
}
|
|
return status
|
|
}
|
|
|
|
/**
|
|
* Synchronize file contents
|
|
*
|
|
* If the datasync parameter is non-zero, then only the user data
|
|
* should be flushed, not the meta data.
|
|
*
|
|
* If this request is answered with an error code of ENOSYS,
|
|
* this is treated as success and future calls to fsync() will
|
|
* succeed automatically without being send to the filesystem
|
|
* process.
|
|
*
|
|
* Valid replies:
|
|
* fuse_reply_err
|
|
*
|
|
* @param req request handle
|
|
* @param ino the inode number
|
|
* @param datasync flag indicating if only data should be flushed
|
|
* @param fi file information
|
|
*/
|
|
func (wfs *WFS) Fsync(cancel <-chan struct{}, in *fuse.FsyncIn) (code fuse.Status) {
|
|
|
|
fh := wfs.GetHandle(FileHandleId(in.Fh))
|
|
if fh == nil {
|
|
return fuse.ENOENT
|
|
}
|
|
|
|
// Fsync is an explicit sync request — always flush synchronously
|
|
return wfs.doFlush(context.Background(), fh, in.Uid, in.Gid, false, metadataFlushTimeout)
|
|
|
|
}
|
|
|
|
// doFlush uploads the handle's dirty data, then commits its metadata to the
|
|
// filer. metadataTimeout is the commitBudget of that commit: what its filer
|
|
// calls may take over all attempts, whatever the uploads before and between
|
|
// them took. 0 sets no overall bound, leaving only the per-call filer
|
|
// timeouts. ctx should carry no deadline of its own: it would run through the
|
|
// uploads as well.
|
|
func (wfs *WFS) doFlush(ctx context.Context, fh *FileHandle, uid, gid uint32, allowAsync bool, metadataTimeout time.Duration) fuse.Status {
|
|
|
|
// flush works at fh level
|
|
fileFullPath := fh.FullPath()
|
|
fh.RememberPath(fileFullPath)
|
|
dir, name := fileFullPath.DirAndName()
|
|
// send the data to the OS
|
|
glog.V(4).Infof("doFlush %s fh %d", fileFullPath, fh.fh)
|
|
|
|
// When writebackCache is enabled and this is a close()-triggered Flush (not fsync),
|
|
// defer the expensive data upload + metadata flush to a background goroutine.
|
|
// This allows the calling process (e.g., rsync) to proceed to the next file immediately.
|
|
// POSIX does not require close() to wait for delayed I/O to complete.
|
|
if allowAsync && wfs.option.WritebackCache && fh.dirtyMetadata {
|
|
if wfs.IsOverQuotaWithUncommitted() {
|
|
return fuse.Status(syscall.ENOSPC)
|
|
}
|
|
fh.asyncFlushPending = true
|
|
fh.asyncFlushUid = uid
|
|
fh.asyncFlushGid = gid
|
|
glog.V(3).Infof("doFlush async deferred %s fh %d", fileFullPath, fh.fh)
|
|
return fuse.OK
|
|
}
|
|
|
|
// Synchronous flush path (normal mode, fsync, or no dirty data)
|
|
fh.asyncFlushPending = false
|
|
publication := &contentPublication{failure: fh.writebackContentError, writes: fh.pageWrites.Load()}
|
|
|
|
// Check quota including uncommitted writes for real-time enforcement
|
|
isOverQuota := wfs.IsOverQuotaWithUncommitted()
|
|
if !isOverQuota {
|
|
if err := fh.dirtyPages.FlushData(); err != nil {
|
|
glog.Errorf("%v doFlush: %v", fileFullPath, err)
|
|
return writeErrorToFuseStatus(err)
|
|
}
|
|
}
|
|
|
|
if !fh.dirtyMetadata {
|
|
return fuse.OK
|
|
}
|
|
|
|
// Skip metadata flush if the file was unlinked while open.
|
|
// The filer entry is already gone; flushing would recreate it.
|
|
if fh.isDeleted {
|
|
glog.V(3).Infof("doFlush %s fh %d: file was unlinked, skipping metadata flush", fileFullPath, fh.fh)
|
|
return fuse.OK
|
|
}
|
|
|
|
if isOverQuota {
|
|
return fuse.Status(syscall.ENOSPC)
|
|
}
|
|
|
|
// The attempts share one budget, charged only for their filer calls.
|
|
var commit *commitBudget
|
|
if metadataTimeout > 0 {
|
|
commit = &commitBudget{left: metadataTimeout}
|
|
}
|
|
|
|
if err := retryMetadataFlushIf(ctx, func() error {
|
|
return wfs.flushMetadataToFiler(ctx, fh, dir, name, uid, gid, commit, publication)
|
|
}, func(err error) bool {
|
|
return !commit.spent() && !isPageUploadError(err) && isRetryableFilerError(err)
|
|
}, func(nextAttempt, totalAttempts int, backoff time.Duration, err error) {
|
|
glog.Warningf("%v fh %d flush: retrying metadata flush (attempt %d/%d) after %v: %v",
|
|
fileFullPath, fh.fh, nextAttempt, totalAttempts, backoff, err)
|
|
}); err != nil {
|
|
if fh.dirtyContent {
|
|
publication.retire(err)
|
|
}
|
|
glog.Errorf("%v fh %d flush: %v", fileFullPath, fh.fh, err)
|
|
return flushErrorToFuseStatus(err)
|
|
}
|
|
|
|
if IsDebugFileReadWrite {
|
|
fh.mirrorFile.Sync()
|
|
}
|
|
|
|
return fuse.OK
|
|
}
|
|
|
|
// pageUploadError is a page upload failure a metadata flush found before it
|
|
// published anything. doFlush reports it as a write error, as it does when its
|
|
// own FlushData fails, instead of retrying it as a filer error.
|
|
type pageUploadError struct{ err error }
|
|
|
|
func (e *pageUploadError) Error() string { return e.err.Error() }
|
|
func (e *pageUploadError) Unwrap() error { return e.err }
|
|
|
|
func isPageUploadError(err error) bool {
|
|
var upload *pageUploadError
|
|
return errors.As(err, &upload)
|
|
}
|
|
|
|
// flushErrorToFuseStatus maps a failed metadata flush to a FUSE status.
|
|
func flushErrorToFuseStatus(err error) fuse.Status {
|
|
var upload *pageUploadError
|
|
if errors.As(err, &upload) {
|
|
return writeErrorToFuseStatus(upload.err)
|
|
}
|
|
return grpcErrorToFuseStatus(err)
|
|
}
|
|
|
|
// flushMetadataToFiler sends the file's chunk references and attributes to the filer.
|
|
// This is shared between the synchronous doFlush path and the async flush completion.
|
|
//
|
|
// When -dlm is enabled, the distributed lock is already held by the FileHandle
|
|
// from open-for-write through close, so no additional distributed lock is
|
|
// needed here. The local fhLockTable lock below serializes within this mount.
|
|
//
|
|
// commit bounds the filer calls made here, over all attempts of one flush; nil
|
|
// leaves them to the per-call filer timeouts.
|
|
func (wfs *WFS) flushMetadataToFiler(ctx context.Context, fh *FileHandle, dir, name string, uid, gid uint32, commit *commitBudget, publications ...*contentPublication) error {
|
|
publication := &contentPublication{writes: fh.pageWrites.Load()}
|
|
if len(publications) > 0 {
|
|
publication = publications[0]
|
|
}
|
|
fileFullPath, dir, name := wfs.flushTarget(fh, dir, name)
|
|
var unlock func()
|
|
var fhActiveLock *util.ActiveLock
|
|
// An unlink or rename can land while the flush waits for its locks.
|
|
for attempt := 0; ; attempt++ {
|
|
// Attribute acknowledgement and local retention must complete before a
|
|
// whole-entry flush can publish. Take this before the handle's active lock.
|
|
unlock = wfs.lockMetadataPublication(fileFullPath)
|
|
fhActiveLock = fh.wfs.fhLockTable.AcquireLock("doFlush", fh.fh, util.ExclusiveLock)
|
|
nowPath, nowDir, nowName := wfs.flushTarget(fh, dir, name)
|
|
if nowPath == fileFullPath && nowDir == dir && nowName == name {
|
|
break
|
|
}
|
|
fh.wfs.fhLockTable.ReleaseLock(fh.fh, fhActiveLock)
|
|
unlock()
|
|
if attempt == 3 {
|
|
return fmt.Errorf("flush %s: name changed while waiting for its locks, now %s", fileFullPath, nowPath)
|
|
}
|
|
fileFullPath, dir, name = nowPath, nowDir, nowName
|
|
}
|
|
defer func() {
|
|
fh.wfs.fhLockTable.ReleaseLock(fh.fh, fhActiveLock)
|
|
unlock()
|
|
}()
|
|
|
|
glog.V(4).Infof("flushMetadataToFiler %s/%s inode %d fh %d", dir, name, fh.inode, fh.fh)
|
|
|
|
// Re-check under the lock: Unlink sets the flag under it, so a flush that
|
|
// was already past the earlier check cannot write the entry back after
|
|
// the delete removed it.
|
|
if fh.isDeleted {
|
|
glog.V(3).Infof("flushMetadataToFiler %s fh %d: file was unlinked, skipping", fileFullPath, fh.fh)
|
|
return nil
|
|
}
|
|
if fh.linkRefresh != nil {
|
|
linkCtx, end := commit.begin(ctx)
|
|
err := wfs.adoptStoredLink(linkCtx, fh, util.NewFullPath(dir, name), fh.linkRefresh)
|
|
end()
|
|
if err != nil {
|
|
return err
|
|
}
|
|
}
|
|
|
|
entry := fh.GetEntry()
|
|
entry.Name = name // this flush may be just after a rename operation
|
|
|
|
if entry.Attributes != nil {
|
|
entry.Attributes.Mime = fh.contentType
|
|
if entry.Attributes.Uid == 0 {
|
|
entry.Attributes.Uid = uid
|
|
}
|
|
if entry.Attributes.Gid == 0 {
|
|
entry.Attributes.Gid = gid
|
|
}
|
|
// Do not stamp mtime/ctime here. Write/SetAttr already maintain
|
|
// them on the entry; overwriting at flush time clobbered user-set
|
|
// mtime (utimes/touch -m -d) once the deferred flush ran.
|
|
}
|
|
|
|
glog.V(4).Infof("%s set chunks: %v", fileFullPath, len(entry.GetChunks()))
|
|
|
|
snapshot := entry.GetChunks()
|
|
manifestChunks, nonManifestChunks := filer.SeparateManifestChunks(snapshot)
|
|
|
|
chunks, _, err := filer.CompactFileChunks(context.Background(), wfs.LookupFn(), nonManifestChunks)
|
|
if err != nil {
|
|
return fmt.Errorf("compact chunks of %s: %w", fileFullPath, err)
|
|
}
|
|
|
|
deferMerge := false
|
|
if _, _, bloated := shouldMergeChunks(chunks, manifestChunks); bloated {
|
|
// The merge replaces every chunk it reads. Uploads attach their chunks
|
|
// without the handle lock, and writes that landed after doFlush drained
|
|
// the pages may still be uploading. A writeback upload retries until
|
|
// storage recovers, so leave the merge to a later flush rather than
|
|
// wait for it under these locks. With none pending, the lock holds
|
|
// writes off and nothing attaches during the merge: take the chunks
|
|
// again. A failed upload fails the flush as a write error, before
|
|
// anything claims the bytes it did not store.
|
|
if fh.dirtyPages.HasPendingUploads() {
|
|
deferMerge = true
|
|
} else {
|
|
if err := fh.dirtyPages.FlushData(); err != nil {
|
|
return &pageUploadError{err: err}
|
|
}
|
|
snapshot = entry.GetChunks()
|
|
manifestChunks, nonManifestChunks = filer.SeparateManifestChunks(snapshot)
|
|
if chunks, _, err = filer.CompactFileChunks(context.Background(), wfs.LookupFn(), nonManifestChunks); err != nil {
|
|
return fmt.Errorf("compact chunks of %s: %w", fileFullPath, err)
|
|
}
|
|
}
|
|
}
|
|
|
|
rewrote := false
|
|
if deferMerge {
|
|
glog.V(3).Infof("%s: uploads pending, leaving the chunk merge to a later flush", fileFullPath)
|
|
} else if mergedChunks, mergeErr := wfs.maybeMergeChunks(fileFullPath, chunks, manifestChunks); mergeErr != nil {
|
|
glog.V(0).Infof("maybeMergeChunks %s: %v", fileFullPath, mergeErr)
|
|
} else if mergedChunks != nil {
|
|
chunks = mergedChunks
|
|
manifestChunks = nil
|
|
fh.noteRewrite()
|
|
rewrote = true
|
|
}
|
|
|
|
// no chunk deleter here: a failed fold reports the blobs it saved
|
|
chunks, manifestErr := filer.MaybeManifestize(wfs.saveDataAsChunk(fileFullPath), nil, chunks)
|
|
if manifestErr != nil {
|
|
// not good, but should be ok
|
|
glog.V(0).Infof("MaybeManifestize: %v", manifestErr)
|
|
}
|
|
if rewrote {
|
|
fh.installRewrittenChunks(snapshot, append(chunks, manifestChunks...))
|
|
} else {
|
|
fh.UpdateEntry(func(e *filer_pb.Entry) {
|
|
e.Chunks = withChunksAttachedSince(snapshot, append(chunks, manifestChunks...), e.Chunks)
|
|
})
|
|
}
|
|
|
|
// Clone the proto entry for the filer request so that mapPbIdFromLocalToFiler
|
|
// does not mutate the file handle's live entry. Without the clone, a concurrent
|
|
// Lookup can observe filer-side uid/gid on the file handle entry and return it
|
|
// to the kernel, which caches it and then rejects opens by the local user.
|
|
// Uploads attach chunks without the handle lock, so clone under the entry lock.
|
|
var requestEntry *filer_pb.Entry
|
|
fh.UpdateEntry(func(e *filer_pb.Entry) { requestEntry = proto.Clone(e).(*filer_pb.Entry) })
|
|
request := &filer_pb.CreateEntryRequest{
|
|
Directory: string(dir),
|
|
Entry: requestEntry,
|
|
Signatures: []int32{wfs.signature},
|
|
SkipCheckParentDirectory: true,
|
|
}
|
|
|
|
// Snapshot with local ids before the request mapping mutates the clone:
|
|
// on ack this becomes the handle's base, judged against future events.
|
|
baseSnapshot := proto.Clone(requestEntry).(*filer_pb.Entry)
|
|
wfs.mapPbIdFromLocalToFiler(request.Entry)
|
|
|
|
// The chunk merge and the manifest above read and upload chunks, which can
|
|
// take longer than the whole commit budget. It is charged only from here.
|
|
// Writes that landed before the lock may still be uploading, so a merge onto a
|
|
// moved entry must keep room for them.
|
|
buffered := fh.dirtyPages.HasPendingUploads()
|
|
commitCtx, end := commit.begin(ctx)
|
|
resp, merged, err := wfs.createContentEntry(commitCtx, fh, request, baseSnapshot, buffered, publication)
|
|
end()
|
|
if err != nil {
|
|
glog.Errorf("fh flush create %s: %v", fileFullPath, err)
|
|
return &contentPublicationError{cause: fmt.Errorf("fh flush create %s: %w", fileFullPath, err), request: proto.Clone(request).(*filer_pb.CreateEntryRequest)}
|
|
}
|
|
if merged != nil {
|
|
fh.rebaseEntry(merged, requestEntry)
|
|
baseSnapshot = merged
|
|
}
|
|
|
|
event := resp.GetMetadataEvent()
|
|
if event == nil {
|
|
event = metadataUpdateEvent(string(dir), request.Entry)
|
|
if event != nil {
|
|
event.TsNs = ackVersionTsNs(resp)
|
|
}
|
|
}
|
|
// The filer acknowledged this state at the event's log position (or, for
|
|
// a no-op create, at the response's log position); older queued
|
|
// subscription events must not roll the handle back.
|
|
fh.setAuthoritativeBase(wfs.ackedBase(resp, baseSnapshot))
|
|
fh.advanceEntryVersion(ackVersionTsNs(resp), resp.GetLogSignature())
|
|
if applyErr := wfs.applyLocalMetadataEvent(context.Background(), event); applyErr != nil {
|
|
glog.Warningf("flush %s: best-effort metadata apply failed: %v", fileFullPath, applyErr)
|
|
wfs.inodeToPath.InvalidateChildrenCache(util.FullPath(dir))
|
|
}
|
|
|
|
if err == nil && fh.pageWrites.Load() == publication.writes {
|
|
fh.pendingAppend = false
|
|
fh.dirtyMetadata = false
|
|
fh.dirtyContent = false
|
|
}
|
|
|
|
return err
|
|
}
|
|
|
|
// flushTarget returns the lock path and the name a flush of fh publishes
|
|
// through: the caller's name while it still names the file, else the name the
|
|
// file has now.
|
|
func (wfs *WFS) flushTarget(fh *FileHandle, dir, name string) (util.FullPath, string, string) {
|
|
fileFullPath := fh.FullPath()
|
|
if resolved := util.FullPath(dir).Child(name); fileFullPath != "" && fileFullPath != resolved {
|
|
if inode, found := wfs.inodeToPath.GetInode(resolved); !found || inode != fh.inode {
|
|
dir, name = fileFullPath.DirAndName()
|
|
}
|
|
}
|
|
return fileFullPath, dir, name
|
|
}
|
|
|
|
// createContentEntry publishes a handle's whole entry only over the filer
|
|
// state the handle observed. Deferred creates expect no entry; existing
|
|
// handles compare the full base, including inode identity. The unary call
|
|
// keeps a refused condition apart from an I/O failure; the mutation stream
|
|
// reduces it to EIO.
|
|
//
|
|
// When the stored entry moved on, the handle's own changes since its base are
|
|
// replayed onto it and published against it, so a peer's acknowledged chunks
|
|
// survive. merged is that entry in local ids; nil when the request went as is.
|
|
// With writes still buffered, the merge must keep room for them.
|
|
func (wfs *WFS) createContentEntry(ctx context.Context, fh *FileHandle, request *filer_pb.CreateEntryRequest, local *filer_pb.Entry, buffered bool, publication *contentPublication) (resp *filer_pb.CreateEntryResponse, merged *filer_pb.Entry, err error) {
|
|
base := fh.baseEntry.Load()
|
|
guard := &filer_pb.WriteCondition_Clause{Kind: filer_pb.WriteCondition_IF_NOT_EXISTS}
|
|
if base != nil {
|
|
expected := proto.Clone(base).(*filer_pb.Entry)
|
|
wfs.mapPbIdFromLocalToFiler(expected)
|
|
guard = &filer_pb.WriteCondition_Clause{Kind: filer_pb.WriteCondition_IF_ENTRY_EQUAL, ExpectedEntry: expected}
|
|
}
|
|
sent := request.Entry
|
|
for attempt := 0; ; attempt++ {
|
|
if guardErr := wfs.checkContentPublication(fh, guard.ExpectedEntry); guardErr != nil {
|
|
return nil, nil, guardErr
|
|
}
|
|
request.Condition = &filer_pb.WriteCondition{Clauses: []*filer_pb.WriteCondition_Clause{guard}}
|
|
err = wfs.withFilerClient(ctx, false, wfs.filerRPCWait(), func(ctx context.Context, client filer_pb.SeaweedFilerClient) error {
|
|
var callErr error
|
|
resp, callErr = filer_pb.CreateEntryWithResponse(ctx, client, request)
|
|
if errors.Is(callErr, filer_pb.ErrPreconditionFailed) {
|
|
publication.refused = true
|
|
}
|
|
return callErr
|
|
})
|
|
if err == nil && request.Entry != sent {
|
|
merged = proto.Clone(request.Entry).(*filer_pb.Entry)
|
|
wfs.mapPbIdFromFilerToLocal(merged)
|
|
}
|
|
if err == nil || base == nil || attempt >= maxContentFlushMerges || !errors.Is(err, filer_pb.ErrPreconditionFailed) {
|
|
return resp, merged, err
|
|
}
|
|
stored := wfs.lookupRefusedEntry(ctx, request.Directory, request.Entry.Name)
|
|
if stored == nil {
|
|
return resp, nil, err
|
|
}
|
|
mark := fh.markSince(base)
|
|
next, ok := mergeContentEntry(stored, base, local, sent, mark.truncatedTo, mark.rewrote)
|
|
if !ok || buffered && !mergeKeepsBuffered(next, local) {
|
|
return resp, nil, err
|
|
}
|
|
glog.V(1).Infof("content flush %s/%s: entry changed since the handle's base, merging", request.Directory, request.Entry.Name)
|
|
request.Entry = next
|
|
guard = &filer_pb.WriteCondition_Clause{Kind: filer_pb.WriteCondition_IF_ENTRY_EQUAL, ExpectedEntry: stored}
|
|
}
|
|
}
|
|
|
|
// lookupRefusedEntry reads the entry a refused create ran into, within one
|
|
// filer call: a read that stalls must not hold or replay the refusal.
|
|
func (wfs *WFS) lookupRefusedEntry(ctx context.Context, dir, name string) (stored *filer_pb.Entry) {
|
|
ctx, cancel := context.WithTimeout(ctx, wfs.filerRPCWait())
|
|
defer cancel()
|
|
_ = wfs.withFilerClient(ctx, false, wfs.filerRPCWait(), func(ctx context.Context, client filer_pb.SeaweedFilerClient) error {
|
|
found, err := filer_pb.LookupEntry(ctx, client, &filer_pb.LookupDirectoryEntryRequest{Directory: dir, Name: name})
|
|
if err == nil {
|
|
stored = found.Entry
|
|
}
|
|
return err
|
|
})
|
|
return stored
|
|
}
|
|
|
|
// maxContentFlushMerges bounds how often one flush rebases onto a moving entry.
|
|
const maxContentFlushMerges = 3
|
|
|
|
// mergeContentEntry replays what local changed since base onto stored: the
|
|
// chunks it added or dropped, and the attributes, extended keys and inline
|
|
// content it changed, taking those values from sent (local in stored's id
|
|
// space). Everything else keeps the stored value. A local truncation keeps
|
|
// the handle's size and clips stored chunks at the lowest size truncated to
|
|
// (truncatedTo, or -1); otherwise the stored size stands unless the handle
|
|
// changed its size or wrote past it.
|
|
// A different file at the path, inline content changed on both sides,
|
|
// inline content next to chunks (reads stop at the inline bytes), a manifest
|
|
// across the truncate (its members keep their sizes), or a local
|
|
// whole-file rewrite (rewrote) over a peer's chunk change cannot be merged:
|
|
// the rewrite carries the handle's stale view.
|
|
func mergeContentEntry(stored, base, local, sent *filer_pb.Entry, truncatedTo int64, rewrote bool) (*filer_pb.Entry, bool) {
|
|
if !sameContentFile(stored, base) {
|
|
return nil, false
|
|
}
|
|
merged := proto.Clone(stored).(*filer_pb.Entry)
|
|
merged.Name = sent.Name
|
|
if !bytes.Equal(local.Content, base.Content) {
|
|
if !bytes.Equal(stored.Content, base.Content) {
|
|
return nil, false
|
|
}
|
|
merged.Content = sent.Content
|
|
}
|
|
|
|
inBase := make(map[string]bool, len(base.Chunks))
|
|
for _, c := range base.Chunks {
|
|
inBase[contentChunkKey(c)] = true
|
|
}
|
|
inLocal := make(map[string]bool, len(local.Chunks))
|
|
for _, c := range local.Chunks {
|
|
inLocal[contentChunkKey(c)] = true
|
|
}
|
|
localSize := filer.FileSize(local)
|
|
if truncatedTo < 0 && localSize < filer.FileSize(base) {
|
|
truncatedTo = int64(localSize)
|
|
}
|
|
peerChanged := len(stored.Chunks) != len(base.Chunks)
|
|
merged.Chunks = merged.Chunks[:0]
|
|
for _, c := range stored.Chunks {
|
|
key := contentChunkKey(c)
|
|
peerChanged = peerChanged || !inBase[key]
|
|
if inBase[key] && !inLocal[key] || truncatedTo >= 0 && c.Offset >= truncatedTo {
|
|
continue
|
|
}
|
|
c = proto.Clone(c).(*filer_pb.FileChunk)
|
|
if truncatedTo >= 0 && c.Offset+int64(c.Size) > truncatedTo {
|
|
if c.IsChunkManifest {
|
|
return nil, false
|
|
}
|
|
c.Size = uint64(truncatedTo - c.Offset)
|
|
}
|
|
merged.Chunks = append(merged.Chunks, c)
|
|
}
|
|
if rewrote && peerChanged {
|
|
return nil, false
|
|
}
|
|
size := int64(filer.FileSize(stored))
|
|
if localSize != filer.FileSize(base) {
|
|
size = max(size, int64(localSize))
|
|
}
|
|
for _, c := range local.Chunks {
|
|
if !inBase[contentChunkKey(c)] {
|
|
merged.Chunks = append(merged.Chunks, proto.Clone(c).(*filer_pb.FileChunk))
|
|
size = max(size, c.Offset+int64(c.Size))
|
|
}
|
|
}
|
|
if len(merged.Content) > 0 && len(merged.Chunks) > 0 {
|
|
return nil, false
|
|
}
|
|
|
|
if merged.Attributes == nil {
|
|
merged.Attributes = &filer_pb.FuseAttributes{}
|
|
}
|
|
baseAttrs, localAttrs := base.GetAttributes(), local.GetAttributes()
|
|
if baseAttrs == nil {
|
|
baseAttrs = &filer_pb.FuseAttributes{}
|
|
}
|
|
if localAttrs == nil {
|
|
localAttrs = &filer_pb.FuseAttributes{}
|
|
}
|
|
m, b, l := merged.Attributes.ProtoReflect(), baseAttrs.ProtoReflect(), localAttrs.ProtoReflect()
|
|
s := proto.Clone(sent.GetAttributes()).(*filer_pb.FuseAttributes).ProtoReflect()
|
|
fields := m.Descriptor().Fields()
|
|
for i := 0; i < fields.Len(); i++ {
|
|
fd := fields.Get(i)
|
|
if fd.Name() == "file_size" || l.Get(fd).Equal(b.Get(fd)) {
|
|
continue
|
|
}
|
|
if s.Has(fd) {
|
|
m.Set(fd, s.Get(fd))
|
|
} else {
|
|
m.Clear(fd)
|
|
}
|
|
}
|
|
|
|
for k, v := range local.Extended {
|
|
if old, found := base.Extended[k]; !found || !bytes.Equal(old, v) {
|
|
if merged.Extended == nil {
|
|
merged.Extended = make(map[string][]byte)
|
|
}
|
|
merged.Extended[k] = sent.Extended[k]
|
|
}
|
|
}
|
|
for k := range base.Extended {
|
|
if _, found := local.Extended[k]; !found {
|
|
delete(merged.Extended, k)
|
|
}
|
|
}
|
|
|
|
if truncatedTo >= 0 {
|
|
size = int64(localSize)
|
|
}
|
|
merged.Attributes.FileSize = uint64(size)
|
|
// A peer truncate must not turn local pending data into a successful hole.
|
|
if filer.FileSize(stored) < filer.FileSize(base) && contentGapAfter(merged, filer.FileSize(stored)) {
|
|
return nil, false
|
|
}
|
|
return merged, true
|
|
}
|
|
|
|
// sameContentFile reports whether stored is still the regular file base was:
|
|
// the same inode, or for an entry without one, the same creation time. A
|
|
// synthesized base with neither cannot tell.
|
|
func sameContentFile(stored, base *filer_pb.Entry) bool {
|
|
if stored == nil || base == nil || stored.IsDirectory || !bytes.Equal(stored.HardLinkId, base.HardLinkId) {
|
|
return false
|
|
}
|
|
if inode := base.GetAttributes().GetInode(); inode != 0 {
|
|
return stored.GetAttributes().GetInode() == inode
|
|
}
|
|
crtime := base.GetAttributes().GetCrtime()
|
|
return crtime == 0 || stored.GetAttributes().GetCrtime() == crtime
|
|
}
|
|
|
|
// contentChunkKey identifies a chunk reference: truncation shrinks a chunk in
|
|
// place, so the same file id at another size is a different reference. Both
|
|
// file id encodings give one key, and the chunk is left as is.
|
|
func contentChunkKey(c *filer_pb.FileChunk) string {
|
|
return fmt.Sprintf("%s@%d+%d", contentFileIdKey(c), c.Offset, c.Size)
|
|
}
|
|
|
|
func contentFileIdKey(c *filer_pb.FileChunk) string {
|
|
fid := c.Fid
|
|
if fid == nil {
|
|
if parsed, err := filer_pb.ToFileIdObject(c.FileId); err == nil {
|
|
fid = parsed
|
|
} else {
|
|
return c.FileId
|
|
}
|
|
}
|
|
return fmt.Sprintf("%d,%x,%x", fid.VolumeId, fid.FileKey, fid.Cookie)
|
|
}
|
|
|
|
// ackedBase is the base an acknowledged create leaves: the entry the filer
|
|
// stored, in local ids. The filer stamps commit_ts_ns, crtime, inode and atime
|
|
// and may rewrite chunks, so the requested entry would never match it again.
|
|
// A filer that sends no event leaves the requested entry.
|
|
func (wfs *WFS) ackedBase(resp *filer_pb.CreateEntryResponse, requested *filer_pb.Entry) *filer_pb.Entry {
|
|
stored := resp.GetMetadataEvent().GetEventNotification().GetNewEntry()
|
|
if stored == nil {
|
|
return requested
|
|
}
|
|
base := proto.Clone(stored).(*filer_pb.Entry)
|
|
wfs.mapPbIdFromFilerToLocal(base)
|
|
return base
|
|
}
|
|
|
|
// shouldMergeChunks reports whether the non-manifest chunks are bloated
|
|
// enough to justify re-reading and re-uploading the file. The condition
|
|
// is: sum of compacted chunk sizes > 2 * logical file size.
|
|
func shouldMergeChunks(compactedChunks []*filer_pb.FileChunk, manifestChunks []*filer_pb.FileChunk) (totalChunkSize, fileSize uint64, merge bool) {
|
|
for _, chunk := range compactedChunks {
|
|
totalChunkSize += chunk.Size
|
|
}
|
|
// Count manifest coverage toward stored total. Each manifest holds
|
|
// sub-chunks on volume servers that cover approximately Size bytes.
|
|
// Without this, overlapping manifests accumulate undetected because
|
|
// the merge condition only saw the (small) non-manifest chunk total.
|
|
for _, chunk := range manifestChunks {
|
|
totalChunkSize += chunk.Size
|
|
}
|
|
allChunks := make([]*filer_pb.FileChunk, 0, len(compactedChunks)+len(manifestChunks))
|
|
allChunks = append(allChunks, compactedChunks...)
|
|
allChunks = append(allChunks, manifestChunks...)
|
|
fileSize = filer.TotalSize(allChunks)
|
|
merge = fileSize > 0 && totalChunkSize > 2*fileSize
|
|
return
|
|
}
|
|
|
|
// maybeMergeChunks re-reads and re-uploads file data as properly sized chunks
|
|
// when the total stored chunk data significantly exceeds the logical file size,
|
|
// which happens after many random writes create partially-overlapping small chunks.
|
|
func (wfs *WFS) maybeMergeChunks(fileFullPath util.FullPath, compactedChunks []*filer_pb.FileChunk, manifestChunks []*filer_pb.FileChunk) ([]*filer_pb.FileChunk, error) {
|
|
totalChunkSize, fileSize, merge := shouldMergeChunks(compactedChunks, manifestChunks)
|
|
if !merge {
|
|
return nil, nil
|
|
}
|
|
|
|
glog.V(0).Infof("%.1fx chunk bloat detected on %s (%d chunks, %d stored vs %d content), merging",
|
|
float64(totalChunkSize)/float64(fileSize), fileFullPath, len(compactedChunks), totalChunkSize, fileSize)
|
|
|
|
ctx := context.Background()
|
|
allChunks := make([]*filer_pb.FileChunk, 0, len(compactedChunks)+len(manifestChunks))
|
|
allChunks = append(allChunks, compactedChunks...)
|
|
allChunks = append(allChunks, manifestChunks...)
|
|
reader := filer.NewChunkStreamReaderFromLookup(ctx, wfs.LookupFn(), allChunks)
|
|
defer reader.Close()
|
|
|
|
saveFunc := wfs.saveDataAsChunk(fileFullPath)
|
|
chunkSize := wfs.option.ChunkSizeLimit
|
|
if int64(fileSize) < chunkSize {
|
|
chunkSize = int64(fileSize)
|
|
}
|
|
|
|
var newChunks []*filer_pb.FileChunk
|
|
var offset int64
|
|
buf := make([]byte, chunkSize)
|
|
|
|
for {
|
|
n, readErr := io.ReadFull(reader, buf)
|
|
if n > 0 {
|
|
chunk, uploadErr := saveFunc(bytes.NewReader(buf[:n]), "", offset, 0, uint64(n))
|
|
if uploadErr != nil {
|
|
return nil, uploadErr
|
|
}
|
|
newChunks = append(newChunks, chunk)
|
|
offset += int64(n)
|
|
}
|
|
if readErr != nil {
|
|
if readErr == io.EOF || readErr == io.ErrUnexpectedEOF {
|
|
break
|
|
}
|
|
return nil, readErr
|
|
}
|
|
}
|
|
|
|
glog.V(0).Infof("merged %s: %d chunks -> %d chunks", fileFullPath, len(compactedChunks)+len(manifestChunks), len(newChunks))
|
|
|
|
return newChunks, nil
|
|
}
|