filer: share one log-buffer window snapshot across all subscriber reads (#10267)

* log_buffer: share one window snapshot across all subscriber reads

Every in-memory read handed each subscriber a private pooled copy of the
window it wanted, so N subscribers reading the same data cost N copies of
up to 8MB each -- and slow consumers (grpc send backpressure) held those
copies live for their whole iteration. With hundreds of mount subscribers
that multiplied into gigabytes of live heap on the filer.

Share the bytes instead of copying per reader:

- Sealed windows get a lazily created GC-owned snapshot, made once by the
  first reader and handed out zero-copy to the rest. The snapshot travels
  with its window when SealBuffer shifts slots, so recycling the sealed
  array never invalidates it.
- The current window keeps a shared snapshot of its append-only prefix
  buf[:pos], extended on demand; each byte is copied once per window
  (writer-rate-bound) instead of once per reader. At seal a fully
  extended prefix becomes the sealed window's snapshot.

ReadFromBuffer now reports whether the returned buffer is a pooled copy
(flush path) or a shared view that must not be released; the read loops
only recycle pooled buffers.

With 200 subscribers consuming at grpc pace over sealed and current
windows, peak live heap drops from 5.2GB to 178MB.

* log_buffer: clear released read buffer so a panic cannot double-free it

The read loops release the previous iteration's pooled buffer and then
call ReadFromBuffer. If that call panicked before reassigning bytesBuf,
the deferred cleanup would put the same buffer into the pool a second
time, letting two future readers share one backing array. Nil the
pointer at the release site so the defer sees nothing to free.

weed/util/log_buffer/log_read.go

@@ -101,6 +101,7 @@ func (logBuffer *LogBuffer) LoopProcessLogData(readerName string, startPosition
 
 	// loop through all messages
 	var bytesBuf *bytes.Buffer
+	var bytesBufPooled bool
 	var batchIndex int64
 	lastReadPosition = startPosition
 	var entryCounter int64
@@ -113,7 +114,7 @@ func (logBuffer *LogBuffer) LoopProcessLogData(readerName string, startPosition
 	// responsive 250ms cadence resumes for active readers.
 	caughtUpToDiskHead := false
 	defer func() {
-		if bytesBuf != nil {
+		if bytesBuf != nil && bytesBufPooled {
 			logBuffer.ReleaseMemory(bytesBuf)
 		}
 		// println("LoopProcessLogData", readerName, "sent messages total", entryCounter)
@@ -121,10 +122,11 @@ func (logBuffer *LogBuffer) LoopProcessLogData(readerName string, startPosition
 
 	for {
 
-		if bytesBuf != nil {
+		if bytesBuf != nil && bytesBufPooled {
 			logBuffer.ReleaseMemory(bytesBuf)
+			bytesBuf = nil // keep the deferred release from double-freeing if ReadFromBuffer panics
 		}
-		bytesBuf, batchIndex, err = logBuffer.ReadFromBuffer(lastReadPosition)
+		bytesBuf, batchIndex, bytesBufPooled, err = logBuffer.ReadFromBuffer(lastReadPosition)
 		if err == ResumeFromDiskError {
 			// Try to read from disk if readFromDiskFn is available
 			if logBuffer.ReadFromDiskFn != nil {
@@ -148,11 +150,11 @@ func (logBuffer *LogBuffer) LoopProcessLogData(readerName string, startPosition
 				// HasData() and ReadFromBuffer lock separately, so a racing write can make
 				// HasData() see data the empty-buffer read missed. Re-read; only bail if the
 				// position is genuinely behind the in-memory window (flushed to disk).
-				reBuf, _, reErr := logBuffer.ReadFromBuffer(lastReadPosition)
+				reBuf, _, rePooled, reErr := logBuffer.ReadFromBuffer(lastReadPosition)
 				if reErr == ResumeFromDiskError {
 					return lastReadPosition, isDone, ResumeFromDiskError
 				}
-				if reBuf != nil {
+				if reBuf != nil && rePooled {
 					logBuffer.ReleaseMemory(reBuf)
 				}
 				continue
@@ -315,13 +317,14 @@ func (logBuffer *LogBuffer) LoopProcessLogDataWithOffset(readerName string, star
 
 	// loop through all messages
 	var bytesBuf *bytes.Buffer
+	var bytesBufPooled bool
 	var offset int64
 	lastReadPosition = startPosition
 	var entryCounter int64
 	// See LoopProcessLogData for the caughtUpToDiskHead invariant.
 	caughtUpToDiskHead := false
 	defer func() {
-		if bytesBuf != nil {
+		if bytesBuf != nil && bytesBufPooled {
 			logBuffer.ReleaseMemory(bytesBuf)
 		}
 		// println("LoopProcessLogDataWithOffset", readerName, "sent messages total", entryCounter)
@@ -335,10 +338,11 @@ func (logBuffer *LogBuffer) LoopProcessLogDataWithOffset(readerName string, star
 			return
 		}
 
-		if bytesBuf != nil {
+		if bytesBuf != nil && bytesBufPooled {
 			logBuffer.ReleaseMemory(bytesBuf)
+			bytesBuf = nil // keep the deferred release from double-freeing if ReadFromBuffer panics
 		}
-		bytesBuf, offset, err = logBuffer.ReadFromBuffer(lastReadPosition)
+		bytesBuf, offset, bytesBufPooled, err = logBuffer.ReadFromBuffer(lastReadPosition)
 		glog.V(4).Infof("ReadFromBuffer for %s returned bytesBuf=%v, offset=%d, err=%v", readerName, bytesBuf != nil, offset, err)
 
 		// Check for buffer corruption error before other error handling
@@ -374,11 +378,11 @@ func (logBuffer *LogBuffer) LoopProcessLogDataWithOffset(readerName string, star
 				// HasData() and ReadFromBuffer lock separately, so a racing write can make
 				// HasData() see data the empty-buffer read missed. Re-read; only bail if the
 				// position is genuinely behind the in-memory window (flushed to disk).
-				reBuf, _, reErr := logBuffer.ReadFromBuffer(lastReadPosition)
+				reBuf, _, rePooled, reErr := logBuffer.ReadFromBuffer(lastReadPosition)
 				if reErr == ResumeFromDiskError {
 					return lastReadPosition, isDone, ResumeFromDiskError
 				}
-				if reBuf != nil {
+				if reBuf != nil && rePooled {
 					logBuffer.ReleaseMemory(reBuf)
 				}
 				continue