mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-30 17:00:48 +01:00
generate vtproto marshalers for filer_pb and use them on the metadata log path (#10337)
* generate vtproto marshalers for filer_pb and use them on the metadata log path Reflection-based proto.Unmarshal allocates a fresh message tree through reflect.New on every call. On the metadata subscription fan-out the same event is decoded once per subscriber, so reflect.New tops the decode churn under many mounts. Generate MarshalVT/UnmarshalVT/SizeVT for filer.proto (a separate filer_vtproto.pb.go, filer.pb.go untouched) and call them on the log entry marshal and the subscribe/replay decode paths. UnmarshalVT allocates message structs directly and copies byte and string fields, so it stays wire-compatible with proto.Unmarshal and preserves the non-aliasing the persisted-log cache depends on. For SubscribeMetadataResponse this cuts decode allocations 69 -> 50 and ~4.5us -> ~2.1us per event; the win scales with subscriber overlap. * marshal log entries directly into the buffer SizeVT is allocation-free and MarshalToSizedBufferVT writes into a pre-sized slice, so the log entry can be marshaled straight into logBuffer.buf. This drops the per-entry MarshalVT allocation and the follow-up copy on the write path. * expand vtproto benchmarks: marshal, decode, and marshal-into-buffer by chunk count Parametrize by nested-message count (chunks per event) and add encode + zero-alloc marshal-into-buffer benchmarks alongside the decode one, so the write-path win from MarshalToSizedBufferVT is measurable too. * keep proto.Unmarshal for metadata events to preserve UTF-8 validation UnmarshalVT skips proto3's UTF-8 validation of string fields, so a SubscribeMetadataResponse with an invalid-UTF-8 string (e.g. Directory "\xff") that proto.Unmarshal rejects would decode and reach path filtering and subscribers. Decode events with proto.Unmarshal again; UnmarshalVT stays on the log entry paths, whose only variable-length fields are bytes and so carry no UTF-8 constraint. Tests cover the codec difference and that a malformed event is skipped before delivery.
This commit is contained in:
1 parent
0ad83d5061
commit
c015cc3939
11 files changed
+24628
-36
No files matched your search
@@ -422,7 +422,7 @@ func parseMessagesFromBuffer(buf []byte, startOffset int64, maxMessages int, max
|
||||
// Parse message
|
||||
entryData := buf[pos+4 : pos+4+int(size)]
|
||||
logEntry := &filer_pb.LogEntry{}
|
||||
if err = proto.Unmarshal(entryData, logEntry); err != nil {
|
||||
if err = logEntry.UnmarshalVT(entryData); err != nil {
|
||||
glog.Warningf("[parseMessages] Failed to unmarshal message: %v", err)
|
||||
pos += 4 + int(size)
|
||||
continue
|
||||
|
||||
Reference in new issue
Block a user