Chris LuandClaude Opus 5.5 8581151376 filer: gzip metadata log pieces before uploading them (#12580) ...

* filer: let the fake-master test helper serve any master

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: gzip metadata log pieces before uploading them

Every metadata event carries the old and new entry with their whole chunk
lists, so a file grown by small fsynced appends logs the same chunk records
again and again: 2000 appends of 256 bytes through weed mount wrote 34.8 MB
of log. The uploader only compresses when the first 128 bytes compress, and
a log piece starts with fresh file ids and ETags, so it stored the log as is.

The filer now gzips each piece itself and keeps the compressed bytes when
they save at least 10%. After the same workload the volumes hold 3.05 MB
instead of 35.2 MB, file data included. Readers already honor the chunk's
IsCompressed flag on every path (filer-side decode, log file refs read
straight from volume servers), and the assign request sizes the bytes
actually stored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

2026-11-12 09:13:50 -08:00

..

abstract_sql

filer: validate bucket names before the SQL DDL hooks (#12547)

2026-11-10 21:22:31 -08:00

arangodb

filer: free orphaned hard-link chunks and remote objects in delete sweeps (#11809)

2026-10-18 12:23:30 -07:00

cassandra

cassandra: do not mask KvGet backend errors as not-found (#10361)

2026-07-18 13:57:24 -07:00

cassandra2

cassandra: do not mask KvGet backend errors as not-found (#10361)

2026-07-18 13:57:24 -07:00

elastic/v7

filer: free orphaned hard-link chunks and remote objects in delete sweeps (#11809)

2026-10-18 12:23:30 -07:00

empty_folder_cleanup

filer: support nanosecond timestamps for conditional delete (#11339) (#12071)

2026-10-26 07:51:33 -07:00

etcd

filer/etcd: serve volume.fsck from one etcd revision (#12555)

2026-11-11 10:11:21 -08:00

foundationdb

filer: guard FoundationDB 100KB value limit and pack earlier (#11161)

2026-09-04 23:12:27 -07:00

hbase

filer: fix prefixed listing when the resume point sorts below the prefix (#12049)

2026-10-24 19:59:19 -07:00

leveldb

filer: reopen a leveldb store wedged by a transient write error (#12565)

2026-11-11 10:23:19 -08:00

leveldb2

filer: reopen a leveldb store wedged by a transient write error (#12565)

2026-11-11 10:23:19 -08:00

leveldb3

filer: reopen a leveldb store wedged by a transient write error (#12565)

2026-11-11 10:23:19 -08:00

leveldb_handle

filer: reopen a leveldb store wedged by a transient write error (#12565)

2026-11-11 10:23:19 -08:00

mongodb

filer: free orphaned hard-link chunks and remote objects in delete sweeps (#11809)

2026-10-18 12:23:30 -07:00

mysql

fix: volume.fsck purges live chunks of files renamed during its filer walk (SQL stores) (#11883)

2026-10-21 20:31:42 -07:00

mysql2

filer: a listing over a hard link no longer deadlocks a bounded SQL pool (#11118)

2026-09-02 23:45:45 -07:00

posixlock

filer/posixlock: remove the unused lock-set serde (#9676)

2026-05-25 13:15:19 -07:00

postgres

fix: volume.fsck purges live chunks of files renamed during its filer walk (SQL stores) (#11883)

2026-10-21 20:31:42 -07:00

postgres2

filer/postgres: create filemeta table on startup via createTable config (#11229)

2026-09-08 16:17:02 -07:00

redis

filer: restore a folder that received an entry while it was deleted (#10783)

2026-08-17 00:04:07 -07:00

redis2

redis2: pipeline entry reads when listing a directory (#12042)

2026-10-24 15:04:37 -07:00

redis3

filer.redis3: accept username and keyPrefix options (#12192)

2026-10-28 22:54:07 -07:00

redis_conf

build(deps): bump github.com/redis/go-redis/v9 from 9.21.0 to 9.22.0 (#11306)

2026-09-14 16:16:35 -07:00

redis_tls

filer: honor the documented TLS options in every redis store (#10425)

2026-07-24 10:26:38 -07:00

rocksdb

filer: fix prefixed listing when the resume point sorts below the prefix (#12049)

2026-10-24 19:59:19 -07:00

sqlite

fix: volume.fsck purges live chunks of files renamed during its filer walk (SQL stores) (#11883)

2026-10-21 20:31:42 -07:00

store_test

fix(weed/filer/store_test): fix dropped errors (#8782)

2026-03-26 12:07:48 -07:00

tarantool

tarantool: fix upsert data corruption and missing context propagation (#10926)

2026-08-24 15:17:08 -07:00

tikv

filer: fix prefixed listing when the resume point sorts below the prefix (#12049)

2026-10-24 19:59:19 -07:00

ydb

filer: escape LIKE wildcards in SQL list prefixes (#11757)

2026-10-15 16:51:15 -07:00

configuration.go

chore: execute goimports to format the code (#7983)

2026-01-07 13:06:08 -08:00

copy_params.go

Use filer-side copy for mounted whole-file copy_file_range (#8747)

2026-03-23 18:35:15 -07:00

entry_codec_atime_test.go

feat(filer): add atime to FuseAttributes + TouchAccessTime RPC (#9556)

2026-05-19 10:22:17 -07:00

entry_codec_attributes_test.go

filer: list directories without decoding chunk lists (#10616)

2026-08-07 12:03:18 -07:00

entry_codec_attributes.go

filer: list directories without decoding chunk lists (#10616)

2026-08-07 12:03:18 -07:00

entry_codec.go

feat(filer): add atime to FuseAttributes + TouchAccessTime RPC (#9556)

2026-05-19 10:22:17 -07:00

entry_s3_expiry_test.go

[s3] force filer apply s3 expiry metadata (#10469)

2026-08-05 23:55:20 -07:00

entry.go

filer: auto-enable per-path commit records when a peer has its own store (#12062)

2026-10-25 11:37:06 -07:00

filechunk_group_test.go

filer: clean up manifest resolve error propagation and add webdav tes… (#11297)

2026-09-13 18:34:09 -07:00

filechunk_group.go

filer.sync: sign proxied chunk I/O from the per-side security file (#11645)

2026-10-08 22:04:06 +08:00

filechunk_manifest_cache_benchmark_test.go

filer.sync: sign proxied chunk I/O from the per-side security file (#11645)

2026-10-08 22:04:06 +08:00

filechunk_manifest_cache_test.go

filer.sync: sign proxied chunk I/O from the per-side security file (#11645)

2026-10-08 22:04:06 +08:00

filechunk_manifest_cache.go

[Mount] Cache Chunk Manifest Resolution for Repeated File Opens (#11266)

2026-09-12 20:06:25 -07:00

filechunk_manifest_resolve_test.go

s3: separate Object Lock configuration permission (#11361)

2026-09-16 15:27:09 -07:00

filechunk_manifest_rollback_test.go

filer: fold small chunks into manifests 256 at a time (#12462)

2026-11-06 13:41:02 -08:00

filechunk_manifest_test.go

filer: fold small chunks into manifests 256 at a time (#12462)

2026-11-06 13:41:02 -08:00

filechunk_manifest.go

filer: fold small chunks into manifests 256 at a time (#12462)

2026-11-06 13:41:02 -08:00

filechunk_section_test.go

chore: execute goimports to format the code (#7983)

2026-01-07 13:06:08 -08:00

filechunk_section.go

[Mount] Add ChunkGroup seeking tests and fix boundary handling (#11223)

2026-09-08 10:50:41 -07:00

filechunks2_test.go

filer: fail the request, not the process, when chunks cannot be resolved (#11685)

2026-10-10 23:39:29 +08:00

filechunks_cancel_test.go

filer: fail the request, not the process, when chunks cannot be resolved (#11685)

2026-10-10 23:39:29 +08:00

filechunks_read_test.go

chore: execute goimports to format the code (#7983)

2026-01-07 13:06:08 -08:00

filechunks_read.go

chore: execute goimports to format the code (#7983)

2026-01-07 13:06:08 -08:00

filechunks_test.go

filer,pb,mount: keep per-chunk cipher keys out of logs (#12085)

2026-10-26 11:52:56 -07:00

filechunks.go

filer,pb,mount: keep per-chunk cipher keys out of logs (#12085)

2026-10-26 11:52:56 -07:00

filer_buckets.go

Prevent bucket renaming in filer, fuse mount, and S3 (#8048)

2026-01-16 19:48:09 -08:00

filer_conf_test.go

filer.conf: merge fsync on presence like worm (#11953)

2026-10-23 20:45:46 -07:00

filer_conf.go

filer.conf: merge fsync on presence like worm (#11953)

2026-10-23 20:45:46 -07:00

filer_delete_collection_test.go

filer: gzip metadata log pieces before uploading them (#12580)

2026-11-12 09:13:50 -08:00

filer_delete_entry_test.go

filer: support nanosecond timestamps for conditional delete (#11339) (#12071)

2026-10-26 07:51:33 -07:00

filer_delete_entry.go

filer: support nanosecond timestamps for conditional delete (#11339) (#12071)

2026-10-26 07:51:33 -07:00

filer_deletion_persist_test.go

filer: persist pending chunk deletions across restarts (durable deletion ledger) (#11550)

2026-10-02 22:37:31 +08:00

filer_deletion_persist.go

filer: persist pending chunk deletions across restarts (durable deletion ledger) (#11550)

2026-10-02 22:37:31 +08:00

filer_deletion_test.go

filer: persist pending chunk deletions across restarts (durable deletion ledger) (#11550)

2026-10-02 22:37:31 +08:00

filer_deletion.go

security: fail closed on gRPC TLS errors, verify dialed server identity (#12263)

2026-10-30 20:04:45 -07:00

filer_hardlink.go

move to https://github.com/seaweedfs/seaweedfs

2022-07-29 00:17:28 -07:00

filer_inode_test.go

filer: shut down test filers' meta-log buffers, fixing the 32-bit OOM (#12104)

2026-10-26 13:24:48 -07:00

filer_inode.go

filer: keep generated inodes inside the positive signed 64-bit range (#11567)

2026-10-03 09:13:59 +08:00

filer_lazy_remote_listing.go

filer: auto-enable per-path commit records when a peer has its own store (#12062)

2026-10-25 11:37:06 -07:00

filer_lazy_remote_test.go

filer: no lazy remote fetch inside a store transaction (#12439)

2026-11-06 13:21:37 -08:00

filer_lazy_remote.go

filer: free orphaned hard-link chunks and remote objects in delete sweeps (#11809)

2026-10-18 12:23:30 -07:00

filer_notify_append_compress_test.go

filer: gzip metadata log pieces before uploading them (#12580)

2026-11-12 09:13:50 -08:00

filer_notify_append_orphan_test.go

filer: release metadata log chunks a failed append leaves unreferenced (#12455)

2026-11-06 13:19:09 -08:00

filer_notify_append_test.go

fix(filer): apply -filer.disk default to metadata log assigns (#10080)

2026-06-24 10:47:11 -07:00

filer_notify_append.go

filer: gzip metadata log pieces before uploading them (#12580)

2026-11-12 09:13:50 -08:00

filer_notify_inflight_test.go

fix(filer): bound aggregated metadata reads by peer watermarks (#10803)

2026-08-19 18:38:46 -07:00

filer_notify_read_minute_test.go

fix(filer): stop skipping recent unflushed events on metadata subscription gaps (#10501)

2026-08-02 00:06:22 -07:00

filer_notify_read_testing.go

fix(filer): stop skipping recent unflushed events on metadata subscription gaps (#10501)

2026-08-02 00:06:22 -07:00

filer_notify_read.go

fix(filer): stop skipping recent unflushed events on metadata subscription gaps (#10501)

2026-08-02 00:06:22 -07:00

filer_notify_test.go

filer: stop an oversized metadata log flush from wedging the change feed (#10430)

2026-07-24 17:59:20 -07:00

filer_notify.go

filer: deliver meta-log events under their stream position, not their stamp (#12084)

2026-10-26 12:18:40 -07:00

filer_on_meta_event_test.go

Adjust rename events metadata format (#8854)

2026-03-30 18:25:11 -07:00

filer_on_meta_event.go

filer,pb,mount: keep per-chunk cipher keys out of logs (#12085)

2026-10-26 11:52:56 -07:00

filer_prefix_object_test.go

s3: route regular copy source reads to the write owner (#12173)

2026-10-28 14:35:39 -07:00

filer_remote_evict_test.go

filer: evict remote-cached objects under storage pressure (#11515)

2026-09-29 22:05:42 +08:00

filer_remote_evict.go

filer: evict remote-cached objects under storage pressure (#11515)

2026-09-29 22:05:42 +08:00

filer_remote_tombstone_test.go

filer: a full remote-deletion tombstone table keeps the lazy fetch off (#12440)

2026-11-06 13:28:47 -08:00

filer_remote_tombstone.go

filer: a full remote-deletion tombstone table keeps the lazy fetch off (#12440)

2026-11-06 13:28:47 -08:00

filer_rename_test.go

filer: reuse IsEqualOrUnder for the rename subdirectory check (#12338)

2026-11-02 10:28:48 -08:00

filer_rename.go

filer: reuse IsEqualOrUnder for the rename subdirectory check (#12338)

2026-11-02 10:28:48 -08:00

filer_search.go

filer: async empty folder cleanup via metadata events (#7614)

2025-12-03 21:12:19 -08:00

filer_shutdown_test.go

filer: bound metadata log flush retries during shutdown (#11596)

2026-10-06 09:51:02 +08:00

filer.go

filer: no lazy remote fetch inside a store transaction (#12439)

2026-11-06 13:21:37 -08:00

filerstore_hardlink.go

fix(filer): update hard link ctime when nlink changes on unlink (#9018)

2026-04-10 11:23:52 -07:00

filerstore_translate_path.go

Add error list each entry func (#7485)

2025-11-25 19:35:19 -08:00

filerstore_wrapper_test.go

filer: fix prefixed listing when the resume point sorts below the prefix (#12049)

2026-10-24 19:59:19 -07:00

filerstore_wrapper.go

filer: fix prefixed listing when the resume point sorts below the prefix (#12049)

2026-10-24 19:59:19 -07:00

filerstore.go

fix: volume.fsck purges live chunks of files renamed during its filer walk (SQL stores) (#11883)

2026-10-21 20:31:42 -07:00

gateway_upload.go

refactor: centralize genUploadUrl in UploadOption (#10164)

2026-06-30 20:45:43 -07:00

interval_list_test.go

chore: execute goimports to format the code (#7983)

2026-01-07 13:06:08 -08:00

interval_list.go

Mount concurrent read (#4400)

2023-04-13 22:32:45 -07:00

manifest_batch_foundationdb.go

filer: guard FoundationDB 100KB value limit and pack earlier (#11161)

2026-09-04 23:12:27 -07:00

manifest_batch.go

filer: guard FoundationDB 100KB value limit and pack earlier (#11161)

2026-09-04 23:12:27 -07:00

meta_aggregator_replay_test.go

filer: deliver meta-log events under their stream position, not their stamp (#12084)

2026-10-26 12:18:40 -07:00

meta_aggregator_test.go

filer: resume a rejoining peer's meta subscription at its last position (#12279)

2026-10-31 11:49:50 -07:00

meta_aggregator_testing.go

filer: end local-only metadata subscriptions when remote peers appear (#11251)

2026-09-09 20:16:50 -07:00

meta_aggregator_watermark_test.go

filer: resume a rejoining peer's meta subscription at its last position (#12279)

2026-10-31 11:49:50 -07:00

meta_aggregator.go

filer: resume a rejoining peer's meta subscription at its last position (#12279)

2026-10-31 11:49:50 -07:00

meta_log_collection_test.go

filer: shut down test filers' meta-log buffers, fixing the 32-bit OOM (#12104)

2026-10-26 13:24:48 -07:00

meta_path_ts_records_test.go

filer: auto-enable per-path commit records when a peer has its own store (#12062)

2026-10-25 11:37:06 -07:00

meta_path_ts.go

filer: auto-enable per-path commit records when a peer has its own store (#12062)

2026-10-25 11:37:06 -07:00

meta_replay.go

filer: deliver meta-log events under their stream position, not their stamp (#12084)

2026-10-26 12:18:40 -07:00

metadata_event_context.go

Adjust rename events metadata format (#8854)

2026-03-30 18:25:11 -07:00

metadata_event_sink_test.go

filer: shut down test filers' meta-log buffers, fixing the 32-bit OOM (#12104)

2026-10-26 13:24:48 -07:00

metadata_event_sink.go

mount: make metadata cache rebuilds snapshot-consistent (#8531)

2026-03-07 09:19:40 -08:00

mount_peer_registry_test.go

peer chunk sharing 2/8: filer mount registry (#9131)

2026-04-18 20:03:23 -07:00

mount_peer_registry.go

peer chunk sharing 2/8: filer mount registry (#9131)

2026-04-18 20:03:23 -07:00

persisted_log_cache_test.go

Cut per-subscriber replay decode and widen replay concurrency (#9917)

2026-06-10 13:08:34 -07:00

persisted_log_cache.go

filer.sync: sign proxied chunk I/O from the per-side security file (#11645)

2026-10-08 22:04:06 +08:00

placement_overlay.go

filer: add a placement overlay seam for the write path (#10437)

2026-07-25 11:20:41 -07:00

read_remote_test.go

preserve Content-Encoding for remote-mounted objects (#10340)

2026-07-15 19:20:00 -07:00

read_remote.go

preserve Content-Encoding for remote-mounted objects (#10340)

2026-07-15 19:20:00 -07:00

read_write_test.go

filer: skip UpdateEntry when inline content is unchanged (#11580)

2026-10-03 21:00:51 +08:00

read_write.go

filer: skip UpdateEntry when inline content is unchanged (#11580)

2026-10-03 21:00:51 +08:00

reader_at_prefetch_window_test.go

filer: keep sequential prefetch within prefetchCount chunks (#11674)

2026-10-12 17:41:01 +08:00

reader_at_shared_test.go

s3: keep small ranged GETs on range reads, no whole-chunk downloads (#11577)

2026-10-03 15:01:04 +08:00

reader_at_test.go

S3: optionally serve remote-mounted objects from remote when the local read fails (#10837)

2026-08-21 11:47:53 -07:00

reader_at.go

s3: keep small ranged GETs on range reads, no whole-chunk downloads (#11577)

2026-10-03 15:01:04 +08:00

reader_cache_budget.go

s3: keep small ranged GETs on range reads, no whole-chunk downloads (#11577)

2026-10-03 15:01:04 +08:00

reader_cache_memory_test.go

filer: free orphaned hard-link chunks and remote objects in delete sweeps (#11809)

2026-10-18 12:23:30 -07:00

reader_cache_test.go

filer: clear a dead test replica's reachability record before closing it (#12387)

2026-11-03 20:47:43 -08:00

reader_cache.go

filer: keep sequential prefetch within prefetchCount chunks (#11674)

2026-10-12 17:41:01 +08:00

reader_pattern_test.go

s3: keep small ranged GETs on range reads, no whole-chunk downloads (#11577)

2026-10-03 15:01:04 +08:00

reader_pattern.go

s3: keep small ranged GETs on range reads, no whole-chunk downloads (#11577)

2026-10-03 15:01:04 +08:00

remote_mapping_test.go

read cold remote objects straight from the origin while caching (#10731)

2026-08-12 23:00:10 -07:00

remote_mapping.go

read cold remote objects straight from the origin while caching (#10731)

2026-08-12 23:00:10 -07:00

remote_storage_test.go

remote: reuse one guarded remote storage client per configuration (#12189)

2026-10-29 00:57:37 -07:00

remote_storage.go

remote: reuse one guarded remote storage client per configuration (#12189)

2026-10-29 00:57:37 -07:00

s3iam_conf_test.go

s3: fix configuring IAM for the same user

2022-08-30 09:37:52 -07:00

s3iam_conf.go

convert error fromating to %w everywhere (#6995)

2025-07-16 23:39:27 -07:00

stream_benchmark_test.go

go fmt

2026-04-10 17:31:14 -07:00

stream_chunk_buffer_test.go

filer: buffer recently fetched chunks so parallel readers do not refetch (#12254)

2026-10-30 08:58:48 -07:00

stream_from_filer_lookup_test.go

filer: let NewChunkStreamReaderFromFiler fall back to the master on a cache miss (#11964)

2026-10-24 07:03:47 -07:00

stream_jwt_test.go

filer: authorize the chunk proxy and the root listing like the rest of the filer port (#11152)

2026-09-04 16:39:36 -07:00

stream_manifest_error_test.go

filer: clean up manifest resolve error propagation and add webdav tes… (#11297)

2026-09-13 18:34:09 -07:00

stream_prefetch_test.go

feat(s3): add concurrent chunk prefetch for large file downloads (#8917)

2026-04-03 19:57:30 -07:00

stream_prefetch.go

read: try a replica that stopped answering last, and relearn its volume's locations (#11130)

2026-09-03 11:51:48 -07:00

stream_source_error_test.go

s3 sink: report the source read error the SDK hides (#11277)

2026-09-11 19:44:30 -07:00

stream_test.go

Fix s3 cancelled read relookup (#12130)

2026-10-27 08:21:32 -07:00

stream.go

filer: buffer recently fetched chunks so parallel readers do not refetch (#12254)

2026-10-30 08:58:48 -07:00

topics.go

merge current message queue code changes (#6201)

2024-11-04 12:08:25 -08:00

zip_extract_test.go

filer: shrink zip spool threshold in tests to fix 32-bit OOM (#11796)

2026-10-17 11:43:55 -07:00

zip_extract.go

filer: shrink zip spool threshold in tests to fix 32-bit OOM (#11796)

2026-10-17 11:43:55 -07:00