Chris LuandClaude Opus 5.5 cdea7e79e0 filer: fold small chunks into manifests 256 at a time (#12462) ...

* filer: fold small chunks into manifests 256 at a time

Every metadata event carries the file's whole chunk list, so a file grown
by small appends re-sent each chunk record on every append until the
list reached ManifestBatch (10000). Chunks under 1 MiB now fold once 256
of them build up; full-size chunks still fold at ManifestBatch, so large
files keep their flat lists and read without a manifest fetch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* filer: merge appended small chunks at the small-chunk fold count

The append path merges small chunks into full-size ones, but only at
1000 of them; the fold now hides small chunks in a manifest at 256, so
the merge would never see enough of them. Merging at the same count
keeps rewriting them into full chunks first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

2026-11-06 13:41:02 -08:00

..

abstract_sql

fix: volume.fsck purges live chunks of files renamed during its filer walk (SQL stores) (#11883)

2026-10-21 20:31:42 -07:00

arangodb

filer: free orphaned hard-link chunks and remote objects in delete sweeps (#11809)

2026-10-18 12:23:30 -07:00

cassandra

cassandra: do not mask KvGet backend errors as not-found (#10361)

2026-07-18 13:57:24 -07:00

cassandra2

cassandra: do not mask KvGet backend errors as not-found (#10361)

2026-07-18 13:57:24 -07:00

elastic/v7

filer: free orphaned hard-link chunks and remote objects in delete sweeps (#11809)

2026-10-18 12:23:30 -07:00

empty_folder_cleanup

filer: support nanosecond timestamps for conditional delete (#11339) (#12071)

2026-10-26 07:51:33 -07:00

etcd

filer: fix prefixed listing when the resume point sorts below the prefix (#12049)

2026-10-24 19:59:19 -07:00

foundationdb

filer: guard FoundationDB 100KB value limit and pack earlier (#11161)

2026-09-04 23:12:27 -07:00

hbase

filer: fix prefixed listing when the resume point sorts below the prefix (#12049)

2026-10-24 19:59:19 -07:00

leveldb

filer: fix prefixed listing when the resume point sorts below the prefix (#12049)

2026-10-24 19:59:19 -07:00

leveldb2

master, volume, filer: start on a read-only mounted directory and serve existing data read-only (#12081)

2026-10-26 09:28:49 -07:00

leveldb3

filer: scope a prefix JWT to the directory a zip extract writes under (#12305)

2026-11-01 18:36:31 -08:00

mongodb

filer: free orphaned hard-link chunks and remote objects in delete sweeps (#11809)

2026-10-18 12:23:30 -07:00

mysql

fix: volume.fsck purges live chunks of files renamed during its filer walk (SQL stores) (#11883)

2026-10-21 20:31:42 -07:00

mysql2

filer: a listing over a hard link no longer deadlocks a bounded SQL pool (#11118)

2026-09-02 23:45:45 -07:00

posixlock

filer/posixlock: remove the unused lock-set serde (#9676)

2026-05-25 13:15:19 -07:00

postgres

fix: volume.fsck purges live chunks of files renamed during its filer walk (SQL stores) (#11883)

2026-10-21 20:31:42 -07:00

postgres2

filer/postgres: create filemeta table on startup via createTable config (#11229)

2026-09-08 16:17:02 -07:00

redis

filer: restore a folder that received an entry while it was deleted (#10783)

2026-08-17 00:04:07 -07:00

redis2

redis2: pipeline entry reads when listing a directory (#12042)

2026-10-24 15:04:37 -07:00

redis3

filer.redis3: accept username and keyPrefix options (#12192)

2026-10-28 22:54:07 -07:00

redis_conf

build(deps): bump github.com/redis/go-redis/v9 from 9.21.0 to 9.22.0 (#11306)

2026-09-14 16:16:35 -07:00

redis_tls

filer: honor the documented TLS options in every redis store (#10425)

2026-07-24 10:26:38 -07:00

rocksdb

filer: fix prefixed listing when the resume point sorts below the prefix (#12049)

2026-10-24 19:59:19 -07:00

sqlite

fix: volume.fsck purges live chunks of files renamed during its filer walk (SQL stores) (#11883)

2026-10-21 20:31:42 -07:00

store_test

fix(weed/filer/store_test): fix dropped errors (#8782)

2026-03-26 12:07:48 -07:00

tarantool

tarantool: fix upsert data corruption and missing context propagation (#10926)

2026-08-24 15:17:08 -07:00

tikv

filer: fix prefixed listing when the resume point sorts below the prefix (#12049)

2026-10-24 19:59:19 -07:00

ydb

filer: escape LIKE wildcards in SQL list prefixes (#11757)

2026-10-15 16:51:15 -07:00

configuration.go

chore: execute goimports to format the code (#7983)

2026-01-07 13:06:08 -08:00

copy_params.go

Use filer-side copy for mounted whole-file copy_file_range (#8747)

2026-03-23 18:35:15 -07:00

entry_codec_atime_test.go

feat(filer): add atime to FuseAttributes + TouchAccessTime RPC (#9556)

2026-05-19 10:22:17 -07:00

entry_codec_attributes_test.go

filer: list directories without decoding chunk lists (#10616)

2026-08-07 12:03:18 -07:00

entry_codec_attributes.go

filer: list directories without decoding chunk lists (#10616)

2026-08-07 12:03:18 -07:00

entry_codec.go

feat(filer): add atime to FuseAttributes + TouchAccessTime RPC (#9556)

2026-05-19 10:22:17 -07:00

entry_s3_expiry_test.go

[s3] force filer apply s3 expiry metadata (#10469)

2026-08-05 23:55:20 -07:00

entry.go

filer: auto-enable per-path commit records when a peer has its own store (#12062)

2026-10-25 11:37:06 -07:00

filechunk_group_test.go

filer: clean up manifest resolve error propagation and add webdav tes… (#11297)

2026-09-13 18:34:09 -07:00

filechunk_group.go

filer.sync: sign proxied chunk I/O from the per-side security file (#11645)

2026-10-08 22:04:06 +08:00

filechunk_manifest_cache_benchmark_test.go

filer.sync: sign proxied chunk I/O from the per-side security file (#11645)

2026-10-08 22:04:06 +08:00

filechunk_manifest_cache_test.go

filer.sync: sign proxied chunk I/O from the per-side security file (#11645)

2026-10-08 22:04:06 +08:00

filechunk_manifest_cache.go

[Mount] Cache Chunk Manifest Resolution for Repeated File Opens (#11266)

2026-09-12 20:06:25 -07:00

filechunk_manifest_resolve_test.go

s3: separate Object Lock configuration permission (#11361)

2026-09-16 15:27:09 -07:00

filechunk_manifest_rollback_test.go

filer: fold small chunks into manifests 256 at a time (#12462)

2026-11-06 13:41:02 -08:00

filechunk_manifest_test.go

filer: fold small chunks into manifests 256 at a time (#12462)

2026-11-06 13:41:02 -08:00

filechunk_manifest.go

filer: fold small chunks into manifests 256 at a time (#12462)

2026-11-06 13:41:02 -08:00

filechunk_section_test.go

chore: execute goimports to format the code (#7983)

2026-01-07 13:06:08 -08:00

filechunk_section.go

[Mount] Add ChunkGroup seeking tests and fix boundary handling (#11223)

2026-09-08 10:50:41 -07:00

filechunks2_test.go

filer: fail the request, not the process, when chunks cannot be resolved (#11685)

2026-10-10 23:39:29 +08:00

filechunks_cancel_test.go

filer: fail the request, not the process, when chunks cannot be resolved (#11685)

2026-10-10 23:39:29 +08:00

filechunks_read_test.go

chore: execute goimports to format the code (#7983)

2026-01-07 13:06:08 -08:00

filechunks_read.go

chore: execute goimports to format the code (#7983)

2026-01-07 13:06:08 -08:00

filechunks_test.go

filer,pb,mount: keep per-chunk cipher keys out of logs (#12085)

2026-10-26 11:52:56 -07:00

filechunks.go

filer,pb,mount: keep per-chunk cipher keys out of logs (#12085)

2026-10-26 11:52:56 -07:00

filer_buckets.go

Prevent bucket renaming in filer, fuse mount, and S3 (#8048)

2026-01-16 19:48:09 -08:00

filer_conf_test.go

filer.conf: merge fsync on presence like worm (#11953)

2026-10-23 20:45:46 -07:00

filer_conf.go

filer.conf: merge fsync on presence like worm (#11953)

2026-10-23 20:45:46 -07:00

filer_delete_collection_test.go

filer: shut down test filers' meta-log buffers, fixing the 32-bit OOM (#12104)

2026-10-26 13:24:48 -07:00

filer_delete_entry_test.go

filer: support nanosecond timestamps for conditional delete (#11339) (#12071)

2026-10-26 07:51:33 -07:00

filer_delete_entry.go

filer: support nanosecond timestamps for conditional delete (#11339) (#12071)

2026-10-26 07:51:33 -07:00

filer_deletion_persist_test.go

filer: persist pending chunk deletions across restarts (durable deletion ledger) (#11550)

2026-10-02 22:37:31 +08:00

filer_deletion_persist.go

filer: persist pending chunk deletions across restarts (durable deletion ledger) (#11550)

2026-10-02 22:37:31 +08:00

filer_deletion_test.go

filer: persist pending chunk deletions across restarts (durable deletion ledger) (#11550)

2026-10-02 22:37:31 +08:00

filer_deletion.go

security: fail closed on gRPC TLS errors, verify dialed server identity (#12263)

2026-10-30 20:04:45 -07:00

filer_hardlink.go

move to https://github.com/seaweedfs/seaweedfs

2022-07-29 00:17:28 -07:00

filer_inode_test.go

filer: shut down test filers' meta-log buffers, fixing the 32-bit OOM (#12104)

2026-10-26 13:24:48 -07:00

filer_inode.go

filer: keep generated inodes inside the positive signed 64-bit range (#11567)

2026-10-03 09:13:59 +08:00

filer_lazy_remote_listing.go

filer: auto-enable per-path commit records when a peer has its own store (#12062)

2026-10-25 11:37:06 -07:00

filer_lazy_remote_test.go

filer: no lazy remote fetch inside a store transaction (#12439)

2026-11-06 13:21:37 -08:00

filer_lazy_remote.go

filer: free orphaned hard-link chunks and remote objects in delete sweeps (#11809)

2026-10-18 12:23:30 -07:00

filer_notify_append_orphan_test.go

filer: release metadata log chunks a failed append leaves unreferenced (#12455)

2026-11-06 13:19:09 -08:00

filer_notify_append_test.go

fix(filer): apply -filer.disk default to metadata log assigns (#10080)

2026-06-24 10:47:11 -07:00

filer_notify_append.go

filer: release metadata log chunks a failed append leaves unreferenced (#12455)

2026-11-06 13:19:09 -08:00

filer_notify_inflight_test.go

fix(filer): bound aggregated metadata reads by peer watermarks (#10803)

2026-08-19 18:38:46 -07:00

filer_notify_read_minute_test.go

fix(filer): stop skipping recent unflushed events on metadata subscription gaps (#10501)

2026-08-02 00:06:22 -07:00

filer_notify_read_testing.go

fix(filer): stop skipping recent unflushed events on metadata subscription gaps (#10501)

2026-08-02 00:06:22 -07:00

filer_notify_read.go

fix(filer): stop skipping recent unflushed events on metadata subscription gaps (#10501)

2026-08-02 00:06:22 -07:00

filer_notify_test.go

filer: stop an oversized metadata log flush from wedging the change feed (#10430)

2026-07-24 17:59:20 -07:00

filer_notify.go

filer: deliver meta-log events under their stream position, not their stamp (#12084)

2026-10-26 12:18:40 -07:00

filer_on_meta_event_test.go

Adjust rename events metadata format (#8854)

2026-03-30 18:25:11 -07:00

filer_on_meta_event.go

filer,pb,mount: keep per-chunk cipher keys out of logs (#12085)

2026-10-26 11:52:56 -07:00

filer_prefix_object_test.go

s3: route regular copy source reads to the write owner (#12173)

2026-10-28 14:35:39 -07:00

filer_remote_evict_test.go

filer: evict remote-cached objects under storage pressure (#11515)

2026-09-29 22:05:42 +08:00

filer_remote_evict.go

filer: evict remote-cached objects under storage pressure (#11515)

2026-09-29 22:05:42 +08:00

filer_remote_tombstone_test.go

filer: a full remote-deletion tombstone table keeps the lazy fetch off (#12440)

2026-11-06 13:28:47 -08:00

filer_remote_tombstone.go

filer: a full remote-deletion tombstone table keeps the lazy fetch off (#12440)

2026-11-06 13:28:47 -08:00

filer_rename_test.go

filer: reuse IsEqualOrUnder for the rename subdirectory check (#12338)

2026-11-02 10:28:48 -08:00

filer_rename.go

filer: reuse IsEqualOrUnder for the rename subdirectory check (#12338)

2026-11-02 10:28:48 -08:00

filer_search.go

filer: async empty folder cleanup via metadata events (#7614)

2025-12-03 21:12:19 -08:00

filer_shutdown_test.go

filer: bound metadata log flush retries during shutdown (#11596)

2026-10-06 09:51:02 +08:00

filer.go

filer: no lazy remote fetch inside a store transaction (#12439)

2026-11-06 13:21:37 -08:00

filerstore_hardlink.go

fix(filer): update hard link ctime when nlink changes on unlink (#9018)

2026-04-10 11:23:52 -07:00

filerstore_translate_path.go

Add error list each entry func (#7485)

2025-11-25 19:35:19 -08:00

filerstore_wrapper_test.go

filer: fix prefixed listing when the resume point sorts below the prefix (#12049)

2026-10-24 19:59:19 -07:00

filerstore_wrapper.go

filer: fix prefixed listing when the resume point sorts below the prefix (#12049)

2026-10-24 19:59:19 -07:00

filerstore.go

fix: volume.fsck purges live chunks of files renamed during its filer walk (SQL stores) (#11883)

2026-10-21 20:31:42 -07:00

gateway_upload.go

refactor: centralize genUploadUrl in UploadOption (#10164)

2026-06-30 20:45:43 -07:00

interval_list_test.go

chore: execute goimports to format the code (#7983)

2026-01-07 13:06:08 -08:00

interval_list.go

Mount concurrent read (#4400)

2023-04-13 22:32:45 -07:00

manifest_batch_foundationdb.go

filer: guard FoundationDB 100KB value limit and pack earlier (#11161)

2026-09-04 23:12:27 -07:00

manifest_batch.go

filer: guard FoundationDB 100KB value limit and pack earlier (#11161)

2026-09-04 23:12:27 -07:00

meta_aggregator_replay_test.go

filer: deliver meta-log events under their stream position, not their stamp (#12084)

2026-10-26 12:18:40 -07:00

meta_aggregator_test.go

filer: resume a rejoining peer's meta subscription at its last position (#12279)

2026-10-31 11:49:50 -07:00

meta_aggregator_testing.go

filer: end local-only metadata subscriptions when remote peers appear (#11251)

2026-09-09 20:16:50 -07:00

meta_aggregator_watermark_test.go

filer: resume a rejoining peer's meta subscription at its last position (#12279)

2026-10-31 11:49:50 -07:00

meta_aggregator.go

filer: resume a rejoining peer's meta subscription at its last position (#12279)

2026-10-31 11:49:50 -07:00

meta_log_collection_test.go

filer: shut down test filers' meta-log buffers, fixing the 32-bit OOM (#12104)

2026-10-26 13:24:48 -07:00

meta_path_ts_records_test.go

filer: auto-enable per-path commit records when a peer has its own store (#12062)

2026-10-25 11:37:06 -07:00

meta_path_ts.go

filer: auto-enable per-path commit records when a peer has its own store (#12062)

2026-10-25 11:37:06 -07:00

meta_replay.go

filer: deliver meta-log events under their stream position, not their stamp (#12084)

2026-10-26 12:18:40 -07:00

metadata_event_context.go

Adjust rename events metadata format (#8854)

2026-03-30 18:25:11 -07:00

metadata_event_sink_test.go

filer: shut down test filers' meta-log buffers, fixing the 32-bit OOM (#12104)

2026-10-26 13:24:48 -07:00

metadata_event_sink.go

mount: make metadata cache rebuilds snapshot-consistent (#8531)

2026-03-07 09:19:40 -08:00

mount_peer_registry_test.go

peer chunk sharing 2/8: filer mount registry (#9131)

2026-04-18 20:03:23 -07:00

mount_peer_registry.go

peer chunk sharing 2/8: filer mount registry (#9131)

2026-04-18 20:03:23 -07:00

persisted_log_cache_test.go

Cut per-subscriber replay decode and widen replay concurrency (#9917)

2026-06-10 13:08:34 -07:00

persisted_log_cache.go

filer.sync: sign proxied chunk I/O from the per-side security file (#11645)

2026-10-08 22:04:06 +08:00

placement_overlay.go

filer: add a placement overlay seam for the write path (#10437)

2026-07-25 11:20:41 -07:00

read_remote_test.go

preserve Content-Encoding for remote-mounted objects (#10340)

2026-07-15 19:20:00 -07:00

read_remote.go

preserve Content-Encoding for remote-mounted objects (#10340)

2026-07-15 19:20:00 -07:00

read_write_test.go

filer: skip UpdateEntry when inline content is unchanged (#11580)

2026-10-03 21:00:51 +08:00

read_write.go

filer: skip UpdateEntry when inline content is unchanged (#11580)

2026-10-03 21:00:51 +08:00

reader_at_prefetch_window_test.go

filer: keep sequential prefetch within prefetchCount chunks (#11674)

2026-10-12 17:41:01 +08:00

reader_at_shared_test.go

s3: keep small ranged GETs on range reads, no whole-chunk downloads (#11577)

2026-10-03 15:01:04 +08:00

reader_at_test.go

S3: optionally serve remote-mounted objects from remote when the local read fails (#10837)

2026-08-21 11:47:53 -07:00

reader_at.go

s3: keep small ranged GETs on range reads, no whole-chunk downloads (#11577)

2026-10-03 15:01:04 +08:00

reader_cache_budget.go

s3: keep small ranged GETs on range reads, no whole-chunk downloads (#11577)

2026-10-03 15:01:04 +08:00

reader_cache_memory_test.go

filer: free orphaned hard-link chunks and remote objects in delete sweeps (#11809)

2026-10-18 12:23:30 -07:00

reader_cache_test.go

filer: clear a dead test replica's reachability record before closing it (#12387)

2026-11-03 20:47:43 -08:00

reader_cache.go

filer: keep sequential prefetch within prefetchCount chunks (#11674)

2026-10-12 17:41:01 +08:00

reader_pattern_test.go

s3: keep small ranged GETs on range reads, no whole-chunk downloads (#11577)

2026-10-03 15:01:04 +08:00

reader_pattern.go

s3: keep small ranged GETs on range reads, no whole-chunk downloads (#11577)

2026-10-03 15:01:04 +08:00

remote_mapping_test.go

read cold remote objects straight from the origin while caching (#10731)

2026-08-12 23:00:10 -07:00

remote_mapping.go

read cold remote objects straight from the origin while caching (#10731)

2026-08-12 23:00:10 -07:00

remote_storage_test.go

remote: reuse one guarded remote storage client per configuration (#12189)

2026-10-29 00:57:37 -07:00

remote_storage.go

remote: reuse one guarded remote storage client per configuration (#12189)

2026-10-29 00:57:37 -07:00

s3iam_conf_test.go

s3: fix configuring IAM for the same user

2022-08-30 09:37:52 -07:00

s3iam_conf.go

convert error fromating to %w everywhere (#6995)

2025-07-16 23:39:27 -07:00

stream_benchmark_test.go

go fmt

2026-04-10 17:31:14 -07:00

stream_chunk_buffer_test.go

filer: buffer recently fetched chunks so parallel readers do not refetch (#12254)

2026-10-30 08:58:48 -07:00

stream_from_filer_lookup_test.go

filer: let NewChunkStreamReaderFromFiler fall back to the master on a cache miss (#11964)

2026-10-24 07:03:47 -07:00

stream_jwt_test.go

filer: authorize the chunk proxy and the root listing like the rest of the filer port (#11152)

2026-09-04 16:39:36 -07:00

stream_manifest_error_test.go

filer: clean up manifest resolve error propagation and add webdav tes… (#11297)

2026-09-13 18:34:09 -07:00

stream_prefetch_test.go

feat(s3): add concurrent chunk prefetch for large file downloads (#8917)

2026-04-03 19:57:30 -07:00

stream_prefetch.go

read: try a replica that stopped answering last, and relearn its volume's locations (#11130)

2026-09-03 11:51:48 -07:00

stream_source_error_test.go

s3 sink: report the source read error the SDK hides (#11277)

2026-09-11 19:44:30 -07:00

stream_test.go

Fix s3 cancelled read relookup (#12130)

2026-10-27 08:21:32 -07:00

stream.go

filer: buffer recently fetched chunks so parallel readers do not refetch (#12254)

2026-10-30 08:58:48 -07:00

topics.go

merge current message queue code changes (#6201)

2024-11-04 12:08:25 -08:00

zip_extract_test.go

filer: shrink zip spool threshold in tests to fix 32-bit OOM (#11796)

2026-10-17 11:43:55 -07:00

zip_extract.go

filer: shrink zip spool threshold in tests to fix 32-bit OOM (#11796)

2026-10-17 11:43:55 -07:00