* filer: enable per-path commit records only when a peer has its own store
Since seaweedfs#11698 every metadata change writes a per-path commit
record into the filer store (an MpTs key, plus an MpDl key on deletes),
and the records stay after the path is deleted. They order the events a
filer replays from peer filers that keep their own store, since replay
only runs when peerSignature != f.Signature.
Filers that all share one store (one Redis, MySQL, TiKV, ...) never
replay each other's events, so the records are never read, yet each
change still costs one or two extra store writes and the key count grows
with every path ever changed — unbounded growth for in-memory stores.
This turns the records off by default and latches them on the first time
a peer reports a different store signature — no option needed. The latch
is persisted in the store (MpRep) so it survives restarts and converges
filers sharing a store even before they discover the peer themselves.
While off, a filer keeps an MpOff marker so enabling later bumps the
records' generation (MpGen, folded into each record key) past the
unrecorded window: a record older than a change made while off cannot
let an older replayed peer event win. Ordering for unrecorded changes
falls back to the entry's mtime, the same as entries written before the
records existed.
Supersedes the manual option approach in seaweedfs#12060.
* filer: keep delete tombstones while records off, harden enable path
Review on the first commit:
- A delete committed while the records were off left no entry and no
usable tombstone: after enabling, an older replayed create could
resurrect the path. Deletes and rename sources are now always recorded
under the generation-0 MpDl key — they are never fenced by a
generation bump, and they cannot go stale because every delete is
recorded. Creates/updates still write nothing while off.
- bumpPathCommitGen could regress MpGen under concurrent bumpers on a
shared store. It now loops: each pass writes only a generation higher
than any it has seen, so concurrent bumps converge upward.
- doSubscribeToOneFiler proceeded into replay when
enablePathCommitRecords failed, running the whole session without
ordering. It now aborts the subscribe so the reconnect retries with
the records on.
* filer: bind pending commit-record repairs to their generation
A failed MpTs write kept in metaTsPending could land after a shared-store
generation bump, writing a pre-bump timestamp as a current-generation
record and overriding newer unrecorded state. Each pending change record
now carries the generation it failed under: repairs write it only while
that generation is current, and ordering reads ignore it otherwise.
Pending delete records need no binding — MpDl keys are never fenced.
* filer: never move the local records' generation backward
The converge loop in bumpPathCommitGen seeded from zero and both it and
syncPathCommitGen stored unconditionally: a stale writer could leave the
stored generation below what this filer already adopted, and the next
sync or bump then walked the local value back — orphaning records it
wrote under the higher generation. The bump now seeds from the local
generation and every store only moves it forward.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: name a records generation after the off marker it consumes
Two filers consuming the same off marker each counted one past the
stored generation, so the second could store N+2 after the first
stored N+1 and write records under a key the other could not read
until the next check. The bump now takes the marker's own timestamp —
filers racing on one marker land on one value — a marker older than
the stored generation is cleaned as a leftover, and a stored value
that already passed the bump is adopted instead of counted past. A
newly enabling filer whose marker was already consumed names its own
off window with the current time, so its unrecorded period is still
fenced.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: tell a fresh off marker from one the generation already consumed
A marker whose timestamp reads at or below the stored generation was
dropped as leftover, but a new off period can also read that old — the
off filer's clock lagging, or a host clock moving back — and dropping
it kept records from before that period live, so stale replay could
win.
The stored generation now names the marker it was bumped for: a marker
the store still shows is consumed unless it is that one, however old
it reads, and only the consumed marker itself is cleared without a
bump. The generation trails in the value, so a whole-slice read still
yields it on stores written before the pair was stored. Enabling
fences the filer's own off window with the current time instead of a
marker a sharer may already have consumed.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: cover racing off-marker consumers settling on one generation
* filer: adopt the stored generation, not the consumed marker it names
A filer syncing with no off marker present read the value's first field —
the consumed marker's timestamp — so a restart after a lagging-clock off
period under-counted the generation and kept records the bump dropped.
* filer: re-check the off marker before consuming it
A sync whose marker read ran behind a later off period's consume cycle
still bumped for its stale read: the stored generation already covered a
different marker, so the bump overwrote the consumed name, counted the
generation past records a newer consumption meant to keep, and cleared
whatever marker was live by then. Each pass now re-reads the marker and
only bumps or clears while it is still the one the pass saw.
* filer: keep a failed generation read from resetting the records' generation
The final generation read ignored errors and then stored whatever gen
held — possibly zero — so a transient store failure could hide records
the current generation wrote. Unexpected read errors now return before
the local generation changes.
Also cover a sync whose marker read runs behind another consumer's
delete: it must adopt the stored generation, not bump for the stale read.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: fail a bump that cannot verify the stored generation
A generation read that failed mid-bump was treated as absent: the
write could land below a generation a racing bumper had stored, or
re-store a marker that bumper had already consumed. Read failures
now abort the pass like every other store error on the path.
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: cover generation reads that fail instead of returning a value
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: carry the commit timestamp on the entry itself
The per-path MpTs record only exists because a commit timestamp had
nowhere else to live. Stamping commit_ts_ns on the entry before its
store write - and reusing that stamp as the emitted event's TsNs -
makes the entry its own commit record: replay and bootstrap traversal
order on it with zero extra writes for creates and updates.
Deletes still need a durable key since a removed entry cannot carry a
timestamp, so MpDl tombstones stay (one write per delete, growth
bounded by delete history). Entries predating the field fall back to
mtime, matching what peers and old records already assume.
This replaces the MpTs/MpOff/MpGen/MpRep machinery entirely: shared-
store filers write nothing extra, and dedicated-store replication
keeps last-writer-wins ordering plus tombstone coverage for deletes
and ancestor drops.
* filer: keep a directory update replay from tombstoning its subtree
A same-path update wrote a delete record before the delete-and-reinsert,
so a directory metadata change read as a subtree delete: a later peer
event for a child older than the update was then skipped even though
nobody deleted the children, leaving files missing on this filer. Only
file updates take the tombstone now — the re-inserted directory entry's
own commit stamp orders the path itself.
* filer: order unstamped entries by their recorded commit before mtime
Entries written before stamping existed have no CommitTsNs, so the
ordering check fell back to the client-settable mtime and dropped the
MpTs record those versions wrote: an entry committed at event time 300
with mtime 100 lost to a delayed peer event at 200 after upgrade. Read
the legacy record for unstamped entries in localCommitTsNs, CommitTsNs,
and the traverse stamp (which also stopped falling back to mtime,
regressing entry_ts_ns to 0).
* filer: release queued rename event stamps when the rename fails
Each moved entry carried an in-flight stamp its queued event would
adopt at emit time. When the move succeeded but a later child move or
the commit failed, the handler rolled back without emitting the queued
events or releasing their stamps, pinning the delivered and flushed
watermarks below the aborted rename forever. Release every queued
stamp on both failure exits in both rename handlers.
* filer: apply a directory update replay with an in-place upsert
Deleting the stored entry before inserting its replacement left a
window where the directory had no entry and no ordering record — a
tombstone cannot stand in for it without reading as a subtree delete
that suppresses the children's own records. A stale peer event accepted
in that gap could replay an older update or delete and sweep the
directory's children. Upsert the replacement instead: the entry keeps
its recorded commit order across the whole update, and a directory has
no chunks or hard links for a delete to release.
* filer: update an existing directory entry on replay instead of upserting
InsertEntry is not an upsert on SQL stores configured without upsert
support — PostgreSQL with enableUpsert=false issues a plain INSERT and
its duplicate-key error does not match the MySQL-only fallback, so a
replay updating a present directory failed on every retry. Pick
UpdateEntry when the entry exists and InsertEntry when it is absent;
UpdateEntry touches no row that is not there.
* filer: detect a shared-store delete racing a replayed directory update
UpdateEntry reports success even when the row was deleted between the
find and the update — the UPDATE changes zero rows and the replayed
change is dropped while the subscription offset advances. Re-read the
entry after the update and retry the write, re-checking the recorded
commit order each pass so a newer peer change wins instead of being
resurrected over.
* filer: declare commit_ts_ns in field-number order
Field 6 sits between extended and hard_link_id; the declaration now
reads in the same order. Wire format unchanged. Also syncs the Java
client's checked-in copy of filer.proto, which the pb Makefile copies
on gen.
* filer: option to skip delete tombstones entirely
filer.options.deleteTombstones=false stops writing MpDl records on
deletes - a single filer or shared-store cluster never replays peer
events, so the tombstones are dead weight. The tradeoff is explicit:
if a dedicated-store peer does replicate, a stale event can resurrect
a deleted path.
* filer: drop delete tombstones entirely
Deletes no longer write MpDl records, and the pending-repair machinery
they needed is gone with them: per-mutation metadata overhead is now
zero for every store topology. Ordering information lives only on the
entry itself via commit_ts_ns.
The accepted tradeoff: a dedicated-store peer replaying an event older
than a delete can resurrect the path, and a kept directory no longer
fences stale child creates. Recorded tombstones and MpTs records left
by older builds are still read, so upgrades keep the protections they
already have.
* filer: stop reading legacy per-path records too
Drops the MpTs/MpDl read path along with the writes: meta_path_ts.go
keeps only the entry-field readers (CommitTsNs, EntryCommitTsNs,
localCommitTsNs). Entries written before stamping existed fall back to
mtime, and tombstones or commit records older builds left behind no
longer fence replayed events.
* filer: trim commit timestamp comments
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer: cover the storage rule TTL on the object transaction write path
An object written through ObjectTransaction used to land with ttlSec 0
even under an fs.configure TTL rule, while the same object written
through CreateEntry got the rule's TTL. Guard the shared stamping so the
two paths cannot drift apart again.
* filer: apply the path's storage rule to an appended entry
AppendToEntry resolved the storage option from the path - so its chunks
land on a TTL volume under an fs.configure TTL rule - but never stamped
the rule's TTL on the entry it creates, leaving an entry that outlives
its data. Route it through applyStorageDefaultsToEntry, which now feeds
the entry's own TTL into the option so the placement an existing entry's
appended chunks get is unchanged.
* filer: apply the path's storage rule to a completed TUS upload
The PATCH path resolves the storage option from the target, so a TUS
upload into an fs.configure TTL prefix writes its chunks to a TTL volume,
but completion built the final entry with ttlSec 0 - the entry outlived
the data it pointed at. Stamp it through applyStorageDefaultsToEntry,
which also subsumes the hand-rolled read-only check and supplies the
rule's name-length limit.
* filer: apply the destination's storage option TTL to a copied entry
The copy handler re-uploads the source's chunks under the destination's
storage option, so a copy into an fs.configure TTL prefix already lands
its data on a TTL volume. The entry, though, carried the source's ttlSec
- 0 for a source outside the prefix, or the source's own TTL where the
two rules differ - so it never expired with the data it pointed at. Take
the TTL from the same option the chunks were placed with, after the
data-only copy has restored the destination's metadata.
* fix: apply S3 Expiry Metadata
* add test Header X-Seaweedfs-Expires-S3
* resolve comments
* test entry lookup by mtime
* filer: skip s3 expiry stamp on versioned entries
The s3 expiry path skips entries carrying a version id, so stamping one
takes away its expiry rather than moving it onto mtime. Files under
.versions/ are written once, so crtime already tracks their needles.
---------
Co-authored-by: Konstantin Lebedev <whitefox@mayflower.work>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* feat(filer): add atime field and TouchAccessTime RPC to filer proto
Introduce POSIX-style access-time tracking on the filer:
- FuseAttributes gains atime (field 22) and atime_ns (field 23).
- New TouchAccessTime RPC (and Touch{Access,Time}{Request,Response})
lets read paths bump atime without going through UpdateEntry's
chunk-rewrite/EqualEntry short-circuit.
Additive proto changes only; zero atime is treated as unset and
existing clients are unaffected. Java client proto is kept in lock
step.
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(filer): wire Atime through Attr codec with mtime fallback
Add Attr.Atime and round-trip it through EntryAttributeToPb /
EntryAttributeToExistingPb / PbToEntryAttribute. A zero proto atime
decodes as Mtime, so legacy entries report a sensible value and
freshly-created/updated entries default Atime to Mtime when callers
do not set it explicitly.
CreateEntry and UpdateEntry stamp Atime = Mtime (or Crtime) when it
is zero. TouchAccessTime later bypasses this path to write atime
alone via Store.UpdateEntry.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(filer): preserve atime in first epoch second on decode
The Atime decode branch previously treated any attr.Atime == 0 as
unset and overwrote it with Mtime, which drops valid timestamps in
the first second of the unix epoch where attr.Atime is 0 but
attr.AtimeNs > 0. Check both fields so we only fall back to Mtime
when both are zero.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* test: add pjdfstest POSIX compliance suite
Adds a script and CI workflow that runs the upstream pjdfstest POSIX
compliance test suite against a SeaweedFS FUSE mount. The script starts
a self-contained `weed mini` server, mounts the filesystem with
`weed mount`, builds pjdfstest from source, and runs it under prove(1).
* fix: address review feedback on pjdfstest setup
- Use github.ref instead of github.head_ref in concurrency group so
push events get a stable group key
- Add explicit timeout check after filer readiness polling loop
- Refresh pjdfstest checkout when PJDFSTEST_REPO or PJDFSTEST_REF are
overridden instead of silently reusing stale sources
* test: add Docker-based pjdfstest for faster iteration
Adds a docker-compose setup that reuses the existing e2e image pattern:
- master, volume, filer services from chrislusf/seaweedfs:e2e
- mount service extended with pjdfstest baked in (Dockerfile extends e2e)
- Tests run via `docker compose exec mount /run.sh`
- CI workflow gains a parallel `pjdfstest (docker)` job
This avoids building Go from scratch on each iteration — just rebuild the
e2e image once and iterate on the compose stack.
* fix: address second round of review feedback
- Use mktemp for WORK_DIR so each run starts with a clean filer state
- Pin PJDFSTEST_REF to immutable commit (03eb257) instead of master
- Use cp -r instead of cp -a to avoid preserving ownership during setup
* fix: address CI failure and third round of review feedback
- Fix docker job: fall back to plain docker build when buildx cache
export is not supported (default docker driver in some CI runners)
- Use /healthz endpoint for filer healthcheck in docker-compose
- Copy logs to a fixed path (/tmp/seaweedfs-pjdfstest-logs/) for
reliable CI artifact upload when WORK_DIR is a mktemp path
* fix(mount): improve POSIX compliance for FUSE mount
Address several POSIX compliance gaps surfaced by the pjdfstest suite:
1. Filename length limit: reduce from 4096 to 255 bytes (NAME_MAX),
returning ENAMETOOLONG for longer names.
2. SUID/SGID clearing on write: clear setuid/setgid bits when a
non-root user writes to a file (POSIX requirement).
3. SUID/SGID clearing on chown: clear setuid/setgid bits when file
ownership changes by a non-root user.
4. Sticky bit enforcement: add checkStickyBit helper and enforce it
in Unlink, Rmdir, and Rename — only file owner, directory owner,
or root may delete entries in sticky directories.
5. ctime (inode change time) tracking: add ctime field to the
FuseAttributes protobuf message and filer.Attr struct. Update
ctime on all metadata-modifying operations (SetAttr, Write/flush,
Link, Create, Mkdir, Mknod, Symlink, Truncate). Fall back to
mtime for backward compatibility when ctime is 0.
* fix: add -T flag to docker compose exec for CI
Disable TTY allocation in the pjdfstest docker job since GitHub
Actions runners have no interactive TTY.
* fix(mount): update parent directory mtime/ctime on entry changes
POSIX requires that a directory's st_mtime and st_ctime be updated
whenever entries are created or removed within it. Add
touchDirMtimeCtime() helper and call it after:
- mkdir, rmdir
- create (including deferred creates), mknod, unlink
- symlink, link
- rename (both source and destination directories)
This fixes pjdfstest failures in mkdir/00, mkfifo/00, mknod/00,
mknod/11, open/00, symlink/00, link/00, and rmdir/00.
* fix(mount): enforce sticky bit on destination directory during rename
POSIX requires sticky-bit enforcement on both source and destination
directories during rename. When the destination directory has the
sticky bit set and a target entry already exists, only the file owner,
directory owner, or root may replace it.
* fix(mount): add in-memory atime tracking for POSIX compliance
Track atime separately from mtime using a bounded in-memory map
(capped at 8192 entries with random eviction). atime is not persisted
to the filer — it's only kept in mount memory to satisfy POSIX stat
requirements for utimensat and related syscalls.
This fixes utimensat/00, utimensat/02, utimensat/04, utimensat/05,
and utimensat/09 pjdfstest failures where atime was incorrectly
aliased to mtime.
* fix(mount): restore long filename support, fix permission checks
- Restore 4096-byte filename limit (was incorrectly reduced to 255).
SeaweedFS stores names as protobuf strings with no ext4-style
constraint — the 255 limit is not applicable.
- Fix AcquireHandle permission check to map filer uid/gid to local
space before calling hasAccess, matching the pattern used in Access().
- Fix hasAccess fallback when supplementary group lookup fails: fall
through to "other" permissions instead of requiring both group AND
other to match, which was overly restrictive for non-existent UIDs.
* fix(mount): fix permission checks and enforce NAME_MAX=255
- Fix AcquireHandle to map uid/gid from filer-space to local-space
before calling hasAccess, consistent with the Access handler.
- Fix hasAccess fallback when supplementary group lookup fails: use
"other" permissions only instead of requiring both group AND other.
- Enforce NAME_MAX=255 with a comment explaining the Linux FUSE kernel
module's VFS-layer limit. Files >255 bytes can be created via direct
FUSE protocol calls but can't be stat'd/chmod'd via normal syscalls.
- Don't call touchDirMtimeCtime for deferred creates to avoid
invalidating the just-cached entry via filer metadata events.
* ci: mark pjdfstest steps as continue-on-error
The pjdfstest suite has known failures (Linux FUSE NAME_MAX=255
limitation, hard link nlink/ctime tracking, nanosecond precision)
that cannot be fixed in the mount layer. Mark the test steps as
continue-on-error so the CI job reports results without blocking.
* ci: increase pjdfstest bare metal timeout to 90 minutes
* fix: use full commit hash for PJDFSTEST_REF in run.sh
Short hashes cannot be resolved by git fetch --depth 1 on shallow
clones. Use the full 40-char SHA.
* test: add pjdfstest known failures skip list
Add known_failures.txt listing 33 test files that cannot pass due to:
- Linux FUSE kernel NAME_MAX=255 (26 files)
- Hard link nlink/ctime tracking requiring filer changes (3 files)
- Parent dir mtime on deferred create (1 file)
- Directory rename permission edge case (1 file)
- rmdir after hard link unlink (1 file)
- Nanosecond timestamp precision (1 file)
Both run.sh and run_inside_container.sh now skip these tests when
running the full suite. Any failure in a non-skipped test will cause
CI to fail, catching regressions immediately.
Remove continue-on-error from CI steps since the skip list handles
known failures.
Result: 204 test files, 8380 tests, all passing.
* ci: remove bare metal pjdfstest job, keep Docker only
The bare metal job consistently gets stuck past its timeout due to
weed processes not exiting cleanly. The Docker job covers the same
tests reliably and runs faster.
* filer: reduce allocations in MatchStorageRule
Optimize MatchStorageRule to avoid allocations in common cases:
- Return singleton emptyPathConf when no rules match (zero allocations)
- Return existing rule directly when only one rule matches (zero allocations)
- Only allocate and merge when multiple rules match (rare case)
Based on heap profile analysis showing 111MB allocated from 1.64M calls
to this function during 180 seconds of operation.
* filer: add fast path for getActualStore when no path-specific stores
Add hasPathSpecificStore flag to FilerStoreWrapper to skip
the MatchPrefix() call and []byte(path) conversion when no
path-specific stores are configured (the common case).
Based on heap profile analysis showing 1.39M calls to this
function during 180 seconds of operation, each requiring a
string-to-byte slice conversion for the MatchPrefix call.
* filer/foundationdb: use sync.Pool for tuple allocation in genKey
Use sync.Pool to reuse tuple.Tuple slices in genKey(), reducing
allocation overhead for every FoundationDB operation.
Based on heap profile analysis showing 102MB allocated from 1.79M
calls to genKey() during 180 seconds of operation. The Pack() call
still allocates internally, but this reduces the tuple slice
allocation overhead by ~50%.
* filer: use sync.Pool for protobuf Entry and FuseAttributes
Add pooling for filer_pb.Entry and filer_pb.FuseAttributes in
EncodeAttributesAndChunks and DecodeAttributesAndChunks to reduce
allocations during filer store operations.
Changes:
- Add pbEntryPool with pre-allocated FuseAttributes
- Add EntryAttributeToExistingPb for in-place attribute conversion
- Update ToExistingProtoEntry to reuse existing Attributes when available
Based on heap profile showing:
- EncodeAttributesAndChunks: 69.5MB cumulative
- DecodeAttributesAndChunks: 46.5MB cumulative
- EntryAttributeToPb: 47.5MB flat allocations
* log_buffer: use sync.Pool for LogEntry in readTs
Add logEntryPool to reuse filer_pb.LogEntry objects in readTs(),
which is called frequently during binary search in ReadFromBuffer.
This function only needs the TsNs field from the unmarshaled entry,
so pooling the LogEntry avoids repeated allocations.
Based on heap profile showing readTs with 188MB cumulative allocations
from timestamp lookups during log buffer reads.
* pb: reduce gRPC metadata allocations in interceptor
Optimize requestIDUnaryInterceptor and WithGrpcClient to reduce
metadata allocations on every gRPC request:
- Use AppendToOutgoingContext instead of NewOutgoingContext + New()
This avoids creating a new map[string]string for single key-value pairs
- Check FromIncomingContext return value before using metadata
Based on heap profile showing metadata operations contributing 0.45GB
(10.5%) of allocations, with requestIDUnaryInterceptor being the main
source at 0.44GB cumulative.
Expected reduction: ~0.2GB from avoiding map allocations per request.
* filer/log_buffer: address code review feedback
- Use proto.Reset() instead of manual field clearing in resetLogEntry
for more idiomatic and comprehensive state clearing
- Add resetPbEntry() call before pool return in error path for
consistency with success path in DecodeAttributesAndChunks
* log_buffer: reduce PreviousBufferCount from 32 to 4
Reduce the number of retained previous buffers from 32 to 4.
Each buffer is 8MB, so this reduces the maximum retained memory
from 256MB to 32MB for previous buffers.
Most subscribers catch up quickly, so 4 buffers (32MB) should
be sufficient while significantly reducing memory footprint.
* filer/foundationdb: use defer for tuple pool cleanup in genKey
Refactor genKey to use defer for returning the pooled tuple.
This ensures the pooled object is always returned even if
store.seaweedfsDir.Pack panics, making the code more robust.
Also simplifies the code by removing the temporary variable.
* filer: early-stop MatchStorageRule prescan after 2 matches
Stop the prescan callback after finding 2 matches since we only
need to know if there are 0, 1, or multiple matches. This avoids
unnecessarily scanning the rest of the trie when many rules exist.
* fix: address critical code review issues
filer_conf.go:
- Remove mutable singleton emptyPathConf that could corrupt shared state
- Return fresh copy for no-match case and cloned copy for single-match case
- Add clonePathConf helper to create shallow copies safely
grpc_client_server.go:
- Remove incorrect AppendToOutgoingContext call in server interceptor
(that API is for outbound client calls, not server-side handlers)
- Rely on request_id.Set and SetTrailer for request ID propagation
* fix: treat FilerConf_PathConf as immutable
Fix callers that were incorrectly mutating the returned PathConf:
- filer_server_handlers_write.go: Use local variable for MaxFileNameLength
instead of mutating the shared rule
- command_s3_bucket_quota_check.go: Create new PathConf explicitly when
modifying config instead of mutating the returned one
This allows MatchStorageRule to safely return the singleton or direct
references without copying, restoring the memory optimization.
Callers must NOT mutate the returned *FilerConf_PathConf.
* filer: add ClonePathConf helper for creating mutable copies
Add reusable ClonePathConf function that creates a mutable copy of
a PathConf. This is useful when callers need to modify config before
calling SetLocationConf.
Update command_s3_bucket_quota_check.go to use the new helper.
Also fix redundant return statement in DeleteLocationConf.
* fmt
* filer: fix protobuf pool reset to clear internal fields
Address code review feedback:
1. resetPbEntry/resetFuseAttributes: Use struct assignment (*e = T{})
instead of field-by-field reset to clear protobuf internal fields
(unknownFields, sizeCache) that would otherwise accumulate across
pool reuses, causing data corruption or memory bloat.
2. EntryAttributeToExistingPb: Add nil guard for attr parameter to
prevent panic if caller passes nil.
* log_buffer: reset logEntry before pool return in error path
For consistency with success path, reset the logEntry before putting
it back in the pool in the error path. This prevents the pooled object
from holding references to partially unmarshaled data.
* filer: optimize MatchStorageRule and document ClonePathConf
1. Avoid double []byte(path) conversion in multi-match case by
converting once and reusing pathBytes.
2. Add IMPORTANT comment to ClonePathConf documenting that it must
be kept in sync with filer_pb.FilerConf_PathConf fields when
the protobuf evolves.
* filer/log_buffer: fix data race and use defer for pool cleanup
1. entry_codec.go EncodeAttributesAndChunks: Fix critical data race -
proto.Marshal may return a slice sharing memory with the message.
Copy the data before returning message to pool to prevent corruption.
2. entry_codec.go DecodeAttributesAndChunks: Use defer for cleaner
pool management, ensuring message is always returned to pool.
3. log_buffer.go readTs: Use defer for pool cleanup, removing
duplicated resetLogEntry/Put calls in success and error paths.
* filer: fix ClonePathConf field order and add comprehensive test
1. Fix field order in ClonePathConf to match protobuf struct definition
(WormGracePeriodSeconds before WormRetentionTimeSeconds).
2. Add TestClonePathConf that constructs a fully-populated PathConf,
calls ClonePathConf, and asserts equality of all exported fields.
This will catch future schema drift when new fields are added.
3. Add TestClonePathConfNil to verify nil handling.
* filer: use reflection in ClonePathConf test to detect schema drift
Replace hardcoded field comparisons with reflection-based comparison.
This automatically catches:
1. New fields added to the protobuf but not copied in ClonePathConf
2. Missing non-zero test values for any exported field
The test iterates over all exported fields using reflect and compares
src vs clone values, failing if any field differs.
* filer: update EntryAttributeToExistingPb comment to reflect nil handling
The function safely handles nil attr by returning early, but the comment
incorrectly stated 'attr must not be nil'. Update comment to accurately
describe the defensive behavior.
* Fix review feedback: restore request ID propagation and remove redundant resets
1. grpc_client_server.go: Restore AppendToOutgoingContext for request ID
so handlers making downstream gRPC calls will automatically propagate
the request ID to downstream services.
2. entry_codec.go: Remove redundant resetPbEntry calls after Get.
The defer block ensures reset before Put, so next Get receives clean object.
3. log_buffer.go: Remove redundant resetLogEntry call after Get for
same reason - defer already handles reset before Put.
* do delete expired entries on s3 list request
https://github.com/seaweedfs/seaweedfs/issues/6837
* disable delete expires s3 entry in filer
* pass opt allowDeleteObjectsByTTL to all servers
* delete on get and head
* add lifecycle expiration s3 tests
* fix opt allowDeleteObjectsByTTL for server
* fix test lifecycle expiration
* fix IsExpired
* fix locationPrefix for updateEntriesTTL
* fix s3tests
* resolv coderabbitai
* GetS3ExpireTime on filer
* go mod
* clear TtlSeconds for volume
* move s3 delete expired entry to filer
* filer delete meta and data
* del unusing func removeExpiredObject
* test s3 put
* test s3 put multipart
* allowDeleteObjectsByTTL by default
* fix pipline tests
* rm dublicate SeaweedFSExpiresS3
* revert expiration tests
* fix updateTTL
* rm log
* resolv comment
* fix delete version object
* fix S3Versioning
* fix delete on FindEntry
* fix delete chunks
* fix sqlite not support concurrent writes/reads
* move deletion out of listing transaction; delete entries and empty folders
* Revert "fix sqlite not support concurrent writes/reads"
This reverts commit 5d5da14e0e.
* clearer handling on recursive empty directory deletion
* handle listing errors
* strut copying
* reuse code to delete empty folders
* use iterative approach with a queue to avoid recursive WithFilerClient calls
* stop a gRPC stream from the client-side callback is to return a specific error, e.g., io.EOF
* still issue UpdateEntry when the flag must be added
* errors join
* join path
* cleaner
* add context, sort directories by depth (deepest first) to avoid redundant checks
* batched operation, refactoring
* prevent deleting bucket
* constant
* reuse code
* more logging
* refactoring
* s3 TTL time
* Safety check
---------
Co-authored-by: chrislu <chris.lu@gmail.com>
