* log buffer: wake the interval loop on shutdown instead of sleeping through it
loopInterval parked in time.Sleep(flushInterval) and only re-checked
IsStopping when it woke, so a buffer shut down early kept both loop
goroutines - and the PreviousBufferCount+1 slabs of BufferSize they
reach - alive for up to a full interval afterwards. Select on shutdownCh
against a ticker instead, and give the loops a WaitGroup so a test can
observe that they exit.
* test: release the filers the server tests build
Every helper here left its filer's meta log buffer running, so each test
pinned PreviousBufferCount+1 buffers of BufferSize for the rest of the
run: ~3.5GB of live heap across the package, which overruns the address
space on linux/386 and kills the 32-bit job with an out-of-memory throw.
Thread the test through the helpers so the buffer is shut down on
cleanup, and shut the subscribe harness's filer down outright - its
deletion loop keeps the whole filer reachable otherwise. That harness
quiesces its flush path first, since Filer.Shutdown closes the store a
flush still in flight would write through.
* s3: routed last-version delete removes the emptied .versions directory
The routed versioned delete (routedDeleteSpecificVersion) repoints the
latest pointer and deletes the version file, but unlike the lock-path
fallback (updateLatestVersionAfterDeletion) it never tears down the
.versions/ directory it just emptied. The residue keeps the key's read
path in the self-heal rescan loop: every GET of the deleted key logs
event=surfaced plus a GetObject error until the background
EmptyFolderCleaner gets to the directory — at least two minutes away on
its delay queue, and possibly never (the queue is in-memory, bounded,
and gated on the bucket's allow-empty-folders policy). Veeam's lock
arbitration probes deleted lock keys continuously, so those windows are
always open and the log spam is chronic.
ObjectMutation DELETE gains remove_empty_parent: after the child delete,
the filer best-effort removes the parent directory in the same locked
transaction. Non-recursive on purpose — a concurrent write that lands a
new version fails the removal instead of being lost with it. The routed
last-version delete sets it on the version-file DELETE, matching the
lock-path fallback's contract.
Claude-Session: https://claude.ai/code/session_014mMYAHXZySkCCUfpRFNtSv
* s3: drain empty .versions residue on read heal and in s3.versions.audit
Directories already stranded by pre-teardown deletes (or dropped from
the EmptyFolderCleaner's bounded in-memory queue) previously re-entered
the self-heal rescan on every GET forever: the heal only cleared the
pointer and nothing ever removed the directory, and s3.versions.audit
counted the state as clean.
When the heal rescan finds no remaining version, remove the directory
outright (non-recursive, so orphan children still block and fall back to
the pointer clear) and log event=healed mode=empty_dir_removed; the next
GET takes the clean not-found path. The audit gains an empty category so
the residue is visible, and -heal removes such directories in bulk.
Claude-Session: https://claude.ai/code/session_014mMYAHXZySkCCUfpRFNtSv
* s3: commit a versioned PutObject and its latest pointer in one transaction
A versioned PutObject wrote the version file and flipped the .versions
latest pointer in two separate routed transactions. Fold the
RECOMPUTE_LATEST into the version file's PUT so both commit atomically
under the object's per-path lock: the recompute, applied after the PUT in
the same transaction, scans the directory and sees the new version. A
crash can no longer leave the version present with a stale pointer.
putToFiler now takes a putFinalize describing the finalize step — routed
mutations folded into the PUT, or an afterCreate run under the object
write lock off the ring. Suspended-versioning keeps its afterCreate-only
form; multipart, copy, and delete-marker finalizes are unchanged.
* s3: trim verbose finalize comments
A non-owner filer forwards the whole transaction to the ring owner of route_key, so the owner's per-path lock stays the single serialization point even when the caller's ring view is stale. is_moved bounds forwarding to one hop. The gateway stamps route_key on every routed builder via the shared objectRouteKey helper. Completes taking S3 object mutations off the distributed lock.
A non-versioned metadata-only self-copy (CopyObject with source == destination
and the REPLACE directive) is a read-modify-write of one entry, which is why it
held the distributed lock. It now routes to the owner as a serialized
PATCH_EXTENDED: the owner merges the new managed metadata (set the replacements,
delete the dropped keys) onto a fresh read of the entry under its per-path lock,
so a concurrent change to non-managed keys (legal hold, retention, version id) is
preserved instead of clobbered, and bumps mtime.
PATCH_EXTENDED gains touch_mtime for the mtime bump. Versioned and suspended
self-copies create a new version (already routed via the copy finalize) and the
no-owner bootstrap keep the lock.
A version-specific DELETE (real version or the null version, including
object-lock WORM-checked ones and governance-bypass) now runs as one routed
transaction on the object's owner instead of holding the distributed lock.
For a real version: recompute the .versions pointer excluding the version
(repoint-before-delete, so a crash leaves a recoverable orphan rather than a
dangling pointer), then delete the version file, under the object's per-path lock.
The null version is the regular object entry, deleted directly (no pointer).
Object-lock buckets gate the delete on the version's WORM guards evaluated on the
owner: legal hold (always) + retention (while not elapsed). Governance bypass
scopes the retention guard to COMPLIANCE mode, so the filer allows a
governance-mode delete while still denying compliance and legal hold — the
gateway never reads the version.
Three primitives make this expressible:
- ObjectTransaction.condition_key: evaluate the condition against a named entry
(the version) while the lock stays on lock_key (the object).
- Recompute.exclude_name: omit a child from the scan, to repoint before delete.
- WriteCondition.Clause gate_key/gate_value: scope IF_EXTENDED_TIME_ELAPSED to a
mode, expressing governance bypass without a gateway-side read.
s3: route versioned PutObject finalize off the distributed lock
A versioned write's finalize (flip the .versions pointer to the newest version,
demote the prior latest) now runs as a single RECOMPUTE_LATEST ObjectTransaction
on the object's owner filer, under its per-path lock, instead of the unserialized
updateLatestVersionInDirectory. The version file is written first; the owner
re-derives the pointer by scanning the directory.
RECOMPUTE_LATEST gains size_to_key / mtime_to_key to cache the chosen version's
size and mtime on the pointer, and demote_key / demote_value to stamp the
displaced prior latest (NoncurrentSinceNs for lifecycle) when the pointer moves.
Falls back to updateLatestVersionInDirectory when no owner is known yet.
* filer: let PATCH_EXTENDED replace Entry.content
PATCH_EXTENDED merges extended attributes under the per-path lock, reading the
entry fresh, so concurrent patches to different keys don't clobber each other.
Some single-key state lives in Entry.content rather than an extended attribute
(e.g. the S3 bucket metadata blob). Add set_content/content to the mutation so a
patch can replace content the same way -- read fresh, set content, preserve the
rest -- letting a content write and an extended-attribute write on the same
entry serialize on the lock instead of racing whole-entry rewrites.
* Update weed/server/filer_grpc_server.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* filer: test set_content FileSize sync; note chosen content-patch approach
Cover the FileSize behavior of a set_content patch: a file's size follows the
new content length (including when it shrinks), a directory's stays zero. Also
document, in the bucket-config design, that extending PATCH_EXTENDED with
set_content is the implemented path for content-backed config.
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
A multi-object delete spans many keys that route to different owner filers. The
gateway groups keys by owner and sends one batch per owner; the filer applies
each transaction under its own per-path lock, independent of the others.
A failed transaction (precondition or mutation error) is reported in its own
response without aborting the rest, matching S3 multi-object semantics where
each key succeeds or fails on its own. There is no cross-key atomicity, which S3
batch delete does not require.
Deleting a specific version that happens to be the latest needs the new latest
re-derived from the remaining versions, and that scan must run under the same
lock as the delete. The gateway can't do it atomically across RPCs.
Add a RECOMPUTE_LATEST mutation: it scans a directory under the transaction
lock, picks the child that sorts last (descending) or first by name, copies the
mapped extended keys from it into a pointer entry, and stores its name under
name_to_key. An empty directory clears the pointer keys. The filer stays
mechanical and S3-agnostic: the caller, which knows the versioning scheme,
supplies the sort direction and the key mappings. A missing pointer entry is a
no-op, so a replayed transaction is idempotent.
A versioned object write touches several entries that must change together: the
main object, a delete marker or version file, and the latest pointer on the
.versions directory. Holding a distributed lock across separate RPCs to do this
is what the per-path lock was meant to replace, but a single CreateEntry only
covers one entry.
Add ObjectTransaction: a request carries a lock_key (the object path), an
optional WriteCondition, and an ordered list of mutations (PUT / DELETE /
PATCH_EXTENDED). The filer holds the per-path lock on lock_key for the whole
call, checks the condition against the entry at lock_key, then applies the
mutations in order. Callers route the object's writes to its owner filer so the
lock is authoritative across all of the object's entries.
DELETE and PATCH of an absent entry are no-ops, so a replayed transaction is
idempotent. PUT entries are metadata-scoped; data-bearing writes (chunks) are
written before the transaction, as today.
Chris Lu