WORM was checked before the entry lock was acquired, so a concurrent
writer could enable it while an ingest, repack, or plain HTTP overwrite
waited, and the commit then replaced a protected entry. Repack now
checks under its lock, and ingest and saveMetaData recheck at commit
time.
gRPC writers, renames, and repack already took the per-path entry lock,
but plain HTTP overwrites committed without it: an overwrite landing
between repack's read and its update was silently replaced, orphaning
its chunks. Take the lock around the saveMetaData and format-ingest
commits, so repack's exclusive hold spans every writer.
Views validated conditional requests against the media entry's ETag,
so re-ingesting identical bytes with a different sidecar changed the
playlist and segment boundaries while clients kept getting 304s. Fold
the encoded layout and the request's view parameters into the ETag the
view serves and checks.
S3 object versions may share one chunk list, so deleting the old chunks
after a repack could corrupt sibling versions; reject those entries
until chunk ownership is tracked.
New chunks also carried the full original TTL, restarting needle
expiry at repack time while entry expiry stayed anchored to creation: a
nearly expired entry left chunks stored for almost a full extra span.
Assign the remaining lifetime instead, and reject entries already past
it.
- namespace the query parameters as format.ingest, format.repack and
format.view, following the mv.from/cp.from dotted convention, so the
general endpoints cannot collide with pass-through client parameters;
requests naming both ingest and repack are rejected
- state Accept-Ranges: none on view responses, which always answer with
whole documents or whole extents
- derive the small-content permission from the boundary source instead
of a second positional bool that a call site could silently swap
- validate the hls-ts layout before returning it, making the formattest
invariant enforced rather than emergent
New chunks were assigned with the TTL the request query implied while
the entry kept its own, so repacking a permanent file with ?ttl= made
its chunks expire under permanent metadata. Force the entry TTL onto
the storage option instead.
Filer.UpdateEntry only writes the store, so metadata subscribers never
heard about the new chunk ids while the old ones were queued for
deletion. Emit the update event the way the gRPC UpdateEntry path does.
Extent chunks are buffered in memory, so an absent limit must not mean
unlimited. Also close the repack chunk reader to release its private
reader cache, and drop the arithmetic capacity hint on the extended-map
allocation.
Three hooks, all on the entry's real path so JWT scopes, WORM, and
read-only rules apply unchanged:
- POST /path?format=<name> ingests a multipart index sidecar plus media
and cuts storage chunks on the extents the sidecar declares
- GET /path?view=<name> serves adapter views; rendered documents and
extent streams both ride the normal prefetch path with entry ETag,
preconditions, and HEAD support
- POST /path?repack=<name> derives the layout from the stored bytes and
rewrites the chunks cut on extent boundaries, swapping the entry under
the entry lock and queueing the old chunks for deletion
The layout is advisory: a stale one 404s its views while plain reads
stay untouched. Repack refuses hard-linked, remote, and SSE entries.
Chris Lu