Sniff sat in the mandatory adapter interface but has exactly one
caller, the repack gate, which only Indexer adapters can reach - the
hls-ts implementation was dead code. Move it to a Sniffer capability
discovered by assertion like the others: parquet keeps it, hls-ts
drops it, and repack skips the gate when an adapter cannot sniff.
The stored size is uint64; converting a larger value wrapped negative,
sizing the repack sniff buffer with make([]byte, -1) - a handler panic -
and passing Validate a negative size, which means skip the size check.
Refuse repack and answer views stale instead.
The identity digests covered only chunks, while the view's extent path
would serve from inline Content when present - a gRPC update could set
Content with the chunks and size unchanged and segments were served
from bytes the layout never described. Format entries are never written
with inline content, so treat it as disqualifying: views answer stale,
repack rejects it up front, the extent path no longer reads it, and the
source identity digests it so it cannot appear mid-repack unnoticed.
The commit-time check compared only the chunk list, so a concurrent
change that kept the chunks - clearing the TTL, moving the expiry
anchor, hard-linking, going remote - passed verification, and the swap
paired that fresh metadata with chunks uploaded under the old inputs: a
permanent entry pointing at chunks that still expire. Digest everything
the repack consumed - chunk fingerprint, file size, TTL, both time
anchors, the S3-expiry flag, hard-link and remote state - and answer
409 when any of it moved.
The entry lock is filer-local, so a writer on another filer could
commit between repack's read and its swap, and repack then restored the
old bytes over an acknowledged update. Re-read the entry and revalidate
the chunk fingerprint and WORM right before the swap, answering 409 on
any change, and build the new entry from the fresh read so concurrent
metadata-only updates are carried forward. This shrinks the unguarded
window from the whole repack to the commit itself; closing it entirely
needs owner routing.
The layout binding hashed only offset and file id, so a mutation that
kept both - a truncate shrinking chunk.Size, then a sparse extend back
to the original length - passed both the size and fingerprint checks
and served a stale view. Digest size, modification timestamp, cipher
key, compression, manifest status, and SSE type as well.
The fallback capped the remainder at MaxInt32 seconds, which the volume
TTL grid encodes as 68 years - about 35 days shorter than the entry's
lifetime. For the narrow band nothing can round up within int32, store
the chunks without a volume TTL instead: they outlive the entry rather
than predecease it.
SecondsToTTL truncates to the volume TTL grid, so 3599 remaining
seconds became 59m and anything under a minute became no TTL at all -
permanent chunks under an expiring entry. Round the remaining lifetime
up to the smallest representable value instead, and anchor it the way
FindEntry expires entries: S3-expiring entries age from Mtime, others
from Crtime, so a recently overwritten S3 object is no longer treated
as nearly expired.
Also bind each layout to a digest of the chunk list it described.
Offset writes and appends keep Extended while changing the chunks, so
a same-size partial write used to leave the old playlist and extents
being served over new bytes; the views now detect the mismatch and
answer 404 until the file is re-ingested or repacked.
WORM was checked before the entry lock was acquired, so a concurrent
writer could enable it while an ingest, repack, or plain HTTP overwrite
waited, and the commit then replaced a protected entry. Repack now
checks under its lock, and ingest and saveMetaData recheck at commit
time.
gRPC writers, renames, and repack already took the per-path entry lock,
but plain HTTP overwrites committed without it: an overwrite landing
between repack's read and its update was silently replaced, orphaning
its chunks. Take the lock around the saveMetaData and format-ingest
commits, so repack's exclusive hold spans every writer.
Views validated conditional requests against the media entry's ETag,
so re-ingesting identical bytes with a different sidecar changed the
playlist and segment boundaries while clients kept getting 304s. Fold
the encoded layout and the request's view parameters into the ETag the
view serves and checks.
S3 object versions may share one chunk list, so deleting the old chunks
after a repack could corrupt sibling versions; reject those entries
until chunk ownership is tracked.
New chunks also carried the full original TTL, restarting needle
expiry at repack time while entry expiry stayed anchored to creation: a
nearly expired entry left chunks stored for almost a full extra span.
Assign the remaining lifetime instead, and reject entries already past
it.
- namespace the query parameters as format.ingest, format.repack and
format.view, following the mv.from/cp.from dotted convention, so the
general endpoints cannot collide with pass-through client parameters;
requests naming both ingest and repack are rejected
- state Accept-Ranges: none on view responses, which always answer with
whole documents or whole extents
- derive the small-content permission from the boundary source instead
of a second positional bool that a call site could silently swap
- validate the hls-ts layout before returning it, making the formattest
invariant enforced rather than emergent
New chunks were assigned with the TTL the request query implied while
the entry kept its own, so repacking a permanent file with ?ttl= made
its chunks expire under permanent metadata. Force the entry TTL onto
the storage option instead.
Filer.UpdateEntry only writes the store, so metadata subscribers never
heard about the new chunk ids while the old ones were queued for
deletion. Emit the update event the way the gRPC UpdateEntry path does.
Extent chunks are buffered in memory, so an absent limit must not mean
unlimited. Also close the repack chunk reader to release its private
reader cache, and drop the arithmetic capacity hint on the extended-map
allocation.
Three hooks, all on the entry's real path so JWT scopes, WORM, and
read-only rules apply unchanged:
- POST /path?format=<name> ingests a multipart index sidecar plus media
and cuts storage chunks on the extents the sidecar declares
- GET /path?view=<name> serves adapter views; rendered documents and
extent streams both ride the normal prefetch path with entry ETag,
preconditions, and HEAD support
- POST /path?repack=<name> derives the layout from the stored bytes and
rewrites the chunks cut on extent boundaries, swapping the entry under
the entry lock and queueing the old chunks for deletion
The layout is advisory: a stale one 404s its views while plain reads
stay untouched. Repack refuses hard-linked, remote, and SSE entries.
Chris Lu