Only flushMetadataToFiler applied linkRefresh. After a peer unlinked one
name, the periodic flush kept publishing over the old link count, so the
filer refused every one until a regular flush ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
flushFileMetadata did not check the handle's delete mark. A file
unlinked while open before its deferred create reached the filer was
created again by the next periodic flush, which expects no entry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A delete event from another client marked the inode's open handle deleted
even when other names still held the hard-linked file, so every later
fsync reported success without saving metadata. The delete event's entry
carries the link count the filer decided on, so an alias delete now drops
only that name, keeps the handle on a name this mount knows, and has the
next flush read the survivor's stored entry before publishing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Lookup mapped a second name of a hard-linked file to the existing inode
without adding it to the inode's names, so GetAllPaths missed it and the
final forget left it mapped to a released inode.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An unlink or rename that lands between the name pick and the locks would
otherwise send the flush to the removed name, where the create guard fails.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A flush could read the removed alias name, find nothing, and drop the
pending read of the surviving link, leaving the stale base in place.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
doFlush resolves the name before uploading the data. An unlink of that
alias, or a rename, in the meantime sent the publish to a name the file no
longer has.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The flush does not restamp times, so a dirty entry carrying an earlier
write's ctime moved the filer's ctime back from the link or unlink.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A link change carried into the handle could replace a base already moved
past it by a later acknowledgment. Use the same clock-domain check as
installAckedEntry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With writeback caching a new file is created on the filer by its queued
close, and a listed directory serves the source from the cache, so a link
right after close updated a name the filer did not have yet and failed
with EIO.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With writeback caching, close only queues the flush. Unlink marked the
handle deleted before waiting for that flush, so the flush skipped its
metadata and the surviving link kept the content from before the close.
When another link keeps the file, wait for the queued flush first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the read of the surviving entry right after an alias unlink fails,
the handle kept a base with the old link count and ctime, and every later
flush was refused even after the filer recovered. Leave the hard link id
on the handle and have the next flush read the stored entry and adopt it
before publishing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Removing one of several hard links lowers the stored link count and
stamps a new ctime, and the handle may still hold the removed name. Look up
the entry at the name the handle flushes through and adopt it, so the next
content flush matches the stored entry, writes the right count, and
attribute saves address the surviving name.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Link rewrites the source entry with the new hard link id, count and times,
but an open handle kept its old base and entry. Its next conditional
content flush was refused (EIO), and an unconditional one would have
dropped the hard link id. Move the handle onto the stored entry when only
the link fields and times changed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Unlink marks the inode's open handle deleted so no flush recreates the
removed name. When the entry was one of several hard links the inode lives
on under another name, yet the mark stayed and every later fsync and close
returned OK without publishing metadata. Lift the unlink's own mark once
the name is gone and the inode still has a path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: sign every operation subresource in signature V2
The V2 canonical resource only kept the subresources inherited from the
original allowlist, so tagging, retention, legal-hold, cors, encryption,
object-lock and the other keys the router dispatches on were left out of
the signature. A V2 signature issued for one request therefore also
verified that request with one of those keys appended, running a
different operation under the signer's identity.
Add every operation-selecting subresource to the signed list. Clients
that already sign tagging, cors, object-lock or replication in V2 now
verify as well.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: sign every value of a repeated signature V2 subresource
canonicalizedResourceV2 kept only the last value of a repeated query key,
while handlers read the first. A request signed for versionId=A could be
sent as versionId=B&versionId=A and act on version B. Sign every value in
order, as V2 clients do, so a second value no longer verifies.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(mount): preserve acknowledged chunks during attribute updates
* mount: keep the WFS field block aligned
Adding the publication-lock fields inside the aligned block made gofmt
re-indent 25 unrelated lines. Put them in their own group after
dirIdleEvict, matching the enterprise layout, so syncs do not conflict there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: finish attribute saves through a FUSE interrupt
saveEntryAttributes derived its RPC context from the syscall's cancel
channel, so an interrupt while the conditional UpdateEntry was at the
owner returned EIO for an update the owner then applied, and skipped the
local install and retention. Keep the metadata deadline but drop the
cancellation, like the other metadata mutations.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: map attribute CAS ids in place instead of cloning the entry
saveEntryAttributes cloned the whole entry three times per attempt only
to rewrite uid/gid for the filer, and each clone is as large as the
chunk list. Switch the ids in place and restore the exact local values
after the call, keeping one clone. The test now checks the reply carries
the local uid.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: compare attribute updates against the filer's own uid/gid
With -map.uid/-map.gid, a filer id that is also a mapped local id reads
back unchanged and maps forward to another filer id. The IF_ENTRY_EQUAL
expected entry then never matched, so chmod, utimes and xattr changes on
such files failed with EAGAIN after three attempts, and an update would
have rewritten the owner. Keep the filer ids read with the entry for the
ids the syscall leaves alone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(mount): align attribute retention with read lock order
* mount: take the handle lock before entryLock in the attribute snapshot
attributeEntrySnapshot held entryLock while it queued for the shared
handle lock. A Read holding that lock waits on entryLock, the snapshot
waits behind a queued exclusive (an invalidation or a Write), and the
exclusive waits on the Read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(mount): complete handle lock order and preserve explicit ID intent
* mount: re-read a refused attribute update from the file's owner filer
The filer checks the IF_ENTRY_EQUAL condition on the owner of the path's
write route key. The retry read any filer, so behind one that had not
applied the owner's latest update, every attempt sent the same stale base
and the syscall failed with EAGAIN. Read from the owner once the lock ring
is known, like lookupFilerEntry, but keep the filer's raw ids.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(mount): condition content flushes on the observed entry
* mount: stop retrying a flush the filer refused on its condition
The filer reports a failed CreateEntry condition as FilerError_PRECONDITION_FAILED
in the response, which surfaces as filer_pb.ErrPreconditionFailed rather than a
FailedPrecondition status. The flush retry loop treated it as transient and
replayed the refused flush three more times over 7 s before returning EIO.
The test filer now answers like the real one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: take the handle base from the entry the filer stored
After an acknowledged flush the handle base was the requested entry, but the
filer stores it with a fresh commit_ts_ns (and fills crtime, inode, atime), so
the next IF_ENTRY_EQUAL flush on the same handle never matched: write, fsync,
write, fsync returned EIO, as did close after a periodic metadata flush and the
first flush after an eager create. Use the stored entry from the
acknowledgement's metadata event.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: condition the periodic metadata flush like the close flush
The periodic metadata flush of an open, dirty handle sent the same whole entry
through an unconditional CreateEntry, so a file held open past the flush
interval still overwrote a peer's acknowledged append, and the close flush then
matched the overwritten state. Both paths now publish through one helper that
carries the handle's write condition.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: republish a refused flush once when the stored entry is unchanged
A handle's base can differ from the stored entry without any peer write: a
rename while the file is open restamps the entry under its new name, and with
-map.uid a filer id that is also a mapped local id does not map back to itself.
Each made every flush of the handle fail with EIO. On a miss, read the stored
entry and, if it still holds the observed state apart from name, commit stamp,
id encoding and derived size, retry once with that entry as the condition.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: take the filer's entry once a write open holds the DLM lock
The lock orders this writer after the previous holder, but the holder's
flush may not have reached the local cache yet, so the handle's base
named the entry from before the wait and the conditional flush refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: share the open-existing path of Create
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: open the file a previous DLM holder created during a deferred create
A deferred create never asks the filer, so a create that waited for the
lock went on to publish over a file the previous holder had just made,
and its flush expecting no entry was refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* master: send the lock ring snapshot to mounts
A mount running -dlm is a lock client like the S3 gateway, and needs the
ring on connect to reach a file's owner filer before the next membership
change is broadcast.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: follow the masters' lock ring under -dlm
The lock client only knew the seed filer, so every lock took the forward
hop and the mount had no way to tell which filer owns a file. Subscribe to
the masters named by the filer, like the S3 gateway, and wait briefly for
the first ring before serving.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: read the DLM-ordered entry from the file's owner filer
The filer checks a flush's condition on the owner of the path's write
route key, and another filer may not have applied the previous holder's
flush yet. Look the entry up on that owner once the ring is known.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rust): untrack queued writes before replying
The durable write worker sent each write's result and only then dropped
the batch's pending-write trackers, so a caller could see its write
finish while the write was still tracked as pending. Go untracks in the
caller before writeNeedle2 returns. Drop the trackers before sending the
results so the Rust queue path matches.
test_replicate_delete_cancels_a_queued_write asserted the count was zero
right after the fsync write returned and failed about 1 run in 100 under
load (Rust Unit Tests on master c2625dd).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume(rust): untrack a write rejected for a remounted volume before replying
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume: skip VolumeDelete when its RPC was cancelled before the delete
A caller that cancels (a lost tiering lease, an expired deadline) still
had its admitted delete run in the Go server, including one that waited
behind the copy guard. The Rust server drops a cancelled handler, so
return the context error before the delete starts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: wait until the queued delete blocks on the guard before cancelling
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: verify signed request bodies against the signed payload hash
A SigV4 signature covers the declared x-amz-content-sha256, but only
getRequestDataReader (PutObject, UploadPart) compared the body with it.
Every other handler parsed the body as received, so a signed request,
DeleteObjects for example, could be replayed with a different body.
After a V4 signature verifies, wrap the body so it is handed out only
once all of it has been read and matched against the hash the signature
covered; a decoder that stops reading early never sees unverified bytes.
getRequestDataReader unwraps it and keeps verifying uploads as they
stream.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: answer XAmzContentSHA256Mismatch for refused DeleteObjects bodies
DeleteObjects answered a body that failed its payload-hash check with
InternalError and CompleteMultipartUpload with MalformedXML. Name the
check that failed instead, as PutObject does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: check Content-MD5 on signed request bodies
Signature V2 signs Content-MD5 rather than a payload hash, and a V4
request sent with UNSIGNED-PAYLOAD may still sign it, but bodies read
outside the upload path never compared it with the body. Check it with
the payload hash once a V2 or V4 signature verifies: BadDigest on a
mismatch, InvalidDigest when it is malformed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: verify presigned uploads against a signed payload hash
A presigned V4 URL can sign X-Amz-Content-Sha256 as a query parameter,
but getRequestDataReader compared upload bodies only with the header of
that name, so the signed hash went unchecked. Verify uploads against the
hash the signature covered.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: check Content-MD5 on bodies buffered for their payload hash
IAM and other non-S3 requests signed without x-amz-content-sha256 have
their body buffered and hashed during signature verification. That
buffer counted as already bound, so their Content-MD5 was never checked.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: check Content-MD5 on directory marker bodies
A PUT of a key ending in "/" stores its body on the directory entry
without going through putToFiler, so its Content-MD5 was never compared
with the body. Answer BadDigest on a mismatch, as putToFiler does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: wrap and unwrap signed bodies through helpers
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* shell: confirm trim survivors still serve the volume before deleting
volume.fix.replication compared the doomed replica's index with every
survivor and then deleted it unconditionally. A survivor that went away
while the indexes were read still let the delete through, leaving fewer
copies than the check had counted. Re-read each survivor's VolumeStatus
after the index checks and skip the trim if any no longer has the volume.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: test a trim whose survivor goes away after its index read
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3: ignore unsigned object headers on POST Object
The POST policy signs form fields only, but the upload read tagging,
user metadata, storage class, object lock and the stored content
headers from the request headers as well. Drop them before the form
fields are applied.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: test POST Object request headers against signed form fields
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: answer PutObjectLegalHold on a missing key with NoSuchKey
In a versioned bucket a key with no latest version resolves to
ErrLatestVersionNotFound, which the legal hold handler did not map, so
the request failed with 500 InternalError. PutObjectRetention already
maps it to NoSuchKey.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: keep failed reads distinct when resolving an object lock target
getObjectEntryForUpdate turned every lookup error into a not-found
sentinel, so a failed filer read made PutObjectRetention and
PutObjectLegalHold answer NoSuchKey for an object that exists. Only a
genuine miss becomes the sentinel now, matching getObjectEntry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: answer object lock calls on a delete marker with NoSuchKey
PutObjectLegalHold and PutObjectRetention on a key whose current
version is a delete marker returned 200 and wrote the lock onto the
marker, and the matching GETs read it back. A key whose current version
is a delete marker does not exist, and delete markers take no lock, so
these calls now answer NoSuchKey like the tagging and ACL handlers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: let a lock recorded on a delete marker not block its removal
Delete markers are not WORM-protected, but a legal hold or retention
written onto one (possible before the previous change) made both the
gateway check and the routed delete's filer-side WORM guard refuse to
remove the marker, so the object could not be restored and lifecycle
could not clean the expired marker up.
Lock enforcement now ignores a lock on a delete marker, and the routed
version deletes in DeleteObject and lifecycle drop the WORM guard when
the version they read is a marker. A version id never changes kind, so
the marker read is the one deleted; the null version keeps its guard
because a PUT can replace it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The master, volume, volume-rust, server, mini, filer and s3 cases and the
fallback forwarded the user arguments as an unquoted $@, so the shell split
each one at whitespace and globbed it against /data. An -admin.password with
a space reached weed as two words, and weed ignored every flag after the
stray word. Quote "$@" as the other cases already do; $ARGS stays unquoted
since the script builds it from its own words.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* replication: let the local sink check a delete against the live source
LocalSink removed the destination path for any delete it was handed. A
delete that reaches the sink after the source recreated the path (a peer
filer's event delivered late, or a historical event replayed) removed the
newer file, and no later event brings it back.
DeleteEntryVersion receives the deleted entry. Before removing an existing
destination, the local sink looks up the source path and keeps the
destination when the source holds an entry of the same kind committed
after the deleted version (commit_ts_ns, or mtime for unstamped entries).
A live entry that is the deleted version itself or older means the source
store lags the event, so the delete proceeds. A failed lookup returns the
error so the event is retried instead of deleting a file that may be live.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.sync, filer.replicate: hand the deleted version to the sink
Pure deletes, renames without a native move and moves out of the
watched directory now go through sink.DeleteEntryVersion with the old
entry, so a sink that checks the live source (the local sink) can keep a
path the source recreated after the deleted version. Other sinks fall
back to DeleteEntry unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* replication: order the local sink check by the delete's mutation time
A source filer can hold a version written concurrently with the delete:
newer than the deleted entry but older than the delete itself. The
source's own replay applies the delete over it, so keeping the backup
copy because it outranks the deleted entry left a file the source no
longer has.
The sink now gets the delete's mutation time (MutationTsNs, else TsNs)
and keeps the destination only when the live entry was committed after
it, the order Filer.Replay keeps. That also covers a stamped replacement
of an unstamped entry whose mtime did not move. filer.replicate events
carry no time, so they keep the deleted entry's stamp as the bound, and
mtime only when nothing is stamped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* replication: count an unstamped live entry from its mtime
A source filer that predates commit stamps stores entries with no
CommitTsNs, so comparing the live stamp with the delete's time never
kept a replacement there. An unstamped live entry now counts from its
mtime against the delete's time.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* replication: keep a stamped live entry when the deleted one is unstamped and the delete has no time
filer.replicate gets no delete time from the notification queue. When the
deleted entry has no commit stamp either, a stamped live entry was written
after it, so its mtime no longer decides.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The line parser only trimmed quotes, so `key = "abc\n"` gave the Rust
volume server the bytes `abc\n` with a literal backslash while the Go
servers sign with a newline, and every JWT failed. Literal strings kept
their quotes and a trailing comment stayed part of the value.
Decode each value with the TOML parser; text it rejects keeps the old
quote trimming.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* operation: probe octet-stream uploads for gzip like untyped ones
The 128-byte gzip probe ran only when the caller passed no MimeType, so an
upload labelled application/octet-stream or binary/octet-stream was never
compressed. S3 PutObject and UploadPart pass the request Content-Type (or the
octet-stream that mimeDetect fills in), so every binary S3 upload skipped
compression that the same bytes get through the filer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* operation: check the probed upload keeps the caller's content type
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* wdclient: keep a shared volume lookup alive when its first caller leaves
Concurrent misses for the same volumes share one provider lookup, but it
ran on the first caller's context, so that caller's cancellation failed
every request waiting on it. The lookup now runs on a context no single
caller can cancel, and each caller stops waiting when its own context ends.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* wdclient: wait for the second caller to join before canceling the first
A fixed sleep did not prove the second caller had joined the shared lookup:
a late caller could hit the filled cache and pass without waiting. The test
now waits until the caller reads its own Done channel, which happens only
after DoChan registered it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: let the fake-master test helper serve any master
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: gzip metadata log pieces before uploading them
Every metadata event carries the old and new entry with their whole chunk
lists, so a file grown by small fsynced appends logs the same chunk records
again and again: 2000 appends of 256 bytes through weed mount wrote 34.8 MB
of log. The uploader only compresses when the first 128 bytes compress, and
a log piece starts with fresh file ids and ETags, so it stored the log as is.
The filer now gzips each piece itself and keeps the compressed bytes when
they save at least 10%. After the same workload the volumes hold 3.05 MB
instead of 35.2 MB, file data included. Readers already honor the chunk's
IsCompressed flag on every path (filer-side decode, log file refs read
straight from volume servers), and the assign request sizes the bytes
actually stored.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* mount: keep the filer status in the flush create error
flushMetadataToFiler formatted the CreateEntry error with %v, so callers
classifying it saw only the message, which includes the file path. Wrap
it so the gRPC status the filer sent stays reachable.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: retry a writeback metadata flush until the filer answers
With -writebackCache, close() returns before the chunk references reach
the filer. The deferred flush gave up after four attempts, about a minute
once each attempt's own filer retries are counted, then released the
handle: a longer filer outage dropped the file even though write() and
close() had succeeded, and its uploaded chunks became orphans.
Keep retrying for as long as the filer is unreachable, as the writeback
chunk uploads already do. A filer that answers with an error still ends
the flush after the bounded attempts, and an unlink of the draining file
still ends it on the next attempt.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: retry only an unanswered filer call without a bound
The unbounded phase used util.IsTransientError, which falls back to
matching the error text. A refusal the filer sends in its response
(read-only, name too long, a plain error string) has no gRPC status, and
the flush error carries the file path, so a file named like
"unavailable-transport.log" kept retrying against a filer that answered.
Retry without a bound only when the call failed with a gRPC Unavailable
status: the dial, the connection or the mount's call deadline gave out.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: deny signed reads that name a version file by its physical path
A stored version lives at <key>.versions/v_<id>, and that path is also a
valid object key. Anonymous reads already refuse it; signed GET, HEAD,
copy sources and object subresources did not, so a caller without
s3:GetObjectVersion, or under a Deny on the logical key, could read any
version through it. Apply the same check in the shared entry recheck.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: recheck a version file key before answering a conditional read
A conditional GET or HEAD answered 304 or 412 with the ETag before the
handler's entry recheck, so the physical version path still confirmed a
version's ETag to signed callers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: look up version file keys in the lookup-based policy recheck
Retention, legal hold, tagging writes and deletes only fetched the entry
for the recheck when the bucket had a policy, so without one they still
acted on a version file named by its physical path. Fetch it whenever
the key has the version file shape.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: refuse a directory-marker delete that names a version file
A trailing slash kept "<key>.versions/v_<id>/" out of the version file
shape check, and the lookup recheck misses the entry for that form, so a
DELETE reached deleteDirectoryMarker and removed a chunkless version or
delete marker without s3:DeleteObjectVersion.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The ring tests used 127.0.0.1:18889 as an unreachable peer, but its gRPC
port 28889 is a common filer port: on a shared CI host a live filer there
accepted the forwarded exclusive create, and
TestCreateEntryExclusiveLeavesNonOwner failed. The peer now listens on a
port the test owns and drops every connection.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* vector: share one Lance session across reads
Each request opened its table in a session of its own, so every query
loaded and decoded the index again. Get, list and query now open in one
session the server keeps, sized by -index-cache-mb and
-metadata-cache-mb; writes keep a session of their own.
Lance keys some of its cached entries, such as a version's index list,
by location and version alone, and an index dropped and created again at
its location numbers its versions from 1 again: a shared session answered
a query on the new table with the old table's index. CreateIndex now
marks each table with an id in its schema metadata, and the session
starts over when a location turns up holding another table.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: share a version lease among the reads of that version
Every get, list and query tagged the version it read and deleted the tag
afterwards: two filer writes and a manifest read per request. Reads of
one version now share one tag, which goes 10 s after the last read took
it; a tag is reused for at most 5 minutes, so a busy version never
reaches the reapers' TTLs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: keep opened tables for the next read
A read built the table's store and read its latest manifest each time.
Reads now keep the opened table by location and encryption, up to 1024,
and move it to the latest version, which lists the versions and takes
the manifest from the session. A handle opened in an earlier session is
not reused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: key shared leases by manifest, and take one per version at once
A table dropped and created again within the linger reused, for a
version number the old table had, a lease whose tag the drop had
deleted. Leases are now keyed by the manifest's ETag as well. Reads that
arrive before a version has a lease wait for the first to take it,
rather than each writing a tag.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: drop kept tables when the shared session starts over
A kept table holds the session it was opened in, so a replaced session
and its caches lived on in the handles of other tables until they were
evicted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: prune shared lease slots only when they double
Every lease lookup scanned all slots under the map's lock, a cost that
grew with the versions read in the last linger.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: shrink the lease slot map after a prune
After a burst the map kept its peak allocation, which every later prune
scanned.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: keep a lease slot a read has taken through a prune
A read that took a slot and had not yet locked it could see the slot
pruned, and then pin through it while the next read pinned through a
new slot, writing a second tag.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* admin: restyle the UI as a calm console
Grey canvas with white sheets, a docked sidebar that marks the current
page with an indigo tint, ink primary buttons, tinted status pills,
dense tables with tabular numbers and spectrum capacity meters. The
styles are design tokens layered over Bootstrap's own variables, with
light and dark values, so the page templates stay unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: add a light, dark and automatic theme switch
The choice is kept in localStorage and applied before the page paints,
on the sign-in page too; automatic follows the system setting.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: drop hard-coded colors from page templates
The volume id link, utilization track and plugin form borders used
fixed light-theme colors; sparklines now use the data palette.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: theme the plugin job detail panel background
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: keep the selected radio dot visible in dark mode
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: cycle the theme from memory when storage is blocked
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: keep success progress bars green
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: keep keyboard focus visible on the toolbar togglers
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: use contrasting text on danger and success buttons in dark mode
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: lead the dashboard with cluster health and one row of tiles
The header now says in one line whether the cluster is healthy and how
many masters, volume servers and filers report. The two rows of stat
tiles become one: the volume size limit and EC volume count move into
footnotes, the mount client card joins the cluster counts, and the
counts link to their pages instead of sitting in nested cards. The
duplicate version list and the bucket shortcut in the header are gone,
and the section menus become plain View all links.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: export the user columns the table actually has
The users table has username, email and access key columns, but the
export waited for six cells per row and so always produced only the
header line.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: drop table titles that repeat the page title
List pages titled their only table with the page title again. The
table now sits right under the summary tiles, and the export actions
that were hidden in the table header menu move to the page header as
an Export button, matching the cluster pages. The policies menu linked
to nothing and is removed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: drop clock and duplicate summary tiles
Buckets, users, policies and service accounts spent a summary tile on
the refresh time, which the page footer already shows (policies gains
that footer). The users page also showed its user count twice.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: show unset bucket settings as a dash
Rows of No owner, No quota and Not configured made the bucket table
hard to scan. Unset cells now show a quiet dash that keeps the wording
as a tooltip and for screen readers, and the name and creation date no
longer wrap (the full timestamp is in the date's tooltip).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: export buckets through the page's API exporter
admin.js loads after the page, so its copy of exportBucketList replaced
the buckets page's own one. That copy scraped the visible rows, so it
exported only the current page, without CSV quoting, and picked up the
display text of the cells. The page's exporter reads every bucket from
the API with full timestamps and escaped fields.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: escape quotes in the user export
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: download the bucket export as a Blob
The exporter put the CSV into a data URL with encodeURI, which leaves #
alone, so a bucket owner containing # ended the download there. Use the
shared downloadCSV helper instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: count masters as configured, not reporting
The master list comes from raft membership, which keeps a follower
that is down, so it does not show which masters are reporting.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: link the Prometheus metrics endpoint from the sidebar
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: drop the sidebar metrics link
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: record the caller on AssumeRole sessions
The session name is the caller's choice; the session now carries the
caller itself as the user it was issued to, as the web identity, LDAP
and credential paths already do.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: take a role session's aws:username from its user, not its name
An assumed-role ARN ends in the session name the caller picked, so
naming a session after another user resolved ${aws:username} and
${aws:userid} to that user in bucket and attached policies. Resolve
them from the user the session was issued to instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iam: take a role session's aws:username from its user, not its name
IsActionAllowed set aws:username to the last segment of an assumed-role
ARN, the caller-chosen session name, for role and bucket policies. Use
the user the session was issued to, and drop a forwarded value when the
session names none.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: pass a role session caller's user on to the session it assumes
A role session that calls AssumeRole is named by its session ID, so the
new session recorded that ID as its user. An STS session caller now
passes on its own recorded user, or none.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: size the session claims map from the request context
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* admin: take master HTTP ports from raft member names
The masters page and dashboard turned each raft member's gRPC address
into an HTTP address by subtracting 10000 from the port, so masters with
a custom -port.grpc showed wrong or negative ports. Raft members are
named after their master address, which carries the HTTP port; use it
and keep the old derivation only for names that are not addresses.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: test raft member address mapping
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: drop cached gRPC channels when a fake filer takes a port
A test that dials a dead owner leaves its failed channel in the
process-wide gRPC connection cache. When a later test's fake filer is
given the same port, its first RPC fails at once with the cached
connection refused, which is how TestRoutedMultipartFinalize failed on
the 32-bit job. The fake filer helpers now share fakeFilerAddress, which
invalidates any cached channel for the new address.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test a fake filer on a dead owner's reused port
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: drop cached channels for dead test filer addresses too
closedFilerAddress can be handed a port whose cached channel belonged to
a fake filer that just stopped and may still look ready; then the dead
owner's transaction is sent on it instead of failing over. Build its
address with fakeFilerAddress as well.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: split upload ACL assembly out of preparePutObjectACL
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: store POST Object acl and grant fields on the object
The form acl and x-amz-grant-* fields were authorized but never assembled,
so the object kept no ACL: acl=private on a public-read bucket stayed
anonymously readable and invalid values were accepted. Build them with the
same rules as PutObject, which also covers the Block Public Access check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test POST Object stores its upload ACL
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: take the POST Object ACL from the signed form only
ACL request headers are not covered by the POST policy, so a holder of a
signed form could add one and have it stored with the upload.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: reject a POST acl field the policy does not cover
The acl form field now sets the stored object ACL, so like the x-amz-*
fields it needs a policy condition, as AWS requires.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test unsigned ACL headers and uncovered acl fields on POST
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: evaluate POST upload policies on the canonical ACL
Policy conditions on the grant headers saw the raw form value, so an
equivalent spelling of a denied grantee passed. Use the PutObject request
shape that the PUT path evaluates.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: check bucket policy for POST ACLs with auth disabled
PutObject applies the bucket policy to an explicit upload ACL when IAM is
off; POST now does the same before storing it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: reject differing acl and x-amz-acl POST fields
Both promote to the canned ACL header, so map order picked the stored ACL.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test POST grant canonicalization, auth-off ACL policy, acl conflicts
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: check the auth-off POST ACL policy after loading the bucket
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: reject an empty acl or x-amz-acl POST field beside a differing one
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test the auth-off POST ACL policy on a cold bucket cache and empty acl aliases
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: load the bucket before per-key object authorization
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test the POST ACL policy on a cold bucket cache with auth on
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: stop per-key object authorization when the bucket fails to load
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test the cold-bucket POST ACL allow path and a failed bucket load
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* lance: require UpdateTable to deregister a table
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: require UpdateTable to register over an existing table
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: test that GetTable alone cannot deregister or re-register a table
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: a register that finds a table the caller cannot read needs UpdateTable to bring it back
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: decide declare write access by whether the table existed, not by what the caller can read
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: test register and declare by a caller that cannot read the table
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: register clears the marker on the create branch only after checking it
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: keep a register whose table was created when the marker check fails
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer/leveldb_handle: reopen a database whose journal latched a write error
goleveldb keeps the first journal write error and returns it for every
later write until the database is reopened, so one transient ENOSPC or
EIO fails all writes until a restart. The handle offers the leveldb.DB
methods the stores use; a write that fails on an OS error probes the
storage and, once it accepts writes and no call holds the database,
closes and reopens it and retries the write once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: open the leveldb stores through leveldb_handle
leveldb, leveldb2 and leveldb3 now recover from a transient storage
write error without a restart.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: leveldb stores return iterator errors from listings and deletes
A listing on a database that cannot be read reported an empty
directory, and DeleteFolderChildren wrote an empty batch, because the
iterator error was never checked.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/leveldb_handle: test a reopen that fails after the probe passes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/leveldb_handle: probe storage before draining readers
A probe that stalls in Create, Write or Sync held mu exclusively and
blocked every read on the handle. Reopen attempts are now serialized by
their own mutex, and the probe runs before the drain.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/leveldb_handle: hold new calls back while a reopen drains
TryLock succeeds only at an instant with no reader, so overlapping
listings could keep a reopen from ever draining while each write spun
for the full drain timeout. The drain now holds new calls back for at
most drainTimeout; a call nested inside a running one waits that out
instead of deadlocking. A drain that times out holds off the next
attempt for busyReopenInterval, so new calls are not held back over and
over.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/leveldb_handle: bound the wait for a running reopen
A write that hit the latched error waited on the reopen mutex for as long
as another caller's storage probe ran, so a stalled probe queued every
writer. Callers now wait at most reopenWait and return their error, and
still retry once a concurrent reopen succeeds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/leveldb_handle: tests wait for the drain and readers explicitly
The nested-read test polled for a drain that could start and end between
polls and then spun forever; it now syncs on the probe, widens the drain
and fails if the put returns first. The overlapping-readers test waits
until every reader holds an iterator before it writes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
