* s3api: deny signed reads that name a version file by its physical path
A stored version lives at <key>.versions/v_<id>, and that path is also a
valid object key. Anonymous reads already refuse it; signed GET, HEAD,
copy sources and object subresources did not, so a caller without
s3:GetObjectVersion, or under a Deny on the logical key, could read any
version through it. Apply the same check in the shared entry recheck.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: recheck a version file key before answering a conditional read
A conditional GET or HEAD answered 304 or 412 with the ETag before the
handler's entry recheck, so the physical version path still confirmed a
version's ETag to signed callers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: look up version file keys in the lookup-based policy recheck
Retention, legal hold, tagging writes and deletes only fetched the entry
for the recheck when the bucket had a policy, so without one they still
acted on a version file named by its physical path. Fetch it whenever
the key has the version file shape.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: refuse a directory-marker delete that names a version file
A trailing slash kept "<key>.versions/v_<id>/" out of the version file
shape check, and the lookup recheck misses the entry for that form, so a
DELETE reached deleteDirectoryMarker and removed a chunkless version or
delete marker without s3:DeleteObjectVersion.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The ring tests used 127.0.0.1:18889 as an unreachable peer, but its gRPC
port 28889 is a common filer port: on a shared CI host a live filer there
accepted the forwarded exclusive create, and
TestCreateEntryExclusiveLeavesNonOwner failed. The peer now listens on a
port the test owns and drops every connection.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* admin: restyle the UI as a calm console
Grey canvas with white sheets, a docked sidebar that marks the current
page with an indigo tint, ink primary buttons, tinted status pills,
dense tables with tabular numbers and spectrum capacity meters. The
styles are design tokens layered over Bootstrap's own variables, with
light and dark values, so the page templates stay unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: add a light, dark and automatic theme switch
The choice is kept in localStorage and applied before the page paints,
on the sign-in page too; automatic follows the system setting.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: drop hard-coded colors from page templates
The volume id link, utilization track and plugin form borders used
fixed light-theme colors; sparklines now use the data palette.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: theme the plugin job detail panel background
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: keep the selected radio dot visible in dark mode
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: cycle the theme from memory when storage is blocked
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: keep success progress bars green
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: keep keyboard focus visible on the toolbar togglers
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: use contrasting text on danger and success buttons in dark mode
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: lead the dashboard with cluster health and one row of tiles
The header now says in one line whether the cluster is healthy and how
many masters, volume servers and filers report. The two rows of stat
tiles become one: the volume size limit and EC volume count move into
footnotes, the mount client card joins the cluster counts, and the
counts link to their pages instead of sitting in nested cards. The
duplicate version list and the bucket shortcut in the header are gone,
and the section menus become plain View all links.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: export the user columns the table actually has
The users table has username, email and access key columns, but the
export waited for six cells per row and so always produced only the
header line.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: drop table titles that repeat the page title
List pages titled their only table with the page title again. The
table now sits right under the summary tiles, and the export actions
that were hidden in the table header menu move to the page header as
an Export button, matching the cluster pages. The policies menu linked
to nothing and is removed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: drop clock and duplicate summary tiles
Buckets, users, policies and service accounts spent a summary tile on
the refresh time, which the page footer already shows (policies gains
that footer). The users page also showed its user count twice.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: show unset bucket settings as a dash
Rows of No owner, No quota and Not configured made the bucket table
hard to scan. Unset cells now show a quiet dash that keeps the wording
as a tooltip and for screen readers, and the name and creation date no
longer wrap (the full timestamp is in the date's tooltip).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: export buckets through the page's API exporter
admin.js loads after the page, so its copy of exportBucketList replaced
the buckets page's own one. That copy scraped the visible rows, so it
exported only the current page, without CSV quoting, and picked up the
display text of the cells. The page's exporter reads every bucket from
the API with full timestamps and escaped fields.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: escape quotes in the user export
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: download the bucket export as a Blob
The exporter put the CSV into a data URL with encodeURI, which leaves #
alone, so a bucket owner containing # ended the download there. Use the
shared downloadCSV helper instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: count masters as configured, not reporting
The master list comes from raft membership, which keeps a follower
that is down, so it does not show which masters are reporting.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: link the Prometheus metrics endpoint from the sidebar
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: drop the sidebar metrics link
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: record the caller on AssumeRole sessions
The session name is the caller's choice; the session now carries the
caller itself as the user it was issued to, as the web identity, LDAP
and credential paths already do.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: take a role session's aws:username from its user, not its name
An assumed-role ARN ends in the session name the caller picked, so
naming a session after another user resolved ${aws:username} and
${aws:userid} to that user in bucket and attached policies. Resolve
them from the user the session was issued to instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iam: take a role session's aws:username from its user, not its name
IsActionAllowed set aws:username to the last segment of an assumed-role
ARN, the caller-chosen session name, for role and bucket policies. Use
the user the session was issued to, and drop a forwarded value when the
session names none.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: pass a role session caller's user on to the session it assumes
A role session that calls AssumeRole is named by its session ID, so the
new session recorded that ID as its user. An STS session caller now
passes on its own recorded user, or none.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: size the session claims map from the request context
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* admin: take master HTTP ports from raft member names
The masters page and dashboard turned each raft member's gRPC address
into an HTTP address by subtracting 10000 from the port, so masters with
a custom -port.grpc showed wrong or negative ports. Raft members are
named after their master address, which carries the HTTP port; use it
and keep the old derivation only for names that are not addresses.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: test raft member address mapping
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: drop cached gRPC channels when a fake filer takes a port
A test that dials a dead owner leaves its failed channel in the
process-wide gRPC connection cache. When a later test's fake filer is
given the same port, its first RPC fails at once with the cached
connection refused, which is how TestRoutedMultipartFinalize failed on
the 32-bit job. The fake filer helpers now share fakeFilerAddress, which
invalidates any cached channel for the new address.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test a fake filer on a dead owner's reused port
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: drop cached channels for dead test filer addresses too
closedFilerAddress can be handed a port whose cached channel belonged to
a fake filer that just stopped and may still look ready; then the dead
owner's transaction is sent on it instead of failing over. Build its
address with fakeFilerAddress as well.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: split upload ACL assembly out of preparePutObjectACL
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: store POST Object acl and grant fields on the object
The form acl and x-amz-grant-* fields were authorized but never assembled,
so the object kept no ACL: acl=private on a public-read bucket stayed
anonymously readable and invalid values were accepted. Build them with the
same rules as PutObject, which also covers the Block Public Access check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test POST Object stores its upload ACL
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: take the POST Object ACL from the signed form only
ACL request headers are not covered by the POST policy, so a holder of a
signed form could add one and have it stored with the upload.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: reject a POST acl field the policy does not cover
The acl form field now sets the stored object ACL, so like the x-amz-*
fields it needs a policy condition, as AWS requires.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test unsigned ACL headers and uncovered acl fields on POST
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: evaluate POST upload policies on the canonical ACL
Policy conditions on the grant headers saw the raw form value, so an
equivalent spelling of a denied grantee passed. Use the PutObject request
shape that the PUT path evaluates.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: check bucket policy for POST ACLs with auth disabled
PutObject applies the bucket policy to an explicit upload ACL when IAM is
off; POST now does the same before storing it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: reject differing acl and x-amz-acl POST fields
Both promote to the canned ACL header, so map order picked the stored ACL.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test POST grant canonicalization, auth-off ACL policy, acl conflicts
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: check the auth-off POST ACL policy after loading the bucket
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: reject an empty acl or x-amz-acl POST field beside a differing one
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test the auth-off POST ACL policy on a cold bucket cache and empty acl aliases
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: load the bucket before per-key object authorization
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test the POST ACL policy on a cold bucket cache with auth on
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: stop per-key object authorization when the bucket fails to load
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: test the cold-bucket POST ACL allow path and a failed bucket load
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* lance: require UpdateTable to deregister a table
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: require UpdateTable to register over an existing table
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: test that GetTable alone cannot deregister or re-register a table
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: a register that finds a table the caller cannot read needs UpdateTable to bring it back
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: decide declare write access by whether the table existed, not by what the caller can read
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: test register and declare by a caller that cannot read the table
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: register clears the marker on the create branch only after checking it
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lance: keep a register whose table was created when the marker check fails
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer/leveldb_handle: reopen a database whose journal latched a write error
goleveldb keeps the first journal write error and returns it for every
later write until the database is reopened, so one transient ENOSPC or
EIO fails all writes until a restart. The handle offers the leveldb.DB
methods the stores use; a write that fails on an OS error probes the
storage and, once it accepts writes and no call holds the database,
closes and reopens it and retries the write once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: open the leveldb stores through leveldb_handle
leveldb, leveldb2 and leveldb3 now recover from a transient storage
write error without a restart.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: leveldb stores return iterator errors from listings and deletes
A listing on a database that cannot be read reported an empty
directory, and DeleteFolderChildren wrote an empty batch, because the
iterator error was never checked.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/leveldb_handle: test a reopen that fails after the probe passes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/leveldb_handle: probe storage before draining readers
A probe that stalls in Create, Write or Sync held mu exclusively and
blocked every read on the handle. Reopen attempts are now serialized by
their own mutex, and the probe runs before the drain.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/leveldb_handle: hold new calls back while a reopen drains
TryLock succeeds only at an instant with no reader, so overlapping
listings could keep a reopen from ever draining while each write spun
for the full drain timeout. The drain now holds new calls back for at
most drainTimeout; a call nested inside a running one waits that out
instead of deadlocking. A drain that times out holds off the next
attempt for busyReopenInterval, so new calls are not held back over and
over.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/leveldb_handle: bound the wait for a running reopen
A write that hit the latched error waited on the reopen mutex for as long
as another caller's storage probe ran, so a stalled probe queued every
writer. Callers now wait at most reopenWait and return their error, and
still retry once a concurrent reopen succeeds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/leveldb_handle: tests wait for the drain and readers explicitly
The nested-read test polled for a drain that could start and end between
polls and then spun forever; it now syncs on the probe, widens the drain
and fails if the put returns first. The overlapping-readers test waits
until every reader holds an iterator before it writes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3: authorize an empty versionId as the current-object action
GET, HEAD and DELETE treat an empty versionId as the current object, but
the action resolver keyed on the parameter's presence and authorized it
as GetObjectVersion/DeleteObjectVersion, so a Deny on GetObject or
DeleteObject did not apply.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: test an empty versionId against a version-only policy
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: cover a bare versionId on HEAD
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume: refuse a tier upload whose .dat grew during the copy
AttachRemoteDatFile only checked that the data backend was still the one
copied, so a needle write acknowledged after the copy read the .dat was
missing from the remote object, and the upload then switched reads to it
and removed the local .dat. Compare the .dat size with the copy under the
same data-file locks; on a mismatch the handler deletes the copy and keeps
the local file.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* seaweed-volume: refuse a tier upload whose .dat grew during the copy
Mirror the Go AttachRemoteDatFile check. The tier-up attach took the
write lease only inside each step, so a needle write acknowledged after
the upload read the .dat was missing from the object the volume then
switched to, and the local .dat holding it was removed. Hold one write
lease across the instance, revision and size checks, the remote-file
update, the .vif save and the backend swap, as tier-down does; a size
mismatch aborts, deleting the object and keeping the local .dat.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: compare the physical .dat size before a tier attach
DiskFile pads its size past an unaligned tail, which a read-only load
keeps, while the upload copies the physical bytes. Stat the file itself
so such a volume is not refused as written during the copy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* shell: parallel moves and a per-run limit for volumeServer.evacuate
-maxParallelization runs up to N EC shard moves at once (default 1), with the
ec.balance scheduler: never two shards of one volume at a time, free workers
go to the least busy targets. The moves are planned first with the same target
choice as before, then applied through volume_move.Mover.MoveEcShards, which
verifies the target registered a shard before deleting it on the source.
-movesPerExec limits how many volumes and EC shards one run moves (default 0
for unlimited), like -volumesPerExec in volume.balance, so a big server can be
emptied in several runs.
With -skipNonMoveable a shard that fails to move (a read error on a failing
disk, for example) no longer drops the other shards of its volume; they are
moved and the failures are listed at the end. Without it the run stops at the
first failure as before. Losing the admin lock stops the run in either mode.
Refs #12501
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: plan evacuation again after failed moves; fail on a lost lock
A failed shard move kept the target slot it was planned on, so a readable
shard that found no destination in the plan stayed on the server although
the room was never used. Failed moves now give their targets back to the
model and the shards without a destination are planned again, up to
maxEcEvacuationRounds times; the failed shards are not tried again.
With -skipNonMoveable a lost admin lock was printed as a skipped failure and
the run returned success. ec.ErrLockLost is now exported and always fails
the run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: volumeServer.evacuate reports -movesPerExec after an applied EC run
The round loop only checked the limit after a failed move, so an applied run
that stopped at -movesPerExec without failures never said to run it again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: volumeServer.evacuate plans again for as long as failed moves leave room
The ten-round cap left readable shards on the server once more than ten
rounds of unreadable shards had taken the free room first, and a rerun
fails the same shards first again. Every round moves or fails each shard it
plans and a failed shard is not planned again, so the rounds end without it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: volumeServer.evacuate gives a failed move back every slot it took
When the target already held the shard in another disk type, adding it to
the planned disk type still took a slot, but the undo gave back only the
explicit one. Record the slots the plan took and restore exactly those.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec: an evacuation move that landed on its target keeps the target slot
A move can fail after the target registered the shard, when only the source
unmount or delete failed. Evacuate then gave the target's slot back and could
plan another shard onto a full server. The runner now checks the target after
a failed move and leaves a shard the target has out of FailedMoves.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: volumeServer.evacuate gives back the slot of a failed move onto a shard the target had
A target that held the shard before the plan reports it after any failed
move, so the runner took such a move for one that landed and evacuate kept
the slot the plan had taken. The runner now lists those moves in
LandedMoves, and evacuate undoes the ones whose target had the shard, then
plans again with the room they gave back.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* util: add WaitUntil, a cond wait bounded by a context and a timeout
The caller holds the cond lock. Cancellation and the timeout wake the
waiter through a broadcast taken under the same lock, so no wakeup is lost.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: bound the wait for an upload slot
A PUT that arrived while the concurrent upload limits were exhausted waited
on the condition variable with no bound. The handler reads nothing from the
body while it waits, so a client that gave up and reset its connection went
unnoticed: the socket and goroutine stayed pinned until a slot freed, and a
stalled volume server grew the gateway's fd count by one per aborted PUT.
The wait now ends on InflightUploadDataTimeout with 503 SlowDown, which S3
SDKs retry, and on request cancellation. An admitted upload takes its slot
under the lock, and a finished upload broadcasts, so freed bytes admit every
waiter that fits instead of one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: bound the wait for an upload slot
The filer's PUT/POST upload limits queued requests the same unbounded way.
Reuse the bounded wait: reject with 429 after InflightUploadDataTimeout, as
the volume server does, and stop on request cancellation.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* command: add -inflightUploadDataTimeout for the S3 gateway and the filer
weed s3 -inflightUploadDataTimeout, weed filer -inflightUploadDataTimeout and
-s3.inflightUploadDataTimeout, weed server and weed mini
-filer.inflightUploadDataTimeout and -s3.inflightUploadDataTimeout. The
default of 60s matches the volume server's flag of the same name; 0 keeps
the old unbounded wait.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: reject a queued upload without reading its body
WriteErrorResponse drains the request body before answering. Behind the
stats recorder the drain has no read deadline, and reading the body sends an
Expect: 100-continue client the go-ahead to upload what is about to be
discarded. Answer the rejection with an empty-body copy of the request so
the drain is skipped; net/http closes the connection when a large unread
body remains.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3, filer: release an upload slot under the cond lock
A waiter checks the counters under the cond lock and then waits. A release
that decremented and broadcast without the lock could land between the two,
so the waiter slept with a slot free until the next release or its timeout.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3, filer: close the HTTP/1 connection when rejecting a queued upload
net/http reads up to 256 KiB of an unread request body before sending a
keep-alive reply, so a client that stalls its body held back the 503 or
429 until the connection idle timeout. Connection: close skips that read.
HTTP/2 is left alone: there it would shut down the whole connection.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* shell: s3.clean.uploads deletes uploads over filer gRPC
Same DeleteEntry the HTTP handler runs, without minting a filer JWT,
and the gRPC path is the one a locked, routed delete needs next.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: s3.clean.uploads frees an open upload under the object write lock
The cleaner deleted a stale upload recursively with no lock, so a
CompleteMultipartUpload could pass its upload-exists check after the
part entries and their chunks were gone, and publish an object over
freed chunks. Do what AbortMultipartUpload does: hold the object's
distributed write lock, and free an open upload in an ObjectTransaction
routed to the object's owner filer, conditioned on the object not
carrying the upload; if it does, delete the metadata only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: s3.clean.uploads keeps the object write lock for the whole cleanup
A large upload's recursive delete can outlast one 15 second lease, and
a completion on the distributed-lock path could then take the lock and
commit over chunks being freed. Renew the lock in the background and
skip the delete once it is lost.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: s3.clean.uploads locks the path a completion commits under
For a trailing-slash key a non-versioned completion writes and locks the
entry inside the directory the key names, not the object path, so the
cleaner's transaction did not serialize with it. Lock and check the
same path the completion uses for the bucket's versioning state; the
route key stays the object's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3api: abort locks the path a completion commits under
A routed abort locked and checked the object path, but for a
trailing-slash key a non-versioned completion commits under the entry
inside the directory the key names. Use the same path the completion
locks for the bucket's versioning state.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: end s3.clean.uploads bucket failure line with a newline
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* cluster: NewBlockingLongLivedLockWithin bounds the wait on a held lock
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: s3.clean.uploads skips an upload whose object lock stays held
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* cluster: share the contended lock filer setup across lock tests
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: s3.clean.uploads reads the bucket's versioning state per upload
The commit path a routed completion locks depends on versioning for a
trailing-slash key. Reading it once per bucket before the run left a
window as long as the run; read it just before the delete, as the
gateway's abort does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* mount: remove the local control socket when the mount exits
weed mount listened on /tmp/seaweedfs-mount-<hash>.sock and removed it only
when a later mount of the same directory started, so every unmount or
SIGTERM left one socket file behind. Remove it on the normal return path and
in the interrupt hook, unless a later mount of the same directory has
already replaced it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: remove the control socket before a failed mount exits
glog.Fatalf and glog.Exitf call os.Exit, which skips the deferred removal,
so a mount that failed after creating its socket (mount point not
available, fusermount failure) still left it behind.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* mount: register the socket cleanup hook before mount setup
A SIGTERM that arrived while the mount was still setting up, for example
waiting on the filer to create the mount root, exited through grace without
running the deferred cleanup. Registering the hook right after the listener
is created makes it run last, after the unmount and filesystem hooks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer/etcd: scan entries from one revision
volume.fsck walked etcd one directory at a time, so a file moved from a
directory not yet listed into one already listed was seen under neither
name and its chunks were reported as orphans. The etcd store now
implements filer.EntryScanner: every page is read at the revision of the
first read, and a compacted revision fails the scan instead of falling
back to the walk.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/etcd: read hard links at the scan revision
Hard-link records resolved during a scan were read at the current
revision, so a link updated meanwhile mixed two instants. Reads made with
the scan's context now use its revision.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: list etcd among the stores volume.fsck reads from a snapshot
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/etcd: skip hard-link records in a scan
Hard-link records live beside the entries under their random ids, so a
root scan could read one as an entry with a path made of random bytes,
which the traversal stream cannot send. Skip hard-link ids and keys whose
directory is not valid UTF-8.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/etcd: recognize a hard-link record by the id it stores
A hard-link id can be valid UTF-8, so the key shape alone let some
records through a root scan as phantom files. The record's value is the
shared entry, whose HardLinkId is the record's own key; no entry key
can equal its own hard-link id.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/etcd: scan entries under an invalid-UTF-8 directory
The filer sanitizes names but stores directories as given, so a path
written over HTTP can leave entries the listing never reaches. Skipping
them let volume.fsck count their chunks as orphans. Send them under the
sanitized directory instead, the substitution the filer uses for names.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer/etcd: fail a scan whose sanitized path names another entry
An entry under an invalid-UTF-8 directory is reported under the sanitized
directory. When that path is also another stored entry at the scan revision,
or another sanitized entry in the same scan, acting on the reported path
(volume.fsck deleting it) would reach the other file, so the scan fails and
names the stored path instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* ec: don't index ev.Shards[0] when the last shard was unmounted
* ec: skip a runtime emptied before the scrub merge reads its shards
FindAllEcVolumes releases each location lock before MergeEcRuntimes runs,
so an unmount can leave a merged runtime with no shards and the width
scan indexed Shards[-1].
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec: treat a scrub merge with no shard left as a vanished volume
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec: read Shards once in ShardSize so a concurrent unmount cannot empty it
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec: read each runtime's Shards once in the scrub merge
The vanished check and the slot construction read Shards separately, so
an unmount between them produced a non-nil view with no shards.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec: anchor the scrub merge on a generation that still holds shards
An unmount can empty the newest runtime while an older one keeps its
shards, which left a merged view anchored on the emptied generation with
no slots and the holder reported as skipped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Mansur Akhmadi <10739460+mansur-akhmadi@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* shell: fs.mergeVolumes leaves in-progress multipart parts in place
CompleteMultipartUpload reads the part entries and later commits the
object with their fids, guarded only by the upload directory. A merge
that moved a part chunk in between deleted the source needle the new
object pointed at. Parts stay where they are until the upload completes
or is cleaned, and their chunks still count as referenced sources.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: note in fs.mergeVolumes help that upload parts stay in place
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: match upload parts when the buckets folder is the root
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: count chunks behind a part's manifest as referenced sources
A skipped part whose chunks on a merge source sit behind a manifest was
not counted, so the merge printed no skip line and warned that nothing
references the source.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: keep a part's direct source references when its manifest is unreadable
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: count chunks resolved before an unreadable part manifest
ResolveChunkManifest returns the data chunks of earlier manifests along
with a later manifest's error; record them before returning so the
unreferenced-source warning does not fire for a source they reference.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: test the partial manifest result without a timing delay
The unreadable manifest is now nested under a readable part manifest, so
the first level resolves fully before the failing lookup runs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer_pb: add a depth-first traversal that can resume after a path
TraverseDfs walks a tree in listing order and reports each finished
path, so a caller can save one path and continue from it later. Pages
are read whole before visiting, and the entries of a page are visited
by up to five workers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.meta.backup: resume an interrupted metadata copy
The walk saves the last finished path every few seconds, and a walk
that continues an unfinished one starts after it. A fresh walk clears
the saved path first, and a finished walk clears it with the walk
start.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.meta.backup: add -entriesPerSecond to throttle the metadata copy
Each entry the full copy reads waits on a rate limiter, and pages are
listed only after the previous page is copied, so the flag also bounds
how fast the copy lists the source filer. 0 keeps it unlimited.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer_pb: report each file of a depth-first walk once all before it are done
A throttled walk only reported progress after a whole listing page, so a
slow copy could go many minutes without a resumable path. Visit entries in
runs that end at a directory and report each file as soon as every entry
before it is done.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* volume: fail a vacuum step that finds a compaction already running
Compact, CompactByIndex and CommitCompact returned success when another
compaction held the volume. A second, uncoordinated vacuum (an expired
admin job still running, a shell volume.vacuum next to the worker) then
went on to commit or clean up the first run's .cpd/.cpx. Return a
"volume N is compacting" error instead, and refuse cleanup while a copy
is in flight, as the Rust volume server already did.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: check the compaction files before a commit closes the volume
CommitCompact closed the needle map and data file and wrote the .cpc
marker before it found .cpd/.cpx missing. A duplicate commit, such as
the second of two overlapping vacuums, then left the volume registered
with no open files: it dropped out of the heartbeat and looked lost to
volume.list and volume.fix.replication until a restart. Check for the
files first so the refused commit leaves the volume as it was.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: run makeupDiff before a commit closes the volume
CommitCompact closed the needle map and data file before makeupDiff, so
a makeupDiff failure (a compaction revision another commit moved, an I/O
error) returned with them closed: the volume stayed registered but
missing from the heartbeat until a restart. makeupDiff reads the live
files through its own handles while the commit holds the write lock, so
close them only once it has succeeded, as the Rust volume server does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: log a volume the heartbeat skips because its files are closed
A volume left without its needle map or data file vanished from the
heartbeat with no log line, so the master saw a replica disappear while
its files were still on disk. Warn on each skip, as the deleted-.dat
case next to it already does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: claim the compaction flag while cleanup removes the temp files
Cleanup only read the flag, so a compaction could start between the
check and the removal and lose its .cpd/.cpx. Claim it for the cleanup
instead, in Go and Rust.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer.sync: move the watermark over jobs that finished ahead of an older one
The watermark only took the finishing job's timestamp when it was the
oldest in flight, so jobs that completed before an older one were never
recorded and a quiet stream kept the offset behind finished events.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.sync: keep one finished timestamp per gap behind a running job
A long job kept every later completion in the done heap. Only the newest
one between two running or failed timestamps can become the watermark.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.sync: test done-heap compaction across several running and failed jobs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3: answer GETs canceled before the first byte
A stream that fails before writing anything now always gets an S3 error
response, canceled errors included. The idle timeout and a shared lookup
canceled by another request both cancel the request context while the
client is still connected, and returning silently sent that client a
200 with Content-Length: 0. A client that has gone away never sees the
write.
Refs #12500
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: mark inline GET write errors as written; bound the idle-cancel test
An inline object's write error comes after the status line, so the
caller must not write an S3 error on top of it. The idle-cancel test
caps the client wait and gives the response write its own deadline,
keeping only the read side on the 200 ms idle timeout.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: test a GET whose request is canceled before the first byte
Drives GetObjectHandler through a chunk lookup that cancels the request
context, so a handler that returns before answering shows up as the empty
200 again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3: check the cancel test reached the chunk lookup
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* volume: forward no Content-MD5 for a JPEG the orientation fix rewrote
With -images.fix.orientation the primary stores and forwards the rotated
bytes, but sent replicas the client's digest of the original upload. The
replica checks Content-MD5 against what it receives and rejects the write,
so replicated JPEG uploads with a digest failed. The Rust volume server
already drops the digest once the body is rewritten.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: test the forwarded Content-MD5 through PostHandler and a replica
The test now posts each upload to a primary whose fake master names a
real replica volume server, so it covers the JPEG gate, ReplicatedWrite
and the replica's own Content-MD5 check, and reads the stored bytes back.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: fail the replica forwarding test instead of hanging when nothing is forwarded
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume: compare the primary and replica bytes in the replica forwarding test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer.copy: compute chunk md5 so files get a real ETag
Chunks uploaded by filer.copy had no ETag, so every multi-chunk file got
the md5 of the empty string plus -N. An overwritten file kept its old ETag
and a revalidating client got 304 for changed content.
* filer.copy: store chunks in offset order
Chunks are collected in upload completion order, and the entry ETag
hashes the chunk ETags in stored order. Identical files copied with
concurrent chunk uploads got different ETags, and a file with swapped
equal-sized chunks could keep the old one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer.copy: test chunk order and chunk md5 on chunked uploads
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* weed/shell: support -noLock flag in volumeServer.evacuate (#12501)
* shell: return flag parse errors from volumeServer.evacuate
A typo such as -noLock=invalid printed the parse error but the command
still succeeded, so a piped script treated the evacuation as done.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: volumeServer.evacuate -h prints usage without failing
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(shell): fs.log.purge purges every filer, not just one
Each filer keeps its own metadata logs in its own store, so purging a
single filer leaves the chunks referenced by the others and the volume
vacuum cannot reclaim anything.
The shell picked that single filer at random (commands.go:208) when no
-filer flag was given, and the admin script worker pinned it to
filers[0] (admin_script_handler.go:574), so the same command purged a
different filer on every run and never the whole cluster.
fs.log.purge now resolves the filer list from the master and purges each
of them, continuing when one is unreachable and reporting the failures
at the end. -filer=<addr>,... restricts the purge to a subset.
Also stops mutating the shared command environment per filer, so a
long-lived shell keeps its original filer.
* fix(shell): bound filer discovery and cover the cluster path
Address review feedback:
- resolveLogPurgeFilers waited on the master with context.Background(): a
shell whose master connection dropped would hang before purging
anything, where it used to purge the connected filer. Bound the master
round trip with a timeout and fall back to the connected filer when the
master cannot be reached or lists no filer.
- the tests only exercised the single-filer fallback. Rename that test to
what it actually covers and add one asserting every filer of the list is
attempted and the shared command environment keeps its original filer.
* shell: reject an fs.log.purge -filer list with no addresses
A comma or whitespace only -filer value fell through to master
discovery and purged every filer instead of the intended subset.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: fall back to the shell's filer without a second master lookup
When the master listed no filer, FilerAddress() ran filer discovery again
and could wait out the shell's ConnectTimeout before failing. Also cover
master discovery with the fake discovery master.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: test the fs.log.purge filer loop against a fake filer
The previous test looped over the addresses itself, so it passed no
matter what Do did.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Martin Morez <martin@morez.org>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: validate bucket names before the SQL DDL hooks
The SQL store builds CREATE TABLE and DROP TABLE statements by formatting
the bucket name into the query. The DML paths already reject names that
fail VerifyS3BucketName, but the bucket-creation and bucket-deletion
hooks ran the DDL without that check, so a name reaching them through the
filer or WebDAV directory path (which skips the check for dot-prefixed
directories) could inject stacked statements. Apply the same validation
the DML paths use.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* filer: check that a valid bucket runs both DDL statements
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* weed-db: end startup when password authentication fails
The failure branches returned the result of writing the error response,
which is nil on success, so the handshake carried on to ReadyForQuery.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-db: test startup with wrong and unknown credentials
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-db: check the auth error code and fail on stray read errors in the startup test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* admin: escape file properties before rendering the modal
The Properties modal concatenated the entry name, path, MIME type and
extended attributes into HTML, so object metadata written over S3 could
inject markup into the admin page.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: keep every property key when escaping
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* security: do not accept a path- or method-scoped filer token as admin
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* security: test that scoped filer tokens fail the admin token check
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: let a table renamed across namespaces keep committing
* s3tables: keep the same-directory exception off a directory another table shares
A record written before metadata locations were confined to their namespace
(6f93f874) and claimed (#12473) can keep its files in another table's
directory, in another namespace. The exception that lets a table renamed
across namespaces keep committing to the directory it left behind accepted
such a record's commits too, where master refused them.
The namespace check now runs first, as on master. Only a location it refuses
is matched against the record's current directory, and the exception holds
only when no other record keeps its files there. Where other records' entries
sit is ignored, so a table created under the renamed table's old name, whose
files go to a directory of its own, does not take the exception away. The
scan runs only on that path, so commits inside the namespace cost what they
did.
Co-Authored-By: Claude <noreply@anthropic.com>
* s3tables: seed the table at the renamed table's old name
A rename may keep the old name taken, so creating a table there can be
refused; seeding the record keeps the test on the update path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
filer: prefix bucket default collection, assign copy collection explicitly
S3 server-side copy paths (CopyObject/UploadPartCopy) assign volumes with an empty collection and relied on the filer's bucket fallback. The fallback used the raw bucket name, while PutObject uses getCollectionName(bucket) = <filerGroup>_<bucket>, so copies landed in a second, unprefixed collection.
Prefix the filer's bucket default with the filer group, and have the S3 copy path pass that collection explicitly, like PutObject. Direct filer writes keep path-rule precedence, so this only aligns S3 copy with S3 put.
Adds regression tests for the fallback, explicit collection precedence, and the S3 copy collection resolver.
* weed/s3api: avoid truncated marker rescan in ListObjects (#12540)
Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com>
* fix(s3): validate marker directory candidates
Respect prefix filters and internal-directory exclusions when recovering marker candidates. Restrict lookups to direct children, propagate read failures other than not-found, and deduplicate directory marker echoes.
Add regression coverage and model promoted objects consistently in the XML-unsafe prefix fixture.
---------
Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3vectors: bound body buffering before authentication
* docs: include vector engine installation and source builds
* s3vectors: prevent MIME sniffing of JSON error responses
* docs: load vector Cargo configuration when building from source
* docs: note the native protoc requirement for Windows vector builds
* docs: make vector source checkout and working directory explicit
* ci: handle deb822 sources during Ubuntu mirror failover
* ci: bound Ubuntu mirror attempts and reject partial index updates
* vector: expose request metrics and registration-aware health endpoints
* vector: report immutable read cache and storage fetch metrics
* docs: describe vector engine health, metrics, and maintenance
* vector: count canceled requests when their handler is dropped
* vector: account for uncached large immutable storage reads
* vector: reliably release the stalled credential reader in tests
* vector: observe large reads as their streamed body is consumed
* vector: bound heartbeats on established master registration streams
* test: exercise hardened vector maintenance on Go and Rust volumes
* vector: register engines in their configured filer group
* vector: make retried index creation preserve compatible datasets
* vector: allow registration to advertise a separate address
* test: release stalled credential readers without blocking FIFO cleanup
* test: allow hardened vector coverage in a named filer group
* vector: discover engines in the same filer group as their gateway
* vector: fall back when the advertised address is empty
* admin: expose vector readiness, maintenance, and management tools (#12520)
* s3vectors: share vector discovery with trusted management clients
* admin: show vector engine readiness and recent maintenance activity
* admin: suggest S3 Vectors IAM actions from the supported operations
* admin: manage vector indexes, policies, tags, encryption, and query data
* admin: validate SSE-S3 support before creating encrypted vector buckets
* admin: discover vector engines in their registration group
* admin: probe vector engines concurrently with independent time budgets
* admin: verify encryption capability on the remote S3 gateway
* admin: defer discovery until after the HTTP server can start
* admin: distinguish unknown vector health services from failed engines
* admin: count workers that can execute vector index maintenance
* admin: allow read-only users to query vectors and inspect tags
* vector: scope gateway and admin discovery to the configured filer group
* admin: refresh policy state after mutations and serialize policy controls
* vector: drain active RPCs before closing replaced discovery connections
* vector: respect request cancellation while waiting for discovery
* vector: prefer another registered engine after a transport failure
* admin: check all masters before reporting no registered vector engines
* iam: honor attached vector policies without broadening legacy grants
* vector: allow short-lived discovery clients to close their connections
* vector: refresh cached master addresses for each discovery lookup
* iam: enforce vector bucket tag conditions in attached policies
* iam: authorize vector bucket creation with the requested tags
* test: exercise vector IAM grants and tag denies on Go and Rust volumes
* admin: reject authenticated vector mutations without an admin role
* test: run vector IAM assertions before the shared engine is stopped
* vector: add native mini startup, shell tools, and a runnable Compose example (#12522)
* s3vectors: share vector discovery with trusted management clients
* admin: show vector engine readiness and recent maintenance activity
* admin: suggest S3 Vectors IAM actions from the supported operations
* admin: manage vector indexes, policies, tags, encryption, and query data
* admin: validate SSE-S3 support before creating encrypted vector buckets
* admin: discover vector engines in their registration group
* admin: probe vector engines concurrently with independent time budgets
* admin: verify encryption capability on the remote S3 gateway
* admin: defer discovery until after the HTTP server can start
* admin: distinguish unknown vector health services from failed engines
* admin: count workers that can execute vector index maintenance
* admin: allow read-only users to query vectors and inspect tags
* vector: scope gateway and admin discovery to the configured filer group
* admin: refresh policy state after mutations and serialize policy controls
* vector: drain active RPCs before closing replaced discovery connections
* vector: respect request cancellation while waiting for discovery
* vector: prefer another registered engine after a transport failure
* admin: check all masters before reporting no registered vector engines
* iam: honor attached vector policies without broadening legacy grants
* vector: allow short-lived discovery clients to close their connections
* vector: refresh cached master addresses for each discovery lookup
* iam: enforce vector bucket tag conditions in attached policies
* iam: authorize vector bucket creation with the requested tags
* test: exercise vector IAM grants and tag denies on Go and Rust volumes
* admin: reject authenticated vector mutations without an admin role
* test: run vector IAM assertions before the shared engine is stopped
* mini: optionally start and supervise the vector engine
* mini: bootstrap vector buckets with existing Lance catalog helpers
* shell: manage vector buckets, indexes, and vector data
* docs: provide a runnable vector and maintenance worker Compose stack
* shell: scope vector discovery to the session and close its connections
* mini: honor the bind address for vector and metrics listeners
* mini: reserve vector ports before assigning gRPC and compatibility listeners
* mini: wait for vector storage setup and master registration
* mini: create requested vector buckets without requiring the Lance HTTP listener
* shell: refresh master addresses during vector discovery
* mini: resolve sibling vector executables with platform extensions
* vector: preserve uncertain creates and fix mini listener allocation (#12525)
* vector: retain catalog records when index creation has an unknown outcome
* mini: keep chosen listener ports reserved throughout allocation
* mini: resolve hostname binds before starting vector listeners
* vector: fix integration test imports and recover interrupted index creation (#12533)
* shell: register vector operations from the command package
* s3vectors: resume interrupted index initialization safely
* mini: exclude disabled WebDAV from listener allocation
* s3vectors: retain uncertain creates across refused retries
* s3vectors: claim pending indexes before retrying initialization
* s3vectors: compare resolved distance metrics during recovery
* s3vectors: preserve metadata keys when retrying initialization
* s3vectors: verify ownership before initializing new catalog records
* topology: list collections without nesting the collection map read lock
Items() already takes the read lock and copies the values. Holding it
around the call made ListCollections take the read lock twice, so a
DeleteCollection waiting for the write lock in between blocked the
second read lock and wedged the master.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* topology: test ListCollections against a concurrent collection delete
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* master: move the KeepConnected stream cleanup into removeClusterNode
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* master: drop a filer from the lock ring only with its last registration
A filer's replacement can register under the same address before the
master ends the old stream. The cluster list counts both registrations,
but the old stream's cleanup removed the address from the lock ring
outright, leaving the live filer out of the ring until it re-registered.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* master: order a stream's cluster list and lock ring changes
The cleanup of an ending stream could see its address removed from the
cluster list, then a new registration of that address add it to the list
and the ring, and only then remove it from the ring.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* master: take a leaving filer's reconnect out of the lock ring
A filer that reconnects while leaving registers with LeaveLockRing set.
Its broken stream can still be open on the master, and since that
stream's end no longer drops an address that is still registered, the
reconnect removes the address from the ring itself.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3: encrypt volume data when the filer is configured to
weed s3 read the filer's configuration at startup (buckets dir, filer
group, masters, metrics, maxMB) but not its -encryptVolumeData, so a
cluster that turned encryption on at the filer still had S3 uploads
stored plain, while uploads through the filer itself were encrypted.
The gateway now encrypts when either its own -encryptVolumeData or the
filer's is set; weed-vector, weed-worker and weed-vector-cli already
follow the filer's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: weed-vector, weed-worker and the CLI take volume encryption from the filer
They already encrypted chunks for the volume servers when the filer's
configuration said so, or when their own --encryptVolumeData did. With
weed s3 following the filer too, the filer's -encryptVolumeData is the
one switch for every writer, so the three flags are removed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* S3 Vectors tests: turn volume encryption on at the filer
VECTOR_ENCRYPT_VOLUMES=1 set -encryptVolumeData on the gateway, the
engine and the worker. It now sets it on the filer alone, so the leg
checks that each of them follows the filer: the gateway's stored chunks
carry a cipher key (with a gateway that ignores the filer, the leg fails
on "the gateway stored a chunk without a cipher key"), and so do the
engine's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker README: volume encryption follows the filer
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3: help text says the filer's -encryptVolumeData turns S3 encryption on
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* S3 Vectors tests: plain legs check the engine stored no cipher key
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3: don't report a version delete as done when the filer lookup failed
deleteSpecificObjectVersion treated any getEntry error as "entry is gone":
- the null-version precheck returned success without deleting anything,
- the recheck after a failed rm returned success while the entry may
still be there,
- a failed .versions lookup was read as "not the latest version", so the
latest blob was removed without repointing the pointer first.
Only filer_pb.ErrNotFound now counts as a miss. Other errors fail the
delete, so the S3 client or the lifecycle worker retries it. The lookups
go through retryFilerOp, like the other .versions reads in this file, so
a single transient filer error doesn't turn into a 500.
* s3: include recheck error when a version delete fails
* s3: recheck a failed version delete once instead of retrying
---------
Co-authored-by: Mansur Akhmadi <10739460+mansur-akhmadi@users.noreply.github.com>
* volume.fsck: batch and deduplicate filer entry deletions
In volume.fsck with -findMissingChunksInFiler and -reallyDeleteFilerEntries, deleting orphaned filer entries was slow and CPU intensive due to tiny worker channel buffers (128), lack of batch processing, redundant HTTP DELETE requests for multi-chunk files, and lock contention on parent directory tracking.
Optimize by:
- Adding startPurgeBatchWorkers with 4096-buffered channels to prevent producer stalls and batch opportunistic deletes up to 64 items.
- Deduplicating missing chunks per file path during volume checks so that multi-chunk files are queued once.
- Processing batches with in-batch path deduplication and aggregating parent directories before updating purgedDirs in a single lock acquisition.
- Buffering temporary fid file writes (64KB) during the traversal to avoid millions of unbuffered syscalls.
* volume.fsck: report a failed write of the filer fid files
The fid files are written through a buffer now, so the last part of
each one reaches the disk only in the deferred Flush. Its error was
dropped: a full temp disk left a short fid file and the collection
still succeeded, and the volume check would then take the chunks
missing from it for orphans.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume.fsck: retry a batched path whose earlier delete failed
A path was marked done before its delete ran, so a second copy of it in
the same batch was skipped after a transient failure and the entry
stayed. Only skip paths already deleted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume.fsck: keep reporting a failed DELETE response read
The non-verbose path now discards the body without checking the copy,
so a connection cut after the headers went unreported.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume.fsck: keep one purge worker starter and one delete function
startPurgeWorkers and the single-path httpDelete were left only as
wrappers the tests called. Give their names to the batch versions and
adapt the tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: give a table created without a location its own directory
An Iceberg table created without a location kept its files under its name
path, so a rename left them where the next table of that name would land.
Such a table now gets s3://<bucket>/<namespace>/<name>-<uuid>, stored as the
record's warehouseLocation and returned by GetTable; its first commit writes
there without moving the root. Records without one keep resolving to their
name path, so existing tables need no rewrite. Lance datasets stay at their
name path, where Lance clients and the vector engine look for them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: create tables and views in a directory of their own
A table or view created without a location took its name path, or a
unique one only when that path was already taken. It now always gets
<name>-<uuid>, so a later rename never leaves its files under a name
another table can take.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: load and commit a table at its warehouse location
A table created through S3 Tables without a location has no metadata until
its first commit; loading or committing it now uses the directory recorded
for it, and only a record without one falls back to its name path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: fail a stage-create whose marker is not saved
The marker is how the commit finds the location the stage-create handed
out. Without it the commit fell back to the name path while the client's
files sat under the returned location.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: expect tables created without a location in their own directory
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: rename a table that keeps its files elsewhere before reusing its name
Enterprise keeps the storage marker on a renamed table's files, so the name
of a table whose files sit under its name path stays taken there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: remove the table directory when a create backs out
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: answer a create that lost the race with the registered table
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: answer a view create that lost the race with the registered view
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: fail a staged commit whose name another create registered
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: remove the directory of a create whose name another table took
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: keep the metadata of a staged commit whose name is taken
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: write the lakekeeper table's data under its warehouse location
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: commit a staged table at the location of its own stage
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: never continue another stage when a commit's own marker is gone
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: test filer honors exclusive creates and token-guarded deletes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
