mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-12 23:09:33 +01:00
a2ffdce2ec0769ee584996818f41bd8ffaef13de
16181
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris LuandClaude Opus 5.5
|
a2ffdce2ec |
volume: skip VolumeDelete when its RPC was cancelled before the delete (#12586)
* volume: skip VolumeDelete when its RPC was cancelled before the delete A caller that cancels (a lost tiering lease, an expired deadline) still had its admitted delete run in the Go server, including one that waited behind the copy guard. The Rust server drops a cancelled handler, so return the context error before the delete starts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: wait until the queued delete blocks on the guard before cancelling Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
6388dfca9b |
s3api: verify signed request bodies against the signed payload hash (#12577)
* s3api: verify signed request bodies against the signed payload hash A SigV4 signature covers the declared x-amz-content-sha256, but only getRequestDataReader (PutObject, UploadPart) compared the body with it. Every other handler parsed the body as received, so a signed request, DeleteObjects for example, could be replayed with a different body. After a V4 signature verifies, wrap the body so it is handed out only once all of it has been read and matched against the hash the signature covered; a decoder that stops reading early never sees unverified bytes. getRequestDataReader unwraps it and keeps verifying uploads as they stream. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: answer XAmzContentSHA256Mismatch for refused DeleteObjects bodies DeleteObjects answered a body that failed its payload-hash check with InternalError and CompleteMultipartUpload with MalformedXML. Name the check that failed instead, as PutObject does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: check Content-MD5 on signed request bodies Signature V2 signs Content-MD5 rather than a payload hash, and a V4 request sent with UNSIGNED-PAYLOAD may still sign it, but bodies read outside the upload path never compared it with the body. Check it with the payload hash once a V2 or V4 signature verifies: BadDigest on a mismatch, InvalidDigest when it is malformed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: verify presigned uploads against a signed payload hash A presigned V4 URL can sign X-Amz-Content-Sha256 as a query parameter, but getRequestDataReader compared upload bodies only with the header of that name, so the signed hash went unchecked. Verify uploads against the hash the signature covered. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: check Content-MD5 on bodies buffered for their payload hash IAM and other non-S3 requests signed without x-amz-content-sha256 have their body buffered and hashed during signature verification. That buffer counted as already bound, so their Content-MD5 was never checked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: check Content-MD5 on directory marker bodies A PUT of a key ending in "/" stores its body on the directory entry without going through putToFiler, so its Content-MD5 was never compared with the body. Answer BadDigest on a mismatch, as putToFiler does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: wrap and unwrap signed bodies through helpers Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
a3b68aa37c |
shell: confirm trim survivors still serve the volume before deleting (#12590)
* shell: confirm trim survivors still serve the volume before deleting volume.fix.replication compared the doomed replica's index with every survivor and then deleted it unconditionally. A survivor that went away while the indexes were read still let the delete through, leaving fewer copies than the check had counted. Re-read each survivor's VolumeStatus after the index checks and skip the trim if any no longer has the volume. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: test a trim whose survivor goes away after its index read Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
4cba4eb7bf |
s3: ignore unsigned object headers on POST Object (#12585)
* s3: ignore unsigned object headers on POST Object The POST policy signs form fields only, but the upload read tagging, user metadata, storage class, object lock and the stored content headers from the request headers as well. Drop them before the form fields are applied. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: test POST Object request headers against signed form fields Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
249f04e63a |
s3api: NoSuchKey for object lock calls on missing or deleted keys (#12592)
* s3api: answer PutObjectLegalHold on a missing key with NoSuchKey In a versioned bucket a key with no latest version resolves to ErrLatestVersionNotFound, which the legal hold handler did not map, so the request failed with 500 InternalError. PutObjectRetention already maps it to NoSuchKey. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: keep failed reads distinct when resolving an object lock target getObjectEntryForUpdate turned every lookup error into a not-found sentinel, so a failed filer read made PutObjectRetention and PutObjectLegalHold answer NoSuchKey for an object that exists. Only a genuine miss becomes the sentinel now, matching getObjectEntry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: answer object lock calls on a delete marker with NoSuchKey PutObjectLegalHold and PutObjectRetention on a key whose current version is a delete marker returned 200 and wrote the lock onto the marker, and the matching GETs read it back. A key whose current version is a delete marker does not exist, and delete markers take no lock, so these calls now answer NoSuchKey like the tagging and ACL handlers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: let a lock recorded on a delete marker not block its removal Delete markers are not WORM-protected, but a legal hold or retention written onto one (possible before the previous change) made both the gateway check and the routed delete's filer-side WORM guard refuse to remove the marker, so the object could not be restored and lifecycle could not clean the expired marker up. Lock enforcement now ignores a lock on a delete marker, and the routed version deletes in DeleteObject and lifecycle drop the WORM guard when the version they read is a marker. A version id never changes kind, so the marker read is the one deleted; the null version keeps its guard because a PUT can replace it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
4ee82977af |
docker: pass entrypoint arguments to weed without splitting them (#12589)
The master, volume, volume-rust, server, mini, filer and s3 cases and the fallback forwarded the user arguments as an unquoted $@, so the shell split each one at whitespace and globbed it against /data. An -admin.password with a space reached weed as two words, and weed ignored every flag after the stray word. Quote "$@" as the other cases already do; $ARGS stays unquoted since the script builds it from its own words. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
bef1b13b3d |
replication: keep a local sink path the source recreated after a delete (#12581)
* replication: let the local sink check a delete against the live source LocalSink removed the destination path for any delete it was handed. A delete that reaches the sink after the source recreated the path (a peer filer's event delivered late, or a historical event replayed) removed the newer file, and no later event brings it back. DeleteEntryVersion receives the deleted entry. Before removing an existing destination, the local sink looks up the source path and keeps the destination when the source holds an entry of the same kind committed after the deleted version (commit_ts_ns, or mtime for unstamped entries). A live entry that is the deleted version itself or older means the source store lags the event, so the delete proceeds. A failed lookup returns the error so the event is retried instead of deleting a file that may be live. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.sync, filer.replicate: hand the deleted version to the sink Pure deletes, renames without a native move and moves out of the watched directory now go through sink.DeleteEntryVersion with the old entry, so a sink that checks the live source (the local sink) can keep a path the source recreated after the deleted version. Other sinks fall back to DeleteEntry unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * replication: order the local sink check by the delete's mutation time A source filer can hold a version written concurrently with the delete: newer than the deleted entry but older than the delete itself. The source's own replay applies the delete over it, so keeping the backup copy because it outranks the deleted entry left a file the source no longer has. The sink now gets the delete's mutation time (MutationTsNs, else TsNs) and keeps the destination only when the live entry was committed after it, the order Filer.Replay keeps. That also covers a stamped replacement of an unstamped entry whose mtime did not move. filer.replicate events carry no time, so they keep the deleted entry's stamp as the bound, and mtime only when nothing is stamped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * replication: count an unstamped live entry from its mtime A source filer that predates commit stamps stores entries with no CommitTsNs, so comparing the live stamp with the delete's time never kept a replacement there. An unstamped live entry now counts from its mtime against the delete's time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * replication: keep a stamped live entry when the deleted one is unstamped and the delete has no time filer.replicate gets no delete time from the notification queue. When the deleted entry has no commit stamp either, a stamped live entry was written after it, so its mtime no longer decides. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
c2625ddf00 |
seaweed-volume: read security.toml values as TOML strings (#12582)
The line parser only trimmed quotes, so `key = "abc\n"` gave the Rust volume server the bytes `abc\n` with a literal backslash while the Go servers sign with a newline, and every JWT failed. Literal strings kept their quotes and a trailing comment stayed part of the value. Decode each value with the TOML parser; text it rejects keeps the old quote trimming. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
b7d6b48921 |
operation: probe octet-stream uploads for gzip like untyped ones (#12584)
* operation: probe octet-stream uploads for gzip like untyped ones The 128-byte gzip probe ran only when the caller passed no MimeType, so an upload labelled application/octet-stream or binary/octet-stream was never compressed. S3 PutObject and UploadPart pass the request Content-Type (or the octet-stream that mimeDetect fills in), so every binary S3 upload skipped compression that the same bytes get through the filer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * operation: check the probed upload keeps the caller's content type Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
02b9747bd8 |
wdclient: keep a shared volume lookup alive when its first caller leaves (#12579)
* wdclient: keep a shared volume lookup alive when its first caller leaves Concurrent misses for the same volumes share one provider lookup, but it ran on the first caller's context, so that caller's cancellation failed every request waiting on it. The lookup now runs on a context no single caller can cancel, and each caller stops waiting when its own context ends. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * wdclient: wait for the second caller to join before canceling the first A fixed sleep did not prove the second caller had joined the shared lookup: a late caller could hit the filled cache and pass without waiting. The test now waits until the caller reads its own Done channel, which happens only after DoChan registered it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
8581151376 |
filer: gzip metadata log pieces before uploading them (#12580)
* filer: let the fake-master test helper serve any master Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: gzip metadata log pieces before uploading them Every metadata event carries the old and new entry with their whole chunk lists, so a file grown by small fsynced appends logs the same chunk records again and again: 2000 appends of 256 bytes through weed mount wrote 34.8 MB of log. The uploader only compresses when the first 128 bytes compress, and a log piece starts with fresh file ids and ETags, so it stored the log as is. The filer now gzips each piece itself and keeps the compressed bytes when they save at least 10%. After the same workload the volumes hold 3.05 MB instead of 35.2 MB, file data included. Readers already honor the chunk's IsCompressed flag on every path (filer-side decode, log file refs read straight from volume servers), and the assign request sizes the bytes actually stored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
e0f8438284 |
mount: retry a writeback metadata flush until the filer answers (#12578)
* mount: keep the filer status in the flush create error flushMetadataToFiler formatted the CreateEntry error with %v, so callers classifying it saw only the message, which includes the file path. Wrap it so the gRPC status the filer sent stays reachable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: retry a writeback metadata flush until the filer answers With -writebackCache, close() returns before the chunk references reach the filer. The deferred flush gave up after four attempts, about a minute once each attempt's own filer retries are counted, then released the handle: a longer filer outage dropped the file even though write() and close() had succeeded, and its uploaded chunks became orphans. Keep retrying for as long as the filer is unreachable, as the writeback chunk uploads already do. A filer that answers with an error still ends the flush after the bounded attempts, and an unlink of the draining file still ends it on the next attempt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: retry only an unanswered filer call without a bound The unbounded phase used util.IsTransientError, which falls back to matching the error text. A refusal the filer sends in its response (read-only, name too long, a plain error string) has no gRPC status, and the flush error carries the file path, so a file named like "unavailable-transport.log" kept retrying against a filer that answered. Retry without a bound only when the call failed with a gRPC Unavailable status: the dial, the connection or the mount's call deadline gave out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
a6a3bdef8d |
s3api: deny signed reads of a version file through its physical path (#12574)
* s3api: deny signed reads that name a version file by its physical path A stored version lives at <key>.versions/v_<id>, and that path is also a valid object key. Anonymous reads already refuse it; signed GET, HEAD, copy sources and object subresources did not, so a caller without s3:GetObjectVersion, or under a Deny on the logical key, could read any version through it. Apply the same check in the shared entry recheck. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: recheck a version file key before answering a conditional read A conditional GET or HEAD answered 304 or 412 with the ETag before the handler's entry recheck, so the physical version path still confirmed a version's ETag to signed callers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: look up version file keys in the lookup-based policy recheck Retention, legal hold, tagging writes and deletes only fetched the entry for the recheck when the bucket had a policy, so without one they still acted on a version file named by its physical path. Fetch it whenever the key has the version file shape. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: refuse a directory-marker delete that names a version file A trailing slash kept "<key>.versions/v_<id>/" out of the version file shape check, and the lookup recheck misses the entry for that form, so a DELETE reached deleteDirectoryMarker and removed a chunkless version or delete marker without s3:DeleteObjectVersion. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
317d907334 |
filer: route tests forward to a peer port the test holds (#12576)
The ring tests used 127.0.0.1:18889 as an unreachable peer, but its gRPC port 28889 is a common filer port: on a shared CI host a live filer there accepted the forwarded exclusive create, and TestCreateEntryExclusiveLeavesNonOwner failed. The peer now listens on a port the test owns and drops every connection. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
github-actions[bot]
|
d51a79335c | docs: regenerate star history chart | ||
|
Chris LuandClaude Opus 5.5
|
13334a59b9 |
vector: reuse the Lance session, version leases and opened tables across reads (#12573)
* vector: share one Lance session across reads Each request opened its table in a session of its own, so every query loaded and decoded the index again. Get, list and query now open in one session the server keeps, sized by -index-cache-mb and -metadata-cache-mb; writes keep a session of their own. Lance keys some of its cached entries, such as a version's index list, by location and version alone, and an index dropped and created again at its location numbers its versions from 1 again: a shared session answered a query on the new table with the old table's index. CreateIndex now marks each table with an id in its schema metadata, and the session starts over when a location turns up holding another table. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: share a version lease among the reads of that version Every get, list and query tagged the version it read and deleted the tag afterwards: two filer writes and a manifest read per request. Reads of one version now share one tag, which goes 10 s after the last read took it; a tag is reused for at most 5 minutes, so a busy version never reaches the reapers' TTLs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: keep opened tables for the next read A read built the table's store and read its latest manifest each time. Reads now keep the opened table by location and encryption, up to 1024, and move it to the latest version, which lists the versions and takes the manifest from the session. A handle opened in an earlier session is not reused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: key shared leases by manifest, and take one per version at once A table dropped and created again within the linger reused, for a version number the old table had, a lease whose tag the drop had deleted. Leases are now keyed by the manifest's ETag as well. Reads that arrive before a version has a lease wait for the first to take it, rather than each writing a tag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: drop kept tables when the shared session starts over A kept table holds the session it was opened in, so a replaced session and its caches lived on in the handles of other tables until they were evicted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: prune shared lease slots only when they double Every lease lookup scanned all slots under the map's lock, a cost that grew with the versions read in the last linger. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: shrink the lease slot map after a prune After a burst the map kept its peak allocation, which every later prune scanned. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: keep a lease slot a read has taken through a prune A read that took a slot and had not yet locked it could see the slot pruned, and then pin through it while the next read pinned through a new slot, writing a second tag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
b924ffabf1 |
admin: calm console restyle with light and dark themes (#12568)
* admin: restyle the UI as a calm console Grey canvas with white sheets, a docked sidebar that marks the current page with an indigo tint, ink primary buttons, tinted status pills, dense tables with tabular numbers and spectrum capacity meters. The styles are design tokens layered over Bootstrap's own variables, with light and dark values, so the page templates stay unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: add a light, dark and automatic theme switch The choice is kept in localStorage and applied before the page paints, on the sign-in page too; automatic follows the system setting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: drop hard-coded colors from page templates The volume id link, utilization track and plugin form borders used fixed light-theme colors; sparklines now use the data palette. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: theme the plugin job detail panel background Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: keep the selected radio dot visible in dark mode Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: cycle the theme from memory when storage is blocked Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: keep success progress bars green Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: keep keyboard focus visible on the toolbar togglers Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: use contrasting text on danger and success buttons in dark mode Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: lead the dashboard with cluster health and one row of tiles The header now says in one line whether the cluster is healthy and how many masters, volume servers and filers report. The two rows of stat tiles become one: the volume size limit and EC volume count move into footnotes, the mount client card joins the cluster counts, and the counts link to their pages instead of sitting in nested cards. The duplicate version list and the bucket shortcut in the header are gone, and the section menus become plain View all links. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: export the user columns the table actually has The users table has username, email and access key columns, but the export waited for six cells per row and so always produced only the header line. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: drop table titles that repeat the page title List pages titled their only table with the page title again. The table now sits right under the summary tiles, and the export actions that were hidden in the table header menu move to the page header as an Export button, matching the cluster pages. The policies menu linked to nothing and is removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: drop clock and duplicate summary tiles Buckets, users, policies and service accounts spent a summary tile on the refresh time, which the page footer already shows (policies gains that footer). The users page also showed its user count twice. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: show unset bucket settings as a dash Rows of No owner, No quota and Not configured made the bucket table hard to scan. Unset cells now show a quiet dash that keeps the wording as a tooltip and for screen readers, and the name and creation date no longer wrap (the full timestamp is in the date's tooltip). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: export buckets through the page's API exporter admin.js loads after the page, so its copy of exportBucketList replaced the buckets page's own one. That copy scraped the visible rows, so it exported only the current page, without CSV quoting, and picked up the display text of the cells. The page's exporter reads every bucket from the API with full timestamps and escaped fields. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: escape quotes in the user export Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: download the bucket export as a Blob The exporter put the CSV into a data URL with encodeURI, which leaves # alone, so a bucket owner containing # ended the download there. Use the shared downloadCSV helper instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: count masters as configured, not reporting The master list comes from raft membership, which keeps a follower that is down, so it does not show which masters are reporting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: link the Prometheus metrics endpoint from the sidebar Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: drop the sidebar metrics link Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
1618d613a3 |
s3: resolve a role session's aws:username from its user, not its session name (#12572)
* s3api: record the caller on AssumeRole sessions The session name is the caller's choice; the session now carries the caller itself as the user it was issued to, as the web identity, LDAP and credential paths already do. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: take a role session's aws:username from its user, not its name An assumed-role ARN ends in the session name the caller picked, so naming a session after another user resolved ${aws:username} and ${aws:userid} to that user in bucket and attached policies. Resolve them from the user the session was issued to instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iam: take a role session's aws:username from its user, not its name IsActionAllowed set aws:username to the last segment of an assumed-role ARN, the caller-chosen session name, for role and bucket policies. Use the user the session was issued to, and drop a forwarded value when the session names none. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: pass a role session caller's user on to the session it assumes A role session that calls AssumeRole is named by its session ID, so the new session recorded that ID as its user. An STS session caller now passes on its own recorded user, or none. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: size the session claims map from the request context Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
3f557b73c8 |
admin: show the right master ports when -port.grpc is not port+10000 (#12570)
* admin: take master HTTP ports from raft member names The masters page and dashboard turned each raft member's gRPC address into an HTTP address by subtracting 10000 from the port, so masters with a custom -port.grpc showed wrong or negative ports. Raft members are named after their master address, which carries the HTTP port; use it and keep the old derivation only for names that are not addresses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: test raft member address mapping Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
a0f95e16bc |
s3api: fix TestRoutedMultipartFinalize flake from a reused fake filer port (#12571)
* s3api: drop cached gRPC channels when a fake filer takes a port A test that dials a dead owner leaves its failed channel in the process-wide gRPC connection cache. When a later test's fake filer is given the same port, its first RPC fails at once with the cached connection refused, which is how TestRoutedMultipartFinalize failed on the 32-bit job. The fake filer helpers now share fakeFilerAddress, which invalidates any cached channel for the new address. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: test a fake filer on a dead owner's reused port Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: drop cached channels for dead test filer addresses too closedFilerAddress can be handed a port whose cached channel belonged to a fake filer that just stopped and may still look ready; then the dead owner's transaction is sent on it instead of failing over. Build its address with fakeFilerAddress as well. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
6c1eec8703 |
s3: store the POST Object acl on the uploaded object (#12567)
* s3api: split upload ACL assembly out of preparePutObjectACL Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: store POST Object acl and grant fields on the object The form acl and x-amz-grant-* fields were authorized but never assembled, so the object kept no ACL: acl=private on a public-read bucket stayed anonymously readable and invalid values were accepted. Build them with the same rules as PutObject, which also covers the Block Public Access check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: test POST Object stores its upload ACL Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: take the POST Object ACL from the signed form only ACL request headers are not covered by the POST policy, so a holder of a signed form could add one and have it stored with the upload. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: reject a POST acl field the policy does not cover The acl form field now sets the stored object ACL, so like the x-amz-* fields it needs a policy condition, as AWS requires. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: test unsigned ACL headers and uncovered acl fields on POST Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: evaluate POST upload policies on the canonical ACL Policy conditions on the grant headers saw the raw form value, so an equivalent spelling of a denied grantee passed. Use the PutObject request shape that the PUT path evaluates. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: check bucket policy for POST ACLs with auth disabled PutObject applies the bucket policy to an explicit upload ACL when IAM is off; POST now does the same before storing it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: reject differing acl and x-amz-acl POST fields Both promote to the canned ACL header, so map order picked the stored ACL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: test POST grant canonicalization, auth-off ACL policy, acl conflicts Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: check the auth-off POST ACL policy after loading the bucket Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: reject an empty acl or x-amz-acl POST field beside a differing one Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: test the auth-off POST ACL policy on a cold bucket cache and empty acl aliases Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: load the bucket before per-key object authorization Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: test the POST ACL policy on a cold bucket cache with auth on Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: stop per-key object authorization when the bucket fails to load Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: test the cold-bucket POST ACL allow path and a failed bucket load Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
eb159f3181 |
lance: deregister and re-register need UpdateTable (#12552)
* lance: require UpdateTable to deregister a table Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: require UpdateTable to register over an existing table Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: test that GetTable alone cannot deregister or re-register a table Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: a register that finds a table the caller cannot read needs UpdateTable to bring it back Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: decide declare write access by whether the table existed, not by what the caller can read Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: test register and declare by a caller that cannot read the table Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: register clears the marker on the create branch only after checking it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: keep a register whose table was created when the marker check fails Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
4cb52d7705 |
filer: reopen a leveldb store wedged by a transient write error (#12565)
* filer/leveldb_handle: reopen a database whose journal latched a write error goleveldb keeps the first journal write error and returns it for every later write until the database is reopened, so one transient ENOSPC or EIO fails all writes until a restart. The handle offers the leveldb.DB methods the stores use; a write that fails on an OS error probes the storage and, once it accepts writes and no call holds the database, closes and reopens it and retries the write once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: open the leveldb stores through leveldb_handle leveldb, leveldb2 and leveldb3 now recover from a transient storage write error without a restart. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: leveldb stores return iterator errors from listings and deletes A listing on a database that cannot be read reported an empty directory, and DeleteFolderChildren wrote an empty batch, because the iterator error was never checked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer/leveldb_handle: test a reopen that fails after the probe passes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer/leveldb_handle: probe storage before draining readers A probe that stalls in Create, Write or Sync held mu exclusively and blocked every read on the handle. Reopen attempts are now serialized by their own mutex, and the probe runs before the drain. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer/leveldb_handle: hold new calls back while a reopen drains TryLock succeeds only at an instant with no reader, so overlapping listings could keep a reopen from ever draining while each write spun for the full drain timeout. The drain now holds new calls back for at most drainTimeout; a call nested inside a running one waits that out instead of deadlocking. A drain that times out holds off the next attempt for busyReopenInterval, so new calls are not held back over and over. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer/leveldb_handle: bound the wait for a running reopen A write that hit the latched error waited on the reopen mutex for as long as another caller's storage probe ran, so a stalled probe queued every writer. Callers now wait at most reopenWait and return their error, and still retry once a concurrent reopen succeeds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer/leveldb_handle: tests wait for the drain and readers explicitly The nested-read test polled for a drain that could start and end between polls and then spun forever; it now syncs on the probe, widens the drain and fails if the put returns first. The overlapping-readers test waits until every reader holds an iterator before it writes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
5150168368 |
s3: authorize an empty versionId as the current-object action (#12566)
* s3: authorize an empty versionId as the current-object action GET, HEAD and DELETE treat an empty versionId as the current object, but the action resolver keyed on the parameter's presence and authorized it as GetObjectVersion/DeleteObjectVersion, so a Deny on GetObject or DeleteObject did not apply. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: test an empty versionId against a version-only policy Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: cover a bare versionId on HEAD Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
0c6c6868de |
volume: refuse a tier upload whose .dat was written during the copy (#12562)
* volume: refuse a tier upload whose .dat grew during the copy AttachRemoteDatFile only checked that the data backend was still the one copied, so a needle write acknowledged after the copy read the .dat was missing from the remote object, and the upload then switched reads to it and removed the local .dat. Compare the .dat size with the copy under the same data-file locks; on a mismatch the handler deletes the copy and keeps the local file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: refuse a tier upload whose .dat grew during the copy Mirror the Go AttachRemoteDatFile check. The tier-up attach took the write lease only inside each step, so a needle write acknowledged after the upload read the .dat was missing from the object the volume then switched to, and the local .dat holding it was removed. Hold one write lease across the instance, revision and size checks, the remote-file update, the .vif save and the backend swap, as tier-down does; a size mismatch aborts, deleting the object and keeping the local .dat. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: compare the physical .dat size before a tier attach DiskFile pads its size past an unaligned tail, which a read-only load keeps, while the upload copies the physical bytes. Stat the file itself so such a volume is not refused as written during the copy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
dc03c3b1f7 |
shell: parallel moves and a per-run limit for volumeServer.evacuate (#12561)
* shell: parallel moves and a per-run limit for volumeServer.evacuate -maxParallelization runs up to N EC shard moves at once (default 1), with the ec.balance scheduler: never two shards of one volume at a time, free workers go to the least busy targets. The moves are planned first with the same target choice as before, then applied through volume_move.Mover.MoveEcShards, which verifies the target registered a shard before deleting it on the source. -movesPerExec limits how many volumes and EC shards one run moves (default 0 for unlimited), like -volumesPerExec in volume.balance, so a big server can be emptied in several runs. With -skipNonMoveable a shard that fails to move (a read error on a failing disk, for example) no longer drops the other shards of its volume; they are moved and the failures are listed at the end. Without it the run stops at the first failure as before. Losing the admin lock stops the run in either mode. Refs #12501 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: plan evacuation again after failed moves; fail on a lost lock A failed shard move kept the target slot it was planned on, so a readable shard that found no destination in the plan stayed on the server although the room was never used. Failed moves now give their targets back to the model and the shards without a destination are planned again, up to maxEcEvacuationRounds times; the failed shards are not tried again. With -skipNonMoveable a lost admin lock was printed as a skipped failure and the run returned success. ec.ErrLockLost is now exported and always fails the run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: volumeServer.evacuate reports -movesPerExec after an applied EC run The round loop only checked the limit after a failed move, so an applied run that stopped at -movesPerExec without failures never said to run it again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: volumeServer.evacuate plans again for as long as failed moves leave room The ten-round cap left readable shards on the server once more than ten rounds of unreadable shards had taken the free room first, and a rerun fails the same shards first again. Every round moves or fails each shard it plans and a failed shard is not planned again, so the rounds end without it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: volumeServer.evacuate gives a failed move back every slot it took When the target already held the shard in another disk type, adding it to the planned disk type still took a slot, but the undo gave back only the explicit one. Record the slots the plan took and restore exactly those. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec: an evacuation move that landed on its target keeps the target slot A move can fail after the target registered the shard, when only the source unmount or delete failed. Evacuate then gave the target's slot back and could plan another shard onto a full server. The runner now checks the target after a failed move and leaves a shard the target has out of FailedMoves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: volumeServer.evacuate gives back the slot of a failed move onto a shard the target had A target that held the shard before the plan reports it after any failed move, so the runner took such a move for one that landed and evacuate kept the slot the plan had taken. The runner now lists those moves in LandedMoves, and evacuate undoes the ones whose target had the shard, then plans again with the room they gave back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Chris LuandClaude Opus 5.5
|
fcfca3fee3 |
s3, filer: bound the wait for an upload slot (#12559)
* util: add WaitUntil, a cond wait bounded by a context and a timeout The caller holds the cond lock. Cancellation and the timeout wake the waiter through a broadcast taken under the same lock, so no wakeup is lost. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: bound the wait for an upload slot A PUT that arrived while the concurrent upload limits were exhausted waited on the condition variable with no bound. The handler reads nothing from the body while it waits, so a client that gave up and reset its connection went unnoticed: the socket and goroutine stayed pinned until a slot freed, and a stalled volume server grew the gateway's fd count by one per aborted PUT. The wait now ends on InflightUploadDataTimeout with 503 SlowDown, which S3 SDKs retry, and on request cancellation. An admitted upload takes its slot under the lock, and a finished upload broadcasts, so freed bytes admit every waiter that fits instead of one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: bound the wait for an upload slot The filer's PUT/POST upload limits queued requests the same unbounded way. Reuse the bounded wait: reject with 429 after InflightUploadDataTimeout, as the volume server does, and stop on request cancellation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * command: add -inflightUploadDataTimeout for the S3 gateway and the filer weed s3 -inflightUploadDataTimeout, weed filer -inflightUploadDataTimeout and -s3.inflightUploadDataTimeout, weed server and weed mini -filer.inflightUploadDataTimeout and -s3.inflightUploadDataTimeout. The default of 60s matches the volume server's flag of the same name; 0 keeps the old unbounded wait. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: reject a queued upload without reading its body WriteErrorResponse drains the request body before answering. Behind the stats recorder the drain has no read deadline, and reading the body sends an Expect: 100-continue client the go-ahead to upload what is about to be discarded. Answer the rejection with an empty-body copy of the request so the drain is skipped; net/http closes the connection when a large unread body remains. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3, filer: release an upload slot under the cond lock A waiter checks the counters under the cond lock and then waits. A release that decremented and broadcast without the lock could land between the two, so the waiter slept with a slot free until the next release or its timeout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3, filer: close the HTTP/1 connection when rejecting a queued upload net/http reads up to 256 KiB of an unread request body before sending a keep-alive reply, so a client that stalls its body held back the 503 or 429 until the connection idle timeout. Connection: close skips that read. HTTP/2 is left alone: there it would shut down the whole connection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
d659228338 |
shell: s3.clean.uploads frees an open upload under the object write lock (#12558)
* shell: s3.clean.uploads deletes uploads over filer gRPC Same DeleteEntry the HTTP handler runs, without minting a filer JWT, and the gRPC path is the one a locked, routed delete needs next. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: s3.clean.uploads frees an open upload under the object write lock The cleaner deleted a stale upload recursively with no lock, so a CompleteMultipartUpload could pass its upload-exists check after the part entries and their chunks were gone, and publish an object over freed chunks. Do what AbortMultipartUpload does: hold the object's distributed write lock, and free an open upload in an ObjectTransaction routed to the object's owner filer, conditioned on the object not carrying the upload; if it does, delete the metadata only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: s3.clean.uploads keeps the object write lock for the whole cleanup A large upload's recursive delete can outlast one 15 second lease, and a completion on the distributed-lock path could then take the lock and commit over chunks being freed. Renew the lock in the background and skip the delete once it is lost. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: s3.clean.uploads locks the path a completion commits under For a trailing-slash key a non-versioned completion writes and locks the entry inside the directory the key names, not the object path, so the cleaner's transaction did not serialize with it. Lock and check the same path the completion uses for the bucket's versioning state; the route key stays the object's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: abort locks the path a completion commits under A routed abort locked and checked the object path, but for a trailing-slash key a non-versioned completion commits under the entry inside the directory the key names. Use the same path the completion locks for the bucket's versioning state. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: end s3.clean.uploads bucket failure line with a newline Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * cluster: NewBlockingLongLivedLockWithin bounds the wait on a held lock Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: s3.clean.uploads skips an upload whose object lock stays held Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * cluster: share the contended lock filer setup across lock tests Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: s3.clean.uploads reads the bucket's versioning state per upload The commit path a routed completion locks depends on versioning for a trailing-slash key. Reading it once per bucket before the run left a window as long as the run; read it just before the delete, as the gateway's abort does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
b11af30cb1 |
mount: remove the local control socket when the mount exits (#12560)
* mount: remove the local control socket when the mount exits weed mount listened on /tmp/seaweedfs-mount-<hash>.sock and removed it only when a later mount of the same directory started, so every unmount or SIGTERM left one socket file behind. Remove it on the normal return path and in the interrupt hook, unless a later mount of the same directory has already replaced it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: remove the control socket before a failed mount exits glog.Fatalf and glog.Exitf call os.Exit, which skips the deferred removal, so a mount that failed after creating its socket (mount point not available, fusermount failure) still left it behind. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: register the socket cleanup hook before mount setup A SIGTERM that arrived while the mount was still setting up, for example waiting on the filer to create the mount root, exited through grace without running the deferred cleanup. Registering the hook right after the listener is created makes it run last, after the unmount and filesystem hooks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
7f18576798 |
filer/etcd: serve volume.fsck from one etcd revision (#12555)
* filer/etcd: scan entries from one revision volume.fsck walked etcd one directory at a time, so a file moved from a directory not yet listed into one already listed was seen under neither name and its chunks were reported as orphans. The etcd store now implements filer.EntryScanner: every page is read at the revision of the first read, and a compacted revision fails the scan instead of falling back to the walk. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer/etcd: read hard links at the scan revision Hard-link records resolved during a scan were read at the current revision, so a link updated meanwhile mixed two instants. Reads made with the scan's context now use its revision. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: list etcd among the stores volume.fsck reads from a snapshot Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer/etcd: skip hard-link records in a scan Hard-link records live beside the entries under their random ids, so a root scan could read one as an entry with a path made of random bytes, which the traversal stream cannot send. Skip hard-link ids and keys whose directory is not valid UTF-8. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer/etcd: recognize a hard-link record by the id it stores A hard-link id can be valid UTF-8, so the key shape alone let some records through a root scan as phantom files. The record's value is the shared entry, whose HardLinkId is the record's own key; no entry key can equal its own hard-link id. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer/etcd: scan entries under an invalid-UTF-8 directory The filer sanitizes names but stores directories as given, so a path written over HTTP can leave entries the listing never reaches. Skipping them let volume.fsck count their chunks as orphans. Send them under the sanitized directory instead, the substitution the filer uses for names. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer/etcd: fail a scan whose sanitized path names another entry An entry under an invalid-UTF-8 directory is reported under the sanitized directory. When that path is also another stored entry at the scan revision, or another sanitized entry in the same scan, acting on the reported path (volume.fsck deleting it) would reach the other file, so the scan fails and names the stored path instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
e23a952878 |
ec: don't index ev.Shards[0] when the last shard was unmounted (#12538)
* ec: don't index ev.Shards[0] when the last shard was unmounted * ec: skip a runtime emptied before the scrub merge reads its shards FindAllEcVolumes releases each location lock before MergeEcRuntimes runs, so an unmount can leave a merged runtime with no shards and the width scan indexed Shards[-1]. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec: treat a scrub merge with no shard left as a vanished volume Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec: read Shards once in ShardSize so a concurrent unmount cannot empty it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec: read each runtime's Shards once in the scrub merge The vanished check and the slot construction read Shards separately, so an unmount between them produced a non-nil view with no shards. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec: anchor the scrub merge on a generation that still holds shards An unmount can empty the newest runtime while an older one keeps its shards, which left a merged view anchored on the emptied generation with no slots and the holder reported as skipped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Mansur Akhmadi <10739460+mansur-akhmadi@users.noreply.github.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
641ab6204b |
shell: fs.mergeVolumes leaves in-progress multipart parts in place (#12556)
* shell: fs.mergeVolumes leaves in-progress multipart parts in place CompleteMultipartUpload reads the part entries and later commits the object with their fids, guarded only by the upload directory. A merge that moved a part chunk in between deleted the source needle the new object pointed at. Parts stay where they are until the upload completes or is cleaned, and their chunks still count as referenced sources. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: note in fs.mergeVolumes help that upload parts stay in place Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: match upload parts when the buckets folder is the root Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: count chunks behind a part's manifest as referenced sources A skipped part whose chunks on a merge source sit behind a manifest was not counted, so the merge printed no skip line and warned that nothing references the source. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: keep a part's direct source references when its manifest is unreadable Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: count chunks resolved before an unreadable part manifest ResolveChunkManifest returns the data chunks of earlier manifests along with a later manifest's error; record them before returning so the unreferenced-source warning does not fire for a source they reference. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: test the partial manifest result without a timing delay The unreadable manifest is now nested under a readable part manifest, so the first level resolves fully before the failing lookup runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
e9606c8ecc |
filer.meta.backup: resume an interrupted metadata copy and add -entriesPerSecond (#12549)
* filer_pb: add a depth-first traversal that can resume after a path TraverseDfs walks a tree in listing order and reports each finished path, so a caller can save one path and continue from it later. Pages are read whole before visiting, and the entries of a page are visited by up to five workers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.meta.backup: resume an interrupted metadata copy The walk saves the last finished path every few seconds, and a walk that continues an unfinished one starts after it. A fresh walk clears the saved path first, and a finished walk clears it with the walk start. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.meta.backup: add -entriesPerSecond to throttle the metadata copy Each entry the full copy reads waits on a rate limiter, and pages are listed only after the previous page is copied, so the flag also bounds how fast the copy lists the source filer. 0 keeps it unlimited. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer_pb: report each file of a depth-first walk once all before it are done A throttled walk only reported progress after a whole listing page, so a slow copy could go many minutes without a resumable path. Visit entries in runs that end at a directory and report each file as soon as every entry before it is done. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
0ef0260844 |
volume: fail a vacuum step that finds the volume already compacting (#12553)
* volume: fail a vacuum step that finds a compaction already running Compact, CompactByIndex and CommitCompact returned success when another compaction held the volume. A second, uncoordinated vacuum (an expired admin job still running, a shell volume.vacuum next to the worker) then went on to commit or clean up the first run's .cpd/.cpx. Return a "volume N is compacting" error instead, and refuse cleanup while a copy is in flight, as the Rust volume server already did. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: check the compaction files before a commit closes the volume CommitCompact closed the needle map and data file and wrote the .cpc marker before it found .cpd/.cpx missing. A duplicate commit, such as the second of two overlapping vacuums, then left the volume registered with no open files: it dropped out of the heartbeat and looked lost to volume.list and volume.fix.replication until a restart. Check for the files first so the refused commit leaves the volume as it was. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: run makeupDiff before a commit closes the volume CommitCompact closed the needle map and data file before makeupDiff, so a makeupDiff failure (a compaction revision another commit moved, an I/O error) returned with them closed: the volume stayed registered but missing from the heartbeat until a restart. makeupDiff reads the live files through its own handles while the commit holds the write lock, so close them only once it has succeeded, as the Rust volume server does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: log a volume the heartbeat skips because its files are closed A volume left without its needle map or data file vanished from the heartbeat with no log line, so the master saw a replica disappear while its files were still on disk. Warn on each skip, as the deleted-.dat case next to it already does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: claim the compaction flag while cleanup removes the temp files Cleanup only read the flag, so a compaction could start between the check and the removal and lose its .cpd/.cpx. Claim it for the cleanup instead, in Go and Rust. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
a4a68c885a |
filer.sync: move the watermark over jobs that finished out of order (#12554)
* filer.sync: move the watermark over jobs that finished ahead of an older one The watermark only took the finishing job's timestamp when it was the oldest in flight, so jobs that completed before an older one were never recorded and a quiet stream kept the offset behind finished events. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.sync: keep one finished timestamp per gap behind a running job A long job kept every later completion in the done heap. Only the newest one between two running or failed timestamps can become the watermark. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.sync: test done-heap compaction across several running and failed jobs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
000baf8dfa |
s3: return an error for GETs canceled before the first byte (#12502)
* s3: answer GETs canceled before the first byte A stream that fails before writing anything now always gets an S3 error response, canceled errors included. The idle timeout and a shared lookup canceled by another request both cancel the request context while the client is still connected, and returning silently sent that client a 200 with Content-Length: 0. A client that has gone away never sees the write. Refs #12500 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: mark inline GET write errors as written; bound the idle-cancel test An inline object's write error comes after the status line, so the caller must not write an S3 error on top of it. The idle-cancel test caps the client wait and gives the response write its own deadline, keeping only the read side on the 200 ms idle timeout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: test a GET whose request is canceled before the first byte Drives GetObjectHandler through a chunk lookup that cancels the request context, so a handler that returns before answering shows up as the empty 200 again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: check the cancel test reached the chunk lookup Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Chris LuandClaude Opus 5.5
|
ea6e9d670a |
volume: forward no Content-MD5 for a JPEG the orientation fix rewrote (#12557)
* volume: forward no Content-MD5 for a JPEG the orientation fix rewrote With -images.fix.orientation the primary stores and forwards the rotated bytes, but sent replicas the client's digest of the original upload. The replica checks Content-MD5 against what it receives and rejects the write, so replicated JPEG uploads with a digest failed. The Rust volume server already drops the digest once the body is rewritten. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: test the forwarded Content-MD5 through PostHandler and a replica The test now posts each upload to a primary whose fake master names a real replica volume server, so it covers the JPEG gate, ReplicatedWrite and the replica's own Content-MD5 check, and reads the stored bytes back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: fail the replica forwarding test instead of hanging when nothing is forwarded Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: compare the primary and replica bytes in the replica forwarding test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
1202c5ba08 |
filer.copy: compute chunk md5 so files get a real ETag (#12497)
* filer.copy: compute chunk md5 so files get a real ETag Chunks uploaded by filer.copy had no ETag, so every multi-chunk file got the md5 of the empty string plus -N. An overwritten file kept its old ETag and a revalidating client got 304 for changed content. * filer.copy: store chunks in offset order Chunks are collected in upload completion order, and the entry ETag hashes the chunk ETags in stored order. Identical files copied with concurrent chunk uploads got different ETags, and a file with swapped equal-sized chunks could keep the old one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.copy: test chunk order and chunk md5 on chunked uploads Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
b9866bae4f |
weed/shell: support -noLock flag in volumeServer.evacuate (#12501) (#12516)
* weed/shell: support -noLock flag in volumeServer.evacuate (#12501) * shell: return flag parse errors from volumeServer.evacuate A typo such as -noLock=invalid printed the parse error but the command still succeeded, so a piped script treated the evacuation as done. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: volumeServer.evacuate -h prints usage without failing Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
09a63031c1 |
helm: set replication and disk on buckets created by the chart (#12543)
* helm: set replication and disk on buckets created by the chart * helm: test bucket replication and disk without ttl * helm: reject unquoted bucket replication values * helm: treat a null bucket replication as unset A blank replication key in a values file failed the render with the quoted-string error, while blank ttl and disk are skipped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * helm: require a three-digit bucket replication Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * helm: note that removing bucket settings keeps the stored rule Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * helm: reject bucket replication above 255 fs.configure rejects an xyz value above 255, so the hook Job failed after the release rendered. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
f068fe7f09 |
fix(shell): fs.log.purge purges every filer, not just one (#12524)
* fix(shell): fs.log.purge purges every filer, not just one Each filer keeps its own metadata logs in its own store, so purging a single filer leaves the chunks referenced by the others and the volume vacuum cannot reclaim anything. The shell picked that single filer at random (commands.go:208) when no -filer flag was given, and the admin script worker pinned it to filers[0] (admin_script_handler.go:574), so the same command purged a different filer on every run and never the whole cluster. fs.log.purge now resolves the filer list from the master and purges each of them, continuing when one is unreachable and reporting the failures at the end. -filer=<addr>,... restricts the purge to a subset. Also stops mutating the shared command environment per filer, so a long-lived shell keeps its original filer. * fix(shell): bound filer discovery and cover the cluster path Address review feedback: - resolveLogPurgeFilers waited on the master with context.Background(): a shell whose master connection dropped would hang before purging anything, where it used to purge the connected filer. Bound the master round trip with a timeout and fall back to the connected filer when the master cannot be reached or lists no filer. - the tests only exercised the single-filer fallback. Rename that test to what it actually covers and add one asserting every filer of the list is attempted and the shared command environment keeps its original filer. * shell: reject an fs.log.purge -filer list with no addresses A comma or whitespace only -filer value fell through to master discovery and purged every filer instead of the intended subset. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: fall back to the shell's filer without a second master lookup When the master listed no filer, FilerAddress() ran filer discovery again and could wait out the shell's ConnectTimeout before failing. Also cover master discovery with the fake discovery master. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: test the fs.log.purge filer loop against a fake filer The previous test looped over the addresses itself, so it passed no matter what Do did. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Martin Morez <martin@morez.org> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 4.8
|
fb5ffb815f |
filer: validate bucket names before the SQL DDL hooks (#12547)
* filer: validate bucket names before the SQL DDL hooks The SQL store builds CREATE TABLE and DROP TABLE statements by formatting the bucket name into the query. The DML paths already reject names that fail VerifyS3BucketName, but the bucket-creation and bucket-deletion hooks ran the DDL without that check, so a name reaching them through the filer or WebDAV directory path (which skips the check for dot-prefixed directories) could inject stacked statements. Apply the same validation the DML paths use. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * filer: check that a valid bucket runs both DDL statements Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
d350a0ad03 |
weed-db: end startup when password authentication fails (#12550)
* weed-db: end startup when password authentication fails The failure branches returned the result of writing the error response, which is nil on success, so the handshake carried on to ReadyForQuery. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-db: test startup with wrong and unknown credentials Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-db: check the auth error code and fail on stray read errors in the startup test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
f3150c3495 |
admin: escape file properties before rendering the modal (#12548)
* admin: escape file properties before rendering the modal The Properties modal concatenated the entry name, path, MIME type and extended attributes into HTML, so object metadata written over S3 could inject markup into the admin page. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: keep every property key when escaping Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
8d96985131 |
security: scoped filer tokens are not admin credentials (#12551)
* security: do not accept a path- or method-scoped filer token as admin Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * security: test that scoped filer tokens fail the admin token check Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
2143761319 |
fix(s3tables): let a table renamed across namespaces keep committing (#12515)
* s3tables: let a table renamed across namespaces keep committing
* s3tables: keep the same-directory exception off a directory another table shares
A record written before metadata locations were confined to their namespace
(
|
||
|
Tsic
|
518009dcd4 |
filer: apply filerGroup prefix to bucket default collection (#12499)
filer: prefix bucket default collection, assign copy collection explicitly S3 server-side copy paths (CopyObject/UploadPartCopy) assign volumes with an empty collection and relied on the filer's bucket fallback. The fallback used the raw bucket name, while PutObject uses getCollectionName(bucket) = <filerGroup>_<bucket>, so copies landed in a second, unprefixed collection. Prefix the filer's bucket default with the filer group, and have the S3 copy path pass that collection explicitly, like PutObject. Direct filer writes keep path-rule precedence, so this only aligns S3 copy with S3 put. Adds regression tests for the fallback, explicit collection precedence, and the S3 copy collection resolver. |
||
|
Ryan Leung
|
7ad8419815 |
shell: preserve completed uploads for bucket-root objects (#12545)
Signed-off-by: rleungx <rleungx@gmail.com> |
||
|
github-actions[bot]
|
9799b82edf | docs: regenerate star history chart | ||
|
Mansur AkhmadiandMansur Akhmadi
|
9886ac0bcf |
topology: take dn.Lock when a heartbeat links a new disk type (#12535)
Co-authored-by: Mansur Akhmadi <10739460+mansur-akhmadi@users.noreply.github.com> |