mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-05 11:37:44 +01:00
814ee75af49870a379cfc5a23842c1fa57777499
1335
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris Lu
|
814ee75af4 |
s3: allow-unordered is a listing parameter, not an unimplemented subresource (#10846)
The guard that stops a bucket GET with an unknown subresource from being answered with a listing does not know about allow-unordered, so it answers 501 NotImplemented - to a parameter the listing handlers already read and already validate against delimiter. This is why test_bucket_list_unordered and test_bucket_listv2_unordered fail in the Ceph s3-tests suite. They fail on master too; this is not a Lance change and can be taken on its own. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm |
||
|
Chris Lu
|
e5edd8be3c |
s3: place multipart part chunks by the destination object's storage rule (#10845)
Multipart parts stage under /buckets/<bucket>/.uploads/<id>/, so the filer resolved filer.conf storage rules against that path when the gateway assigned volumes for them. A rule scoped to a key prefix - fs.configure -locationPrefix=/buckets/b/data/ -ttl=30d - then matched a small object but not the parts of a large one, so an object whose entry carried the rule's TTL had its bytes spread over TTL-less volumes. Assign part chunks against the destination object's filer path instead, the way the x-seaweedfs-destination header made the filer resolve it before the S3 write path moved off the filer proxy. Covers PutObjectPart and both UploadPartCopy paths. The part entry itself is still written under .uploads, so a read-only rule there still rejects it. The lifecycle XML Expiration.Days TTL keeps passing 0 for parts: that rule targets the user-visible object key and would start its clock before CompleteMultipartUpload. |
||
|
Chris Lu
|
abd61de52c |
S3: stamp the gateway's own uid/gid on PutObject and copy entries (#10844)
* fix(s3): stamp the gateway's own ids on single-shot PutObject entries putToFiler builds the entry in the gateway now instead of proxying a PUT to the filer, and it hardcoded Uid/Gid 0 while every sibling write path stamps filer_pb.OS_UID/OS_GID. On a non-root deployment that leaves single-shot PUTs and multipart parts owned by root while directories and completed multipart objects keep the real ids, so a mount reader can list the tree but gets EACCES on every open once objects are not world-readable. * fix(s3): stamp mode and ownership on copy destinations CopyObject and UploadPartCopy build the destination attributes themselves and then assign them over the entry filer_pb.MkFile just stamped, so the copy landed with mode 0000 and uid/gid 0 - unreadable on a mount even by the filer's own user. Build the destination with the same mode PutObject resolves for the request and the gateway's own ids. |
||
|
Chris Lu
|
1354b58675 |
s3: stop unrouted bucket subresources from being answered with a listing (#10814)
* s3: answer GetBucketReplication, GetBucketWebsite and GetBucketNotificationConfiguration None of the three had a route, so they reached the unconstrained ListObjectsV1 catch-all and a client asking for a bucket's replication config got 200 and a <ListBucketResult> back. Replication and website report their configuration as absent the way AWS does; notification returns the empty configuration AWS returns for a bucket with no events wired up. * s3: stop an unrouted bucket subresource from being answered with a listing ListObjectsV1 is the catch-all GET on a bucket, so every subresource without a route of its own - ?torrent today, whatever AWS adds next - came back 200 with a <ListBucketResult>. A client that asked for a configuration and got a listing either fails its XML decode in a way that reads like corruption, or worse, tolerantly parses it. Refuse the request instead. The allow-list is the ListObjects parameters rather than the subresources, so a new one fails closed. Presigned URLs sign their credentials into the query string, so X-Amz-* and the SigV2 trio have to stay listable. |
||
|
Chris Lu
|
5d5ea63b3f |
Fix what the Go 1.26 language bump breaks (#10794)
* worker: log the balance move stage through a constant format string Go 1.26's printf analyzer now follows printf wrappers reached through an interface, so passing the stage straight to Logger.Info is a vet failure. * s3api: bracket the IPv6 host in the signature test URL A bare IPv6 literal is legal in a Host header but never in a URL. Go 1.26 stopped parsing it leniently, so carry the two forms separately and set r.Host to the value the client would actually have signed. * mini: bracket IPv6 addresses in the readiness probe URLs An IPv6-only host hands mini a bare literal, and %s:%d pasted it into a URL unbracketed. Under Go 1.26 that URL no longer parses, so waiting for the admin server never succeeds and mini refuses to start. |
||
|
Chris Lu
|
7522e17b6d |
iceberg: vend table-scoped credentials to clients that ask for delegation (#10777)
* iceberg: vend table-scoped credentials to clients that ask for delegation The catalog recognised X-Iceberg-Access-Delegation: vended-credentials and then deliberately said nothing, because it had nothing to vend: it withheld even the S3 endpoint so the client would keep the credentials it was configured with. That left every engine expecting the catalog to hand out access - Snowflake, Databricks, Trino with vending, any multi-tenant setup - needing static S3 keys distributed out of band. Mint an STS session per request instead, scoped by a session policy to the table's own prefix plus the bucket listing needed to resolve it, and return it in the load response config and storage-credentials. The role to assume is named by -s3.iceberg.credentialRole; its trust policy is what decides whether a caller may assume it, and vending stays off until it is set. A failed mint falls back to the old silence rather than handing back an endpoint the client cannot sign for. * iceberg: keep vended credentials inside the table prefix Review follow-ups on credential vending: Listing was granted on the bucket ARN with no condition, so a credential vended for one table could enumerate every other table's object names. Constrain s3:prefix to the table's own prefix, which the S3 gateway already populates for list requests. A table location carrying * or ? would have gone into the policy's resource pattern unescaped and widened the session to sibling prefixes. Refuse to vend for such a location rather than escaping it; nothing the catalog generates contains those characters. DurationSeconds skipped the 900..43200 bounds the other assume-role paths enforce, so -s3.iceberg.credentialDurationSeconds could ask for a session outside them. The check is now shared by all three entry points. * iceberg: return the vended credentials from buildFileIOConfig itself buildStorageConfig was a second name for what buildFileIOConfig already did; it now returns the storage credentials alongside the properties, and callers that only want the properties drop them. * iceberg: split the vended bucket grants, and refuse a whole-bucket scope The prefix condition sat on a statement that also granted GetBucketLocation and ListBucketMultipartUploads, neither of which carries an s3:prefix to satisfy it, so both were denied for every vended credential. GetBucketLocation moves to its own unconditioned statement. ListBucketMultipartUploads is dropped: Iceberg writers complete and abort by upload id, and granting it either leaks in-flight keys bucket-wide or breaks on the same missing prefix. A table whose location has no prefix - one registered at the bucket root - would have been vended read and write over every other table in the bucket. Refuse, the way a location with wildcards is refused. |
||
|
Chris Lu
|
ec37ef5aaa |
iceberg: add view rename, scan-report and snapshots=refs to the catalog (#10776)
* iceberg: add view rename, scan-report and snapshots=refs to the catalog Three gaps against the REST spec that clients hit in normal use: Views had no rename, though tables did and views are stored the same way, so the move is the same catalog-only pointer move. Tables and views share a namespace directory, so both renames now refuse the other kind instead of moving it. Engines POST a scan or commit report after planning; a 404 there turns into an error line per query. Accept the report and discard it - the catalog keeps no metrics store. LoadTable ignored ?snapshots=refs and always returned the whole snapshot history, which is what clients use the parameter to avoid on long-lived tables. * iceberg: authorize view rename against the view ARN, tighten the metrics endpoint Review follow-ups: The shared rename checked the source against a table ARN whatever the kind, so a policy scoped to a view's own ARN never matched and one written for a table ARN was evaluated for a view. The entry kind now carries the ARN builder. The metrics endpoint truncated a report at 1 MiB and then failed to parse it, answering 400 for a query that had actually succeeded. Read one byte past the limit to tell "fits" from "cut short", and discard an oversized report instead of rejecting it. Empty bodies and reports without a report-type are now rejected, which the REST schema requires. ?snapshots= is defined for LoadTable, so it no longer filters what CreateTable echoes back. |
||
|
Chris Lu
|
d044839ab2 |
iceberg: make a table commit a compare-and-swap (#10775)
* iceberg: make a table commit a compare-and-swap
The catalog validated the caller's version token, ran its authorization
checks, and only then wrote the new metadata xattr. Two engines
committing against the same base both passed that check and both wrote,
so the second silently dropped the first one's snapshot. Both also derive
the same v{N}.metadata.json name and the file write overwrote, leaving
the surviving pointer aimed at the loser's metadata - and the loser's
conflict cleanup then deleted the winner's file.
Write the metadata file with an exclusive create and update the xattr
conditionally on the bytes the handler read, the way the maintenance
worker already commits. A writer that lost the race re-reads and retries,
and reports 409 CommitFailedException once out of attempts.
* iceberg: stage a commit under a unique name when the versioned one is taken
Two follow-ups from review of the commit compare-and-swap:
Refusing to overwrite v{N}.metadata.json also refused to get past a file
left behind by a commit that died between staging and updating the
pointer. Every later commit derived the same name, saw the collision, and
reported a conflict, so the table stayed uncommittable until an orphan
sweep removed the file. Stage under v{N}-{uuid} instead: neither writer's
file is overwritten and the catalog pointer still decides who won, which
is how the maintenance worker has always staged its own metadata.
metadataVersionFromLocation learned to read the version back out of that
name.
The conditional update guarded only the metadata attribute while the
write replaced the whole entry, so a policy or tag written in the same
window was silently reverted. Guard every catalog attribute, which turns
that into a conflict the caller retries on fresh state.
* iceberg: give saveMetadataFile the exclusive flag instead of a second name
saveNewMetadataFile, saveMetadataBlobExclusive and uniqueMetadataFileName
were three new names around one existing helper. The flag now rides on
saveMetadataFile and saveMetadataBlob, and the unique-name construction
sits where it is used.
* iceberg: reuse the filer CAS helpers #10773 added, and stage transactions exclusively
#10773 landed mutateEntryExtended, which already writes an entry back under a
whole-entry precondition and retries. Drop the helper this branch added and
route the table commit through it: the check that the metadata is still the
one this request read now lives in the mutation, where it sees current state.
The policy the request was authorized against is asserted too, so an
administrator restricting it mid-commit sends the caller back through
authorization instead of having a stale decision applied. Bucket and
namespace policies live on other entries and a single-entry precondition
cannot cover them.
Multi-table transactions stage their metadata exclusively for the same
reason single-table commits do, and carry the name they landed on into the
pointer flip.
|
||
|
Chris Lu
|
eef6f3d1e6 |
s3tables: add the maintenance configuration APIs (#10773)
* s3tables: add the maintenance configuration APIs Stores the configuration verbatim as the wire shape under a new s3tables.maintenance extended attribute, so Get hands back what Put took and no translation layer can drift from the AWS model. Nothing reads the configuration yet. Put merges a single type into the stored map so configuring compaction does not drop snapshot management, and asserts the attribute's prior value so two concurrent Puts cannot silently clobber each other. * iceberg: apply the maintenance configuration in the worker The worker now reads the per-table and per-bucket maintenance configuration written by the control plane, so the wildcard plugin config is a default rather than the only setting a table can have. Table properties still win by default, since a table declaring its own layout is what every engine honours and the compactor has to agree with whoever writes the files. Clearing table_properties_override makes the maintenance configuration authoritative instead. Status is not part of that contest: a disabled type drops its operations and no property can re-enable them, so the operator's kill switch always holds. Manifest and delete-file rewrites have no AWS equivalent and ride with compaction. Detection reads both attributes from entries it already lists. * s3tables: report maintenance job status The worker records the outcome of each run in its own extended attribute, separate from the configuration so operator and worker writes do not contend, and GetTableMaintenanceJobStatus reads it back. Only the types a run touched are written, so a partial run cannot erase what an earlier one recorded. The reader fills in the rest: Disabled when the configuration switched a type off, Not_Yet_Run otherwise. Status is advisory, so a lost race is logged rather than failing a job whose work already committed. * s3tables: route the maintenance APIs over REST The five actions were only reachable by X-Amz-Target dispatch, which the AWS CLI and SDK do not use for this service. They address the operations by path, so the APIs were unreachable from any official client. * s3tables: fix the table bucket ARN field name GetTableBucketMaintenanceConfiguration emitted tableBucketArn where the wire field is tableBucketARN, as every other response in this package already spells it. Official SDK deserializers ignore the unknown key, so the required field came back unset. * s3tables: carry the compaction strategy through to the worker IcebergCompactionSettings modelled only targetFileSizeMB, so a request naming a strategy was accepted and then dropped on the way to storage. The worker now maps binpack and sort onto its own rewrite strategy and lets auto defer to the worker configuration. z-order is rejected rather than accepted and quietly binpacked. * s3tables: report bucket-level maintenance status GetTableMaintenanceJobStatus read only the table's configuration, so unreferenced file removal — which is configured on the bucket — reported Not_Yet_Run or a stale success after an operator disabled it. The merge helper now lives in this package and the worker shares it. * iceberg: delete orphans only after the non-current window AWS marks a file non-current once it has been unreferenced for unreferencedDays, then deletes it a further nonCurrentDays later. The cutoff was taken from unreferencedDays alone, so a 3/10 configuration hard-deleted on day three and threw away the ten day recovery window. remove_orphans deletes in one step rather than marking, so the cutoff is now the sum of the two. * s3tables: assert every attribute when rewriting an entry UpdateEntry writes the whole entry back from the snapshot the caller read, and its precondition only covers the keys the caller names. Both maintenance writers named one key, so a job status write could revert a maintenance configuration an operator had just disabled, turning an advisory write into a silent re-enable. Both now assert the entry's full attribute set, including the target key when absent so a concurrent create also fails the precondition. * s3tables: assert absent attributes when rewriting an entry The precondition covered the attributes present when the writer read the entry, so an attribute created between that read and the write was absent from it. A first-time PutTableMaintenanceConfiguration disabling a type therefore lands, passes the per-key checks, and is then deleted by the stale whole-entry write. Every attribute this package stores is now asserted, absent ones included. The metadata commit and planning index writers rewrite the same entries and had the same exposure, so both use the shared snapshot too. * iceberg: implement the auto compaction strategy auto was accepted, stored and read back, but left the worker on its own default, so a sorted table configured as auto was compacted with binpack. AWS defines auto as sorting tables that declare a sort order and bin-packing the rest. That needs the table metadata, so the choice is made where the rewrite plan is resolved: an unsorted table falls back to binpack rather than failing the way an explicit sort request does. * s3tables: validate the maintenance setting ranges PUT accepted zero, negative and oversized values for every numeric setting. The worker then ignores a non-positive value and saturates an oversized one, so the configuration read back was not the one that ran. AWS bounds all five to 1..2147483647, which is now enforced. The fields are pointers so an explicit zero is distinguishable from an omitted one and can be rejected rather than silently ignored. * s3tables: give every entry writer the same compare-and-swap updateExtendedAttribute asserted the entry's attributes, but the helpers behind the metadata, policy and tag handlers still wrote the whole entry unconditionally. Any of them could land on a stale snapshot and delete a maintenance configuration an operator had just written. They all share one read-modify-write loop now, so the precondition and the bounded retry apply wherever an entry is rewritten. * s3tables: move the maintenance configuration with a renamed table RenameTable carried the metadata, version, policy and tags to the new name but left the maintenance configuration and job status behind. A table with snapshot management disabled came back enabled under its new name, and the stale configuration stayed on the old name where a table created there would inherit it. The decoupled-delete cleanup left the same two attributes behind. * s3tables: accept every AWS partition in ARNs The route regexes and the ARN patterns both hardcoded arn:aws, so valid aws-cn and aws-us-gov ARNs never reached a handler. The router now shares the partition-tolerant prefix with the parser, and a generated ARN uses the partition its region belongs to so it parses back. * s3tables: generate ARNs in the region's partition The handler's own ARN generators still formatted arn:aws directly rather than going through the partition-aware builder, so a China or GovCloud deployment routed the request but then returned a commercial ARN and matched IAM policies against it. The round-trip test missed this because parsing accepts any partition, so it now asserts the prefix the region implies. * s3tables: complete the ARN partition table aws-iso-e, aws-iso-f and aws-eusc were missing, so eu-isoe-*, us-isof-* and eusc-* regions fell through to the commercial partition. * s3tables: do not let a rename swallow a concurrent maintenance write Rename copied the source attributes early and cleared the source at the end, so a Put landing in between missed the copy to the destination and was then deleted by the cleanup. It succeeded and vanished. The cleanup now clears the source only while it still holds exactly what was copied, and returns a conflict otherwise. Put checks the catalog identity inside the same conditional mutation, so it also cannot write to a name that a rename or delete has already soft-deleted. |
||
|
Chris Lu
|
f66d6ffc4a |
s3: option to disable bucket auto-creation on upload (#10759)
* s3: add option to disable bucket auto-creation on upload * command: expose -autoCreateBucket in s3, filer, server, and mini * s3: apply the bucket auto-create policy to directory marker uploads * s3: validate the bucket name before the auto-create disabled check * s3: cover the disabled auto-create gate at all three upload entry points |
||
|
Chris Lu
|
db5a086d04 |
read cold remote objects straight from the origin while caching (#10731)
* refactor: extract remote mount resolution into shared helpers * refactor: share the adaptive remote cache wait policy * filer: stream cold remote reads from the origin while caching * s3: stream cold remote reads from the origin instead of 503 retries * test: cover the S3 origin stream-through path * remote mounts: match on path components and prefer the longest mount * fail short origin streams instead of silently truncating * s3: try the origin before failing a cold read on a local cache error * s3: gate origin streaming on the entry's resolved version * return the cache RPC's NotFound as a canonical status and classify it everywhere * filer: keep multipart-range cold reads on the retry path |
||
|
Chris Lu
|
7c87d78ea2 |
s3: a key deleted after enabling versioning must leave the listing (#10684)
* s3: a null object wins over a rescan when the latest-version pointer is absent The read path already resolves an absent pointer this way; the listing-path counterpart scanned .versions/ first and could surface an old version or delete marker over the current suspended-versioning null object. * s3: dedup a key against its .versions sibling in suspended buckets too A suspended bucket keeps its .versions directories, so a suspended-versioning null object and its .versions sibling emitted the same key twice. * s3: retract a null object from the listing when a delete marker shadows it Deleting a key whose null version predates versioning leaves the base-path entry in place and records the delete marker under <key>.versions. The listing appended the base-path entry and relied on the .versions sibling to replace it, but a delete-marker current version emitted nothing, so the deleted key stayed visible to ListObjects while GET and HEAD returned 404. * s3: keep a key's .versions sibling on the same page as the key When the page quota ran out between a base-path entry and its .versions directory, the page ended with the stale entry and the next page skipped the directory as a marker echo, so the replacement or retraction never happened. * s3: the null version is not latest when the .versions pointer names a newer one ListObjectVersions stamped IsLatest on every base-path null object, so a key deleted after enabling versioning reported IsLatest on both the delete marker and the null version. * s3: test listing after a pre-versioning null object is delete-marked * s3: find a key's earlier page entry by scan, not by adjacency A key such as k.bak sorts between k and k.versions, so the entry a .versions sibling replaces or retracts is not always the last one on the page. Scan back through the page for the key, and insert a late resolution in sorted position instead of at the end. * s3: settle trailing null objects by lookup when a page fills The quota can run out while keys still sit between a null object and its .versions sibling, and the sibling-adjacent page-boundary exception never fires for those. Track the trailing null objects whose sibling has not been ruled out and look each one up before declaring the page full; a retraction reopens the quota. * s3: do not resolve a .versions sibling its page has already moved past A page resuming from a marker inside the base key's extension region has already listed and settled the base null object on an earlier page, so resolving the .versions directory again re-emitted the key. * s3: test listing with keys between a null object and its .versions sibling * s3: pick the newer of the null object and the scanned versions Making the null object win outright whenever the pointer is absent misread multi-filer pointer lag: version files replicate ahead of the pointer, and a key overwritten or delete-marked after pre-versioning days would list its stale null again. The suspended-versioning write that legitimately makes the null current is also the newer entry, so mtime tells the two apart. * s3: a delete-marked null object no longer keeps its prefix alive The hidden-entries probe took any plain file as proof of a listable key, but a null object shadowed by its .versions sibling's delete marker is not one. Hold plain files pending until the sibling settles them either way. * s3: settle an evicted pending null instead of dropping it Nested keys like k, k!, k!! can hold more pending nulls than the cap. A silently evicted one could close the page unsettled, and the resume skip would then keep the stale entry for good. * s3: test deleted-prefix hiding and the pending-null cap * s3: cover the reported '!' intervening key with a live version * s3: an unstamped same-second version outranks the null object Second-resolution mtimes cannot order same-second writes, so the tie went to the stale null when the pointer lagged. The suspended write that makes a null current stamps the version it displaces before clearing the pointer, so the stamp is the authoritative signal and a tie without it goes to the version. * s3: a pointer-less versions listing still checks what replicated ListObjectVersions took a missing pointer as proof the null object is latest, but under pointer lag the sibling can already hold newer replicated versions or markers. Apply the same nullObjectWins rule as the listing recovery. * s3: a failed null-object settlement fails the listing Every getEntry error read as a missing sibling, so a transient filer error at a page boundary committed the unsettled null and the next page skipped its sibling for good. Only a definitive not-found means the null is live; other failures are retained on eviction and fail the request at page close. * s3: retract a CommonPrefix whose only backers were delete-marked nulls The directory probe settles this for the / delimiter, but any other delimiter derives prefixes from base-path keys directly, and a prefix built solely from null objects survived their delete markers. Count the unsettled null backers behind the newest prefix and retract it when the last one settles as a marker; a live resolution or any listable contributor confirms the prefix instead. * s3: test custom-delimiter prefix retraction * s3: an explicit signal marks the null object current, not the demotion stamp The NoncurrentSinceNs stamp survives promotion: delete the version that demoted another and the promoted one is current yet still stamped, so a lagging replica would resurrect the stale null. A suspended-versioning write now records Seaweed-X-Amz-Null-Version-Is-Latest on the .versions directory when it clears the pointer, every pointer update removes it, and the recovery paths trust the signal instead of the stamp. * s3: a filer failover retry rebuilds the listing page from scratch The failover wrapper reruns the callback on another filer after a transport error, and the partially built page, spent quota, and advanced marker leaked into the retry, which could then return a stale or duplicated page as success. * s3: only a prefix's own backers can debit it A delete marker for a version-only key (no base object) derived the same prefix as its neighbors and decremented backing it never contributed, retracting a prefix that a live null object still backed. Track backers by key so settlement is idempotent and only debits what was counted. * s3: test a version-only marker against a null-backed prefix * s3: a pointer recompute clears the null-current signal The routed finalize for delete markers, COPY, and multipart rewrites the .versions pointer through RECOMPUTE_LATEST, which left a suspended-era null-current signal in place. Version files never carry the signal, so mapping it in CopyExtended deletes it whenever the pointer recomputes. * s3: the pointer outranks the null-current signal in the versions listing The signal check guarded the pointer check, so a stale signal a recompute had not cleared yet would have let the null claim IsLatest alongside the pointed-at version. |
||
|
Chris Lu
|
c8cc56be91 | iceberg: route unprefixed requests to the first table bucket (#10675) | ||
|
Chris Lu
|
a2ff9cca27 |
master: let VolumeList ask for the volumes it wants (#10674)
* master: let VolumeList ask for the volumes it wants The request carried nothing, so every caller was answered with the whole cluster. A dashboard opening one volume's page, or a capacity probe adding up one bucket, was served all 800k of them and threw away the rest -- and the master built every one of those messages first. The topology, its disks and their counters are still reported in full: a caller reading free space or replica placement needs the cluster whichever volumes it asked about. Only what is listed under a disk is selected, ec shards included. An empty collection and a zero volume id take everything, the way volume.list already reads its own -collectionPattern and -volumeId, so a caller that forgets to narrow is answered too much rather than answered wrongly. That leaves the default collection unnameable, since it is the one the empty string names, so it gets a field of its own. An older client sends none of it and is answered exactly as before. * admin: ask the master for the volume the page is showing A volume's detail page was pulling every volume in the cluster to find one and its replicas, and discarding the rest. * admin: ask the master for the ec volume the page is showing Same as the volume detail page: one volume's shards were found by pulling every ec shard in the cluster. * s3: ask the master for the bucket's own collection The SOSAPI capacity probe summed one collection's volumes out of a listing of every volume in the cluster. Cluster capacity still comes out the same: it is read from the disk counters, which a filtered listing reports in full. * topology: read the disk usage counters atomically They are written with atomic.AddInt64 from heartbeats but were read plainly by the two listings and by FreeSpace, and the map they sit in was iterated without the lock its neighbour takes. Under -race a listing concurrent with a heartbeat trips on both. |
||
|
Chris Lu
|
567052bfb6 |
s3: take bucket sizes from the master's summary (#10664)
* pb: ask the master what each collection holds Callers tracking usage were sent every volume in the cluster to add up themselves, which is the master's largest single allocation. * topology: summarise what each collection holds One pass over the topology, allocating per collection rather than per volume. Regular volumes count once each for logical totals and once per replica for physical, taken from the lookup index, which is already keyed by volume and so needs no set of seen ids. Ec shards are node-local so their sizes sum, while the file and delete counts describe the volume and resolve once every holder has been seen. Replicas of one volume disagree while a write is landing or a heartbeat is late. Walking a full listing took whichever replica the map iteration reached first, so the answer moved between runs; this takes the largest, which is stable and never reports usage below what some replica already holds. * s3: take bucket sizes from the master's summary The bucket size metrics pulled the whole volume list once a minute and added it up, which cost the master 184.6MB of allocation and 17.8MB on the wire for six numbers per collection. VolumeList over 550k volumes 184.6 MB allocated, 17.8 MB on the wire CollectionStatistics 176 bytes allocated, 47 bytes on the wire The aggregation moves to the master with it, so the cases the removed tests covered are now asserted against it directly. * topology: count the replica holding the most live data Quotas are enforced on size less deletions, and the replica with the biggest raw size can be the one that has deleted the most. Counting it reported a bucket smaller than it is and would leave one writable over its quota, which is the opposite of what picking the largest was meant to guarantee. * topology: cap a volume's deletions at what it holds Live usage is read as a collection's size less its deletions, so a volume reporting more deleted bytes than it has cancels live bytes belonging to other volumes in the same bucket and reports it smaller than it is. Replica selection already floored that volume's own live size at zero; the totals have to agree with it. |
||
|
Chris Lu
|
923d0bd20c |
iceberg: repair non-compliant manifests at commit (#10641)
* iceberg: stamp a default name mapping on new tables * iceberg: repair non-compliant manifests at commit * s3tables: verify ClickHouse writes read back through PyIceberg * iceberg: carry the manifest-list content into repaired manifests * iceberg: refresh the default name mapping on schema evolution * iceberg: merge historical names into the refreshed name mapping * iceberg: never fail a commit on repair fallout * iceberg: harden manifest repair against writer dialects * s3tables: keep PyIceberg reader stderr out of row data * iceberg: keep name mappings unambiguous across field id reassignment * iceberg: align existing manifest content metadata with the list entry |
||
|
Chris Lu
|
2d9ea0285c |
s3: add the RenameObject endpoint (#10659)
* s3: add the RenameObject endpoint
PUT /{bucket}/{key}?renameObject with x-amz-rename-source moves an object
through the filer's AtomicRenameEntry, so no bytes are read or rewritten and
the ETag, tags and SSE keys travel with the entry.
Only unversioned buckets: a versioned rename would have to rebuild the
.versions chain, and AWS offers RenameObject on directory buckets, which
cannot be versioned. The source arrives in a header, so it is authorized
separately for read and delete; both keys are locked, in key order, across the
precondition checks and the move.
* s3: let a matched source ETag precondition settle its date precondition
RFC 7232 has an ETag precondition outrank the date precondition on its own
side, and AWS documents the same for CopyObject: a matching
x-amz-copy-source-if-match with a failing x-amz-copy-source-if-unmodified-since
copies rather than returning 412. The source check evaluated all four headers in
sequence, so the date header could still veto a decided ETag match.
validateConditionalHeadersForReads already applies this precedence; the source
path now matches it.
* s3: cover a rename source named as a directory without a trailing slash
Renaming a directory would move a whole subtree, so it has to stay a missing
key whether or not the caller wrote the trailing slash.
* s3: accept a bare object key as the RenameObject source
AWS spells x-amz-rename-source both ways. Its CLI, Java and Rust examples pass
the bare source key, and only a second CLI example and the boto3 conditional
example pass bucket/key; the API reference's own example is a bare key too. The
header was read as bucket/key only, so the form AWS leads with was rejected with
InvalidArgument and the endpoint was unusable as documented.
A value is now read as a literal key first — the only reading that can never
name the wrong object — and as bucket-qualified second, when it carries the
request's own bucket and the literal key does not exist. That costs one extra
lookup only for a source that starts with the bucket's own name.
Another bucket's name in the source is no longer a distinct error: RenameObject
moves within one bucket, so it is simply part of a key this bucket does not
hold, and it reports NoSuchKey.
* s3: only a proven absence picks the other reading of a rename source
A source that resolves to a directory is not a miss to fall through on: the
literal path is still what the caller named, so answering for it beats renaming
a different object under the bucket-qualified reading. With a directory at
bucket/source.txt and an object at source.txt, a rename naming the former moved
the latter.
A failed lookup is not a proof of absence either, so a blip can no longer
redirect a rename to the other reading.
|
||
|
hsdfatandChris Lu
|
0dd33bd7ec |
s3api: fix ListObjectsV2 dropping objects under a partial prefix (#10656)
* s3api: fix ListObjectsV2 dropping objects under a partial prefix
ListObjectsV2 with a prefix that names only part of a directory name -
"data/a" matching both "data/a/..." and "data/ab/..." - dropped objects
when paginating with a small max-keys, reporting IsTruncated=false before
all matching keys had been returned.
On resume, normalizePrefixMarker took the listing directory from the
continuation marker ("data/a/") and discarded the "a" name prefix, so once
the marker's subtree was exhausted the sibling directories that also
matched the prefix were never listed.
Resolve the listing directory and name prefix from the prefix instead of
the marker, so siblings sharing the name prefix are still listed after the
marker's subtree.
Fixes #10652
* s3api: cover a sibling-directory marker in the partial prefix tests
* s3api: resume a truncated delimited listing from the emitted CommonPrefix
Rebuilding the continuation token from the request dir plus the listing prefix
assumes that prefix names a directory. With a partial name prefix like "data/a"
it produced "data/a/ab/" for the sibling prefix "data/ab/", so the next page
resumed in the wrong subtree and repeated it forever.
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
|
||
|
Lars Lehtonen
|
2ff3dda7cd | chore(weed/s3api/policy_engine): prune dead code (#10599) | ||
|
|
d1f503181b |
[s3] force filer apply s3 expiry metadata (#10469)
* fix: apply S3 Expiry Metadata * add test Header X-Seaweedfs-Expires-S3 * resolve comments * test entry lookup by mtime * filer: skip s3 expiry stamp on versioned entries The s3 expiry path skips entries carrying a version id, so stamping one takes away its expiry rather than moving it onto mtime. Files under .versions/ are written once, so crtime already tracks their needles. --------- Co-authored-by: Konstantin Lebedev <whitefox@mayflower.work> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
a5e8254ffd |
s3: give a versioned metadata-only copy its own chunks (#10594)
* s3: give a versioned metadata-only copy its own chunks A self-copy that only rewrites metadata clones the source entry, chunk fids and all, and writes the clone back. With no versioning that is exactly right: the clone replaces the entry it came from, so one entry owns the needles the whole time. Under versioning the clone lands in a new .versions/ file and the source stays live, and nothing refcounts a plain shared chunk list -- deleting either version (a NoncurrentVersionExpiration rule, say) frees needles the other still points at, and the next vacuum makes that permanent. rclone hits this on every upload, since it stamps mtime with exactly this copy. Take the metadata-only path only where the write replaces the entry it read: the bare key of a bucket without versioning. Versioned, suspended, and versionId-pinned copies fall through to the regular copy path, which gives the destination its own chunks. * s3: reencrypt a versioned SSE-KMS key rotation instead of reusing the chunks A same-object copy that changes the KMS key id hands the source chunks straight back, on the assumption that the copy overwrites the entry they came from. A versioned bucket writes a new version beside the source instead, so the two end up sharing needles that nothing refcounts, and deleting either one frees the other's data. Reuse the chunks only when the destination really is the source entry; otherwise fall through to the reencrypt path, which also gives the new version the key it asked for rather than leaving it on the old one. * s3: make one predicate decide whether a copy replaces its source The metadata-only branch and the key-rotation strategy both answer the same question -- does this copy write back to the entry it read -- so let them share one predicate instead of pairing a same-destination check with it separately at each site. * test(s3): fail the copy regression tests when the vacuum does not run The helper swallowed a failed or non-200 request to the master, so a vacuum that never ran turned both chunk-ownership assertions into no-ops: the tombstoned needles were still readable and the surviving version looked fine either way. Require the endpoint, the request, and a 200. * ci(s3): run every versioning test in the regression gate The gate named the tests it wanted, so a new regression test sat there uncovered until someone remembered this file -- it fooled me into thinking two tests added in this PR never ran anywhere, when the comprehensive job had them all along. Invert it: run everything, and name a test only to keep it out. The delete job beside this one already works that way, and the suite costs about two minutes. Only the pagination stress tests are excluded; they build 1500+ versions, skip themselves without ENABLE_STRESS_TESTS, and have their own make target. Go's regexp has no negation, so the pattern is still assembled from a listing, the way the volume-server integration workflow does it. Note the trailing $$: make eats a lone trailing $ and takes the anchor with it. |
||
|
Chris Lu
|
e8020910db |
iam: authorize IAM management actions as IAM actions (#10593)
* s3: keep a non-S3 action out of the request-shape resolver
ResolveS3Action reads the request shape before it looks at the base action, so
an iam: or sts: action on a request that happens to carry an S3 query parameter
came back as the S3 action for that parameter. An action that already names its
service is resolved; there is no S3 request shape to read for it.
* iam: authorize the standalone IAM server's actions as IAM, not as S3
The standalone `weed iam` server wrapped its single POST / route in the generic
S3 Auth middleware with ACTION_ADMIN. The route has no {bucket}, so the check
ran with an empty bucket and resolved to a coarse S3 action rather than the IAM
one. The embedded IAM surface checks iam:<Action>; the standalone one was never
updated to match.
Both now go through one authorization function, so they cannot drift apart
again. It also rejects the anonymous identity, which has no user of its own to
run a self-service action against, and reads UserName from the body only, where
the handlers read it from.
|
||
|
Chris Lu
|
c2b47967bd |
s3: retire the suspended null marker only once the PUT has committed (#10589)
The suspended PUT dropped the null delete marker before writing, so a failed write left the .versions pointer naming a marker that was gone. The read path heals a dangling pointer by promoting the newest survivor, so a key the caller had deleted came back serving an older version, and the heal persisted that pointer. Move the retire into afterCreate via the shared finalize, which also brings the ownership check the copy and multipart paths already have. |
||
|
Chris Lu
|
f09bc14165 |
s3: report the effective ownership when a bucket has none stored (#10591)
* s3: report the effective ownership when a bucket has none stored GetBucketOwnershipControls read Seaweed-X-Amz-Ownership straight out of the bucket entry, so a bucket that never had one written reported an empty ObjectOwnership. The object write path defaults the same missing attribute to BucketOwnerEnforced, so the API contradicted the behavior it describes. Resolve the stored value through one helper both readers share, and let PutBucketOwnershipControls persist unconditionally so setting the default value still gives DeleteBucketOwnershipControls something to remove. * test: cover the bucket ownership controls round trip Pins the behaviors the ownership default fix depends on: a bucket that never had ownership controls written reports BucketOwnerEnforced, and putting that same value on such a bucket still persists it, so the delete that follows has something to remove. The put-then-delete case gets its own bucket -- run after an ObjectWriter put, it would pass against an implementation that skips only the initial write. The acl workflow already runs this package against a live weed mini, so it needs no wiring. |
||
|
Chris Lu
|
5269d93fa8 |
s3: let a suspended-versioning multipart completion replace the null delete marker (#10585)
* s3: let a suspended-versioning multipart completion replace the null delete marker In a versioning-suspended bucket a DELETE writes a null delete marker into the key's .versions directory. CompleteMultipartUpload then writes the new null version at the regular path but left that marker in place, so the completion returned 200 and the object listed while HEAD and GET kept resolving the marker and answered NoSuchKey. PutObject already handles this; do the same on the multipart path. * s3: order the suspended-versioning null cleanup behind the multipart write Removing the null delete marker before writing left a failed completion having already published the key's newest real version: the marker was gone, the pointer still named it, so reads rescanned .versions and promoted the older version. Do both fixups only once the write commits, pointer first so reads never see a pointer aimed at a marker that is no longer there, and fail the completion when the pointer cannot be cleared instead of returning 200 for an object HEAD and GET still miss - a non-ErrNone finalize keeps the upload directory, so the caller's retry replays it. Also cover a pre-suspension real version in the regression test. * s3: skip the suspended null cleanup when a concurrent write won the key The completion's .versions fixups are unconditional rewrites of shared state and the routed path runs off the object write lock, so a DELETE landing between the multipart write and the cleanup had its own null delete marker erased - leaving a successfully deleted key readable as an older retained version. Re-read the object first and leave the cleanup alone unless it is still the one we wrote. This narrows the window rather than closing it; a compare-and-set pointer flip is the real answer and wants its own change. * s3: re-read the completed object from the filer that took the write The guard compared the object against our upload id through the routed read, which skips an owner it recently found unreachable and falls back local-first. A write that just landed on the owner could then read as superseded on another filer, skipping the cleanup and leaving the key unreadable - the bug this set out to fix. Read back from the filer the write went to instead. * s3: trim the suspended-completion comments to the non-obvious why * s3: lift the suspended null-write finalize into a named helper The pointer-then-marker ordering is policy shared by every suspended null write, not something the multipart path should be stating on its own; putSuspendedVersioningObject and the copy path each restate it today. Give it a home next to the versioned finalize helpers, and reuse the canonical key normalizer and the existing test helpers rather than open-coding both. * s3: retire the null delete marker on a suspended-versioning copy The suspended CopyObject branch cleared the .versions latest pointer but left the null delete marker a preceding DELETE wrote. While the regular-path object owns the null slot that marker is shadowed, so it reads and lists correctly - but it resurfaces as a phantom delete for a key nobody deleted once that null version goes away. Route the branch through the shared finalize. * s3: keep the suspended null cleanup from erasing a concurrent delete Retiring the marker on the copy path reopened the race the multipart path had already closed: a DELETE landing between the write and the cleanup lost its own marker, so a rescan promoted an older version under a deleted key. Move the ownership check into the shared finalize, keyed on the attribute that identifies the caller's write, so both paths get it. |
||
|
Chris Lu
|
7063b3e14c |
s3 lifecycle: bound the daily-replay pass so a quiet cluster stops wedging the job (#10578)
* s3 lifecycle: bound the daily-replay subscription at the pass boundary A pass opens one meta-log subscription and 16 shard drains, then waits on all of them. Nothing told the subscription where the pass ends, so the only exit was the fan-out spotting an event past runNow — i.e. some unrelated write landing under /buckets after the pass started. On a cluster that goes quiet the reader parks in Recv, every shard drain starves on an empty channel, and Run never returns. The job sits at stage "starting" with the executor slot held and no log line, so expiry stops cluster-wide until someone restarts the worker. The pass covers (globalStartTsNs, runNow], so say that: UntilNs on the subscribe request makes the filer end the stream once it has shipped that range. The reader then closes the event channel on the way out, which is what unblocks the fan-out and the drains when the stream finishes on its own rather than by cancellation. Same fix retires the other silent hang: a reader that failed early (subscribe error, stream error) also left every drain waiting forever. * s3 lifecycle: keep a halted shard from starving the shared fan-out A drain that halts mid-stream (BLOCKED / RETRY_LATER / an RPC error on dispatch) returns while the fan-out is still routing that shard's events. After 256 of them the per-shard buffer is full and the fan-out blocks on the send, so no other shard sees another event. Run's WaitGroup never drains, and the teardown that would cancel the reader sits behind that wait — the pass wedges exactly like an idle subscription did, with one S3 hiccup as the trigger. Keep discarding the channel after runShard returns. The events are past this shard's saved cursor and get re-scanned next pass anyway. * s3 lifecycle: assert the starved shard actually made progress The fan-out test only checked that Run returned, which a version that quietly dropped the second shard's events would also satisfy. Assert the dispatch landed and the cursor moved. recordingClient gains a per-object outcome map: the two shards dispatch from separate goroutines, so pinning BLOCKED by call index was a race waiting to pick the wrong shard. * s3 lifecycle: fail the pass when the shared subscription dies Closing the event channel on reader exit is what unblocks the shard drains, but it also means a subscribe that never opened, or a stream that broke mid-pass, now ends every drain cleanly. Run logged that at V(2) and returned the shard result — so a filer failure produced a green lifecycle job that had processed nothing. Surface it as the pass error. Cursors still hold what was processed and tomorrow resumes there; what changes is that the job stops claiming success. Cancellation has to stay a non-error — the shell driver's -runtime cap is a truncated pass, not a failed one — and a canceled gRPC stream arrives as a status code, not a wrapped context.Canceled, so isCanceled checks both forms the way the rest of the tree does. * s3 lifecycle: decide reader cancellation by intent, not status code A stream we cancel and a stream the filer cancels both arrive as codes.Canceled, so classifying the reader's exit by its error let a truncated pass report success whenever the failure happened to carry a cancellation status. Intent is knowable exactly, so read that instead: the pass stops on purpose only when the caller's context ended (the shell driver's -runtime cap) or the fan-out hit the pass boundary itself. Everything else is a broken subscription and fails the pass. TestRun_ServerSideCancelFailsThePass and TestRun_CappedPassIsNotAFailure are the same codes.Canceled from the reader with opposite verdicts — the pair only passes because the decision no longer looks at the error. * s3 lifecycle: time out a subscription that stops delivering UntilNs ends a healthy stream and gRPC keepalive catches a dead connection, but neither reaches a filer that keeps answering pings while its handler has stopped producing. The pass would wait on that forever, since s3_lifecycle is the one job type with no execution timeout. Bound the wait for each response at 20 minutes, and opt into the filer's idle heartbeats so a caught-up stream proves liveness instead of looking stalled. The default sits above the filer's 15-minute metadata-gap recovery budget, so a subscriber legitimately parked on a gap is never mistaken for a stalled one. Recv is only interruptible by killing the RPC, so it moves to its own goroutine behind a per-response deadline. The timer covers only the wait on the filer — dispatch to Events happens outside it, so a slow consumer can't trip the watchdog. Approach and the 20-minute figure are from #10577 by way of comparing the two fixes; the wiring differs because the reader here ends the pass by closing its event channel rather than cancelling the fan-out. * s3 lifecycle: trim the comments added by this branch Keep the non-obvious why, drop the prose restating what the code says. * s3 lifecycle: snapshot reader intent where the reader stops Sampling ctx.Err() during teardown reads it after the drains and cursor saves have run. A reader that failed while the deadline was still live, on a pass whose teardown then outlives that deadline, was classified as an intentional stop and reported success. Sampling earlier in Run is not the fix either: before the shard wait, a legitimately capped pass has not reached its deadline yet and would be misclassified the other way. Intent belongs where the reader actually stops, so the reader goroutine records it next to the error it returns. Reported by greptile on #10578. * s3 lifecycle: cover the worker-dispatched pass with nothing due The e2e suite drives the shell command in 14 of 15 files; the one test on the real admin->worker path backdates an object, so its own delete pushes a meta-log event past the pass boundary and ends the pass. The branch where a pass has nothing to dispatch was never exercised through the worker. Cover it, asserting the pass returns on its own: no admin cancellation, and the executor slot free for the next one. This is not a regression test for the wedge. A pass used to end when any write landed past its boundary, and on a shared test cluster something usually does — the whole suite passes on the unfixed build, verified. The deterministic guards stay the dailyrun unit tests; this one would catch a pass that hangs unconditionally. |
||
|
Chris Lu
|
7d6c83b126 |
s3: stop treating a directory marker as a versioned object (#10573)
* s3: delete a directory marker instead of versioning it The key "dir/" is stored as the filer directory itself, so a delete marker cannot stand in for it without hiding the children underneath, and its history has to sit inside the directory it describes, where listings keep meeting it. Delete it the way an unversioned bucket already does: remove the directory when nothing is left under it, demote it to a plain directory when children remain, and drop a history an older build recorded for it. * s3: stop resolving directory markers through a version history Nothing records one for them any more, so the lookups that read it are dead weight - and the one in the listing was a filer round trip per directory marker returned, which for a bucket that keeps a marker per directory is the whole listing cost. A listing reads what a directory stands for straight off the entry it already has; a unit test pins that N markers cost one ListEntries rather than N+1. The guard that keeps a history left inside a directory by an older build from surfacing as a key named after it stays. * s3: do not let deleting "dir/" destroy the object at "dir" Writing under an existing object turns that object's entry into a directory while it keeps its data, so the keys "m2" and "m2/" end up sharing one entry. Stripping the entry to delete "m2/" therefore wiped the object at "m2" - a different key, and in a versioned bucket one no delete marker records. Leave a directory holding uploaded data alone; "m2/" does not name it. * s3: make the directory-marker delete fail closed and take the write lock The guard that spares a promoted file only fired when the entry read succeeded, so a transient filer error fell through to the delete and could destroy the object at "dir" anyway. Fail the request instead, take the object write lock so the entry cannot change between the check and the delete, and report a stale history that cannot be removed rather than leaving it to keep naming the key in ListObjectVersions. * s3: check If-Match inside the directory-marker delete lock The lock belongs to the caller: taking it inside the delete nested it under the batch handler's own lock, and since every lock from a gateway shares one owner the inner release would have freed it while the outer caller still assumed it held it. Both callers now own the lock, the single-object path re-checks If-Match inside it the way the other delete paths do, and a batch delete of a trailing-slash key in an unversioned bucket goes through the same marker path instead of the raw delete. A history lookup that fails now fails the delete. |
||
|
Chris Lu
|
d448e9db7b |
iceberg: withhold the S3 endpoint from credential-vending clients (#10570)
* iceberg: withhold the S3 endpoint from credential-vending clients A client that sends X-Iceberg-Access-Delegation: vended-credentials builds its storage credential out of the LoadTable config and drops the one it was configured with. We vend no credentials, so the endpoint we advertised left DuckDB signing nothing: every metadata and data file came back 403, and its attempt to refresh the empty credential 404ed on stage-created tables. Answer those clients with no config at all so they keep their own credentials. Clients that do not ask for delegation still get the endpoint. * iceberg: mark load responses as varying on the delegation header The FileIO config in a table or view load response now depends on whether the client asked for vended credentials, so a cache between us and the client must key on that header rather than on the URL alone. * test: cover the DuckDB vended-credentials access pattern Runs weed mini with -s3.externalUrl, which is what makes the catalog advertise an endpoint at all, and checks both halves: a plain LoadTable still gets the endpoint, while one asking for vended credentials never gets an endpoint without the credentials to sign with. The DuckDB round trip creates a table from a query and reads it back, which is the flow that failed with 403 on every data file. |
||
|
Chris Lu
|
474a0713b0 |
s3: honor the version history of a directory marker (#10571)
* s3: stop listing a directory marker whose latest version is a delete marker A directory marker is stored as the filer directory itself, so deleting the key "dir/" writes its delete marker into dir/.versions while the directory keeps its mime and stays a key object. Every listing kept reporting the key. Consult that history before treating the entry as a key, and demote it in memory when it is delete-marked so live children still hold the prefix. Also skip the container's own .versions entry while listing inside it: the suffix match read it as the history of a nested object named "", which surfaces as a phantom dir/dir key as soon as a live directory version exists. * s3: a directory marker with version history is not also the latest null version The directory entry behind the key "dir/" is that key's null version, but list-object-versions reported it with IsLatest hardcoded true. After a delete the key came back twice, once as the delete marker and once as a null version, both claiming to be latest. Read the pointer under the directory instead. * s3: resolve directory markers through their version history on GET and HEAD GET and HEAD short-circuit any trailing-slash key straight to the filer directory, so a directory marker kept answering 200 after its delete marker was written. Resolve the key from dir/.versions first when the bucket is versioned: a delete-marked current version answers 404 with x-amz-delete-marker, a named delete-marker version answers 405, and a key with no history keeps today's directory-probe behavior untouched. * s3: re-creating a directory marker retires its delete marker PutObject on a trailing-slash key never looked at the bucket's versioning state, so re-creating a marker after a delete left the latest-version pointer on the delete marker and the key stayed invisible to every versioned read. Point the key back at the directory entry, which is its null version, and drop the null version .versions may still hold — the same two steps a suspended write already takes, now shared. * s3: fail a directory-marker request whose version history cannot be read Every lookup of dir/.versions treated any error as "no history", so a filer hiccup served the directory entry for a key whose current version may be a delete marker, reported a null version as latest over one, and let a PUT report success without retiring the delete marker it was meant to retire. Only a confirmed absence takes the no-history path now. * s3: cancel the directory-marker probe stream instead of abandoning it The probe answers off the first entry and returns, leaving the ListEntries stream open for the life of the parent context. Give it a context of its own. |
||
|
Chris Lu
|
b452a5e41b |
s3: honor a bucket owner recorded as an identity (#10567)
* s3: resolve a bucket owner recorded as an identity The admin UI and weed shell record a bucket's owner as an identity name in s3-identity-id and never write the account id the S3 API stores alongside it, so such a bucket looked unowned: its ACL owner fell back to the default admin account, and under the default BucketOwnerEnforced ownership every object uploaded to it was stamped with that account instead of the bucket owner. Resolve the identity to its account when no account id is recorded, in the one place both the bucket metadata and the bucket config derive the owner from. * s3: drop the recorded account when the bucket owner is reassigned Changing the owner of a bucket created through the S3 API left its old account id behind, and that outranks the identity when the owner is resolved, so the new owner never took effect for object ownership or the bucket ACL. |
||
|
Chris Lu
|
c191b2fe01 |
iceberg: let clients select their table bucket as the catalog warehouse (#10549)
* iceberg: accept bare bucket names and ARNs as the catalog warehouse Only s3://<bucket>/ was recognized. A warehouse spelled as a bare table bucket name or as the s3tables bucket ARN -- the two forms users reach for first, the latter being what AWS S3 Tables itself takes -- was silently dropped, so every call landed on the default "warehouse" bucket and failed with "table bucket warehouse not found". * iceberg: report a missing table bucket as 404, not 500 Pointing a client at a table bucket that does not exist -- which every client with no warehouse set does, since the default bucket "warehouse" rarely exists -- returned InternalServerError with a message naming a bucket the client never asked for. Answer 404 and say how to select one. * admin: show the warehouse in the PyIceberg example The example connected without one, so it always resolved to the default table bucket and every client that copied it failed on the first call. * test: pin bearer auth against a table bucket that exists The subtest called the catalog with no warehouse and accepted 500 as proof that auth had passed, since the default bucket does not exist. A missing table bucket now answers 404, which the test read as an auth failure. Give it a real table bucket so only 200 passes. * test: assert the missing-bucket guidance reaches the client The status and error type were checked but not the message, which is the part of the mapping that tells a user how to select a table bucket. * test: encode the warehouse query value The ARN case pasted raw colons and slashes into the query string. Go's parser tolerates them, so the test passed without modelling how a client actually sends the request. |
||
|
Chris Lu
|
cc2775d9f2 |
s3: register an identity's inline account instead of collapsing it into admin (#10548)
* s3: register an identity's inline account instead of collapsing it into admin Credential stores persist an account inline on the identity and never emit a top-level accounts list, so every user created through the IAM API or the admin UI with an email hit the "non exist account ID" branch and was given the shared admin account. Distinct users then presented the same owner id, so ownership checks could not tell them apart and each passed for the others' buckets. Treat an id missing from the account map as undeclared rather than invalid: register it, keeping an email another account already claimed. Both load paths now resolve the account through one helper. * s3: refresh an undeclared account from the identity that carries it The merge path starts from the live account cache, so an identity upserted with the same account id but a new email or display name kept the cached copy: the new address never reached the email index and the replaced one still resolved. Changing a user's email through the admin UI takes exactly that path. An account registered from an inline block is only described by the identity carrying it, so refresh it and move its email claim. Accounts from a top-level list and the predefined defaults are marked declared and stay authoritative. * s3: let an account reclaim an email once its holder moves away Two identities can carry the same email, and the second to load leaves the lookup with the first. Returning early when the incoming metadata matches the cached account meant the loser never re-ran the claim, so an address freed by the holder's update resolved to nobody until the loser itself changed. Re-index on the unchanged path, which is a no-op while another account still holds the address. |
||
|
Chris Lu
|
c2701955c7 |
s3: cover three untested STS paths (#10521)
* s3: test the GetCallerIdentity handler The handler had no test, only XML marshalling, so nothing pinned that a caller presenting session credentials is reported as the assumed role rather than the user who minted the session. * s3: drive AssumeRoleWithWebIdentity over HTTP with a real OIDC token Coverage reached the OIDC path either at the IAMManager service layer or through the Authorization: Bearer shortcut. Nothing exercised the public STS entry point an AWS SDK actually calls, which is where parameter parsing, the IAMManager dispatch and the XML response shape live. * s3: test that every STS route emits an audit entry STS responses go out through WriteXMLResponse, which never calls PostLog, so track() is the only thing that logs them. A route registered outside it would mint credentials with no audit trail and nothing would notice. STS has three routing layers, so a new action is easy to attach to the wrong one. * s3: make the STS tests assert what they claim to cover The audit routing test ran against an uninitialized STS service, so every case answered 503 and a non-404 status was the only evidence the request had reached STS at all - the POST-body case could have been served by the dispatcher's IAM branch and still passed. Back it with a real STS service and assert the STS response namespace, which IAM and S3 responses do not carry. The session policy case checked that the policy travelled in the token rather than that it restricted anything; assert the narrowed bucket is allowed and another bucket is refused. Give the forged-token case the same claim set a valid token gets, so it cannot pass for want of a claim, and cover both rejection paths: a key we do not publish, and a key id absent from the JWKS. |
||
|
Chris Lu
|
1ce106e69d |
s3: audit the assumed-role principal and the STS caller (#10519)
* s3: log the requester's principal ARN in the audit entry An STS session authenticates as an opaque session subject, so requester alone gave an operator no way back to the assumed role or the session name. Record the principal ARN next to the identity name and emit it as requester_arn. * s3: record the caller identity in the STS handlers AssumeRole, GetFederationToken and GetCallerIdentity verify the caller themselves and are not wrapped by the auth middleware that records the identity, so every audit entry for minting a session had an empty requester. * s3: resolve the audit principal ARN the way policy evaluation does A JWT-authenticated identity carries no PrincipalArn — the auth layer hands the principal over in a request header — so reading the field directly left requester_arn empty for OIDC callers. buildPrincipalARN is the resolver the policy path already uses: header first, then the identity's own ARN, then a synthesized user ARN for legacy identities that have none. |
||
|
Chris Lu
|
fa432f9a6a |
s3: keep an admin's role session scoped to the role (#10520)
* s3: keep an admin's role session scoped to the role AssumeRole copied the caller's admin standing into the minted session as the is_admin claim, which short-circuits base policy evaluation. An admin assuming a scoped-down role therefore kept full access and the role's attached policies, explicit denies included, were never evaluated. Only a session the caller assumed for itself carries the claim now — a legacy static admin has no IAM policies for such a session to inherit. * s3: name the caller when it assumes a session for itself An identity that carries no principal ARN left the self-assumed session with an empty role name in its assumed-role ARN. callerPrincipalArn synthesizes the canonical user ARN for that case. |
||
|
Chris Lu
|
d8d29c4ede |
s3: carry storage class in the cached listing metadata (#10516)
A listing on a versioned bucket is served from metadata cached on the .versions directory entry so the whole listing is a single scan. The cache carried size, mtime, ETag, owner and the delete-marker flag but not the storage class, so newListEntry found none and fell back to STANDARD. The result was that HEAD and the listings disagreed about the same object: HEAD reported the class the object was stored with, while ListObjectsV2 and ListObjectVersions reported STANDARD for every object. Clients that filter or tier on storage class act on the listing. Caches the class alongside the other listing fields, clears it with them, and copies it in the routed RECOMPUTE_LATEST path so both finalize paths agree. |
||
|
Chris Lu
|
f582c8451b |
s3: report a peer that went away as ClientDisconnected, not IncompleteBody (#10511)
* s3: report a peer that went away as ClientDisconnected, not IncompleteBody A streaming PUT whose body ends early is always reported as IncompleteBody (400). That collapses two cases with opposite causes: the peer vanished mid-upload, and the peer sent fewer bytes than it promised while still connected. The first points at the network path, the second at the client, and once merged they cannot be told apart from the logs. Split out ClientDisconnected (499) and select it when the request context shows the peer is gone. The upload itself keeps running on a background context so chunks still finish, which means cancellation races the read error; a missed signal degrades to IncompleteBody exactly as before. * s3: note what request-context cancellation is taken to mean |
||
|
Chris Lu
|
63d5140485 |
s3: allow copying an object onto itself in a versioned bucket (#10497)
* s3: allow copying an object onto itself in a versioned bucket The copy writes a new version instead of overwriting in place, which is how an earlier version is restored. Buckets with versioning off or suspended keep rejecting a self-copy that changes nothing. * s3: cover the suspended-versioning self-copy rejection Suspended versioning overwrites the null version in place, so a self-copy that changes nothing stays rejected. Pin that alongside the never-versioned case. |
||
|
Chris Lu
|
e7a678fa72 |
s3: keep the list marker exclusive for versioned objects (#10496)
* s3: keep the list marker exclusive for versioned objects A versioned object lives in a "<key>.versions" directory, so the entry name never matched the marker and start-after/marker returned the marker key itself. * s3: match the list marker against the raw entry name too A backend that echoes the marker it was given returns the ".versions" directory name, which no longer matched once the comparison used the object name alone. Cover both, and unit test each half. |
||
|
Chris Lu
|
4149346bb7 |
s3: register the advertised ip with the master (#10482)
* s3: register the advertised ip with the master The cluster address came from the bind ip, falling back to the auto-detected interface, so -ip never reached the S3 registration. weed mini -ip=localhost binds the wildcard and ended up registering whatever interface happened to sort first -- on a host with VPN interfaces, an address that stops routing once the tunnel drops. IAM changes are pushed to registered S3 servers over gRPC, so every mutation then blocked the full 10s propagation deadline before logging a failure, and cluster.ps and the admin UI listed a node nothing could reach. Identities still arrived through the /etc/iam metadata subscription, so this cost latency and visibility, not credentials. Add an advertise ip to the gateway option, preferring it over the bind address, and wire the parent -ip through server, filer and mini. * s3: treat any unspecified bind address as a wildcard net.ParseIP + IsUnspecified covers ::, [::] and the expanded IPv6 forms instead of only the 0.0.0.0 literal, so an IPv6 wildcard bind no longer registers an address peers cannot dial. Host names parse as nil and stay addresses in their own right. Apply the same guard to the advertised ip. |
||
|
Chris Lu
|
0002e5cc7f |
s3api: load document-style policies from the advanced IAM config (#10481)
* s3api: load document-style policies from the advanced IAM config The advanced IAM file doubles as the S3 identity config when only -s3.iam.config is given. protojson drops its "document" field, so every policy landed with empty content and warned "skipping invalid policy" on each reload. Worse, if the same file also declares identities the empty content sticks in the policy map and fails the whole runtime policy sync into the IAM manager, so policies created later never reach it. * iam: skip an unparsable policy instead of failing the whole runtime sync One policy the engine cannot parse aborted SyncRuntimePolicies before it touched anything, so every other policy stayed unsynced and the engine kept serving whatever it last held. * s3api: reject a non-role RoleArn in AssumeRole as a bad request arn:aws:iam:::user/name can never resolve to a role, but the handler ran it through the trust-policy check and answered "not authorized to assume role", pointing the caller at a permission problem they do not have. * s3api: build the policy content before touching the entry Deleting "document" up front meant a marshal failure left the policy with neither field, so a later rewrite would emit it with no definition at all. * iam: pin the fail-closed handling of an unparsable policy Say in the comment that dropping it from the desired set deletes it from the engine on purpose, and cover it with a test. * s3api: widen the non-role RoleArn test to canonical ARN shapes The reported ARN omits the account id; a user ARN that carries one, and a non-principal ARN, must be rejected the same way. |
||
|
Chris Lu
|
4b0d09683a |
iceberg: read manifest lists that omit the Avro format version (#10475)
* s3tables: read Iceberg manifest lists that omit the Avro format version The Iceberg spec pins the Avro header metadata of manifest files but says nothing about manifest lists, so writers disagree. Java and PyIceberg record "format-version"; DuckDB writes no header metadata at all. iceberg-go reads a missing entry as v1, so every v2 manifest listed in a DuckDB-written list is rejected with manifest file's 'format-version' metadata indicates version 2, but entry from manifest list indicates version 1 and, because v1 has no "content" field, delete manifests silently decode as data manifests. ReadManifestList derives the version from the record schema the writer embedded - v2 added "content" and the sequence numbers, v3 added "first_row_id" - and splices it into the header before handing the bytes to iceberg-go. Lists that already carry the entry, and input that is not a parseable Avro container, go through untouched. * iceberg: parse DuckDB-written manifest lists in maintenance and data preview Every manifest list read - the four maintenance operations and the admin table data preview - went straight to iceberg-go, so tables written by DuckDB failed detection and all of compact, remove_orphans, rewrite_manifests and expire_snapshots before they touched anything. Route them through s3tables.ReadManifestList, which recovers the format version the writer left out of the Avro header. This also restores the manifest content type on those tables: with the list read as v1 every delete manifest looked like a data manifest, which hid deletes from the compaction guard and made the preview report a table with position deletes as having none. |
||
|
Chris Lu
|
ac6f3c92ef |
s3api/iceberg: report the reason a table schema was rejected (#10473)
* s3api/iceberg: report the reason a table schema was rejected newTableMetadata swallowed the iceberg-go error and returned nil, so every schema the metadata builder refused came back as a bare 500 "Failed to build table metadata". A v3-only column type is the common case: creating a table with a variant field but no format-version 3 property leaves the client with nothing, while "variant is not supported until v3" sits in the server log. Return the error instead and classify it. Schema, spec and argument failures are the caller's input, so they answer 400 with the underlying reason; the rest stay 500. Paths that build placeholder metadata with no schema keep their existing 500 via newEmptyTableMetadata. * s3api/iceberg: fail LoadTable when placeholder metadata cannot be built buildLoadTableResult dropped a nil from the placeholder path straight into the response. That serializes as "metadata":null under HTTP 200, which no Iceberg client can parse -- a worse outcome than the 500 the nil was meant to signal. Return an error instead and let the five callers answer 500. The nil-return convention goes away with it, so the commit and transaction paths check an error rather than a sentinel. * s3api/iceberg: route rejected schemas through writeManagerError The two helpers added here duplicated work the package already does. writeManagerError is the canonical error-to-response mapper -- it already downgrades client-input failures to 400 and defaults the rest to 500 -- so teach it the iceberg-go schema and spec sentinels instead of standing up a parallel classifier. The placeholder wrapper was a pure alias for newTableMetadata with nil arguments; call that directly. No behavior change beyond the 500 message, which now reads err.Error() like every other manager error rather than carrying its own prefix. |
||
|
Lars Lehtonen
|
2bea4dd610 | chore(weed/s3api): prune dead code (#10462) | ||
|
Chris Lu
|
62c4333074 |
s3: list the buckets an attached IAM policy grants (#10458)
* s3: list the buckets an attached IAM policy grants ListBuckets served an identity authorized by an attached IAM policy only the buckets it had created itself. A user granted s3:ListBucket on a bucket someone else provisioned could GetObject and ListObjectsV2 against it, but the bucket never showed up in the listing any S3 client uses to build its bucket picker. The owner-index fast path is only valid for an identity whose grants name every bucket it can reach, and the routing check assumed a policy could never be enumerated. Read the names out of the policy instead: statements that allow s3:ListBucket on a concrete bucket ARN become candidates, and the per-bucket permission re-check still decides what is listed. A policy that can reach a bucket it does not name -- a wildcard resource, a policy variable, a NotResource, an STS session policy -- falls back to the full scan, which evaluates the policy per bucket. * s3: share the attached policy name lookup authorizeWithIAM and the ListBuckets enumeration both built an identity's policy names the same way, its own plus the ones from its enabled groups. Pull that into one helper so group eligibility is decided in a single place. * s3: read policy actions the way the IAM authorizer matches them The IAM authorizer matches action names case-insensitively, so a policy granting "S3:LISTBUCKET" or "S3:*" authorizes a list. The ListBuckets classifier read those actions with the local case-sensitive matcher and found no grant, so once the owner index was ready the buckets that policy allows dropped out of the listing. Match the action the looser way in the classifier: case-insensitive, and true for any pattern holding a policy variable. Over-matching only costs a candidate the per-bucket permission check then rejects, while under-matching hides a bucket the caller can read. * s3: infer a multipart grant in any case The classifier matches action patterns case-insensitively but looked the requested action up in a canonical-cased set, so "S3:UPLOADPART" missed the s3:PutObject inference that the authorizer makes. Key the set for lookup in lower case, matching how the IAM authorizer holds it. |
||
|
Chris Lu
|
6b6e6d8547 |
s3: apply filer identity changes despite a static config file (#10392)
* s3: apply filer identity changes despite a static config file A -config file with inline identities disabled the metadata-subscription reload entirely, leaving the best-effort filer->s3 push as the only way s3.configure changes could reach a running gateway. Reload on IAM events regardless: the merge keeps the file's identities protected, and a full credential-manager snapshot now also drops dynamic identities the store no longer has, so revocation works without a restart. * s3: log identity propagation failures as warnings * s3: retry failed IAM reloads and reconcile policies and groups An event-driven reload that fails now hands off to a coalescing retry loop, so a transient filer error cannot strand a revoked credential until the next IAM event. Full-state merges also drop dynamic policies the store no longer has, keeping the static file's, and treat the group snapshot as authoritative even when empty. * s3: serialize IAM configuration loads The SIGHUP file reload, subscription reloads, the retry loop, and the postgres poll run on different goroutines. Without an end-to-end lock a load holding an older store snapshot can commit after a newer one and revert it. Hold reloadMu from snapshot through commit in both load entry points; partial merges from pushed updates stay lock-free and self-heal through the next event-driven reload. * s3: keep static-file groups through full-state reconciliation Group names from the static config file are tracked like identities and policies, and a full snapshot that does not carry them keeps the current definition and its memberships instead of dropping them. * credential: include groups in postgres configuration snapshots Full-state reconciliation treats absent groups as deleted, so a snapshot that never carries them would erase every dynamic group. * s3: revoke static-file groups dropped from the config file A file reload is authoritative for the file's group set while keeping dynamic groups, mirroring how full snapshots are authoritative for dynamic groups while keeping the file's. * credential: fail filer snapshots on unreadable entries A skipped identity or policy file made the load report success with an incomplete snapshot, which reconciliation reads as deletion and the retry loop never sees. Unparseable content is still skipped: it is durable, matches boot behavior, and must not block reloads forever. * s3: ignore groups in static config files Groups are managed through the IAM API and the dynamic store; no deployment defines them in a bootstrap config file. Ignoring them with a warning removes the two-directional group merge: full snapshots are plainly authoritative and file reloads never touch groups. |
||
|
Chris Lu
|
a7f4b88a61 |
s3: require a bucket-policy action to write a bucket policy (#10444)
* s3: require a bucket-policy action to write a bucket policy PutBucketPolicy and DeleteBucketPolicy were gated on ACTION_WRITE, the same action that grants object writes. An explicit Allow in a bucket policy short-circuits IAM entirely -- authRequestWithAuthType sets policyAllows and skips VerifyActionPermission -- so anyone who could write an object could author a policy granting itself, or anonymous, anything on the bucket. That is what separates a bucket policy from the sibling bucket controls also gated on ACTION_WRITE: rewriting cors or lifecycle can destroy data, but only a policy hands out access. Give the two verbs their own actions, mapped to the AWS names that were already defined but unrouted. ACTION_ADMIN would also have closed it, but it resolves to s3:* for IAM identities, forcing a blanket grant on a user holding a precise s3:PutBucketPolicy. Admins are unaffected, since isAdmin short-circuits CanDo, and an operator can delegate with PutBucketPolicy:bucket. The route binding is asserted from the router source: checking the action constants alone still passes when the route says ACTION_WRITE. * s3: also read the action from a direct iam.Auth call in the route test Routes read iam.Auth(cb.Limit(handler, ACTION)), a multi-value pass-through: Limit returns (http.HandlerFunc, Action) and those become Auth's parameters, so the action Auth authorizes on is Limit's second argument and the two cannot disagree -- Auth(Limit(h, X), Y) does not compile. A route that skipped Limit and called Auth with its own action would compile, though, and the test reported that as a missing route rather than as the wrong action. Recognise the two-argument Auth form so it names the action instead. * s3: make the bucket-policy actions grantable through an IAM policy The new actions close the escalation only if an operator can grant them, and they were not reachable: MapToStatementAction had no entry for PutBucketPolicy, so an IAM policy naming s3:PutBucketPolicy was rejected outright with "not a valid action". GetBucketPolicy was unmapped the same way. DeleteBucketPolicy was mapped, but to ACTION_ADMIN -- granting an identity permission to delete a bucket policy handed it full administrative access. Map all three to the actions the router now uses, and add the reverse direction so an identity holding them renders back as a policy statement instead of a bare "s3:". * admin: offer the bucket-policy permissions in the user editor The two new actions are otherwise only grantable by hand-editing identity JSON or by calling the IAM API, so an operator using the UI cannot delegate bucket policy management without granting Admin. Regenerating this file also picks up codegen the repo has not taken yet: the checked-in _templ.go files were produced by templ v0.3.1001 while go.mod pins v0.3.1020, so the generator rewrites the attribute-value calls. That churn is confined to this one file; running `make generate` in weed/admin reproduces it across all 36. |
||
|
Chris Lu
|
6824619c16 |
s3: chunk uploads at the filer's maxMB (#10439)
The S3 write path cut fixed 8MB chunks, so an object stored through S3 chunked differently from the same bytes stored through the filer, WebDAV or a mount, and -maxMB had no effect on it. Read maxMB from the filer configuration at startup and use it, falling back to 8MB when the filer reports none. |
||
|
Chris Lu
|
c392f45705 |
s3: stop listing prefixes whose objects are all delete-marked (#10419)
Deleting the only object under a prefix in a versioned bucket writes a delete marker and keeps the version history, so the filer directory survives with nothing a current-version listing would return. A delimited ListObjects kept reporting that path in CommonPrefixes, because the prefixes come from the directory tree rather than from the keys, while a listing scoped inside the prefix correctly came back empty. Probe a directory before reporting it: one that holds entries but no key the listing returns is neither a CommonPrefix nor a path the trailing-slash probe answers for. Empty directories keep the meaning they have today, and the probe only runs for buckets with versioning configured, the only ones that can reach this state. |
||
|
Chris Lu
|
b4b0346f95 |
iceberg maintenance: resolve table files from the recorded location (#10418)
The worker assumed every file of a table sits under its catalog path, so loadFileByIcebergPath stripped the scheme off a recorded location and joined the remainder onto /buckets/<bucket>/<ns>/<table>. A table the REST catalog placed elsewhere in the bucket — which is what a client gets whenever the catalog path is already occupied — then resolves to a doubled path: lookup /buckets/lake/source/t/lake/source/t-0cd81bca-.../metadata/snap-.avro so the very first manifest list read fails and the job fails again on every scan interval, indefinitely. Resolve absolute references (s3:// URIs and /buckets paths) from the bucket root and keep relative ones under the table's own directory; the bucket-relative form is now the canonical key everywhere references are compared. That directory comes from the metadata location the catalog stores, so reads, writes and deletes all land where the table's other files are instead of splitting it across two trees. References outside the table's bucket are rejected rather than silently misresolved. Rewritten position-delete files now name their data file by absolute URI, the way the table itself names it, instead of a path relative to the table. |