1348 Commits

Include renames

Author	SHA1	Message	Date
Chris Lu	69cc2869ad	Fixes from the review of the admin bucket policy UI (#10907) ... * admin: treat a missing S3 Tables policy as an empty load, not an error The bucket/table policy GET relayed the backend's 404 NoSuchPolicy to the dialog, whose loader treats any non-OK response as a load failure and keeps Save and Delete blocked. A bucket or table without a policy could never be given one. Return policy null instead, the same contract ShowBucketPolicy uses for classic buckets. * admin: reject policy documents the structured editor would misread A top-level JSON array passed the object guard (typeof [] is 'object') and loaded as a zero-statement policy, which the next commit would rewrite to an empty document. Object elements in Action/Resource were coerced to '[object Object]' and saved that way on the s3tables surface, which stores policies verbatim. Both now throw, which routes the document to the JSON tab like other unrepresentable shapes. * admin: let the JSON tab save documents the structured editor can't model Save with the JSON tab active required a round-trip through policyDocToEditorState, so exactly the documents the dialogs shunt to 'JSON tab only' mode (unrepresentable Effect, Resource+NotResource, and the like) could never be saved - Delete was the only mutation left. Invalid JSON still blocks; an unrepresentable document now saves and the editor state stays marked unparsed. * admin: pin the policy editor to what each consumer's backend supports The s3tables evaluator has no NotResource/NotPrincipal fields - it silently drops them, turning Allow+NotResource into allow-everything and making Deny+NotPrincipal inert - and it only matches s3tables: actions against s3tables ARNs, while the editor suggested s3: actions and arn:aws:s3::: resources. New registerPolicyEditor knobs: allowNegation hides the Not* modes and routes documents using them to the JSON tab; resourceSuggestions pins the Resource autocomplete to the open resource's ARN; the S3 Tables dialogs get an s3tables-only action datalist. requirePrincipal now also hides NotPrincipal, which policy_engine.ValidateBucketPolicy always rejects, and the client-side check requires Principal specifically to match that server rule. * admin: save S3 Tables policies from a button, not form submission The multi-input structured editor sits inside a form whose Save button was type=submit, so Enter in any single-line editor input - accepting an autocomplete suggestion, say - implicitly submitted whatever half-built statement the editor held, and the backend stores the document verbatim. A lone statement with no Principal matches nobody, locking out every non-owner. Save is now an ordinary button and the form ignores submission. * admin: block zero-statement policy saves Committing the active tab before the emptiness check made 'Policy JSON is required' dead code: an empty editor serializes to {"Statement":[]}, which the s3tables backend stores verbatim - evaluated default-deny for every non-owner, while the statement-count column keeps showing 'Not configured'. All three policy dialogs now refuse a save with no statements and point at Delete instead. The classic bucket modal only gained a clearer message; the server already rejected the document. * admin: guard S3 Tables policy mutations against stale and overlapping requests The save/delete completions ran against whatever resource the shared modal happened to show by then: a slow PUT for one bucket would hide the modal mid-edit of another and misattribute its alerts, a late DELETE cleared the shared textarea over the newly opened resource with its loaded flag set, and nothing stopped a double-click from firing two overlapping mutations. Ported the classic modal's pattern: capture the target on start, flag the mutation in flight with the buttons disabled, and only touch the UI when the completion still matches the open resource. Success now reloads the page, which also keeps the Policy column's statement count honest. * admin: confirm before deleting an S3 Tables policy Delete Policy sat next to Save and fired on a single click; with default-allow enabled one stray click silently dropped the resource policy and left the bucket open to every principal. Same confirmation the classic bucket modal already has. * admin: let a corrupt stored bucket policy be shown, fixed, and deleted A stored document the decoder rejects made the policy GET 500, and with the loaded flag never set the modal blocked both Save and Delete - the one policy an operator most needs to remove was the one they couldn't, even though the delete path never reads the document. The GET now returns the raw bytes alongside a null policy; the dialog hands them to the JSON tab and unblocks the buttons. * admin: url-encode the bucket name in the policy API calls The filer lists any directory under the buckets path, names S3 would never allow included; one carrying '#' or '%' broke the fetch URL or addressed a different name than the modal shows. * admin: drop stale edit-policy responses on the IAM policies page The same race the bucket and S3 Tables dialogs already guard against: open one policy's editor while its GET stalls, open another, and the late response populates the editor under the second policy's name - Update then saves the first policy's statements over the second. * admin: warn before a bucket policy save drops unsupported fields The editor tracks unmodeled top-level keys precisely so confirmPolicyFieldDiscard can warn before the server's Version+Statement decode discards them, but only the IAM page called it; the bucket modal saved a pasted document with e.g. a console-generated Id without a word while the editor kept displaying the field. * s3: enforce the bucket policy size cap on both surfaces The 20KB cap lived only in the admin UI, so a larger policy stored via the S3 API displayed there but could never be re-saved, desyncing the two writers the cap comment claimed could not desync. The constant now lives in policy_engine next to the shared validator and PutBucketPolicy rejects oversized documents with PolicyTooLarge, matching AWS. * admin: ship the policy editor's fieldset styles with the editor The .policy-stmt-* rules that undo Bootstrap's full-width legend reset stayed behind in policies.templ when the editor markup moved to the shared script, so the bucket and S3 Tables dialogs rendered Actions/ Resource/Principal as full-width jumbo headings. PolicyDatalists is the component every consumer already renders once; the styles live there now. * s3: mirror bucket policy changes into the IAM store from the metadata subscription The advanced-IAM path appends the bucket-policy:<bucket> document to every STS/session evaluation, but only this gateway's own PutBucketPolicy maintained that mirror - a policy tightened or created through the admin UI (or another gateway) never reached it, so revoked access stayed live indefinitely, and the delete side was an unimplemented TODO in any case. The metadata subscription now diffs the stored policy on every bucket entry change and updates or removes the mirror, covering all writers and deletion with one mechanism; IAMManager gains the missing RemoveBucketPolicy. * admin: deduplicate the bucket policy write path Set and Delete carried line-for-line identical filer closures; bucketPolicyMutation already treats nil as clear-the-key. The shared helper sits below Set's validation, since ValidatePolicy cannot take the nil document Delete passes. * s3: drop ValidateBucketPolicy's re-checks of ValidatePolicy rules Both callers run ValidatePolicy first, which already enforces the version and at-least-one-statement rules; the duplicates were dead code with drifted error text. * admin: seed a new statement's Resource from the pinned suggestions A fresh statement on the S3 Tables dialogs started with no resource row at all; seed it with the broadest pinned ARN the same way cfg.bucket already seeds the classic modal. * admin: refuse to save Not* fields the backend would silently drop Hiding the NotResource/NotPrincipal modes was not enough where negation is disallowed: the JSON tab accepts any valid document (that is its job), and a statement's Advanced-fields box can reintroduce the keys, so an s3tables save could still store fields the evaluator drops - turning Allow+NotResource into allow-everything. commitPolicyActiveTab now runs a final document-level check over what would actually be saved; Delete stays available for cleanup. * s3: move the IAM bucket policy mirror on a bucket rename A same-directory rename delivers one event carrying both entries, and the byte-equality short-circuit skipped the new name's mirror when the policy was unchanged - while the replayed delete for the old name removed its mirror, leaving the renamed bucket unmirrored. The mirror decision is now a pure function that removes the old name and writes the new one regardless of byte equality, with the rename cases unit tested. * s3: backfill the IAM bucket policy mirror on lazy bucket loads The metadata subscription only mirrors changes, so a policy that predates the IAM integration never reached the bucket-policy:<bucket> mirror and its grants did not bind on the IAM path until the policy was next modified. The gateway is deliberately lazy at startup (nothing lists all buckets), so the backfill hooks the same place a bucket's policy first becomes known: the cold bucket-config load. EnsureBucketPolicy writes only when no mirror is stored, so repeat loads cost one cached read. * s3: reconcile the bucket policy backfill against concurrent changes The backfill's check-then-write could race an event-driven mirror update or removal and re-store bytes that were already stale, with no later event to heal it. EnsureBucketPolicy now reports whether it wrote, and a write is reconciled against a fresh authoritative entry read: a changed policy is re-mirrored, a removed one is removed. Anything changing after that read fires its own event, which finds the backfill's write already present and supersedes it. The backfill also carries the entry's raw bytes rather than a re-marshaled document, so the reconcile can byte-compare. * s3: prime the bucket policy mirror before advanced-IAM authorization The backfill ran from the lazy bucket-config load, but IAM authorization evaluates the bucket-policy:<bucket> mirror before any handler runs - a grant carried only by a not-yet-mirrored policy denied forever, and the denied request never reached the code that would have loaded the bucket. authorizeWithIAM now primes the bucket config first (an in-memory cache hit once warm), and the backfill runs synchronously on the cold load so the very first authorization already sees the mirror.	2026-08-24 00:52:01 -07:00
Mathieu Arnold	e931cccc7b	Manage bucket policies via the admin ui (#10895) ... * admin: manage S3 bucket policies from the admin UI Bucket policies were only manageable through the S3 PutBucketPolicy API; the admin UI had no equivalent to the quota/owner/lifecycle editors it already offers. Add GET/PUT/DELETE for a bucket's policy, sharing the exact validation the S3 gateway uses. - Extract validateBucketPolicy/validateResourceForBucket out of s3api_bucket_policy_handlers.go into policy_engine.ValidateBucketPolicy / ResourceMatchesBucket so both the S3 API and the admin UI enforce identical rules. - weed/admin/dash/bucket_policy.go: Get/Set/DeleteBucketPolicy, writing through ObjectTransaction + PATCH_EXTENDED (the lifecycle pattern) so a concurrent owner/quota/lifecycle change on the same bucket entry isn't clobbered. Propagation to every S3 gateway is automatic via the existing filer metadata log subscription. The S3 gateway's IAM policy mirror is deliberately not replicated here (its delete path is already an unimplemented TODO on the S3 side). - New GET/PUT/DELETE /api/s3/buckets/{bucket}/policy routes, CSRF-guarded on writes. - Bucket list and details modal now show a statement-count badge, read from the entry already fetched (no extra RPC). - UI: a JSON-textarea policy editor modal, matching the lifecycle modal's structure. * admin: reuse the visual policy editor for bucket policies Extract the structured policy editor (add/remove statement, action/ resource/principal rows with autocomplete, JSON tab kept in sync) out of policies.templ's inline script into a shared weed/admin/static/js/policy_editor.js, and wire the bucket policy modal in s3_buckets.templ up to it instead of a bare JSON textarea. - registerPolicyEditor(which, config) replaces the hardcoded create/edit id derivation with a per-instance config (textarea/tab/body ids, datalist ids, requirePrincipal, bucket). The IAM policies page keeps its exact pre-extraction ids via two registerPolicyEditor calls, so its markup is unchanged. - New policy_datalists.templ exposes the three shared <datalist>s (actions/resources/principals) as @PolicyDatalists(), now rendered by both policies.templ and s3_buckets.templ. - requirePrincipal seeds new bucket-policy statements with Principal: "*" and adds a client-side check before save (the server, via policy_engine.ValidateBucketPolicy, remains the actual authority); the bucket config pins the Resource autocomplete to the open bucket instead of fetching every bucket in the cluster. - layout.templ loads policy_editor.js globally, after admin.js/ modal-alerts.js (basePath/escapeHtml/showAlert) which it depends on. 3a (the extraction) is a byte-preserving move verified against the unchanged policies.templ behavior before layering 3b's parameterization and the bucket-policy wiring on top. * admin: migrate S3 Tables bucket/table policy editors to the shared editor Third consumer of the shared visual policy editor: the S3 Tables bucket and table policy modals (a bare JSON textarea each) now get the same structured Editor/JSON tabs as the bucket policy and IAM policy pages, via registerPolicyEditor('s3tablesBucketPolicy'/'s3tablesTablePolicy', { textareaId: ... }). Storage and validation are untouched - S3 Tables policies still go through their own s3tables.PolicyDocument type and the s3tables.policy extended attribute, unrelated to policy_engine and s3-bucket-policy; only the editor UI is shared. Fix a real bug surfaced by adding this second load path: the bucket policy modal (and the naive first draft of this s3tables port) called commitPolicyTextareaToEditor() right after a GET and then force-switched to the Editor tab. commitPolicyTextareaToEditor() is designed to leave the current tab in place and the editor state untouched when a document fails to parse (so an in-progress edit survives a bad tab switch), so forcing the Editor tab afterwards could show empty/stale editor state that a careless Save would then serialize over a perfectly valid but structurally-unusual stored policy. Add loadPolicyTextareaIntoEditor(which) to policy_editor.js, which has no "current tab" to defer to and instead falls back to the JSON tab with an alert on a document the structured editor can't represent - the same safety editPolicy already had in policies.templ - and use it at all three "populate the editor right after a GET" call sites (bucket policy, S3 Tables bucket policy, S3 Tables table policy). * admin: show policy statement count on the S3 Tables buckets page Mirrors the "Policy" column already added to the classic S3 buckets list: a clickable badge with the statement count when the table bucket has a resource policy, "Not configured" otherwise. S3 Tables policies are a separate mechanism (s3tables.PolicyDocument under the s3tables.policy extended attribute) from the S3 bucket policy work elsewhere in this branch (policy_engine.PolicyDocument / s3-bucket-policy), so this is a parallel implementation of the same pattern rather than shared code. - S3TablesBucketSummary gains PolicyStatementCount, populated in GetS3TablesBucketsData from entry.Entry.Extended[s3tables.ExtendedKeyPolicy] via the new extractS3TablesPolicyStatementCountFromEntry - no extra RPC, the entry is already fetched for ExtendedKeyMetadata. - The badge reuses the existing .s3tables-bucket-policy-btn class, so it opens the same policy modal as the row's action button with no JS changes. * admin: don't let a failed policy GET open the door to an empty overwrite loadS3TablesBucketPolicy/loadS3TablesTablePolicy cleared the textarea, then unconditionally called loadPolicyTextareaIntoEditor() regardless of whether the GET actually succeeded - including when fetch() rejected or the response was not ok, silently logged to console only. That leaves the structured editor holding a legitimate-looking empty policy ({version, statements: []}), with the Editor tab active by default. If Save is then clicked, commitPolicyActiveTab() serializes that empty state into the textarea as `{"Version":"2012-10-17","Statement":[]}` - a non-empty string - before the "Policy JSON is required" guard ever sees it, so the guard passes and the transient load failure gets written over whatever policy was actually stored. Add s3tablesBucketPolicyLoaded/s3tablesTablePolicyLoaded, set true only once a GET has actually completed (ok, including a genuinely empty policy) and false on any failure path (fetch rejection or a non-ok response, which previously fell through silently). Both submit handlers now check the flag before touching the editor at all, and a failed load surfaces via alert() instead of only a console.error - the user previously had no visible indication the load had failed. Verified with a jsdom simulation driving the real rendered page against a stubbed fetch: a failed GET followed by Save now sends no PUT at all (previously it sent Statement: []); a successful GET followed by Save still PUTs the loaded policy unchanged. * admin: address code review findings on the policy editor 1. policy_editor.js: policyEditors is only pre-populated for 'create'/ 'edit'; every other `which` (bucket, s3tablesBucket, s3tablesTable) stays undefined until its first successful async load. Nothing in this file enforces that a page hide its Editor/JSON tabs and Add-statement button until that load completes - the S3 Tables policy modals don't - so a click in that window (e.g. Add statement, or switching to the JSON tab) threw "Cannot read properties of undefined (reading 'unparsed')". Add policyEditorState(which), which lazily initializes a default state, and route addPolicyStatement, the jsonTabBtn 'show.bs.tab' handler, commitPolicyActiveTab, and renderPolicyEditor through it. Verified with a jsdom simulation against a never-resolving fetch: the exact click threw on the pre-fix code and no longer does. 2. s3_buckets.templ: the bucket-policy Save handler checked the textarea for emptiness before calling commitPolicyActiveTab(), which is what actually serializes the structured Editor tab's fields into that textarea. A policy entered entirely through the Editor tab (the primary path - never touching the JSON tab) left the textarea at whatever it was at load time, so creating a new policy this way hit "Enter a policy document" and Save silently did nothing. Move the commit before the emptiness check, preserving the existing alert and early-return. Verified with a jsdom simulation: Add-statement then Save (no tab switch) now PUTs the entered statement; before the fix the same sequence never reached fetch(). 3. s3tables_buckets.templ / s3tables_tables.templ: the policy Editor/ JSON nav-tabs were missing the ARIA roles Bootstrap's own tab pattern expects (role="tab"/"tabpanel", aria-selected, aria-controls, aria-labelledby) - screen readers had no way to tell these were tabs or which pane went with which button. Added the standard Bootstrap 5 tab markup to both. * admin: guard policy load/save flows against overlapping requests 1. s3tables.js: loadS3TablesBucketPolicy/loadS3TablesTablePolicy had no protection against overlapping loads. Opening one bucket's (or table's) policy dialog and then another's before the first GET resolved let the late response write its document into the shared textarea and mark the dialog "loaded" while it was now targeting the second resource - a subsequent Save would then push the first resource's policy onto the second. Add a per-load monotonic sequence number (s3tablesBucketPolicyRequestSeq / s3tablesTablePolicyRequestSeq, the same pattern already used for the classic bucket-policy load in s3_buckets.templ); a response is only applied - textarea, loaded flag, editor state - if its captured sequence still matches the latest one issued. Verified with a jsdom simulation: bucket A's policy load (artificially slow) followed immediately by bucket B's (fast) previously left A's policy in the textarea once A's late response landed; it now correctly keeps B's. 2. s3_buckets.templ: the bucket-policy Save button lives outside the (initially hidden) editor wrapper, so it stays clickable while a load is still in flight - the existing policyRequestSeq guard only protects the *load* from a stale response, not Save from firing before any load for the current bucket has completed. Add bucketPolicyLoaded, reset before each GET and set only once the matching response lands, and check it at the top of the Save handler. Verified with a jsdom simulation: clicking Save immediately after opening the dialog, before a (deliberately never-resolving) GET settles, now sends no PUT; a normal load-then-save sequence still PUTs the loaded policy unchanged. * admin: address further code review findings on the policy editor 1. s3tables.js: loadS3TablesBucketPolicy/loadS3TablesTablePolicy only reset the JSON textarea when a new load starts; the structured editor kept showing the previously loaded resource's statements (Editor tab is the default active one) until the new fetch resolved. Call loadPolicyTextareaIntoEditor() against the now-cleared textarea immediately, so switching resources visibly resets the editor right away instead of only once its own load completes. Verified with jsdom: opening bucket A (loads fully) then bucket B (GET never resolves) no longer leaves A's statements visible in B's editor. 2. s3tables.js: deleteS3TablesBucketPolicy/deleteS3TablesTablePolicy had no loaded-state check, so a failed GET (which already blocks Save) left Delete fully able to remove the resource's stored policy sight unseen. Add the same s3tablesBucketPolicyLoaded/s3tablesTablePolicyLoaded guard Save already uses. Verified with jsdom: delete after a failed load now sends no DELETE; delete after a successful load is unaffected. 3. s3_buckets.templ: the bucket-policy Editor/JSON nav-tabs were missing the same ARIA roles already added to the S3 Tables policy tabs in an earlier round (role="tab"/"tabpanel", aria-selected, aria-controls, aria-labelledby) - this instance was out of scope for that review comment but is the same gap. Bootstrap's own tab.js already manages aria-selected on tab switch once the attribute exists, so no extra JS was needed. 4. s3_buckets.templ: neither the bucket-policy Save nor Delete handler guarded against a double-click, or against firing while the other was still in flight - two overlapping PUT/DELETE requests for the same bucket could land in either order. Add a shared bucketPolicyMutationInFlight flag: set (and both buttons disabled) before each fetch, cleared (and buttons re-enabled) on failure so the user can retry, left set through the existing success hide-and-reload path, and also reset when a new bucket's dialog opens so an abandoned in-flight request from a closed dialog can't leave the buttons stuck disabled. Verified with jsdom: double-clicking Save now sends exactly one PUT, and a Delete click while that PUT is still pending sends no DELETE. * admin: scope bucket-policy mutation completions to the bucket that started them 1. The previous round's fix reset bucketPolicyMutationInFlight whenever a new bucket's policy dialog opened, to avoid leaving Save/Delete stuck disabled if the modal was closed mid-request. That traded one bug for a worse one: if bucket A's PUT/DELETE was still in flight when the user opened bucket B's dialog, the reset let B's Save/Delete fire immediately, and A's completion handler - unaware anything had changed - would still hide the (now B's) modal and reload the page out from under whatever the user was doing with B, on success, or alert a message with no bucket context, on failure. Stop resetting on reopen, so a pending mutation for a previous bucket keeps this bucket's Save/Delete blocked until it settles (matches the "preventing overlapping mutations" the review comment describes). Instead, capture policyEditorBucket as targetBucket right before each fetch and compare it against policyEditorBucket again in the completion handler: the in-flight flag is always released so the buttons never get stuck, but the modal-hide/reload/alert only fire if this bucket is still the one showing; a stale completion for an abandoned bucket just logs to the console instead. Verified with a jsdom simulation: opening bucket B while bucket A's Save is still pending leaves B's Save button disabled and a click on it a no-op; once A's PUT resolves, B's button re-enables but no modal.hide()/reload() fires (previously both fired unconditionally). 2. bucketPolicyDeleteBtn had no bucketPolicyLoaded check, unlike Save - a failed GET blocked Save but left Delete free to remove a policy the client never actually saw (the same gap already fixed for the S3 Tables policy modals in an earlier round). Added the same guard, ahead of the confirm() dialog. Verified with jsdom: Delete after a failed load now sends no DELETE request. * admin: fix spelling mistake	2026-08-23 22:11:18 -07:00
孙超	c80664ec21	s3: propagate storage rule fsync to volume server uploads (#10906) ... The storage rule's fsync decision was computed by the filer (detectStorageOption -> rule.Fsync) and applied on the filer's own HTTP write path, but was never carried onto the chunk uploads S3 issues: the AssignVolumeResponse had no fsync field, so the s3api client could not learn the decision, and the chunked upload URL was hardcoded without it. Every S3 write to a path with fsync configured went to the volume server as a non-fsync write. Carry the decision through the assign response: - filer.proto: AssignVolumeResponse gains bool fsync, filled from the storage option the assign resolved. - operation.AssignResult gains Fsync, so uploadChunk can append ?fsync=true to the volume server upload URL (single and replica fan-out paths). - The S3 PUT/UploadPart assignFunc, the S3 copy path, the admin file browser upload, and the Iceberg worker assign functions all forward the response field. Adds TestUploadReaderInChunksAppendsFsyncWhenAssigned.	2026-08-23 22:11:08 -07:00
Chris Lu	36c97344ef	s3: confine a Lance catalog table location to the caller's own bucket (#10901) ... The Lance namespace gateway took the request-body location field, trimmed a trailing slash, and passed it straight to the marker sink. That location feeds TableDataDirFromMetadataLocation, which joins it under /buckets and collapses any ../ segments, and writeMarker's CreateEntry then auto-creates every missing parent. A caller could point the location at another tenant's bucket, or escape /buckets entirely, and plant a fixed-name marker (recursively creating the parents) or hide a victim's live table with .lance-deregistered. Confine the declared location the way the Iceberg gateway already does: require an s3:// URI whose bucket is the caller's own and whose path carries no traversal segment, on both the declare and register handlers.	2026-08-23 11:49:52 -07:00
Chris Lu	cf0dba334c	s3api: no filer failover after the callback has consumed part of a response (#10902) ... s3api: no filer failover after fn has consumed part of a response withFilerClientFailover replays fn verbatim on the next filer, so a filer that died mid-stream followed by a healthy peer returned success with the callback's closure-captured accumulator holding the dead filer's prefix twice; the per-attempt accumulator in listWithRetry could not close this, because the replay happens inside a single attempt. Track delivery on the connection handed to fn: once a unary reply or streamed message has reached the callback, surface the transport error unwrapped instead of failing over, and let callers replay from a clean slate. A filer that fails before delivering anything fails over exactly as before.	2026-08-23 11:30:43 -07:00
Chris Lu	8d8a25b1cf	s3api: remove the duplicated listing retry helpers left by overlapping merges	2026-08-23 00:26:02 -07:00
Chris LuandJunker der Provinz	c58795354a	s3api: retry a transient filer failure on metadata listings (#10890) ... * s3api: retry a transient failure when listing multipart uploads/parts A blip on the way to the filer failed the whole ListMultipartUploads or ListParts request. Both reported failure points sit inside one streaming listing: the ListEntries call that opens the stream, and the stream.Recv calls that drain it. Neither retried, so a single Unavailable answer from a filer that was restarting turned into a 500 for the S3 client. Replay the listing instead, bounded to three attempts with a 100ms backoff that doubles. Only a transient failure is replayed. A not-found answer stays authoritative so the empty-list branch still works, and every other error still reaches the client on the first attempt. This is scoped to (*S3ApiServer).list rather than added inside DoSeaweedListWithSnapshot, which mount, the shell and the other object listings share, and where a retry after a partial stream would re-deliver entries the callback had already seen. Within one call to list, a replay is safe: it collects into a fresh slice each time, so it can neither duplicate nor drop entries. That guarantee does not extend past this function. withFilerClientFailover already re-runs its callback against the next filer on any non-NotFound error without resetting the caller's accumulator, so on a multi-filer gateway a mid-listing failover can itself produce a duplicated result with err == nil, independent of this change and not fixed by it. Noted in the PR rather than silently left for someone to rediscover. Fixes #7221 References #7235 * s3api: move the listing retry inside list itself --------- Co-authored-by: Junker der Provinz <jdp@braethoria.com>	2026-08-22 23:42:33 -07:00
Junker der Provinz	f710b6003a	s3api: retry a transient failure when listing multipart uploads/parts (RFC on layering) (#10886) ... s3api: retry a transient failure when listing multipart uploads/parts A blip on the way to the filer failed the whole ListMultipartUploads or ListParts request. Both reported failure points sit inside one streaming listing: the ListEntries call that opens the stream, and the stream.Recv calls that drain it. Neither retried, so a single Unavailable answer from a filer that was restarting turned into a 500 for the S3 client. Replay the listing instead, bounded to three attempts with a 100ms backoff that doubles. Only a transient failure is replayed. A not-found answer stays authoritative so the empty-list branch still works, and every other error still reaches the client on the first attempt. This is scoped to (*S3ApiServer).list rather than added inside DoSeaweedListWithSnapshot, which mount, the shell and the other object listings share, and where a retry after a partial stream would re-deliver entries the callback had already seen. Within one call to list, a replay is safe: it collects into a fresh slice each time, so it can neither duplicate nor drop entries. That guarantee does not extend past this function. withFilerClientFailover already re-runs its callback against the next filer on any non-NotFound error without resetting the caller's accumulator, so on a multi-filer gateway a mid-listing failover can itself produce a duplicated result with err == nil, independent of this change and not fixed by it. Noted in the PR rather than silently left for someone to rediscover. Fixes #7221 References #7235	2026-08-22 23:00:50 -07:00
3 people	641fc8b031	admin: add visual iam policy editor (#10878) ... * admin: add visual iam policy editor Add a structured, tabbed editor (Editor / JSON) for creating and editing IAM policies in the admin dashboard, alongside the existing raw-JSON textarea: - policies.templ: per-statement cards for Sid, Effect, Action, and Resource, with unmanaged fields (Principal, NotPrincipal, NotResource, Condition, or anything else) preserved verbatim in a per-statement "advanced fields" JSON box so nothing is lost on round-trip. Switching tabs commits and reparses in both directions. Restored the "Use Sample Policy" button, now filling both the structured editor and the JSON tab. The "Validate" button now calls the existing but previously unused POST /api/object-store/policies/validate endpoint instead of doing JS-only checks. - Progressive Resource ARN autocomplete: suggests bucket names first, then once "bucket/" is typed, suggests bucket/* plus the bucket's direct subfolders, drilling down one path segment at a time as the user types further "/" characters. - New GET /api/files/list-folders endpoint (file_browser_handlers.go) backing the folder autocomplete: wraps the existing file browser data function and returns just the subdirectory names as JSON, scoped to paths under /buckets. - Action-name suggestions (datalist) for the Action field, sourced from the existing s3_constants.S3_ACTION_* constants plus new s3_constants.S3TABLES_ACTION_* constants (extracted from the s3tables operation dispatch switch) so the suggestion list can't drift from the strings the engines actually understand. - policy_handlers.go: ValidatePolicy now accepts a statement with only NotResource set (previously required Resource), matching policy_engine.validateStatement and the fact the new editor makes such statements reachable from the UI. - Tests: ValidatePolicy behavior, route registration for the policy API and the new list-folders endpoint, list-folders path scoping, and the action-suggestion list's shape. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * admin: fix XSS, cache poisoning, and cap overshoot in policy editor Address code review findings on the IAM policy editor added in the previous commit: - policies.templ (displayPolicyDetails): escape every interpolated policy value (Sid, Effect, Action, Resource, policy name, and the raw JSON document) before assigning to innerHTML. Policy documents can come from other admins or an import, so an unescaped field could execute script when the "View" modal renders it. - policies.templ (policyEditorStateToDoc): reject JSON arrays in a statement's "advanced fields" box, not just invalid JSON. `typeof [] === 'object'` was true, so a JSON array was assigned to the statement; subsequent property assignments (Sid, Effect, ...) landed on the array object but JSON.stringify of an array only serializes numeric indices, silently dropping them. - policies.templ (loadPolicyFolderNames): on a failed folder lookup, remove the cache entry instead of permanently caching the empty fallback, so a transient network/server error doesn't block retries for the rest of the page's lifetime. - file_browser_handlers.go (ListFolders): stop appending directory names as soon as the running count reaches maxListFoldersEntries, instead of only checking the cap after a full page is processed, so the returned list never exceeds the configured cap. Regenerated policies_templ.go with the already-stamped templ v0.3.1001 to keep the diff scoped to this file. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * admin: stop policy editor from clobbering the active tab and dropping malformed advanced fields Address two review findings on the IAM policy editor (Issue 3, stored-XSS in displayPolicyDetails, was already fixed by the previous commit and is unchanged here): - createPolicy, updatePolicy, and validatePolicyDocument always committed the structured editor's (possibly stale) state into the JSON textarea before submitting, even when the user had just edited the JSON tab directly. That silently discarded the user's JSON edits and validated/saved the old structured-editor state instead, which could leave broader permissions in force than intended. Added commitPolicyActiveTab(which), which commits whichever tab is currently visible into the other side instead of unconditionally overwriting the JSON tab from the editor: if the JSON tab is active it parses that JSON back into the structured editor (without touching the textarea itself), otherwise it serializes the structured editor into the textarea as before. All three call sites, plus the JSON-tab "show.bs.tab" handler, now use this and abort with an alert if the currently active tab's content can't be committed. - policyEditorStateToDoc silently continued with an empty object when a statement's "advanced fields" box held invalid JSON, so switching tabs, validating, or saving would drop Principal/NotResource/Condition from that statement without telling the user. It now throws (with the statement number and parse error) on invalid or non-object JSON there, and callers surface that via showAlert and abort instead of proceeding. Regenerated policies_templ.go with the already-stamped templ v0.3.1001. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * admin: keep unmanaged top-level policy fields across editor tab switches policyDocToEditorState only carried Version and Statement into editor state, so any other top-level key (e.g. Id) present in the JSON tab was silently rewritten away as soon as the user switched to the Editor tab and back. Capture those keys in state.otherFields and merge them back in policyEditorStateToDoc before Version and Statement are written, so the two tabs stay faithful to each other and the editor never rewrites text the user typed. Note this is editor fidelity only: the admin API's policy_engine.PolicyDocument carries just Version and Statement, and DocumentJSON is never populated, so such fields are still discarded by the server once a policy is saved. Making them survive a save would require a backend change, which is out of scope here. Regenerated policies_templ.go with the already-stamped templ v0.3.1001. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * admin: warn before a policy save discards unsupported top-level fields The editor round-trips unmanaged top-level keys (e.g. Id) between the Editor and JSON tabs, but the admin API's policy_engine.PolicyDocument carries only Version and Statement, so the server drops them on save and the user saw no indication. Added confirmPolicyFieldDiscard(), called from createPolicy and updatePolicy after the active tab is committed (so the field list is accurate whichever tab is showing). It names the fields that will be lost and lets the user confirm or cancel. Not wired into validatePolicyDocument, which doesn't persist anything. Chose the warning over the alternative of persisting these fields through the backend: policy_engine.PolicyDocument is shared by the S3 bucket-policy engine and IAM evaluation, so extending it would change the stored document shape for every policy in the codebase - far beyond the scope of this editor. Regenerated policies_templ.go with the already-stamped templ v0.3.1001. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * admin: reject malformed Effect and Resource/NotResource conflicts in policy editor Two review findings on the IAM policy editor: - policyDocToEditorState defaulted any non-"Deny" Effect (missing, misspelled, wrong case) to "Allow". A statement meant to be "Deny" with a typo like "deny" would silently become a permissive "Allow" instead of being rejected. It now throws on anything but an exact "Allow" or "Deny", naming the offending statement and value. commitPolicyTextareaToEditor catches this the same way it already catches invalid JSON: alert the user and keep the JSON tab active instead of switching to the Editor tab with wrong data. - policyEditorStateToDoc could save a statement with both Resource (from the structured field) and NotResource (surviving in the "advanced fields" extras from before the user switched to using Resource) set at once - a contradictory combination neither the admin's ValidatePolicy handler nor policy_engine's evaluator rejected. When the structured Resource field is non-empty it now deletes any leftover NotResource from extras, consistent with the file's existing rule that structured fields take precedence over extras. Mirrored the existing Principal/NotPrincipal exclusivity check in weed/admin/handlers/policy_handlers.go's ValidatePolicy to reject the same combination server-side, since create/update perform no validation at all. Deliberately left policy_engine.validateStatement (used by the S3 bucket-policy PUT handler for every bucket policy in the product) unchanged - extending that shared validator is a larger, separate change outside this admin-editor fix's scope. Added a handler test for the new Resource+NotResource rejection. Regenerated policies_templ.go with the already-stamped templ v0.3.1001. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * admin: add NotResource support to the visual policy editor Since Resource and NotResource are mutually exclusive (enforced by a previous fix), NotResource could previously only be set through the raw JSON in a statement's "advanced fields" box. Promote it to a first-class mode of the structured editor: - The static "Resources" label is now a Resource/NotResource dropdown; the same list of values underneath is reused for either key depending on the selected mode, with a short form-text explaining the semantics. - NotResource is added to POLICY_STATEMENT_KNOWN_KEYS, since it's now a managed field like Resource rather than something that falls through to extras. - policyDocToEditorState derives resourceMode from which key is present on load, and throws (same handling as the existing malformed-Effect case: alert, keep the JSON tab active) if a hand-edited document has both Resource and NotResource on one statement, since that can't be represented by the dropdown. - policyEditorStateToDoc writes only the key matching the selected mode, replacing the previous one-directional "delete NotResource whenever Resource is set" fix with mode-driven logic that also deletes Resource when NotResource is selected. - displayPolicyDetails (the read-only View modal) now shows the actual NotResource values with a distinct label instead of a static "(NotResource used instead)" placeholder. No backend changes: the server-side "cannot specify both" check added previously in policy_handlers.go's ValidatePolicy already covers this. Regenerated policies_templ.go with the already-stamped templ v0.3.1001. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Eb2a51LciCsyY35sqDoGNe * admin: reject non-object policy documents; catch Principal/NotPrincipal conflicts server-side Two review findings: - policyDocToEditorState treated a top-level JSON value that wasn't an object (null, or a bare string/number/boolean) as an empty statement list instead of failing explicitly. If the user typed e.g. "hello" or 42 in the JSON tab and switched to the Editor tab, their input was silently discarded and replaced with an empty policy - the same class of "guess instead of reject" bug fixed for malformed Effect and Resource/NotResource conflicts previously. Added an explicit check that throws for null/scalar input, while leaving array and object document shapes accepted exactly as before. - weed/admin/handlers/policy_handlers.go's ValidatePolicy checked the Resource/NotResource conflict by non-empty length (len(...Strings()) > 0), which misses a statement where Resource is explicitly present but an empty list (e.g. "Resource": []) alongside a non-empty NotResource. Switched that check to field presence (!= nil), matching how policy_engine's own validateStatement already treats Principal/NotPrincipal exclusivity. Also added the equivalent Principal/NotPrincipal presence check to this handler, which had none before - the advanced-fields box in the visual editor lets a user set both today, and nothing server-side caught it. The existing non-empty "Resource or NotResource is required" check is left as a length check, since an empty array shouldn't count as "provided". Added test cases for both conflict checks in policy_handlers_test.go. Regenerated policies_templ.go with the already-stamped templ v0.3.1001. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Eb2a51LciCsyY35sqDoGNe * admin: add Principal/NotPrincipal support to the visual policy editor (v1, AWS-only) Adds a first, deliberately narrow structured editor for a statement's Principal/NotPrincipal, left out when NotResource support was added: - A Principal/NotPrincipal mode dropdown mirrors the existing Resource/NotResource one (same mutual-exclusivity handling: the two fields can't be set at once, and switching modes reuses the same value list). - A simple repeatable text-value list feeds a single {"AWS": [...]} object on save - always the AWS type, never the "bare" (untyped) SeaweedFS-extension shape. Per policy_engine's allowedPrincipalKeys, Service/Federated/CanonicalUser also parse successfully, but nothing in the S3 bucket-policy evaluation path ever sets a real caller's principal to a service name, an OIDC provider ARN, or a canonical user ID, so only AWS is functionally meaningful today - out of scope for this v1. - On load, only the exact {"AWS": ...} single-key shape is unwrapped into the structured field and removed from "extras". Anything else (bare string/array, a different single type key, or several type keys at once) is left untouched in "extras" exactly as before, with a visible warning under the dropdown so the user knows a Principal/NotPrincipal exists but isn't shown there. Saving with the structured field left empty never touches whatever's already in extras, so a preserved complex form isn't silently dropped just because the user didn't touch this field. - The read-only View modal now displays Principal/NotPrincipal for any shape (via a small generic summarizer), not just the AWS-simple one. - Generalized the action/resource field-to-state-key mapping (used by commitPolicyEditorForm and the add/remove-item click handler) into a shared lookup table instead of stacking another ternary, now that a third field (principal) exists. Regenerated policies_templ.go with the already-stamped templ v0.3.1001. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Eb2a51LciCsyY35sqDoGNe * admin: support the bare "*" wildcard Principal in the visual editor "Principal": "*" (and NotPrincipal: "*") is the standard AWS shorthand for "everyone" and is common in real bucket policies, but the v1 Principal/NotPrincipal editor only recognized the {"AWS": ...} object form, leaving a bare "*" statement's principal hidden in Advanced fields. parseSimpleAwsPrincipal now also accepts the bare string "*" as a simple, structurally-editable value. On save, a principal value list containing exactly ["*"] is written back as the bare "*" string (matching the common convention) rather than wrapped as {"AWS": "*"}; anything else still wraps under AWS as before. Updated the field's form-text hint accordingly. Regenerated policies_templ.go with the already-stamped templ v0.3.1001. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Eb2a51LciCsyY35sqDoGNe * admin: add Principal field autocomplete backed by users + IAM roles Adds a datalist-backed autocomplete for the policy editor's Principal/ NotPrincipal text fields, sourced from a new API listing existing identities: - weed/admin/dash/principal_suggestions.go: AdminServer.GetPrincipalSuggestions combines S3 user ARNs (via the existing GetObjectStoreUsers + iam.UserArn) with IAM role ARNs (via integration.NewFilerRoleStore / ListRoles, reusing the exact same construction already used in iam_manager.go - no new dependency risk introduced). Role ARNs are reconstructed from the role name using SeaweedFS's default arn:aws:iam::role/<name> convention rather than fetching each role's stored definition, since this only backs a suggestion list. Role listing failures are logged and swallowed rather than failing the whole request - an incomplete suggestion list is fine, blocking policy editing over it is not. Service accounts are deliberately not listed separately: a service account's ARN is identical to its parent user's, already covered by the user list. - weed/admin/handlers/policy_handlers.go: GetPrincipalSuggestions handler exposing this as {"principals": [...]}. - Route registered at the API root (GET /api/principals) rather than under policyApi's "/object-store/policies" prefix, since that subrouter's existing "/{name}" GET route would shadow any single-segment GET route registered after it (the same class of gotcha previously seen with "/validate"). - weed/admin/view/app/policies.templ: a shared, lazily-fetched-once policyPrincipalSuggestions datalist (flat list - unlike the progressive per-folder Resource ARN autocomplete, users/roles aren't hierarchical), wired into policyListRowHtml for field:"principal" and populated on input/focus, with "*" always offered first. Added tests for the new ARN-construction helper and route registration. Regenerated policies_templ.go with the already-stamped templ v0.3.1001. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Eb2a51LciCsyY35sqDoGNe * admin: fix fieldset/legend styling in the structured policy editor Bootstrap's form reset stretches <legend> to the fieldset's full width (float: left; width: 100%), which loses the native "notch in the border" look and makes each section's label bar as wide as the card. Add two scoped classes: .policy-stmt-fieldset (border, rounded corners, spacing between sections) and .policy-stmt-legend (undoes the float/width so the legend hugs its content, with a little padding). Applied to the three per-statement sections (Actions, Resource/NotResource, Principal/NotPrincipal), replacing the ad hoc "border rounded" utility classes that were doubling up with the fieldset's own border. Also gave the "Advanced fields" <details> a small top margin to match the new spacing. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Eb2a51LciCsyY35sqDoGNe * admin: suggest bucket/* alongside the bucket itself in Resource autocomplete At the bucket-name stage of the Resource field's progressive autocomplete, only "arn:aws:s3:::bucket" was offered. Add "arn:aws:s3:::bucket/*" right alongside it, since granting access to everything in a bucket is the more common case and previously required typing a "/" first to reach the folder-level "*" suggestion. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Eb2a51LciCsyY35sqDoGNe * admin: keep an unparseable policy in the JSON tab instead of wiping it editPolicy() built the structured state inside the fetch .then, so a policy the editor cannot model threw into the sibling .catch, which alerted and called hide(). Showing the alert at that moment left the modal on screen with an empty editor and the document only in the JSON tab, and Save Changes then serialized the empty state over the policy. Reachable two ways, since neither create path rejects these: the admin API never validates on create, so "Effect":"allow" is stored as typed, and policy_engine.validateStatement lets Resource and NotResource sit in the same statement. Hand the document to the JSON tab instead, which is what that tab is for, and mark the state so nothing serializes the placeholder over it. * admin: validate a policy document before saving it Validation was wired only to the Validate button, so nothing stopped a document the server's own validator rejects from being stored. With the structured editor supplying the boilerplate and required dropped from the textarea, opening the modal, typing a name and clicking Create Policy was enough to save a statement-less policy. Share validatePolicyJSON with the two save paths and abort on failure. * admin: bound the folder autocomplete listing maxListFoldersEntries caps the folders collected, but nothing capped the entries paged through to find them, so a bucket holding only flat object keys - no subfolders to count - was walked to the end, 200 entries per round trip, behind one keystroke. Measured against an in-process filer: 6 entries 0.5ms, 3k entries 7.7ms, 30k entries 53ms, all of it linear in the directory rather than in the answer. Cap the scan as well, and let GetFileBrowser take a prefix so the segment the user is still typing is filtered by the filer instead of by paging. The same 30k directory now answers in 0.6ms once a prefix is typed. * admin: clean the path before scoping list-folders to /buckets util.CleanWindowsPath only rewrites backslashes, so "/buckets/../etc" walked straight past the prefix check the endpoint relies on for its scope. Nothing leaked - filer paths are literal keys, so the traversal resolved to nothing - but the check reads as a boundary and wasn't one, and the test asserting it didn't cover the one input that would try. validateAndCleanFilePath in the same file already does this. * admin: stringify policy values before escaping them escapeHtml calls text.replace directly, and the Sid, the per-item action and resource inputs, and the View modal's Resource/NotResource all pass values straight out of JSON.parse. A policy carrying "Sid": 5 or "Action": [1] threw "text.replace is not a function" and took the render with it. escapedJoin already coerced; use it everywhere and coerce the editor state at the point it's built. * admin: only show the NotResource hint in NotResource mode The hint rendered unconditionally, so it sat under a selector reading "Resource" telling the user the statement applies to everything except what they'd listed. Redraw the card when the selector changes so it follows the mode. --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-08-22 12:33:08 -07:00
3 people	df93d01c06	admin: add bucket lifecycle rule editing (#10860) ... * admin: add bucket lifecycle rule editing * address greptile's comments * more small fixes * coderabbit's comments * more comment fixes * more fixes * more * maybe last * last ? * 14850 * 14851 * filer: stamp the content MD5 on every SaveInsideFiler write An entry's ETag falls back to Attributes.Md5, so conditional writers key IF_ETAG_MATCH off it. SaveInsideFiler carried the looked-up attributes forward without refreshing the hash, leaving it describing whatever the previous writer stored: a later conditional write matched the stale hash and overwrote content that had already changed. * s3api: give the bucket lifecycle constants and the write route key one definition each The extended-attribute keys, the XML size cap and the object-write ring key prefix were each spelled out in two places, so the admin dashboard's copies could drift from the gateway's. Move them to the packages both sides already import and alias them where the short local name reads better. * admin: patch the bucket entry's lifecycle keys instead of rewriting the entry The save read the bucket entry, edited its extended map and wrote the whole entry back, guarded by IF_UNMODIFIED_SINCE. Nothing that writes a bucket entry advances its mtime - not the S3 gateway's patchBucketEntry, not SetBucketOwner, not SetBucketQuota - so the guard never fired and the stale snapshot reverted whatever else had changed since the lookup. Send the PATCH_EXTENDED mutation the S3 gateway already uses for these keys: the filer re-reads and merges under the bucket path lock, so only the two lifecycle keys move. That removes the reason for the mtime snapshot, the verification retry loop and the compensating restore of the cleared day-TTL rules, which the migration now logs instead. * s3api: run the delete-lifecycle day-TTL migration through the shared helper DeleteBucketLifecycleHandler kept its own copy of the read-strip-write sequence the put handler now shares, including a missing return that let a ToText failure persist a truncated filer.conf and write a second response. It also wrote the whole file back unconditionally, reverting any concurrent edit; the shared helper writes conditionally. * admin: answer 404 when a lifecycle request names a bucket that does not exist Every SetBucketLifecycle failure came back as 500, including the lookup miss for an unknown bucket, so a client or monitor read a caller error as a server fault and retried it. * s3api: emit lifecycle XML a client would recognize Two changes to what MarshalCanonical writes, both visible through GetBucketLifecycleConfiguration, which replays the stored bytes verbatim: stamp the S3 namespace on the root, and put a size range under <And>. A <Filter> carries one predicate, so two size bounds side by side is a shape AWS does not document. Parsing still accepts either. * admin: fix the lifecycle editor's handling of stored status, deletes and empty saves Four things the editor got wrong: A stored <Status> the S3 API never validated, say 'enabled', left both radio buttons unchecked, so reading the form threw on a null querySelector result and Save did nothing. Collapse anything but an exact 'Enabled' to 'Disabled', which is what the engine already does with it. Deleting a rule re-rendered an open edit form from the snapshot taken when editing began, discarding what had been typed; every other transition folds the form in first. The Transition warning only matched a bare <Transition>, missing the form with attributes, self-closed or namespace-prefixed. Saving an emptied rule list clears the configuration through a path with no prompt, next to a Delete-all-rules button that asks. Also collapses the three divergent copies of formatBytes on this page to one. * filer: stop the day-TTL migration from deleting an operator's path rule The migration removed every rule under the bucket's path that carried a day TTL in the bucket's collection. The add path it is retiring used AddLocationConf, which merged its TTL onto whatever already sat at the prefix, so a rule can hold operator settings the lifecycle path never wrote - a disk type, WORM retention, a read-only flag, a placement pin. Deleting the whole rule to retire its TTL took those with it, leaving objects under that prefix on defaults nobody asked for. Delete only rules shaped like ones the add path created from scratch; anything else keeps its settings and loses just the TTL. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>	2026-08-21 23:42:26 -07:00
Chris Lu	96304b6870	S3: source config credentials from the environment, and let the chart point at an existing secret (#10868) ... * s3: resolve ${VAR} in static config credentials from the environment A deployment that keeps its S3 keys in a secret store had no way to hand them to the gateway: -config takes a file, so the keys had to be written into that file. Let a key in the static config name an environment variable instead, and drop any credential whose reference stays unset so the placeholder never becomes a usable key. * helm: source the generated s3 identities from an existing secret The only way to reuse credentials that already live in a Secret was to hand-author the whole seaweedfs_s3_config JSON, since the literal keys in values.yaml end up in git and a lookup-based keyRef renders empty under helm template and Argo CD. Let s3.credentials.admin/read name a Secret and its keys instead: the generated config references them as ${VAR} and the gateway resolves them from the environment, so nothing is read from the cluster at render time. * s3: treat an empty environment value as an unresolved credential reference A secret store can hand over a key that exists but is blank. Resolving it would leave an access key whose signing secret is empty, so count it as unresolved and drop the credential. * helm: render the s3 secret when only the all-in-one auth flag is set The all-in-one deployment mounts the s3 secret whenever any of the three enableAuth flags is set, but the secret itself only rendered for the s3 and filer flags, so allInOne.s3.enableAuth on its own left the pod waiting on a secret nothing creates. * helm ci: check the credential wiring on every workload that mounts it The render check only looked at the standalone s3 deployment and only at one of the four variables, so a helper that bound a variable to the wrong secret key would still pass. * helm: create the all-in-one s3 secret for every flag that mounts it The all-in-one pod mounts the secret on any of the three enableAuth flags, so keying its creation off allInOne.s3.enableAuth alone still left filer.s3.enableAuth without filer.s3.enabled pointing at a secret nothing creates. Mirror the deployment's own condition instead, and check each flag renders both the mount and the secret. * s3: reject a malformed credential reference instead of keying on it A typo such as ${MY-VAR} matches no substitution, so it survived expansion and the placeholder itself became the access key the gateway accepted. Require every ${ in a static credential to open a well-formed reference.	2026-08-21 22:32:47 -07:00
3 people	930603eb74	S3: optionally serve remote-mounted objects from remote when the local read fails (#10837) ... * feat(s3): serve from remote on local read failure When a locally-cached chunk of a remote-mounted object becomes unreadable (volume server down/restarting, or an evicted needle 404ing under retry-backoff), fall back to serving the object from its mounted remote instead of erroring. A bounded pre-flight probe makes a stuck volume trip the timeout rather than stalling the request. Gated by -localReadFallbackToRemote (default off) with -localReadFallbackTimeout (2s default), so existing deployments are unaffected until they opt in. * fix(s3): register local-read-fallback flags for mini/server/filer The mini, server and filer launchers build S3Options directly and only populate the flag pointers they register. Without registering the two new flags there, startS3Server dereferenced nil pointers and crashed at boot, failing every integration suite that runs `weed mini`. * fix(s3): treat a zero-byte probe read as unreadable A read that returns no byte -- whether it reports io.EOF or no error at all -- means the offset is not locally readable, so the probe must fall back to the remote rather than proceeding to stream a truncated response. Only a returned byte (including the object's final byte with a trailing io.EOF) counts as readable. * s3: finish a mid-stream local read failure from the remote mount The pre-flight probe only proves the byte at the requested offset readable. A multi-chunk object can still lose a later chunk after the 200/206 and its Content-Length are committed, which truncated the body with no fallback. Resume from the mounted remote at the byte the local copy stopped at, so the response still carries the declared length. A short local read that surfaces as a clean EOF is treated the same way instead of silently truncating. * s3: fall back to the remote mount without a CLI switch Serving a remote-mounted object from its authoritative remote is what the read should have done all along -- the alternative is a 500 on an object the cluster can still reach -- so make it the behavior instead of two new flags, with the probe bounded by a constant. * s3: trim the comments on the fallback path * filer: report only the contiguous prefix when a parallel chunk read fails The parallel branch of doReadAt fans the chunk reads straight into their own windows of the output buffer, then sums every task's bytesRead. A middle chunk failing while a later one succeeds therefore returned a length covering a hole the reader never filled, handing the caller zeros in the middle of otherwise valid data. * s3: only splice the remote onto a local prefix while it is the cached generation Eligibility establishes a size match, not byte identity: a remote key overwritten with same-size content between the cache fill and the fallback would have finished the response with bytes from a second generation, under the first one's ETag. Stat the remote before resuming and keep the local error when it no longer matches -- a truncated body is a visible failure, a spliced one is not. --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-08-21 11:47:53 -07:00
Chris Lu	8c7d714d5e	Lance catalog, and a Rust plugin worker to maintain it (#10841) ... * iceberg: skip tables the maintenance worker does not own A Lance dataset registered through the Lance namespace's Iceberg REST adapter arrives as an Iceberg table with a placeholder schema and table_type=lance, and keeps its fragments under data/ - the same subdirectory the orphan cleaner walks. Every fragment is unreferenced by the Iceberg metadata, so a maintenance pass deletes the dataset. Views share the entry shape and were only skipped because parsing their metadata happened to fail first. Gate the scan and the execution path on the entry actually being an Iceberg table. Maintenance is off by default, so this was latent rather than live. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tables: let a table declare a format the catalog does not interpret CreateTable accepted ICEBERG and nothing else. A Lance table has no metadata file for the catalog to maintain - the entry records a name and the dataset root, and the client owns everything under it - so accept LANCE, and carry the declared format on the entry instead of hardcoding it back on the way out. ListTables now reports format and metadataLocation, so listing a catalog that holds both kinds takes one pass rather than a GetTable per row. AWS omits both fields; adding them is additive. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tables: move the in-memory filer into its own package The Lance namespace tests need the same harness, and copying it would leave two of them to keep in step. Extracted as it was, plus the two fidelity gaps that only surface once a paginating caller uses it: ListEntries ignored startFromFileName and limit, so a caller that paginates re-read the first page until it hit its own cap and reported the same entry over and over, and GetFilerConfiguration was missing, which CreateTableBucket needs to resolve the buckets directory. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: serve the Lance Namespace REST spec A second catalog surface beside the Iceberg one, over the same table buckets: the namespace and table metadata operations, the $-delimited identifier codec, the spec's numeric error model, the directory-catalog marker files, and storage_options vending through the STS path the Iceberg catalog already uses. Listens on -port.lance, 9101 by default, and inherits ARNs, policies and tags from the storage layer, so a Lance table needs no second permission model. Identifiers map bucket / namespace / table onto the three levels Lance clients already use, which is why there is no warehouse selector to invent. The data plane needs Lance format support that does not exist in Go and answers with the spec's Unsupported code rather than a bare 404. Two things it deliberately will not do: create a table bucket as a side effect of creating a namespace inside one, since a bucket carries its own policy and lifecycle, and resolve an Iceberg table's location for a Lance client, which would hand it a table another engine owns. The design note this follows is in design-lance-catalog.md, including the .lance directory suffix it proposed and this does not implement. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * mini: give the Lance port the same treatment as the Iceberg one The flag was registered but nothing else knew about it, so mini would start the server without reserving its port, waiting for it, or saying where it is. Adds it to the startup service list, the conflict resolver, the gRPC allocator's reserved set, the readiness wait, the stop reporting and the banner. The admin server still takes only the Iceberg port, because there is no Lance page for it to link to. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: stop deregister and repoint from deleting the dataset Deregistering preserves data by definition, and this did the opposite: the catalog entry is the dataset directory, so DeleteTable took the files with it. Registering over an existing name had the same shape, destroying the dataset the name used to hold. Found by driving the running server rather than the in-memory filer, where both looked like success because the table did stop being listed. Deregistering is now a state on the entry - the marker file hides it, and declaring or registering the name again brings it back. Repointing a name at another dataset is an UpdateTable against the version token, so neither dataset loses files. Drop is left alone; it is the operation that does remove data. The storage endpoint now falls back to the advertised -ip where the Iceberg derivation gives up. An Iceberg client brings its own s3.endpoint and advertising the wrong one hijacks it, but storage_options is the only place a Lance client learns where the store is, and without it object_store quietly talks to real AWS. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tables: refuse to create a table over one of another format Creating a table that already exists is idempotent, and that path returned the existing table without looking at its format. A Lance declare over an Iceberg table answered 200 and handed back a directory Iceberg owns, so the client would write its dataset on top. The view check immediately above it already guards the same class of collision. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tables: let a table bucket hold a format other than Iceberg The S3 door validated every object written into a table bucket against Iceberg's file layout, so a Lance client could not write its dataset at all: it got 403 on data/*.lance, on _versions/, and on the _transactions/ directory it turned out to write as well. Table buckets were only neutral containers by intention; in practice they were Iceberg-shaped and enforced as such. The allowed set is now the union of what the supported formats write, because the validator runs where the table's format is not in hand. Underscore-prefixed directories are treated as belonging to the format, since enumerating them means guessing at the next one - _transactions is exactly the one this missed - and their contents are checked only for traversal. Iceberg writes none of them, so it loses nothing. Marker files at the table root are admitted too, which the namespace/table/dir/file shape had rejected as too shallow. Describe also honours the request-body spellings of with_table_uri, load_detailed_metadata and check_declared. The spec puts them in the query string, but real clients send both. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: record what the implementation found The table bucket being an Iceberg-shaped container, enforced at the S3 door, was the premise this design never questioned and the one that had to change before anything worked end to end. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * iceberg: prove the data loss the foreign-format guard prevents The guard landed with a unit test for the predicate and nothing showing what it saves. These seed what the Lance namespace's Iceberg REST adapter actually leaves behind - an Iceberg table with a placeholder schema and table_type=lance whose directory holds a Lance dataset - and assert both halves: orphan collection does flag the dataset's fragments, because the Iceberg metadata beside them references nothing, and the scan never reaches the table. An ordinary Iceberg table in the same shape is still scanned, so the guard is not just skipping everything. Confirmed against a running gateway first: our Iceberg catalog accepts the adapter's registration, and a real Lance client then writes a dataset into that table's location. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tablestest: make the in-memory filer safe to race against Two gaps that only matter once a test drives concurrent writers, which is what an exclusive create has to be tested with: the entry map had no lock, and CreateEntry ignored O_EXCL entirely, so both writers of the same name would have won and the test would have passed while proving nothing. The BeforeUpdate hook runs before the lock is taken. Its whole purpose is to land a competing write in a handler's read-to-write window, and that write needs the lock the hook would otherwise be holding. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: make the namespace an external manifest store Lance commits a version by writing _versions/{v}.manifest with put-if-not-exists. The S3 layer in front of this same filer evaluates If-None-Match by looking the entry up and then writing without a precondition, so two writers can both pass the check and one commit is lost. The filer itself has the primitive: CreateEntry with o_excl. Adds the four version operations a Lance client actually calls - create, list, describe and batch-delete - recording one entry per version under _lance_versions/, and advertises managed_versioning so the client routes its commits here. Reserving a version is the exclusive create, so exactly one of several racing writers wins and the rest rebase. Off by default, behind -lance.managedVersioning. Turning it on moves where a table's version history lives, and a reader that does not come through this namespace no longer sees all of it; that is the operator's call, not a default. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: record what managed versioning does and does not reach The first commit through a namespace-backed store works and is recorded the way the protocol specifies. Later commits do not, because lance 4.0.0 refuses put_if_exists on that path in its own code, so the feature is capped upstream rather than here. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * test: integration tests for the Lance namespace Everything this surface got wrong so far - a deregister that deleted the dataset, an S3 door that refused every Lance file, a version reservation that could not actually be exclusive - passed against an in-memory filer first. So these run against a live gateway, and where the claim is about data they check storage rather than visibility. Five Go tests on the shared harness: namespace and table lifecycle including that deregister keeps the bytes and drop removes them, that a Lance client cannot resolve or declare over an Iceberg table, that a Lance dataset's files get past the table-bucket layout guard while junk still does not, and that eight writers racing for one version produce exactly one winner. One Docker-gated test drives the real Lance client, which is the only way to check that the location and storage_options the namespace vends are between them enough to write and read a dataset. It overrides the endpoint with the container's view of the same gateway, because the shared harness binds a wildcard address and so vends none. The harness gains a Lance port and turns managed versioning on; the flag touches nothing outside that surface. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tables: a directory with no namespace metadata is a missing namespace Three callers resolved a namespace by reading its metadata attribute and each tested only for a missing entry, so a directory that carried no metadata came back as an internal error saying "attribute not found". Creating a table under a namespace that does not exist answered 500. Collapses the three copies into one helper that reports both conditions as absent, which is what they are: a directory without namespace metadata is not a namespace. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * iceberg: stop reporting storage-layer refusals as server faults writeManagerError recognised a missing table bucket and sent everything else to 500, so a missing namespace, a duplicate name and a commit conflict all reached the client as InternalServerError with nothing to act on. Creating a table in a namespace that does not exist is the case that turned up: 500 where the spec wants 404 NoSuchNamespaceException. Maps the storage error types onto the exception names this package already uses, and keeps the existing bucket message, which explains how to select a table bucket. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * iceberg: skip a foreign-format table by name, not by failing to parse it A table the namespace created as LANCE carries no Iceberg metadata, so the worker skipped it only because the parse failed, and logged that as damaged metadata. The catalog records the format on the entry and this never read it. Reading it turns an accident into a decision, and separates a mixed catalog from a corrupt one in the logs. The property check beside it still covers the other shape: a real Iceberg table wearing table_type=lance, which is what the Lance namespace's Iceberg REST adapter writes. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: answer whether a Lance table needs maintenance It does, and index optimization has no Iceberg equivalent: rows written after an index was built are not covered by it, so a vector search quietly misses them. None of the three jobs can run in the Go worker, and there is no useful subset, because deciding what an old version still references means parsing Lance manifests. Version cleanup at least has an answer that needs nothing from us - Lance can enable it on the dataset itself. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: the Lance maintenance worker is a plugin worker, in Rust Framing it as a sidecar was wrong. plugin.proto already defines a language-agnostic gRPC contract for external maintenance workers, and "weed worker -admin=..." is the Go reference implementation of it from outside the admin process. seaweed-volume already compiles protos out of weed/pb with tonic_build, so a Lance worker is that build plus plugin.proto and the lance crate. Scheduling, retries, dedupe, progress and the admin settings page all come from the protocol: a worker that answers RequestConfigSchema with a descriptor gets its configuration form rendered without a line of Go. The data plane is the part that genuinely does need a process answering HTTP, and this had the two conflated. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * seaweed-worker: Rust plugin worker workspace, with Lance as the first one plugin.proto is language-agnostic and the Rust toolchain was already in the tree, so a Lance maintenance worker needs no new integration surface: core is the contract and nothing else, and a worker crate beside it supplies handlers and a binary. A second worker is a new member here rather than a fork of the protocol, which is why this is seaweed-worker and not seaweed-lance-worker. Verified against a running admin: it connects, is accepted, and admin prefetches descriptors for lance_compact, lance_optimize_indices and lance_cleanup_versions, so their settings pages render from the Rust side without a line of Go. The stream stays up across heartbeats. The job bodies are stubs that report failure. Doing the work means adding the lance crate and opening the dataset, and claiming success before that would be worse than saying so. Two things running it caught that reading the proto did not: the admin address has to be converted to the gRPC port the way pb.ServerToGrpcAddress does, or the dial fails as an h2 frame error; and the generated field names differ from the Go ones in several places, so JobCompleted carries success rather than a state enum. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: implement compaction Detection lists tables from the namespace, opens each one, and proposes a job for any with more fragments than the policy allows; opening a dataset reads its manifest and not its data, so a sweep stays cheap. Execution re-resolves the table rather than trusting what detection saw - it may have been repointed, and the vended credentials expire - then compacts and reports the fragment counts either side. Verified against a live gateway: a twelve-fragment dataset became one fragment with all twelve rows intact. The test drives the handler directly and skips unless WEED_LANCE_NAMESPACE names a namespace, the way the Go integration tests skip without Docker. Running it turned up a gap the design had not: a gateway without STS vends no credentials at all, so the worker could not open anything and detection quietly proposed nothing. --access-key/--secret-key are the fallback, and whatever the namespace vends still wins over them. Two API assumptions did not survive contact either. Datasets open through DatasetBuilder::with_storage_options, not ReadParams, and lance 10's ObjectStoreParams has no storage_options field at all. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: implement index optimization and version cleanup Index optimization is the job with no Iceberg equivalent: rows appended after an index was built are invisible to a search of it until this runs. Detection reads num_unindexed_rows from each index's statistics and proposes a table once more rows sit outside its indices than the budget allows; a table with no indices is skipped, which is different from one whose indices have fallen behind. Cleanup applies a retention window, refusing rather than silently dropping a tagged version, and leaving unverified files alone because they may belong to a commit still in flight. Both verified against a live gateway: 512 uncovered rows became 0, and a fourteen-version table lost its old ones. Each test now seeds what it needs, including building an IVF_PQ index and appending rows outside it. The first version of these depended on state a script had left, so the second run found the work already done and asserted nothing - a test that passes by doing nothing is worse than no test. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: answer an empty catalog with an empty list, not null ListAllTables built its result from a nil slice, so a namespace holding no tables answered {"tables":null} on a field the spec marks required. A generated client may decode that differently from an empty list. Found running the namespace on a dev box, where the catalog was empty. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: give Lance maintenance its own scheduler lane Lane assignment is a hardcoded map, so the three lance_* job types fell through to the default lane. That lane serialises its work under the cluster admin lock because volume management shares global state, which would queue a table's compaction behind volume balancing for no reason - Iceberg has its own lock-free lane for exactly this. Adds the lane, maps the three job types to it, and puts it in the sidebar beside Iceberg and Lifecycle. The lane routes were already generic, so only the nav was hand-written. The lane-coverage test spelled out the three known lanes, so a fourth failed it. It now checks against AllLanes(), which is the property it was reaching for and does not need editing next time. Found by connecting the Rust worker to a real admin: it registered fine and its job types were known, but they were filed under "default" and had no page. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: log what detection saw "Detection proposed nothing" and "the worker could not read the table" look identical from the admin side, and the second is what a missing credential produces. One line per table separates them. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: fix a leaked heartbeat and a silent reconnect loop spawn_heartbeat returned a handle to an empty task rather than the ticker it had just spawned, so aborting it aborted nothing and every reconnect left another heartbeat running against a dead channel. A stream that admin closes cleanly is not an error, but reconnecting in silence hides why. Two workers sharing an id evict each other forever and the log shows nothing but a login every five seconds - which is exactly how this presented on a dev box, and it took a look at the admin's own log to see it. The message now names the id to check. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: a namespace cannot be created without its parent Storage keeps a namespace's parts flattened, so creating "a.b" with no "a" was accepted and left an intermediate that only existed inside a name. Listing derives child names by slicing those parts, so it reported "a", while describe and exists on "a" both answered 404 - a client walking the tree got a 404 on something the listing had just handed it. The spec asks for NamespaceNotFound when the parent is missing, which is also what keeps listing and describe telling the same story. Namespaces created through the S3 Tables API still bypass this, so listing keeps deriving intermediates rather than hiding whatever is already there. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: say why a non-Iceberg table shows no schema The table pages read Iceberg metadata for schema and snapshots, and a Lance table has none, so both panels rendered "No schema available" - which reads as an empty table rather than a table this page cannot describe. The dataset behind the one that prompted this holds 1024 rows. The format is already on the entry and shown two rows above, so the empty states now use it: the catalog records where a LANCE table lives, not what is in it. Reading the schema for real needs Lance format code, which is the same wall as the data plane. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * seaweed-worker: run rustfmt over the workspace Committed the crates unformatted, so `cargo fmt --all --check` failed on files nothing had touched since. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * plugin: let a worker report what it saw about an object Admin cannot read a Lance table: it knows where the dataset lives and nothing else, so the details page had a location and two empty panels. The worker already opens every dataset during detection to decide whether it needs compacting, so it knows the schema, the row count and the fragment count at that moment. It just had no way to say so. Add a WorkerObservations body to the worker stream. Admin caches the last observation per object and serves it back, timestamped, for display; nothing schedules from it. The Lance compaction sweep reports what it opened, and the S3 Tables details page fills its schema panel from the cache when it has no metadata of its own, badged with when the worker looked and which worker it was. Nothing about this is Lance-specific past the reporting side, which is the point: any format admin cannot parse can describe itself the same way. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: record the observation channel Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * plugin: ask a worker for sample rows of a table admin cannot read Browse Data reads an Iceberg table's Parquet files directly, so it shows real rows. For a Lance table it showed "Table has no Iceberg metadata" and an empty grid, because there is no Go Lance reader and never will be one worth maintaining. The worker has the reader. Add RequestObjectPreview / ObjectPreviewResponse to the stream, mirroring the config-schema round trip that already exists, and give the Rust worker a PreviewProvider that scans the dataset and formats the rows with Arrow's own formatter, so a vector column reads as a vector. Admin picks the worker from the observation store: whichever one last described this table is the one that can read it. Unlike an observation the rows are not cached. They are the table's data rather than a description of it, and a copy sitting in admin would be both stale and nobody's business. The page fetches on load, bounded at 200 rows and a 15 second round trip, and drops the snapshot and data-file panels that only mean something for Iceberg. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: record the preview channel Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * test: disable the lance listener when two gateways share a host * test: keep AllocatePorts away from the lance default port * s3tables: let a table bucket declare the format it holds A bucket is a catalog, and a catalog serves one protocol. Format was recorded per table, so nothing could answer "where do I point a client at this bucket" without opening a table first, and an empty bucket had no answer at all. CreateTableBucket takes an optional format, stored with the rest of the bucket metadata and returned by Get and List. Empty means ICEBERG, which is what AWS S3 Tables serves and therefore what an SDK that has never heard of the field means. CreateTable refuses a table of another format, and CreateView refuses outright in a bucket that is not Iceberg, since a view is Iceberg metadata. Buckets that already exist carry no declaration and keep accepting anything, so nothing is migrated and nothing that worked stops working. The Lance namespace declares LANCE for the buckets it creates, which is what stops one of them being described to a client as an Iceberg catalog. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: take the Lance port the way it takes the Iceberg one The UI cannot name the endpoint that serves a Lance bucket without it, and every format-aware page below needs to. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: show which format a table bucket holds The bucket list printed an Iceberg endpoint for every bucket, including ones holding Lance datasets, where that endpoint serves nothing. It was the most visible place the UI assumed one format. The list gains a Format column and its endpoint column follows the bucket's declaration. The banner names both endpoints rather than asserting everything is Iceberg, and says so only for the servers that are actually running. Create Bucket picks a format with two cards rather than a dropdown, since what matters is not the name but which clients can read the result, and the endpoint under them updates as you choose so the operator leaves the modal knowing where to point one. A bucket from before the declaration existed shows "unset" in an outline badge, explained on hover. It is a fact about the bucket's age, not a fault, so nothing nags about it. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: carry the bucket's format into the pages inside it Namespaces and tables are reached through a bucket, so both now say which catalog they belong to rather than making you go back up to find out. The tables list gains a Format column and a Rows column filled from what a worker last observed, since for a format admin cannot read that is the only row count there is; a table nothing has looked at shows a dash, not a zero. Create Table stops offering a choice the bucket has already made: in a declared bucket the format is fixed and says why, and only an undeclared one still offers both. Before this the select had exactly one option, hardcoded, which made a Lance table impossible to create from the UI at all. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: let the table page speak the table's own format Partitions and Snapshot History are Iceberg's shape. Rendering them empty for a Lance table reads as a fault; a Lance table has neither, and says so by not showing them. In their place is a Versions panel, which is what that format calls its history, carrying the worker's timestamp so it is clear the numbers are a cached look rather than something read live. The breadcrumb carries the format badge, so the page names what it is looking at before you read a panel and wonder why it is empty. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: show how to connect to either catalog, and group the two format workers The client examples on the buckets page were Iceberg's alone, so the one thing an operator wants after creating a Lance bucket - what to type to reach it - was not written down anywhere in the UI. Both formats now get a pair of snippets, and only for a server that is running. In the Workers menu, Iceberg moves below Lifecycle so it sits next to Lance: the two table-format workers together, the two cluster-wide ones above them. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * shell: create a table bucket of either format s3tables.bucket -create takes -format, so a Lance bucket can be made without going through the UI. The integration harness passes it too: its Lance tests were creating Iceberg buckets and getting away with it only because nothing checked. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: record that a bucket declares its format Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: drop managed versioning; the store already orders commits The namespace offered itself as an external manifest store, so that a commit could reserve a version through a real put-if-not-exists. That was designed around a gateway that no longer exists: If-None-Match: * is reduced to a filer WriteCondition and evaluated at the object's owner under its per-path lock, or under the object write lock on the fallback path. Sixteen writers racing one fresh key get a single 200 and fifteen 412s, every time. Lance needs nothing else. commit_handler_from_url hands every s3:// dataset a ConditionalPutCommitHandler, which puts with PutMode::Create, which object_store sends as If-None-Match: *. So the feature solved a problem this store does not have, while moving a table's version history out of the dataset and into the catalog - and lance could not use it past the first commit anyway, since its own namespace-backed store answers "put_if_not_exists is not supported" to the second. The version operations answer Unsupported with the rest, managed_versioning is false, and the flag is gone. In place of the reserve-once test there is one that races eight writers at the manifest key through S3, which is the path a commit actually takes. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: honour the version floor, the slot limits, and a shutdown Five findings from review, all of them things the worker claimed to do and did not. The version floor was checked when a cleanup job was proposed and ignored when it ran, so a table whose versions had aged past the retention window in between could be taken below the count the operator asked to keep. Execution now computes the floor itself and passes it as before_version; CleanupPolicy ANDs its clauses, so a version has to be both too old and below the floor to go. Both settings are clamped to the range the form offers, since Duration::hours panics on a large enough value and a negative min-versions wraps to a huge usize. Admin's shutdown was answered by returning from the stream, which the reconnect loop read as a healthy close and logged straight back in: the worker could not be stopped. serve_once now says which of the two happened. The advertised concurrency limits bounded nothing - every request spawned a task - and the heartbeat reported zero slots in use whatever was running. Both now go through semaphores sized from the limits, with the permits held for the life of the request and reported in the heartbeat. A namespace call had no timeout, so a gateway that accepted the connection and went quiet held a detection slot forever. And one table whose stats could not be read failed the whole sweep, losing the proposals for every table already scanned; it is now skipped and warned about, like a table that cannot be opened. The tests drove one shared catalog concurrently, which is why one of them asserted "no proposals at all" and passed by luck. They now take a lock and judge only their own tables. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: fix the review findings on the format-aware pages The endpoint hint in Create Bucket built its HTML by concatenating the bucket name the operator is typing, so a name like <img onerror=...> ran in the admin origin as they typed it. It is built from DOM nodes now. A preview reply looked its channel up under the lock and then sent outside it, which Shutdown can close in between: a Gosched in that gap panics with "send on closed channel" every time. The send now happens under the lock. Observations were looked up by path alone, so a table dropped and remade in another format at the same path was described by the observation left behind. Lookups now have to agree on the format. Also: the Lance namespace caps a request body rather than reading whatever arrives; the details action no longer says "Iceberg" over a Lance table; mini stops advertising a catalog port when it is not running S3; a format whose server this cluster does not run cannot be picked in the modal or accepted by the API, since a bucket nothing can reach is not worth creating; and the unused catalogPortFor helper is gone. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: let the control stream use mTLS The channel was hardcoded to http://, so off loopback the stream carried preview rows and execution commands in the clear - and a cluster with grpc TLS turned on would refuse the worker outright. --tls-ca, --tls-cert and --tls-key take the same certificates the Go worker reads from the [grpc.worker] section of security.toml, and must be given together: a CA on its own would quietly mean one-way TLS, which a mutual setup rejects anyway. Without them the stream stays plaintext, which is what the Go worker also does when nothing is configured. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: answer null properties rather than an empty map The catalog does not keep a table's properties. Declare echoed the request's back and describe answered {}, both of which claim they were stored and are empty. Null says the catalog does not keep them, which is what the spec distinguishes and what is true here. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: test the slot accounting The heartbeat reporting and the waiting are the two things the semaphores are for, and neither is observable from outside without catching a sweep mid-flight. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * test: fix the mixed-format catalog test, and name the binary it drives The integration suite passed locally and failed in CI on TestLanceRefusesIcebergTables. Both were right: CI builds the binary first, my tree had one from the day before, so locally the test drove a gateway with no format enforcement at all. The test itself no longer holds as written. It made a bucket, put an Iceberg table in it, and checked the Lance surface hid it - but a bucket that declares LANCE now refuses the Iceberg table outright. The invariant still matters from the other side, so it starts from an Iceberg bucket instead: Lance must not describe or list a table whose format it does not serve, and must refuse to declare one beside it. The harness now prints which weed binary it is about to run and when that was built. `make test` rebuilds first; a plain `go test` will happily drive a weeks-old binary and report a pass for code it never ran, which is exactly what happened here. Also make the row-limit conversion in the preview request explicitly bounded: CodeQL flagged the int-to-int32 conversion, and clamping by reassignment beforehand is not a form it recognises. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: prove concurrent commits are kept, and preselect the only format on offer Two more from review. The commit test asserted that exactly one writer wins the conditional PUT, which is the mechanism, not the claim. The claim is that nothing is lost: the losers see the conflict, rebase and commit again. So there is now a test that has eight writers append to one dataset at once and counts the rows afterwards - all eight batches survive. That is also the sequence managed versioning could not finish, since its store refuses the second commit outright. And when Iceberg's endpoint is not running, the format picker offered two options with neither selected, so Create Bucket submitted no format at all, fell back to ICEBERG, and was refused by the guard added last round. Lance is preselected when it is the only format this cluster serves. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * Clamp the remaining worker settings, and bootstrap buckets in a served format Compaction and index optimization read their thresholds and cast straight to usize and u64, so a negative arrives as an enormous number and turns the threshold into "never": compaction and reindexing both go quiet with nothing to say. The cleanup job was fixed last round; these are the same bug. Clamped to the values that stay meaningful rather than to what the form offers - zero uncovered rows is a real setting, meaning reindex as soon as anything is not covered, so the floor there is zero and not the form's thousand. mini pre-creates the buckets named by -tableBucket, and did so without a format, which now means Iceberg. Started with the Iceberg endpoint off and the Lance one on, that left buckets nothing could reach and which refused every Lance table. It takes the format from the endpoint that is actually running, and creates nothing when neither is. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3: allow-unordered is a listing parameter, not an unimplemented subresource The guard that stops a bucket GET with an unknown subresource from being answered with a listing does not know about allow-unordered, so it answers 501 NotImplemented - to a parameter the listing handlers already read and already validate against delimiter. This is why test_bucket_list_unordered and test_bucket_listv2_unordered fail in the Ceph s3-tests suite. They fail on master too; this is not a Lance change and can be taken on its own. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm	2026-08-19 22:59:56 -07:00
Chris Lu	814ee75af4	s3: allow-unordered is a listing parameter, not an unimplemented subresource (#10846) ... The guard that stops a bucket GET with an unknown subresource from being answered with a listing does not know about allow-unordered, so it answers 501 NotImplemented - to a parameter the listing handlers already read and already validate against delimiter. This is why test_bucket_list_unordered and test_bucket_listv2_unordered fail in the Ceph s3-tests suite. They fail on master too; this is not a Lance change and can be taken on its own. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm	2026-08-19 20:09:23 -07:00
Chris Lu	e5edd8be3c	s3: place multipart part chunks by the destination object's storage rule (#10845) ... Multipart parts stage under /buckets/<bucket>/.uploads/<id>/, so the filer resolved filer.conf storage rules against that path when the gateway assigned volumes for them. A rule scoped to a key prefix - fs.configure -locationPrefix=/buckets/b/data/ -ttl=30d - then matched a small object but not the parts of a large one, so an object whose entry carried the rule's TTL had its bytes spread over TTL-less volumes. Assign part chunks against the destination object's filer path instead, the way the x-seaweedfs-destination header made the filer resolve it before the S3 write path moved off the filer proxy. Covers PutObjectPart and both UploadPartCopy paths. The part entry itself is still written under .uploads, so a read-only rule there still rejects it. The lifecycle XML Expiration.Days TTL keeps passing 0 for parts: that rule targets the user-visible object key and would start its clock before CompleteMultipartUpload.	2026-08-19 20:07:57 -07:00
Chris Lu	abd61de52c	S3: stamp the gateway's own uid/gid on PutObject and copy entries (#10844) ... * fix(s3): stamp the gateway's own ids on single-shot PutObject entries putToFiler builds the entry in the gateway now instead of proxying a PUT to the filer, and it hardcoded Uid/Gid 0 while every sibling write path stamps filer_pb.OS_UID/OS_GID. On a non-root deployment that leaves single-shot PUTs and multipart parts owned by root while directories and completed multipart objects keep the real ids, so a mount reader can list the tree but gets EACCES on every open once objects are not world-readable. * fix(s3): stamp mode and ownership on copy destinations CopyObject and UploadPartCopy build the destination attributes themselves and then assign them over the entry filer_pb.MkFile just stamped, so the copy landed with mode 0000 and uid/gid 0 - unreadable on a mount even by the filer's own user. Build the destination with the same mode PutObject resolves for the request and the gateway's own ids.	2026-08-19 18:57:37 -07:00
Chris Lu	1354b58675	s3: stop unrouted bucket subresources from being answered with a listing (#10814) ... * s3: answer GetBucketReplication, GetBucketWebsite and GetBucketNotificationConfiguration None of the three had a route, so they reached the unconstrained ListObjectsV1 catch-all and a client asking for a bucket's replication config got 200 and a <ListBucketResult> back. Replication and website report their configuration as absent the way AWS does; notification returns the empty configuration AWS returns for a bucket with no events wired up. * s3: stop an unrouted bucket subresource from being answered with a listing ListObjectsV1 is the catch-all GET on a bucket, so every subresource without a route of its own - ?torrent today, whatever AWS adds next - came back 200 with a <ListBucketResult>. A client that asked for a configuration and got a listing either fails its XML decode in a way that reads like corruption, or worse, tolerantly parses it. Refuse the request instead. The allow-list is the ListObjects parameters rather than the subresources, so a new one fails closed. Presigned URLs sign their credentials into the query string, so X-Amz-* and the SigV2 trio have to stay listable.	2026-08-18 16:03:08 -07:00
Chris Lu	5d5ea63b3f	Fix what the Go 1.26 language bump breaks (#10794) ... * worker: log the balance move stage through a constant format string Go 1.26's printf analyzer now follows printf wrappers reached through an interface, so passing the stage straight to Logger.Info is a vet failure. * s3api: bracket the IPv6 host in the signature test URL A bare IPv6 literal is legal in a Host header but never in a URL. Go 1.26 stopped parsing it leniently, so carry the two forms separately and set r.Host to the value the client would actually have signed. * mini: bracket IPv6 addresses in the readiness probe URLs An IPv6-only host hands mini a bare literal, and %s:%d pasted it into a URL unbracketed. Under Go 1.26 that URL no longer parses, so waiting for the admin server never succeeds and mini refuses to start.	2026-08-17 14:46:25 -07:00
Chris Lu	7522e17b6d	iceberg: vend table-scoped credentials to clients that ask for delegation (#10777) ... * iceberg: vend table-scoped credentials to clients that ask for delegation The catalog recognised X-Iceberg-Access-Delegation: vended-credentials and then deliberately said nothing, because it had nothing to vend: it withheld even the S3 endpoint so the client would keep the credentials it was configured with. That left every engine expecting the catalog to hand out access - Snowflake, Databricks, Trino with vending, any multi-tenant setup - needing static S3 keys distributed out of band. Mint an STS session per request instead, scoped by a session policy to the table's own prefix plus the bucket listing needed to resolve it, and return it in the load response config and storage-credentials. The role to assume is named by -s3.iceberg.credentialRole; its trust policy is what decides whether a caller may assume it, and vending stays off until it is set. A failed mint falls back to the old silence rather than handing back an endpoint the client cannot sign for. * iceberg: keep vended credentials inside the table prefix Review follow-ups on credential vending: Listing was granted on the bucket ARN with no condition, so a credential vended for one table could enumerate every other table's object names. Constrain s3:prefix to the table's own prefix, which the S3 gateway already populates for list requests. A table location carrying * or ? would have gone into the policy's resource pattern unescaped and widened the session to sibling prefixes. Refuse to vend for such a location rather than escaping it; nothing the catalog generates contains those characters. DurationSeconds skipped the 900..43200 bounds the other assume-role paths enforce, so -s3.iceberg.credentialDurationSeconds could ask for a session outside them. The check is now shared by all three entry points. * iceberg: return the vended credentials from buildFileIOConfig itself buildStorageConfig was a second name for what buildFileIOConfig already did; it now returns the storage credentials alongside the properties, and callers that only want the properties drop them. * iceberg: split the vended bucket grants, and refuse a whole-bucket scope The prefix condition sat on a statement that also granted GetBucketLocation and ListBucketMultipartUploads, neither of which carries an s3:prefix to satisfy it, so both were denied for every vended credential. GetBucketLocation moves to its own unconditioned statement. ListBucketMultipartUploads is dropped: Iceberg writers complete and abort by upload id, and granting it either leaks in-flight keys bucket-wide or breaks on the same missing prefix. A table whose location has no prefix - one registered at the bucket root - would have been vended read and write over every other table in the bucket. Refuse, the way a location with wildcards is refused.	2026-08-16 12:57:12 -07:00
Chris Lu	ec37ef5aaa	iceberg: add view rename, scan-report and snapshots=refs to the catalog (#10776) ... * iceberg: add view rename, scan-report and snapshots=refs to the catalog Three gaps against the REST spec that clients hit in normal use: Views had no rename, though tables did and views are stored the same way, so the move is the same catalog-only pointer move. Tables and views share a namespace directory, so both renames now refuse the other kind instead of moving it. Engines POST a scan or commit report after planning; a 404 there turns into an error line per query. Accept the report and discard it - the catalog keeps no metrics store. LoadTable ignored ?snapshots=refs and always returned the whole snapshot history, which is what clients use the parameter to avoid on long-lived tables. * iceberg: authorize view rename against the view ARN, tighten the metrics endpoint Review follow-ups: The shared rename checked the source against a table ARN whatever the kind, so a policy scoped to a view's own ARN never matched and one written for a table ARN was evaluated for a view. The entry kind now carries the ARN builder. The metrics endpoint truncated a report at 1 MiB and then failed to parse it, answering 400 for a query that had actually succeeded. Read one byte past the limit to tell "fits" from "cut short", and discard an oversized report instead of rejecting it. Empty bodies and reports without a report-type are now rejected, which the REST schema requires. ?snapshots= is defined for LoadTable, so it no longer filters what CreateTable echoes back.	2026-08-16 12:56:45 -07:00
Chris Lu	d044839ab2	iceberg: make a table commit a compare-and-swap (#10775) ... * iceberg: make a table commit a compare-and-swap The catalog validated the caller's version token, ran its authorization checks, and only then wrote the new metadata xattr. Two engines committing against the same base both passed that check and both wrote, so the second silently dropped the first one's snapshot. Both also derive the same v{N}.metadata.json name and the file write overwrote, leaving the surviving pointer aimed at the loser's metadata - and the loser's conflict cleanup then deleted the winner's file. Write the metadata file with an exclusive create and update the xattr conditionally on the bytes the handler read, the way the maintenance worker already commits. A writer that lost the race re-reads and retries, and reports 409 CommitFailedException once out of attempts. * iceberg: stage a commit under a unique name when the versioned one is taken Two follow-ups from review of the commit compare-and-swap: Refusing to overwrite v{N}.metadata.json also refused to get past a file left behind by a commit that died between staging and updating the pointer. Every later commit derived the same name, saw the collision, and reported a conflict, so the table stayed uncommittable until an orphan sweep removed the file. Stage under v{N}-{uuid} instead: neither writer's file is overwritten and the catalog pointer still decides who won, which is how the maintenance worker has always staged its own metadata. metadataVersionFromLocation learned to read the version back out of that name. The conditional update guarded only the metadata attribute while the write replaced the whole entry, so a policy or tag written in the same window was silently reverted. Guard every catalog attribute, which turns that into a conflict the caller retries on fresh state. * iceberg: give saveMetadataFile the exclusive flag instead of a second name saveNewMetadataFile, saveMetadataBlobExclusive and uniqueMetadataFileName were three new names around one existing helper. The flag now rides on saveMetadataFile and saveMetadataBlob, and the unique-name construction sits where it is used. * iceberg: reuse the filer CAS helpers #10773 added, and stage transactions exclusively #10773 landed mutateEntryExtended, which already writes an entry back under a whole-entry precondition and retries. Drop the helper this branch added and route the table commit through it: the check that the metadata is still the one this request read now lives in the mutation, where it sees current state. The policy the request was authorized against is asserted too, so an administrator restricting it mid-commit sends the caller back through authorization instead of having a stale decision applied. Bucket and namespace policies live on other entries and a single-entry precondition cannot cover them. Multi-table transactions stage their metadata exclusively for the same reason single-table commits do, and carry the name they landed on into the pointer flip.	2026-08-16 12:55:42 -07:00
Chris Lu	eef6f3d1e6	s3tables: add the maintenance configuration APIs (#10773) ... * s3tables: add the maintenance configuration APIs Stores the configuration verbatim as the wire shape under a new s3tables.maintenance extended attribute, so Get hands back what Put took and no translation layer can drift from the AWS model. Nothing reads the configuration yet. Put merges a single type into the stored map so configuring compaction does not drop snapshot management, and asserts the attribute's prior value so two concurrent Puts cannot silently clobber each other. * iceberg: apply the maintenance configuration in the worker The worker now reads the per-table and per-bucket maintenance configuration written by the control plane, so the wildcard plugin config is a default rather than the only setting a table can have. Table properties still win by default, since a table declaring its own layout is what every engine honours and the compactor has to agree with whoever writes the files. Clearing table_properties_override makes the maintenance configuration authoritative instead. Status is not part of that contest: a disabled type drops its operations and no property can re-enable them, so the operator's kill switch always holds. Manifest and delete-file rewrites have no AWS equivalent and ride with compaction. Detection reads both attributes from entries it already lists. * s3tables: report maintenance job status The worker records the outcome of each run in its own extended attribute, separate from the configuration so operator and worker writes do not contend, and GetTableMaintenanceJobStatus reads it back. Only the types a run touched are written, so a partial run cannot erase what an earlier one recorded. The reader fills in the rest: Disabled when the configuration switched a type off, Not_Yet_Run otherwise. Status is advisory, so a lost race is logged rather than failing a job whose work already committed. * s3tables: route the maintenance APIs over REST The five actions were only reachable by X-Amz-Target dispatch, which the AWS CLI and SDK do not use for this service. They address the operations by path, so the APIs were unreachable from any official client. * s3tables: fix the table bucket ARN field name GetTableBucketMaintenanceConfiguration emitted tableBucketArn where the wire field is tableBucketARN, as every other response in this package already spells it. Official SDK deserializers ignore the unknown key, so the required field came back unset. * s3tables: carry the compaction strategy through to the worker IcebergCompactionSettings modelled only targetFileSizeMB, so a request naming a strategy was accepted and then dropped on the way to storage. The worker now maps binpack and sort onto its own rewrite strategy and lets auto defer to the worker configuration. z-order is rejected rather than accepted and quietly binpacked. * s3tables: report bucket-level maintenance status GetTableMaintenanceJobStatus read only the table's configuration, so unreferenced file removal — which is configured on the bucket — reported Not_Yet_Run or a stale success after an operator disabled it. The merge helper now lives in this package and the worker shares it. * iceberg: delete orphans only after the non-current window AWS marks a file non-current once it has been unreferenced for unreferencedDays, then deletes it a further nonCurrentDays later. The cutoff was taken from unreferencedDays alone, so a 3/10 configuration hard-deleted on day three and threw away the ten day recovery window. remove_orphans deletes in one step rather than marking, so the cutoff is now the sum of the two. * s3tables: assert every attribute when rewriting an entry UpdateEntry writes the whole entry back from the snapshot the caller read, and its precondition only covers the keys the caller names. Both maintenance writers named one key, so a job status write could revert a maintenance configuration an operator had just disabled, turning an advisory write into a silent re-enable. Both now assert the entry's full attribute set, including the target key when absent so a concurrent create also fails the precondition. * s3tables: assert absent attributes when rewriting an entry The precondition covered the attributes present when the writer read the entry, so an attribute created between that read and the write was absent from it. A first-time PutTableMaintenanceConfiguration disabling a type therefore lands, passes the per-key checks, and is then deleted by the stale whole-entry write. Every attribute this package stores is now asserted, absent ones included. The metadata commit and planning index writers rewrite the same entries and had the same exposure, so both use the shared snapshot too. * iceberg: implement the auto compaction strategy auto was accepted, stored and read back, but left the worker on its own default, so a sorted table configured as auto was compacted with binpack. AWS defines auto as sorting tables that declare a sort order and bin-packing the rest. That needs the table metadata, so the choice is made where the rewrite plan is resolved: an unsorted table falls back to binpack rather than failing the way an explicit sort request does. * s3tables: validate the maintenance setting ranges PUT accepted zero, negative and oversized values for every numeric setting. The worker then ignores a non-positive value and saturates an oversized one, so the configuration read back was not the one that ran. AWS bounds all five to 1..2147483647, which is now enforced. The fields are pointers so an explicit zero is distinguishable from an omitted one and can be rejected rather than silently ignored. * s3tables: give every entry writer the same compare-and-swap updateExtendedAttribute asserted the entry's attributes, but the helpers behind the metadata, policy and tag handlers still wrote the whole entry unconditionally. Any of them could land on a stale snapshot and delete a maintenance configuration an operator had just written. They all share one read-modify-write loop now, so the precondition and the bounded retry apply wherever an entry is rewritten. * s3tables: move the maintenance configuration with a renamed table RenameTable carried the metadata, version, policy and tags to the new name but left the maintenance configuration and job status behind. A table with snapshot management disabled came back enabled under its new name, and the stale configuration stayed on the old name where a table created there would inherit it. The decoupled-delete cleanup left the same two attributes behind. * s3tables: accept every AWS partition in ARNs The route regexes and the ARN patterns both hardcoded arn:aws, so valid aws-cn and aws-us-gov ARNs never reached a handler. The router now shares the partition-tolerant prefix with the parser, and a generated ARN uses the partition its region belongs to so it parses back. * s3tables: generate ARNs in the region's partition The handler's own ARN generators still formatted arn:aws directly rather than going through the partition-aware builder, so a China or GovCloud deployment routed the request but then returned a commercial ARN and matched IAM policies against it. The round-trip test missed this because parsing accepts any partition, so it now asserts the prefix the region implies. * s3tables: complete the ARN partition table aws-iso-e, aws-iso-f and aws-eusc were missing, so eu-isoe-*, us-isof-* and eusc-* regions fell through to the commercial partition. * s3tables: do not let a rename swallow a concurrent maintenance write Rename copied the source attributes early and cleared the source at the end, so a Put landing in between missed the copy to the destination and was then deleted by the cleanup. It succeeded and vanished. The cleanup now clears the source only while it still holds exactly what was copied, and returns a conflict otherwise. Put checks the catalog identity inside the same conditional mutation, so it also cannot write to a name that a rename or delete has already soft-deleted.	2026-08-16 10:36:59 -07:00
Chris Lu	f66d6ffc4a	s3: option to disable bucket auto-creation on upload (#10759) ... * s3: add option to disable bucket auto-creation on upload * command: expose -autoCreateBucket in s3, filer, server, and mini * s3: apply the bucket auto-create policy to directory marker uploads * s3: validate the bucket name before the auto-create disabled check * s3: cover the disabled auto-create gate at all three upload entry points	2026-08-14 10:58:45 -07:00
Chris Lu	db5a086d04	read cold remote objects straight from the origin while caching (#10731) ... * refactor: extract remote mount resolution into shared helpers * refactor: share the adaptive remote cache wait policy * filer: stream cold remote reads from the origin while caching * s3: stream cold remote reads from the origin instead of 503 retries * test: cover the S3 origin stream-through path * remote mounts: match on path components and prefer the longest mount * fail short origin streams instead of silently truncating * s3: try the origin before failing a cold read on a local cache error * s3: gate origin streaming on the entry's resolved version * return the cache RPC's NotFound as a canonical status and classify it everywhere * filer: keep multipart-range cold reads on the retry path	2026-08-12 23:00:10 -07:00
Chris Lu	7c87d78ea2	s3: a key deleted after enabling versioning must leave the listing (#10684) ... * s3: a null object wins over a rescan when the latest-version pointer is absent The read path already resolves an absent pointer this way; the listing-path counterpart scanned .versions/ first and could surface an old version or delete marker over the current suspended-versioning null object. * s3: dedup a key against its .versions sibling in suspended buckets too A suspended bucket keeps its .versions directories, so a suspended-versioning null object and its .versions sibling emitted the same key twice. * s3: retract a null object from the listing when a delete marker shadows it Deleting a key whose null version predates versioning leaves the base-path entry in place and records the delete marker under <key>.versions. The listing appended the base-path entry and relied on the .versions sibling to replace it, but a delete-marker current version emitted nothing, so the deleted key stayed visible to ListObjects while GET and HEAD returned 404. * s3: keep a key's .versions sibling on the same page as the key When the page quota ran out between a base-path entry and its .versions directory, the page ended with the stale entry and the next page skipped the directory as a marker echo, so the replacement or retraction never happened. * s3: the null version is not latest when the .versions pointer names a newer one ListObjectVersions stamped IsLatest on every base-path null object, so a key deleted after enabling versioning reported IsLatest on both the delete marker and the null version. * s3: test listing after a pre-versioning null object is delete-marked * s3: find a key's earlier page entry by scan, not by adjacency A key such as k.bak sorts between k and k.versions, so the entry a .versions sibling replaces or retracts is not always the last one on the page. Scan back through the page for the key, and insert a late resolution in sorted position instead of at the end. * s3: settle trailing null objects by lookup when a page fills The quota can run out while keys still sit between a null object and its .versions sibling, and the sibling-adjacent page-boundary exception never fires for those. Track the trailing null objects whose sibling has not been ruled out and look each one up before declaring the page full; a retraction reopens the quota. * s3: do not resolve a .versions sibling its page has already moved past A page resuming from a marker inside the base key's extension region has already listed and settled the base null object on an earlier page, so resolving the .versions directory again re-emitted the key. * s3: test listing with keys between a null object and its .versions sibling * s3: pick the newer of the null object and the scanned versions Making the null object win outright whenever the pointer is absent misread multi-filer pointer lag: version files replicate ahead of the pointer, and a key overwritten or delete-marked after pre-versioning days would list its stale null again. The suspended-versioning write that legitimately makes the null current is also the newer entry, so mtime tells the two apart. * s3: a delete-marked null object no longer keeps its prefix alive The hidden-entries probe took any plain file as proof of a listable key, but a null object shadowed by its .versions sibling's delete marker is not one. Hold plain files pending until the sibling settles them either way. * s3: settle an evicted pending null instead of dropping it Nested keys like k, k!, k!! can hold more pending nulls than the cap. A silently evicted one could close the page unsettled, and the resume skip would then keep the stale entry for good. * s3: test deleted-prefix hiding and the pending-null cap * s3: cover the reported '!' intervening key with a live version * s3: an unstamped same-second version outranks the null object Second-resolution mtimes cannot order same-second writes, so the tie went to the stale null when the pointer lagged. The suspended write that makes a null current stamps the version it displaces before clearing the pointer, so the stamp is the authoritative signal and a tie without it goes to the version. * s3: a pointer-less versions listing still checks what replicated ListObjectVersions took a missing pointer as proof the null object is latest, but under pointer lag the sibling can already hold newer replicated versions or markers. Apply the same nullObjectWins rule as the listing recovery. * s3: a failed null-object settlement fails the listing Every getEntry error read as a missing sibling, so a transient filer error at a page boundary committed the unsettled null and the next page skipped its sibling for good. Only a definitive not-found means the null is live; other failures are retained on eviction and fail the request at page close. * s3: retract a CommonPrefix whose only backers were delete-marked nulls The directory probe settles this for the / delimiter, but any other delimiter derives prefixes from base-path keys directly, and a prefix built solely from null objects survived their delete markers. Count the unsettled null backers behind the newest prefix and retract it when the last one settles as a marker; a live resolution or any listable contributor confirms the prefix instead. * s3: test custom-delimiter prefix retraction * s3: an explicit signal marks the null object current, not the demotion stamp The NoncurrentSinceNs stamp survives promotion: delete the version that demoted another and the promoted one is current yet still stamped, so a lagging replica would resurrect the stale null. A suspended-versioning write now records Seaweed-X-Amz-Null-Version-Is-Latest on the .versions directory when it clears the pointer, every pointer update removes it, and the recovery paths trust the signal instead of the stamp. * s3: a filer failover retry rebuilds the listing page from scratch The failover wrapper reruns the callback on another filer after a transport error, and the partially built page, spent quota, and advanced marker leaked into the retry, which could then return a stale or duplicated page as success. * s3: only a prefix's own backers can debit it A delete marker for a version-only key (no base object) derived the same prefix as its neighbors and decremented backing it never contributed, retracting a prefix that a live null object still backed. Track backers by key so settlement is idempotent and only debits what was counted. * s3: test a version-only marker against a null-backed prefix * s3: a pointer recompute clears the null-current signal The routed finalize for delete markers, COPY, and multipart rewrites the .versions pointer through RECOMPUTE_LATEST, which left a suspended-era null-current signal in place. Version files never carry the signal, so mapping it in CopyExtended deletes it whenever the pointer recomputes. * s3: the pointer outranks the null-current signal in the versions listing The signal check guarded the pointer check, so a stale signal a recompute had not cleared yet would have let the null claim IsLatest alongside the pointed-at version.	2026-08-10 11:04:06 -07:00
Chris Lu	c8cc56be91	iceberg: route unprefixed requests to the first table bucket (#10675)	2026-08-09 22:20:13 -07:00
Chris Lu	a2ff9cca27	master: let VolumeList ask for the volumes it wants (#10674) ... * master: let VolumeList ask for the volumes it wants The request carried nothing, so every caller was answered with the whole cluster. A dashboard opening one volume's page, or a capacity probe adding up one bucket, was served all 800k of them and threw away the rest -- and the master built every one of those messages first. The topology, its disks and their counters are still reported in full: a caller reading free space or replica placement needs the cluster whichever volumes it asked about. Only what is listed under a disk is selected, ec shards included. An empty collection and a zero volume id take everything, the way volume.list already reads its own -collectionPattern and -volumeId, so a caller that forgets to narrow is answered too much rather than answered wrongly. That leaves the default collection unnameable, since it is the one the empty string names, so it gets a field of its own. An older client sends none of it and is answered exactly as before. * admin: ask the master for the volume the page is showing A volume's detail page was pulling every volume in the cluster to find one and its replicas, and discarding the rest. * admin: ask the master for the ec volume the page is showing Same as the volume detail page: one volume's shards were found by pulling every ec shard in the cluster. * s3: ask the master for the bucket's own collection The SOSAPI capacity probe summed one collection's volumes out of a listing of every volume in the cluster. Cluster capacity still comes out the same: it is read from the disk counters, which a filtered listing reports in full. * topology: read the disk usage counters atomically They are written with atomic.AddInt64 from heartbeats but were read plainly by the two listings and by FreeSpace, and the map they sit in was iterated without the lock its neighbour takes. Under -race a listing concurrent with a heartbeat trips on both.	2026-08-09 21:59:42 -07:00
Chris Lu	567052bfb6	s3: take bucket sizes from the master's summary (#10664) ... * pb: ask the master what each collection holds Callers tracking usage were sent every volume in the cluster to add up themselves, which is the master's largest single allocation. * topology: summarise what each collection holds One pass over the topology, allocating per collection rather than per volume. Regular volumes count once each for logical totals and once per replica for physical, taken from the lookup index, which is already keyed by volume and so needs no set of seen ids. Ec shards are node-local so their sizes sum, while the file and delete counts describe the volume and resolve once every holder has been seen. Replicas of one volume disagree while a write is landing or a heartbeat is late. Walking a full listing took whichever replica the map iteration reached first, so the answer moved between runs; this takes the largest, which is stable and never reports usage below what some replica already holds. * s3: take bucket sizes from the master's summary The bucket size metrics pulled the whole volume list once a minute and added it up, which cost the master 184.6MB of allocation and 17.8MB on the wire for six numbers per collection. VolumeList over 550k volumes 184.6 MB allocated, 17.8 MB on the wire CollectionStatistics 176 bytes allocated, 47 bytes on the wire The aggregation moves to the master with it, so the cases the removed tests covered are now asserted against it directly. * topology: count the replica holding the most live data Quotas are enforced on size less deletions, and the replica with the biggest raw size can be the one that has deleted the most. Counting it reported a bucket smaller than it is and would leave one writable over its quota, which is the opposite of what picking the largest was meant to guarantee. * topology: cap a volume's deletions at what it holds Live usage is read as a collection's size less its deletions, so a volume reporting more deleted bytes than it has cancels live bytes belonging to other volumes in the same bucket and reports it smaller than it is. Replica selection already floored that volume's own live size at zero; the totals have to agree with it.	2026-08-09 00:00:19 -07:00
Chris Lu	923d0bd20c	iceberg: repair non-compliant manifests at commit (#10641) ... * iceberg: stamp a default name mapping on new tables * iceberg: repair non-compliant manifests at commit * s3tables: verify ClickHouse writes read back through PyIceberg * iceberg: carry the manifest-list content into repaired manifests * iceberg: refresh the default name mapping on schema evolution * iceberg: merge historical names into the refreshed name mapping * iceberg: never fail a commit on repair fallout * iceberg: harden manifest repair against writer dialects * s3tables: keep PyIceberg reader stderr out of row data * iceberg: keep name mappings unambiguous across field id reassignment * iceberg: align existing manifest content metadata with the list entry	2026-08-08 21:24:37 -07:00
Chris Lu	2d9ea0285c	s3: add the RenameObject endpoint (#10659) ... * s3: add the RenameObject endpoint PUT /{bucket}/{key}?renameObject with x-amz-rename-source moves an object through the filer's AtomicRenameEntry, so no bytes are read or rewritten and the ETag, tags and SSE keys travel with the entry. Only unversioned buckets: a versioned rename would have to rebuild the .versions chain, and AWS offers RenameObject on directory buckets, which cannot be versioned. The source arrives in a header, so it is authorized separately for read and delete; both keys are locked, in key order, across the precondition checks and the move. * s3: let a matched source ETag precondition settle its date precondition RFC 7232 has an ETag precondition outrank the date precondition on its own side, and AWS documents the same for CopyObject: a matching x-amz-copy-source-if-match with a failing x-amz-copy-source-if-unmodified-since copies rather than returning 412. The source check evaluated all four headers in sequence, so the date header could still veto a decided ETag match. validateConditionalHeadersForReads already applies this precedence; the source path now matches it. * s3: cover a rename source named as a directory without a trailing slash Renaming a directory would move a whole subtree, so it has to stay a missing key whether or not the caller wrote the trailing slash. * s3: accept a bare object key as the RenameObject source AWS spells x-amz-rename-source both ways. Its CLI, Java and Rust examples pass the bare source key, and only a second CLI example and the boto3 conditional example pass bucket/key; the API reference's own example is a bare key too. The header was read as bucket/key only, so the form AWS leads with was rejected with InvalidArgument and the endpoint was unusable as documented. A value is now read as a literal key first — the only reading that can never name the wrong object — and as bucket-qualified second, when it carries the request's own bucket and the literal key does not exist. That costs one extra lookup only for a source that starts with the bucket's own name. Another bucket's name in the source is no longer a distinct error: RenameObject moves within one bucket, so it is simply part of a key this bucket does not hold, and it reports NoSuchKey. * s3: only a proven absence picks the other reading of a rename source A source that resolves to a directory is not a miss to fall through on: the literal path is still what the caller named, so answering for it beats renaming a different object under the bucket-qualified reading. With a directory at bucket/source.txt and an object at source.txt, a rename naming the former moved the latter. A failed lookup is not a proof of absence either, so a blip can no longer redirect a rename to the other reading.	2026-08-08 21:24:30 -07:00
hsdfatandChris Lu	0dd33bd7ec	s3api: fix ListObjectsV2 dropping objects under a partial prefix (#10656) ... * s3api: fix ListObjectsV2 dropping objects under a partial prefix ListObjectsV2 with a prefix that names only part of a directory name - "data/a" matching both "data/a/..." and "data/ab/..." - dropped objects when paginating with a small max-keys, reporting IsTruncated=false before all matching keys had been returned. On resume, normalizePrefixMarker took the listing directory from the continuation marker ("data/a/") and discarded the "a" name prefix, so once the marker's subtree was exhausted the sibling directories that also matched the prefix were never listed. Resolve the listing directory and name prefix from the prefix instead of the marker, so siblings sharing the name prefix are still listed after the marker's subtree. Fixes #10652 * s3api: cover a sibling-directory marker in the partial prefix tests * s3api: resume a truncated delimited listing from the emitted CommonPrefix Rebuilding the continuation token from the request dir plus the listing prefix assumes that prefix names a directory. With a partial name prefix like "data/a" it produced "data/a/ab/" for the sibling prefix "data/ab/", so the next page resumed in the wrong subtree and repeated it forever. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-08-08 20:24:44 -07:00
Lars Lehtonen	2ff3dda7cd	chore(weed/s3api/policy_engine): prune dead code (#10599)	2026-08-06 16:07:59 -07:00
3 people	d1f503181b	[s3] force filer apply s3 expiry metadata (#10469) ... * fix: apply S3 Expiry Metadata * add test Header X-Seaweedfs-Expires-S3 * resolve comments * test entry lookup by mtime * filer: skip s3 expiry stamp on versioned entries The s3 expiry path skips entries carrying a version id, so stamping one takes away its expiry rather than moving it onto mtime. Files under .versions/ are written once, so crtime already tracks their needles. --------- Co-authored-by: Konstantin Lebedev <whitefox@mayflower.work> Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-08-05 23:55:20 -07:00
Chris Lu	a5e8254ffd	s3: give a versioned metadata-only copy its own chunks (#10594) ... * s3: give a versioned metadata-only copy its own chunks A self-copy that only rewrites metadata clones the source entry, chunk fids and all, and writes the clone back. With no versioning that is exactly right: the clone replaces the entry it came from, so one entry owns the needles the whole time. Under versioning the clone lands in a new .versions/ file and the source stays live, and nothing refcounts a plain shared chunk list -- deleting either version (a NoncurrentVersionExpiration rule, say) frees needles the other still points at, and the next vacuum makes that permanent. rclone hits this on every upload, since it stamps mtime with exactly this copy. Take the metadata-only path only where the write replaces the entry it read: the bare key of a bucket without versioning. Versioned, suspended, and versionId-pinned copies fall through to the regular copy path, which gives the destination its own chunks. * s3: reencrypt a versioned SSE-KMS key rotation instead of reusing the chunks A same-object copy that changes the KMS key id hands the source chunks straight back, on the assumption that the copy overwrites the entry they came from. A versioned bucket writes a new version beside the source instead, so the two end up sharing needles that nothing refcounts, and deleting either one frees the other's data. Reuse the chunks only when the destination really is the source entry; otherwise fall through to the reencrypt path, which also gives the new version the key it asked for rather than leaving it on the old one. * s3: make one predicate decide whether a copy replaces its source The metadata-only branch and the key-rotation strategy both answer the same question -- does this copy write back to the entry it read -- so let them share one predicate instead of pairing a same-destination check with it separately at each site. * test(s3): fail the copy regression tests when the vacuum does not run The helper swallowed a failed or non-200 request to the master, so a vacuum that never ran turned both chunk-ownership assertions into no-ops: the tombstoned needles were still readable and the surviving version looked fine either way. Require the endpoint, the request, and a 200. * ci(s3): run every versioning test in the regression gate The gate named the tests it wanted, so a new regression test sat there uncovered until someone remembered this file -- it fooled me into thinking two tests added in this PR never ran anywhere, when the comprehensive job had them all along. Invert it: run everything, and name a test only to keep it out. The delete job beside this one already works that way, and the suite costs about two minutes. Only the pagination stress tests are excluded; they build 1500+ versions, skip themselves without ENABLE_STRESS_TESTS, and have their own make target. Go's regexp has no negation, so the pattern is still assembled from a listing, the way the volume-server integration workflow does it. Note the trailing $$: make eats a lone trailing $ and takes the anchor with it.	2026-08-05 20:29:28 -07:00
Chris Lu	e8020910db	iam: authorize IAM management actions as IAM actions (#10593) ... * s3: keep a non-S3 action out of the request-shape resolver ResolveS3Action reads the request shape before it looks at the base action, so an iam: or sts: action on a request that happens to carry an S3 query parameter came back as the S3 action for that parameter. An action that already names its service is resolved; there is no S3 request shape to read for it. * iam: authorize the standalone IAM server's actions as IAM, not as S3 The standalone `weed iam` server wrapped its single POST / route in the generic S3 Auth middleware with ACTION_ADMIN. The route has no {bucket}, so the check ran with an empty bucket and resolved to a coarse S3 action rather than the IAM one. The embedded IAM surface checks iam:<Action>; the standalone one was never updated to match. Both now go through one authorization function, so they cannot drift apart again. It also rejects the anonymous identity, which has no user of its own to run a self-service action against, and reads UserName from the body only, where the handlers read it from.	2026-08-05 17:08:21 -07:00
Chris Lu	c2b47967bd	s3: retire the suspended null marker only once the PUT has committed (#10589) ... The suspended PUT dropped the null delete marker before writing, so a failed write left the .versions pointer naming a marker that was gone. The read path heals a dangling pointer by promoting the newest survivor, so a key the caller had deleted came back serving an older version, and the heal persisted that pointer. Move the retire into afterCreate via the shared finalize, which also brings the ownership check the copy and multipart paths already have.	2026-08-05 14:31:50 -07:00
Chris Lu	f09bc14165	s3: report the effective ownership when a bucket has none stored (#10591) ... * s3: report the effective ownership when a bucket has none stored GetBucketOwnershipControls read Seaweed-X-Amz-Ownership straight out of the bucket entry, so a bucket that never had one written reported an empty ObjectOwnership. The object write path defaults the same missing attribute to BucketOwnerEnforced, so the API contradicted the behavior it describes. Resolve the stored value through one helper both readers share, and let PutBucketOwnershipControls persist unconditionally so setting the default value still gives DeleteBucketOwnershipControls something to remove. * test: cover the bucket ownership controls round trip Pins the behaviors the ownership default fix depends on: a bucket that never had ownership controls written reports BucketOwnerEnforced, and putting that same value on such a bucket still persists it, so the delete that follows has something to remove. The put-then-delete case gets its own bucket -- run after an ObjectWriter put, it would pass against an implementation that skips only the initial write. The acl workflow already runs this package against a live weed mini, so it needs no wiring.	2026-08-05 13:15:28 -07:00
Chris Lu	5269d93fa8	s3: let a suspended-versioning multipart completion replace the null delete marker (#10585) ... * s3: let a suspended-versioning multipart completion replace the null delete marker In a versioning-suspended bucket a DELETE writes a null delete marker into the key's .versions directory. CompleteMultipartUpload then writes the new null version at the regular path but left that marker in place, so the completion returned 200 and the object listed while HEAD and GET kept resolving the marker and answered NoSuchKey. PutObject already handles this; do the same on the multipart path. * s3: order the suspended-versioning null cleanup behind the multipart write Removing the null delete marker before writing left a failed completion having already published the key's newest real version: the marker was gone, the pointer still named it, so reads rescanned .versions and promoted the older version. Do both fixups only once the write commits, pointer first so reads never see a pointer aimed at a marker that is no longer there, and fail the completion when the pointer cannot be cleared instead of returning 200 for an object HEAD and GET still miss - a non-ErrNone finalize keeps the upload directory, so the caller's retry replays it. Also cover a pre-suspension real version in the regression test. * s3: skip the suspended null cleanup when a concurrent write won the key The completion's .versions fixups are unconditional rewrites of shared state and the routed path runs off the object write lock, so a DELETE landing between the multipart write and the cleanup had its own null delete marker erased - leaving a successfully deleted key readable as an older retained version. Re-read the object first and leave the cleanup alone unless it is still the one we wrote. This narrows the window rather than closing it; a compare-and-set pointer flip is the real answer and wants its own change. * s3: re-read the completed object from the filer that took the write The guard compared the object against our upload id through the routed read, which skips an owner it recently found unreachable and falls back local-first. A write that just landed on the owner could then read as superseded on another filer, skipping the cleanup and leaving the key unreadable - the bug this set out to fix. Read back from the filer the write went to instead. * s3: trim the suspended-completion comments to the non-obvious why * s3: lift the suspended null-write finalize into a named helper The pointer-then-marker ordering is policy shared by every suspended null write, not something the multipart path should be stating on its own; putSuspendedVersioningObject and the copy path each restate it today. Give it a home next to the versioned finalize helpers, and reuse the canonical key normalizer and the existing test helpers rather than open-coding both. * s3: retire the null delete marker on a suspended-versioning copy The suspended CopyObject branch cleared the .versions latest pointer but left the null delete marker a preceding DELETE wrote. While the regular-path object owns the null slot that marker is shadowed, so it reads and lists correctly - but it resurfaces as a phantom delete for a key nobody deleted once that null version goes away. Route the branch through the shared finalize. * s3: keep the suspended null cleanup from erasing a concurrent delete Retiring the marker on the copy path reopened the race the multipart path had already closed: a DELETE landing between the write and the cleanup lost its own marker, so a rescan promoted an older version under a deleted key. Move the ownership check into the shared finalize, keyed on the attribute that identifies the caller's write, so both paths get it.	2026-08-05 11:49:36 -07:00
Chris Lu	7063b3e14c	s3 lifecycle: bound the daily-replay pass so a quiet cluster stops wedging the job (#10578) ... * s3 lifecycle: bound the daily-replay subscription at the pass boundary A pass opens one meta-log subscription and 16 shard drains, then waits on all of them. Nothing told the subscription where the pass ends, so the only exit was the fan-out spotting an event past runNow — i.e. some unrelated write landing under /buckets after the pass started. On a cluster that goes quiet the reader parks in Recv, every shard drain starves on an empty channel, and Run never returns. The job sits at stage "starting" with the executor slot held and no log line, so expiry stops cluster-wide until someone restarts the worker. The pass covers (globalStartTsNs, runNow], so say that: UntilNs on the subscribe request makes the filer end the stream once it has shipped that range. The reader then closes the event channel on the way out, which is what unblocks the fan-out and the drains when the stream finishes on its own rather than by cancellation. Same fix retires the other silent hang: a reader that failed early (subscribe error, stream error) also left every drain waiting forever. * s3 lifecycle: keep a halted shard from starving the shared fan-out A drain that halts mid-stream (BLOCKED / RETRY_LATER / an RPC error on dispatch) returns while the fan-out is still routing that shard's events. After 256 of them the per-shard buffer is full and the fan-out blocks on the send, so no other shard sees another event. Run's WaitGroup never drains, and the teardown that would cancel the reader sits behind that wait — the pass wedges exactly like an idle subscription did, with one S3 hiccup as the trigger. Keep discarding the channel after runShard returns. The events are past this shard's saved cursor and get re-scanned next pass anyway. * s3 lifecycle: assert the starved shard actually made progress The fan-out test only checked that Run returned, which a version that quietly dropped the second shard's events would also satisfy. Assert the dispatch landed and the cursor moved. recordingClient gains a per-object outcome map: the two shards dispatch from separate goroutines, so pinning BLOCKED by call index was a race waiting to pick the wrong shard. * s3 lifecycle: fail the pass when the shared subscription dies Closing the event channel on reader exit is what unblocks the shard drains, but it also means a subscribe that never opened, or a stream that broke mid-pass, now ends every drain cleanly. Run logged that at V(2) and returned the shard result — so a filer failure produced a green lifecycle job that had processed nothing. Surface it as the pass error. Cursors still hold what was processed and tomorrow resumes there; what changes is that the job stops claiming success. Cancellation has to stay a non-error — the shell driver's -runtime cap is a truncated pass, not a failed one — and a canceled gRPC stream arrives as a status code, not a wrapped context.Canceled, so isCanceled checks both forms the way the rest of the tree does. * s3 lifecycle: decide reader cancellation by intent, not status code A stream we cancel and a stream the filer cancels both arrive as codes.Canceled, so classifying the reader's exit by its error let a truncated pass report success whenever the failure happened to carry a cancellation status. Intent is knowable exactly, so read that instead: the pass stops on purpose only when the caller's context ended (the shell driver's -runtime cap) or the fan-out hit the pass boundary itself. Everything else is a broken subscription and fails the pass. TestRun_ServerSideCancelFailsThePass and TestRun_CappedPassIsNotAFailure are the same codes.Canceled from the reader with opposite verdicts — the pair only passes because the decision no longer looks at the error. * s3 lifecycle: time out a subscription that stops delivering UntilNs ends a healthy stream and gRPC keepalive catches a dead connection, but neither reaches a filer that keeps answering pings while its handler has stopped producing. The pass would wait on that forever, since s3_lifecycle is the one job type with no execution timeout. Bound the wait for each response at 20 minutes, and opt into the filer's idle heartbeats so a caught-up stream proves liveness instead of looking stalled. The default sits above the filer's 15-minute metadata-gap recovery budget, so a subscriber legitimately parked on a gap is never mistaken for a stalled one. Recv is only interruptible by killing the RPC, so it moves to its own goroutine behind a per-response deadline. The timer covers only the wait on the filer — dispatch to Events happens outside it, so a slow consumer can't trip the watchdog. Approach and the 20-minute figure are from #10577 by way of comparing the two fixes; the wiring differs because the reader here ends the pass by closing its event channel rather than cancelling the fan-out. * s3 lifecycle: trim the comments added by this branch Keep the non-obvious why, drop the prose restating what the code says. * s3 lifecycle: snapshot reader intent where the reader stops Sampling ctx.Err() during teardown reads it after the drains and cursor saves have run. A reader that failed while the deadline was still live, on a pass whose teardown then outlives that deadline, was classified as an intentional stop and reported success. Sampling earlier in Run is not the fix either: before the shard wait, a legitimately capped pass has not reached its deadline yet and would be misclassified the other way. Intent belongs where the reader actually stops, so the reader goroutine records it next to the error it returns. Reported by greptile on #10578. * s3 lifecycle: cover the worker-dispatched pass with nothing due The e2e suite drives the shell command in 14 of 15 files; the one test on the real admin->worker path backdates an object, so its own delete pushes a meta-log event past the pass boundary and ends the pass. The branch where a pass has nothing to dispatch was never exercised through the worker. Cover it, asserting the pass returns on its own: no admin cancellation, and the executor slot free for the next one. This is not a regression test for the wedge. A pass used to end when any write landed past its boundary, and on a shared test cluster something usually does — the whole suite passes on the unfixed build, verified. The deterministic guards stay the dailyrun unit tests; this one would catch a pass that hangs unconditionally.	2026-08-05 08:41:37 -07:00
Chris Lu	7d6c83b126	s3: stop treating a directory marker as a versioned object (#10573) ... * s3: delete a directory marker instead of versioning it The key "dir/" is stored as the filer directory itself, so a delete marker cannot stand in for it without hiding the children underneath, and its history has to sit inside the directory it describes, where listings keep meeting it. Delete it the way an unversioned bucket already does: remove the directory when nothing is left under it, demote it to a plain directory when children remain, and drop a history an older build recorded for it. * s3: stop resolving directory markers through a version history Nothing records one for them any more, so the lookups that read it are dead weight - and the one in the listing was a filer round trip per directory marker returned, which for a bucket that keeps a marker per directory is the whole listing cost. A listing reads what a directory stands for straight off the entry it already has; a unit test pins that N markers cost one ListEntries rather than N+1. The guard that keeps a history left inside a directory by an older build from surfacing as a key named after it stays. * s3: do not let deleting "dir/" destroy the object at "dir" Writing under an existing object turns that object's entry into a directory while it keeps its data, so the keys "m2" and "m2/" end up sharing one entry. Stripping the entry to delete "m2/" therefore wiped the object at "m2" - a different key, and in a versioned bucket one no delete marker records. Leave a directory holding uploaded data alone; "m2/" does not name it. * s3: make the directory-marker delete fail closed and take the write lock The guard that spares a promoted file only fired when the entry read succeeded, so a transient filer error fell through to the delete and could destroy the object at "dir" anyway. Fail the request instead, take the object write lock so the entry cannot change between the check and the delete, and report a stale history that cannot be removed rather than leaving it to keep naming the key in ListObjectVersions. * s3: check If-Match inside the directory-marker delete lock The lock belongs to the caller: taking it inside the delete nested it under the batch handler's own lock, and since every lock from a gateway shares one owner the inner release would have freed it while the outer caller still assumed it held it. Both callers now own the lock, the single-object path re-checks If-Match inside it the way the other delete paths do, and a batch delete of a trailing-slash key in an unversioned bucket goes through the same marker path instead of the raw delete. A history lookup that fails now fails the delete.	2026-08-05 00:24:54 -07:00
Chris Lu	d448e9db7b	iceberg: withhold the S3 endpoint from credential-vending clients (#10570) ... * iceberg: withhold the S3 endpoint from credential-vending clients A client that sends X-Iceberg-Access-Delegation: vended-credentials builds its storage credential out of the LoadTable config and drops the one it was configured with. We vend no credentials, so the endpoint we advertised left DuckDB signing nothing: every metadata and data file came back 403, and its attempt to refresh the empty credential 404ed on stage-created tables. Answer those clients with no config at all so they keep their own credentials. Clients that do not ask for delegation still get the endpoint. * iceberg: mark load responses as varying on the delegation header The FileIO config in a table or view load response now depends on whether the client asked for vended credentials, so a cache between us and the client must key on that header rather than on the URL alone. * test: cover the DuckDB vended-credentials access pattern Runs weed mini with -s3.externalUrl, which is what makes the catalog advertise an endpoint at all, and checks both halves: a plain LoadTable still gets the endpoint, while one asking for vended credentials never gets an endpoint without the credentials to sign with. The DuckDB round trip creates a table from a query and reads it back, which is the flow that failed with 403 on every data file.	2026-08-04 18:14:34 -07:00
Chris Lu	474a0713b0	s3: honor the version history of a directory marker (#10571) ... * s3: stop listing a directory marker whose latest version is a delete marker A directory marker is stored as the filer directory itself, so deleting the key "dir/" writes its delete marker into dir/.versions while the directory keeps its mime and stays a key object. Every listing kept reporting the key. Consult that history before treating the entry as a key, and demote it in memory when it is delete-marked so live children still hold the prefix. Also skip the container's own .versions entry while listing inside it: the suffix match read it as the history of a nested object named "", which surfaces as a phantom dir/dir key as soon as a live directory version exists. * s3: a directory marker with version history is not also the latest null version The directory entry behind the key "dir/" is that key's null version, but list-object-versions reported it with IsLatest hardcoded true. After a delete the key came back twice, once as the delete marker and once as a null version, both claiming to be latest. Read the pointer under the directory instead. * s3: resolve directory markers through their version history on GET and HEAD GET and HEAD short-circuit any trailing-slash key straight to the filer directory, so a directory marker kept answering 200 after its delete marker was written. Resolve the key from dir/.versions first when the bucket is versioned: a delete-marked current version answers 404 with x-amz-delete-marker, a named delete-marker version answers 405, and a key with no history keeps today's directory-probe behavior untouched. * s3: re-creating a directory marker retires its delete marker PutObject on a trailing-slash key never looked at the bucket's versioning state, so re-creating a marker after a delete left the latest-version pointer on the delete marker and the key stayed invisible to every versioned read. Point the key back at the directory entry, which is its null version, and drop the null version .versions may still hold — the same two steps a suspended write already takes, now shared. * s3: fail a directory-marker request whose version history cannot be read Every lookup of dir/.versions treated any error as "no history", so a filer hiccup served the directory entry for a key whose current version may be a delete marker, reported a null version as latest over one, and let a PUT report success without retiring the delete marker it was meant to retire. Only a confirmed absence takes the no-history path now. * s3: cancel the directory-marker probe stream instead of abandoning it The probe answers off the first entry and returns, leaving the ListEntries stream open for the life of the parent context. Give it a context of its own.	2026-08-04 18:14:00 -07:00
Chris Lu	b452a5e41b	s3: honor a bucket owner recorded as an identity (#10567) ... * s3: resolve a bucket owner recorded as an identity The admin UI and weed shell record a bucket's owner as an identity name in s3-identity-id and never write the account id the S3 API stores alongside it, so such a bucket looked unowned: its ACL owner fell back to the default admin account, and under the default BucketOwnerEnforced ownership every object uploaded to it was stamped with that account instead of the bucket owner. Resolve the identity to its account when no account id is recorded, in the one place both the bucket metadata and the bucket config derive the owner from. * s3: drop the recorded account when the bucket owner is reassigned Changing the owner of a bucket created through the S3 API left its old account id behind, and that outranks the identity when the owner is resolved, so the new owner never took effect for object ownership or the bucket ACL.	2026-08-04 16:00:04 -07:00
Chris Lu	c191b2fe01	iceberg: let clients select their table bucket as the catalog warehouse (#10549) ... * iceberg: accept bare bucket names and ARNs as the catalog warehouse Only s3://<bucket>/ was recognized. A warehouse spelled as a bare table bucket name or as the s3tables bucket ARN -- the two forms users reach for first, the latter being what AWS S3 Tables itself takes -- was silently dropped, so every call landed on the default "warehouse" bucket and failed with "table bucket warehouse not found". * iceberg: report a missing table bucket as 404, not 500 Pointing a client at a table bucket that does not exist -- which every client with no warehouse set does, since the default bucket "warehouse" rarely exists -- returned InternalServerError with a message naming a bucket the client never asked for. Answer 404 and say how to select one. * admin: show the warehouse in the PyIceberg example The example connected without one, so it always resolved to the default table bucket and every client that copied it failed on the first call. * test: pin bearer auth against a table bucket that exists The subtest called the catalog with no warehouse and accepted 500 as proof that auth had passed, since the default bucket does not exist. A missing table bucket now answers 404, which the test read as an auth failure. Give it a real table bucket so only 200 passes. * test: assert the missing-bucket guidance reaches the client The status and error type were checked but not the message, which is the part of the mapping that tells a user how to select a table bucket. * test: encode the warehouse query value The ARN case pasted raw colons and slashes into the query string. Go's parser tolerates them, so the test passed without modelling how a client actually sends the request.	2026-08-03 13:25:37 -07:00
Chris Lu	cc2775d9f2	s3: register an identity's inline account instead of collapsing it into admin (#10548) ... * s3: register an identity's inline account instead of collapsing it into admin Credential stores persist an account inline on the identity and never emit a top-level accounts list, so every user created through the IAM API or the admin UI with an email hit the "non exist account ID" branch and was given the shared admin account. Distinct users then presented the same owner id, so ownership checks could not tell them apart and each passed for the others' buckets. Treat an id missing from the account map as undeclared rather than invalid: register it, keeping an email another account already claimed. Both load paths now resolve the account through one helper. * s3: refresh an undeclared account from the identity that carries it The merge path starts from the live account cache, so an identity upserted with the same account id but a new email or display name kept the cached copy: the new address never reached the email index and the replaced one still resolved. Changing a user's email through the admin UI takes exactly that path. An account registered from an inline block is only described by the identity carrying it, so refresh it and move its email claim. Accounts from a top-level list and the predefined defaults are marked declared and stay authoritative. * s3: let an account reclaim an email once its holder moves away Two identities can carry the same email, and the second to load leaves the lookup with the first. Returning early when the incoming metadata matches the cached account meant the loser never re-ran the claim, so an address freed by the holder's update resolved to nobody until the loser itself changed. Re-index on the unchanged path, which is a no-op while another account still holds the address.	2026-08-03 12:45:42 -07:00
Chris Lu	c2701955c7	s3: cover three untested STS paths (#10521) ... * s3: test the GetCallerIdentity handler The handler had no test, only XML marshalling, so nothing pinned that a caller presenting session credentials is reported as the assumed role rather than the user who minted the session. * s3: drive AssumeRoleWithWebIdentity over HTTP with a real OIDC token Coverage reached the OIDC path either at the IAMManager service layer or through the Authorization: Bearer shortcut. Nothing exercised the public STS entry point an AWS SDK actually calls, which is where parameter parsing, the IAMManager dispatch and the XML response shape live. * s3: test that every STS route emits an audit entry STS responses go out through WriteXMLResponse, which never calls PostLog, so track() is the only thing that logs them. A route registered outside it would mint credentials with no audit trail and nothing would notice. STS has three routing layers, so a new action is easy to attach to the wrong one. * s3: make the STS tests assert what they claim to cover The audit routing test ran against an uninitialized STS service, so every case answered 503 and a non-404 status was the only evidence the request had reached STS at all - the POST-body case could have been served by the dispatcher's IAM branch and still passed. Back it with a real STS service and assert the STS response namespace, which IAM and S3 responses do not carry. The session policy case checked that the policy travelled in the token rather than that it restricted anything; assert the narrowed bucket is allowed and another bucket is refused. Give the forged-token case the same claim set a valid token gets, so it cannot pass for want of a claim, and cover both rejection paths: a key we do not publish, and a key id absent from the JWKS.	2026-07-31 23:56:04 -07:00
Chris Lu	1ce106e69d	s3: audit the assumed-role principal and the STS caller (#10519) ... * s3: log the requester's principal ARN in the audit entry An STS session authenticates as an opaque session subject, so requester alone gave an operator no way back to the assumed role or the session name. Record the principal ARN next to the identity name and emit it as requester_arn. * s3: record the caller identity in the STS handlers AssumeRole, GetFederationToken and GetCallerIdentity verify the caller themselves and are not wrapped by the auth middleware that records the identity, so every audit entry for minting a session had an empty requester. * s3: resolve the audit principal ARN the way policy evaluation does A JWT-authenticated identity carries no PrincipalArn — the auth layer hands the principal over in a request header — so reading the field directly left requester_arn empty for OIDC callers. buildPrincipalARN is the resolver the policy path already uses: header first, then the identity's own ARN, then a synthesized user ARN for legacy identities that have none.	2026-07-31 19:51:03 -07:00
Chris Lu	fa432f9a6a	s3: keep an admin's role session scoped to the role (#10520) ... * s3: keep an admin's role session scoped to the role AssumeRole copied the caller's admin standing into the minted session as the is_admin claim, which short-circuits base policy evaluation. An admin assuming a scoped-down role therefore kept full access and the role's attached policies, explicit denies included, were never evaluated. Only a session the caller assumed for itself carries the claim now — a legacy static admin has no IAM policies for such a session to inherit. * s3: name the caller when it assumes a session for itself An identity that carries no principal ARN left the self-assumed session with an empty role name in its assumed-role ARN. callerPrincipalArn synthesizes the canonical user ARN for that case.	2026-07-31 19:50:59 -07:00
Chris Lu	d8d29c4ede	s3: carry storage class in the cached listing metadata (#10516) ... A listing on a versioned bucket is served from metadata cached on the .versions directory entry so the whole listing is a single scan. The cache carried size, mtime, ETag, owner and the delete-marker flag but not the storage class, so newListEntry found none and fell back to STANDARD. The result was that HEAD and the listings disagreed about the same object: HEAD reported the class the object was stored with, while ListObjectsV2 and ListObjectVersions reported STANDARD for every object. Clients that filter or tier on storage class act on the listing. Caches the class alongside the other listing fields, clears it with them, and copies it in the routed RECOMPUTE_LATEST path so both finalize paths agree.	2026-07-31 19:48:17 -07:00
Chris Lu	f582c8451b	s3: report a peer that went away as ClientDisconnected, not IncompleteBody (#10511) ... * s3: report a peer that went away as ClientDisconnected, not IncompleteBody A streaming PUT whose body ends early is always reported as IncompleteBody (400). That collapses two cases with opposite causes: the peer vanished mid-upload, and the peer sent fewer bytes than it promised while still connected. The first points at the network path, the second at the client, and once merged they cannot be told apart from the logs. Split out ClientDisconnected (499) and select it when the request context shows the peer is gone. The upload itself keeps running on a background context so chunks still finish, which means cancellation races the read error; a missed signal degrades to IncompleteBody exactly as before. * s3: note what request-context cancellation is taken to mean	2026-07-31 19:43:45 -07:00