6 Commits

Include renames

Author	SHA1	Message	Date
Chris Lu	69cc2869ad	Fixes from the review of the admin bucket policy UI (#10907) ... * admin: treat a missing S3 Tables policy as an empty load, not an error The bucket/table policy GET relayed the backend's 404 NoSuchPolicy to the dialog, whose loader treats any non-OK response as a load failure and keeps Save and Delete blocked. A bucket or table without a policy could never be given one. Return policy null instead, the same contract ShowBucketPolicy uses for classic buckets. * admin: reject policy documents the structured editor would misread A top-level JSON array passed the object guard (typeof [] is 'object') and loaded as a zero-statement policy, which the next commit would rewrite to an empty document. Object elements in Action/Resource were coerced to '[object Object]' and saved that way on the s3tables surface, which stores policies verbatim. Both now throw, which routes the document to the JSON tab like other unrepresentable shapes. * admin: let the JSON tab save documents the structured editor can't model Save with the JSON tab active required a round-trip through policyDocToEditorState, so exactly the documents the dialogs shunt to 'JSON tab only' mode (unrepresentable Effect, Resource+NotResource, and the like) could never be saved - Delete was the only mutation left. Invalid JSON still blocks; an unrepresentable document now saves and the editor state stays marked unparsed. * admin: pin the policy editor to what each consumer's backend supports The s3tables evaluator has no NotResource/NotPrincipal fields - it silently drops them, turning Allow+NotResource into allow-everything and making Deny+NotPrincipal inert - and it only matches s3tables: actions against s3tables ARNs, while the editor suggested s3: actions and arn:aws:s3::: resources. New registerPolicyEditor knobs: allowNegation hides the Not* modes and routes documents using them to the JSON tab; resourceSuggestions pins the Resource autocomplete to the open resource's ARN; the S3 Tables dialogs get an s3tables-only action datalist. requirePrincipal now also hides NotPrincipal, which policy_engine.ValidateBucketPolicy always rejects, and the client-side check requires Principal specifically to match that server rule. * admin: save S3 Tables policies from a button, not form submission The multi-input structured editor sits inside a form whose Save button was type=submit, so Enter in any single-line editor input - accepting an autocomplete suggestion, say - implicitly submitted whatever half-built statement the editor held, and the backend stores the document verbatim. A lone statement with no Principal matches nobody, locking out every non-owner. Save is now an ordinary button and the form ignores submission. * admin: block zero-statement policy saves Committing the active tab before the emptiness check made 'Policy JSON is required' dead code: an empty editor serializes to {"Statement":[]}, which the s3tables backend stores verbatim - evaluated default-deny for every non-owner, while the statement-count column keeps showing 'Not configured'. All three policy dialogs now refuse a save with no statements and point at Delete instead. The classic bucket modal only gained a clearer message; the server already rejected the document. * admin: guard S3 Tables policy mutations against stale and overlapping requests The save/delete completions ran against whatever resource the shared modal happened to show by then: a slow PUT for one bucket would hide the modal mid-edit of another and misattribute its alerts, a late DELETE cleared the shared textarea over the newly opened resource with its loaded flag set, and nothing stopped a double-click from firing two overlapping mutations. Ported the classic modal's pattern: capture the target on start, flag the mutation in flight with the buttons disabled, and only touch the UI when the completion still matches the open resource. Success now reloads the page, which also keeps the Policy column's statement count honest. * admin: confirm before deleting an S3 Tables policy Delete Policy sat next to Save and fired on a single click; with default-allow enabled one stray click silently dropped the resource policy and left the bucket open to every principal. Same confirmation the classic bucket modal already has. * admin: let a corrupt stored bucket policy be shown, fixed, and deleted A stored document the decoder rejects made the policy GET 500, and with the loaded flag never set the modal blocked both Save and Delete - the one policy an operator most needs to remove was the one they couldn't, even though the delete path never reads the document. The GET now returns the raw bytes alongside a null policy; the dialog hands them to the JSON tab and unblocks the buttons. * admin: url-encode the bucket name in the policy API calls The filer lists any directory under the buckets path, names S3 would never allow included; one carrying '#' or '%' broke the fetch URL or addressed a different name than the modal shows. * admin: drop stale edit-policy responses on the IAM policies page The same race the bucket and S3 Tables dialogs already guard against: open one policy's editor while its GET stalls, open another, and the late response populates the editor under the second policy's name - Update then saves the first policy's statements over the second. * admin: warn before a bucket policy save drops unsupported fields The editor tracks unmodeled top-level keys precisely so confirmPolicyFieldDiscard can warn before the server's Version+Statement decode discards them, but only the IAM page called it; the bucket modal saved a pasted document with e.g. a console-generated Id without a word while the editor kept displaying the field. * s3: enforce the bucket policy size cap on both surfaces The 20KB cap lived only in the admin UI, so a larger policy stored via the S3 API displayed there but could never be re-saved, desyncing the two writers the cap comment claimed could not desync. The constant now lives in policy_engine next to the shared validator and PutBucketPolicy rejects oversized documents with PolicyTooLarge, matching AWS. * admin: ship the policy editor's fieldset styles with the editor The .policy-stmt-* rules that undo Bootstrap's full-width legend reset stayed behind in policies.templ when the editor markup moved to the shared script, so the bucket and S3 Tables dialogs rendered Actions/ Resource/Principal as full-width jumbo headings. PolicyDatalists is the component every consumer already renders once; the styles live there now. * s3: mirror bucket policy changes into the IAM store from the metadata subscription The advanced-IAM path appends the bucket-policy:<bucket> document to every STS/session evaluation, but only this gateway's own PutBucketPolicy maintained that mirror - a policy tightened or created through the admin UI (or another gateway) never reached it, so revoked access stayed live indefinitely, and the delete side was an unimplemented TODO in any case. The metadata subscription now diffs the stored policy on every bucket entry change and updates or removes the mirror, covering all writers and deletion with one mechanism; IAMManager gains the missing RemoveBucketPolicy. * admin: deduplicate the bucket policy write path Set and Delete carried line-for-line identical filer closures; bucketPolicyMutation already treats nil as clear-the-key. The shared helper sits below Set's validation, since ValidatePolicy cannot take the nil document Delete passes. * s3: drop ValidateBucketPolicy's re-checks of ValidatePolicy rules Both callers run ValidatePolicy first, which already enforces the version and at-least-one-statement rules; the duplicates were dead code with drifted error text. * admin: seed a new statement's Resource from the pinned suggestions A fresh statement on the S3 Tables dialogs started with no resource row at all; seed it with the broadest pinned ARN the same way cfg.bucket already seeds the classic modal. * admin: refuse to save Not* fields the backend would silently drop Hiding the NotResource/NotPrincipal modes was not enough where negation is disallowed: the JSON tab accepts any valid document (that is its job), and a statement's Advanced-fields box can reintroduce the keys, so an s3tables save could still store fields the evaluator drops - turning Allow+NotResource into allow-everything. commitPolicyActiveTab now runs a final document-level check over what would actually be saved; Delete stays available for cleanup. * s3: move the IAM bucket policy mirror on a bucket rename A same-directory rename delivers one event carrying both entries, and the byte-equality short-circuit skipped the new name's mirror when the policy was unchanged - while the replayed delete for the old name removed its mirror, leaving the renamed bucket unmirrored. The mirror decision is now a pure function that removes the old name and writes the new one regardless of byte equality, with the rename cases unit tested. * s3: backfill the IAM bucket policy mirror on lazy bucket loads The metadata subscription only mirrors changes, so a policy that predates the IAM integration never reached the bucket-policy:<bucket> mirror and its grants did not bind on the IAM path until the policy was next modified. The gateway is deliberately lazy at startup (nothing lists all buckets), so the backfill hooks the same place a bucket's policy first becomes known: the cold bucket-config load. EnsureBucketPolicy writes only when no mirror is stored, so repeat loads cost one cached read. * s3: reconcile the bucket policy backfill against concurrent changes The backfill's check-then-write could race an event-driven mirror update or removal and re-store bytes that were already stale, with no later event to heal it. EnsureBucketPolicy now reports whether it wrote, and a write is reconciled against a fresh authoritative entry read: a changed policy is re-mirrored, a removed one is removed. Anything changing after that read fires its own event, which finds the backfill's write already present and supersedes it. The backfill also carries the entry's raw bytes rather than a re-marshaled document, so the reconcile can byte-compare. * s3: prime the bucket policy mirror before advanced-IAM authorization The backfill ran from the lazy bucket-config load, but IAM authorization evaluates the bucket-policy:<bucket> mirror before any handler runs - a grant carried only by a not-yet-mirrored policy denied forever, and the denied request never reached the code that would have loaded the bucket. authorizeWithIAM now primes the bucket config first (an in-memory cache hit once warm), and the backfill runs synchronously on the cold load so the very first authorization already sees the mirror.	2026-08-24 00:52:01 -07:00
Mathieu Arnold	e931cccc7b	Manage bucket policies via the admin ui (#10895) ... * admin: manage S3 bucket policies from the admin UI Bucket policies were only manageable through the S3 PutBucketPolicy API; the admin UI had no equivalent to the quota/owner/lifecycle editors it already offers. Add GET/PUT/DELETE for a bucket's policy, sharing the exact validation the S3 gateway uses. - Extract validateBucketPolicy/validateResourceForBucket out of s3api_bucket_policy_handlers.go into policy_engine.ValidateBucketPolicy / ResourceMatchesBucket so both the S3 API and the admin UI enforce identical rules. - weed/admin/dash/bucket_policy.go: Get/Set/DeleteBucketPolicy, writing through ObjectTransaction + PATCH_EXTENDED (the lifecycle pattern) so a concurrent owner/quota/lifecycle change on the same bucket entry isn't clobbered. Propagation to every S3 gateway is automatic via the existing filer metadata log subscription. The S3 gateway's IAM policy mirror is deliberately not replicated here (its delete path is already an unimplemented TODO on the S3 side). - New GET/PUT/DELETE /api/s3/buckets/{bucket}/policy routes, CSRF-guarded on writes. - Bucket list and details modal now show a statement-count badge, read from the entry already fetched (no extra RPC). - UI: a JSON-textarea policy editor modal, matching the lifecycle modal's structure. * admin: reuse the visual policy editor for bucket policies Extract the structured policy editor (add/remove statement, action/ resource/principal rows with autocomplete, JSON tab kept in sync) out of policies.templ's inline script into a shared weed/admin/static/js/policy_editor.js, and wire the bucket policy modal in s3_buckets.templ up to it instead of a bare JSON textarea. - registerPolicyEditor(which, config) replaces the hardcoded create/edit id derivation with a per-instance config (textarea/tab/body ids, datalist ids, requirePrincipal, bucket). The IAM policies page keeps its exact pre-extraction ids via two registerPolicyEditor calls, so its markup is unchanged. - New policy_datalists.templ exposes the three shared <datalist>s (actions/resources/principals) as @PolicyDatalists(), now rendered by both policies.templ and s3_buckets.templ. - requirePrincipal seeds new bucket-policy statements with Principal: "*" and adds a client-side check before save (the server, via policy_engine.ValidateBucketPolicy, remains the actual authority); the bucket config pins the Resource autocomplete to the open bucket instead of fetching every bucket in the cluster. - layout.templ loads policy_editor.js globally, after admin.js/ modal-alerts.js (basePath/escapeHtml/showAlert) which it depends on. 3a (the extraction) is a byte-preserving move verified against the unchanged policies.templ behavior before layering 3b's parameterization and the bucket-policy wiring on top. * admin: migrate S3 Tables bucket/table policy editors to the shared editor Third consumer of the shared visual policy editor: the S3 Tables bucket and table policy modals (a bare JSON textarea each) now get the same structured Editor/JSON tabs as the bucket policy and IAM policy pages, via registerPolicyEditor('s3tablesBucketPolicy'/'s3tablesTablePolicy', { textareaId: ... }). Storage and validation are untouched - S3 Tables policies still go through their own s3tables.PolicyDocument type and the s3tables.policy extended attribute, unrelated to policy_engine and s3-bucket-policy; only the editor UI is shared. Fix a real bug surfaced by adding this second load path: the bucket policy modal (and the naive first draft of this s3tables port) called commitPolicyTextareaToEditor() right after a GET and then force-switched to the Editor tab. commitPolicyTextareaToEditor() is designed to leave the current tab in place and the editor state untouched when a document fails to parse (so an in-progress edit survives a bad tab switch), so forcing the Editor tab afterwards could show empty/stale editor state that a careless Save would then serialize over a perfectly valid but structurally-unusual stored policy. Add loadPolicyTextareaIntoEditor(which) to policy_editor.js, which has no "current tab" to defer to and instead falls back to the JSON tab with an alert on a document the structured editor can't represent - the same safety editPolicy already had in policies.templ - and use it at all three "populate the editor right after a GET" call sites (bucket policy, S3 Tables bucket policy, S3 Tables table policy). * admin: show policy statement count on the S3 Tables buckets page Mirrors the "Policy" column already added to the classic S3 buckets list: a clickable badge with the statement count when the table bucket has a resource policy, "Not configured" otherwise. S3 Tables policies are a separate mechanism (s3tables.PolicyDocument under the s3tables.policy extended attribute) from the S3 bucket policy work elsewhere in this branch (policy_engine.PolicyDocument / s3-bucket-policy), so this is a parallel implementation of the same pattern rather than shared code. - S3TablesBucketSummary gains PolicyStatementCount, populated in GetS3TablesBucketsData from entry.Entry.Extended[s3tables.ExtendedKeyPolicy] via the new extractS3TablesPolicyStatementCountFromEntry - no extra RPC, the entry is already fetched for ExtendedKeyMetadata. - The badge reuses the existing .s3tables-bucket-policy-btn class, so it opens the same policy modal as the row's action button with no JS changes. * admin: don't let a failed policy GET open the door to an empty overwrite loadS3TablesBucketPolicy/loadS3TablesTablePolicy cleared the textarea, then unconditionally called loadPolicyTextareaIntoEditor() regardless of whether the GET actually succeeded - including when fetch() rejected or the response was not ok, silently logged to console only. That leaves the structured editor holding a legitimate-looking empty policy ({version, statements: []}), with the Editor tab active by default. If Save is then clicked, commitPolicyActiveTab() serializes that empty state into the textarea as `{"Version":"2012-10-17","Statement":[]}` - a non-empty string - before the "Policy JSON is required" guard ever sees it, so the guard passes and the transient load failure gets written over whatever policy was actually stored. Add s3tablesBucketPolicyLoaded/s3tablesTablePolicyLoaded, set true only once a GET has actually completed (ok, including a genuinely empty policy) and false on any failure path (fetch rejection or a non-ok response, which previously fell through silently). Both submit handlers now check the flag before touching the editor at all, and a failed load surfaces via alert() instead of only a console.error - the user previously had no visible indication the load had failed. Verified with a jsdom simulation driving the real rendered page against a stubbed fetch: a failed GET followed by Save now sends no PUT at all (previously it sent Statement: []); a successful GET followed by Save still PUTs the loaded policy unchanged. * admin: address code review findings on the policy editor 1. policy_editor.js: policyEditors is only pre-populated for 'create'/ 'edit'; every other `which` (bucket, s3tablesBucket, s3tablesTable) stays undefined until its first successful async load. Nothing in this file enforces that a page hide its Editor/JSON tabs and Add-statement button until that load completes - the S3 Tables policy modals don't - so a click in that window (e.g. Add statement, or switching to the JSON tab) threw "Cannot read properties of undefined (reading 'unparsed')". Add policyEditorState(which), which lazily initializes a default state, and route addPolicyStatement, the jsonTabBtn 'show.bs.tab' handler, commitPolicyActiveTab, and renderPolicyEditor through it. Verified with a jsdom simulation against a never-resolving fetch: the exact click threw on the pre-fix code and no longer does. 2. s3_buckets.templ: the bucket-policy Save handler checked the textarea for emptiness before calling commitPolicyActiveTab(), which is what actually serializes the structured Editor tab's fields into that textarea. A policy entered entirely through the Editor tab (the primary path - never touching the JSON tab) left the textarea at whatever it was at load time, so creating a new policy this way hit "Enter a policy document" and Save silently did nothing. Move the commit before the emptiness check, preserving the existing alert and early-return. Verified with a jsdom simulation: Add-statement then Save (no tab switch) now PUTs the entered statement; before the fix the same sequence never reached fetch(). 3. s3tables_buckets.templ / s3tables_tables.templ: the policy Editor/ JSON nav-tabs were missing the ARIA roles Bootstrap's own tab pattern expects (role="tab"/"tabpanel", aria-selected, aria-controls, aria-labelledby) - screen readers had no way to tell these were tabs or which pane went with which button. Added the standard Bootstrap 5 tab markup to both. * admin: guard policy load/save flows against overlapping requests 1. s3tables.js: loadS3TablesBucketPolicy/loadS3TablesTablePolicy had no protection against overlapping loads. Opening one bucket's (or table's) policy dialog and then another's before the first GET resolved let the late response write its document into the shared textarea and mark the dialog "loaded" while it was now targeting the second resource - a subsequent Save would then push the first resource's policy onto the second. Add a per-load monotonic sequence number (s3tablesBucketPolicyRequestSeq / s3tablesTablePolicyRequestSeq, the same pattern already used for the classic bucket-policy load in s3_buckets.templ); a response is only applied - textarea, loaded flag, editor state - if its captured sequence still matches the latest one issued. Verified with a jsdom simulation: bucket A's policy load (artificially slow) followed immediately by bucket B's (fast) previously left A's policy in the textarea once A's late response landed; it now correctly keeps B's. 2. s3_buckets.templ: the bucket-policy Save button lives outside the (initially hidden) editor wrapper, so it stays clickable while a load is still in flight - the existing policyRequestSeq guard only protects the *load* from a stale response, not Save from firing before any load for the current bucket has completed. Add bucketPolicyLoaded, reset before each GET and set only once the matching response lands, and check it at the top of the Save handler. Verified with a jsdom simulation: clicking Save immediately after opening the dialog, before a (deliberately never-resolving) GET settles, now sends no PUT; a normal load-then-save sequence still PUTs the loaded policy unchanged. * admin: address further code review findings on the policy editor 1. s3tables.js: loadS3TablesBucketPolicy/loadS3TablesTablePolicy only reset the JSON textarea when a new load starts; the structured editor kept showing the previously loaded resource's statements (Editor tab is the default active one) until the new fetch resolved. Call loadPolicyTextareaIntoEditor() against the now-cleared textarea immediately, so switching resources visibly resets the editor right away instead of only once its own load completes. Verified with jsdom: opening bucket A (loads fully) then bucket B (GET never resolves) no longer leaves A's statements visible in B's editor. 2. s3tables.js: deleteS3TablesBucketPolicy/deleteS3TablesTablePolicy had no loaded-state check, so a failed GET (which already blocks Save) left Delete fully able to remove the resource's stored policy sight unseen. Add the same s3tablesBucketPolicyLoaded/s3tablesTablePolicyLoaded guard Save already uses. Verified with jsdom: delete after a failed load now sends no DELETE; delete after a successful load is unaffected. 3. s3_buckets.templ: the bucket-policy Editor/JSON nav-tabs were missing the same ARIA roles already added to the S3 Tables policy tabs in an earlier round (role="tab"/"tabpanel", aria-selected, aria-controls, aria-labelledby) - this instance was out of scope for that review comment but is the same gap. Bootstrap's own tab.js already manages aria-selected on tab switch once the attribute exists, so no extra JS was needed. 4. s3_buckets.templ: neither the bucket-policy Save nor Delete handler guarded against a double-click, or against firing while the other was still in flight - two overlapping PUT/DELETE requests for the same bucket could land in either order. Add a shared bucketPolicyMutationInFlight flag: set (and both buttons disabled) before each fetch, cleared (and buttons re-enabled) on failure so the user can retry, left set through the existing success hide-and-reload path, and also reset when a new bucket's dialog opens so an abandoned in-flight request from a closed dialog can't leave the buttons stuck disabled. Verified with jsdom: double-clicking Save now sends exactly one PUT, and a Delete click while that PUT is still pending sends no DELETE. * admin: scope bucket-policy mutation completions to the bucket that started them 1. The previous round's fix reset bucketPolicyMutationInFlight whenever a new bucket's policy dialog opened, to avoid leaving Save/Delete stuck disabled if the modal was closed mid-request. That traded one bug for a worse one: if bucket A's PUT/DELETE was still in flight when the user opened bucket B's dialog, the reset let B's Save/Delete fire immediately, and A's completion handler - unaware anything had changed - would still hide the (now B's) modal and reload the page out from under whatever the user was doing with B, on success, or alert a message with no bucket context, on failure. Stop resetting on reopen, so a pending mutation for a previous bucket keeps this bucket's Save/Delete blocked until it settles (matches the "preventing overlapping mutations" the review comment describes). Instead, capture policyEditorBucket as targetBucket right before each fetch and compare it against policyEditorBucket again in the completion handler: the in-flight flag is always released so the buttons never get stuck, but the modal-hide/reload/alert only fire if this bucket is still the one showing; a stale completion for an abandoned bucket just logs to the console instead. Verified with a jsdom simulation: opening bucket B while bucket A's Save is still pending leaves B's Save button disabled and a click on it a no-op; once A's PUT resolves, B's button re-enables but no modal.hide()/reload() fires (previously both fired unconditionally). 2. bucketPolicyDeleteBtn had no bucketPolicyLoaded check, unlike Save - a failed GET blocked Save but left Delete free to remove a policy the client never actually saw (the same gap already fixed for the S3 Tables policy modals in an earlier round). Added the same guard, ahead of the confirm() dialog. Verified with jsdom: Delete after a failed load now sends no DELETE request. * admin: fix spelling mistake	2026-08-23 22:11:18 -07:00
Chris Lu	813c439af6	admin: regenerate the gzipped static mirror (#10859) ... The toast and modal changes edited static/js/ without rerunning gen_static_gz.go, so the embedded assets served to browsers still carry the old scripts and TestStaticGzMirror fails on master.	2026-08-21 10:10:10 -07:00
Chris Lu	8c7d714d5e	Lance catalog, and a Rust plugin worker to maintain it (#10841) ... * iceberg: skip tables the maintenance worker does not own A Lance dataset registered through the Lance namespace's Iceberg REST adapter arrives as an Iceberg table with a placeholder schema and table_type=lance, and keeps its fragments under data/ - the same subdirectory the orphan cleaner walks. Every fragment is unreferenced by the Iceberg metadata, so a maintenance pass deletes the dataset. Views share the entry shape and were only skipped because parsing their metadata happened to fail first. Gate the scan and the execution path on the entry actually being an Iceberg table. Maintenance is off by default, so this was latent rather than live. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tables: let a table declare a format the catalog does not interpret CreateTable accepted ICEBERG and nothing else. A Lance table has no metadata file for the catalog to maintain - the entry records a name and the dataset root, and the client owns everything under it - so accept LANCE, and carry the declared format on the entry instead of hardcoding it back on the way out. ListTables now reports format and metadataLocation, so listing a catalog that holds both kinds takes one pass rather than a GetTable per row. AWS omits both fields; adding them is additive. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tables: move the in-memory filer into its own package The Lance namespace tests need the same harness, and copying it would leave two of them to keep in step. Extracted as it was, plus the two fidelity gaps that only surface once a paginating caller uses it: ListEntries ignored startFromFileName and limit, so a caller that paginates re-read the first page until it hit its own cap and reported the same entry over and over, and GetFilerConfiguration was missing, which CreateTableBucket needs to resolve the buckets directory. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: serve the Lance Namespace REST spec A second catalog surface beside the Iceberg one, over the same table buckets: the namespace and table metadata operations, the $-delimited identifier codec, the spec's numeric error model, the directory-catalog marker files, and storage_options vending through the STS path the Iceberg catalog already uses. Listens on -port.lance, 9101 by default, and inherits ARNs, policies and tags from the storage layer, so a Lance table needs no second permission model. Identifiers map bucket / namespace / table onto the three levels Lance clients already use, which is why there is no warehouse selector to invent. The data plane needs Lance format support that does not exist in Go and answers with the spec's Unsupported code rather than a bare 404. Two things it deliberately will not do: create a table bucket as a side effect of creating a namespace inside one, since a bucket carries its own policy and lifecycle, and resolve an Iceberg table's location for a Lance client, which would hand it a table another engine owns. The design note this follows is in design-lance-catalog.md, including the .lance directory suffix it proposed and this does not implement. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * mini: give the Lance port the same treatment as the Iceberg one The flag was registered but nothing else knew about it, so mini would start the server without reserving its port, waiting for it, or saying where it is. Adds it to the startup service list, the conflict resolver, the gRPC allocator's reserved set, the readiness wait, the stop reporting and the banner. The admin server still takes only the Iceberg port, because there is no Lance page for it to link to. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: stop deregister and repoint from deleting the dataset Deregistering preserves data by definition, and this did the opposite: the catalog entry is the dataset directory, so DeleteTable took the files with it. Registering over an existing name had the same shape, destroying the dataset the name used to hold. Found by driving the running server rather than the in-memory filer, where both looked like success because the table did stop being listed. Deregistering is now a state on the entry - the marker file hides it, and declaring or registering the name again brings it back. Repointing a name at another dataset is an UpdateTable against the version token, so neither dataset loses files. Drop is left alone; it is the operation that does remove data. The storage endpoint now falls back to the advertised -ip where the Iceberg derivation gives up. An Iceberg client brings its own s3.endpoint and advertising the wrong one hijacks it, but storage_options is the only place a Lance client learns where the store is, and without it object_store quietly talks to real AWS. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tables: refuse to create a table over one of another format Creating a table that already exists is idempotent, and that path returned the existing table without looking at its format. A Lance declare over an Iceberg table answered 200 and handed back a directory Iceberg owns, so the client would write its dataset on top. The view check immediately above it already guards the same class of collision. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tables: let a table bucket hold a format other than Iceberg The S3 door validated every object written into a table bucket against Iceberg's file layout, so a Lance client could not write its dataset at all: it got 403 on data/*.lance, on _versions/, and on the _transactions/ directory it turned out to write as well. Table buckets were only neutral containers by intention; in practice they were Iceberg-shaped and enforced as such. The allowed set is now the union of what the supported formats write, because the validator runs where the table's format is not in hand. Underscore-prefixed directories are treated as belonging to the format, since enumerating them means guessing at the next one - _transactions is exactly the one this missed - and their contents are checked only for traversal. Iceberg writes none of them, so it loses nothing. Marker files at the table root are admitted too, which the namespace/table/dir/file shape had rejected as too shallow. Describe also honours the request-body spellings of with_table_uri, load_detailed_metadata and check_declared. The spec puts them in the query string, but real clients send both. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: record what the implementation found The table bucket being an Iceberg-shaped container, enforced at the S3 door, was the premise this design never questioned and the one that had to change before anything worked end to end. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * iceberg: prove the data loss the foreign-format guard prevents The guard landed with a unit test for the predicate and nothing showing what it saves. These seed what the Lance namespace's Iceberg REST adapter actually leaves behind - an Iceberg table with a placeholder schema and table_type=lance whose directory holds a Lance dataset - and assert both halves: orphan collection does flag the dataset's fragments, because the Iceberg metadata beside them references nothing, and the scan never reaches the table. An ordinary Iceberg table in the same shape is still scanned, so the guard is not just skipping everything. Confirmed against a running gateway first: our Iceberg catalog accepts the adapter's registration, and a real Lance client then writes a dataset into that table's location. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tablestest: make the in-memory filer safe to race against Two gaps that only matter once a test drives concurrent writers, which is what an exclusive create has to be tested with: the entry map had no lock, and CreateEntry ignored O_EXCL entirely, so both writers of the same name would have won and the test would have passed while proving nothing. The BeforeUpdate hook runs before the lock is taken. Its whole purpose is to land a competing write in a handler's read-to-write window, and that write needs the lock the hook would otherwise be holding. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: make the namespace an external manifest store Lance commits a version by writing _versions/{v}.manifest with put-if-not-exists. The S3 layer in front of this same filer evaluates If-None-Match by looking the entry up and then writing without a precondition, so two writers can both pass the check and one commit is lost. The filer itself has the primitive: CreateEntry with o_excl. Adds the four version operations a Lance client actually calls - create, list, describe and batch-delete - recording one entry per version under _lance_versions/, and advertises managed_versioning so the client routes its commits here. Reserving a version is the exclusive create, so exactly one of several racing writers wins and the rest rebase. Off by default, behind -lance.managedVersioning. Turning it on moves where a table's version history lives, and a reader that does not come through this namespace no longer sees all of it; that is the operator's call, not a default. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: record what managed versioning does and does not reach The first commit through a namespace-backed store works and is recorded the way the protocol specifies. Later commits do not, because lance 4.0.0 refuses put_if_exists on that path in its own code, so the feature is capped upstream rather than here. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * test: integration tests for the Lance namespace Everything this surface got wrong so far - a deregister that deleted the dataset, an S3 door that refused every Lance file, a version reservation that could not actually be exclusive - passed against an in-memory filer first. So these run against a live gateway, and where the claim is about data they check storage rather than visibility. Five Go tests on the shared harness: namespace and table lifecycle including that deregister keeps the bytes and drop removes them, that a Lance client cannot resolve or declare over an Iceberg table, that a Lance dataset's files get past the table-bucket layout guard while junk still does not, and that eight writers racing for one version produce exactly one winner. One Docker-gated test drives the real Lance client, which is the only way to check that the location and storage_options the namespace vends are between them enough to write and read a dataset. It overrides the endpoint with the container's view of the same gateway, because the shared harness binds a wildcard address and so vends none. The harness gains a Lance port and turns managed versioning on; the flag touches nothing outside that surface. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tables: a directory with no namespace metadata is a missing namespace Three callers resolved a namespace by reading its metadata attribute and each tested only for a missing entry, so a directory that carried no metadata came back as an internal error saying "attribute not found". Creating a table under a namespace that does not exist answered 500. Collapses the three copies into one helper that reports both conditions as absent, which is what they are: a directory without namespace metadata is not a namespace. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * iceberg: stop reporting storage-layer refusals as server faults writeManagerError recognised a missing table bucket and sent everything else to 500, so a missing namespace, a duplicate name and a commit conflict all reached the client as InternalServerError with nothing to act on. Creating a table in a namespace that does not exist is the case that turned up: 500 where the spec wants 404 NoSuchNamespaceException. Maps the storage error types onto the exception names this package already uses, and keeps the existing bucket message, which explains how to select a table bucket. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * iceberg: skip a foreign-format table by name, not by failing to parse it A table the namespace created as LANCE carries no Iceberg metadata, so the worker skipped it only because the parse failed, and logged that as damaged metadata. The catalog records the format on the entry and this never read it. Reading it turns an accident into a decision, and separates a mixed catalog from a corrupt one in the logs. The property check beside it still covers the other shape: a real Iceberg table wearing table_type=lance, which is what the Lance namespace's Iceberg REST adapter writes. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: answer whether a Lance table needs maintenance It does, and index optimization has no Iceberg equivalent: rows written after an index was built are not covered by it, so a vector search quietly misses them. None of the three jobs can run in the Go worker, and there is no useful subset, because deciding what an old version still references means parsing Lance manifests. Version cleanup at least has an answer that needs nothing from us - Lance can enable it on the dataset itself. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: the Lance maintenance worker is a plugin worker, in Rust Framing it as a sidecar was wrong. plugin.proto already defines a language-agnostic gRPC contract for external maintenance workers, and "weed worker -admin=..." is the Go reference implementation of it from outside the admin process. seaweed-volume already compiles protos out of weed/pb with tonic_build, so a Lance worker is that build plus plugin.proto and the lance crate. Scheduling, retries, dedupe, progress and the admin settings page all come from the protocol: a worker that answers RequestConfigSchema with a descriptor gets its configuration form rendered without a line of Go. The data plane is the part that genuinely does need a process answering HTTP, and this had the two conflated. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * seaweed-worker: Rust plugin worker workspace, with Lance as the first one plugin.proto is language-agnostic and the Rust toolchain was already in the tree, so a Lance maintenance worker needs no new integration surface: core is the contract and nothing else, and a worker crate beside it supplies handlers and a binary. A second worker is a new member here rather than a fork of the protocol, which is why this is seaweed-worker and not seaweed-lance-worker. Verified against a running admin: it connects, is accepted, and admin prefetches descriptors for lance_compact, lance_optimize_indices and lance_cleanup_versions, so their settings pages render from the Rust side without a line of Go. The stream stays up across heartbeats. The job bodies are stubs that report failure. Doing the work means adding the lance crate and opening the dataset, and claiming success before that would be worse than saying so. Two things running it caught that reading the proto did not: the admin address has to be converted to the gRPC port the way pb.ServerToGrpcAddress does, or the dial fails as an h2 frame error; and the generated field names differ from the Go ones in several places, so JobCompleted carries success rather than a state enum. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: implement compaction Detection lists tables from the namespace, opens each one, and proposes a job for any with more fragments than the policy allows; opening a dataset reads its manifest and not its data, so a sweep stays cheap. Execution re-resolves the table rather than trusting what detection saw - it may have been repointed, and the vended credentials expire - then compacts and reports the fragment counts either side. Verified against a live gateway: a twelve-fragment dataset became one fragment with all twelve rows intact. The test drives the handler directly and skips unless WEED_LANCE_NAMESPACE names a namespace, the way the Go integration tests skip without Docker. Running it turned up a gap the design had not: a gateway without STS vends no credentials at all, so the worker could not open anything and detection quietly proposed nothing. --access-key/--secret-key are the fallback, and whatever the namespace vends still wins over them. Two API assumptions did not survive contact either. Datasets open through DatasetBuilder::with_storage_options, not ReadParams, and lance 10's ObjectStoreParams has no storage_options field at all. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: implement index optimization and version cleanup Index optimization is the job with no Iceberg equivalent: rows appended after an index was built are invisible to a search of it until this runs. Detection reads num_unindexed_rows from each index's statistics and proposes a table once more rows sit outside its indices than the budget allows; a table with no indices is skipped, which is different from one whose indices have fallen behind. Cleanup applies a retention window, refusing rather than silently dropping a tagged version, and leaving unverified files alone because they may belong to a commit still in flight. Both verified against a live gateway: 512 uncovered rows became 0, and a fourteen-version table lost its old ones. Each test now seeds what it needs, including building an IVF_PQ index and appending rows outside it. The first version of these depended on state a script had left, so the second run found the work already done and asserted nothing - a test that passes by doing nothing is worse than no test. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: answer an empty catalog with an empty list, not null ListAllTables built its result from a nil slice, so a namespace holding no tables answered {"tables":null} on a field the spec marks required. A generated client may decode that differently from an empty list. Found running the namespace on a dev box, where the catalog was empty. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: give Lance maintenance its own scheduler lane Lane assignment is a hardcoded map, so the three lance_* job types fell through to the default lane. That lane serialises its work under the cluster admin lock because volume management shares global state, which would queue a table's compaction behind volume balancing for no reason - Iceberg has its own lock-free lane for exactly this. Adds the lane, maps the three job types to it, and puts it in the sidebar beside Iceberg and Lifecycle. The lane routes were already generic, so only the nav was hand-written. The lane-coverage test spelled out the three known lanes, so a fourth failed it. It now checks against AllLanes(), which is the property it was reaching for and does not need editing next time. Found by connecting the Rust worker to a real admin: it registered fine and its job types were known, but they were filed under "default" and had no page. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: log what detection saw "Detection proposed nothing" and "the worker could not read the table" look identical from the admin side, and the second is what a missing credential produces. One line per table separates them. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: fix a leaked heartbeat and a silent reconnect loop spawn_heartbeat returned a handle to an empty task rather than the ticker it had just spawned, so aborting it aborted nothing and every reconnect left another heartbeat running against a dead channel. A stream that admin closes cleanly is not an error, but reconnecting in silence hides why. Two workers sharing an id evict each other forever and the log shows nothing but a login every five seconds - which is exactly how this presented on a dev box, and it took a look at the admin's own log to see it. The message now names the id to check. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: a namespace cannot be created without its parent Storage keeps a namespace's parts flattened, so creating "a.b" with no "a" was accepted and left an intermediate that only existed inside a name. Listing derives child names by slicing those parts, so it reported "a", while describe and exists on "a" both answered 404 - a client walking the tree got a 404 on something the listing had just handed it. The spec asks for NamespaceNotFound when the parent is missing, which is also what keeps listing and describe telling the same story. Namespaces created through the S3 Tables API still bypass this, so listing keeps deriving intermediates rather than hiding whatever is already there. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: say why a non-Iceberg table shows no schema The table pages read Iceberg metadata for schema and snapshots, and a Lance table has none, so both panels rendered "No schema available" - which reads as an empty table rather than a table this page cannot describe. The dataset behind the one that prompted this holds 1024 rows. The format is already on the entry and shown two rows above, so the empty states now use it: the catalog records where a LANCE table lives, not what is in it. Reading the schema for real needs Lance format code, which is the same wall as the data plane. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * seaweed-worker: run rustfmt over the workspace Committed the crates unformatted, so `cargo fmt --all --check` failed on files nothing had touched since. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * plugin: let a worker report what it saw about an object Admin cannot read a Lance table: it knows where the dataset lives and nothing else, so the details page had a location and two empty panels. The worker already opens every dataset during detection to decide whether it needs compacting, so it knows the schema, the row count and the fragment count at that moment. It just had no way to say so. Add a WorkerObservations body to the worker stream. Admin caches the last observation per object and serves it back, timestamped, for display; nothing schedules from it. The Lance compaction sweep reports what it opened, and the S3 Tables details page fills its schema panel from the cache when it has no metadata of its own, badged with when the worker looked and which worker it was. Nothing about this is Lance-specific past the reporting side, which is the point: any format admin cannot parse can describe itself the same way. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: record the observation channel Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * plugin: ask a worker for sample rows of a table admin cannot read Browse Data reads an Iceberg table's Parquet files directly, so it shows real rows. For a Lance table it showed "Table has no Iceberg metadata" and an empty grid, because there is no Go Lance reader and never will be one worth maintaining. The worker has the reader. Add RequestObjectPreview / ObjectPreviewResponse to the stream, mirroring the config-schema round trip that already exists, and give the Rust worker a PreviewProvider that scans the dataset and formats the rows with Arrow's own formatter, so a vector column reads as a vector. Admin picks the worker from the observation store: whichever one last described this table is the one that can read it. Unlike an observation the rows are not cached. They are the table's data rather than a description of it, and a copy sitting in admin would be both stale and nobody's business. The page fetches on load, bounded at 200 rows and a 15 second round trip, and drops the snapshot and data-file panels that only mean something for Iceberg. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: record the preview channel Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * test: disable the lance listener when two gateways share a host * test: keep AllocatePorts away from the lance default port * s3tables: let a table bucket declare the format it holds A bucket is a catalog, and a catalog serves one protocol. Format was recorded per table, so nothing could answer "where do I point a client at this bucket" without opening a table first, and an empty bucket had no answer at all. CreateTableBucket takes an optional format, stored with the rest of the bucket metadata and returned by Get and List. Empty means ICEBERG, which is what AWS S3 Tables serves and therefore what an SDK that has never heard of the field means. CreateTable refuses a table of another format, and CreateView refuses outright in a bucket that is not Iceberg, since a view is Iceberg metadata. Buckets that already exist carry no declaration and keep accepting anything, so nothing is migrated and nothing that worked stops working. The Lance namespace declares LANCE for the buckets it creates, which is what stops one of them being described to a client as an Iceberg catalog. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: take the Lance port the way it takes the Iceberg one The UI cannot name the endpoint that serves a Lance bucket without it, and every format-aware page below needs to. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: show which format a table bucket holds The bucket list printed an Iceberg endpoint for every bucket, including ones holding Lance datasets, where that endpoint serves nothing. It was the most visible place the UI assumed one format. The list gains a Format column and its endpoint column follows the bucket's declaration. The banner names both endpoints rather than asserting everything is Iceberg, and says so only for the servers that are actually running. Create Bucket picks a format with two cards rather than a dropdown, since what matters is not the name but which clients can read the result, and the endpoint under them updates as you choose so the operator leaves the modal knowing where to point one. A bucket from before the declaration existed shows "unset" in an outline badge, explained on hover. It is a fact about the bucket's age, not a fault, so nothing nags about it. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: carry the bucket's format into the pages inside it Namespaces and tables are reached through a bucket, so both now say which catalog they belong to rather than making you go back up to find out. The tables list gains a Format column and a Rows column filled from what a worker last observed, since for a format admin cannot read that is the only row count there is; a table nothing has looked at shows a dash, not a zero. Create Table stops offering a choice the bucket has already made: in a declared bucket the format is fixed and says why, and only an undeclared one still offers both. Before this the select had exactly one option, hardcoded, which made a Lance table impossible to create from the UI at all. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: let the table page speak the table's own format Partitions and Snapshot History are Iceberg's shape. Rendering them empty for a Lance table reads as a fault; a Lance table has neither, and says so by not showing them. In their place is a Versions panel, which is what that format calls its history, carrying the worker's timestamp so it is clear the numbers are a cached look rather than something read live. The breadcrumb carries the format badge, so the page names what it is looking at before you read a panel and wonder why it is empty. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: show how to connect to either catalog, and group the two format workers The client examples on the buckets page were Iceberg's alone, so the one thing an operator wants after creating a Lance bucket - what to type to reach it - was not written down anywhere in the UI. Both formats now get a pair of snippets, and only for a server that is running. In the Workers menu, Iceberg moves below Lifecycle so it sits next to Lance: the two table-format workers together, the two cluster-wide ones above them. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * shell: create a table bucket of either format s3tables.bucket -create takes -format, so a Lance bucket can be made without going through the UI. The integration harness passes it too: its Lance tests were creating Iceberg buckets and getting away with it only because nothing checked. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: record that a bucket declares its format Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: drop managed versioning; the store already orders commits The namespace offered itself as an external manifest store, so that a commit could reserve a version through a real put-if-not-exists. That was designed around a gateway that no longer exists: If-None-Match: * is reduced to a filer WriteCondition and evaluated at the object's owner under its per-path lock, or under the object write lock on the fallback path. Sixteen writers racing one fresh key get a single 200 and fifteen 412s, every time. Lance needs nothing else. commit_handler_from_url hands every s3:// dataset a ConditionalPutCommitHandler, which puts with PutMode::Create, which object_store sends as If-None-Match: *. So the feature solved a problem this store does not have, while moving a table's version history out of the dataset and into the catalog - and lance could not use it past the first commit anyway, since its own namespace-backed store answers "put_if_not_exists is not supported" to the second. The version operations answer Unsupported with the rest, managed_versioning is false, and the flag is gone. In place of the reserve-once test there is one that races eight writers at the manifest key through S3, which is the path a commit actually takes. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: honour the version floor, the slot limits, and a shutdown Five findings from review, all of them things the worker claimed to do and did not. The version floor was checked when a cleanup job was proposed and ignored when it ran, so a table whose versions had aged past the retention window in between could be taken below the count the operator asked to keep. Execution now computes the floor itself and passes it as before_version; CleanupPolicy ANDs its clauses, so a version has to be both too old and below the floor to go. Both settings are clamped to the range the form offers, since Duration::hours panics on a large enough value and a negative min-versions wraps to a huge usize. Admin's shutdown was answered by returning from the stream, which the reconnect loop read as a healthy close and logged straight back in: the worker could not be stopped. serve_once now says which of the two happened. The advertised concurrency limits bounded nothing - every request spawned a task - and the heartbeat reported zero slots in use whatever was running. Both now go through semaphores sized from the limits, with the permits held for the life of the request and reported in the heartbeat. A namespace call had no timeout, so a gateway that accepted the connection and went quiet held a detection slot forever. And one table whose stats could not be read failed the whole sweep, losing the proposals for every table already scanned; it is now skipped and warned about, like a table that cannot be opened. The tests drove one shared catalog concurrently, which is why one of them asserted "no proposals at all" and passed by luck. They now take a lock and judge only their own tables. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: fix the review findings on the format-aware pages The endpoint hint in Create Bucket built its HTML by concatenating the bucket name the operator is typing, so a name like <img onerror=...> ran in the admin origin as they typed it. It is built from DOM nodes now. A preview reply looked its channel up under the lock and then sent outside it, which Shutdown can close in between: a Gosched in that gap panics with "send on closed channel" every time. The send now happens under the lock. Observations were looked up by path alone, so a table dropped and remade in another format at the same path was described by the observation left behind. Lookups now have to agree on the format. Also: the Lance namespace caps a request body rather than reading whatever arrives; the details action no longer says "Iceberg" over a Lance table; mini stops advertising a catalog port when it is not running S3; a format whose server this cluster does not run cannot be picked in the modal or accepted by the API, since a bucket nothing can reach is not worth creating; and the unused catalogPortFor helper is gone. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: let the control stream use mTLS The channel was hardcoded to http://, so off loopback the stream carried preview rows and execution commands in the clear - and a cluster with grpc TLS turned on would refuse the worker outright. --tls-ca, --tls-cert and --tls-key take the same certificates the Go worker reads from the [grpc.worker] section of security.toml, and must be given together: a CA on its own would quietly mean one-way TLS, which a mutual setup rejects anyway. Without them the stream stays plaintext, which is what the Go worker also does when nothing is configured. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: answer null properties rather than an empty map The catalog does not keep a table's properties. Declare echoed the request's back and describe answered {}, both of which claim they were stored and are empty. Null says the catalog does not keep them, which is what the spec distinguishes and what is true here. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: test the slot accounting The heartbeat reporting and the waiting are the two things the semaphores are for, and neither is observable from outside without catching a sweep mid-flight. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * test: fix the mixed-format catalog test, and name the binary it drives The integration suite passed locally and failed in CI on TestLanceRefusesIcebergTables. Both were right: CI builds the binary first, my tree had one from the day before, so locally the test drove a gateway with no format enforcement at all. The test itself no longer holds as written. It made a bucket, put an Iceberg table in it, and checked the Lance surface hid it - but a bucket that declares LANCE now refuses the Iceberg table outright. The invariant still matters from the other side, so it starts from an Iceberg bucket instead: Lance must not describe or list a table whose format it does not serve, and must refuse to declare one beside it. The harness now prints which weed binary it is about to run and when that was built. `make test` rebuilds first; a plain `go test` will happily drive a weeks-old binary and report a pass for code it never ran, which is exactly what happened here. Also make the row-limit conversion in the preview request explicitly bounded: CodeQL flagged the int-to-int32 conversion, and clamping by reassignment beforehand is not a form it recognises. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: prove concurrent commits are kept, and preselect the only format on offer Two more from review. The commit test asserted that exactly one writer wins the conditional PUT, which is the mechanism, not the claim. The claim is that nothing is lost: the losers see the conflict, rebase and commit again. So there is now a test that has eight writers append to one dataset at once and counts the rows afterwards - all eight batches survive. That is also the sequence managed versioning could not finish, since its store refuses the second commit outright. And when Iceberg's endpoint is not running, the format picker offered two options with neither selected, so Create Bucket submitted no format at all, fell back to ICEBERG, and was refused by the guard added last round. Lance is preselected when it is the only format this cluster serves. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * Clamp the remaining worker settings, and bootstrap buckets in a served format Compaction and index optimization read their thresholds and cast straight to usize and u64, so a negative arrives as an enormous number and turns the threshold into "never": compaction and reindexing both go quiet with nothing to say. The cleanup job was fixed last round; these are the same bug. Clamped to the values that stay meaningful rather than to what the form offers - zero uncovered rows is a real setting, meaning reindex as soon as anything is not covered, so the floor there is zero and not the form's thousand. mini pre-creates the buckets named by -tableBucket, and did so without a format, which now means Iceberg. Started with the Iceberg endpoint off and the Lance one on, that left buckets nothing could reach and which refused every Lance table. It takes the format from the endpoint that is actually running, and creates nothing when neither is. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3: allow-unordered is a listing parameter, not an unimplemented subresource The guard that stops a bucket GET with an unknown subresource from being answered with a listing does not know about allow-unordered, so it answers 501 NotImplemented - to a parameter the listing handlers already read and already validate against delimiter. This is why test_bucket_list_unordered and test_bucket_listv2_unordered fail in the Ceph s3-tests suite. They fail on master too; this is not a Lance change and can be taken on its own. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm	2026-08-19 22:59:56 -07:00
ruriandChris Lu	340f9951ac	admin: make paths relative (#10709) ... * admin: make paths relative * admin: make filer browser link and nav path checks prefix-relative * admin: add isCurrentPath and currentPathStartsWith helpers --------- Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-08-10 22:31:55 -07:00
Chris Lu	e56a1c4c05	admin: pre-gzip embedded static assets, add cache headers (#9918) ... The admin UI served embedded static files uncompressed and without cache headers: embed.FS has zero mod times, so no Last-Modified, no ETag, no 304s -- every page load re-downloaded ~700KB of css/js in full, which gets painful over slow or tunneled links. Gzip the static tree at generation time (go generate ./weed/admin) and embed only the compressed mirror, shrinking the binary ~1.5MB. The handler hands the pre-compressed bytes to gzip-capable clients, decompresses for the rest, and sets Cache-Control, per-variant content-hash ETags and Vary so repeat loads revalidate with a 304. bootstrap.min.css goes 232KB -> 30KB on the wire. A drift test keeps static_gz/ in sync with static/.	2026-06-10 12:54:36 -07:00