mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-11 09:00:45 +02:00
c2cdefd06d89f4510dfe06508eee7ea5aa49d607
85
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris Lu
|
34bb444f33 |
test: drive the Lance namespace with Spark (#10864)
* test: drive the Lance namespace with Spark
The counterpart of catalog_spark, which does this for the Iceberg REST
catalog. Spark is the engine most likely to be pointed at a lakehouse,
and it reaches the Lance catalog through the connector's DSV2 catalog -
org.lance.spark.LanceNamespaceSparkCatalog with impl=rest - over the same
routes every other client uses.
SHOW NAMESPACES -> ['`sparklance-lcephd80`.ml']
SHOW TABLES -> ['sparklance-lcephd80$ml$embeddings']
count -> 3
filtered -> [(2, 'two'), (3, 'three')]
count after a second commit -> 4
The second insert is there on purpose: a store that cannot order commits
fails on the second one, not the first.
Two things the run settled that were guesses beforehand. CREATE TABLE
works, because the connector declares through the namespace and writes the
data itself rather than pushing Arrow at the server. And SHOW TABLES
returns the namespace's own identifiers - bucket, namespace and name
joined by the delimiter - not bare Spark table names.
Credentials go under the catalog's storage.* prefix, which is handed to
lance as object_store options; a gateway without STS vends none, the same
trap the LanceDB suite documents.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: verify the Lance table bucket was actually created
weed shell prints a command's own failure and still exits 0, so the harness
would go on to blame Spark for a bucket that was never made.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: bound the Docker probe
An unhealthy daemon makes docker version hang, and the probe runs before the
test has a timeout of its own.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: keep the ivy cache under the user's cache directory
It is mounted into a container running as root, so a shared temp path lets
another local user pre-create it and choose what Spark loads.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: assert the vector column's type, not only its name
A column that came back as array<double> or array<string> would still be
called vector and still pass.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: read the dataset off its location for real
The catalog being optional is the property that lets duckdb and pandas read
these tables; it was asserted in a comment and printed, never exercised.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: do not persist credentials in the Spark Lance checkout
The job only uploads a log on failure; nothing in it pushes.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: say the hosts in the README are placeholders
The suite passes dynamically allocated host.docker.internal ports.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
|
||
|
Chris Lu
|
5e7ab43ddd |
test: read Lance tables from DuckDB (#10866)
* test: read Lance tables from DuckDB
The LanceDB and Spark suites go through the catalog. DuckDB does not: its
lance extension reaches the data over S3 with no namespace involved, which
exercises the other half of the design - a table bucket's layout is a
valid Lance dataset directory, so a table stays readable when the catalog
is not in the path.
scan_rows=128
scan_columns=id,title,vector
filtered_rows=5
nearest=1,0,2
It also pins the one place the layout costs us. DuckDB's replacement scan
recognises a dataset by a .lance path suffix, and tables created through
this catalog deliberately have none: the catalog entry is the dataset
directory, a table name may not contain a dot, and a suffix would leak
into ARNs and policies. So __lance_scan is the way in, and the bare
SELECT ... FROM 's3://...' form does not see these tables.
The test asserts both halves - a suffixed path is read, a suffix-less one
is not - so if the extension ever recognises a bare directory, it fails
and says to update the documentation rather than leaving it wrong.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: require the catalog error from the suffix-less read
Any failure satisfied the old check - a missing extension, bad credentials,
an unreachable endpoint - so the assertion could pass without the
replacement scan ever classifying the path.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: verify the Lance table bucket was actually created
weed shell prints a command's own failure and still exits 0, so the harness
would go on to blame DuckDB for a bucket that was never made.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: bound the Docker probe
An unhealthy daemon makes docker version hang, and the probe runs before the
test has a timeout of its own.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: order the aggregates the assertions read
string_agg over an unordered relation may return the names, and the vector
search's ids, in any order, so the expectations could fail on a run where
nothing changed.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: do not persist credentials in the DuckDB Lance checkout
The job only uploads a log on failure; nothing in it pushes.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
|
||
|
Chris Lu
|
0dfaa103d0 |
test: take a table through its whole life, for Iceberg and Lance (#10862)
* lance worker: share the integration tests' scaffolding The recorder that keeps what a handler sent, the config builder and the storage-option fallback all lived inside compaction.rs, so a second test binary would have had to copy them. They move to tests/common. The fallback now reads AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_ENDPOINT_URL from the environment, defaulting to what it used before. A harness can then point these tests at a gateway that checks what it is given rather than one that accepts anything. * lance worker: maintain one named table, for a harness to drive Compacts and cleans up whatever WEED_LANCE_TABLE names, through the handlers' own detect-then-execute path: a proposal the worker would not have made is not one worth running. The existing tests seed the tables they check. This one deliberately does not, so a harness that has already written a table and knows what is in it can have the real handlers maintain it and then read it back. * test: take a table through its whole life, for Iceberg and Lance Created in the catalog, filled by a real client, maintained by the worker, read again, dropped. The step nothing was checking is the read after maintenance: compaction once rewrote every dictionary-encoded column onto a single value and shipped, because the maintenance tests were thorough about sequence numbers, manifest entries and metadata versions and none of them opened the parquet file the worker had just written. So the assertion is a tally - row count, the cardinality of each dictionary-encoded column, and an md5 over whole rows - taken before maintenance and again after, required to be equal. The cardinalities name the failure that happened; the digest catches a rewrite that keeps every column's cardinality and hands the values to the wrong rows. A compaction that merged nothing fails rather than passes, or the read afterwards is checking a file the worker never wrote. The Iceberg half runs two clients. DuckDB is the one the corruption was reported against and the only one here that writes the deprecated PLAIN_DICTIONARY encoding, which parquet-go normalizes away on write, so a Go writer cannot produce it. PyIceberg writes the modern spelling. Pinning parquet-go back to v0.30.1 fails the DuckDB half and passes the PyIceberg one, which is why both are here. Lance maintenance lives in the Rust worker, so it runs there where cargo is installed and through the two lance calls those handlers wrap where it is not. WEED_LANCE_MAINTENANCE picks one instead of letting the test guess. * ci: run the table lifecycle tests CI maintains the Lance table through the lance library rather than the worker: a cold build of the lance crate costs more than the glue it would be checking, and the worker's own tests cover its handlers. The suite drives the Iceberg maintenance worker, so a change to it now triggers this workflow too. * test: let the lifecycle harness fail instead of skipping Setup failures all exited zero, so a cluster that would not come up, or a port allocation that lost, reported a green run for code nothing had executed. That is the failure mode this whole directory exists to close, and it was in the harness itself. Only a checkout without a weed binary skips now, and it runs the tests so each one says so rather than the package quietly passing. Everything else fails. The filer existence probe gets a deadline while I am here: it ran without one, so an unresponsive filer would hang the suite past every timeout the clients have. * test: make the lifecycle checks check what they claim to Three of them could pass without having looked. The DuckDB skip matched "syntax error", "not implemented" and "Failed to load" anywhere in the output, in any phase. A parse error in the SQL this test generates, or a refusal from our own catalog, would have taken the only coverage of the PLAIN_DICTIONARY encoding out of CI and left it green. It now matches the extension failing to install, and only in the phase that installs it. Everything past LOAD is ours and fails. The digests covered id, category and value. Compaction rewrites the whole row, so a defect confined to ts, or to a Lance vector, changed nothing either side of maintenance. Every persisted column goes in now, ts as microseconds so no timezone sits between the two runs. The Lance drop check caught every exception as proof the dataset was gone. pylance turns credential and transport failures into the same ValueError, so it only accepts the message that means not found. * docs: say up front which maintenance path the Lance half takes The opening summary said the worker maintains both tables. It maintains the Iceberg one always and the Lance one only where cargo is installed, which is not what CI does. |
||
|
Chris Lu
|
83753ccdad |
test: drive the Lance namespace with LanceDB (#10850)
* test: drive the Lance namespace with LanceDB
The Iceberg catalog is checked against Spark, Trino, ClickHouse, Doris,
Dremio and RisingWave. The Lance one had only its own reference client,
which is the same thing as checking it against ourselves.
LanceDB connects with connect_namespace("rest", ...), which speaks the
routes this catalog implements, so the suite exercises the protocol rather
than our idea of it: list the catalog, open a table through it, read the
schema, run a vector search and a filtered scan, create a table, and read
the same dataset straight off its URI with no catalog at all.
table_names -> ['lancedb-p0guidmm$ml$embeddings']
open_table -> 64 rows
search -> [1, 0, 2]
create_table -> 4 rows, listed by the catalog
direct read without the catalog -> 64 rows
Seeding is pylance, because the namespace records where a table lives and
does not carry its data. That split is the design rather than a limit of
the test.
One interop note the test encodes: a gateway without STS vends
storage_options carrying an endpoint and a region but no credentials, and
LanceDB uses what the namespace vends on some paths. The container gets
credentials in its environment as well, which is what a deployment without
STS would do.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: pin the LanceDB client, and index before searching
Three from review.
The client's dependencies were unpinned, so an unrelated upstream release
could change what an old commit reproduces. Pinned to the versions this
suite was verified against; the client is as much the thing under test as
the server.
The search was called ANN and was not: without an index LanceDB scans.
The test now builds an IVF_PQ index over 1024 rows first, which is worth
more than the wording fix - an index writes into a directory of the table
that the S3 door has to admit, and that guard has refused a Lance
directory before. It builds, covers all 1024 rows, and searches.
The assertion moved with it. Demanding the exact nearest neighbour was
right for a brute-force scan and wrong for a quantized index, which
answered 0 as readily as 1; both are correct, so the check is now the
neighbourhood.
And the pushdown check accepted any failure. It now requires the refusal
to be the catalog's Unsupported and requires that nothing was left behind,
or, when the client falls back, that the table is complete.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
|
||
|
Chris Lu
|
8c7d714d5e |
Lance catalog, and a Rust plugin worker to maintain it (#10841)
* iceberg: skip tables the maintenance worker does not own A Lance dataset registered through the Lance namespace's Iceberg REST adapter arrives as an Iceberg table with a placeholder schema and table_type=lance, and keeps its fragments under data/ - the same subdirectory the orphan cleaner walks. Every fragment is unreferenced by the Iceberg metadata, so a maintenance pass deletes the dataset. Views share the entry shape and were only skipped because parsing their metadata happened to fail first. Gate the scan and the execution path on the entry actually being an Iceberg table. Maintenance is off by default, so this was latent rather than live. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tables: let a table declare a format the catalog does not interpret CreateTable accepted ICEBERG and nothing else. A Lance table has no metadata file for the catalog to maintain - the entry records a name and the dataset root, and the client owns everything under it - so accept LANCE, and carry the declared format on the entry instead of hardcoding it back on the way out. ListTables now reports format and metadataLocation, so listing a catalog that holds both kinds takes one pass rather than a GetTable per row. AWS omits both fields; adding them is additive. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tables: move the in-memory filer into its own package The Lance namespace tests need the same harness, and copying it would leave two of them to keep in step. Extracted as it was, plus the two fidelity gaps that only surface once a paginating caller uses it: ListEntries ignored startFromFileName and limit, so a caller that paginates re-read the first page until it hit its own cap and reported the same entry over and over, and GetFilerConfiguration was missing, which CreateTableBucket needs to resolve the buckets directory. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: serve the Lance Namespace REST spec A second catalog surface beside the Iceberg one, over the same table buckets: the namespace and table metadata operations, the $-delimited identifier codec, the spec's numeric error model, the directory-catalog marker files, and storage_options vending through the STS path the Iceberg catalog already uses. Listens on -port.lance, 9101 by default, and inherits ARNs, policies and tags from the storage layer, so a Lance table needs no second permission model. Identifiers map bucket / namespace / table onto the three levels Lance clients already use, which is why there is no warehouse selector to invent. The data plane needs Lance format support that does not exist in Go and answers with the spec's Unsupported code rather than a bare 404. Two things it deliberately will not do: create a table bucket as a side effect of creating a namespace inside one, since a bucket carries its own policy and lifecycle, and resolve an Iceberg table's location for a Lance client, which would hand it a table another engine owns. The design note this follows is in design-lance-catalog.md, including the .lance directory suffix it proposed and this does not implement. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * mini: give the Lance port the same treatment as the Iceberg one The flag was registered but nothing else knew about it, so mini would start the server without reserving its port, waiting for it, or saying where it is. Adds it to the startup service list, the conflict resolver, the gRPC allocator's reserved set, the readiness wait, the stop reporting and the banner. The admin server still takes only the Iceberg port, because there is no Lance page for it to link to. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: stop deregister and repoint from deleting the dataset Deregistering preserves data by definition, and this did the opposite: the catalog entry is the dataset directory, so DeleteTable took the files with it. Registering over an existing name had the same shape, destroying the dataset the name used to hold. Found by driving the running server rather than the in-memory filer, where both looked like success because the table did stop being listed. Deregistering is now a state on the entry - the marker file hides it, and declaring or registering the name again brings it back. Repointing a name at another dataset is an UpdateTable against the version token, so neither dataset loses files. Drop is left alone; it is the operation that does remove data. The storage endpoint now falls back to the advertised -ip where the Iceberg derivation gives up. An Iceberg client brings its own s3.endpoint and advertising the wrong one hijacks it, but storage_options is the only place a Lance client learns where the store is, and without it object_store quietly talks to real AWS. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tables: refuse to create a table over one of another format Creating a table that already exists is idempotent, and that path returned the existing table without looking at its format. A Lance declare over an Iceberg table answered 200 and handed back a directory Iceberg owns, so the client would write its dataset on top. The view check immediately above it already guards the same class of collision. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tables: let a table bucket hold a format other than Iceberg The S3 door validated every object written into a table bucket against Iceberg's file layout, so a Lance client could not write its dataset at all: it got 403 on data/*.lance, on _versions/, and on the _transactions/ directory it turned out to write as well. Table buckets were only neutral containers by intention; in practice they were Iceberg-shaped and enforced as such. The allowed set is now the union of what the supported formats write, because the validator runs where the table's format is not in hand. Underscore-prefixed directories are treated as belonging to the format, since enumerating them means guessing at the next one - _transactions is exactly the one this missed - and their contents are checked only for traversal. Iceberg writes none of them, so it loses nothing. Marker files at the table root are admitted too, which the namespace/table/dir/file shape had rejected as too shallow. Describe also honours the request-body spellings of with_table_uri, load_detailed_metadata and check_declared. The spec puts them in the query string, but real clients send both. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: record what the implementation found The table bucket being an Iceberg-shaped container, enforced at the S3 door, was the premise this design never questioned and the one that had to change before anything worked end to end. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * iceberg: prove the data loss the foreign-format guard prevents The guard landed with a unit test for the predicate and nothing showing what it saves. These seed what the Lance namespace's Iceberg REST adapter actually leaves behind - an Iceberg table with a placeholder schema and table_type=lance whose directory holds a Lance dataset - and assert both halves: orphan collection does flag the dataset's fragments, because the Iceberg metadata beside them references nothing, and the scan never reaches the table. An ordinary Iceberg table in the same shape is still scanned, so the guard is not just skipping everything. Confirmed against a running gateway first: our Iceberg catalog accepts the adapter's registration, and a real Lance client then writes a dataset into that table's location. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tablestest: make the in-memory filer safe to race against Two gaps that only matter once a test drives concurrent writers, which is what an exclusive create has to be tested with: the entry map had no lock, and CreateEntry ignored O_EXCL entirely, so both writers of the same name would have won and the test would have passed while proving nothing. The BeforeUpdate hook runs before the lock is taken. Its whole purpose is to land a competing write in a handler's read-to-write window, and that write needs the lock the hook would otherwise be holding. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: make the namespace an external manifest store Lance commits a version by writing _versions/{v}.manifest with put-if-not-exists. The S3 layer in front of this same filer evaluates If-None-Match by looking the entry up and then writing without a precondition, so two writers can both pass the check and one commit is lost. The filer itself has the primitive: CreateEntry with o_excl. Adds the four version operations a Lance client actually calls - create, list, describe and batch-delete - recording one entry per version under _lance_versions/, and advertises managed_versioning so the client routes its commits here. Reserving a version is the exclusive create, so exactly one of several racing writers wins and the rest rebase. Off by default, behind -lance.managedVersioning. Turning it on moves where a table's version history lives, and a reader that does not come through this namespace no longer sees all of it; that is the operator's call, not a default. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: record what managed versioning does and does not reach The first commit through a namespace-backed store works and is recorded the way the protocol specifies. Later commits do not, because lance 4.0.0 refuses put_if_exists on that path in its own code, so the feature is capped upstream rather than here. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * test: integration tests for the Lance namespace Everything this surface got wrong so far - a deregister that deleted the dataset, an S3 door that refused every Lance file, a version reservation that could not actually be exclusive - passed against an in-memory filer first. So these run against a live gateway, and where the claim is about data they check storage rather than visibility. Five Go tests on the shared harness: namespace and table lifecycle including that deregister keeps the bytes and drop removes them, that a Lance client cannot resolve or declare over an Iceberg table, that a Lance dataset's files get past the table-bucket layout guard while junk still does not, and that eight writers racing for one version produce exactly one winner. One Docker-gated test drives the real Lance client, which is the only way to check that the location and storage_options the namespace vends are between them enough to write and read a dataset. It overrides the endpoint with the container's view of the same gateway, because the shared harness binds a wildcard address and so vends none. The harness gains a Lance port and turns managed versioning on; the flag touches nothing outside that surface. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tables: a directory with no namespace metadata is a missing namespace Three callers resolved a namespace by reading its metadata attribute and each tested only for a missing entry, so a directory that carried no metadata came back as an internal error saying "attribute not found". Creating a table under a namespace that does not exist answered 500. Collapses the three copies into one helper that reports both conditions as absent, which is what they are: a directory without namespace metadata is not a namespace. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * iceberg: stop reporting storage-layer refusals as server faults writeManagerError recognised a missing table bucket and sent everything else to 500, so a missing namespace, a duplicate name and a commit conflict all reached the client as InternalServerError with nothing to act on. Creating a table in a namespace that does not exist is the case that turned up: 500 where the spec wants 404 NoSuchNamespaceException. Maps the storage error types onto the exception names this package already uses, and keeps the existing bucket message, which explains how to select a table bucket. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * iceberg: skip a foreign-format table by name, not by failing to parse it A table the namespace created as LANCE carries no Iceberg metadata, so the worker skipped it only because the parse failed, and logged that as damaged metadata. The catalog records the format on the entry and this never read it. Reading it turns an accident into a decision, and separates a mixed catalog from a corrupt one in the logs. The property check beside it still covers the other shape: a real Iceberg table wearing table_type=lance, which is what the Lance namespace's Iceberg REST adapter writes. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: answer whether a Lance table needs maintenance It does, and index optimization has no Iceberg equivalent: rows written after an index was built are not covered by it, so a vector search quietly misses them. None of the three jobs can run in the Go worker, and there is no useful subset, because deciding what an old version still references means parsing Lance manifests. Version cleanup at least has an answer that needs nothing from us - Lance can enable it on the dataset itself. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: the Lance maintenance worker is a plugin worker, in Rust Framing it as a sidecar was wrong. plugin.proto already defines a language-agnostic gRPC contract for external maintenance workers, and "weed worker -admin=..." is the Go reference implementation of it from outside the admin process. seaweed-volume already compiles protos out of weed/pb with tonic_build, so a Lance worker is that build plus plugin.proto and the lance crate. Scheduling, retries, dedupe, progress and the admin settings page all come from the protocol: a worker that answers RequestConfigSchema with a descriptor gets its configuration form rendered without a line of Go. The data plane is the part that genuinely does need a process answering HTTP, and this had the two conflated. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * seaweed-worker: Rust plugin worker workspace, with Lance as the first one plugin.proto is language-agnostic and the Rust toolchain was already in the tree, so a Lance maintenance worker needs no new integration surface: core is the contract and nothing else, and a worker crate beside it supplies handlers and a binary. A second worker is a new member here rather than a fork of the protocol, which is why this is seaweed-worker and not seaweed-lance-worker. Verified against a running admin: it connects, is accepted, and admin prefetches descriptors for lance_compact, lance_optimize_indices and lance_cleanup_versions, so their settings pages render from the Rust side without a line of Go. The stream stays up across heartbeats. The job bodies are stubs that report failure. Doing the work means adding the lance crate and opening the dataset, and claiming success before that would be worse than saying so. Two things running it caught that reading the proto did not: the admin address has to be converted to the gRPC port the way pb.ServerToGrpcAddress does, or the dial fails as an h2 frame error; and the generated field names differ from the Go ones in several places, so JobCompleted carries success rather than a state enum. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: implement compaction Detection lists tables from the namespace, opens each one, and proposes a job for any with more fragments than the policy allows; opening a dataset reads its manifest and not its data, so a sweep stays cheap. Execution re-resolves the table rather than trusting what detection saw - it may have been repointed, and the vended credentials expire - then compacts and reports the fragment counts either side. Verified against a live gateway: a twelve-fragment dataset became one fragment with all twelve rows intact. The test drives the handler directly and skips unless WEED_LANCE_NAMESPACE names a namespace, the way the Go integration tests skip without Docker. Running it turned up a gap the design had not: a gateway without STS vends no credentials at all, so the worker could not open anything and detection quietly proposed nothing. --access-key/--secret-key are the fallback, and whatever the namespace vends still wins over them. Two API assumptions did not survive contact either. Datasets open through DatasetBuilder::with_storage_options, not ReadParams, and lance 10's ObjectStoreParams has no storage_options field at all. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: implement index optimization and version cleanup Index optimization is the job with no Iceberg equivalent: rows appended after an index was built are invisible to a search of it until this runs. Detection reads num_unindexed_rows from each index's statistics and proposes a table once more rows sit outside its indices than the budget allows; a table with no indices is skipped, which is different from one whose indices have fallen behind. Cleanup applies a retention window, refusing rather than silently dropping a tagged version, and leaving unverified files alone because they may belong to a commit still in flight. Both verified against a live gateway: 512 uncovered rows became 0, and a fourteen-version table lost its old ones. Each test now seeds what it needs, including building an IVF_PQ index and appending rows outside it. The first version of these depended on state a script had left, so the second run found the work already done and asserted nothing - a test that passes by doing nothing is worse than no test. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: answer an empty catalog with an empty list, not null ListAllTables built its result from a nil slice, so a namespace holding no tables answered {"tables":null} on a field the spec marks required. A generated client may decode that differently from an empty list. Found running the namespace on a dev box, where the catalog was empty. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: give Lance maintenance its own scheduler lane Lane assignment is a hardcoded map, so the three lance_* job types fell through to the default lane. That lane serialises its work under the cluster admin lock because volume management shares global state, which would queue a table's compaction behind volume balancing for no reason - Iceberg has its own lock-free lane for exactly this. Adds the lane, maps the three job types to it, and puts it in the sidebar beside Iceberg and Lifecycle. The lane routes were already generic, so only the nav was hand-written. The lane-coverage test spelled out the three known lanes, so a fourth failed it. It now checks against AllLanes(), which is the property it was reaching for and does not need editing next time. Found by connecting the Rust worker to a real admin: it registered fine and its job types were known, but they were filed under "default" and had no page. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: log what detection saw "Detection proposed nothing" and "the worker could not read the table" look identical from the admin side, and the second is what a missing credential produces. One line per table separates them. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: fix a leaked heartbeat and a silent reconnect loop spawn_heartbeat returned a handle to an empty task rather than the ticker it had just spawned, so aborting it aborted nothing and every reconnect left another heartbeat running against a dead channel. A stream that admin closes cleanly is not an error, but reconnecting in silence hides why. Two workers sharing an id evict each other forever and the log shows nothing but a login every five seconds - which is exactly how this presented on a dev box, and it took a look at the admin's own log to see it. The message now names the id to check. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: a namespace cannot be created without its parent Storage keeps a namespace's parts flattened, so creating "a.b" with no "a" was accepted and left an intermediate that only existed inside a name. Listing derives child names by slicing those parts, so it reported "a", while describe and exists on "a" both answered 404 - a client walking the tree got a 404 on something the listing had just handed it. The spec asks for NamespaceNotFound when the parent is missing, which is also what keeps listing and describe telling the same story. Namespaces created through the S3 Tables API still bypass this, so listing keeps deriving intermediates rather than hiding whatever is already there. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: say why a non-Iceberg table shows no schema The table pages read Iceberg metadata for schema and snapshots, and a Lance table has none, so both panels rendered "No schema available" - which reads as an empty table rather than a table this page cannot describe. The dataset behind the one that prompted this holds 1024 rows. The format is already on the entry and shown two rows above, so the empty states now use it: the catalog records where a LANCE table lives, not what is in it. Reading the schema for real needs Lance format code, which is the same wall as the data plane. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * seaweed-worker: run rustfmt over the workspace Committed the crates unformatted, so `cargo fmt --all --check` failed on files nothing had touched since. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * plugin: let a worker report what it saw about an object Admin cannot read a Lance table: it knows where the dataset lives and nothing else, so the details page had a location and two empty panels. The worker already opens every dataset during detection to decide whether it needs compacting, so it knows the schema, the row count and the fragment count at that moment. It just had no way to say so. Add a WorkerObservations body to the worker stream. Admin caches the last observation per object and serves it back, timestamped, for display; nothing schedules from it. The Lance compaction sweep reports what it opened, and the S3 Tables details page fills its schema panel from the cache when it has no metadata of its own, badged with when the worker looked and which worker it was. Nothing about this is Lance-specific past the reporting side, which is the point: any format admin cannot parse can describe itself the same way. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: record the observation channel Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * plugin: ask a worker for sample rows of a table admin cannot read Browse Data reads an Iceberg table's Parquet files directly, so it shows real rows. For a Lance table it showed "Table has no Iceberg metadata" and an empty grid, because there is no Go Lance reader and never will be one worth maintaining. The worker has the reader. Add RequestObjectPreview / ObjectPreviewResponse to the stream, mirroring the config-schema round trip that already exists, and give the Rust worker a PreviewProvider that scans the dataset and formats the rows with Arrow's own formatter, so a vector column reads as a vector. Admin picks the worker from the observation store: whichever one last described this table is the one that can read it. Unlike an observation the rows are not cached. They are the table's data rather than a description of it, and a copy sitting in admin would be both stale and nobody's business. The page fetches on load, bounded at 200 rows and a 15 second round trip, and drops the snapshot and data-file panels that only mean something for Iceberg. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: record the preview channel Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * test: disable the lance listener when two gateways share a host * test: keep AllocatePorts away from the lance default port * s3tables: let a table bucket declare the format it holds A bucket is a catalog, and a catalog serves one protocol. Format was recorded per table, so nothing could answer "where do I point a client at this bucket" without opening a table first, and an empty bucket had no answer at all. CreateTableBucket takes an optional format, stored with the rest of the bucket metadata and returned by Get and List. Empty means ICEBERG, which is what AWS S3 Tables serves and therefore what an SDK that has never heard of the field means. CreateTable refuses a table of another format, and CreateView refuses outright in a bucket that is not Iceberg, since a view is Iceberg metadata. Buckets that already exist carry no declaration and keep accepting anything, so nothing is migrated and nothing that worked stops working. The Lance namespace declares LANCE for the buckets it creates, which is what stops one of them being described to a client as an Iceberg catalog. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: take the Lance port the way it takes the Iceberg one The UI cannot name the endpoint that serves a Lance bucket without it, and every format-aware page below needs to. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: show which format a table bucket holds The bucket list printed an Iceberg endpoint for every bucket, including ones holding Lance datasets, where that endpoint serves nothing. It was the most visible place the UI assumed one format. The list gains a Format column and its endpoint column follows the bucket's declaration. The banner names both endpoints rather than asserting everything is Iceberg, and says so only for the servers that are actually running. Create Bucket picks a format with two cards rather than a dropdown, since what matters is not the name but which clients can read the result, and the endpoint under them updates as you choose so the operator leaves the modal knowing where to point one. A bucket from before the declaration existed shows "unset" in an outline badge, explained on hover. It is a fact about the bucket's age, not a fault, so nothing nags about it. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: carry the bucket's format into the pages inside it Namespaces and tables are reached through a bucket, so both now say which catalog they belong to rather than making you go back up to find out. The tables list gains a Format column and a Rows column filled from what a worker last observed, since for a format admin cannot read that is the only row count there is; a table nothing has looked at shows a dash, not a zero. Create Table stops offering a choice the bucket has already made: in a declared bucket the format is fixed and says why, and only an undeclared one still offers both. Before this the select had exactly one option, hardcoded, which made a Lance table impossible to create from the UI at all. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: let the table page speak the table's own format Partitions and Snapshot History are Iceberg's shape. Rendering them empty for a Lance table reads as a fault; a Lance table has neither, and says so by not showing them. In their place is a Versions panel, which is what that format calls its history, carrying the worker's timestamp so it is clear the numbers are a cached look rather than something read live. The breadcrumb carries the format badge, so the page names what it is looking at before you read a panel and wonder why it is empty. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: show how to connect to either catalog, and group the two format workers The client examples on the buckets page were Iceberg's alone, so the one thing an operator wants after creating a Lance bucket - what to type to reach it - was not written down anywhere in the UI. Both formats now get a pair of snippets, and only for a server that is running. In the Workers menu, Iceberg moves below Lifecycle so it sits next to Lance: the two table-format workers together, the two cluster-wide ones above them. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * shell: create a table bucket of either format s3tables.bucket -create takes -format, so a Lance bucket can be made without going through the UI. The integration harness passes it too: its Lance tests were creating Iceberg buckets and getting away with it only because nothing checked. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: record that a bucket declares its format Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: drop managed versioning; the store already orders commits The namespace offered itself as an external manifest store, so that a commit could reserve a version through a real put-if-not-exists. That was designed around a gateway that no longer exists: If-None-Match: * is reduced to a filer WriteCondition and evaluated at the object's owner under its per-path lock, or under the object write lock on the fallback path. Sixteen writers racing one fresh key get a single 200 and fifteen 412s, every time. Lance needs nothing else. commit_handler_from_url hands every s3:// dataset a ConditionalPutCommitHandler, which puts with PutMode::Create, which object_store sends as If-None-Match: *. So the feature solved a problem this store does not have, while moving a table's version history out of the dataset and into the catalog - and lance could not use it past the first commit anyway, since its own namespace-backed store answers "put_if_not_exists is not supported" to the second. The version operations answer Unsupported with the rest, managed_versioning is false, and the flag is gone. In place of the reserve-once test there is one that races eight writers at the manifest key through S3, which is the path a commit actually takes. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: honour the version floor, the slot limits, and a shutdown Five findings from review, all of them things the worker claimed to do and did not. The version floor was checked when a cleanup job was proposed and ignored when it ran, so a table whose versions had aged past the retention window in between could be taken below the count the operator asked to keep. Execution now computes the floor itself and passes it as before_version; CleanupPolicy ANDs its clauses, so a version has to be both too old and below the floor to go. Both settings are clamped to the range the form offers, since Duration::hours panics on a large enough value and a negative min-versions wraps to a huge usize. Admin's shutdown was answered by returning from the stream, which the reconnect loop read as a healthy close and logged straight back in: the worker could not be stopped. serve_once now says which of the two happened. The advertised concurrency limits bounded nothing - every request spawned a task - and the heartbeat reported zero slots in use whatever was running. Both now go through semaphores sized from the limits, with the permits held for the life of the request and reported in the heartbeat. A namespace call had no timeout, so a gateway that accepted the connection and went quiet held a detection slot forever. And one table whose stats could not be read failed the whole sweep, losing the proposals for every table already scanned; it is now skipped and warned about, like a table that cannot be opened. The tests drove one shared catalog concurrently, which is why one of them asserted "no proposals at all" and passed by luck. They now take a lock and judge only their own tables. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: fix the review findings on the format-aware pages The endpoint hint in Create Bucket built its HTML by concatenating the bucket name the operator is typing, so a name like <img onerror=...> ran in the admin origin as they typed it. It is built from DOM nodes now. A preview reply looked its channel up under the lock and then sent outside it, which Shutdown can close in between: a Gosched in that gap panics with "send on closed channel" every time. The send now happens under the lock. Observations were looked up by path alone, so a table dropped and remade in another format at the same path was described by the observation left behind. Lookups now have to agree on the format. Also: the Lance namespace caps a request body rather than reading whatever arrives; the details action no longer says "Iceberg" over a Lance table; mini stops advertising a catalog port when it is not running S3; a format whose server this cluster does not run cannot be picked in the modal or accepted by the API, since a bucket nothing can reach is not worth creating; and the unused catalogPortFor helper is gone. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: let the control stream use mTLS The channel was hardcoded to http://, so off loopback the stream carried preview rows and execution commands in the clear - and a cluster with grpc TLS turned on would refuse the worker outright. --tls-ca, --tls-cert and --tls-key take the same certificates the Go worker reads from the [grpc.worker] section of security.toml, and must be given together: a CA on its own would quietly mean one-way TLS, which a mutual setup rejects anyway. Without them the stream stays plaintext, which is what the Go worker also does when nothing is configured. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: answer null properties rather than an empty map The catalog does not keep a table's properties. Declare echoed the request's back and describe answered {}, both of which claim they were stored and are empty. Null says the catalog does not keep them, which is what the spec distinguishes and what is true here. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: test the slot accounting The heartbeat reporting and the waiting are the two things the semaphores are for, and neither is observable from outside without catching a sweep mid-flight. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * test: fix the mixed-format catalog test, and name the binary it drives The integration suite passed locally and failed in CI on TestLanceRefusesIcebergTables. Both were right: CI builds the binary first, my tree had one from the day before, so locally the test drove a gateway with no format enforcement at all. The test itself no longer holds as written. It made a bucket, put an Iceberg table in it, and checked the Lance surface hid it - but a bucket that declares LANCE now refuses the Iceberg table outright. The invariant still matters from the other side, so it starts from an Iceberg bucket instead: Lance must not describe or list a table whose format it does not serve, and must refuse to declare one beside it. The harness now prints which weed binary it is about to run and when that was built. `make test` rebuilds first; a plain `go test` will happily drive a weeks-old binary and report a pass for code it never ran, which is exactly what happened here. Also make the row-limit conversion in the preview request explicitly bounded: CodeQL flagged the int-to-int32 conversion, and clamping by reassignment beforehand is not a form it recognises. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: prove concurrent commits are kept, and preselect the only format on offer Two more from review. The commit test asserted that exactly one writer wins the conditional PUT, which is the mechanism, not the claim. The claim is that nothing is lost: the losers see the conflict, rebase and commit again. So there is now a test that has eight writers append to one dataset at once and counts the rows afterwards - all eight batches survive. That is also the sequence managed versioning could not finish, since its store refuses the second commit outright. And when Iceberg's endpoint is not running, the format picker offered two options with neither selected, so Create Bucket submitted no format at all, fell back to ICEBERG, and was refused by the guard added last round. Lance is preselected when it is the only format this cluster serves. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * Clamp the remaining worker settings, and bootstrap buckets in a served format Compaction and index optimization read their thresholds and cast straight to usize and u64, so a negative arrives as an enormous number and turns the threshold into "never": compaction and reindexing both go quiet with nothing to say. The cleanup job was fixed last round; these are the same bug. Clamped to the values that stay meaningful rather than to what the form offers - zero uncovered rows is a real setting, meaning reindex as soon as anything is not covered, so the floor there is zero and not the form's thousand. mini pre-creates the buckets named by -tableBucket, and did so without a format, which now means Iceberg. Started with the Iceberg endpoint off and the Lance one on, that left buckets nothing could reach and which refused every Lance table. It takes the format from the endpoint that is actually running, and creates nothing when neither is. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3: allow-unordered is a listing parameter, not an unimplemented subresource The guard that stops a bucket GET with an unknown subresource from being answered with a listing does not know about allow-unordered, so it answers 501 NotImplemented - to a parameter the listing handlers already read and already validate against delimiter. This is why test_bucket_list_unordered and test_bucket_listv2_unordered fail in the Ceph s3-tests suite. They fail on master too; this is not a Lance change and can be taken on its own. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm |
||
|
Chris Lu
|
a1d3fe236f |
iceberg: let table properties override the worker config (#10772)
* iceberg: carry snapshot retention in milliseconds Config stored retention as hours, so any sub-hour value would have to be truncated to 0 and then clamped back up to the 168 hour default. Keep the plugin config key in hours and convert once at parse time. * iceberg: let table properties override the worker config Every other Iceberg implementation lets a table's own properties win over engine defaults; the worker ignored them entirely. A writer honouring write.target-file-size-bytes and a compactor rewriting to the plugin config's size would rewrite each other's output forever. Resolved once per job rather than per operation, so compaction committing new metadata mid-job cannot change the settings underneath it. * iceberg: clamp the orphan cutoff so it cannot overflow collectOrphanCandidates converts the cutoff to a time.Duration. Past roughly 2.5 million hours that multiplication wraps negative, putting the cutoff in the future so every file walked looks like an orphan and gets deleted, including data a concurrent writer has not yet committed. Reachable today through orphan_older_than_hours. |
||
|
Chris Lu
|
4500bdf88e |
iceberg: accept lowercase parquet file format when planning compaction (#10751)
* iceberg: accept lowercase parquet file format when planning compaction * iceberg: expect absolute added-file paths in compaction integration test |
||
|
Chris Lu
|
790e8d3fd6 |
clickhouse catalog test: cover latest ClickHouse and catalog-side CREATE TABLE (#10707)
* clickhouse catalog test: cover latest ClickHouse and catalog-side CREATE TABLE * verify catalog registration structurally and fix README image wording |
||
|
Chris Lu
|
923d0bd20c |
iceberg: repair non-compliant manifests at commit (#10641)
* iceberg: stamp a default name mapping on new tables * iceberg: repair non-compliant manifests at commit * s3tables: verify ClickHouse writes read back through PyIceberg * iceberg: carry the manifest-list content into repaired manifests * iceberg: refresh the default name mapping on schema evolution * iceberg: merge historical names into the refreshed name mapping * iceberg: never fail a commit on repair fallout * iceberg: harden manifest repair against writer dialects * s3tables: keep PyIceberg reader stderr out of row data * iceberg: keep name mappings unambiguous across field id reassignment * iceberg: align existing manifest content metadata with the list entry |
||
|
Chris Lu
|
213eb4c23a |
s3tables: add ClickHouse iceberg catalog integration test (#10637)
* s3tables: add ClickHouse iceberg catalog integration test * ci: run the ClickHouse iceberg catalog test * s3tables: bound setup HTTP calls in the ClickHouse test * s3tables: pin the ClickHouse writer image dependencies |
||
|
Chris Lu
|
d448e9db7b |
iceberg: withhold the S3 endpoint from credential-vending clients (#10570)
* iceberg: withhold the S3 endpoint from credential-vending clients A client that sends X-Iceberg-Access-Delegation: vended-credentials builds its storage credential out of the LoadTable config and drops the one it was configured with. We vend no credentials, so the endpoint we advertised left DuckDB signing nothing: every metadata and data file came back 403, and its attempt to refresh the empty credential 404ed on stage-created tables. Answer those clients with no config at all so they keep their own credentials. Clients that do not ask for delegation still get the endpoint. * iceberg: mark load responses as varying on the delegation header The FileIO config in a table or view load response now depends on whether the client asked for vended credentials, so a cache between us and the client must key on that header rather than on the URL alone. * test: cover the DuckDB vended-credentials access pattern Runs weed mini with -s3.externalUrl, which is what makes the catalog advertise an endpoint at all, and checks both halves: a plain LoadTable still gets the endpoint, while one asking for vended credentials never gets an endpoint without the credentials to sign with. The DuckDB round trip creates a table from a query and reads it back, which is the flow that failed with 403 on every data file. |
||
|
Chris Lu
|
c191b2fe01 |
iceberg: let clients select their table bucket as the catalog warehouse (#10549)
* iceberg: accept bare bucket names and ARNs as the catalog warehouse Only s3://<bucket>/ was recognized. A warehouse spelled as a bare table bucket name or as the s3tables bucket ARN -- the two forms users reach for first, the latter being what AWS S3 Tables itself takes -- was silently dropped, so every call landed on the default "warehouse" bucket and failed with "table bucket warehouse not found". * iceberg: report a missing table bucket as 404, not 500 Pointing a client at a table bucket that does not exist -- which every client with no warehouse set does, since the default bucket "warehouse" rarely exists -- returned InternalServerError with a message naming a bucket the client never asked for. Answer 404 and say how to select one. * admin: show the warehouse in the PyIceberg example The example connected without one, so it always resolved to the default table bucket and every client that copied it failed on the first call. * test: pin bearer auth against a table bucket that exists The subtest called the catalog with no warehouse and accepted 500 as proof that auth had passed, since the default bucket does not exist. A missing table bucket now answers 404, which the test read as an auth failure. Give it a real table bucket so only 200 passes. * test: assert the missing-bucket guidance reaches the client The status and error type were checked but not the message, which is the part of the mapping that tells a user how to select a table bucket. * test: encode the warehouse query value The ARN case pasted raw colons and slashes into the query string. Go's parser tolerates them, so the test passed without modelling how a client actually sends the request. |
||
|
Yaroslav HalchenkoandClaude Code 2.1.217 / Claude Opus 4.7
|
490379bff3 |
Add codespell support with configuration and typo fixes (#10393)
* Add GitHub Actions workflow for codespell on master * Add rudimentary codespell config * Tune codespell config: skip generated code, ignore camelCase, whitelist domain terms Add camelCase/PascalCase regex to ignore common Go/Rust/JS identifiers like allLocations, publishErr, ReadInside, FlushInterval. Also skip templ-generated *_templ.go files, and whitelist a handful of short/domain-specific words (visibles, fo, te, ser, bject, unparseable, keep-alives, tread, anc, ue) that show up as false positives across the tree. Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Fix ambiguous typos and protect false positives Fixes typos that codespell reports with multiple candidate suggestions (so `codespell -w` cannot auto-apply them), plus one inline pragma and one config entry to protect legitimate identifiers. Manual fixes (single correct answer chosen from context): - pattens -> patterns (5x) in filer/upload/shell flag help strings - finded -> found (2x) in tarantool storage.lua comment - spacify -> specify (2x) in helm chart values.yaml comment - wether -> whether in skiplist.go docstring - simpe -> simple in mq schema test case name False-positive protection: - Add `//codespell:ignore` next to `source GET's` (possessive of HTTP verb) in s3api_object_handlers_copy_stream.go - Whitelist `auther` in .codespellrc — it's a local variable meaning "authenticator" in weed/security/tls.go, not a typo of "author". Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Extend codespell ignore list: .git-meta path and thirdparty groupId Also skip `.git-meta` (scratch dir for commit messages that may contain typo words verbatim) and whitelist `thirdparty` — it appears as the literal Maven groupId `org.apache.hadoop.thirdparty` in hdfs3 poms and cannot be renamed. Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com> * [DATALAD RUNCMD] Fix non-ambiguous typos with codespell -w Auto-applied fixes to the 44 remaining single-suggestion typos across docs, comments, log messages, tests, config, and one Java pom. === Do not change lines below === { "chain": [], "cmd": "uvx codespell -w", "exit": 0, "extra_inputs": [], "inputs": [], "outputs": [], "pwd": "." } ^^^ Do not change lines above ^^^ * Revert breaking codespell fixes; whitelist unknwon and atleast Two of the auto-applied `codespell -w` fixes were false positives that would break the build/tests: - go.mod: `github.com/unknwon/goconfig` is a real Go module path — the upstream author's GitHub handle is literally `unknwon`. Renaming to `unknown` would fail dependency resolution. - test/benchmark/fuse_db/bin/{sqlite_verify.py,run_mysql.sh,run_sqlite.sh}: `atleast` is a literal CLI mode value (a string constant compared and passed as a positional argument). Rewriting to `at least` splits it into two arguments and breaks the mode check. Reverted those files and whitelisted both words in .codespellrc so future runs won't re-suggest the same broken fixes. Co-Authored-By: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Code 2.1.217 / Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Chris Lu
|
60e7b30009 |
admin: browse Iceberg table data (#10227)
* admin: move volume-server read JWT helper into dash The Iceberg data preview page needs the same per-fileId read token the file browser uses when streaming chunks from volume servers. Claude-Session: https://claude.ai/code/session_015n3oKLTjnPjcnZtfigNKur * admin: add Iceberg table data preview page The admin UI browses the Iceberg catalog down to table details but not the data itself. Add a Browse Data page per table that walks the selected snapshot's manifests and shows sample rows from its Parquet data files, plus the data file list with per-file preview, a snapshot switcher, and a row limit selector. Rows are read through a ranged ReaderAt over stream-content so only the Parquet footer and needed pages are fetched, with the volume read JWT applied when configured. Iceberg locations resolve into /buckets with traversal guards, and the file parameter must match a manifest-listed data file. Snapshots with delete files get a warning that raw rows are shown. Claude-Session: https://claude.ai/code/session_015n3oKLTjnPjcnZtfigNKur * admin: integration test for Iceberg catalog and data preview pages Starts a weed mini cluster with the admin UI, creates a table bucket, namespace, and tables via the S3 Tables manager, uploads real Parquet files via S3, writes manifests and snapshots with iceberg-go, and asserts on the rendered pages: catalog browsing, table details, current and historical snapshot previews, per-file preview, row limits, unknown snapshot and file errors, and a metadata-less table. Claude-Session: https://claude.ai/code/session_015n3oKLTjnPjcnZtfigNKur * admin: write Iceberg preview chunk reads straight into the caller slice ReadAt wrapped the caller's buffer in a bytes.Buffer, which would silently allocate a fresh backing array and drop bytes if it ever grew. Copy directly into the destination slice and reject negative offsets so the ReaderAt contract holds. Claude-Session: https://claude.ai/code/session_015n3oKLTjnPjcnZtfigNKur * admin: link to snapshot history when the preview switcher truncates The snapshot switcher caps at 25 entries; add a trailing item pointing at the table details page so older snapshots stay reachable. Claude-Session: https://claude.ai/code/session_015n3oKLTjnPjcnZtfigNKur * test: hoist mini cluster context assignment out of the goroutine Set MiniClusterCtx before launching the cluster goroutine and clear it in stop(), so the assignment is not buried in the command loop. Claude-Session: https://claude.ai/code/session_015n3oKLTjnPjcnZtfigNKur |
||
|
Chris Lu
|
3b9e196e5f |
sts: enforce session-policy explicit deny during role chaining (#10103)
* sts: enforce session-policy explicit deny during role chaining A chained AssumeRole caller authenticates with an STS session token whose inline session policy can explicitly deny sts:AssumeRole. The deny check only evaluated the caller's named policies, so such a session could still chain into any role its trust policy admits. Validate the session token in the deny check and honor an explicit Deny in the inline session policy too. * test(sts): integration coverage for AssumeRole authorization Add an end-to-end AssumeRole authorization test (real weed mini + boto3): a non-admin caller assumes a role its trust policy admits, an explicit identity-side deny is blocked, and a session policy's explicit deny blocks role chaining. * sts: skip OIDC tokens and reject revoked sessions in the chaining deny check Review follow-ups on the session-policy deny check: - Guard session validation with !isOIDCToken so a bearer token our STS service cannot validate does not error into a false deny. - Reject a revoked session before evaluating its policy, restoring the revocation enforcement the AssumeRole path lost when it stopped routing through IsActionAllowed. |
||
|
Chris Lu
|
caadd6ca79 |
ci(s3tables): stop Lakekeeper flaking on Docker Hub pull timeouts (#9920)
* ci(s3tables): drop docker pre-pull from Lakekeeper job The lakekeeper repro is pure Go against the local weed binary; the job kept failing on Docker Hub timeouts pulling python:3 and localstack images the test never runs. Also drop the stale python-in-docker comments left from the old harness. * ci(s3tables): serve python:3 from GHA cache in the STS job Retried pulls still die when both mirror.gcr.io and registry-1.docker.io are unreachable from the runner. Cache the saved image tarball under a weekly key: an exact hit skips the registry entirely, a miss pulls fresh and refreshes the cache, and a stale tarball from a previous week is the fallback when Docker Hub is down. * ci(spark): pre-pull the spark tag the test actually runs The workflow warmed apache/spark:3.5.8 with retries while the testcontainers setup runs apache/spark:3.5.1, so the real image was pulled at test time with no retry at all. |
||
|
Chris Lu
|
f72983c1fd |
fix(s3): stop S3 Tables routes from swallowing buckets named "buckets" or "get-table" (#9566)
* fix(s3): stop S3 Tables routes from swallowing buckets named "buckets" or "get-table"
The S3 Tables REST endpoints share top-level paths with the regular S3
API (/buckets for ListTableBuckets/CreateTableBucket, /get-table for
GetTable). They are registered first on the same router as the bucket
subrouter, so a path-style request such as GET /buckets?list-type=2 on
a bucket actually named "buckets" matched ListTableBuckets and returned
JSON. AWS SDK V2 (and Hadoop s3a / Spark) then failed XML parsing with
"Unexpected character '{' (code 123) in prolog".
Disambiguate by requiring the AWS V4 credential scope to name the
s3tables service on the colliding routes. Regular S3 SDKs sign with
service=s3, S3 Tables SDKs sign with service=s3tables, and the scope is
present in both the Authorization header and the X-Amz-Credential query
parameter for presigned URLs, so the matcher works for both flavors.
ARN-bearing S3 Tables routes (/buckets/<arn>, /namespaces/<arn>, etc.)
already cannot collide because colons are not valid in bucket names, so
they are left untouched.
* fix(s3): accept AWS JSON RPC content type as S3 Tables intent signal
The Iceberg catalog integration tests send unsigned PUT /buckets with
Content-Type: application/x-amz-json-1.1 to create table buckets. With
only the credential-scope check, those requests fell through to the
regular S3 CreateBucket handler and the suite went red on this branch.
Extend the matcher so a request is recognized as S3 Tables when either:
- its AWS V4 credential scope names SERVICE=s3tables; or
- it carries the canonical AWS JSON RPC 1.1 content type and is
unsigned (a request explicitly signed for SERVICE=s3 still wins).
The regular S3 SDKs do not send application/x-amz-json-1.1, so the
signal is safe for the colliding paths (/buckets, /get-table).
Also add an AWS SDK V2 for Go integration test under
test/s3/sdk_v2_routing/ that drives the SDK's own XML deserializer
against a bucket literally named "buckets" and "get-table" — the SDK
errors before the test asserts if the server returns the wrong body
shape. Wired up via .github/workflows/s3-sdk-v2-routing-tests.yml,
mirroring the etag/acl workflow.
* s3api: extend service matcher to all S3 Tables routes; simplify scope check
- Apply serviceMatcher to every S3 Tables route, not just the bare-path
ones. ARN-bearing paths could otherwise be hit by an S3 object key
that starts with arn:aws:s3tables:..., inside a bucket named
"buckets", "namespaces", "tables", or "tag". One matcher everywhere
closes both collision classes.
- Replace strings.Split + index lookup with strings.Contains for the
credential-scope check. The scope shape is fixed at
AK/DATE/REGION/SERVICE/aws4_request, slashes only delimit components,
and access keys are alphanumeric — so /s3tables/ matches iff SERVICE
is exactly s3tables. Existing unit cases (including the
access-key-substring case) still pass.
- Read the GetObject body in the SDK v2 routing test with io.ReadAll;
the single Read could return short and make the equality check flaky.
* s3api: drop content-type fallback; sign s3 tables harness traffic instead
The content-type fallback in isS3TablesSignedRequest let an anonymous
regular-S3 request whose body type is application/x-amz-json-1.1 hit
an S3 Tables route when the path-style object key happened to be
shaped like an S3 Tables ARN (e.g. PutObject on bucket "buckets"
with key arn:aws:s3tables:.../bucket/foo/policy). Narrow the matcher
back to the AWS V4 credential scope so only requests signed for
SERVICE=s3tables match the S3 Tables routes.
Update the Iceberg catalog test harness — the only caller still
sending unsigned PUT /buckets — to sign with SERVICE=s3tables. The
mini instance runs in default-allow mode, so the signature itself is
not verified; only the credential scope matters for the route match.
Drop the stale unit cases for the JSON-RPC content-type signal and
the routing test that exercised unsigned harness traffic.
|
||
|
Chris Lu
|
a761441926 |
fix(test): reserve mini ports on all interfaces; bound risingwave cleanup shell (#9545)
The 127.0.0.1-only reservation in AllocateMiniPorts/AllocatePortSet let another process hold the gRPC port on a different interface, so weed mini's isPortAvailable check failed and it shifted master.grpc. weed shell -master=<HTTP> still derives grpc as HTTP+10000 and dialed the unused port, hanging until the 30s context deadline killed it. Bind the reservation listeners on :port to match mini's check. Also bound listFilerContents in catalog_risingwave with a 30s exec.CommandContext so a hung weed shell during failure-cleanup can't burn the 20-minute test budget. |
||
|
Chris Lu
|
a769c938ec |
test(s3tables): Unity Catalog OSS integration tests against SeaweedFS (#9308)
* test(s3tables): add Unity Catalog OSS integration test against SeaweedFS Mirrors the configuration used by the upstream playground at data-engineering-helpers/mds-in-a-box/unitycatalog-playground. Three test variants under test/s3tables/unity_catalog: - TestUnityCatalogDeltaIntegration: aws.masterRoleArn empty / static keys; catalog/schema/EXTERNAL Delta CRUD + temporary-table-credentials S3 round-trip (the playground's working configuration). - TestUnityCatalogMasterRoleIntegration: aws.masterRoleArn set to a SeaweedFS-side role with a permissive trust policy; UC's StsClient is pinned at SeaweedFS via AWS_ENDPOINT_URL_STS, and the test asserts the vended creds carry a session_token and a non-static access key, proving the role-vended path the playground notes as not-yet-working actually does work today. - TestUnityCatalogDeltaRsRoundTrip: writes/reads a real Delta table at the registered storage_location using delta-rs in a slim Python container, with temporary credentials fetched from UC. All three self-skip without Docker or a weed binary, matching the sibling lakekeeper / polaris tests. * test(s3tables): tighten Unity Catalog tests against actual UC OSS behavior After running the suite locally, ground the assertions in what the upstream UC OSS Docker image actually does against SeaweedFS today. - Static-key playground configuration (TestUnityCatalogDeltaIntegration): catalog/schema/EXTERNAL Delta CRUD pass against the SeaweedFS-backed warehouse. The temporary-table- credentials subtest is renamed and inverted to assert the failure mode the playground reports -- UC's AwsCredentialVendor falls through to an internal StsClient.assumeRole when masterRoleArn and sessionToken are both empty, which has no real STS to talk to. Bucket path is also fixed to match UC's getStorageBase() lookup (s3://lakehouse vs the playground's s3://lakehouse/warehouse, which the upstream code never matches). - Master-role variant (TestUnityCatalogMasterRoleIntegration): split into two passing slices. Slice 1 proves SeaweedFS' STS endpoint vending UnityCatalogVendedRole works via the Go AWS SDK and the vended creds round-trip on S3. Slice 2 boots UC with aws.masterRoleArn set and verifies catalog/schema/Delta CRUD. The third hop -- UC's Java StsClient actually reaching SeaweedFS' STS handler during /temporary-table-credentials -- is logged but not asserted, since the AWS Java SDK's STS request currently lands on a SeaweedFS S3 path rather than the STS handler. - Delta-RS round-trip (TestUnityCatalogDeltaRsRoundTrip): gated on UC_DELTA_RS_RUN=1 since it depends on the master-role STS handoff above. The Dockerfile / writer script stay in tree so the test runs end-to-end the moment that hop is fixed. README rewritten to be explicit about what each test validates today and what is still pending. Result: `go test -run TestUnityCatalog ./test/s3tables/unity_catalog/...` passes cleanly with weed + Docker available, and self-skips otherwise. * test(s3tables): exercise unity catalog integrations * ci: run Unity Catalog integration tests on PRs Adds a unity-catalog-integration-tests job to s3-tables-tests.yml, modeled on the existing lakekeeper / dremio jobs. Pre-pulls the UC image and python:3.11-slim (used by the delta-rs writer container) and runs `go test ./test/s3tables/unity_catalog`. Format-check and go-vet jobs already recurse into ./test/s3tables/... so the new package is covered there too. * test/ci: address PR review Tighten the UC readiness probe to require 200, not <500, so a 401/403/404 during startup surfaces immediately instead of being treated as ready (CodeRabbit). Pin the UC image to v0.4.0 in both the workflow and the test default, matching the pinned-tag convention the rest of s3-tables-tests.yml uses (CodeRabbit). Use UC_IMAGE=unitycatalog/unitycatalog:main to re-test against current upstream. * docs: separate UC static-key vs master-role failure modes The README mixed the two together. Static-key empty-sessionToken short-circuits with "S3 bucket configuration not found." before UC even fires an STS call; the AccessDenied I described is what happens in the master-role variant where UC's Java StsClient actually reaches SeaweedFS. Cross-link the playground PR that fixes the static-key vending side. Also drop the "what most playground users actually run" hand-wave under MANAGED tables. * docs: trim README Drop the playground cross-reference and the "two layers fail independently" framing. * docs: pin down what's actually pending Investigated the master-role STS handoff with a sniffer in front of SeaweedFS' STS port. UC's StsClient is constructed without an endpointOverride and never reads aws.endpoint or AWS_ENDPOINT_URL_STS; verified by pointing AWS_ENDPOINT_URL_STS at port 1 and seeing the same real-AWS InvalidClientTokenId 403 with zero traffic to SeaweedFS. The fix is upstream in UC. Updated the README and the master-role test's t.Logf to say so precisely, and dropped the stale "Spark client" bullet (delta-rs covers that path). * test(s3tables): use BaseEndpoint instead of deprecated resolver EndpointResolverWithOptions is deprecated in aws-sdk-go-v2; the supported way to override a service endpoint is via the per-service Options.BaseEndpoint. Switch the assume-role helper to that pattern so the test stops compiling against deprecated API and the resolver boilerplate disappears. Addresses gemini review on PR #9308. * test(s3tables): drop unused splitS3URI helper Helper had no callers; gemini caught it on PR #9308. Easy to bring back from git history if needed. * test(s3tables): extract last token of docker run output as container ID docker run -d may prefix the container ID with image-pull progress when the image isn't cached locally. strings.TrimSpace on the whole output then gave a multi-line string, not the ID. Take the last whitespace-separated token so the ID survives a fresh CI runner. Addresses gemini review on PR #9308. * test(s3tables): cap Unity Catalog response body reads at 10 MiB io.ReadAll without a limit could OOM the test runner if the UC container hands back an unexpectedly large body. 10 MiB is well above any well-formed catalog response and turns a misbehaving server into a test failure instead of a runner crash. Addresses gemini review on PR #9308. * docs: link UC fix PR and call out UC's mocked-Sts test pattern UC's own credential-vending tests substitute StsClient with an in-process EchoAwsStsClient (BaseCRUDTestWithMockCredentials) or Mockito.mockStatic (CloudCredentialVendorTest), so the wire path between UC's Java SDK and a real STS server is untested -- which is why the missing endpointOverride slipped through upstream. Linked the upstream fix at unitycatalog/unitycatalog#1532. |
||
|
Chris Lu
|
1de741737d |
test(s3tables): add Apache Doris Iceberg catalog integration test (#9307)
* test(s3tables): add Apache Doris Iceberg catalog integration test Adds an end-to-end smoke test that boots the apache/doris all-in-one container, registers SeaweedFS as an external Iceberg REST catalog (OAuth2 client_credentials), and validates metadata visibility plus the parquet read path against tables seeded via the Iceberg REST API and a PyIceberg writer container, mirroring the existing Trino, Spark, and Dremio coverage. Wires the test into a new s3-tables-tests workflow job. * test(s3tables): document weed shell -master flag format and fill in helper docstrings Restores the explanatory comment on createTableBucket about the host:port.grpcPort ServerAddress format used by `weed shell -master` (produced by pb.NewServerAddress) so the dot separator isn't mistaken for a typo, and adds doc comments for createIcebergNamespace, createIcebergTable, doIcebergJSONRequest, requireDorisRuntime, and hasDocker. |
||
|
Chris Lu
|
87bb0a4115 |
test(s3tables): capture weed mini stdout/stderr in catalog_spark tests
Without this, when "weed mini" fails to bind its master port within the 45s startup timeout the only diagnostic is the timeout message itself, making flakes impossible to root-cause. The sibling catalog, catalog_dremio, and catalog_trino packages already wire stdout/stderr through; bring catalog_spark in line. |
||
|
Chris Lu
|
fc75f16c30 |
test(s3tables): expand Dremio Iceberg catalog test coverage (#9303)
* test(s3tables): expand Dremio Iceberg catalog test coverage
Restructure TestDremioIcebergCatalog into subtests and add three new
checks that go beyond a connectivity smoke test:
- ColumnProjection: SELECT id, label proves Dremio parsed the schema
served by the SeaweedFS REST catalog (the previous SELECT COUNT(*)
passed without exercising any column metadata).
- InformationSchemaColumns: verifies the table's columns are listed in
Dremio's INFORMATION_SCHEMA.COLUMNS in the expected ordinal order.
- InformationSchemaTables: verifies the table is registered in
INFORMATION_SCHEMA.TABLES.
All subtests share a single Dremio container startup, so total
runtime is unchanged.
* test(s3tables): exercise multi-level Iceberg namespaces from Dremio
Seed a 2-level Iceberg namespace (and a table inside it) via the REST
catalog before bootstrapping Dremio, then add a MultiLevelNamespace
subtest that scans the nested table by its dot-separated reference.
This relies on isRecursiveAllowedNamespaces=true (already set in the
Dremio source config) to surface the nested levels as folders. A
regression in either the SeaweedFS namespace path encoding (#8959-style)
or Dremio's recursive-namespace discovery would surface here.
Adds two helpers to keep the existing single-level call sites unchanged:
- createIcebergNamespaceLevels: namespace creation with []string levels
- createIcebergTableInLevels: table creation with []string levels and
unit-separator (0x1F) URL encoding for the namespace path component
* test(s3tables): verify Dremio reads PyIceberg-written rows
The previous Dremio subtests only scanned empty tables, so they did not
exercise the data path - just the catalog/metadata path. Add a
PyIceberg-based writer that materializes parquet files plus a snapshot
on a separate table before Dremio bootstraps, and two new subtests:
- ReadWrittenDataCount: SELECT COUNT(*) returns 3.
- ReadWrittenDataValues: SELECT id, label ORDER BY id returns the three
written rows with the expected (id, label) pairs.
The writer runs in a small image (Dockerfile.writer) built locally on
demand. It pip-installs pyiceberg+pyarrow once and reuses the layer
cache on subsequent runs. The CI workflow pre-pulls python:3.11-slim
to keep cold runs predictable.
The writer authenticates via the OAuth2 client_credentials flow that
SeaweedFS already exposes at /v1/oauth/tokens, mirroring the Go-side
helper used for REST-API table creation.
* test(s3tables): fix Dremio writer required-field schema mismatch
PyIceberg's append() compatibility check rejects an arrow column whose
nullability does not match the Iceberg field. The table schema declares
id as `required long`, but the default pyarrow int64 column is nullable
- so the writer failed with:
1: id: required long vs. 1: id: optional long
Declare an explicit pyarrow schema with nullable=False on id and
nullable=True on label to match the Iceberg side.
|
||
|
Chris Lu
|
1f6f473995 |
refactor(worker): co-locate plugin handlers with their task packages (#9301)
* refactor(worker): co-locate plugin handlers with their task packages
Move every per-task plugin handler from weed/plugin/worker/ into the
matching weed/worker/tasks/<name>/ package, so each task owns its
detection, scheduling, execution, and plugin handler in one place.
Step 0 (within pluginworker, no behavior change): extract shared helpers
that previously lived inside individual handler files into dedicated
files and export the ones now consumed across packages.
- activity.go: BuildExecutorActivity, BuildDetectorActivity
- config.go: ReadStringConfig/Double/Int64/Bytes/StringList, MapTaskPriority
- interval.go: ShouldSkipDetectionByInterval
- volume_state.go: VolumeState + consts, FilterMetricsByVolumeState/Location
- collection_filter.go: CollectionFilterMode + consts
- volume_metrics.go: export CollectVolumeMetricsFromMasters,
MasterAddressCandidates, FetchVolumeList
- testing_senders_test.go: shared test stubs
Phase 1: move the per-task plugin handlers (and the iceberg subpackage)
into their task packages.
weed/plugin/worker/vacuum_handler.go -> weed/worker/tasks/vacuum/plugin_handler.go
weed/plugin/worker/ec_balance_handler.go -> weed/worker/tasks/ec_balance/plugin_handler.go
weed/plugin/worker/erasure_coding_handler.go -> weed/worker/tasks/erasure_coding/plugin_handler.go
weed/plugin/worker/volume_balance_handler.go -> weed/worker/tasks/balance/plugin_handler.go
weed/plugin/worker/iceberg/ -> weed/worker/tasks/iceberg/
weed/plugin/worker/handlers/handlers.go now blank-imports all five
task subpackages so their init() registrations fire.
weed/command/mini.go and the worker tests construct the handler with
vacuum.DefaultMaxExecutionConcurrency (the constant moved with the
vacuum handler).
admin_script remains in weed/plugin/worker/ because there is no
underlying weed/worker/tasks/admin_script/ package to merge with.
* refactor(worker): update test/plugin_workers imports for moved handlers
Three handler constructors moved out of pluginworker into their task
packages — update the integration test files in test/plugin_workers/
to import from the new locations:
pluginworker.NewVacuumHandler -> vacuum.NewVacuumHandler
pluginworker.NewVolumeBalanceHandler -> balance.NewVolumeBalanceHandler
pluginworker.NewErasureCodingHandler -> erasure_coding.NewErasureCodingHandler
The pluginworker import is kept where the file still uses
pluginworker.WorkerOptions / pluginworker.JobHandler.
* refactor(worker): update test/s3tables iceberg import path
The iceberg subpackage moved from weed/plugin/worker/iceberg/ to
weed/worker/tasks/iceberg/. test/s3tables/maintenance/maintenance_integration_test.go
still imported the old path, breaking S3 Tables / RisingWave / Trino /
Spark / Iceberg-catalog / STS integration test builds.
Mirrors the OSS-side fix needed by every job in the run that
transitively imports test/s3tables/maintenance.
* chore: gofmt PR-touched files
The S3 Tables Format Check job runs `gofmt -l` over weed/s3api/s3tables
and test/s3tables, then fails if anything is unformatted. Files this
PR moved or modified had import-grouping and trailing-spacing issues
introduced by perl-based renames; reformat them with gofmt -w.
Touched files:
test/plugin_workers/erasure_coding/{detection,execution}_test.go
test/s3tables/maintenance/maintenance_integration_test.go
weed/plugin/worker/handlers/handlers.go
weed/worker/tasks/{balance,ec_balance,erasure_coding,vacuum}/plugin_handler*.go
* refactor(worker): bounds-checked int conversions for plugin config values
CodeQL flagged 18 go/incorrect-integer-conversion warnings on the moved
plugin handler files: results of pluginworker.ReadInt64Config (which
ultimately calls strconv.ParseInt with bit size 64) were being narrowed
to int32/uint32/int without an upper-bound check, so a malicious or
malformed admin/worker config value could overflow the target type.
Add three helpers in weed/plugin/worker/config.go that wrap
ReadInt64Config and clamp out-of-range values back to the caller's
fallback:
ReadInt32Config (math.MinInt32 .. math.MaxInt32)
ReadUint32Config (0 .. math.MaxUint32)
ReadIntConfig (math.MinInt32 .. math.MaxInt32, platform-portable)
Update each flagged call site in the four moved task packages to use
the bounds-checked helper. For protobuf uint32 fields (volume IDs)
the variable type also becomes uint32, removing the trailing
uint32(volumeID) casts and changing the "missing volume_id" check
from `<= 0` to `== 0`.
Touched files:
weed/plugin/worker/config.go
weed/worker/tasks/balance/plugin_handler.go
weed/worker/tasks/erasure_coding/plugin_handler.go
weed/worker/tasks/vacuum/plugin_handler.go
* refactor(worker): use ReadIntConfig for clamped derive-worker-config helpers
CodeQL still flagged three call sites where ReadInt64Config was being
narrowed to int after a value-range clamp (max_concurrent_moves <= 50,
batch_size <= 100, min_server_count >= 2). The clamp is correct but
CodeQL's flow analysis didn't recognize the bound, so it flagged them
as unbounded narrowing.
Switch to ReadIntConfig (already int32-bounded by the helper) for
those three sites, drop the now-redundant int64 intermediate variables.
Also drops the now-unused `> math.MaxInt32` clamp in
ec_balance.deriveECBalanceWorkerConfig (the helper covers it).
|
||
|
Chris Lu
|
b2f4ebb776 |
test(s3tables): add Dremio Iceberg catalog integration tests (#9299)
* test(s3tables): add Dremio Iceberg catalog integration tests
Add comprehensive integration tests for Dremio with SeaweedFS's Iceberg
REST Catalog, following the same patterns as existing Spark and Trino tests.
Tests include:
- Basic catalog connectivity and schema operations
- Table creation, insertion, and querying (CRUD)
- Deterministic table location specification
- Multi-level namespace support
Implementation includes:
- dremio_catalog_test.go: Core test environment and basic operations
- dremio_crud_operations_test.go: Schema and table CRUD testing
- dremio_deterministic_location_test.go: Location and namespace testing
- Comprehensive README and implementation documentation
CI/CD:
- Added dremio-iceberg-catalog-tests job to s3-tables-tests.yml
- Pre-pulls Dremio image, runs with 25m timeout
- Uploads artifacts on failure
* add docstrings to Dremio integration tests and fix CI image pre-pull
- Add function docstrings to all test functions and helper functions
in dremio_catalog_test.go, dremio_crud_operations_test.go, and
dremio_deterministic_location_test.go to improve code documentation
and satisfy CodeRabbit's docstring coverage requirements.
- Make Dremio Docker image pre-pull non-critical in CI workflow.
The pre-pull was failing with access denied error, but the image
can still be pulled at runtime. Using continue-on-error to allow
tests to proceed.
* fix: correct YAML syntax in Dremio CI workflow
Use multi-line run command with pipe operator (|) instead of
inline command with || operator to avoid YAML parsing errors.
The || operator was causing 'mapping values are not allowed here'
syntax errors in the YAML parser.
* make Dremio tests gracefully skip if container unavailable
Modify startDremioContainer and waitForDremio to return boolean values
instead of fataling. Tests now skip gracefully if:
- Dremio Docker image is unavailable
- Container fails to start
- Container doesn't become ready within timeout
This prevents CI failure when Dremio image is not accessible while
still testing the integration when it is available.
* Revert "make Dremio tests gracefully skip if container unavailable"
This reverts commit
|
||
|
Chris Lu
|
d92c5e057a |
test(iceberg): cross-engine regression coverage for deterministic table locations (#9074) (#9253)
* test(trino): regression for unique-table-location=false (#9074) With #9246's namespace-location property, Trino's REST catalog can resolve table locations even when the connector is configured with `iceberg.unique-table-location=false`, and CREATE/CTAS lands at the deterministic <namespace-location>/<tableName> path with no UUID-suffixed sibling. Lock that in: - writeTrinoConfig now parameterizes the unique-table-location flag via a withDeterministicTableLocation() option. - setupTrinoTest forwards config options. - TestTrinoDeterministicLocationCTAS exercises a fresh CREATE TABLE + CTAS with the flag flipped off and asserts the on-disk layout has no UUID-suffixed sibling under the namespace dir, proving each table occupies a single dir. Refs #9074 * test(spark): regression for CTAS without explicit location (#9074) iceberg-spark has no equivalent of Trino's unique-table-location flag — its REST catalog interactions always produce the deterministic <namespace-location>/<tableName> path. Without #9246's namespace-location property, Spark cannot resolve a table location for a CREATE TABLE that omits an explicit LOCATION clause; with it, the operation succeeds and the table lands at the expected single-dir-per-table layout. TestSparkDeterministicLocationCTAS walks the same scenario as the Trino test: CREATE TABLE without LOCATION, INSERT, CTAS, SELECT count, then asserts via S3 ListObjectsV2 that no UUID-suffixed sibling directory appears under the namespace. Refs #9074 * test(duckdb): read table at deterministic location via REST catalog (#9074) DuckDB's iceberg extension is a read-only consumer in this flow — there is no client-side UUID-suffixing toggle to test. The relevant question post-#9246 is whether DuckDB can ATTACH the REST catalog and resolve a table at a deterministic <bucket>/<namespace>/<tableName> path produced by writers that don't suffix UUIDs (iceberg-spark, pyiceberg, Trino with unique-table-location=false). TestDuckDBDeterministicLocationRead creates a namespace + minimal table via direct REST API calls (so no client-side UUID is added), confirms the catalog returns a deterministic location URL, then runs DuckDB through ATTACH ... TYPE 'ICEBERG' and DESCRIBE on the table. Asserting DESCRIBE succeeds proves DuckDB walked the catalog → metadata → schema chain against the deterministic on-disk path. The test skips gracefully when the DuckDB image lacks the iceberg extension or the ATTACH-iceberg syntax, so older base images don't fail the suite. Refs #9074 |
||
|
Chris Lu
|
f50917224a |
fix(iceberg): default namespace location so fresh CTAS does not race metadata write (#9074) (#9246)
* fix(iceberg): advertise default namespace location for REST clients
Trino's REST catalog has two code paths for CREATE TABLE depending on
whether a table location can be resolved before the catalog call:
// TrinoRestCatalog.newCreateTableTransaction (Trino 479)
if (location.isEmpty()) {
return tableBuilder.create().newTransaction(); // EAGER: REST POST now
}
return tableBuilder.withLocation(...).createTransaction(); // DEFERRED
`tableLocation` resolves to null when the user does not pass
`location = '...'` AND `defaultTableLocation` returns null. The latter
happens whenever the namespace's `loadNamespaceMetadata` response has no
`location` property — and our handler returned exactly that.
In the eager branch Trino calls REST POST /v1/.../tables immediately, so
our handleCreateTable persists `<location>/metadata/v1.metadata.json` to
the filer. Trino's IcebergMetadata.beginCreateTable then runs
`fileSystem.listFiles(location).hasNext()` on the same path, finds the
metadata file we just wrote, and throws "Cannot create a table on a
non-empty location" — even on a brand-new schema and table name.
Synthesize a default `location` of `s3://<bucket>/<flattened-namespace>`
on Get/Create namespace responses when one is not stored. With a non-
null `defaultTableLocation`, Trino takes the deferred branch, picks a
unique `<namespace-location>/<table>-<UUID>` path (UUID added by the
standard `iceberg.unique-table-location=true` setting), and the empty-
location check passes. The actual REST POST is deferred to commit time,
so our metadata write lands alongside the data files Trino has already
produced.
Existing namespaces with an explicit `location` property are untouched —
the synthesis only kicks in when the property is absent.
Refs #9074
* test(trino): regression for fresh CREATE TABLE without explicit location
Exercises the follow-up scenario reported on issue #9074: a CREATE TABLE
on a brand-new schema and brand-new table name with NO `location = '...'`
clause, followed by a CTAS on top — exactly the SQL pattern from the
report. Before advertising a default namespace location, the first
CREATE TABLE failed with
Cannot create a table on a non-empty location:
s3://iceberg-tables/<schema>/<table>, set
'iceberg.unique-table-location=true' in your Iceberg catalog
properties to use unique table locations for every table.
even though the bucket was genuinely empty. The bug surfaced because our
handleCreateTable persists `<location>/metadata/v1.metadata.json` during
Trino's eager-create branch, and Trino's post-create listFiles
emptiness check then trips on the metadata file we just wrote.
The test asserts the CTAS succeeds AND the resulting table contains the
source rows, since the reporter saw the table get created with empty
data when the query failed.
Refs #9074
|
||
|
Chris Lu
|
96e5fea08e |
test(catalog_spark): bound weed shell invocation with 30s timeout
createTableBucket ran `weed shell` via exec.Command with no deadline. When the shell's first command retries on a transient master connection blip, the trailing `exit` on stdin never gets processed and the subprocess blocks until the outer 20m `go test` timeout fires — the surfacing symptom is a flaky 20m panic with no diagnostic output. Wrap the invocation in exec.CommandContext with a 30s timeout, matching the existing pattern in test/s3tables/catalog_risingwave/setup_test.go. |
||
|
Chris Lu
|
8857dbfb74 |
fix(test): drop host port mapping from risingwave catalog test to kill TOCTOU flake
The random host port allocated by MustFreeMiniPorts was released before docker run bound it, occasionally losing the race to another process and failing with "address already in use". The sidecar already reaches RisingWave via shared netns (--network container:...), so the host -p mapping and the corresponding WaitForPort check were unused. |
||
|
Chris Lu
|
9554e259dd |
fix(iceberg): route catalog clients to the right bucket and vend S3 endpoint (#9109)
* fix(iceberg): route catalog clients to the right bucket and vend S3 endpoint DuckDB ATTACH 's3://<bucket>/' AS cat (TYPE 'ICEBERG', ...) was failing with "schema does not exist" because GET /v1/config ignored the warehouse query parameter and returned no overrides.prefix, so subsequent requests fell through to the hard-coded "warehouse" default bucket instead of the one the client attached. LoadTable also returned an empty config, forcing clients to discover the S3 endpoint out-of-band and producing 403s on direct iceberg_scan calls. - handleConfig now echoes overrides.prefix = bucket and defaults.warehouse when ?warehouse=s3://<bucket>/ is supplied. - getBucketFromPrefix honors a warehouse query parameter as a fallback for clients that skip the /v1/config handshake. - LoadTable responses advertise s3.endpoint and s3.path-style-access so clients can reach data files without separate configuration. Refs #9103 * address review feedback on iceberg S3 endpoint vending - deriveS3AdvertisedEndpoint is now a method on S3Options; honors externalUrl / S3_EXTERNAL_URL, switches to https when -s3.key.file is set, uses the https port when configured, and brackets IPv6 literals via util.JoinHostPort. - handleCreateTable returns s.buildFileIOConfig() in both its staged and final LoadTableResult branches so create and load flows see the same FileIO hints. - Add unit test coverage for the endpoint derivation scenarios. * address CI and review feedback for #9109 - DuckDB integration test now runs under its own newOAuthTestEnv (with a valid IAM config) so the OAuth2 client_credentials flow DuckDB requires actually works; the shared env has no registered credentials, which was the cause of the CI failure. Helper createTableWithToken was added to create tables via Bearer auth. - Tighten TestIssue9103_LoadTableDoesNotVendS3FileIOCredentials to also assert s3.path-style-access = "true", so a partial regression where the endpoint is vended but path-style is dropped still fails. - deriveS3AdvertisedEndpoint now logs a startup hint when it infers the host from os.Hostname because the bind IP is a wildcard, pointing operators at -s3.externalUrl / S3_EXTERNAL_URL for reverse-proxy deployments where the inferred name is not externally reachable. - handleConfig has a comment explaining that any sub-path in the warehouse URL is dropped because catalog routing is bucket-scoped. * fix(iceberg): make advertised S3 endpoint strictly opt-in; add region The wildcard-bind fallback to os.Hostname() in deriveS3AdvertisedEndpoint was hijacking correctly-configured clients: on the CI runner it produced http://runnervmrc6n4:<port>, which Spark (running in Docker) could not resolve, so Spark iceberg tests began failing after the endpoint started being vended in LoadTable responses. Change the rule so advertising is opt-in and never guesses a host that might not be routable: - -s3.externalUrl / S3_EXTERNAL_URL wins (covers reverse-proxy). - Otherwise, only an explicit, non-wildcard -s3.bindIp is used. - Wildcard / empty bind returns "" so no FileIO endpoint is vended and existing clients keep using their own configuration. buildFileIOConfig additionally vends s3.region (defaulting to the same value baked into table bucket ARNs) whenever it vends an endpoint, so DuckDB's attach does not fail with "No region was provided via the vended credentials" when the operator has opted in. The DuckDB issue-9103 integration test runs under an env with a wildcard bind, so it explicitly sets AWS_REGION in the docker run to pick up the same default. The HTTP-level LoadTable-vending test was dropped because its expectation is now conditional and already covered by unit tests in iceberg_issue_9103_test.go. |
||
|
Chris Lu
|
2818251dd5 |
fix(iceberg): clean stale data before creating a table (#9074) (#9077)
* fix(iceberg): clean stale data before creating a table CREATE TABLE AS from Trino fails against the SeaweedFS Iceberg REST catalog with "Cannot create a table on a non-empty location". The catalog assigns every new table the deterministic <ns>/<table> path, and Trino's pre-write check rejects the CTAS whenever leftover objects live there — typically files from a prior DROP that did not purge the data, or an earlier aborted CTAS. Make the catalog the authority for table existence: before writing any metadata, look up the table in the S3 Tables catalog. If it is already registered, return the existing definition (idempotent create). If it is not registered, any objects still sitting at the target location are stale, so purge them before proceeding. Live tables are never touched — the cleanup path is guarded by the catalog lookup. Fixes #9074 * test(trino): regression for create/drop/recreate table (#9074) Exercises the exact sequence from the reported bug: CREATE TABLE without an explicit location, INSERT, DROP, then CREATE again with the same name, followed by a CTAS on top. Previously the recreate failed with "Cannot create a table on a non-empty location" because stale data files from the dropped table lingered at the deterministic <schema>/<table> path. The test also asserts the recreated table does not see the dropped data and that a drop/recreate CTAS cycle works. * fix(iceberg): purge dropped table location on DROP TABLE Recreate-at-same-location was failing with "Cannot create a table on a non-empty location" because DROP TABLE only removed the catalog entry and left data files behind. Trino's pre-write emptiness check then rejected the subsequent CREATE. Look up the table's storage location before deleting the catalog entry, and after a successful DeleteTable, purge the filer subtree at that location. The lookup uses the catalog — the authoritative owner of the name→location mapping — so cleanup is gated on a live table having existed; failed lookups skip cleanup and leave storage alone. Also pin the regression test to an explicit, fixed location so the recreate genuinely targets the same path the drop was supposed to free. * refactor(iceberg): use errors.As in isNoSuchTableError * fix(iceberg): drop create-preflight cleanup; harden cleanup path - Remove the destructive cleanupStaleTableLocation call from the CreateTable preflight. Storage cleanup now lives exclusively on the DROP path, so CreateTable has no side effects on storage beyond the new table's own metadata write. - Validate tablePath in cleanupStaleTableLocation by rejecting empty, ".", "..", or backslash-bearing segments before joining with the bucket prefix, so a crafted location cannot escape the bucket subtree. path.Clean would silently collapse traversal, so segments are checked raw. * test(trino): defer table drops in recreate test for failure-safe cleanup |
||
|
Chris Lu
|
2b8c16160f |
feat(iceberg): add OAuth2 token endpoint for DuckDB compatibility (#9017)
* feat(iceberg): add OAuth2 token endpoint for DuckDB compatibility (#9015) DuckDB's Iceberg connector uses OAuth2 client_credentials flow, hitting POST /v1/oauth/tokens which was not implemented, returning 404. Add the OAuth2 token endpoint that accepts S3 access key / secret key as client_id / client_secret, validates them against IAM, and returns a signed JWT bearer token. The Auth middleware now accepts Bearer tokens in addition to S3 signature auth. * fix(test): use weed shell for table bucket creation with IAM enabled The S3 Tables REST API requires SigV4 auth when IAM is configured. Use weed shell (which bypasses S3 auth) to create table buckets, matching the pattern used by the Trino integration tests. * address review feedback: access key in JWT, full identity in Bearer auth - Include AccessKey in JWT claims so token verification uses the exact credential that signed the token (no ambiguity with multi-key identities) - Return full Identity object from Bearer auth so downstream IAM/policy code sees an authenticated request, not anonymous - Replace GetSecretKeyForIdentity with GetCredentialByAccessKey for unambiguous credential lookup - DuckDB test now tries the full SQL script first (CREATE SECRET + catalog access), falling back to simple CREATE SECRET if needed - Tighten bearer auth test assertion to only accept 200/500 Addresses review comments from coderabbitai and gemini-code-assist. * security: use PostFormValue, bind signing key to access key, fix port conflict - Use r.PostFormValue instead of r.FormValue to prevent credentials from leaking via query string into logs and caches - Reject client_secret in URL query parameters explicitly - Include access key in HMAC signing key derivation to prevent cross-credential token forgery when secrets happen to match - Allocate dedicated webdav port in OAuth test env to avoid port collision with the shared TestMain cluster |
||
|
Chris Lu
|
fbe758efa8 |
test: consolidate port allocation into shared test/testutil package (#8982)
* test: consolidate port allocation into shared test/testutil package Move duplicated port allocation logic from 15+ test files into a single shared package at test/testutil/. This fixes a port collision bug where independently allocated ports could overlap via the gRPC offset (port+10000), causing weed mini to reject the configuration. The shared package provides: - AllocatePorts: atomic allocation of N unique ports - AllocateMiniPorts/MustFreeMiniPorts: gRPC-offset-aware allocation that prevents port A+10000 == port B collisions - WaitForPort, WaitForService, FindBindIP, WriteIAMConfig, HasDocker * test: address review feedback and fix FUSE build - Revert fuse_integration change: it has its own go.mod and cannot import the shared testutil package - AllocateMiniPorts: hold all listeners open until the entire batch is allocated, preventing race conditions where other processes steal ports - HasDocker: add 5s context timeout to avoid hanging on stalled Docker - WaitForService: only treat 2xx HTTP status codes as ready * test: use global rand in AllocateMiniPorts for better seeding Go 1.20+ auto-seeds the global rand generator. Using it avoids identical sequences when multiple tests call at the same nanosecond. * test: revert WaitForService status code check S3 endpoints return non-2xx (e.g. 403) on bare GET requests, so requiring 2xx caused the S3 integration test to time out. Any HTTP response is sufficient proof that the service is running. * test: fix gofmt formatting in s3tables test files |
||
|
Chris Lu
|
761ec7da00 |
fix(iceberg): use dot separator for namespace paths instead of unit separator (#8960)
* fix(iceberg): use dot separator for namespace paths instead of unit separator The Iceberg REST Catalog handler was using \x1F (unit separator) to join multi-level namespaces when constructing S3 location and filer paths. The S3 Tables storage layer uses "." (dot) as the namespace separator, causing tables created via the Iceberg REST API to point to different paths than where S3 Tables actually stores them. Fixes #8959 * fix(iceberg): use dot separator in log messages for readable namespace output * fix(iceberg): use path.Join for S3 location path segments Use path.Join to construct the namespace/table path segments in fallback S3 locations for robustness and consistency with handleCreateTable. * test(iceberg): add multi-level namespace integration tests for Spark and Trino Add regression tests for #8959 that create a two-level namespace (e.g. "analytics.daily"), create a table under it, insert data, and query it back. This exercises the dot-separated namespace path construction and verifies that Spark/Trino can actually read the data at the S3 location returned by the Iceberg REST API. * fix(test): enable nested namespace in Trino Iceberg catalog config Trino requires `iceberg.rest-catalog.nested-namespace-enabled=true` to support multi-level namespaces. Without this, CREATE SCHEMA with a dotted name fails with "Nested namespace is not enabled for this catalog". * fix(test): parse Trino COUNT(*) output as integer instead of substring match Avoids false matches from strings.Contains(output, "3") by parsing the actual numeric result with strconv.Atoi and asserting equality. * fix(test): use separate Trino config for nested namespace test The nested-namespace-enabled=true setting in Trino changes how SHOW SCHEMAS works, causing "Internal error" for all tests sharing that catalog config. Move the flag to a dedicated config used only by TestTrinoMultiLevelNamespace. * fix(iceberg): support parent query parameter in ListNamespaces for nested namespaces Add handling for the Iceberg REST spec's `parent` query parameter in handleListNamespaces. When Trino has nested-namespace-enabled=true, it sends `GET /v1/namespaces?parent=<ns>` to list child namespaces. The parent value is decoded from the Iceberg unit separator format and converted to a dot-separated prefix for the S3 Tables layer. Also simplify TestTrinoMultiLevelNamespace to focus on namespace operations (create, list, show tables) rather than data operations, since Trino's REST catalog has a non-empty location check that conflicts with server-side metadata creation. * fix(test): expand Trino multi-level namespace test and merge config helpers - Expand TestTrinoMultiLevelNamespace to create a table with explicit location, insert rows, query them back, and verify the S3 file path contains the dot-separated namespace (not \x1F). This ensures the original #8959 bug would be caught by the Trino integration test. - Merge writeTrinoConfig and writeTrinoNestedNamespaceConfig into a single parameterized function using functional options. |
||
|
Chris Lu
|
e7fc243ee1 |
Fix flaky s3tables tests: allocate all ports atomically
The test port allocation had a TOCTOU race where GetFreePort() would open a listener, grab the port number, then immediately close it. When called repeatedly, the OS could recycle a just-released port, causing two services (e.g. Filer and S3) to be assigned the same port. Replace per-call GetFreePort() with batch AllocatePorts() that holds all listeners open until every port is obtained, matching the pattern already used in test/volume_server/framework/cluster.go. |
||
|
Chris Lu
|
d5068b3ee6 | test: harden weed mini readiness checks | ||
|
Chris LuandCopilot
|
a00eddb525 |
Iceberg table maintenance Phase 3: multi-spec compaction, delete handling, and metrics (#8643)
* Add multi-partition-spec compaction and delete-aware compaction (Phase 3) Multi-partition-spec compaction: - Add SpecID to compactionBin struct and group by spec+partition key - Remove the len(specIDs) > 1 skip that blocked spec-evolved tables - Write per-spec manifests in compaction commit using specByID map - Use per-bin PartitionSpec when calling NewDataFileBuilder Delete-aware compaction: - Add ApplyDeletes config (default: true) with readBoolConfig helper - Implement position delete collection (file_path + pos Parquet columns) - Implement equality delete collection (field ID to column mapping) - Update mergeParquetFiles to filter rows via position deletes (binary search) and equality deletes (hash set lookup) - Smart delete manifest carry-forward: drop when all data files compacted - Fix EXISTING/DELETED entries to include sequence numbers Tests for multi-spec bins, delete collection, merge filtering, and end-to-end compaction with position/equality/mixed deletes. * Add structured metrics and per-bin progress to iceberg maintenance - Change return type of all four operations from (string, error) to (string, map[string]int64, error) with structured metric counts (files_merged, snapshots_expired, orphans_removed, duration_ms, etc.) - Add onProgress callback to compactDataFiles for per-bin progress - In Execute, pass progress callback that sends JobProgressUpdate with per-bin stage messages - Accumulate per-operation metrics with dot-prefixed keys (e.g. compact.files_merged) into OutputValues on completion - Update testing_api.go wrappers and integration test call sites - Add tests: TestCompactDataFilesMetrics, TestExpireSnapshotsMetrics, TestExecuteCompletionOutputValues * Address review feedback: group equality deletes by field IDs, use metric constants - Group equality deletes by distinct equality_ids sets so different delete files with different equality columns are handled correctly - Use length-prefixed type-aware encoding in buildEqualityKey to avoid ambiguity between types and collisions from null bytes - Extract metric key strings into package-level constants * Fix buildEqualityKey to use length-prefixed type-aware encoding The previous implementation used plain String() concatenation with null byte separators, which caused type ambiguity (int 123 vs string "123") and separator collisions when values contain null bytes. Now each value is serialized as "kind:length:value" for unambiguous composite keys. This fix was missed in the prior cherry-pick due to a merge conflict. * Address nitpick review comments - Document patchManifestContentToDeletes workaround: explain that iceberg-go WriteManifest cannot create delete manifests, and note the fail-fast validation on pattern match - Document makeTestEntries: note that specID field is ignored and callers should use makeTestEntriesWithSpec for multi-spec testing * fmt * Fix path normalization, manifest threshold, and artifact filename collisions - Normalize file paths in position delete collection and lookup so that absolute S3 URLs and relative paths match correctly - Fix rewriteManifests threshold check to count only data manifests (was including delete manifests in the count and metric) - Add random suffix to artifact filenames in compactDataFiles and rewriteManifests to prevent collisions between concurrent runs - Sort compaction bins by SpecID then PartitionKey for deterministic ordering across specs * Fix pos delete read, deduplicate column resolution, minor cleanups - Remove broken Column() guard in position delete reading that silently defaulted pos to 0; unconditionally extract Int64() instead - Deduplicate column resolution in readEqualityDeleteFile by calling resolveEqualityColIndices instead of inlining the same logic - Add warning log in readBoolConfig for unrecognized string values - Fix CompactDataFiles call site in integration test to capture 3 return values * Advance progress on all bins, deterministic manifest order, assert metrics - Call onProgress for every bin iteration including skipped/failed bins so progress reporting never appears stalled - Sort spec IDs before iterating specEntriesMap to produce deterministic manifest list ordering across runs - Assert expected metric keys in CompactDataFiles integration test --------- Co-authored-by: Copilot <copilot@github.com> |
||
|
Chris Lu
|
e24630251c |
iceberg: handle filer-backed compaction inputs (#8638)
* iceberg: handle filer-backed compaction inputs * iceberg: preserve upsert creation times * iceberg: align compaction test schema * iceberg: tighten compact output assertion * iceberg: document compact output match * iceberg: clear stale chunks in upsert helper * iceberg: strengthen compaction integration coverage |
||
|
|
8cde3d4486 |
Add data file compaction to iceberg maintenance (Phase 2) (#8503)
* Add iceberg_maintenance plugin worker handler (Phase 1) Implement automated Iceberg table maintenance as a new plugin worker job type. The handler scans S3 table buckets for tables needing maintenance and executes operations in the correct Iceberg order: expire snapshots, remove orphan files, and rewrite manifests. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add data file compaction to iceberg maintenance handler (Phase 2) Implement bin-packing compaction for small Parquet data files: - Enumerate data files from manifests, group by partition - Merge small files using parquet-go (read rows, write merged output) - Create new manifest with ADDED/DELETED/EXISTING entries - Commit new snapshot with compaction metadata Add 'compact' operation to maintenance order (runs before expire_snapshots), configurable via target_file_size_bytes and min_input_files thresholds. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix memory exhaustion in mergeParquetFiles by processing files sequentially Previously all source Parquet files were loaded into memory simultaneously, risking OOM when a compaction bin contained many small files. Now each file is loaded, its rows are streamed into the output writer, and its data is released before the next file is loaded — keeping peak memory proportional to one input file plus the output buffer. * Validate bucket/namespace/table names against path traversal Reject names containing '..', '/', or '\' in Execute to prevent directory traversal via crafted job parameters. * Add filer address failover in iceberg maintenance handler Try each filer address from cluster context in order instead of only using the first one. This improves resilience when the primary filer is temporarily unreachable. * Add separate MinManifestsToRewrite config for manifest rewrite threshold The rewrite_manifests operation was reusing MinInputFiles (meant for compaction bin file counts) as its manifest count threshold. Add a dedicated MinManifestsToRewrite field with its own config UI section and default value (5) so the two thresholds can be tuned independently. * Fix risky mtime fallback in orphan removal that could delete new files When entry.Attributes is nil, mtime defaulted to Unix epoch (1970), which would always be older than the safety threshold, causing the file to be treated as eligible for deletion. Skip entries with nil Attributes instead, matching the safer logic in operations.go. * Fix undefined function references in iceberg_maintenance_handler.go Use the exported function names (ShouldSkipDetectionByInterval, BuildDetectorActivity, BuildExecutorActivity) matching their definitions in vacuum_handler.go. * Remove duplicated iceberg maintenance handler in favor of iceberg/ subpackage The IcebergMaintenanceHandler and its compaction code in the parent pluginworker package duplicated the logic already present in the iceberg/ subpackage (which self-registers via init()). The old code lacked stale-plan guards, proper path normalization, CAS-based xattr updates, and error-returning parseOperations. Since the registry pattern (default "all") makes the old handler unreachable, remove it entirely. All functionality is provided by iceberg.Handler with the reviewed improvements. * Fix MinManifestsToRewrite clamping to match UI minimum of 2 The clamp reset values below 2 to the default of 5, contradicting the UI's advertised MinValue of 2. Clamp to 2 instead. * Sort entries by size descending in splitOversizedBin for better packing Entries were processed in insertion order which is non-deterministic from map iteration. Sorting largest-first before the splitting loop improves bin packing efficiency by filling bins more evenly. * Add context cancellation check to drainReader loop The row-streaming loop in drainReader did not check ctx between iterations, making long compaction merges uncancellable. Check ctx.Done() at the top of each iteration. * Fix splitOversizedBin to always respect targetSize limit The minFiles check in the split condition allowed bins to grow past targetSize when they had fewer than minFiles entries, defeating the OOM protection. Now bins always split at targetSize, and a trailing runt with fewer than minFiles entries is merged into the previous bin. * Add integration tests for iceberg table maintenance plugin worker Tests start a real weed mini cluster, create S3 buckets and Iceberg table metadata via filer gRPC, then exercise the iceberg.Handler operations (ExpireSnapshots, RemoveOrphans, RewriteManifests) against the live filer. A full maintenance cycle test runs all operations in sequence and verifies metadata consistency. Also adds exported method wrappers (testing_api.go) so the integration test package can call the unexported handler methods. * Fix splitOversizedBin dropping files and add source path to drainReader errors The runt-merge step could leave leading bins with fewer than minFiles entries (e.g. [80,80,10,10] with targetSize=100, minFiles=2 would drop the first 80-byte file). Replace the filter-based approach with an iterative merge that folds any sub-minFiles bin into its smallest neighbor, preserving all eligible files. Also add the source file path to drainReader error messages so callers can identify which Parquet file caused a read/write failure. * Harden integration test error handling - s3put: fail immediately on HTTP 4xx/5xx instead of logging and continuing - lookupEntry: distinguish NotFound (return nil) from unexpected RPC errors (fail the test) - writeOrphan and orphan creation in FullMaintenanceCycle: check CreateEntryResponse.Error in addition to the RPC error * go fmt --------- Co-authored-by: Copilot <copilot@github.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Chris Lu
|
fb944f0071 |
test: add Polaris S3 tables integration tests (#8489)
* test: add polaris integration test harness * test: add polaris integration coverage * ci: run polaris s3 tables tests * test: harden polaris harness * test: DRY polaris integration tests * ci: pre-pull Polaris image * test: extend Polaris pull timeout * test: refine polaris credentials selection * test: keep Polaris tables inside allowed location * test: use fresh context for polaris cleanup * test: prefer specific Polaris storage credential * test: tolerate Polaris credential variants * test: request Polaris vended credentials * test: load Polaris table credentials * test: allow polaris vended access via bucket policy * test: align Polaris object keys with table location * test: rename Polaris vended role references * test: simplify Polaris vended credential extraction * test: marshal Polaris bucket policy |
||
|
Chris Lu
|
340339f678 |
Add Apache Polaris integration tests (#8478)
* test: add polaris integration test harness * test: add polaris integration coverage * ci: run polaris s3 tables tests * test: harden polaris harness * test: DRY polaris integration tests * ci: pre-pull Polaris image * test: extend Polaris pull timeout * test: refine polaris credentials selection * test: keep Polaris tables inside allowed location * test: use fresh context for polaris cleanup * test: prefer specific Polaris storage credential * test: tolerate Polaris credential variants * test: request Polaris vended credentials * test: load Polaris table credentials * test: allow polaris vended access via bucket policy * test: align Polaris object keys with table location |
||
|
Chris LuandCopilot
|
c5d5b517f6 |
Add lakekeeper table bucket integration test (#8470)
* Add lakekeeper table bucket integration test Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Convert lakekeeper table bucket test to Go Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add multipart upload to lakekeeper table bucket test Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove lakekeeper test skips Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Convert lakekeeper repro to Go Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Chris Lu
|
91f59e73e5 | close ports | ||
|
Chris Lu
|
36c469e34e |
Enforce IAM for S3 Tables bucket creation (#8388)
* Enforce IAM for s3tables bucket creation * Prefer IAM path when policies exist * Ensure IAM enforcement honors default allow * address comments * Reused the precomputed principal when setting tableBucketMetadata.OwnerAccountID, avoiding the redundant getAccountID call. * get identity * fix * dedup * fix * comments * fix tests * update iam config * go fmt * fix ports * fix flags * mini clean shutdown * Revert "update iam config" This reverts commit |
||
|
Chris Lu
|
0d8588e3ae |
S3: Implement IAM defaults and STS signing key fallback (#8348)
* S3: Implement IAM defaults and STS signing key fallback logic * S3: Refactor startup order to init SSE-S3 key manager before IAM * S3: Derive STS signing key from KEK using HKDF for security isolation * S3: Document STS signing key fallback in security.toml * fix(s3api): refine anonymous access logic and secure-by-default behavior - Initialize anonymous identity by default in `NewIdentityAccessManagement` to prevent nil pointer exceptions. - Ensure `ReplaceS3ApiConfiguration` preserves the anonymous identity if not present in the new configuration. - Update `NewIdentityAccessManagement` signature to accept `filerClient`. - In legacy mode (no policy engine), anonymous defaults to Deny (no actions), preserving secure-by-default behavior. - Use specific `LookupAnonymous` method instead of generic map lookup. - Update tests to accommodate signature changes and verify improved anonymous handling. * feat(s3api): make IAM configuration optional - Start S3 API server without a configuration file if `EnableIam` option is set. - Default to `Allow` effect for policy engine when no configuration is provided (Zero-Config mode). - Handle empty configuration path gracefully in `loadIAMManagerFromConfig`. - Add integration test `iam_optional_test.go` to verify empty config behavior. * fix(iamapi): fix signature mismatch in NewIdentityAccessManagementWithStore * fix(iamapi): properly initialize FilerClient instead of passing nil * fix(iamapi): properly initialize filer client for IAM management - Instead of passing `nil`, construct a `wdclient.FilerClient` using the provided `Filers` addresses. - Ensure `NewIdentityAccessManagementWithStore` receives a valid `filerClient` to avoid potential nil pointer dereferences or limited functionality. * clean: remove dead code in s3api_server.go * refactor(s3api): improve IAM initialization, safety and anonymous access security * fix(s3api): ensure IAM config loads from filer after client init * fix(s3): resolve test failures in integration, CORS, and tagging tests - Fix CORS tests by providing explicit anonymous permissions config - Fix S3 integration tests by setting admin credentials in init - Align tagging test credentials in CI with IAM defaults - Added goroutine to retry IAM config load in iamapi server * fix(s3): allow anonymous access to health targets and S3 Tables when identities are present * fix(ci): use /healthz for Caddy health check in awscli tests * iam, s3api: expose DefaultAllow from IAM and Policy Engine This allows checking the global "Open by Default" configuration from other components like S3 Tables. * s3api/s3tables: support DefaultAllow in permission logic and handler Updated CheckPermissionWithContext to respect the DefaultAllow flag in PolicyContext. This enables "Open by Default" behavior for unauthenticated access in zero-config environments. Added a targeted unit test to verify the logic. * s3api/s3tables: propagate DefaultAllow through handlers Propagated the DefaultAllow flag to individual handlers for namespaces, buckets, tables, policies, and tagging. This ensures consistent "Open by Default" behavior across all S3 Tables API endpoints. * s3api: wire up DefaultAllow for S3 Tables API initialization Updated registerS3TablesRoutes to query the global IAM configuration and set the DefaultAllow flag on the S3 Tables API server. This completes the end-to-end propagation required for anonymous access in zero-config environments. Added a SetDefaultAllow method to S3TablesApiServer to facilitate this. * s3api: fix tests by adding DefaultAllow to mock IAM integrations The IAMIntegration interface was updated to include DefaultAllow(), breaking several mock implementations in tests. This commit fixes the build errors by adding the missing method to the mocks. * env * ensure ports * env * env * fix default allow * add one more test using non-anonymous user * debug * add more debug * less logs |
||
|
Chris Lu
|
49a64f50f1 |
Add session policy support to IAM (#8338)
* Add session policy support to IAM - Implement policy evaluation for session tokens in policy_engine.go - Add session_policy field to session claims for tracking applied policies - Update STS service to include session policies in token generation - Add IAM integration tests for session policy validation - Update IAM manager to support policy attachment to sessions - Extend S3 API STS endpoint to handle session policy restrictions * fix: optimize session policy evaluation and add documentation * sts: add NormalizeSessionPolicy helper for inline session policies * sts: support inline session policies for AssumeRoleWithWebIdentity and credential-based flows * s3api: parse and normalize Policy parameter for STS HTTP handlers * tests: add session policy unit tests and integration tests for inline policy downscoping * tests: add s3tables STS inline policy integration * iam: handle user principals and validate tokens * sts: enforce inline session policy size limit * tests: harden s3tables STS integration config * iam: clarify principal policy resolution errors * tests: improve STS integration endpoint selection |
||
|
Chris LuandCopilot
|
796f23f68a |
Fix STS InvalidAccessKeyId and request body consumption issues (#8328)
* Fix STS InvalidAccessKeyId and request body consumption in Lakekeeper integration test * Remove debug prints * Add Lakekeeper integration tests to CI * Fix connection refused in CI by binding to 0.0.0.0 * Add timeout to docker run in Lakekeeper integration test * Update weed/s3api/auth_credentials.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Chris Lu
|
0082c47e04 |
Test: Add RisingWave DML verification test (#8317)
* Test: Verify RisingWave DML operations (INSERT, UPDATE, DELETE) support * Test: Refine RisingWave DML test (remove sleeps, use polling) |
||
|
Chris Lu
|
c1a9263e37 |
Fix STS AssumeRole with POST body param (#8320)
* Fix STS AssumeRole with POST body param and add integration test * Add STS integration test to CI workflow * Address code review feedback: fix HPP vulnerability and style issues * Refactor: address code review feedback - Fix HTTP Parameter Pollution vulnerability in UnifiedPostHandler - Refactor permission check logic for better readability - Extract test helpers to testutil/docker.go to reduce duplication - Clean up imports and simplify context setting * Add SigV4-style test variant for AssumeRole POST body routing - Added ActionInBodyWithSigV4Style test case to validate real-world scenario - Test confirms routing works correctly for AWS SigV4-signed requests - Addresses code review feedback about testing with SigV4 signatures * Fix: always set identity in context when non-nil - Ensure UnifiedPostHandler always calls SetIdentityInContext when identity is non-nil - Only call SetIdentityNameInContext when identity.Name is non-empty - This ensures downstream handlers (embeddedIam.DoActions) always have access to identity - Addresses potential issue where empty identity.Name would skip context setting |
||
|
Chris Lu
|
b8ef48c8f1 |
Add RisingWave catalog tests (#8308)
* Add RisingWave catalog tests for S3 tables * Add RisingWave catalog integration tests to CI workflow * Refactor RisingWave catalog tests based on PR feedback * Address PR feedback: optimize checks, cleanup logs * fix tests * consistent |
||
|
Chris Lu
|
7151181d54 | fix flaky tests |