mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-08 15:41:15 +02:00
2d4b730a2f744c5e608e0a1ff51c6ef59d05a49d
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 dependabot[bot]andChris Lu
|
2d4b730a2f |
build(deps): bump github.com/twmb/avro from 1.7.2 to 1.8.0 (#11210)
* build(deps): bump github.com/twmb/avro from 1.7.2 to 1.8.0 Bumps [github.com/twmb/avro](https://github.com/twmb/avro) from 1.7.2 to 1.8.0. - [Commits](https://github.com/twmb/avro/compare/v1.7.2...v1.8.0) --- updated-dependencies: - dependency-name: github.com/twmb/avro dependency-version: 1.8.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * iceberg: adapt to twmb/avro v1.8.0 and iceberg-go defensive copies avro v1.8.0 changes Schema.Root() to return *SchemaNode, which breaks iceberg-go v0.6.0's internal avro_schemas.go. The fix (apache/iceberg-go#1843) is only on iceberg-go's main branch, unreleased, so bump iceberg-go to that commit (c210509) alongside the avro bump. That iceberg-go revision also changes two behaviors seaweedfs worked around: - It now infers a manifest list's format version from the embedded writer schema, so a list missing the "format-version" header entry (DuckDB's shape is read as v2, not v1. ReadManifestList's header patching is now a redundant safety net; tests updated to expect v2. - It returns defensive copies from DataFile.Partition(), so the ReadManifest shim's in-place partition normalization was silently discarded. Rebuild the entry through NewDataFileBuilder when any partition value is normalized, copying every other DataFile field so manifest round-trips are preserved. - It converts day-transform partitions to iceberg.Date on read (applyDayTransformDates), so the day-partition cases the shim and tests guarded now convert without help; tests updated to expect iceberg.Date from the raw read. EOF ) * iceberg: accept assert-ref-snapshot-id without snapshot-id iceberg-go's new nullableInt64 parser rejects an assert-ref-snapshot-id requirement whose "snapshot-id" field is absent from the JSON, even though the Iceberg REST spec makes it optional (null means the ref must not already exist). v0.6.0 used a plain *int64, so absent was nil and accepted. ClickHouse sends the requirement without snapshot-id when asserting a branch does not yet exist, so its writes fail with "missing required field \"snapshot-id\"". normalizeRequirements splices an explicit null into any assert-ref-snapshot-id requirement missing the field before handing the JSON to iceberg-go's parser, restoring the v0.6.0 behavior across both iceberg-go versions. * iceberg: fix v1 block_size_in_bytes default in rebuilt manifest entries rebuildManifestEntry set block_size_in_bytes to 0, but the v1 manifest schema requires the default of 64 MiB ("Always write default in v1"). The original value is not exposed on the DataFile interface, so use the spec default. Also clarify the fallback comment to note that empty (zero-record / zero-byte) files also trigger it, not just a nil spec. Add a round-trip test that writes a rebuilt entry as v1 and verifies block_size_in_bytes is 64 MiB via Avro decoding. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
d044839ab2 |
iceberg: make a table commit a compare-and-swap (#10775)
* iceberg: make a table commit a compare-and-swap
The catalog validated the caller's version token, ran its authorization
checks, and only then wrote the new metadata xattr. Two engines
committing against the same base both passed that check and both wrote,
so the second silently dropped the first one's snapshot. Both also derive
the same v{N}.metadata.json name and the file write overwrote, leaving
the surviving pointer aimed at the loser's metadata - and the loser's
conflict cleanup then deleted the winner's file.
Write the metadata file with an exclusive create and update the xattr
conditionally on the bytes the handler read, the way the maintenance
worker already commits. A writer that lost the race re-reads and retries,
and reports 409 CommitFailedException once out of attempts.
* iceberg: stage a commit under a unique name when the versioned one is taken
Two follow-ups from review of the commit compare-and-swap:
Refusing to overwrite v{N}.metadata.json also refused to get past a file
left behind by a commit that died between staging and updating the
pointer. Every later commit derived the same name, saw the collision, and
reported a conflict, so the table stayed uncommittable until an orphan
sweep removed the file. Stage under v{N}-{uuid} instead: neither writer's
file is overwritten and the catalog pointer still decides who won, which
is how the maintenance worker has always staged its own metadata.
metadataVersionFromLocation learned to read the version back out of that
name.
The conditional update guarded only the metadata attribute while the
write replaced the whole entry, so a policy or tag written in the same
window was silently reverted. Guard every catalog attribute, which turns
that into a conflict the caller retries on fresh state.
* iceberg: give saveMetadataFile the exclusive flag instead of a second name
saveNewMetadataFile, saveMetadataBlobExclusive and uniqueMetadataFileName
were three new names around one existing helper. The flag now rides on
saveMetadataFile and saveMetadataBlob, and the unique-name construction
sits where it is used.
* iceberg: reuse the filer CAS helpers #10773 added, and stage transactions exclusively
#10773 landed mutateEntryExtended, which already writes an entry back under a
whole-entry precondition and retries. Drop the helper this branch added and
route the table commit through it: the check that the metadata is still the
one this request read now lives in the mutation, where it sees current state.
The policy the request was authorized against is asserted too, so an
administrator restricting it mid-commit sends the caller back through
authorization instead of having a stale decision applied. Bucket and
namespace policies live on other entries and a single-entry precondition
cannot cover them.
Multi-table transactions stage their metadata exclusively for the same
reason single-table commits do, and carry the name they landed on into the
pointer flip.
|
||
|
Chris Lu
|
ac6f3c92ef |
s3api/iceberg: report the reason a table schema was rejected (#10473)
* s3api/iceberg: report the reason a table schema was rejected newTableMetadata swallowed the iceberg-go error and returned nil, so every schema the metadata builder refused came back as a bare 500 "Failed to build table metadata". A v3-only column type is the common case: creating a table with a variant field but no format-version 3 property leaves the client with nothing, while "variant is not supported until v3" sits in the server log. Return the error instead and classify it. Schema, spec and argument failures are the caller's input, so they answer 400 with the underlying reason; the rest stay 500. Paths that build placeholder metadata with no schema keep their existing 500 via newEmptyTableMetadata. * s3api/iceberg: fail LoadTable when placeholder metadata cannot be built buildLoadTableResult dropped a nil from the placeholder path straight into the response. That serializes as "metadata":null under HTTP 200, which no Iceberg client can parse -- a worse outcome than the 500 the nil was meant to signal. Return an error instead and let the five callers answer 500. The nil-return convention goes away with it, so the commit and transaction paths check an error rather than a sentinel. * s3api/iceberg: route rejected schemas through writeManagerError The two helpers added here duplicated work the package already does. writeManagerError is the canonical error-to-response mapper -- it already downgrades client-input failures to 400 and defaults the rest to 500 -- so teach it the iceberg-go schema and spec sentinels instead of standing up a parallel classifier. The placeholder wrapper was a pure alias for newTableMetadata with nil arguments; call that directly. No behavior change beyond the 500 message, which now reads err.Error() like every other manager error rather than carrying its own prefix. |
||
|
Chris Lu
|
1ca628d3e9 |
iceberg: support multi-table transaction commit (#10066)
* iceberg: support multi-table transaction commit Add handleCommitTransaction for POST /v1/transactions/commit. Validation is atomic across all table-changes (resolve, load, evaluate every requirement before any write); metadata writes and pointer flips are best-effort with rollback, so this is not crash-atomic. * iceberg: route transactions/commit with and without prefix * iceberg: test transaction commit request decoding * iceberg: restore full prior table state on transaction rollback * iceberg: test transaction rollback restores full prior table state * iceberg: only clean up metadata for rolled-back tables |