Files
seaweedfs/weed/s3api/iceberg/requirements_test.go
T
dependabot[bot]andChris Lu 2d4b730a2f build(deps): bump github.com/twmb/avro from 1.7.2 to 1.8.0 (#11210)
* build(deps): bump github.com/twmb/avro from 1.7.2 to 1.8.0

Bumps [github.com/twmb/avro](https://github.com/twmb/avro) from 1.7.2 to 1.8.0.
- [Commits](https://github.com/twmb/avro/compare/v1.7.2...v1.8.0)

---
updated-dependencies:
- dependency-name: github.com/twmb/avro
  dependency-version: 1.8.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* iceberg: adapt to twmb/avro v1.8.0 and iceberg-go defensive copies

avro v1.8.0 changes Schema.Root() to return *SchemaNode, which breaks
iceberg-go v0.6.0's internal avro_schemas.go. The fix (apache/iceberg-go#1843)
is only on iceberg-go's main branch, unreleased, so bump iceberg-go to
that commit (c210509) alongside the avro bump.

That iceberg-go revision also changes two behaviors seaweedfs worked
around:

- It now infers a manifest list's format version from the embedded
  writer schema, so a list missing the "format-version" header entry
  (DuckDB's shape is read as v2, not v1. ReadManifestList's header
  patching is now a redundant safety net; tests updated to expect v2.

- It returns defensive copies from DataFile.Partition(), so the
  ReadManifest shim's in-place partition normalization was silently
  discarded. Rebuild the entry through NewDataFileBuilder when any
  partition value is normalized, copying every other DataFile field so
  manifest round-trips are preserved.

- It converts day-transform partitions to iceberg.Date on read
  (applyDayTransformDates), so the day-partition cases the shim and
  tests guarded now convert without help; tests updated to expect
  iceberg.Date from the raw read.
EOF
)

* iceberg: accept assert-ref-snapshot-id without snapshot-id

iceberg-go's new nullableInt64 parser rejects an assert-ref-snapshot-id
requirement whose "snapshot-id" field is absent from the JSON, even
though the Iceberg REST spec makes it optional (null means the ref must
not already exist). v0.6.0 used a plain *int64, so absent was nil and
accepted. ClickHouse sends the requirement without snapshot-id when
asserting a branch does not yet exist, so its writes fail with
"missing required field \"snapshot-id\"".

normalizeRequirements splices an explicit null into any
assert-ref-snapshot-id requirement missing the field before handing
the JSON to iceberg-go's parser, restoring the v0.6.0 behavior across
both iceberg-go versions.

* iceberg: fix v1 block_size_in_bytes default in rebuilt manifest entries

rebuildManifestEntry set block_size_in_bytes to 0, but the v1 manifest
schema requires the default of 64 MiB ("Always write default in v1").
The original value is not exposed on the DataFile interface, so use the
spec default. Also clarify the fallback comment to note that empty
(zero-record / zero-byte) files also trigger it, not just a nil spec.

Add a round-trip test that writes a rebuilt entry as v1 and verifies
block_size_in_bytes is 64 MiB via Avro decoding.

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-09-07 15:58:24 -07:00

95 lines
2.8 KiB
Go

package iceberg
import (
"encoding/json"
"testing"
"github.com/apache/iceberg-go/table"
)
func TestNormalizeRequirementsAddsNullSnapshotID(t *testing.T) {
// ClickHouse sends assert-ref-snapshot-id without snapshot-id when
// asserting a branch does not yet exist. iceberg-go's parser rejects
// an absent field; normalizeRequirements splices in an explicit null.
raw := json.RawMessage(`[{"type":"assert-ref-snapshot-id","ref":"main"}]`)
// Without normalization, iceberg-go rejects the requirement.
var reqs table.Requirements
if err := json.Unmarshal(raw, &reqs); err == nil {
t.Fatal("iceberg-go accepted an assert-ref-snapshot-id without snapshot-id; expected an error")
}
normalized, err := normalizeRequirements(raw)
if err != nil {
t.Fatalf("normalizeRequirements: %v", err)
}
if err := json.Unmarshal(normalized, &reqs); err != nil {
t.Fatalf("unmarshal normalized requirements: %v", err)
}
if len(reqs) != 1 {
t.Fatalf("got %d requirements, want 1", len(reqs))
}
if reqs[0].GetType() != "assert-ref-snapshot-id" {
t.Fatalf("requirement type = %s, want assert-ref-snapshot-id", reqs[0].GetType())
}
}
func TestNormalizeRequirementsPreservesPresentSnapshotID(t *testing.T) {
raw := json.RawMessage(`[{"type":"assert-ref-snapshot-id","ref":"main","snapshot-id":42}]`)
normalized, err := normalizeRequirements(raw)
if err != nil {
t.Fatalf("normalizeRequirements: %v", err)
}
var reqs table.Requirements
if err := json.Unmarshal(normalized, &reqs); err != nil {
t.Fatalf("unmarshal: %v", err)
}
if len(reqs) != 1 {
t.Fatalf("got %d requirements, want 1", len(reqs))
}
}
func TestNormalizeRequirementsPreservesExplicitNullSnapshotID(t *testing.T) {
raw := json.RawMessage(`[{"type":"assert-ref-snapshot-id","ref":"main","snapshot-id":null}]`)
normalized, err := normalizeRequirements(raw)
if err != nil {
t.Fatalf("normalizeRequirements: %v", err)
}
var reqs table.Requirements
if err := json.Unmarshal(normalized, &reqs); err != nil {
t.Fatalf("unmarshal: %v", err)
}
if len(reqs) != 1 {
t.Fatalf("got %d requirements, want 1", len(reqs))
}
}
func TestNormalizeRequirementsLeavesOtherTypesUnchanged(t *testing.T) {
raw := json.RawMessage(`[{"type":"assert-create"},{"type":"assert-table-uuid","uuid":"00000000-0000-0000-0000-000000000000"}]`)
normalized, err := normalizeRequirements(raw)
if err != nil {
t.Fatalf("normalizeRequirements: %v", err)
}
var reqs table.Requirements
if err := json.Unmarshal(normalized, &reqs); err != nil {
t.Fatalf("unmarshal: %v", err)
}
if len(reqs) != 2 {
t.Fatalf("got %d requirements, want 2", len(reqs))
}
}
func TestNormalizeRequirementsEmptyInput(t *testing.T) {
normalized, err := normalizeRequirements(nil)
if err != nil {
t.Fatalf("normalizeRequirements(nil): %v", err)
}
if normalized != nil {
t.Fatalf("expected nil output for nil input, got %s", normalized)
}
}