Files
seaweedfs/weed/s3api/iceberg/requirements.go
T
dependabot[bot]andChris Lu 2d4b730a2f build(deps): bump github.com/twmb/avro from 1.7.2 to 1.8.0 (#11210)
* build(deps): bump github.com/twmb/avro from 1.7.2 to 1.8.0

Bumps [github.com/twmb/avro](https://github.com/twmb/avro) from 1.7.2 to 1.8.0.
- [Commits](https://github.com/twmb/avro/compare/v1.7.2...v1.8.0)

---
updated-dependencies:
- dependency-name: github.com/twmb/avro
  dependency-version: 1.8.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* iceberg: adapt to twmb/avro v1.8.0 and iceberg-go defensive copies

avro v1.8.0 changes Schema.Root() to return *SchemaNode, which breaks
iceberg-go v0.6.0's internal avro_schemas.go. The fix (apache/iceberg-go#1843)
is only on iceberg-go's main branch, unreleased, so bump iceberg-go to
that commit (c210509) alongside the avro bump.

That iceberg-go revision also changes two behaviors seaweedfs worked
around:

- It now infers a manifest list's format version from the embedded
  writer schema, so a list missing the "format-version" header entry
  (DuckDB's shape is read as v2, not v1. ReadManifestList's header
  patching is now a redundant safety net; tests updated to expect v2.

- It returns defensive copies from DataFile.Partition(), so the
  ReadManifest shim's in-place partition normalization was silently
  discarded. Rebuild the entry through NewDataFileBuilder when any
  partition value is normalized, copying every other DataFile field so
  manifest round-trips are preserved.

- It converts day-transform partitions to iceberg.Date on read
  (applyDayTransformDates), so the day-partition cases the shim and
  tests guarded now convert without help; tests updated to expect
  iceberg.Date from the raw read.
EOF
)

* iceberg: accept assert-ref-snapshot-id without snapshot-id

iceberg-go's new nullableInt64 parser rejects an assert-ref-snapshot-id
requirement whose "snapshot-id" field is absent from the JSON, even
though the Iceberg REST spec makes it optional (null means the ref must
not already exist). v0.6.0 used a plain *int64, so absent was nil and
accepted. ClickHouse sends the requirement without snapshot-id when
asserting a branch does not yet exist, so its writes fail with
"missing required field \"snapshot-id\"".

normalizeRequirements splices an explicit null into any
assert-ref-snapshot-id requirement missing the field before handing
the JSON to iceberg-go's parser, restoring the v0.6.0 behavior across
both iceberg-go versions.

* iceberg: fix v1 block_size_in_bytes default in rebuilt manifest entries

rebuildManifestEntry set block_size_in_bytes to 0, but the v1 manifest
schema requires the default of 64 MiB ("Always write default in v1").
The original value is not exposed on the DataFile interface, so use the
spec default. Also clarify the fallback comment to note that empty
(zero-record / zero-byte) files also trigger it, not just a nil spec.

Add a round-trip test that writes a rebuilt entry as v1 and verifies
block_size_in_bytes is 64 MiB via Avro decoding.

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-09-07 15:58:24 -07:00

56 lines
1.6 KiB
Go

package iceberg
import (
"encoding/json"
)
// normalizeRequirements repairs requirement JSON that iceberg-go's parser
// rejects but the Iceberg REST spec permits.
//
// iceberg-go's assert-ref-snapshot-id parser uses a nullableInt64 that tracks
// whether the "snapshot-id" field was present in the JSON. An absent field
// (set == false) is rejected as "missing required field", even though the spec
// makes snapshot-id optional — null means the ref must not already exist.
// iceberg-go v0.6.0 used a plain *int64, so absent was nil and accepted.
//
// Clients like ClickHouse send assert-ref-snapshot-id without snapshot-id when
// asserting a branch does not yet exist. Splicing in an explicit null before
// unmarshaling restores the v0.6.0 behavior across both iceberg-go versions.
func normalizeRequirements(raw json.RawMessage) (json.RawMessage, error) {
if len(raw) == 0 {
return raw, nil
}
var reqs []json.RawMessage
if err := json.Unmarshal(raw, &reqs); err != nil {
return raw, err
}
changed := false
for i, req := range reqs {
var obj map[string]json.RawMessage
if err := json.Unmarshal(req, &obj); err != nil {
continue // let the downstream parser report the error
}
var typ string
if rawType, ok := obj["type"]; ok {
_ = json.Unmarshal(rawType, &typ)
}
if typ != "assert-ref-snapshot-id" {
continue
}
if _, hasSnapshotID := obj["snapshot-id"]; hasSnapshotID {
continue
}
obj["snapshot-id"] = json.RawMessage("null")
fixed, err := json.Marshal(obj)
if err != nil {
continue
}
reqs[i] = fixed
changed = true
}
if !changed {
return raw, nil
}
return json.Marshal(reqs)
}