mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-10 16:27:47 +02:00
c2cdefd06d89f4510dfe06508eee7ea5aa49d607
9
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Dmitriy PavlovandChris Lu
|
1d0b97f4c6 |
avro: field time.Time <> iceberg.date (#11091)
* iceberg: normalize foreign day partitions during manifest rewrite * test: cover manifest rewrite with foreign day partitions * iceberg: restore every foreign partition value, not just day transforms iceberg-go takes a partition field's logical type from the last branch of its Avro union, so a writer that spells an optional partition [<type>, null] rather than [null, <type>] leaves the value as whatever the Avro decoder produced. A day or date partition then arrives as a time.Time the manifest writer cannot encode, and a time partition is worse: time.Duration converts to int64 nanoseconds and silently records the wrong value. ReadManifest sits next to ReadManifestList, the other shim for what foreign writers put on the wire, and converts each partition value back to the Iceberg representation for its field type. Claude-Session: https://claude.ai/code/session_01FdQyRuWF9SuCnPn21iH9yR * iceberg: read manifests that carry partition values through the shim Compaction, delete rewrite and their detection passes read entries and write the same partition values back into new manifests, so they fail on a foreign day partition exactly as manifest rewrite does. Where filters see it too: literalMatchesActual falls through to fmt.Sprint, so a time.Time renders as a timestamp and never matches the day the user asked for. The two remaining readers, orphan collection and the admin preview, only look at file paths and stay on iceberg.ReadManifest. Claude-Session: https://claude.ai/code/session_01FdQyRuWF9SuCnPn21iH9yR * iceberg: convert partition values before the writer rebinds logical types Dimonyga checked the manifests of a live Doris table: every input spells the partition union null-first, with the date logical type present, so the union ordering is not what breaks the merge. The conversion is lazy. iceberg-go converts what the Avro decoder returned on the first Partition() call, using the logical types read from the manifest being parsed, and ManifestWriter.addEntry rebinds them to the manifest it is about to write before it makes that call. A day partition is where the two disagree -- iceberg-go's day transform reports an int32 result type, so the manifest it writes carries no date logical type at all -- and an entry nobody looked at in between converts against that and keeps its time.Time. That is why only rewrite_manifests failed: compaction and delete rewrite group entries by partitionKey(df.Partition()) first, which converts them, and a where filter does the same. Reading every entry's partition here converts them all while the manifest's own logical types are still in place. Claude-Session: https://claude.ai/code/session_01FdQyRuWF9SuCnPn21iH9yR --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
5f6dd4d3e5 |
iceberg maintenance: keep the snapshots that branches and tags pin (#10774)
* iceberg maintenance: keep the snapshots that branches and tags pin expireSnapshots only ever protected the current snapshot, so a snapshot held by a tag or a non-main branch was expired once it aged out of the retention window. iceberg-go's RemoveSnapshots drops any ref whose snapshot is gone without complaint, so the tag disappeared and the files behind it were deleted as unreferenced. Protect every ref target, and honour a branch's own min-snapshots-to-keep / max-snapshot-age-ms over the ancestors behind its head. Detection skips pinned snapshots for the same reason: proposing a job whose only outcome is a no-op keeps the worker busy forever. * iceberg maintenance: re-plan when a ref appears mid-commit, and stop proposing no-op expiry Three follow-ups from review of the ref-aware expiry: The commit guard only compared the table head, so a tag created between planning and commit could pin a snapshot the plan was about to expire. Re-check the refs against the metadata the commit actually reads. Detection now asks snapshotsToExpire what execution would remove instead of approximating with its own count-and-age rules. Expiry always requires a snapshot past the retention window, so a table over the quota whose snapshots are all young was being proposed for a job that could only no-op. The branch retention test could not tell "retained the whole lineage" from "honoured min-snapshots-to-keep", because the branch had exactly as many ancestors as the count. Give it one more, and cover max-snapshot-age-ms too. Both need snapshots genuinely older than a retention window, which iceberg-go will not accept at build time, so the fixture backdates the metadata after building it. * iceberg maintenance: fold the metadata test builders back into one buildTestMetadata, buildTestMetadataWithRefs, buildTestMetadataAged and buildTestMetadataNow were four names for one thing. Keep the original and give it the refs and age it needs. |
||
|
Chris Lu
|
a1d3fe236f |
iceberg: let table properties override the worker config (#10772)
* iceberg: carry snapshot retention in milliseconds Config stored retention as hours, so any sub-hour value would have to be truncated to 0 and then clamped back up to the 168 hour default. Keep the plugin config key in hours and convert once at parse time. * iceberg: let table properties override the worker config Every other Iceberg implementation lets a table's own properties win over engine defaults; the worker ignored them entirely. A writer honouring write.target-file-size-bytes and a compactor rewriting to the plugin config's size would rewrite each other's output forever. Resolved once per job rather than per operation, so compaction committing new metadata mid-job cannot change the settings underneath it. * iceberg: clamp the orphan cutoff so it cannot overflow collectOrphanCandidates converts the cutoff to a time.Duration. Past roughly 2.5 million hours that multiplication wraps negative, putting the cutoff in the future so every file walked looks like an orphan and gets deleted, including data a concurrent writer has not yet committed. Reachable today through orphan_older_than_hours. |
||
|
Chris Lu
|
4b0d09683a |
iceberg: read manifest lists that omit the Avro format version (#10475)
* s3tables: read Iceberg manifest lists that omit the Avro format version The Iceberg spec pins the Avro header metadata of manifest files but says nothing about manifest lists, so writers disagree. Java and PyIceberg record "format-version"; DuckDB writes no header metadata at all. iceberg-go reads a missing entry as v1, so every v2 manifest listed in a DuckDB-written list is rejected with manifest file's 'format-version' metadata indicates version 2, but entry from manifest list indicates version 1 and, because v1 has no "content" field, delete manifests silently decode as data manifests. ReadManifestList derives the version from the record schema the writer embedded - v2 added "content" and the sequence numbers, v3 added "first_row_id" - and splices it into the header before handing the bytes to iceberg-go. Lists that already carry the entry, and input that is not a parseable Avro container, go through untouched. * iceberg: parse DuckDB-written manifest lists in maintenance and data preview Every manifest list read - the four maintenance operations and the admin table data preview - went straight to iceberg-go, so tables written by DuckDB failed detection and all of compact, remove_orphans, rewrite_manifests and expire_snapshots before they touched anything. Route them through s3tables.ReadManifestList, which recovers the format version the writer left out of the Avro header. This also restores the manifest content type on those tables: with the list read as v1 every delete manifest looked like a data manifest, which hid deletes from the compaction guard and made the preview report a table with position deletes as having none. |
||
|
Chris Lu
|
1e1b2bb2f9 |
iceberg maintenance: record file counters on the snapshots it commits (#10420)
A maintenance snapshot carried only its own labels — merged-files, delete-groups and friends — and no summary counters, so every engine that reads a table's size out of the current snapshot summary reported nothing for it: PyIceberg's inspect.snapshots, Trino's $snapshots and Spark's DESCRIBE all read total-records, total-data-files and total-files-size verbatim, and a table lost them the moment compaction touched it. Accumulate the files each operation adds and removes, and render them the way the spec defines: the added-*/removed-* counters from the files themselves, then the totals carried over from the parent snapshot. Carry a total only when the parent recorded it. Iceberg treats a missing total as zero, which turns a compaction replacing two files with one into a negative total-data-files, or a table with millions of rows into total-records: 0. Leaving the field out lets a reader fall back to the manifests instead of believing a made-up number. Compaction also accounts for the delete files it consumes, so a run that folds every delete into the rewritten data reports them as removed. |
||
|
Chris Lu
|
b4b0346f95 |
iceberg maintenance: resolve table files from the recorded location (#10418)
The worker assumed every file of a table sits under its catalog path, so loadFileByIcebergPath stripped the scheme off a recorded location and joined the remainder onto /buckets/<bucket>/<ns>/<table>. A table the REST catalog placed elsewhere in the bucket — which is what a client gets whenever the catalog path is already occupied — then resolves to a doubled path: lookup /buckets/lake/source/t/lake/source/t-0cd81bca-.../metadata/snap-.avro so the very first manifest list read fails and the job fails again on every scan interval, indefinitely. Resolve absolute references (s3:// URIs and /buckets paths) from the bucket root and keep relative ones under the table's own directory; the bucket-relative form is now the canonical key everywhere references are compared. That directory comes from the metadata location the catalog stores, so reads, writes and deletes all land where the table's other files are instead of splitting it across two trees. References outside the table's bucket are rejected rather than silently misresolved. Rewritten position-delete files now name their data file by absolute URI, the way the table itself names it, instead of a path relative to the table. |
||
|
Chris Lu
|
07c9e0db85 |
iceberg maintenance: write absolute s3:// locations into table metadata (#10363)
The maintenance operations built manifest, manifest-list, data-file and
metadata-log paths with a bare path.Join("metadata", ...), so a single
maintenance run wrote scheme-less relative paths into the new snapshot,
the metadata-log and the table xattr. SeaweedFS reads those back fine
because normalizeIcebergPath accepts both forms, but strict readers
resolve every location through S3FileIO and fail with "Invalid S3 URI,
cannot determine scheme", leaving the whole table unreadable after any
maintenance commit.
Add absoluteIcebergPath, the inverse of normalizeIcebergPath, and apply
it at every site that authors locations: rewrite_manifests, compact,
rewrite_position_delete_files, and the shared commit path. The base is
derived from the bucket and table path since that is where the worker
physically writes, matching the locations the REST catalog generates.
|
||
|
dependabot[bot]andChris Lu
|
36f2ddcaea |
build(deps): bump github.com/apache/iceberg-go from 0.5.0 to 0.6.0 (#10038)
* build(deps): bump github.com/apache/iceberg-go from 0.5.0 to 0.6.0 Bumps [github.com/apache/iceberg-go](https://github.com/apache/iceberg-go) from 0.5.0 to 0.6.0. - [Release notes](https://github.com/apache/iceberg-go/releases) - [Commits](https://github.com/apache/iceberg-go/compare/v0.5.0...v0.6.0) --- updated-dependencies: - dependency-name: github.com/apache/iceberg-go dependency-version: 0.6.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * iceberg: adapt worker to iceberg-go 0.6.0 API Fields() now yields iter.Seq2 (index, value); SortField.SourceID and PartitionField.SourceID are methods backed by SourceIDs; RemoveSnapshots takes a postCommit flag (false here, file cleanup runs through the filer). --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
1f6f473995 |
refactor(worker): co-locate plugin handlers with their task packages (#9301)
* refactor(worker): co-locate plugin handlers with their task packages
Move every per-task plugin handler from weed/plugin/worker/ into the
matching weed/worker/tasks/<name>/ package, so each task owns its
detection, scheduling, execution, and plugin handler in one place.
Step 0 (within pluginworker, no behavior change): extract shared helpers
that previously lived inside individual handler files into dedicated
files and export the ones now consumed across packages.
- activity.go: BuildExecutorActivity, BuildDetectorActivity
- config.go: ReadStringConfig/Double/Int64/Bytes/StringList, MapTaskPriority
- interval.go: ShouldSkipDetectionByInterval
- volume_state.go: VolumeState + consts, FilterMetricsByVolumeState/Location
- collection_filter.go: CollectionFilterMode + consts
- volume_metrics.go: export CollectVolumeMetricsFromMasters,
MasterAddressCandidates, FetchVolumeList
- testing_senders_test.go: shared test stubs
Phase 1: move the per-task plugin handlers (and the iceberg subpackage)
into their task packages.
weed/plugin/worker/vacuum_handler.go -> weed/worker/tasks/vacuum/plugin_handler.go
weed/plugin/worker/ec_balance_handler.go -> weed/worker/tasks/ec_balance/plugin_handler.go
weed/plugin/worker/erasure_coding_handler.go -> weed/worker/tasks/erasure_coding/plugin_handler.go
weed/plugin/worker/volume_balance_handler.go -> weed/worker/tasks/balance/plugin_handler.go
weed/plugin/worker/iceberg/ -> weed/worker/tasks/iceberg/
weed/plugin/worker/handlers/handlers.go now blank-imports all five
task subpackages so their init() registrations fire.
weed/command/mini.go and the worker tests construct the handler with
vacuum.DefaultMaxExecutionConcurrency (the constant moved with the
vacuum handler).
admin_script remains in weed/plugin/worker/ because there is no
underlying weed/worker/tasks/admin_script/ package to merge with.
* refactor(worker): update test/plugin_workers imports for moved handlers
Three handler constructors moved out of pluginworker into their task
packages — update the integration test files in test/plugin_workers/
to import from the new locations:
pluginworker.NewVacuumHandler -> vacuum.NewVacuumHandler
pluginworker.NewVolumeBalanceHandler -> balance.NewVolumeBalanceHandler
pluginworker.NewErasureCodingHandler -> erasure_coding.NewErasureCodingHandler
The pluginworker import is kept where the file still uses
pluginworker.WorkerOptions / pluginworker.JobHandler.
* refactor(worker): update test/s3tables iceberg import path
The iceberg subpackage moved from weed/plugin/worker/iceberg/ to
weed/worker/tasks/iceberg/. test/s3tables/maintenance/maintenance_integration_test.go
still imported the old path, breaking S3 Tables / RisingWave / Trino /
Spark / Iceberg-catalog / STS integration test builds.
Mirrors the OSS-side fix needed by every job in the run that
transitively imports test/s3tables/maintenance.
* chore: gofmt PR-touched files
The S3 Tables Format Check job runs `gofmt -l` over weed/s3api/s3tables
and test/s3tables, then fails if anything is unformatted. Files this
PR moved or modified had import-grouping and trailing-spacing issues
introduced by perl-based renames; reformat them with gofmt -w.
Touched files:
test/plugin_workers/erasure_coding/{detection,execution}_test.go
test/s3tables/maintenance/maintenance_integration_test.go
weed/plugin/worker/handlers/handlers.go
weed/worker/tasks/{balance,ec_balance,erasure_coding,vacuum}/plugin_handler*.go
* refactor(worker): bounds-checked int conversions for plugin config values
CodeQL flagged 18 go/incorrect-integer-conversion warnings on the moved
plugin handler files: results of pluginworker.ReadInt64Config (which
ultimately calls strconv.ParseInt with bit size 64) were being narrowed
to int32/uint32/int without an upper-bound check, so a malicious or
malformed admin/worker config value could overflow the target type.
Add three helpers in weed/plugin/worker/config.go that wrap
ReadInt64Config and clamp out-of-range values back to the caller's
fallback:
ReadInt32Config (math.MinInt32 .. math.MaxInt32)
ReadUint32Config (0 .. math.MaxUint32)
ReadIntConfig (math.MinInt32 .. math.MaxInt32, platform-portable)
Update each flagged call site in the four moved task packages to use
the bounds-checked helper. For protobuf uint32 fields (volume IDs)
the variable type also becomes uint32, removing the trailing
uint32(volumeID) casts and changing the "missing volume_id" check
from `<= 0` to `== 0`.
Touched files:
weed/plugin/worker/config.go
weed/worker/tasks/balance/plugin_handler.go
weed/worker/tasks/erasure_coding/plugin_handler.go
weed/worker/tasks/vacuum/plugin_handler.go
* refactor(worker): use ReadIntConfig for clamped derive-worker-config helpers
CodeQL still flagged three call sites where ReadInt64Config was being
narrowed to int after a value-range clamp (max_concurrent_moves <= 50,
batch_size <= 100, min_server_count >= 2). The clamp is correct but
CodeQL's flow analysis didn't recognize the bound, so it flagged them
as unbounded narrowing.
Switch to ReadIntConfig (already int32-bounded by the helper) for
those three sites, drop the now-redundant int64 intermediate variables.
Also drops the now-unused `> math.MaxInt32` clamp in
ec_balance.deriveECBalanceWorkerConfig (the helper covers it).
|