mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-04 11:07:35 +01:00
5dfda3645ead2003ff85cd2f236bb296af39fc22
13
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
0d00e92178 |
iceberg maintenance: match percent-decoded keys when checking file references (#11978)
* iceberg maintenance: match percent-decoded keys when checking file references Writers may percent-escape partition directory names in the file paths they record in manifests (e.g. "part%3D1"), while the S3 layer decodes the key before storing the object. remove_orphans compared the escaped reference to the decoded filer path, counted the live file as an orphan, and deleted it once it passed the age window — readers then got NoSuchKey on the next scan. Match both spellings of every referenced path in remove_orphans, apply the same coverage to the kept-file set in expire_snapshots, and delete expired files at their decoded key so the object the S3 layer stored is the one removed. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: retry filer lookups at the percent-decoded key A manifest reference carrying an escaped segment resolves to a filer name that does not exist, because the S3 layer decoded the key on write. When a lookup misses and the reference contains escapes, retry once at the decoded spelling so maintenance can still read the file. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: resolve referenced files verbatim-first, decoded fallback filer_pb.LookupEntry reports misses as ErrNotFound rather than a gRPC NotFound, so the escaped-reference retry never fired against a real filer. Both maintenance reads and expired-snapshot deletes now resolve through one verbatim-first/decoded-fallback lookup, and expiry refuses to delete a resolved path outside the table's data directory. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: test expiry of a file stored at a literal percent key * iceberg maintenance: resolve referenced files in one filer lookup lookupFileByIcebergPath re-fetched the entry resolveFilerPath had just looked up, doubling the RPC per referenced file during compaction and expiration. The resolver now returns the entry alongside the path so a hit costs one lookup and a decoded fallback two. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: keep every decoded spelling of a referenced path The kept-reference sets recorded a path plus a single decode of it, but lookupResolvedEntry decodes recursively until the spelling is stable. A doubly-escaped reference could resolve to a spelling absent from the set and be deleted while still referenced. addFilerPathSpellings now walks the same decode chain the resolver does. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: keep every decoded spelling of a referenced path * iceberg maintenance: test expiry keeps a doubly-escaped kept reference * iceberg maintenance: clean decoded spellings in the referenced-path set * iceberg maintenance: test expiry keeps a kept reference resolved via decoded parent segment * iceberg maintenance: decode only the reference in the spelling set Decoding the composed filer path also decoded the table directory, so a table at a literal percent-escaped path produced a kept set missing the spelling the resolver returns, and expiry could delete a live file. * test: gofmt the s3tables cluster helpers Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
610346ae8c |
iceberg: keep version-hint.text out of remove_orphans (#11867) (#11871)
Generated with [Devin](https://devin.ai) Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
50a9312e1e |
iceberg maintenance: v3 row lineage so v3 tables compact and rewrite manifests (#11653)
* iceberg maintenance: commit v3 row lineage on the three commit paths
iceberg-go 0.7.0 refuses every snapshot on a format-version 3 table
that lacks first-row-id and added-rows ("invalid row lineage"), so no
v3 table was ever compacted or had its manifests rewritten; worse,
passing 0 to the manifest list would have handed new files row IDs
existing rows already hold.
Each commit now plans first-row-id from the table's next-row-id, lets
the v3 manifest-list writer assign manifest first_row_id from there,
and reports the assigned range as the snapshot's added-rows so
next-row-id advances by exactly that. The commit guard also treats a
moved next-row-id or a changed format version as a stale plan, like a
moved head: a commit to another branch advances next-row-id too, and
manifests written for the old version must not be committed.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: keep row identity through v3 compaction
A rewrite that moves rows into a new data file must not renumber them.
On v3 tables the merge now appends _row_id and
_last_updated_sequence_number (spec field ids 2147483540/2147483539)
to the output and fills each row with the value its input
materializes, else the inherited one: the input file's first_row_id
plus the row's position before any delete, and the manifest entry's
data sequence number. Inputs are matched with their lineage columns
set aside, so a file an earlier compaction wrote merges with one an
engine wrote; a duplicate lineage column or a reserved name under
another field id fails the bin instead of writing a doubled column.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: skip rewrite_position_delete_files on v3 tables
The spec forbids adding position-delete files to a format-version 3
table (v3 writes deletion vectors), and this rewrite adds new ones.
Detection no longer proposes the operation and the operation returns a
skip result; compaction still applies and removes existing position
deletes.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: keep Puffin deletion vectors out of position-delete reads
A Puffin file in a delete manifest is a deletion vector, not a
position-delete file: reading it as Parquet fails, and merging the data
file it covers without applying the vector resurrects deleted rows.
Skip Puffin entries in the position-delete collection and the delete
rewrite group scan, and exclude the files a vector covers from
compaction candidates - in compactDataFiles and in eligibility
detection alike. A vector whose referenced_data_file is missing or
unresolvable fails the run rather than silently compacting whatever it
points at.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: name a merged file's own first row ID
Rewritten rows keep their IDs, so a merged data file's first_row_id is
the _row_id of its first written row - not the next free range the
manifest list would otherwise assign to the entry at commit. Read the
first materialized value back off the merged output and stamp it on the
manifest entry.
Add an end-to-end test that populates a v3 table whose entries carry no
first_row_id, so the compact path exercises manifest first_row_id
inheritance through the stored manifest list, then commits and reads
the result back through the real manifest readers.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: locate the _row_id column by index, not row position
Repeated columns shift parquet.Row positions, so leaf.ColumnIndex is not
the value's slot — a first row containing a list or map saved an unrelated
value as the merged file's first_row_id.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* iceberg maintenance: read the returned row before treating EOF as empty
parquet-go can return a row alongside io.EOF; a one-row merged file then
lost its first_row_id and the manifest list would allocate a fresh range.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: avoid nil dereference in first-row-id failure message
---------
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
||
|
|
4111aa4dd8 |
iceberg maintenance: keep statistics files the table metadata references (#11651)
Orphan cleanup built its referenced set from snapshot manifests plus the current and previous metadata files, then walked all of metadata/ and data/. Table statistics (Puffin) and partition statistics files are listed in the table metadata's statistics and partition-statistics lists, not in any snapshot, so once they passed the safety window remove_orphans deleted them while the metadata still pointed at them. Mark the files in both lists referenced as well. Generated with [Devin](https://devin.ai) Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
1d0b97f4c6 |
avro: field time.Time <> iceberg.date (#11091)
* iceberg: normalize foreign day partitions during manifest rewrite * test: cover manifest rewrite with foreign day partitions * iceberg: restore every foreign partition value, not just day transforms iceberg-go takes a partition field's logical type from the last branch of its Avro union, so a writer that spells an optional partition [<type>, null] rather than [null, <type>] leaves the value as whatever the Avro decoder produced. A day or date partition then arrives as a time.Time the manifest writer cannot encode, and a time partition is worse: time.Duration converts to int64 nanoseconds and silently records the wrong value. ReadManifest sits next to ReadManifestList, the other shim for what foreign writers put on the wire, and converts each partition value back to the Iceberg representation for its field type. Claude-Session: https://claude.ai/code/session_01FdQyRuWF9SuCnPn21iH9yR * iceberg: read manifests that carry partition values through the shim Compaction, delete rewrite and their detection passes read entries and write the same partition values back into new manifests, so they fail on a foreign day partition exactly as manifest rewrite does. Where filters see it too: literalMatchesActual falls through to fmt.Sprint, so a time.Time renders as a timestamp and never matches the day the user asked for. The two remaining readers, orphan collection and the admin preview, only look at file paths and stay on iceberg.ReadManifest. Claude-Session: https://claude.ai/code/session_01FdQyRuWF9SuCnPn21iH9yR * iceberg: convert partition values before the writer rebinds logical types Dimonyga checked the manifests of a live Doris table: every input spells the partition union null-first, with the date logical type present, so the union ordering is not what breaks the merge. The conversion is lazy. iceberg-go converts what the Avro decoder returned on the first Partition() call, using the logical types read from the manifest being parsed, and ManifestWriter.addEntry rebinds them to the manifest it is about to write before it makes that call. A day partition is where the two disagree -- iceberg-go's day transform reports an int32 result type, so the manifest it writes carries no date logical type at all -- and an entry nobody looked at in between converts against that and keeps its time.Time. That is why only rewrite_manifests failed: compaction and delete rewrite group entries by partitionKey(df.Partition()) first, which converts them, and a where filter does the same. Reading every entry's partition here converts them all while the manifest's own logical types are still in place. Claude-Session: https://claude.ai/code/session_01FdQyRuWF9SuCnPn21iH9yR --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
|
5f6dd4d3e5 |
iceberg maintenance: keep the snapshots that branches and tags pin (#10774)
* iceberg maintenance: keep the snapshots that branches and tags pin expireSnapshots only ever protected the current snapshot, so a snapshot held by a tag or a non-main branch was expired once it aged out of the retention window. iceberg-go's RemoveSnapshots drops any ref whose snapshot is gone without complaint, so the tag disappeared and the files behind it were deleted as unreferenced. Protect every ref target, and honour a branch's own min-snapshots-to-keep / max-snapshot-age-ms over the ancestors behind its head. Detection skips pinned snapshots for the same reason: proposing a job whose only outcome is a no-op keeps the worker busy forever. * iceberg maintenance: re-plan when a ref appears mid-commit, and stop proposing no-op expiry Three follow-ups from review of the ref-aware expiry: The commit guard only compared the table head, so a tag created between planning and commit could pin a snapshot the plan was about to expire. Re-check the refs against the metadata the commit actually reads. Detection now asks snapshotsToExpire what execution would remove instead of approximating with its own count-and-age rules. Expiry always requires a snapshot past the retention window, so a table over the quota whose snapshots are all young was being proposed for a job that could only no-op. The branch retention test could not tell "retained the whole lineage" from "honoured min-snapshots-to-keep", because the branch had exactly as many ancestors as the count. Give it one more, and cover max-snapshot-age-ms too. Both need snapshots genuinely older than a retention window, which iceberg-go will not accept at build time, so the fixture backdates the metadata after building it. * iceberg maintenance: fold the metadata test builders back into one buildTestMetadata, buildTestMetadataWithRefs, buildTestMetadataAged and buildTestMetadataNow were four names for one thing. Keep the original and give it the refs and age it needs. |
||
|
|
a1d3fe236f |
iceberg: let table properties override the worker config (#10772)
* iceberg: carry snapshot retention in milliseconds Config stored retention as hours, so any sub-hour value would have to be truncated to 0 and then clamped back up to the 168 hour default. Keep the plugin config key in hours and convert once at parse time. * iceberg: let table properties override the worker config Every other Iceberg implementation lets a table's own properties win over engine defaults; the worker ignored them entirely. A writer honouring write.target-file-size-bytes and a compactor rewriting to the plugin config's size would rewrite each other's output forever. Resolved once per job rather than per operation, so compaction committing new metadata mid-job cannot change the settings underneath it. * iceberg: clamp the orphan cutoff so it cannot overflow collectOrphanCandidates converts the cutoff to a time.Duration. Past roughly 2.5 million hours that multiplication wraps negative, putting the cutoff in the future so every file walked looks like an orphan and gets deleted, including data a concurrent writer has not yet committed. Reachable today through orphan_older_than_hours. |
||
|
|
4b0d09683a |
iceberg: read manifest lists that omit the Avro format version (#10475)
* s3tables: read Iceberg manifest lists that omit the Avro format version The Iceberg spec pins the Avro header metadata of manifest files but says nothing about manifest lists, so writers disagree. Java and PyIceberg record "format-version"; DuckDB writes no header metadata at all. iceberg-go reads a missing entry as v1, so every v2 manifest listed in a DuckDB-written list is rejected with manifest file's 'format-version' metadata indicates version 2, but entry from manifest list indicates version 1 and, because v1 has no "content" field, delete manifests silently decode as data manifests. ReadManifestList derives the version from the record schema the writer embedded - v2 added "content" and the sequence numbers, v3 added "first_row_id" - and splices it into the header before handing the bytes to iceberg-go. Lists that already carry the entry, and input that is not a parseable Avro container, go through untouched. * iceberg: parse DuckDB-written manifest lists in maintenance and data preview Every manifest list read - the four maintenance operations and the admin table data preview - went straight to iceberg-go, so tables written by DuckDB failed detection and all of compact, remove_orphans, rewrite_manifests and expire_snapshots before they touched anything. Route them through s3tables.ReadManifestList, which recovers the format version the writer left out of the Avro header. This also restores the manifest content type on those tables: with the list read as v1 every delete manifest looked like a data manifest, which hid deletes from the compaction guard and made the preview report a table with position deletes as having none. |
||
|
|
1e1b2bb2f9 |
iceberg maintenance: record file counters on the snapshots it commits (#10420)
A maintenance snapshot carried only its own labels — merged-files, delete-groups and friends — and no summary counters, so every engine that reads a table's size out of the current snapshot summary reported nothing for it: PyIceberg's inspect.snapshots, Trino's $snapshots and Spark's DESCRIBE all read total-records, total-data-files and total-files-size verbatim, and a table lost them the moment compaction touched it. Accumulate the files each operation adds and removes, and render them the way the spec defines: the added-*/removed-* counters from the files themselves, then the totals carried over from the parent snapshot. Carry a total only when the parent recorded it. Iceberg treats a missing total as zero, which turns a compaction replacing two files with one into a negative total-data-files, or a table with millions of rows into total-records: 0. Leaving the field out lets a reader fall back to the manifests instead of believing a made-up number. Compaction also accounts for the delete files it consumes, so a run that folds every delete into the rewritten data reports them as removed. |
||
|
|
b4b0346f95 |
iceberg maintenance: resolve table files from the recorded location (#10418)
The worker assumed every file of a table sits under its catalog path, so loadFileByIcebergPath stripped the scheme off a recorded location and joined the remainder onto /buckets/<bucket>/<ns>/<table>. A table the REST catalog placed elsewhere in the bucket — which is what a client gets whenever the catalog path is already occupied — then resolves to a doubled path: lookup /buckets/lake/source/t/lake/source/t-0cd81bca-.../metadata/snap-.avro so the very first manifest list read fails and the job fails again on every scan interval, indefinitely. Resolve absolute references (s3:// URIs and /buckets paths) from the bucket root and keep relative ones under the table's own directory; the bucket-relative form is now the canonical key everywhere references are compared. That directory comes from the metadata location the catalog stores, so reads, writes and deletes all land where the table's other files are instead of splitting it across two trees. References outside the table's bucket are rejected rather than silently misresolved. Rewritten position-delete files now name their data file by absolute URI, the way the table itself names it, instead of a path relative to the table. |
||
|
|
07c9e0db85 |
iceberg maintenance: write absolute s3:// locations into table metadata (#10363)
The maintenance operations built manifest, manifest-list, data-file and
metadata-log paths with a bare path.Join("metadata", ...), so a single
maintenance run wrote scheme-less relative paths into the new snapshot,
the metadata-log and the table xattr. SeaweedFS reads those back fine
because normalizeIcebergPath accepts both forms, but strict readers
resolve every location through S3FileIO and fail with "Invalid S3 URI,
cannot determine scheme", leaving the whole table unreadable after any
maintenance commit.
Add absoluteIcebergPath, the inverse of normalizeIcebergPath, and apply
it at every site that authors locations: rewrite_manifests, compact,
rewrite_position_delete_files, and the shared commit path. The base is
derived from the bucket and table path since that is where the worker
physically writes, matching the locations the REST catalog generates.
|
||
|
|
36f2ddcaea |
build(deps): bump github.com/apache/iceberg-go from 0.5.0 to 0.6.0 (#10038)
* build(deps): bump github.com/apache/iceberg-go from 0.5.0 to 0.6.0 Bumps [github.com/apache/iceberg-go](https://github.com/apache/iceberg-go) from 0.5.0 to 0.6.0. - [Release notes](https://github.com/apache/iceberg-go/releases) - [Commits](https://github.com/apache/iceberg-go/compare/v0.5.0...v0.6.0) --- updated-dependencies: - dependency-name: github.com/apache/iceberg-go dependency-version: 0.6.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * iceberg: adapt worker to iceberg-go 0.6.0 API Fields() now yields iter.Seq2 (index, value); SortField.SourceID and PartitionField.SourceID are methods backed by SourceIDs; RemoveSnapshots takes a postCommit flag (false here, file cleanup runs through the filer). --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
|
1f6f473995 |
refactor(worker): co-locate plugin handlers with their task packages (#9301)
* refactor(worker): co-locate plugin handlers with their task packages
Move every per-task plugin handler from weed/plugin/worker/ into the
matching weed/worker/tasks/<name>/ package, so each task owns its
detection, scheduling, execution, and plugin handler in one place.
Step 0 (within pluginworker, no behavior change): extract shared helpers
that previously lived inside individual handler files into dedicated
files and export the ones now consumed across packages.
- activity.go: BuildExecutorActivity, BuildDetectorActivity
- config.go: ReadStringConfig/Double/Int64/Bytes/StringList, MapTaskPriority
- interval.go: ShouldSkipDetectionByInterval
- volume_state.go: VolumeState + consts, FilterMetricsByVolumeState/Location
- collection_filter.go: CollectionFilterMode + consts
- volume_metrics.go: export CollectVolumeMetricsFromMasters,
MasterAddressCandidates, FetchVolumeList
- testing_senders_test.go: shared test stubs
Phase 1: move the per-task plugin handlers (and the iceberg subpackage)
into their task packages.
weed/plugin/worker/vacuum_handler.go -> weed/worker/tasks/vacuum/plugin_handler.go
weed/plugin/worker/ec_balance_handler.go -> weed/worker/tasks/ec_balance/plugin_handler.go
weed/plugin/worker/erasure_coding_handler.go -> weed/worker/tasks/erasure_coding/plugin_handler.go
weed/plugin/worker/volume_balance_handler.go -> weed/worker/tasks/balance/plugin_handler.go
weed/plugin/worker/iceberg/ -> weed/worker/tasks/iceberg/
weed/plugin/worker/handlers/handlers.go now blank-imports all five
task subpackages so their init() registrations fire.
weed/command/mini.go and the worker tests construct the handler with
vacuum.DefaultMaxExecutionConcurrency (the constant moved with the
vacuum handler).
admin_script remains in weed/plugin/worker/ because there is no
underlying weed/worker/tasks/admin_script/ package to merge with.
* refactor(worker): update test/plugin_workers imports for moved handlers
Three handler constructors moved out of pluginworker into their task
packages — update the integration test files in test/plugin_workers/
to import from the new locations:
pluginworker.NewVacuumHandler -> vacuum.NewVacuumHandler
pluginworker.NewVolumeBalanceHandler -> balance.NewVolumeBalanceHandler
pluginworker.NewErasureCodingHandler -> erasure_coding.NewErasureCodingHandler
The pluginworker import is kept where the file still uses
pluginworker.WorkerOptions / pluginworker.JobHandler.
* refactor(worker): update test/s3tables iceberg import path
The iceberg subpackage moved from weed/plugin/worker/iceberg/ to
weed/worker/tasks/iceberg/. test/s3tables/maintenance/maintenance_integration_test.go
still imported the old path, breaking S3 Tables / RisingWave / Trino /
Spark / Iceberg-catalog / STS integration test builds.
Mirrors the OSS-side fix needed by every job in the run that
transitively imports test/s3tables/maintenance.
* chore: gofmt PR-touched files
The S3 Tables Format Check job runs `gofmt -l` over weed/s3api/s3tables
and test/s3tables, then fails if anything is unformatted. Files this
PR moved or modified had import-grouping and trailing-spacing issues
introduced by perl-based renames; reformat them with gofmt -w.
Touched files:
test/plugin_workers/erasure_coding/{detection,execution}_test.go
test/s3tables/maintenance/maintenance_integration_test.go
weed/plugin/worker/handlers/handlers.go
weed/worker/tasks/{balance,ec_balance,erasure_coding,vacuum}/plugin_handler*.go
* refactor(worker): bounds-checked int conversions for plugin config values
CodeQL flagged 18 go/incorrect-integer-conversion warnings on the moved
plugin handler files: results of pluginworker.ReadInt64Config (which
ultimately calls strconv.ParseInt with bit size 64) were being narrowed
to int32/uint32/int without an upper-bound check, so a malicious or
malformed admin/worker config value could overflow the target type.
Add three helpers in weed/plugin/worker/config.go that wrap
ReadInt64Config and clamp out-of-range values back to the caller's
fallback:
ReadInt32Config (math.MinInt32 .. math.MaxInt32)
ReadUint32Config (0 .. math.MaxUint32)
ReadIntConfig (math.MinInt32 .. math.MaxInt32, platform-portable)
Update each flagged call site in the four moved task packages to use
the bounds-checked helper. For protobuf uint32 fields (volume IDs)
the variable type also becomes uint32, removing the trailing
uint32(volumeID) casts and changing the "missing volume_id" check
from `<= 0` to `== 0`.
Touched files:
weed/plugin/worker/config.go
weed/worker/tasks/balance/plugin_handler.go
weed/worker/tasks/erasure_coding/plugin_handler.go
weed/worker/tasks/vacuum/plugin_handler.go
* refactor(worker): use ReadIntConfig for clamped derive-worker-config helpers
CodeQL still flagged three call sites where ReadInt64Config was being
narrowed to int after a value-range clamp (max_concurrent_moves <= 50,
batch_size <= 100, min_server_count >= 2). The clamp is correct but
CodeQL's flow analysis didn't recognize the bound, so it flagged them
as unbounded narrowing.
Switch to ReadIntConfig (already int32-bounded by the helper) for
those three sites, drop the now-redundant int64 intermediate variables.
Also drops the now-unused `> math.MaxInt32` clamp in
ec_balance.deriveECBalanceWorkerConfig (the helper covers it).
|