mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-21 21:44:22 +02:00
f8cfea0dcfed45ba1b04aa5fdef204fa1bbce3e9
15607
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 jsasandjames
|
f8cfea0dcf |
gcs: size the upload buffer from the entry instead of reserving the default 16 MiB chunk per writer (#11854)
Co-authored-by: james <james@medable.com> |
||
|
Jay Hemnani
|
c90da999f2 | s3: answer a conditional write on a missing object with 404, not 412 (#11882) | ||
|
Chris LuandDevin
|
379abbe278 |
volume: repair torn tails on canDelete volumes; reject damaged 1-3 byte records (#11876)
- Go/Rust: run the unindexed .dat tail repair on noWriteCanDelete volumes too — deletes still append tombstone records, so the same crash window applies and they previously degraded to fully read-only. - Go readTailNeedle: reject a positive size of 1-3 for v2+ as corrupted before ReadBytes slices the body's 4-byte data-size field and panics; a torn last record is now cut instead of crashing the load. - Tests: TestVolumeLoadRepairsTailOnCanDeleteVolume and TestVolumeLoadCutsDamagedSmallSizeTailOnCanDeleteVolume (Go), and the Rust mirrors volume_load_repairs_tail_on_can_delete_volume / volume_load_cuts_damaged_small_size_tail. Generated with [Devin](https://devin.ai) Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
GAURIandGAURI
|
c601712cc4 |
README: describe OLake Go and fix its wiki link (#11880)
README: describe OLake Go and point its link at the renamed wiki page Co-authored-by: GAURI <174253719+gauriiiiiiiiiiii@users.noreply.github.com> |
||
|
Chris LuandDevin
|
ab738bfd33 |
rust volume: repair an unindexed .dat tail at load (#11878)
* rust volume: repair an unindexed .dat tail at load Mirror Go's repairUnindexedDatTail (#11870): a crash between the .dat\nappend and the .idx append leaves a .dat tail no index entry covers,\nwhich sent the volume read-only on the next load and left acknowledged\nneedles unreadable. Before the index loads, re-index the complete\nneedles in the tail (a size 0 record as the tombstone or empty write\nresolveEmptyTailRecords reads it as) and cut only an incomplete last\nrecord. A tail an interrupted append does not explain is left as it is\nand the integrity check quarantines the volume read-only as before. Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: tests for the unindexed .dat tail repair Reload volumes through the real load path after dropping .idx entries\nand cutting or damaging the .dat, mirroring Go's\nvolume_checking_tail_repair_test.go: complete unindexed needles are\nre-indexed (deletes as tombstones, empty writes as empty needles), an\nincomplete last record is cut, and a tail an interrupted append does\nnot explain is left untouched with the volume read-only. Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: re-index a torn size 0 record as a live empty needle A delete and a write with no data append the identical size 0 record, so a recovered one cannot be told apart from the prior index state alone: the write path accepts an empty overwrite of a live key and indexes it with size 0, while a delete of the same key publishes a tombstone. Inferring a tombstone from prior live state turned a lost index row after an empty overwrite into a deletion -- the object read back not-found although the write was answered. Every re-indexed record is now indexed at its on-disk size, the entry publishWriteRequest writes for an empty write. The other misread is bounded: a recovered delete reads its key back as an empty object rather than not-found, and the size 0 record carries no data to resurrect. Review feedback on the Rust mirror (#11878): the same bug exists here; fixing both sides identically. Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: re-index a torn size 0 record as a live empty needle Mirrors the Go fix in this pull request. A delete and a write with no data append the identical size 0 record, so resolve_empty_tail_records could not tell them apart from the prior index state: it treated a zero-size record of a live id as a tombstone, although the write path indexes the same bytes as a live size 0 entry. A crash after an empty overwrite of a live key therefore recovered the object as deleted. Every re-indexed record is now indexed at its on-disk size, the entry publish_write writes for an empty write. A recovered delete reads its key back as an empty object rather than not-found; the size 0 record carries no data to resurrect. Review: greptile-apps[bot] and devin-ai-integration[bot] on #11878. Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: repair a .dat tail left past an empty .idx The writable index loader creates the .idx before the first needle append, so a crash after that append leaves a volume whose data nothing indexes. CheckVolumeDataIntegrity skipped an empty index outright, so the volume loaded writable and the surviving first needle stayed invisible while later writes appended past it. A zero-size index now runs the same repair with tailEntryPos -1: the walk starts at the first record past the superblock with no anchor needle and no append timestamp to compare against, gets the same complete-record validation and incomplete-tail policy, and re-verifies through the normal tail check once it wrote entries. A tail the repair cannot explain fails the check, so the volume loads read-only instead of writable on an unaccounted-for .dat. Review feedback on the Rust mirror (#11878): the same bug exists here; fixing both sides identically. Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: repair a .dat tail left past an empty .idx Mirrors the Go fix in this pull request. The writable index loader creates the .idx before the first needle append, so a crash after that append leaves a volume whose data nothing indexes. Both the repair and the integrity check skipped an empty index, so the volume loaded writable and the surviving first needle stayed invisible while later writes appended past it. An empty .idx now runs the same repair with no anchor: the walk starts at the first record past the superblock, and a tail it cannot explain fails the load like any other repair failure, since every later check skips an empty index. Review: devin-ai-integration[bot] on #11878. Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: sync the .dat before publishing re-indexed .idx rows A power loss after the repaired .idx rows become durable but before the .dat tail records they name are flushed leaves the index pointing past the end of the data file; the next load then keeps the volume read-only until an operator steps in. Sync the .dat first so the rows never outlive the records they reference. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: sync the .dat before publishing re-indexed .idx rows Mirrors the Go fix: a power loss after the repaired .idx rows become durable but before the .dat tail records they name are flushed leaves the index pointing past the end of the data file. Sync the .dat first. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
222e96d6ab |
iceberg maintenance: merge bin files across parquet schema spellings (#11875)
* iceberg maintenance: compare parquet schemas by field id in bin-pack merge parquet.EqualNodes ignores Iceberg field ids, so a column dropped and\nre-added under the same name and type compared equal and was merged,\nfiling the new column's values under the dropped field id. Require the\nfield ids to agree wherever both sides carry one; the v3 layout's\nper-field check uses the same test.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: merge bin files whose schemas differ by field id A bin used to take the first file's Parquet schema as the writer's and\nrefuse every file that was not structurally identical, so a table never\ncompacted once files had a different column set or REQUIRED-vs-OPTIONAL\nrepetition. A file whose flat columns the writer's schema covers by\nfield id is converted into it (a column the file lacks reads as null);\na file that covers the writer's schema restarts the bin with it, so no\ncolumn is dropped. Equality-delete columns resolve on each file's own\nschema. Renames, type changes, nested columns and files without field\nids stay refused. The sorted and spatial merges are unchanged.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: test bin-pack merge across parquet schema spellings Regression coverage for the field-id merge: an added column in either\norder, REQUIRED vs OPTIONAL, equality deletes resolved per file, the\nre-added column refused on v1/v2 and v3, and the refusals that stay\n(rename, type change, one-sided columns, an added required column, no\nfield ids).\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: resolve missing delete columns as null in bin-pack merge An equality delete on an optional column added after a data file was\nwritten no longer refuses the bin: the file reads the column as null,\nso a real delete key cannot match its rows and a null key deletes them.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: write bin-pack merges under the files' union schema The writer schema is now the union of the bin's file schemas by field\nid, computed in a schema-only pass before any row is written: a column\nstays REQUIRED only where every file has it and spells it so, and the\nmerge refuses only true name/type/id conflicts. Files that need a union\nneither side covers (an added column plus a REQUIRED/OPTIONAL\ndifference, disjoint added columns) now merge in a single pass instead\nof refusing, and progressively wider bins no longer reread and rewrite\nevery earlier file per restart.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: spell the re-added column's test file for codespell Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: resolve equality-delete columns by field id A column renamed after its data and delete files were written still carries its field id there; resolving it by id instead of the current name keeps a rename from turning every equality key null. The missing-column null-key fallback is now for data files only: an equality-delete file that lacks its equality column is refused, as the spec requires it to carry them. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: read only the parquet footer choosing a bin's writer schema The union-schema pass downloaded every input in full, then the merge downloaded each again. The footer suffices for the schema, and a chunked file's ReadAt pulls only the chunks the footer spans. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
a724f4296c |
s3, s3tables: create buckets and namespaces exclusively (#11872)
* filer_pb: add MkdirExclusive A taken name leaves the callback as a result, not an error, so filer failover does not count a lost race as a filer failure. * s3tables: create the entry with its attributes in one exclusive create CreateTableBucket and CreateNamespace checked that the name was free and then wrote the entry in several calls, so a second create in the window replaced the first caller's entry including its owner. The create now carries the marker, metadata and tags in one O_EXCL CreateEntry that answers ErrEntryAlreadyExists with the same 409. * s3: PutBucket uses an exclusive mkdir The same window let a plain PutBucket replace a table bucket created concurrently. The already-exists branch now answers from the entry itself, since the registry just cached the name as absent, and the admin auto-create path is exclusive too. * s3: read the winning entry when a create loses the race The concurrent-create branch called existingBucketError on a name the registry had just cached as absent, answering NoSuchBucket, and the auto-create fallback could assign an S3 owner to a table bucket it met there. The negative cache is cleared before the existing-bucket error is built, and a table bucket keeps its owner-less shape. * s3: fail an auto-create that met a table bucket A table bucket carries no S3 owner and must not gain one, and the upload that auto-create ran for must not continue into it either. * s3: fail an auto-create when the winning entry cannot be read An unreadable existing entry fell through to success, so the write continued into an unvalidated entry, possibly a table bucket. |
||
|
Chris Lu
|
f05d743459 |
s3: resolve versionId tagging requests to the version-tagging actions (#11874)
* s3: resolve versionId tagging requests to the version-tagging actions A request for an object's tags with versionId acted on a historical version but authorized against the current-object tagging action, so an explicit Deny on s3:*ObjectVersionTagging never applied. * iam: map PutObjectVersionTagging and DeleteObjectVersionTagging to the tagging action * s3: only resolve tagging when the query value is empty The tagging routes match ?tagging with an empty value, so a non-empty one is not a tagging request and would resolve a tagging grant onto the plain read or version read that actually serves it. * s3: keep resolving tagging for any query value The routes match ?tagging whenever the key is present, so tagging=x is a tagging request served by the tagging handlers and must resolve to the tagging actions. |
||
|
Chris Lu
|
03a75cbc52 |
s3tables: only the admin repairs a table bucket that has no owner metadata (#11873)
* s3tables: only the admin repairs a table bucket that has no owner metadata CreateNamespace wrote bucket metadata naming the caller as owner before any permission check ran, so an ordinary caller that found the marker without metadata was written in as the bucket's owner and passed every check after. A bucket in that state is answered with the operation's own denial; the admin or a trusted internal handler may still repair it. * s3tables: repair the marker-only bucket under the entry lock A creator finishing its own create between the missing-metadata check and the repair write would have its owner overwritten. The mutation now adopts whatever metadata is there and writes only while it is still absent. * s3tables: gate the repair on the Admin action, not an operation grant hasAdminAction also answers s3tables:*, so a non-admin identity holding the wildcard operation grant could pass the repair check and write itself in as the bucket's owner. |
||
|
dependabot[bot]
|
efa9d304d2 |
build(deps): bump github.com/klauspost/compress from 1.18.0 to 1.18.7 in /test/kafka/kafka-client-loadtest (#11879)
build(deps): bump github.com/klauspost/compress Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.18.0 to 1.18.7. - [Release notes](https://github.com/klauspost/compress/releases) - [Commits](https://github.com/klauspost/compress/compare/v1.18.0...v1.18.7) --- updated-dependencies: - dependency-name: github.com/klauspost/compress dependency-version: 1.18.7 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
github-actions[bot]
|
b7f5d04428 | docs: regenerate star history chart | ||
|
Chris LuandDevin
|
ece2f7a7c2 |
iam: targeted writes for single-user IAM actions (#11877)
* s3api: persist single-identity IAM changes through the credential manager CreateAccessKey, DeleteAccessKey, UpdateAccessKey, PutUserPolicy, DeleteUserPolicy, SetUserStatus, UpdateUser and DeleteUser used to load the whole configuration, change one identity and then SaveConfiguration, rewriting and re-propagating every identity. They now write only the identity (and, for rename/delete, the groups and service accounts that referenced it) through the credential manager, which propagates that identity alone. Static identities and skipPersist keep the previous full-save/no-write contract. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iamapi: persist single-identity IAM changes through the credential manager Same targeted-write change as the embedded IAM API for the standalone weed iam dispatcher: UpdateUser, DeleteUser, CreateAccessKey, DeleteAccessKey, UpdateAccessKey, PutUserPolicy and DeleteUserPolicy write one identity through the credential manager instead of saving the whole configuration. CreateAccessKey keeps the full save when the handler created a user the store does not know. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * credential/memory: re-key identity on UpdateUser rename UpdateUser stored the identity under the old username key even when identity.Name changed. Re-key it the way filer_etc does (delete old, save new) so the targeted rename path sees the moved record. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: cover targeted single-identity IAM writes Recording-store tests assert each rerouted action uses the single-identity credential store methods and never SaveConfiguration, in both the embedded S3 IAM dispatcher and the standalone iamapi dispatcher, plus the skipPersist and static-identity contracts and a memory-store rename case. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * credential: keep not-found and rename-collision semantics in stores IamGrpcStore.DeleteAccessKey translates gRPC NotFound to ErrAccessKeyNotFound so a repeat delete stays the no-op the IAM handlers advertise. MemoryStore.UpdateUser rejects a rename onto an existing user instead of overwriting it, matching RenameUser. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
610346ae8c |
iceberg: keep version-hint.text out of remove_orphans (#11867) (#11871)
Generated with [Devin](https://devin.ai) Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Javier de la TorreandJavier de la Torre
|
df89afbef4 |
volume: repair an unindexed .dat tail at load instead of loading read-only (#11870)
* volume: repair an unindexed .dat tail at load instead of loading read-only A needle write appends the needle to the .dat, then its entry to the .idx, then answers the client. A crash between the two appends leaves a .dat that ends past the last indexed needle; on the next start CheckVolumeDataIntegrity reports "actual N bytes expected M bytes" and the volume loads read-only, and a complete needle in that tail cannot be read because nothing indexes it. After a process crash those bytes belong to a write that was never answered; after a power loss a complete needle can survive in the .dat while its .idx entry did not, although the client was answered. So complete needles are re-indexed rather than cut. CheckVolumeDataIntegrity now repairs the tail before the existing check, which then runs on the repaired files as before. The repair: - acts only on a local, version 3 volume within MaxPossibleVolumeSize, with a whole number of .idx entries, that is about to load writable (neither noWriteOrDelete nor noWriteCanDelete is set); - requires the needle at the highest indexed offset to match its entry (id, size, whole record, CRC); - walks from the end of that needle to the real end of the file and gives every complete needle (header, body, checksum, timestamp and padding inside the file, CRC valid, append timestamp later than the needle before it) the .idx entry the volume server would have written, in .dat order; a size 0 record, which is how a delete is appended, is indexed as a tombstone; - cuts only an incomplete last record: fewer bytes than a header, a record running past the end of the file, or a last record whose CRC fails; - changes nothing otherwise (a damaged record with more data after it, a header with id 0 or a negative size, append timestamps going backwards), so the volume loads read-only exactly as before. The .idx entries are written and synced before the .dat is truncated and synced, so a crash during the repair leaves the needles indexed and the remainder still past them; the next load cuts it. One warning per repaired volume names the needles re-indexed and the bytes cut. Tests: volume_checking_tail_repair_test.go reloads volumes through the normal load path. Repaired: one and two complete unindexed needles, an unindexed delete, a partial header, a partial body, a complete needle followed by a partial one, a last needle with a bad CRC, a partial needle after an indexed tombstone; each reloads clean afterwards. Left read-only and unchanged: the last indexed needle with a wrong id or a bad CRC, a bad CRC followed by more data, zero bytes past the last needle, an index past the data, a volume persisted read-only, a partial .idx entry, and a clean volume. TestCheckVolumeDataIntegrityVerifiesDeletionTail now also checks that on a writable volume its 64 zero bytes are reported and kept. Fixes #11864 * volume: fail the integrity check when a tail repair cannot complete A repair that wrote the .idx or cut the .dat and then failed to sync only logged the error; the check that followed saw the repaired files and the volume loaded writable on a repair that may not be on disk. Any repair error other than an unexplained tail now fails the check, so the volume loads read-only, and the next load repairs again. A failed .idx append could leave part of an entry behind, which made every later load reject the .idx as a partial entry. The appended bytes are truncated away again before the error is returned. * volume: do not cut a tail record that could not be read The walk cut a last record whose read failed for any reason, so an I/O error (EIO) on a complete record erased it. Only a record whose bytes are damaged (needle.ErrorCorrupted: a CRC mismatch or a body that does not parse) is cut now; any other read error is returned with both files left as they are, and the volume loads read-only. * volume: treat a sized tail record with no data as damaged A needle with no data is written with size 0. A tail record whose header gives a body but whose data size field reads 0 parsed as an empty needle, and ReadBytes checks the CRC only when there is data, so the repair indexed it with its positive size and reads returned an empty body. Such a record is now damaged like one with a bad CRC: cut when it is the last record, otherwise the tail is left alone and the volume loads read-only. * volume: keep an unindexed empty write as an empty needle A delete and a write with no data both append a size 0 record, and the repair indexed every size 0 record in the tail as a tombstone, so an acknowledged empty write whose .idx entry was lost read back as deleted. A delete is only appended for a needle that is live, so a size 0 record of a needle that is absent or deleted (in the .idx and in the tail before it) is an empty write and is re-indexed with size 0. One of a live needle stays a tombstone, the reading weed fix and the backup's index regeneration already give it. The .idx is scanned only when the tail holds a size 0 record. * volume: cut torn-tail test records inside their body, not at +40 The partial-record cases cut the .dat 40 bytes into a record. A needle with up to 7 bytes of data is a 40-byte record, so newRandomNeedle's random length (1..1023) left it whole about once in 150 runs per case: the repair then correctly re-indexed it and cut nothing, and the test failed on the .dat size. The cut now lands halfway through the record's body, which is always past its header and short of its end. --------- Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com> |
||
|
dependabot[bot]
|
1253b97303 |
build(deps): bump github.com/klauspost/compress from 1.15.9 to 1.18.7 in /test/kafka/kafka-client-loadtest/tools (#11866)
build(deps): bump github.com/klauspost/compress Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.15.9 to 1.18.7. - [Release notes](https://github.com/klauspost/compress/releases) - [Commits](https://github.com/klauspost/compress/compare/v1.15.9...v1.18.7) --- updated-dependencies: - dependency-name: github.com/klauspost/compress dependency-version: 1.18.7 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
a1d8739464 |
build(deps): bump github.com/klauspost/compress from 1.17.0 to 1.18.7 in /test/kafka/simple-consumer (#11869)
build(deps): bump github.com/klauspost/compress Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.17.0 to 1.18.7. - [Release notes](https://github.com/klauspost/compress/releases) - [Commits](https://github.com/klauspost/compress/compare/v1.17.0...v1.18.7) --- updated-dependencies: - dependency-name: github.com/klauspost/compress dependency-version: 1.18.7 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
1cfd9bbb4c |
volume: run the durable sync outside the reader-visible lock (#11863)
* volume: run the durable sync outside the reader-visible lock Reads of a volume waited on that volume's fsync because the write lock was held across Sync()/sync_all. A new per-volume lock (Go dataFileSyncLock, Rust data_file_sync_lock) now serializes every .dat mutation and lifecycle op across append+publish+sync+rollback, while the fsync itself runs with dataFileAccessLock/the io write guard dropped — so readers proceed during a sync, and a failed sync's rollback still cannot clobber another write or a compaction swap. Same change in Go (volume_write.go syncWrite/processBatch, SyncToDisk) and Rust (write_needles_grouped/write_synced_run). Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: publish after sync; fix Close/CommitCompact lock order Address review on #11863: - Close took dataFileSyncLock before waiting on isCompactionInProgress while CommitCompact claims the flag then takes the lock — a deadlock. Close now claims the flag before taking the locks (flag -> sync -> access). - Go now publishes needle-map entries only after the .dat sync, matching Rust's ordering: appendWriteRequest/appendDeleteRequest stage the bytes, publishWriteRequest/publishDeleteRequest index them after Sync. A failed sync rolls back bytes no reader ever resolved — the streamed-read offset-invalidation window is gone. processBatch splits runs at repeated needle ids (Rust parity) so dedup/cookie checks stay sequential; the old snapshot/restore rollback machinery is removed. - Rust commit_compact (store-lock caller) takes data_file_sync_lock via try_lock -> WouldBlock before the write lease; commit_compact_held and the tier-down _held paths document/require the caller-held sync lock, and the two vol.instance().write_lock() grpc sites take it first. The swap can no longer land between a durable write's append and publish. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: restore write watermarks when an inline fsync fails The syncWrite rollback truncated the .dat but left lastAppendAtNs, lastWriteAppendAtNs, lastWriteNeedleKey, lastWriteDeleted and lastModifiedTsSeconds describing the rejected write; recovery and compaction could read a watermark for a record that no longer exists. Snapshot and restore them on failure, as processBatchRun already does. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
0de6347b11 |
s3: mask anonymous misses behind AccessDenied unless the bucket is listable (#11857)
* s3: mask anonymous misses behind AccessDenied unless the bucket is listable Deferred anonymous GET/HEAD authorization only runs once an entry is selected, so misses returned 404 while existing keys returned 403 — an unauthenticated existence oracle for bucket names and object keys. Route not-found responses in Get/HeadObjectHandler through a check that only discloses 404 when anonymous listing is allowed (policy, identity or a public-read bucket); otherwise the miss is indistinguishable from a denied read, matching AWS and 4.48 behavior. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: mask missing-directory misses for anonymous reads too Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: check a fixed request shape when masking anonymous misses The miss check passed the caller's request through, so query parameters could retarget the resolved action (?policy) or satisfy an s3:prefix condition on an unrelated subtree. Each check now runs on a request with a fixed shape: a ListBucket scoped to the probed key and a plain read of the key, so the miss only shows when the caller could learn the same fact directly. * s3: mask conditional-header misses for anonymous reads An If-Match conditional on a missing key wrote NoSuchKey directly from processConditionalHeaders, before the handlers reached the masked miss path, so anonymous callers could still distinguish a missing key from a denied read. * s3: keep the versionId when masking anonymous misses The read leg must evaluate the action the caller's request resolves to: a versioned miss is s3:GetObjectVersion, and dropping versionId would let a plain GetObject grant disclose versions the caller may not read. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
6936bbdc3e |
rust volume: score EC shard placement with the volume's own ratio (#11862)
The auto-select branches of ReceiveFile (is_ec_volume) and VolumeEcShardsCopy passed the build-default DATA_SHARDS_COUNT into find_ec_shard_target_location, so its free-slot maths scored every volume as 10+4 even when the volume's .vif declares a custom ratio (e.g. 6+2). A wider-ratio volume's shards could be steered onto a disk too full to hold them all, and a narrower one would be refused slots it actually has. Add Store::lookup_ec_data_shards(collection, vid), which answers from the mounted EC volume first and then from the .vif on whichever disk holds it (via read_ec_shard_config_across_dirs, so split -dir/-dir.idx and rebuild-staging layouts are covered), falling back to the default when no metadata exists yet. Mirrors the enterprise fix (seaweed-mono c22fa8a15). |
||
|
Chris Lu
|
4d06742664 |
Detect and quarantine duplicate volume ids (#11763) (#11828)
* master: detect and quarantine duplicate volume ids across layouts (#11763) * shell: volume.check.duplicate_ids audit command (#11763) * master: persist duplicate-vid quarantine across heartbeats (#11763) * master: quarantine all copies of a duplicated volume id Near-simultaneous registrations let each side quarantine only the other, so arrival order decided which copy stayed writable. A confirmed cross-layout collision means either copy could be the impostor, so flag every colliding copy read-only regardless of registration order. * topology: return a copy from VolumeLayout.Lookup Callers iterating the returned slice raced with registrations and removals replacing the location list in place. * master: release duplicate-vid quarantine when the conflict resolves A cross-node volume.move briefly shows the same id on two nodes under different disk layouts; permanent quarantine would pin a valid target read-only. Keep the first-reported state per quarantined copy and lift the marks once the conflicting copy unregisters, so only a persistent collision stays read-only. * shell: include ec shards in volume.check.duplicate_ids Ec-only volumes share the global id space. Compare shard collections against regular ones while allowing shards and regular copies of the same encoded volume. * master: keep quarantine release current and cover all removal paths Track the latest server-reported read-only flag while quarantined so a release restores the server s true state, and run the release check on collection deletion and data-node removal too, not only explicit volume unregistration. * shell: do not flag ec-only volumes as duplicates An ec-only volume has no regular copy, so its shard collections had nothing to compare against and always read as foreign. * master: quarantine every conflicting layout at the layout level (#11763) - collect all conflicting layouts, not just the first, so a vid spanning three layouts does not leave one writable - mark quarantine on the location list only: folding it into the stored report made the volume digest permanently disagree with the server and forced a full volume list every heartbeat - isAllWritable now honors the location read-only mark so a quarantined copy stays out of writables through resyncs and re-registration - re-check the conflict before keeping marks, and do not clear a node's quarantine while it still holds the vid in another layout - log the full layout keys on both sides of the conflict * master: push duplicate-vid quarantine to the volume server (#11763) A quarantined copy stayed writable on the volume server itself for a client holding an existing file id, since the master-only mark never reached the server's local write path. Quarantine now also calls VolumeMarkReadonly (can_delete so expiry still drains), is re-pushed if the server comes back reporting it writable, and VolumeMarkWritable runs when the conflict resolves and the server had reported writable. * master: retry unenforced quarantine marks and restore server writability on release (#11763) Dial the advertised grpc port, return mark errors, and keep a quarantine entry until the writable push succeeds so a released copy does not stay read-only on the server. * master: treat rp-only layout differences as an in-flight replication change (#11763) volume.configure.replication rolls through nodes one at a time, so a volume legitimately reports the old and new replica placement while it settles. Layouts differing only in rp within the same collection are a supported transition, not a reused id. * shell: group ec shards by collection in the duplicate id audit (#11763) EC shards of one volume legitimately span disk types when DiskTypePrefer spills, so grouping by disk type reports healthy volumes as duplicates. |
||
|
Chris Lu
|
2874bd05fb |
Freeze volume growth on topology id mismatch (#11763) (#11820)
* pb: ResumeVolumeGrowth rpc for operator-driven growth resume (#11763) * master: freeze volume growth on topology id mismatch (#11763) * shell: volume.growth.resume command (#11763) * master: replicate the growth freeze and gate it on veteran heartbeats (#11763) * master: block allocation while the topology id is being minted DeferAllocation only armed after the topology id persisted, so growth requests arriving during init ran ahead of the warmup window. Gate NextVolumeId on an init-pending flag spanning the whole mint. * master: stash reported topology ids until the local id is minted (#11763) * drain stashed topology id reports when the id arrives outside the mint path * persist the growth-freeze flag in the mirror file and raft snapshots * log hashicorp raft marshal and apply errors in applyGrowthFreeze |
||
|
Chris LuandDevin
|
1144bf15d4 |
rust volume: group-commit fsync uploads on the direct write path (#11860)
* rust volume: group-commit fsync uploads on the direct write path Each fsync=true upload used to run its own .dat/.idx sync when the write queue was off, which is the default. Route durable uploads through the per-volume write workers so a batch shares one sync, matching Go's writeNeedle2: non-fsync writes, and writes while the server is stopping, stay on the synchronous path. The queue is now always constructed and SEAWEED_WRITE_QUEUE is gone. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: document post_handler write routing Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
8d2b417c69 |
iceberg, lance: vend read-only credentials to callers that may only read (#11859)
* s3tables: share the UpdateTable authorization gate Extract the validation, load, and permission check from handleUpdateTable into authorizeUpdateTable so the table an update was authorized against is the same snapshot the mutation writes to. Manager gains AuthorizeUpdateTable, mirroring AuthorizeCreateTable, so callers can ask the question without updating anything. * s3tables: share the UpdateView authorization gate Extract the lookup, policy load, and permission check from handleUpdateView into authorizeUpdateView, mirroring authorizeUpdateTable, and expose it as Manager.AuthorizeUpdateView for callers that need the answer without the update. * s3api: scope vended table credentials to read-only on request VendTableCredentials takes a readOnly flag and tablePrefixSessionPolicy emits only s3:GetObject for the table prefix when it is set, keeping the conditioned ListBucket and unconditioned GetBucketLocation statements that reads still need. * iceberg: vend read-only credentials when the caller may not write LoadTable and friends vend storage credentials only for the authority the caller holds: the vend now checks UpdateTable/UpdateView permission and a caller that may only read the catalog entry gets a read-only session. Paths that just passed a write gate (stage-create, view commit) keep the full session. * lance: vend read-only credentials when the caller may not write storage_options vends the same read-only session as the Iceberg catalog when the caller lacks UpdateTable on the table. * lance: keep write credentials for callers authorized to declare A Lance commit is a direct S3 write that never revisits the catalog, so a caller that just passed CreateTable still gets a writable session; the UpdateTable gate applies to describe, the read path. * iceberg, lance: log a failed write check instead of only reading it as denied |
||
|
|
7ed26deaae |
s3: list keys in byte order around directory markers (#11832)
* s3: list keys in byte order around directory markers * s3: order deferred dir blocks by slash name and hold quota * s3: fix deferred dir blocks at page and marker boundaries Review follow-ups on the marker byte-order change: - A marker inside an emitted "<name>/" block re-descended and re-listed its folded contents under delimiter "/" instead of skipping the block. - A prefix object's "<name>/" common prefix was emitted eagerly at the name position, outranking siblings like "a.txt" and repeating across pages; the plain key now emits alone at the name position and the block emits the common prefix when children exist. - A recursion starved on quota before listing a deferred subtree returned without truncation, dropping keys silently; the block now reports a resumable truncation when the subtree still holds entries. - Clone prefix-object entries with proto.Clone rather than copying the protobuf message by value. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: close the hasEntries probe stream on return * s3: resume delimited listings at the dir block boundary A deferred directory block emits at its "<name>/" position, but emitDir left nextMarker at the bare name, which sorts inside the (name, name/) band and reads as "block pending". Every pass of the listing loop then re-emitted the same common prefix until maxKeys ran out (999 prefixes, IsTruncated), and a "<name>/" continuation token trimmed to "<name>" by adjustMarkerForDelimiter reopened the block and paged forever. Advance nextMarker to the block boundary the same way the recursive branch does, and keep a "/" -ending marker intact — it already names the bound past the block. --------- Co-authored-by: Devin <devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
GAURIandGAURI
|
da0d423bd9 |
Add OLake to README and Admin UI; fix catalog_olake path-style docs (#11858)
* README: mention OLake in the Iceberg lakehouse section * catalog_olake: fix path-style prose; admin UI: list OLake as a REST catalog client * admin UI: restore generated file import block, keep only the string change --------- Co-authored-by: GAURI <174253719+gauriiiiiiiiiiii@users.noreply.github.com> |
||
|
Chris LuandDevin
|
2095c905ba |
server: give heartbeat-options test a topology (#11853)
heartbeatResponse now reads MaxVolumeId and TopologyId off ms.Topo, but the test built a MasterServer without one and panicked, failing the package for every PR. Generated with [Devin](https://devin.ai) Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
6b6a268903 |
volume balance: stop replica-fallback moves between equally full servers (#11851)
* volume balance: unify the per-move convergence guard Evaluate the post-move utilization comparison through serverUtilization so the guard also applies when servers lack MaxVolumeCount info (raw-count mode), where it previously did not run at all. * volume balance: restrict fallback destinations to improving servers The score-based fallback inside createBalanceTask could redirect a move to any valid server, including ones as full as the source — between equally full servers the move only relocates the imbalance and the next detection cycle reverses it (#11839). Exclude from eligibleTargets any server whose post-move utilization would exceed the source's, so a fallback destination satisfies the same convergence guard as the loop-chosen target. * volume balance: skip a volume when the chosen target violates placement When the detection loop's target resolves but cannot host the replica (it already holds one, or is out of the filtered scope), createBalanceTask now returns nil so the loop advances to the next volume — matching shell volume.balance, which continues to the next candidate instead of redirecting the move to a different server. The score-based fallback is kept for the case where the target cannot be resolved at all; its allowed set is already limited to servers the move would improve. * volume balance: regression test for replica-constrained targets (#11839) Reproduces the reported loop: the emptiest server holds the second rp=001 replica of the lowest-ID volumes on the fuller servers, so it is never a legal target for them. Detection must skip those volumes and send the movable ones to the under-utilized server, and re-running detection after applying the plan must never move the same volume twice. |
||
|
Chris LuandDevin
|
79daffb2d2 |
test: retry retryable 503s in cold-start assign test (#11852)
The master intentionally sheds an assign that triggered volume growth once its wait window elapses (503 + Retry-After); on a slow CI box cold-start growth can outlast it, so the single-shot assertion flaked. Honor the documented retry contract like a real client, within a 60s deadline. Generated with [Devin](https://devin.ai) Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
github-actions[bot]
|
207d06863a | docs: regenerate star history chart | ||
|
Chris LuandDevin
|
7e348951c2 |
s3: strip internal metadata headers on auth entry (#11850)
* s3: strip internal object-lock and version headers on auth entry putToFiler persists Seaweed-X-Amz-Object-Lock-Mode, -Retention-Until-Date, -Legal-Hold and -Version-Id request headers into authoritative entry metadata without a permission check, so an s3:PutObject-only writer could pin COMPLIANCE retention and legal hold on its own uploads, bypassing the dedicated-permission gate on the standard x-amz-object-lock-* headers. The server only sets these headers from validated inputs, so drop them in authenticateRequestInternal alongside the identity headers already scrubbed there. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: strip internal SSE headers on auth entry X-SeaweedFS-SSE-S3-Key-Data, -SSE-S3-Base-IV and -SSE-KMS-Base-IV are read from the request to drive multipart encryption, so a client could inject its own key material and IVs. The server sets them itself between handler stages of a request; drop inbound values in authenticateRequestInternal. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: scrub internal headers before the auth-enabled early return Auth and AuthPostPolicy skip authenticateRequestInternal when IAM is disabled, so client-supplied internal headers still reached putToFiler. Hoist the scrub into scrubInternalHeaders and run it unconditionally in both wrappers as well as the shared auth entry point. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: test internal header scrub through the auth-disabled wrapper Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
8b09c96cec |
s3: keep a signed content-length in the canonical request (#11849)
Co-authored-by: Devin <devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
0fb6c0e77d |
filer.sync: unstick sync on chunks the source no longer has (#11845)
* filer.sync: keep the needle-404 clock from resetting on live volumes The give-up clock for a source chunk that cannot be produced is kept per volume so a vacuumed volume's files all inherit one wait. But a needle 404 and a volume lookup failure shared that clock, and sourceServed cleared it whenever any needle on the same volume read fine, so on a busy cluster the grace never elapsed for a single deleted needle on a healthy volume and filer.sync retried it forever (#11841). Give needle not-found answers their own per-volume clock that serving siblings does not reset. * filer.sync: an answered 404 needs no liveness probe before skipping After the missing-chunk gate gives up, onReplicateChunkError only skips the entry when a re-read of the most recently served chunk succeeds. That probe can never have been armed on a fresh sync, and its single stored file id can itself be vacuumed since — either way the entry stays pinned and the offset never advances. A needle 404 was answered by a live volume server, which is itself proof the source is serving requests, so skip on it directly. Keep the probe for the lookup-failure flavor, where no server answered and the silence can still mean a source outage. * filer.sync: skip a replayed version the source rewrote in place Supersession was decided by entry mtime alone, so a source entry whose chunks were swapped without an mtime bump — remote cache eviction and re-cache, an in-place repair — never read as superseded, and its vacuumed chunks were retried indefinitely. When a chunk is missing, compare the replayed chunk's file id against the live entry's resolved chunk set: absent means the replayed version is obsolete and its event is skipped on the first failure instead of after the give-up window. The same live-entry comparison now also decides the entry-level skip in onCorruptChunk/onReplicateChunkError. * filer.sync: give each missing needle its own grace clock A 404 is one needle's answer, but the clock was kept per volume: once the first needle's wait expired, every other needle on that volume inherited an already-elapsed clock and could be written off on its first sighting. Track the wait per file id and clear it only when that needle itself serves again, so a live volume's healthy needles neither shorten nor restart it. * filer.sync: keep the probe for filer-proxied 404s ReadPart proxied through the source filer collapses "no volume locations" into the same 404 a missing needle returns, so that answer cannot stand in for the liveness probe the way a direct volume read's can. Only treat 404 as an answered needle when the sink reads volume servers directly. * filer.sync: only a not-older live entry can supersede a replay Chunk-list comparison must respect the same ordering the mtime check does: a live entry older than the replayed version means the source was rolled back, so its differing chunks say nothing about whether the replayed data is still needed. Gate the content check on live mtime >= replayed mtime in both sourceMovedOn and sourceDroppedChunk. * filer.sync: compare chunk ids without resolving manifests resolvedChunkFileIds downloaded manifest files to expand chunk lists, so a manifest sitting on a dead volume made the supersession check itself fail and left the offset pinned behind a replica the source cannot read. File ids are immutable needle identities — the unresolved sets already compare chunk content exactly, and the check no longer performs I/O. * filer.sync: judge nested chunks by their top-level chunk A manifest child is not in the entry's chunk list even when the manifest still references it, so checking the live entry for the child id reported it dropped and returned the raw 404 — unwrapped, so the offset still pinned. Thread the entry-level chunk id down the manifest recursion and consult it for the dropped-chunk check instead. * filer.sync: let the manifest-child test exit through the gate Run the worker with a shortened grace period and assert the returned error is marked errSourceChunkMissing, instead of racing the worker still running against a restored RetryWaitTime. |
||
|
Jay HemnaniandChris Lu
|
60b06b6881 |
s3: sort ListMultipartUploads by key and honor key-marker (#11833)
* s3: sort ListMultipartUploads by key and honor key-marker * s3: skip upload folders without an object key in ListMultipartUploads * s3: order same-key multipart uploads by initiation time The upload id suffix is now a time-ordered UUIDv7, so sorting by upload id lists a key's uploads oldest first, like S3. Pagination is unchanged: ordering and the upload-id-marker comparison share the same key. selectMultipartUploads now builds the whole result, including NextKeyMarker and NextUploadIdMarker, and the pagination test pages with the response markers. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris LuandDevin
|
dbcf6850b6 |
ci: fail the awscli multipart retry loop on the last attempt (#11848)
Co-authored-by: Devin <devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
04eff7d613 |
ci: retry the awscli multipart download on transient failures (#11846)
The HEAD on the completed multipart object occasionally gets a 403 from a signature verification mismatch before the object read settles; retry the download instead of failing the run. Co-authored-by: Devin <devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
4feb30deff |
Federated volume-id bound via heartbeat stamps (#11763) (#11819)
* pb: heartbeat fields for the federated volume-id bound (#11763) Heartbeat carries max_volume_id_seen and last_seen_topology_id up, and HeartbeatResponse echoes max_volume_id and topology_id down, so the allocation bound is replicated to every volume server. * master: echo the volume-id bound and fold reported bounds in (#11763) Every heartbeat response carries the leader current max volume id and topology id; every heartbeat reported max_volume_id_seen is folded into the topology bound via UpAdjustMaxVolumeId. * volume: persist a per-dir vid stamp of the highest id ever seen (#11763) The store keeps max_volume_id_seen + last_seen_topology_id in a .vid_stamp file per volume dir, fed by heartbeat echoes, AllocateVolume, and local volume/EC creation, seeded at boot from stamps and local holdings, and reported in every heartbeat. A rebuilt master recovers the bound from the first surviving veteran report. * volume-rust: mirror the vid stamp and heartbeat echo handling (#11763) Same mechanism as the Go volume server: .vid_stamp per dir, bound folded in from add_location scans and allocate/mount calls, echoed fields applied on heartbeat responses, reported on every heartbeat. * volume: serialize vid stamp writes and fsync the directory (#11763) * volume-rust: serialize vid stamp writes and fsync the directory (#11763) * master: echo the allocation bound to connected servers on change The bound was only sent at connect time or on digest mismatch, so a healthy stream never learned later increases. If the high-id server then went offline and the master lost state, survivors restored a stale bound and the id could be reissued. Resend the heartbeat response whenever the bound or topology id moved. * volume: retry failed vid stamp saves on later updates A transient write failure left the in-memory bound raised but the stamp stale, and repeat updates at the same bound never retried. Track a dirty flag so any later update re-attempts the save. * volume: cover stamp bootstrap for pre-stamp dirs in test * volume-rust: retry failed vid stamp saves, mirroring Go * volume-rust: stamp every dir with the combined bound on add_location A dir whose own stamp sat below the combined bound kept its stale value after fold; losing the higher-stamped disk then restarted the server below ids it had already learned. Rewrite all stamps after each location folds in. * volume: stamp mounted volume and ec shard ids (#11763) * volume-rust: stamp mounted volume ids, mirroring Go (#11763) * volume: retry dirty vid stamp save on unchanged topology id (#11763) * volume-rust: retry dirty vid stamp save on unchanged topology id (#11763) * master: record echoed bound from the sent heartbeat response (#11763) |
||
|
Chris Lu
|
117ea58a83 |
s3tables: accept .puffin deletion vectors in table data/ (#11844)
* s3tables: accept .puffin data files for Iceberg v3 deletion vectors Iceberg v3 writes deletion vectors as Puffin files under a table's data/ directory, so merge-on-read DELETE and UPDATE on v3 tables were refused with AccessDenied by the table-bucket upload validator. Fixes #11842 * s3tables: name all accepted data formats in rejection error The message stopped at .avro even though .lance and now .puffin are accepted. * s3tables: cover deletion-vector filenames in data layout test Uses the names real engines emit: Spark/Iceberg Java {seq}-{task}-{uuid}-{counter}-deletes.puffin, UUID-suffixed names, and the same under a partition directory. |
||
|
dependabot[bot]
|
ce93660abb |
build(deps): bump github.com/linxGnu/grocksdb from 1.10.8 to 1.11.1 (#11836)
Bumps [github.com/linxGnu/grocksdb](https://github.com/linxGnu/grocksdb) from 1.10.8 to 1.11.1. - [Release notes](https://github.com/linxGnu/grocksdb/releases) - [Commits](https://github.com/linxGnu/grocksdb/compare/v1.10.8...v1.11.1) --- updated-dependencies: - dependency-name: github.com/linxGnu/grocksdb dependency-version: 1.11.1 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
73391a2e2f |
build(deps): bump github.com/tidwall/gjson from 1.19.0 to 1.20.0 (#11834)
Bumps [github.com/tidwall/gjson](https://github.com/tidwall/gjson) from 1.19.0 to 1.20.0. - [Commits](https://github.com/tidwall/gjson/compare/v1.19.0...v1.20.0) --- updated-dependencies: - dependency-name: github.com/tidwall/gjson dependency-version: 1.20.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
3a30ee177a |
build(deps): bump github.com/klauspost/compress from 1.20.1 to 1.20.3 (#11837)
Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.20.1 to 1.20.3. - [Release notes](https://github.com/klauspost/compress/releases) - [Commits](https://github.com/klauspost/compress/compare/v1.20.1...v1.20.3) --- updated-dependencies: - dependency-name: github.com/klauspost/compress dependency-version: 1.20.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
109dd24f52 |
build(deps): bump github/codeql-action from 4.38.3 to 4.39.0 (#11840)
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.38.3 to 4.39.0. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/v4.38.3...v4.39.0) --- updated-dependencies: - dependency-name: github/codeql-action dependency-version: 4.39.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
a2f8f4943c |
build(deps): bump gocloud.dev/pubsub/rabbitpubsub from 0.46.0 to 0.47.0 (#11835)
--- updated-dependencies: - dependency-name: gocloud.dev/pubsub/rabbitpubsub dependency-version: 0.47.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
4909843748 |
build(deps): bump modernc.org/sqlite from 1.57.0 to 1.60.1 (#11838)
Bumps [modernc.org/sqlite](https://gitlab.com/cznic/sqlite) from 1.57.0 to 1.60.1. - [Changelog](https://gitlab.com/cznic/sqlite/blob/master/CHANGELOG.md) - [Commits](https://gitlab.com/cznic/sqlite/compare/v1.57.0...v1.60.1) --- updated-dependencies: - dependency-name: modernc.org/sqlite dependency-version: 1.60.1 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
0a96972a99 |
s3: verify X-Forwarded-Prefix only from trusted proxies (#11843)
* s3api: check the direct peer against the trusted-proxy allowlist * s3: verify X-Forwarded-Prefix only from trusted proxies * s3: note that trusted proxies must sanitize forwarded headers |
||
|
Jay Hemnani
|
4e5fed9e9e | s3: list the size of a directory marker that has a body (#11830) | ||
|
Chris LuandDevin
|
e7923c14d1 |
s3tables: honor resource policy Deny on remaining operations (#11825)
* s3tables: honor a table policy Deny for maintenance operations authorizeMaintenanceTable evaluated only the bucket policy, so a Deny in the table policy never vetoed PutTableMaintenanceConfiguration, GetTableMaintenanceConfiguration or GetTableMaintenanceJobStatus even though they act on the table ARN. The stored table policy is now checked for a matching Deny first via deniedByPolicy, a veto helper that keeps the admin exemption, and resource and bucket tags are loaded into the policy context so tag-conditioned Denies evaluate. readBucketPolicy is renamed to readPolicy and reused for the table path. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: honor the stored policy's Deny on policy management operations PutTablePolicy, GetTablePolicy and DeleteTablePolicy only evaluated the bucket policy, so a Deny in the table's own policy was ignored and a bucket-level Allow could strip or replace the policy guarding a table. The existing table policy is now loaded and a matching Deny vetoes the call. PutTableBucketPolicy likewise consults the existing bucket policy for a Deny before replacing it. Resource and bucket tags populate the policy context so tag-conditioned Denies evaluate, and policy writes assert the policy authorized against is still in force, returning a conflict when it changed mid-request. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: apply a table policy Deny to tag operations TagResource, ListTagsForResource and UntagResource evaluated only the bucket policy; a Deny in the table's own policy is now honored when the target ARN is a table. ListTagsForResource and UntagResource also returned early when the resource carried no tags attribute, skipping the permission check entirely; the check now always runs. Policy evaluation uses the canonical resource ARN rebuilt from stored metadata, so a request ARN with a forged account or region cannot dodge the Deny. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: regression tests for resource policy denies Cover the case where a bucket policy grants an identity s3tables:* while the table policy denies it: the policy, tag and maintenance operations must stay denied so the protection cannot be stripped. Also cover a bucket policy that denies PutTableBucketPolicy under the open fallback, and a tag-conditioned Deny, which vetoes only while the stored tag matches. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
51a4c092c8 |
rust volume: release the store write lock during needle I/O (#11823)
* volume: give Volume interior locks mirroring Go's Volume's mutable state moves behind interior locks matching Go's per-volume lock domains, so the store can hand out shared handles instead of requiring exclusive &mut access: - io: RwLock<VolumeIo> owns the .dat/remote handles, needle map, append cursor, compaction bookkeeping and write watermarks — Go's dataFileAccessLock-adjacent mutable set; helpers take the guard as &mut VolumeIo so nothing re-locks. - super_block and volume_info each get a RwLock (superBlockAccessLock, volumeInfoRWLock); dir_idx gets its own so path building never contends with I/O. - no_write packs Go's noWriteLock pair into AtomicU8; last_modified_ts_seconds and compaction_byte_per_second become atomics. The data-file write lease (dataFileAccessLock) now covers every mutating entry that previously relied on borrow exclusivity: writes, deletes, blob writes, set_writable's flag/index swap, remote-file updates, local-backend swaps, index relocation, replica-placement rewrite, put_needle_index, close and destroy. set_read_only_persist leases only its can_delete idx-attach branch because mark_io_unavailable reaches it under a held lease. commit_compact takes try_write_lock — callers hold the store lock, so blocking would stall every volume behind one slow reader; WouldBlock surfaces as COMMIT_WAITS_FOR_READS for the caller to retry. Lock order is fixed: lease -> io -> volume_info/super_block/dir_idx. * storage: mount volumes as shared Arc<Volume> handles DiskLocation's map owns Arc<Volume> so the store can hand a caller a handle that stays valid across store-guard releases, instead of borrowing &mut Volume under the store write lock. - volumes: HashMap<VolumeId, Arc<Volume>>; find_volume returns a cloned Arc. - find_volume_mut and volumes_mut are gone — every Volume method is &self now, so shared handles cover what exclusive borrows did. - writable_volume_mut -> writable_volume(&self) -> Result<Arc<Volume>> keeps the same admission checks (exists, disk-space-low). - write_volume_needle/write_volume_needles/delete_volume_needle are &self; lifecycle methods that mutate the map or mounted state stay &mut self. * server: run needle I/O with the store guard released The store write lock previously wrapped needle writes and deletes end-to-end: process_batch held it across every grouped fsync, and the HTTP/gRPC fallbacks, tail receiver and fetch-and-write paths did the same. One volume's slow I/O then stalled admission and metadata on every other volume. - write_queue resolves each group's handle via writable_volume under a brief store read, then runs write_needles_grouped after the guard drops. - HTTP write/DELETE fallbacks, the gRPC batch-delete EC recheck + delete, tail receiver and fetch_and_write local writes, blob writes and the quarantine/sync/set_writable/remote-file paths all run under store.read() now that every volume method they reach is &self. - Lifecycle calls that mutate the volume map or mounted state (mount/unmount/delete, EC shard ops, compaction commits, load_new_volumes, heartbeat volume actions) keep store.write(). * test: cross-volume admission proceeds during another volume's write Regression coverage for the store-lock fix: - write_queue's test parks a volume-1 write inside its .dat append via a new needle_write_hook seam (mirroring needle_read_hook), then proves the store still hands out a lifecycle writer and a volume-2 handle — while volume 1's data-file lease still excludes same-volume writers. - tests/store_lock_repro.rs exercises the public API: concurrent read-guarded writes on two volumes, and a stale Arc<Volume> handle failing closed after unmount instead of writing into torn-down files. * test: drop store write guards that shared handles made read-only Test setup sites held store.write() purely to reach find_volume_mut; with Arc<Volume> and &self methods the guard is read-only (clippy readonly_write_lock). The volume lease still serializes the writes inside. * volume: drop a windows-only read_exact_at that shadows the io helper cargo's windows build failed E0255: the free fn copied with the interior-lock conversion collided with the unconditional crate::storage::io::read_exact_at import, which already carries the platform branches internally. * server: release the store read guard before needle I/O The read guard still ran through write_needle_blob_and_index, the tail receiver write, the fetch-and-write local write, and the HTTP write and DELETE fallbacks, so a slow append blocked mount/unmount writers (and, on writer-preferring locks, readers on other volumes). Resolve the Arc<Volume> under a short guard and write through the handle after it drops — the same scoping process_batch already used. * volume: serialize the read-only mark and its .vif save under the lease set_read_only_persist only leased the idx-attach branch, so concurrent mark-readonly and mark-writable calls could interleave flag updates and .vif writes — including two save_vif calls truncating and writing the file at once. The lease now spans the whole flag change and save, with a _held variant for mark_io_unavailable, which reaches this from write paths already holding the lease (Go: markVolumeReadonly runs under dataFileAccessLock). * volume,server: span the tier-down transition under one write lease update_remote_files cleared the remote-only write restriction and released its lease before open_local_dat_backend attached the local file, so a write could slip into the gap — pass check_writable, then fail appending to a missing dat_file. The handler now holds the lease across the reference trim, .vif save, and backend swap (and the rollback re-insert), via _held variants; the store read guard is scoped to the handle lookup. The swap still lands after the .vif trim — the ordering that keeps a crash from pointing the volume at a deleted remote object. * server: run a batch's volume groups on separate scoped threads A single batch can hold writes for several volumes; running the groups sequentially made every later volume wait behind a stalled append on an earlier one — the same head-of-line blocking the store guard release fixed, one level up. Each group now gets a scoped thread, preserving order within a group while independent volumes progress in parallel. * storage: close a volume handle a mount slot replaces With shared handles, inserting a new Volume over an existing map entry left the old handle alive in callers — two Volume instances could then write the same files with separate in-memory indexes. set_volume and create_volume now close() the replaced handle so its writers fail closed, matching what unload_volume already does. * test: key the write hook off a reserved needle id The hook registry is process-global and keyed by needle id, and other parallel tests write NeedleId(1) — an unrelated write could enter the barriers and leave the test's real writer parked forever. * server: wait out readers for vacuum commit without the store guard commit_compact try-locked the write lease because callers held the store write lock, returning COMMIT_WAITS_FOR_READS while a download was in flight — and the master reads that as a failed vacuum, discarding completed compaction work. The commit RPC now resolves the handle under a brief store read and blocks on the volume's lease, matching Go's CommitCompact under dataFileAccessLock: a slow reader stalls only this volume's vacuum. The try-lock path stays for callers that still hold the store lock. * server: scope the remaining store read guards to volume lookup Same class as the write-path fix: the batch-delete cookie check and tombstone, the query read, volume_needle_status, and the local chunk read all kept the read guard through the needle operation, so slow I/O there could still stall store writers. Each now resolves the Arc<Volume> under a short guard; the EC rechecks keep their guards. * seaweed-volume: fail closed on detached volume handles A handle resolved under a brief store read can outlive its mount: an unmount+remount between lookup and lease leaves the Arc pointing at a closed Volume while a new instance owns the files. Mark the handle in close() and reject it under the write lease in commit_compact_held, delete_needle, check_writable, set_read_only_persist and update_remote_files_held; the tier-down handler also checks the flag before deleting a remote object its replacement still reads. * seaweed-volume: drop the store guard before mark-readonly waits on the lease mark_volume_readonly held a store read across set_read_only_persist, which waits out the volume's leased readers; a slow download would stall every lifecycle operation behind the guard. Resolve the handle under a short read, then mark off the guard; a detached handle now maps to not_found. * seaweed-volume: retry batch deletes that raced an EC mount The store guard now drops before delete_needle, so ec.encode can mount shards between the pre-delete recheck and the tombstone; the encode then removes the .dat and the needle resurrects from pre-tombstone shards. Re-check has_ec_volume after each successful delete and return the retriable 503 so the filer requeues onto the EC path. * seaweed-volume: one write-queue worker per volume A stalled batch kept later batches for other volumes queued behind it: the single worker awaited process_batch before receiving again. Go's volume_write.go keeps one batching worker per volume, so give each volume its own FIFO worker spawned lazily, bounded by a global concurrent grouped-I/O semaphore and a global queued-request bound; workers exit only when their channel closes, and submit refuses unknown volume ids before spawning. * seaweed-volume: move the queued-write permit inside the request submit held the global queued-slot permit across the send and the response wait, so a canceled submit freed the slot while the worker still held the needle; repeated cancels could overflow the bound. The request now owns the permit until the worker drops it. * seaweed-volume: retire write workers whose volume left the store worker_for kept a sender per volume forever, so servers that mount and retire volumes accumulated idle workers. A worker now polls while idle and retires once its volume leaves the store: close the channel so racing senders fail fast, drain accepted writes, then release the map slot — a replacement worker can never start while this one still holds queued writes, so same-volume FIFO survives retirement. |
||
|
Chris Lu
|
0b3ed4502f |
Design + master mirror of max volume id outside raft (#11763) (#11818)
* design: durable volume-id allocation boundary (#11763) * master: mirror max volume id to a file outside raft (#11763) The allocation bound lived only in raft state, so a master that lost it rebuilt the bound from whatever volume servers were connected and could reissue ids held by offline servers. NextVolumeId now persists the bound to <mdir>/max_volume_id before the raft apply, and both raft implementations persist on apply and recovery so follower mirrors stay current. The file also carries the topology id, which survives raft state cleanup the same way the snapshot recovery did. * master: serialize and fully sync the max volume id mirror (#11763) * master: rewrite the mirror when the topology id changes at the same bound A same-bound raft apply carrying a new topology id was skipped by the lastPersistedMaxVolumeId check, leaving the mirrored topology id empty until the next allocation. Track the mirrored topology id too and rewrite when either value differs. * drop the design doc from the PR stack |
||
|
Chris Lu
|
ab559bca76 |
storage: hash index-metric bloom filter with xxhash
The reverse index walk dedupes seen NeedleIds with a BoomFilters bloom filter. Its default kernel is FNV-1, whose final step only XORs the last input byte into the hash's low bits, so keys sharing the high 7 bytes of a big-endian NeedleId land on correlated probe positions and false positives arrive in bursts of hundreds instead of the ~0.001 target rate. Each false positive hides a live-latest key and inflates the rebuilt deletion counters, which is what made TestFastLoadingNeedleMapMetrics flake past its +256 bound in CI (observed DeletedCount drift of 267). Swap the kernel for xxhash, which fully diffuses the key bytes; the observed false-positive count in 300 local runs of the test drops to max 2 (avg 0.23). |
||
|
|
d04d838ebd |
ec: compare live idx NeedleId sets before encode (#11812)
* ec: compare live idx NeedleId sets before encode Refuse encode when replicas' live .idx NeedleId sets differ. Walk only live rows (skip deleted and Size<=0) and compare NeedleId, ignoring Size so MIME/flags drift does not abort a matching id set. Mark the volume read-only for encode and reject writes after that. * fix(ec): compare all replicas then pick capacity source Log matching live idx sets with "replicas are consistent" for the shell CI check. After readonly, compare every replica's live NeedleId set, then require the encode source to have FreeVolumeCount>=2. Empty live-idx vs non-empty refuses encode instead of skipping superblock stubs. Drop the FileCount rebind after the idx gate. * volume: mark readonly under the write lock so checked appends finish first * ec: fix readonly deadlock and restore worker-host source preference MarkVolumeReadonly held dataFileAccessLock while calling reopenIdxForWrite, which re-acquires the same mutex and deadlocks TestMarkVolumeReadonlyCanDelete. Split out reopenIdxForWriteLocked for callers already holding the lock, matching the existing Locked pattern. The worker task lost its worker-host source preference when the idx gate replaced the replica union. Late-bind the source to a replica on the worker host using only the checked location set, with no FileCount rebinding. Drop the now-unused helpers, trim comments added by the earlier commits, and rename tests whose names still claimed convergence while asserting refusal. --------- Co-authored-by: Chepurnov Simon <195125934+CHEPURNOV-S-S@users.noreply.github.com> Co-authored-by: Devin <devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |