mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-06 14:31:57 +02:00
f0afcf904d7584449a8babb8e2f432a7ba490a3e
15252
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
f0afcf904d |
volume: an EC volume needs an .ecx to mount, and a 0-byte stub never outranks a real index (#11415)
* volume: an EC volume needs a non-empty .ecx to mount Two gaps against Go in how the Rust volume server treats the .ecx. EcVolume::new mounted with no index at all. The per-shard VolumeEcShardsMount path picks the disk by shard file alone, so a shard whose .ecx was on no local directory still registered and was advertised to the master; every VolumeEcShardRead then failed with "ecx file not open", and add_shard's 0-byte guard was neutralised because ecx_file_size stayed 0. Go's NewEcVolume returns an error wrapping os.ErrNotExist. EcVolume::new now fails with NotFound, and Store::mount_ec_shard looks up the .ecx owner across all disks first (findEcxIdxDirForVolume) so a shard on a sibling disk of its index still mounts instead of turning into a hard failure. A 0-byte .ecx stub, as left by a failed EC distribute copy, counted as a valid index. Go requires Size() > 0 wherever the file steers a decision: HasEcxFileOnDisk, findEcxIdxDirForVolume, indexEcxOwners (shared by reconcile and mirror), and VolumeEcShardsCopy removes a copied 0-byte .ecx and fails the copy. Mirror each through one is_usable_ecx_file helper. NewEcVolume itself still accepts a lone 0-byte .ecx as a legitimate empty index, but prefers a non-empty copy, local directory first, over a stub in the other directory; the resolution in EcVolume::new now follows the same order. Tests that mounted EC volumes without any .ecx get a real fixture. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * volume: mount_ec_shard tries every disk; reconcile ignores a 0-byte local .ecx mount_ec_shard returned the first disk's error, so an unusable shard copy (a 0-byte .ecNN left by an interrupted move) hid a good copy on the next disk. Like Go's MountEcShards, keep scanning: NotFound means "not this disk", any other failure is collected, and an all-disks-fail error names every disk tried. "No .ecx on any local disk" is now told apart from "shard not on this server". The orphan-shard reconcile took its locally-mirrored fast path whenever a local .ecx existed at all. A 0-byte stub there registered the shards against an empty index while the owner index skipped that same stub. Go gates the fast path on HasEcxFileOnDisk; do the same. ec_local_ecx_path loses its last production caller and becomes test-only. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * volume: match Go's mount error text and skip the owner stat on the owning disk MountEcShards in Go skips the HasEcxFileOnDisk stat when the disk's own directories already hold the .ecx, dedups a shared -dir.idx across locations in findEcxIdxDirForVolume, and reports "load failures" with the same wording. Also drop two issue-number references from comments. --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Eliah RusinandClaude Fable 5.1
|
c1ccbcda13 |
volume server: the write queue answers uploads with the needle's real ETag (#11414)
With SEAWEED_WRITE_QUEUE=1 every upload came back with ETag "00000000". The upload handler built the needle with Needle::default(), so its checksum was CRC(0), and handed a clone of it to the queue. The CRC was only computed in the write path, on the worker's clone, and WriteResult carries no checksum back, so n.etag() in the handler formatted the zero checksum. The direct path writes through &mut n and was correct. Compute the checksum in the handler while building the needle, the way Go's CreateNeedleFromRequest does, over the same bytes the write path hashes (the stored data, gzipped or not). The ETag and the has-name flag are read before the write, so the needle is moved into the queue instead of cloned, which also drops a full payload copy per queued upload. Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
|
94a68fa9b9 |
volume: walk_index_file keeps row alignment across short reads (#11412)
* volume: walk_index_file keeps row alignment across short reads walk_index_file issued one Read::read per batch and decoded whatever came back. Read::read may legally return a short count that is not a multiple of the 17-byte entry size (FUSE and network filesystems, a BufReader whose capacity is not a multiple of 17). The split entry at the end of the batch was dropped with no carry and the next read started mid-entry, so every later row was decoded from misaligned bytes and fed to the index as a garbage key/offset/size. This function backs every in-memory index load. Go's WalkIndexFile is immune because it reads through io.ReaderAt, which returns a full buffer or an error. Fill the batch buffer until it is full or the reader reports EOF, retrying ErrorKind::Interrupted, and only then decode whole entries. Reads stay batched at ROWS_TO_READ entries. EOF semantics are unchanged and match Go: on io.EOF Go decodes the whole entries in the final buffer, ignores a trailing partial entry and returns nil. A torn final entry is still skipped without an error here. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * volume: trim walk_index_file comments The batch-fill loop and the ShortReader test helper each carried a paragraph where a sentence suffices. --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Chris Lu
|
b3a8701989 |
lance: authenticate the catalog with Bearer tokens and x-api-key (#11431)
* lance: accept OAuth2 bearer tokens for catalog auth Lance and LanceDB clients can only send OAuth2 / Bearer / API-Key headers on catalog calls, never SigV4, so behind an auth-enabled S3 gateway every namespace request failed with 403 Access Denied. Mirror the Iceberg catalog's OAuth2 support: POST /oauth/token accepts an S3 access key / secret key as client_id / client_secret, validates them against IAM, and returns a signed JWT. The Auth middleware accepts that token as a Bearer credential before falling through to SigV4. Closes #11430 * lance: accept x-api-key header carrying an S3 credential The Lance namespace spec's third auth scheme maps api_key onto the x-api-key header. Accept "access_key:secret_key" there and validate it against IAM, so clients that only hold static headers can authenticate without minting a token first. * lance: answer invalid_client with the Basic challenge RFC 6749 5.2 requires a 401 from the token endpoint to carry WWW-Authenticate matching the scheme the client used, so it knows how to retry. * lance: cap the token endpoint request body /oauth/token is unauthenticated, so ParseForm needs the same size bound decodeBody applies to every other catalog request. * lance: keep query strings out of request logs /oauth/token rejects a client_secret sent in the query, but the logging middleware and the catch-all wrote RequestURI to the log before that rejection ran. Log the path alone so a mis-sent secret never reaches the log. * lance: log the escaped path, not the decoded one URL.Path decodes percent escapes, so a request like /%0aFORGED could split log lines. EscapedPath keeps the encoding while still dropping the query string. |
||
|
ssshr-66andChris Lu
|
b9ad62fc16 |
[Volume] Keep DAT and index state consistent after async batch Sync failure (#11425)
* fix 11400 * persist failed-recovery quarantine and harden rollback - record the unavailable state in a .unavailable marker, fsync it, and re-arm it on load so a restart cannot serve an unverified pair - quarantine the volume so heartbeats stop advertising it - block MarkVolumeWritable while unavailable, rechecked under noWriteLock - fail every request of a failed batch, not only the succeeded ones - restore the needle map and truncate .dat on inline fsync rollback failure - add truncateIndex for the sorted-file needle map - mirror the fail-closed semantics in the Rust volume server * volume: erase rolled-back mappings instead of leaving tombstones A rolled-back batch or failed inline write used Delete() to undo a needle that did not exist beforehand, leaving a tombstoned map entry whose stale offset makes the next write to that needle fail reading a header that no longer exists. Add removeMapping/restoreMapping to the mappers so recovery erases entries that were absent before the batch and reinstates the exact prior offset/size for ones that were, including tombstones. The index row still goes through Delete so a replay forgets the needle. * volume: gate bulk readers on unavailable and fsync the marker's dir - fsync_dir(&self.dir) synced the volume dir's parent, not the dir holding .unavailable; pass the marker path so the create survives a host crash - export UnavailableError and check it in ReadAllNeedles, VolumeTailSender, VolumeIncrementalCopy, and IncrementalBackup so replica-sync paths cannot stream or append data from an unverified .dat/.idx pair; mirror on the Rust side via read_dat_slice, read_all_needles, dat_scan_plan, and the incremental-copy handler * volume: drop issue references from comments near touched code * volume: stop active scans when the volume becomes unavailable The stream entry-point checks ran once per RPC, so a volume quarantined by a failed recovery mid-scan kept serving data. Recheck availability per needle/chunk on the detached read paths: tail scan and heartbeat, read-all, incremental copy, incremental backup writes, and the Rust StreamingBody chunk reads. Rust incremental copy also rejects a quarantined volume before sync_to_disk touches the backend. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
d848b8ed00 |
rust volume: regression test for makeup_diff replay across a 32 GiB offset boundary (#11410) (#11427)
* rust volume: test makeup_diff replay across a 32 GiB offset boundary Issue #11410 corrupted a replayed write's index offset in Go's makeupDiff by patching only four of the five offset bytes. The Rust makeup_diff already encodes the whole offset through idx_entry_to_bytes and Offset::from_actual_offset; this adds the mirror of TestConcurrentWriteCrossesOffsetBoundary so a regression would fail here the same way it does under -tags=5BytesOffset on the Go side. Sparse-truncate the .dat to 64 GiB, compact, write, commit: the index offset must equal the .cpd size and the needle must stay readable through a second vacuum. Gated on the 5bytes feature since a 64 GiB .dat exceeds the 32 GiB range of 4-byte offsets. * rust volume: skip the offset-boundary replay test on Windows Windows set_len allocates the full 64 GiB extension instead of a sparse range, so the test fails with StorageFull on CI runners. Gate it to unix, where set_len leaves the extension unallocated. |
||
|
github-actions[bot]
|
196c71b613 | docs: regenerate star history chart | ||
|
dependabot[bot]
|
559ec33498 |
build(deps): bump github/codeql-action from 4.38.0 to 4.38.1 (#11423)
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.38.0 to 4.38.1. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/v4.38.0...v4.38.1) --- updated-dependencies: - dependency-name: github/codeql-action dependency-version: 4.38.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
d6397fdc50 |
build(deps): bump dorny/paths-filter from 3 to 4 (#11422)
Bumps [dorny/paths-filter](https://github.com/dorny/paths-filter) from 3 to 4. - [Release notes](https://github.com/dorny/paths-filter/releases) - [Changelog](https://github.com/dorny/paths-filter/blob/master/CHANGELOG.md) - [Commits](https://github.com/dorny/paths-filter/compare/v3...v4) --- updated-dependencies: - dependency-name: dorny/paths-filter dependency-version: '4' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
264030c08c |
build(deps): bump docker/setup-qemu-action from 4.3.0 to 4.4.0 (#11421)
Bumps [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action) from 4.3.0 to 4.4.0. - [Release notes](https://github.com/docker/setup-qemu-action/releases) - [Commits](https://github.com/docker/setup-qemu-action/compare/v4.3.0...v4.4.0) --- updated-dependencies: - dependency-name: docker/setup-qemu-action dependency-version: 4.4.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
8099e71934 |
build(deps): bump golang.org/x/crypto from 0.56.0 to 0.57.0 (#11419)
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.56.0 to 0.57.0. - [Commits](https://github.com/golang/crypto/compare/v0.56.0...v0.57.0) --- updated-dependencies: - dependency-name: golang.org/x/crypto dependency-version: 0.57.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
63eb67c9da |
build(deps): bump github.com/aws/aws-sdk-go-v2/service/sts from 1.50.0 to 1.51.0 (#11420)
build(deps): bump github.com/aws/aws-sdk-go-v2/service/sts Bumps [github.com/aws/aws-sdk-go-v2/service/sts](https://github.com/aws/aws-sdk-go-v2) from 1.50.0 to 1.51.0. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.50.0...service/s3/v1.51.0) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/service/sts dependency-version: 1.51.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
93efc64af6 |
build(deps): bump go.uber.org/atomic from 1.11.0 to 1.12.0 (#11416)
Bumps [go.uber.org/atomic](https://github.com/uber-go/atomic) from 1.11.0 to 1.12.0. - [Release notes](https://github.com/uber-go/atomic/releases) - [Changelog](https://github.com/uber-go/atomic/blob/master/CHANGELOG.md) - [Commits](https://github.com/uber-go/atomic/compare/v1.11.0...v1.12.0) --- updated-dependencies: - dependency-name: go.uber.org/atomic dependency-version: 1.12.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
f2b08e47e5 |
build(deps): bump github.com/tarantool/go-tarantool/v3 from 3.0.1 to 3.0.2 (#11417)
build(deps): bump github.com/tarantool/go-tarantool/v3 Bumps [github.com/tarantool/go-tarantool/v3](https://github.com/tarantool/go-tarantool) from 3.0.1 to 3.0.2. - [Release notes](https://github.com/tarantool/go-tarantool/releases) - [Changelog](https://github.com/tarantool/go-tarantool/blob/master/CHANGELOG.md) - [Commits](https://github.com/tarantool/go-tarantool/compare/v3.0.1...v3.0.2) --- updated-dependencies: - dependency-name: github.com/tarantool/go-tarantool/v3 dependency-version: 3.0.2 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
5079d926e4 |
build(deps): bump cloud.google.com/go/kms from 1.33.0 to 1.34.0 (#11418)
Bumps [cloud.google.com/go/kms](https://github.com/googleapis/google-cloud-go) from 1.33.0 to 1.34.0. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/documentai/CHANGES.md) - [Commits](https://github.com/googleapis/google-cloud-go/compare/kms/v1.33.0...dlp/v1.34.0) --- updated-dependencies: - dependency-name: cloud.google.com/go/kms dependency-version: 1.34.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
github-actions[bot]
|
a9e1b57dcd | docs: regenerate star history chart | ||
|
Chris Lu
|
e2608edda4 |
volume: fix 5-byte index offset corruption in makeupDiff (#11411)
* volume: encode all offset bytes when makeupDiff replays a write makeupDiff patched only bytes 8:12 of the index entry, so under the 5BytesOffset build the fifth byte kept the old offset's high bits and the replayed needle's index pointed 32 GiB-aligned ranges away from its body. A later vacuum then dropped the entry as unreadable. Rebuild the entry with needle_map.ToBytes, the same encoder the tombstone branch just below uses. * volume: test makeupDiff replay across a 32 GiB offset boundary Sparse-file test: truncate the .dat to 64 GiB after one write, compact, write a second needle, commit, and assert the index offset matches the .compacted size and the needle stays readable through a second vacuum. Only runs under -tags=5BytesOffset. |
||
|
|
0f2ecb766f |
volume server: reject non-ASCII input instead of panicking (#11406)
* volume server: reject non-ASCII input instead of panicking
Three parsers sliced attacker-supplied strings by byte offset, so a
multi-byte character split inside itself and panicked the task:
- parse_needle_id_cookie took the last 8 bytes as the cookie and the
rest as the needle id. Reachable from VolumeServer.BatchDelete,
whose file_ids come straight off the wire as protobuf strings;
that handler already answers 400 per bad fid, so the guard turns a
panicked RPC into the error it was already written to return.
- TTL::read took the unit as the last byte and the count as
everything before it, so "?ttl=5<multi-byte>" split mid-character.
The HTTP upload path does TTL::read(..).ok() and drops an invalid
TTL; AllocateVolume maps the Err to InvalidArgument.
Both now reject non-ASCII up front. Hex and a digits-plus-unit TTL are
ASCII by definition, so no accepted input changes -- covered by tests
alongside the rejection cases.
The six response-* header overrides were inserted with
parse().unwrap(). They come from the query string, so
"?response-cache-control=%0Aevil" decodes to a value HeaderValue
rejects and the unwrap panicked the connection task,
unauthenticated. They now skip the override, matching the if-let the
chunked-response path in the same file already uses.
ReplicaPlacement::from_string was reported as a fourth site but is not
one: reaching chars[2] requires chars[0] and chars[1] to be ASCII
digits, which forces the padded string to be three single-byte
characters, so a multi-byte character always lands on a to_digit()
None first. Kept as a regression test rather than a change.
Each fix was confirmed against the unfixed code first: the parser
tests panic with "byte index N is not a char boundary", and the
integration tests panic at handlers.rs:1413 and ttl.rs:88.
Not a vector, contrary to the report: the HTTP request line. The path
is not percent-decoded before parsing, so "%C3%A9" stays ASCII and
fails the length check.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* volume server: fall back to needle MIME when response-content-type is invalid
Skipping an unparseable override left the response without any
Content-Type because the override had already bypassed the normal MIME
selection. Also correct a test comment that described a chars[2] panic
which cannot be reached.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
|
||
|
Chris Lu
|
6848cdf9e1 |
s3: close seaweedfs-quota policy-confusion gap (#11409)
* s3: count seaweedfs-quota as an operation subresource PUT /bucket?policy&seaweedfs-quota was not rejected by hasAmbiguousSubresource because operationSubresources omitted the seaweedfs-quota key. The router then picks the policy route (registered first) while the IAM action resolver may resolve the request to s3:PutBucketQuota, letting a quota-only identity write a bucket policy. Reject the combination before routing, matching the fix for policy&tagging (#10987). * s3: resolve seaweedfs-quota after other bucket subresources The quota routes are registered last among the bucket subresource routes, but the action resolver found seaweedfs-quota inside the unordered bucketQueryActions map, so a request carrying it alongside another selector could be authorized as the quota operation while the router served the earlier-registered handler. Resolve it explicitly at the end so the resolver agrees with the router, mirroring how list-type is handled. * s3: count resolver subresources in the ambiguity guard hasAmbiguousSubresource only counted operationSubresources, so adding a query parameter to the action resolver without updating that list reopened the authorize-one-serve-another gap. Count bucketQueryActions keys as operation selectors too, and add a test that walks the registered routes and fails on any query key that is neither an operation subresource nor a known modifier. |
||
|
|
ca62d4297b |
volume: load the .ecj deletion journal in chunks, and repair a torn tail (#11408)
* volume: load the .ecj deletion journal in chunks, and repair a torn tail Two independent defects in the EC deletion journal's load path. 1. The loader issued one NEEDLE_ID_SIZE-byte positional read per entry. That is fine for a healthy journal -- kilobytes -- and pathological for a large one. A `.ecj` is semantically a SET of deleted needle ids but is written as an append-only log that nothing dedupes, and several paths append a peer's ENTIRE journal onto the local one (VolumeEcShardsCopy with copy_ecj_file, EC index recovery, and ec_decode's deliberate cross-holder merge), so a volume whose shards are repeatedly balanced between two servers grows the file without bound. Observed in production: 1.51 TB and 1.30 TB on the two holders of one 10+4 volume containing ~100 distinct ids. At that size the per-entry loop is ~188e9 syscalls, run synchronously while holding the deleted_needles write lock and before the HTTP port opens. The process sits at 100% of one core with a small RSS -- the set stays tiny because the ids repeat -- reading at a few MiB/s because 8-byte reads defeat readahead, logs nothing after "Adding storage location", and ignores SIGTERM. The master then unregisters every volume it holds and reads of them fail. 4.46 and 4.47 are both affected. Read in 1 MiB chunks and build into a local set, merging once at the end so the write lock is not held for the whole scan. Measured on a 256 MiB journal of 100 distinct ids: 33,554,500 syscalls -> 257, identical resulting set. 2. A torn tail silently corrupted later deletes. The journal handle is in append mode, so writes land at the physical end regardless of alignment. A trailing partial record therefore pushed every later append out of alignment: the loader skipped the partial bytes, but the next mount decoded them together with the leading bytes of the following entry, producing one garbage id and dropping the delete that came after the tear -- after acknowledging it. Truncate to a whole number of records at mount, before anything can append. The repair uses its own read+write (non-append) handle: on Windows, append(true) requests FILE_APPEND_DATA without FILE_WRITE_DATA (and .write(true) is subsumed by .append(true)), so SetEndOfFile through the journal handle fails with ERROR_ACCESS_DENIED. The same trap exists in journal_delete's recovery path, which calls set_len on the append handle to roll back a partial write whose sync failed. It is error-handled rather than fatal, so on Windows that rollback silently does not happen. Untouched here; worth a separate fix. Bounding the journal's growth needs compaction, which is deliberately not in this change: replacing the file under a store that can hold several EcVolume instances for one volume id requires coordinating with the other holders, and that belongs at the store layer. Sent separately. Tests: a journal spanning several read chunks loads every entry; a trailing partial record is ignored rather than panicking; a torn tail is truncated at mount and a delete taken afterwards survives a remount. * volume: roll back a failed .ecj append through a dedicated write handle The append handle lacks FILE_WRITE_DATA on Windows, so the set_len rollback after a failed sync silently did nothing and the journal could drift one record past deleted_needles. Same trap as the torn-tail repair in this file; fix it the same way. Also format the new tests. * volume: mirror chunked .ecj load and torn-tail repair in Go --------- Co-authored-by: chrislusf <chrislusf@users.noreply.github.com> Co-authored-by: Devin <devin@cognition.ai> |
||
|
|
4bb40732bb |
volume server: ReceiveFile loses bytes and hides fsync failures (#11407)
* volume server: ReceiveFile loses bytes and hides fsync failures
Three defects in one handler, all on the path that receives a pushed
.dat/.idx/.vif or EC shard:
- `f.write(&content)` never compared the return to content.len().
A short write (ENOSPC, NFS) counted only the bytes that landed,
so every later chunk was written at a shifted offset and the RPC
answered error: "" with a byte count that looked right. Go's
os.File.Write loops. Now write_all.
- `let _ = f.sync_all();` discarded EIO and answered success with
the full byte count. Go omits the check too, but
ReceiveFileResponse carries an `error` field and the caller
renames the staged file into place on success -- so a silent
fsync failure publishes a file whose data never reached the
platter. Flush and fsync failures are now reported.
- Both the per-chunk write and the final fsync were blocking
std::fs calls inside the async fn, on the runtime worker that is
also driving the stream. Switched to tokio::fs + BufWriter, the
shape `drain_copy_stream_to_file` in this same file already uses
and documents. The partial-file cleanup on the error path moves
to tokio::fs::remove_file for the same reason.
The handler had no test at all, which is how the short-write bug
survived. Added a round-trip over a real connection with ragged chunk
boundaries, asserting the bytes on disk and not only the reported
count -- a dropped or reordered chunk changes the file even when
bytes_written still adds up.
That test guards the rewrite; it does not reproduce the original
faults. ENOSPC and EIO need fault injection that this suite has no
harness for, so the short-write and fsync paths are argued from the
code, not demonstrated by a failing test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* volume server: remove the staged file on every ReceiveFile error reply
Flush and fsync failures returned early and left the partial .copying or
shard file behind, as did the pre-existing write-error path. Route all
response-level errors through one cleanup block, matching Go's
close-and-remove on a failed write.
* volume server: tighten ReceiveFile comments
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <devin@cognition.ai>
|
||
|
|
8ff2e0777e |
volume server: HTTP DELETE on a distributed EC volume (#11405)
* volume server: HTTP DELETE on a distributed EC volume
The delete handler validated the cookie with EcVolume::read_ec_shard_needle,
which reads only locally-mounted shards and errors "ec shard N not available
locally" for any interval held by a peer. Every Err was mapped to 500 and no
.ecj tombstone was appended, so on a standard 10+4 spread over 14 servers an
HTTP delete of an EC needle could not succeed. The GET path already goes
through read_ec_shard_needle_distributed.
Route the delete's read through the same distributed reader. It does a
local-first pass in its snapshot phase, so the all-shards-local case costs
what it did before, and no store guard is held across the await (the reader
takes its own; RwLockReadGuard is !Send).
Two smaller corrections fall out of the new return type:
- the reader reports both "needle not in the index" and "volume vanished
between the has_ec check and the snapshot" as Ok(None), which collapses
the old Some(Ok(None)) and None arms into one 404;
- an io::ErrorKind::NotFound now answers 404 rather than 500, matching the
GET path. Telling a caller to retry a delete that can never succeed was
half the bug.
The cookie check and its ordering before the journal append are unchanged.
Not addressed here: Rust journals the tombstone locally while Go routes it to
the primary shard holder. That is a separate behaviour change and belongs in
its own PR against the same issue-10 checkbox.
The regression test mounts 13 of 14 shards, leaving out the one holding the
needle's interval. The distributed reader seeds its Reed-Solomon buffers from
locally mounted siblings, so with >= 10 survivors it reconstructs with no peer
fan-out -- which makes the bug reproducible on a single node. Against the
unfixed handler the test fails with 500 vs 202.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* volume server: fail the delete when the EC volume unmounts mid-request
find_ec_volume_mut returning None used to fall through to a 202 with no
.ecj tombstone written, reporting success for a delete that did not
happen. Answer 404 like the other volume-vanished arms so the caller can
retry after a remount.
* volume server: forward EC needle deletes to a primary-shard holder
Mirror Go's doDeleteNeedleFromAtLeastOneRemoteEcShards: the tombstone is
journaled on one holder of the needle's primary data shard via
VolumeEcBlobDelete (or the local journal when this server holds the
shard), falling back to any other shard holder when the primary has
none. Journaling only on the node that received the DELETE scattered
tombstones across whichever server took the request.
* volume server: route BatchDelete EC deletes through the same forwarding
BatchDelete had the same local-journal divergence as HTTP DELETE, plus a
gap the old code admitted in a comment: the .ecx index cannot supply the
needle's cookie, so EC deletes ran with no cookie check at all. A
distributed read now fills the needle for every EC entry — matching Go's
DeleteEcShardNeedle, which reads and compares the fid cookie even when
skip_cookie_check is set — and the tombstone forwards via
delete_ec_shard_needle_distributed. A needle deleted between read and
journal reports 304 like Go's ErrorDeleted; a vanished volume reports
500 so the filer retries.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <devin@cognition.ai>
|
||
|
Eliah RusinandClaude Fable 5.1
|
ac876eef21 |
rust volume: build outgoing gRPC clients in one place and give every dial a connect timeout (#11354)
server/grpc_client.rs stopped at build_grpc_endpoint() -> Endpoint, so all 13 production call sites hand-wrote the same .connect() + X::with_interceptor() + two max_*_message_size() lines. Four of them -- VolumeCopy, VolumeTailReceiver, VolumeEcShardsCopy and the HTTP chunk batch-delete fan-out -- dialed with no timeout at all, so an unreachable peer whose TCP handshake never completes (SYN dropped, blackholed route, host behind a silent firewall) left the operation waiting on the kernel's own retry budget, minutes long. Add GrpcDialOptions (unary / long / stream presets), connect_channel(), and volume_server_client() / master_client() / filer_client() constructors that attach the request-id interceptor and lift both message-size limits, then route all 13 sites through them. build_grpc_endpoint is private again, so connect_channel is the only way out of the module and no call site can dial without picking up a bound. Each site's existing timeouts are preserved exactly; the four bare dials gain a 5 s connect timeout and nothing else. No per-request deadline was added to any streaming call: Endpoint::timeout is a per-request bound on time-to-first-response-headers for every request the channel carries, so a value picked for one short call would also be the header deadline for the whole-volume transfer sharing the dial. The new bound covers the TCP handshake only -- tonic hands connect_timeout to HttpConnector::set_connect_timeout. A peer that completes the handshake and then stalls in the TLS or HTTP/2 exchange is still unbounded at those four sites, as are the RPCs themselves. That is why the three ping_* helpers keep their outer tokio::time::timeout: replacing it with connect_timeout would have narrowed a whole-connect bound they already had. main.rs no longer re-declares GRPC_MAX_MESSAGE_SIZE and the three keepalive/window constants; it imports them from grpc_client.rs so the inbound server and the outgoing clients cannot drift apart. Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
github-actions[bot]
|
ddf009ffe7 | docs: regenerate star history chart | ||
|
|
818f3bb71b |
rust volume: share the I/O-error tracker between Volume and EcVolume (#11351)
* rust volume: share the I/O-error tracker between Volume and EcVolume Volume and EcVolume each carried the same three fields - a mutex-held last error, a consecutive count and a sticky quarantine flag - and the same four methods over them, identical except for the path qualifier on is_storage_io_error. The tolerance the count is compared against was a fourth copy: heartbeat.rs held VOLUME_IO_ERROR_TOLERANCE for volumes, ec_volume.rs held IO_ERROR_TOLERANCE for EC, and the volume test helper open-coded the same 3, so the two paths could drift apart silently. Go keeps this in one place already: weed/storage/io_error.go holds IoErrorTracker, IoErrorTolerance and isStorageIoError, and Volume embeds the tracker. Go's EcVolume has to re-implement it only because those fields are unexported and EC lives in another package. storage::io_error::IoErrorTracker now owns that state, with record / state / should_quarantine / mark_quarantined / reset and the single IO_ERROR_TOLERANCE. is_storage_io_error moves into the same file, so it sits with the tracker that is now its only caller, the way io_error.go is laid out. Both volume kinds embed one tracker and keep their existing method names as delegates, so the ~16 internal call sites and the readers in heartbeat.rs, store.rs and grpc_server.rs change only where the two threshold comparisons become should_quarantine(). Volume::last_io_error and EcVolume::reset_io_error_state had no callers and are gone. Unchanged: what counts as a storage-media error - is_storage_io_error changed file, not body, and is still the single predicate both volume kinds share, where Go's EcVolume tests EIO directly and so misses the Windows codes. Also unchanged: the tolerance value, the metric increment on every counted error, and the sticky quarantine - a success clears the count and the last error but never the flag, which only reset lifts. In the heartbeat the state read moved inside the quarantine branch, so the common path no longer takes the tracker's mutex or clones the last-error string; should_quarantine's two relaxed loads run either way. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * rust volume: hoist absolute_display_path into server handlers.rs and ui.rs each held a byte-identical copy of the helper that turns a configured -dir into an absolute path for display. The status JSON and the status page are meant to show the same directory, so the two copies had to be edited together to stay that way. The helper now lives in server/mod.rs as pub(crate) and both callers use it. No behaviour change: same body, same call sites. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * rust volume: keep EcVolume::reset_io_error_state Moving both volume types onto the shared IoErrorTracker dropped EcVolume's public reset while Volume kept its own, so the two sides of the tracker drifted apart. mark_quarantined is sticky: a later successful read clears the error count through record(), but the quarantine flag only comes down through reset(). Without the delegate an EC volume that hit sustained media errors could not be returned to service in place once the storage was repaired. Go exposes the same method as EcVolume.ResetIoErrorState (weed/storage/erasure_coding/ec_volume.go:114). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * rust volume: name the shared tracker after Go's IoErrorTracker - check_read_write_error, get_io_error_state, mark_io_quarantined, reset_io_error_state match weed/storage/io_error.go one to one - io_error module is pub(crate) like the io module beside it - restore EcVolume::reset_io_error_state so both volume kinds expose the same recovery surface - trim comments that restate the code Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
7643f4f541 |
rust: a seaweed-common crate for the address and TLS helpers both crates carry (#11358)
* rust: a seaweed-common crate for the address and TLS helpers both crates carry seaweed-volume and seaweed-worker are separate cargo trees with separate lockfiles and no root manifest, so anything both of them need has had to be written twice. Two of those copies are a correctness risk rather than a typing cost, and this crate is where they stop being copies. address.rs is the HTTP<->gRPC port rule: `host:port` means gRPC on port+10000, `host:port.grpcPort` names it outright. The two copies had already drifted — the worker's bracketed IPv6 literals, the volume server's did not — so the rule lives here once, returning a typed AddressError whose Display text is the volume server's original wording, with join_host_port public beside it. A test asserts two of those messages in full rather than by substring, because the wording is the contract its callers hand to a Status or an io::Error; the other three end in a std ParseIntError message, which is std's to reword. The enum is #[non_exhaustive] so a future variant is not a breaking change for either consumer. The tests are both crates' cases together, plus the IPv6, already-bracketed and normalisation cases neither copy covered on its own. tls.rs is install_default_crypto_provider. Both binaries link aws-lc-rs and ring transitively, so rustls cannot auto-select and tonic's client TLS panics on first use; each binary has to pin one and it has to be the same one, which is exactly the kind of choice that should not exist twice. It is safe to share because `cargo tree -i rustls` resolves a single rustls in each tree (0.23.37 in seaweed-volume, 0.23.43 in seaweed-worker) and cargo unifies all semver-compatible `rustls = "0.23"` requirements into one crate per binary, so this crate writes the same process-wide static its consumer reads. rustls is already in both graphs — directly in the volume server, through tonic's tls-aws-lc in seaweed-worker-core — so the dependency adds no crate to either. rust-version is 1.91.1, the lower of the two consumers' floors, so depending on this crate cannot raise either tree's MSRV; verified with `cargo +1.91.1 check --all-targets`. The lockfile is committed even though this is a library: CI builds it directly, so a committed lock is what makes those runs reproducible and their caches stable. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * rust: take the address and TLS helpers from seaweed-common Both public signatures are kept, so no caller outside the two wrapper files changes. parse_grpc_address stays `Result<String, String>` and maps the typed error through Display; server_to_grpc_address stays `Option<String>` and drops it with .ok(). Their doc comments and the volume server's 13 call sites are otherwise untouched. Three behaviours change, each in the direction of the copy that was already right: - The volume server now brackets IPv6 literals. `::1:19333` used to come back as `::1:29333`, which build_grpc_endpoint rejects with "invalid gRPC endpoint http://::1:19333: invalid authority" — an IPv6 master or EC peer could not be dialled at all. Two tests in grpc_client.rs pin it, one on the string and one on the endpoint the string builds. - The volume server now emits the *parsed* gRPC port of the dotted form instead of the original text it had just validated, so `host:8080.018080` and `host:8080.+18080` come back as `host:18080` rather than as authorities the URI parser rejects. Same port either way; only malformed spellings change. - The worker's dotted form now validates the HTTP port it discards. `server_to_grpc_address("host:abc.18080")` used to answer Some("host:18080"); it now answers None, which is what the volume server's copy has always done. install_default_crypto_provider becomes a re-export in both trees, so `crate::security::tls::install_default_crypto_provider` and `weed_lance_worker::tls::install_default_crypto_provider` still resolve. The lance crate's `rustls = "0.23"` was its only direct use of rustls and goes away with the body; seaweed-common states the same requirement, so neither the resolved version nor the enabled features move in either lockfile. The PEM test fixtures stay where they are. The two tests that use them are not duplicates: the volume server's exercises build_grpc_endpoint, and the lance one exists precisely because aws-lc-rs and ring are both linked in that crate's graph. Only the literals are shared, and exporting test fixtures from a library to dedupe two constants costs more than it saves. A path dependency outside both trees means every build context that copies one crate directory has to copy the other. The repo has one: the Rust source-build stage of docker/Dockerfile.go_build, which now copies seaweed-common beside seaweed-volume. Every workflow whose `paths:` filter keys on a crate directory gains `seaweed-common/**` — the two Rust test workflows, rust_binaries_dev, container_dev and performance. The tag- and dispatch-triggered ones (rust_binaries_release, container_release_unified, container_latest) have no `paths:` filter and need nothing. The two Rust test workflows also run `cargo test` in seaweed-common, from their unit-test job, because a path dependency is not a workspace member and neither tree's own `cargo test` reaches it. Each step builds into its job's cached target directory, and both cache keys now hash seaweed-common/Cargo.lock as well so a change there invalidates the cache it would otherwise silently reuse. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docker: keep go_build working for BRANCH revisions without seaweed-common The rust_builder stage copies seaweed-common unconditionally now that seaweed-volume path-depends on it, but BRANCH can name any revision — including ones that predate the crate. Create the directory in the builder stage so the COPY always has a source; an empty dir beside an old seaweed-volume is harmless. --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
|
f1ed270942 |
rust volume: one S3 tier registry instead of two kept in sync by hand (#11357)
* rust volume: one S3 tier registry instead of two kept in sync by hand `VolumeServerState.s3_tier_registry` and `global_s3_tier_registry()` held the same S3 tier backends. `apply_storage_backends` — the only production writer — registered every backend into both, and each half of the tiering code then read a different one: the gRPC tier-move handlers resolved the backend from the per-server field, while `Volume`'s remote mount and destroy paths resolved it from the global registry, because a `Volume` has no handle to the server state. Two registries that must agree, kept in agreement by a duplicated `register_s3_backend` call and a comment in a test constructor explaining the hand-sync. Delete the field and let both tier-move handlers resolve from the global registry, so `apply_storage_backends` registers once and no longer needs the server state at all. Injecting a registry handle through `VolumeSpec` instead was considered and rejected here: it would touch every `Volume` constructor for no functional gain, and the process-wide registry is what `Volume` already uses. Behaviour is unchanged: the same names were registered in both registries, so every lookup resolves exactly as before. The tier-down test now registers its backend only in the global registry — before this change it fails with `remote storage s3.tier_down_delete not found from supported: []`. The tier-up handler had no test at all, so it gets a cheap probe: register a backend only in the global registry, ask for that destination, and check the call gets past the lookup — the response is dropped straight away, so the transfer sees a departed caller and never opens a connection. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * rust volume: await the tier-up probe terminal error instead of racing it Dropping the response left it to chance whether the detached transfer saw the closed channel before its initial check; if it won that race it went on to attempt the multipart upload with no one waiting on the outcome. Hold the stream and read until the dead endpoint fails the upload — the terminal error proves the task ran and finished, so no background network work outlives the test. --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Eliah RusinandClaude Fable 5.1
|
7dbbdac030 |
rust volume: keep the EC shard-location map, its refresh time and stale mark under one lock (#11356)
The per-EcVolume shard-location cache was three fields under three locks: an RwLock<HashMap> for the map, a Mutex<Option<Instant>> for the time it was last refreshed, and a Mutex<bool> for the stale mark. Nothing tied them together. merge_shard_locations published the merged map, released the write lock, and only then stamped the refresh time; both readers (scrub_ec_volume_distributed's snapshot and build_snapshot) took the two guards one after the other. A reader landing between the two writes paired a freshly merged map with the previous lookup's timestamp -- and that pair is exactly what needs_refresh judges, so a read went back to the master for a map that had just been refreshed. Go keeps the same state in one struct behind one ShardLocationsLock. replace_shard_locations documented itself as "a single observable step" while being two. Fold the three fields into one ShardLocationCache behind a single RwLock. merge_shard_locations upserts and stamps in one write section, shard_locations_snapshot returns the map and its time from one read section, and mark_shard_locations_stale / claim_shard_locations_refresh move the mark's read-and-consume onto the cache. The three zero-caller accessors -- set_shard_locations, replace_shard_locations, get_shard_locations -- are deleted, and the field is now private, so the invariant cannot be sidestepped from outside the module. The two test seeding sites go through merge_shard_locations, which already produces the state they were writing by hand. Unchanged: the freshness rule. needs_refresh keeps its thresholds and still judges the caller's snapshot -- the map that caller will actually read from, not whatever is cached by the time the claim runs -- so only the stale mark is read from under the new lock. The master lookup, the completeness guard in write_back_shard_locations and the per-shard upsert semantics are untouched. Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
|
6d676eda67 |
rust volume: typed errors for store compaction so gRPC can answer NotFound (#11355)
* rust volume: typed errors for store compaction so gRPC can answer NotFound The vacuum entry points on `Store` returned `Result<_, String>`, so the gRPC layer had nothing to branch on and answered `Status::internal` for every failure. A vacuum loop that races a volume being moved or deleted saw the same code as a disk going bad, and `weed shell` could only tell the two apart by matching on the message text. `VolumeError` gains `VolumeNotFound(VolumeId)` — the existing `NotFound` is needle-level and carries no payload — and `InsufficientSpace`, and `compact_volume`, `commit_compact_volume`, `cleanup_compact_volume` and `delete_collection` return it. `impl From<VolumeError> for tonic::Status` in `server/mod.rs` maps not-found to `not_found`, read-only to `failed_precondition`, insufficient space to `resource_exhausted`, already-exists to `already_exists`, and everything else to `internal`; the four RPCs prefix their own context with `status_with_context`, so a message reads "commit compact volume 7: volume id 7 is not found". The store-side "during compact" / "during commit compact" / "during cleaning up" suffixes are gone, and the free-space message drops the volume id the prefix already supplies. `check_compact_volume` had no callers — `VacuumVolumeCheck` computes the garbage level from its own `find_volume` — and is deleted. `compact_volume` folded the size estimate into its first lookup, dropping the `unwrap()` re-lookup that only existed to dodge a borrow. `ascending_visit` on `CompactNeedleMap`, `RedbNeedleMap`, `SortedFileNeedleMap` and the `NeedleMap` dispatch is now generic over the visitor's error type, like `CompactMap::ascending_visit` already was. The three signatures that can fail on their own bound `E: From<String>` to carry those failures; the in-memory walk in `iter_entries` names `Infallible`, which says in the type what its comment used to say in prose. No Go shell command matches on the old error text: the strings exist only in weed/storage/store_vacuum.go. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * volume server: trim comments and answer the same codes from Go - vacuum_volume_check reports VolumeError::VolumeNotFound like the other vacuum RPCs instead of its own "not found volume id" wording - drop doc comments that restate what the code says - Go volume server wraps ErrVolumeNotFound/ErrInsufficientSpace from store_vacuum.go so VacuumVolumeCheck/Compact/Commit/Cleanup and DeleteCollection answer NotFound/ResourceExhausted, matching the Rust volume server; volumeDeleteStatusError generalized to volumeStatusError Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume server: prefix operation context on vacuum errors Lower-level errors forwarded by CompactVolume, CommitCompactVolume, CommitCleanupVolume and DeleteCollection carry no volume id or operation name. Wrap with %w so the status mapping still sees the sentinel chain, matching the context the Rust server's status_with_context adds. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume server: map NotEmpty to FailedPrecondition, share mapper in VolumeDelete Go's volumeStatusError maps ErrVolumeNotEmpty to FailedPrecondition; the Rust Status conversion was missing it and volume_delete kept a hand-rolled match. Route it through status_with_context like the vacuum handlers. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Eliah RusinandClaude Fable 5.1
|
d002481037 |
rust volume: derive has_remote_file instead of mirroring it (#11353)
Volume carried `pub has_remote_file: bool` next to `pub volume_info`, and the bool was only ever the answer to `!volume_info.files.is_empty()`: outside the two constructors, `refresh_remote_write_mode` was the single writer. Both fields being public made the pair a convention rather than an invariant. Every caller that touched `volume_info.files` — load_vif twice, the tier-up handler, the tier-down handler and its rollback — had to remember to call `refresh_remote_write_mode` afterwards, and a caller that forgot would leave the volume advertising a write mode its .vif contradicts, or serving a remote .dat through a writable needle map. The bool becomes `has_remote_file()`, computed from the list, so it cannot drift. `volume_info` becomes private with a `volume_info()` reader, and edits to the reference list go through `update_remote_files(|files| ...)`, which applies the closure and then refreshes the derived write mode and the needle map. With no caller left outside the module, `refresh_remote_write_mode` is private. Unchanged: the refresh logic itself, the order of operations in both tier handlers, and the tier-down rollback semantics. The rollback still snapshots the removed reference before the refresh runs, restores it on failure, and re-refreshes unconditionally on the error path — the second `update_remote_files` call runs with a no-op closure when there was nothing to restore, exactly as the old code re-ran the refresh whether or not it had re-inserted a reference. Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Eliah RusinandClaude Fable 5.1
|
1df8c05bc3 |
rust volume: document every unsafe block and stop mutating the process env in tests (#11352)
Three production `unsafe` blocks carried no `// SAFETY:` comment at all (`libc::fallocate`, `libc::sysinfo`, `libc::statvfs`), and nothing made that an error: `clippy::undocumented_unsafe_blocks` is a `restriction` lint, allow-by-default, and appeared nowhere in either crate. Turn it on in `seaweed-volume`'s `[lints.clippy]` and in the worker workspace's `[workspace.lints.clippy]`, then document what each block relies on. `memory_status.rs` and `disk_location.rs` get their blocks narrowed to the `zeroed()` and the libc call, so each comment sits next to the operation it justifies and the arithmetic is outside the block. Both turn the success test into an early return on failure; the casts, the multiplication order and the values returned on either path are unchanged. The bigger problem was in `config.rs`'s tests. `with_temp_env_var` and `with_cleared_security_env` called `std::env::set_var`/`remove_var`, claiming soundness because every caller holds `process_state_lock()`. That mutex only serialises the fourteen annotated tests in this module. The same lib test binary runs the `grpc_server.rs` tests, which bind a `TcpListener`, dial loopback and drive a multi-thread tokio runtime, and tonic/hyper/rustls/aws-sdk all read the environment lazily on those threads — which is exactly the race Rust 2024 made these calls unsafe for. `restore_env_var` had no SAFETY comment at all. `#[serial]` would not have helped: it serialises annotated tests, which the mutex already did. So the config layer no longer reads the environment implicitly. An `EnvLookup<'a> = &'a dyn Fn(&str) -> Option<OsString>` is threaded from the public entry points down to every reader — `HOME`, `USERPROFILE`, the twenty-four `WEED_*` keys and `SEAWEED_WRITE_QUEUE`. `parse_cli` and `parse_security_config` keep their signatures and pass `process_env`, a thin wrapper over `std::env::var_os`; `resolve_config` becomes `resolve_config_with_env` (private, one caller). Tests build one with `fake_env` instead, so no test touches the real environment and every `unsafe` in the module is gone. `process_state_lock()` stays, with a smaller job: `set_current_dir` is safe but still process-global, so the tests that move the working directory are still serialised against the ones that read it. Tests naming an explicit config file never reach that search and no longer take the lock. No production behaviour changes: the same keys are read in the same order with the same precedence, and `env_string` reproduces `std::env::var(key).ok()` — absent and non-UTF-8 both read as unset. Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Chris Lu
|
44ba070d83 |
helm: document admin.allowInsecureBind and drop stale comment
- README and values.yaml now describe the allowInsecureBind escape hatch alongside the non-loopback bind guard - remove a PR reference from the CI test comment |
||
|
Yongun SeongandClaude Sonnet 5
|
062238bb5c |
helm: let admin.allowInsecureBind satisfy the admin auth render guard
PR #11185 made `weed admin` refuse to bind a non-loopback address without -adminPassword or mTLS. PR #11228 added -allowInsecureNoAuth as an explicit opt-out for operators who restrict admin access some other way (e.g. a NetworkPolicy plus an authenticating reverse proxy). The chart's render-time guard added by #11236 (admin-statefulset.yaml, seaweedfs.admin.authEnabled) predates -allowInsecureNoAuth and only recognizes password-based auth, so there was no values.yaml path to express that choice: the chart would fail(...) even though the binary itself would start fine with a warning. Add admin.allowInsecureBind (default false) to the seaweedfs.admin.authEnabled helper's checks; when true it renders -allowInsecureNoAuth on the admin command and satisfies the render guard alongside the existing password-based checks. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
Chris Lu
|
b77c42ff32 | test: use BaseEndpoint for the checksum test client | ||
|
Chris Lu
|
26fc90187e |
s3: accept x-amz-checksum-mode from the query string, case-insensitively
Presigned HeadObject/GetObject requests hoist x-amz-checksum-mode into the signed query string, so a strict header-only check would withhold stored checksums on presigned reads that AWS honors. |
||
|
Chris Lu
|
ef463fe1af |
s3: read complete-request checksum values from headers or query
Presigned CompleteMultipartUpload requests hoist x-amz-checksum-type and the full-object checksum header into the signed query string, so a header-only lookup would skip BadDigest validation for them. |
||
|
Chris Lu
|
17ad5a1419 |
s3: accept FULL_OBJECT checksums without per-part checksums at complete
COMPOSITE uploads must still carry every part checksum in the complete request, but FULL_OBJECT uploads may instead supply the whole-object checksum in an x-amz-checksum-* request header. Compare that header against the computed object checksum and return BadDigest on mismatch, matching AWS. |
||
|
Chris Lu
|
b03419ee92 |
s3: reject UploadPart checksum algorithms conflicting with the upload
An UploadPart that explicitly selects a different checksum algorithm than the one declared at CreateMultipartUpload would store a checksum CompleteMultipartUpload could never accept. Reject the conflict up front with InvalidRequest, matching AWS. |
||
|
Chris Lu
|
f849b7c823 |
s3: validate per-part checksums in CompleteMultipartUpload
Parse the Checksum* elements of each completed part and enforce what AWS does for uploads created with x-amz-checksum-algorithm: every part must carry a checksum in the complete request (InvalidRequest when missing, BadDigest when it differs from the stored part checksum), and an x-amz-checksum-type header must match the upload resolved checksum type (BadDigest). Add the issue-11401 reproduction as a regression test. |
||
|
Chris Lu
|
f15b980976 |
s3: UploadPart inherits the checksum algorithm of its multipart upload
AWS computes a checksum for every part of an upload created with x-amz-checksum-algorithm, even when the part request carries no checksum headers. Mirror that: when the part request specifies no algorithm, apply the one stored on the upload entry so the part entry keeps a checksum CompleteMultipartUpload can fold into the object checksum. |
||
|
hsdfat
|
110b485bae |
fix(volume): stop ScanVolumeFileFrom at a header it cannot advance past (#11398)
fix(volume): stop scans at a header they cannot advance past A corrupt .dat header with a very negative size gives a record length (NeedleHeaderSize + NeedleBodyLength) of zero or less: v3 sizes -43..-36 and v2 sizes -35..-28 give exactly zero, and smaller sizes give a negative length. ScanVolumeFileFrom advanced by that length, so it re-read the same header forever or stepped back into the record before it. weed fix, weed export, weed compact, incremental weed backup and the tail sender behind volume.move and volume.merge could hang on such a volume, and weed compact could also finish with a .cpx that had dropped every needle after the header. Return an error wrapping needle.ErrorCorrupted instead. The check runs after the visitor has seen the record, so the rebuild scanner still stops quietly with io.EOF. Smaller negative sizes whose record length is positive are still stepped over, preserving the salvage behavior compaction relies on. Mirror the guard into the Rust volume scans: DatScanPlan::scan and read_all_needles fail on a non-positive record length, as does scan_dat_head, so a corrupt header cannot stall a tail pass or leave the repair scan walking stale offsets. |
||
|
hsdfatandChris Lu
|
06dda12e4b |
fix(volume): validate sizes in ReadNeedleBlob and WriteNeedleBlob (#11399)
* fix(volume): reject negative sizes in ReadNeedleBlob and WriteNeedleBlob A ReadNeedleBlob RPC with a size of -44 or below (-36 on v2 volumes) panics in makeslice inside needle.ReadNeedleBlob. The volume gRPC server has no recovery interceptor, so one request kills the process. Smaller negative sizes return bytes that are not a record. WriteNeedleBlob accepted a negative size whenever the blob header carried the same value: it appended the blob to .dat and indexed the needle with that size, which reads as deleted. Reject size < 0 in both Volume methods. Size 0 still passes, since delete records carry it. The Rust volume server got the same storage guards in #11345. * fix(volume): reject needle blobs whose length does not match their size WriteNeedleBlob appends the blob as is. A blob that is not the length its size implies leaves .dat off the 8-byte grid, and every later ordinary write to the volume is indexed at a truncated offset and reads back as EOF. A blob off by 8 bytes keeps the grid but leaves bytes that a .dat scan reads as the next record. The in-tree callers already send exact lengths. The one case this newly refuses is a copy between volumes of different needle versions, and that case already writes a broken record: a v3 record lands on a v2 volume with 8 extra bytes, and a v2 record on a v3 volume either fails the timestamp check or lands 8 bytes short. This is separate from the negative-size guards, whose Rust counterpart is #11345. The Rust server does not check the length yet. * fix(volume): guard the blob buffer allocation in needle.ReadNeedleBlob Volume.ReadNeedleBlob rejected negative sizes, but needle.ReadNeedleBlob still sized its buffer from the size and is called directly by vacuum and other paths. Reject a deletion marker before make() there too, and use size.IsDeleted() in the volume-level checks. * fix(volume): mirror the blob length check in the rust volume server write_needle_blob_and_index checked the size against the blob header but appended the blob verbatim, so a blob that is not the length its size implies still leaves .dat off the record grid. Match the Go check. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
hsdfatandChris Lu
|
a93a1ab2eb |
fix(volume): return an error instead of 201 when a write lands on no volume (#11397)
* fix(volume): return an error instead of 201 when a write lands on no volume ReplicatedWrite only writes locally when this server holds the volume. For a volume id no server holds, the master lookup returns no locations, so the write went nowhere and the upload still got 201 Created. The same happened for a type=replicate write to a server without the volume, so the primary, or the S3 chunk fan-out, counted a replica that was never written. A server without the volume still forwards the write to the replicas the master lists. When there is nothing to forward to, fail with "volume N not found on host:port". PostHandler returns that as 500, the status the Rust volume server already returns here, and uploaders re-assign on 5xx. Fixes #6609 * volume: reuse Store.HasVolume, drop issue ref from test comment --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
ssshr-66andChris Lu
|
cd1e738422 |
[Volume] Scrub local deletion tombstones during FULL scrub (#11396)
* fix 11388 * fix(volume): scrub validates local deletion tombstones TombstoneFileSize (-1) is an .idx-only sentinel; the physical record it points at carries a zero-sized body. Normalize deleted index sizes to 0 via onDiskSize before computing disk usage and calling ReadData, so corrupted or truncated tombstone records are detected instead of skipped. Offset-zero entries (remote logical deletes, no .dat record) remain skipped, and the physical needle id is checked against the index key. Mirror the behavior in the Rust volume server. * fix(volume): scrub preserves physical size of deleted non-tombstone entries Size.Raw()/raw() already encodes the index-to-disk mapping: tombstone (-1) -> 0, other negative sizes -> their absolute value (the offset then points at the original record, per the ReadDeleted path). Use it instead of mapping every deleted size to 0. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
github-actions[bot]
|
3dec359d6b | docs: regenerate star history chart | ||
|
Chris Lu
|
f6a3286b32 |
fix(volume): derive needle body tail bound from the version layout (#11395)
* fix(volume): derive needle body tail bound from the version layout The size guard in ReadNeedleBodyBytes computed the tail length as checksum, plus timestamp only for Version3. Forks and future on-disk formats whose tail carries more fields would silently under-check and still panic in readNeedleTail on a truncated body. Derive the tail from NeedleBodyLength minus data and padding so the bound stays exact for every version. Iterate IsSupportedVersion in the new tests instead of hardcoding v1-v3 so downstream formats get covered automatically, and skip versions the build cannot write rather than failing on them. * test: skip needle write only on the unsupported-version error A blanket skip would hide a real writer regression. Skip the version subtest only when the writer reports the version is not supported in this build (the error text differs between builds), and fail on any other write error. |
||
|
Chris Lu
|
01bb3b3053 |
s3api: add Snowflake s3compat API integration tests (#11394)
* s3api: add Snowflake s3compat API integration tests Run the upstream snowflakedb/snowflake-s3compat-api-test-suite against a local SeaweedFS server in CI. test/s3/snowflake/run.sh starts weed server with S3 (-s3.autoCreateBucket=false so missing-bucket PUTs return NoSuchBucket), prepares the fixtures the suite needs (versioned bucket, deny-all-policy bucket, >1000-object prefix), clones the suite, patches it to path-style addressing, and runs mvn -Dtest=S3CompatApiTest. The suite also exposed that GetBucketLocation returned 404 NoSuchBucket for a malformed bucket name; validate the name first and return 400 InvalidBucketName like AWS. * test: harden snowflake s3compat runner per review - Pin the upstream suite to a tested commit (SUITE_REV) instead of the moving default branch - Bind the test server to loopback only - Require the AccessDenied error code when verifying the denied bucket - Fix README so go install runs in a subshell - checkout with persist-credentials: false - Make the concurrency group unique per PR, and widen path filters to the storage/operation/wdclient/cluster/pb packages the S3 stack uses * test: advertise loopback ip for snowflake test server -ip.bind 127.0.0.1 alone left the volume server advertising the host's primary address, so chunk uploads were refused. Also set -ip 127.0.0.1 and disable the Iceberg/Lance listeners so the harness is loopback-only and does not collide with other local services. |
||
|
hsdfat
|
5769057af3 |
fix(volume): return an error instead of panicking on a corrupt needle size (#11393)
ReadNeedleBodyBytes sliced the needle body with the size from the needle header without checking it. A corrupted .dat header carrying size -1 still gets a positive body length (16 bytes on v3), so vacuum compaction read that body and panicked with "slice bounds out of range [:-1]". Writers never put a negative size in a .dat header: a delete appends a size-0 record, and TombstoneFileSize only lives in the .idx. Reject a size that is negative or leaves no room for the checksum/timestamp tail with an error wrapping ErrorCorrupted. ScanVolumeFileFrom already logs body read errors and moves on, so compaction now skips the record like any other corrupt needle. Fixes #6763 |
||
|
github-actions[bot]
|
4160b92864 | docs: regenerate star history chart | ||
|
Chris Lu
|
37bf1cd91d |
volume: validate copy/tail source addresses before dialing (#11390)
* pb: stop exiting the process on malformed server addresses ServerToGrpcAddress and GrpcAddressToServerAddress called glog.Fatalf when hostAndPort could not parse the port, which os.Exit(255)ed the whole process. A caller-supplied copy or tail source address reached this path synchronously in the serving goroutine, so one anonymous VolumeCopy with a non-numeric port terminated the volume server. Log the parse error and return the input unchanged instead: the dial or request that consumes the address then fails as an ordinary error. * volume: validate copy and tail source addresses before dialing VolumeCopy, VolumeEcShardsCopy and VolumeTailReceiver dial a caller-supplied source address (SourceDataNode / SourceVolumeServer) with no endpoint validation, so an anonymous caller could aim the volume server at loopback, link-local (cloud metadata) or other unintended destinations and read dial behavior back as a connectivity oracle. Apply the same peer-target deny list FetchAndWriteNeedle uses for replica targets: the source must be a bare host:port whose host is not loopback, link-local or unspecified; cluster peers stay reachable on private networks, and -volume.allowUntrustedRemoteEndpoints opts out. The loopback-using copy tests set the flag to keep exercising the copy path in process. * rust volume: validate copy and tail source addresses before dialing Mirror the Go guard on the Rust volume server: volume_copy, volume_ec_shards_copy and volume_tail_receiver dial a caller-supplied source address, so run it through validate_replica_target first (bare host:port; no loopback, link-local or unspecified hosts; private peers stay allowed). --volume.allowUntrustedRemoteEndpoints opts out; the test fixture and the Rust test-cluster launcher set it so loopback sources in tests keep working. * volume: pin validated copy/tail source addresses at dial time validateReplicaTarget resolves the source hostname once, but the gRPC client resolved it again at connect, leaving a DNS-rebinding window for hostname sources. The copy and tail source dials now run through the same guardedDialerPolicy the remote-storage path uses, so every resolved address is re-checked against the replica deny list (private peers allowed) immediately before the TCP connect. guardedDialerPolicy also moves to util.OutboundDialContext so the guarded path keeps the -ip.bind source binding the default gRPC dialer had. The Rust volume server mirrors this with connect_guarded, a tonic connector that resolves, re-checks each address, and connects to the first passing IP; handlers use it whenever the untrusted-endpoint opt-out is off. A handler-level test now exercises the enabled validation branches for all three source-taking RPCs. * pb: return empty server address for malformed grpc addresses GrpcAddressToServerAddress used to return the unparseable input on a hostAndPort failure, so a malformed raft address (e.g. "host:abc") flowed into admin dashboard master maps unchanged. Return an empty string instead, skip empty conversions at the two raft-cluster merge sites, and drop the now-stale comment about the fatal exit the earlier commit removed. * test: opt erasure-coding loopback clusters out of the remote endpoint guard The erasure-coding suites drive VolumeEcShardsCopy / VolumeCopy between volume servers bound to 127.0.0.1, which the copy/tail source guard now rejects by default. Pass -volume.allowUntrustedRemoteEndpoints to the test volume launches, matching what the volume_server framework harnesses already do. * admin: only claim fallback master leadership on an empty raft response A nonempty RaftListClusterServers response whose entries were all rejected left masterMap empty, so the fallback marked the reachable current master as leader the same way a genuinely empty (non-raft) response does. Track whether the successful response returned zero servers and only promote the fallback master then. |