mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-08 15:41:15 +02:00
361fd6b263d86d7d7bae0f53210eb4af49bf0b60
15085
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 ssshr-66andChris Lu
|
361fd6b263 |
[Filer] Parallelize Chunk Manifest Resolution to Reduce Large File Read Latency (#11215)
* fix issue-11214 * fix(filer): cancel sibling manifest reads on failure * fix(filer): scope manifest cancellation to read batch and propagate context to encrypted reads Address PR review comments on #11215: - Scope cancellation to each parallel read batch instead of the resolver-wide context, so a later manifest failure does not cancel recursive work for an earlier successful manifest (CodeRabbit #3952446554). - Propagate the resolver context through GetAuthenticatedWithContext so encrypted sibling reads observe cancellation and stop promptly when another manifest fails (Greptile #3952422343). - Use net.ListenConfig.Listen with an explicit context in the test fixture to satisfy the noctx linter (CodeRabbit #3952111696). - Add regression tests for encrypted sibling cancellation and for preserving earlier manifest children on later failure. * fix(filer): return known manifest errors without blocking on earlier children Address Greptile review comment on #11215: After all parallel reads complete, pre-scan slots for the first real (non-internal-cancel) error before recursing into earlier manifests' children. If a later manifest already failed, return its error promptly with data chunks already in hand, instead of blocking on recursive network reads of earlier manifests' children. Updated the regression test to verify the error returns within 1 second when an earlier manifest's child has a 2-second delay, and that the child is never loaded. * fix(filer): filter partial child chunks by requested range on error path Address CodeRabbit review comment on #11215: The pre-scan error path appended non-manifest child chunks from earlier manifests without applying the [startOffset, stopOffset) overlap check used for top-level chunks. A child outside the requested range could be returned in dataChunks alongside the later manifest's error. Apply the same range predicate before appending. Add regression test with an out-of-range child chunk. * fix(filer): buffer job channel and abort submission on batch cancellation Address Greptile review comment on #11215: - Use a buffered job channel (capacity 128) so submission does not block when all workers are busy. This ensures a promptly-failing manifest is always queued and can cancel stalled sibling reads once a worker picks it up, instead of blocking the caller on the unbuffered channel send. - Add batchCtx.Done() to the submit select so submission aborts promptly when the batch is already cancelled by a sibling failure. - Add regression test with 5 manifests (4 stalled + 1 failing) verifying the failing job is queued and picked up after a stalled worker is freed. * fix(filer): avoid double WaitGroup decrement on batch cancellation in submit Address Devin review comment on #11215: When batchCtx.Done() fired in submit, it called job.done.Done() and returned false. The caller in resolve also called reads.Done() on the same WaitGroup, causing a double decrement that would panic with a negative counter. Fix: submit sets the result error but does not decrement the WaitGroup. The caller always owns the decrement and skips overwriting the result when submit already set it. * fix(filer): overflow execution when job queue buffer is full Address Greptile follow-up review comment on #11215: With a 128-entry buffer, if more than 132 in-range manifests (4 workers + 128 buffer) stall at one level, a promptly-failing manifest beyond the buffer cannot be submitted and cannot cancel the stalled reads. Fix: when the buffer is full, run the job directly in a goroutine instead of blocking on the channel send. This only triggers for >132 manifests at one level (exceedingly rare), so the bounded concurrency guarantee (4 workers) holds for all normal workloads. Extracted executeJob method shared by both workers and overflow goroutines. * fix(filer): bound overflow execution with a semaphore Address Devin review comment on #11215: The unbounded overflow goroutines could create thousands of concurrent reads for large files, defeating the four-worker resource bound. Fix: add a semaphore (capacity = maxChunkManifestResolveWorkers) that overflow goroutines must acquire before doing the read. While waiting for the semaphore, they also watch batchCtx and r.ctx so they exit promptly on cancellation. Total concurrency is now bounded to 2 * workers (4 workers + 4 overflow) in the degenerate case. * refactor(http): add ctx to GetAuthenticated signature instead of new function Reuse the existing GetAuthenticated name by adding ctx as the first parameter, matching the pattern of ReadUrl, ReadUrlAsStream, and RetriedFetchChunkData. Removes the GetAuthenticatedWithContext wrapper. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
75ec5ec193 |
admin: allow setting volume read-only and read/write modes (#11217)
* admin: support setting volume read-only and read/write modes
* admin: address PR review on volume access-mode persistence
Reject trailing JSON values in the SetVolumeReadOnly handler so
requests like {"read_only":true}{} no longer pass validation, and add
a trailing-value case to the invalid-request test.
Propagate .vif persistence failures through the access-mode chain.
PersistReadOnly now returns the SaveVolumeInfo error and rolls back
the in-memory volumeInfo on failure; Store.MarkVolumeReadonly and
Store.MarkVolumeWritable propagate that error and roll back their
noWrite flags, so the API reports failure instead of success while
restart would revert the mode.
* admin: make .vif persistence atomic and preserve error chain
SaveVolumeInfo now writes to a .vif.tmp file, syncs it, renames it
over the target, and fsyncs the directory. A write/sync/close failure
leaves the existing .vif intact, so the PersistReadOnly in-memory
rollback matches the durable state instead of diverging from a
partially written file that restart would apply.
Switch the error wrappers in PersistReadOnly, MarkVolumeReadonly, and
MarkVolumeWritable from %v to %w so callers can use errors.Is and
errors.As to classify persistence failures.
* admin: treat post-rename dir fsync failure as a warning
After os.Rename commits the new .vif, the on-disk file already holds
the requested mode. A directory fsync failure only risks losing the
rename across a crash; returning an error here would make
PersistReadOnly roll back in-memory state while the durable file keeps
the new mode, splitting the replica. Log the failure as a warning
instead, matching the best-effort nature of FsyncDir (already skipped
on Windows).
* admin: distinguish post-rename durability failures and use unique temp files
SaveVolumeInfo now uses os.CreateTemp for the staging file, preventing
concurrent saves for the same volume from colliding on a shared .tmp
path.
A directory fsync failure after os.Rename returns a
NotCrashDurableError instead of being silently swallowed. The rename
already committed the new metadata to disk, so PersistReadOnly,
MarkVolumeReadonly, and MarkVolumeWritable skip the in-memory rollback
for this error type (keeping state aligned with the durable file) while
still propagating the failure to the API. Pre-commit failures continue
to roll back as before.
* admin: continue post-commit work after NotCrashDurableError
MarkVolumeWritable now clears the EIO quarantine and the gRPC handlers
(makeVolumeReadonly step 3, makeVolumeWritable master notification)
proceed with their post-commit work when SaveVolumeInfo returns a
NotCrashDurableError, instead of aborting and leaving the volume
unavailable or the master unaware of the mode change. The durability
warning is still propagated to the API caller. Pre-commit failures
continue to abort early as before.
* admin: handle NotCrashDurableError in tier and EC callers
VolumeTierMoveDatFromRemote and VolumeEcShardsGenerate now check for
NotCrashDurableError from SaveVolumeInfo. When the rename has already
committed the new .vif, they continue with their post-commit work
(backend switch, remote deletion, keeping generated EC shards) instead
of aborting and leaving the on-disk metadata inconsistent with the
file layout. The durability warning is logged for the operator.
|
||
|
github-actions[bot]
|
53a18ecadd | docs: regenerate star history chart | ||
|
dependabot[bot]andChris Lu
|
2d4b730a2f |
build(deps): bump github.com/twmb/avro from 1.7.2 to 1.8.0 (#11210)
* build(deps): bump github.com/twmb/avro from 1.7.2 to 1.8.0 Bumps [github.com/twmb/avro](https://github.com/twmb/avro) from 1.7.2 to 1.8.0. - [Commits](https://github.com/twmb/avro/compare/v1.7.2...v1.8.0) --- updated-dependencies: - dependency-name: github.com/twmb/avro dependency-version: 1.8.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * iceberg: adapt to twmb/avro v1.8.0 and iceberg-go defensive copies avro v1.8.0 changes Schema.Root() to return *SchemaNode, which breaks iceberg-go v0.6.0's internal avro_schemas.go. The fix (apache/iceberg-go#1843) is only on iceberg-go's main branch, unreleased, so bump iceberg-go to that commit (c210509) alongside the avro bump. That iceberg-go revision also changes two behaviors seaweedfs worked around: - It now infers a manifest list's format version from the embedded writer schema, so a list missing the "format-version" header entry (DuckDB's shape is read as v2, not v1. ReadManifestList's header patching is now a redundant safety net; tests updated to expect v2. - It returns defensive copies from DataFile.Partition(), so the ReadManifest shim's in-place partition normalization was silently discarded. Rebuild the entry through NewDataFileBuilder when any partition value is normalized, copying every other DataFile field so manifest round-trips are preserved. - It converts day-transform partitions to iceberg.Date on read (applyDayTransformDates), so the day-partition cases the shim and tests guarded now convert without help; tests updated to expect iceberg.Date from the raw read. EOF ) * iceberg: accept assert-ref-snapshot-id without snapshot-id iceberg-go's new nullableInt64 parser rejects an assert-ref-snapshot-id requirement whose "snapshot-id" field is absent from the JSON, even though the Iceberg REST spec makes it optional (null means the ref must not already exist). v0.6.0 used a plain *int64, so absent was nil and accepted. ClickHouse sends the requirement without snapshot-id when asserting a branch does not yet exist, so its writes fail with "missing required field \"snapshot-id\"". normalizeRequirements splices an explicit null into any assert-ref-snapshot-id requirement missing the field before handing the JSON to iceberg-go's parser, restoring the v0.6.0 behavior across both iceberg-go versions. * iceberg: fix v1 block_size_in_bytes default in rebuilt manifest entries rebuildManifestEntry set block_size_in_bytes to 0, but the v1 manifest schema requires the default of 64 MiB ("Always write default in v1"). The original value is not exposed on the DataFile interface, so use the spec default. Also clarify the fallback comment to note that empty (zero-record / zero-byte) files also trigger it, not just a nil spec. Add a round-trip test that writes a rebuilt entry as v1 and verifies block_size_in_bytes is 64 MiB via Avro decoding. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
|
005012edcf |
rust volume: quick-repair redb on durable checkpoints (#11203)
* rust volume: insert redb rebuilds in needle-id order Unlink the .rdb before create (create does not truncate). Collapse last-write-wins, then insert live keys sorted so 4.2.0 packs leaves. * rust volume: rebuild redb from a BTreeMap and clear leftover keys Peak rebuild memory is one ordered map instead of HashMap + Vec + stable-sort scratch. Unlink stays best-effort: if it fails, retain clears the leftover table before sorted insert. Compute idx metrics before the write so a read error does not unlink a committed .rdb. * rust volume: drop the extra redb read transaction on put/delete put uses insert()'s previous value. delete gets then inserts the tombstone in the same write transaction. Truncate the .idx row on any failed redb write after the append. * rust volume: unpack redb blobs through packed_to_needle_value save_to_idx, ascending_visit, and collect_entries used the same length-check copy as get. Route them through the helper so a wrong-length value is absent everywhere, not a panic. * rust volume: quick-repair redb on durable checkpoints set_quick_repair(true) on the durable checkpoint transaction so an OOM-killed volume server opens without a full-file repair scan. * rust volume: reopen redb from .idx on non-poisoned commit error redb 4.2.0 can make a Durability::None commit visible before returning Err(CommitError::Storage(..)). In that state the database refuses further write transactions, so truncating the .idx row (the old behavior) would leave a redb-only put or tombstone that the stored idx_size makes the reload skip. Distinguish CommitError::TransactionPoisoned (txn rolled back, db still usable -- truncate the orphan .idx row as before) from other commit errors (change may be visible, db refuses writes -- keep the .idx row, close the database, and reopen from .idx to repair redb's internal state). db becomes Option<Database> so reopen_from_idx can drop the old file lock before load_from_idx opens the same path. rdb_path, version, and cache_bytes are stored so the reopen uses the same configuration. * rust volume: truncate .idx row when redb is closed in put put appends the .idx entry before acquiring the write transaction. When db_or_err() fails (db is None after a failed reopen), the ? returned without calling truncate_idx_to_offset, so a write reported failed remained in the authoritative .idx and was replayed on restart. Handle db_or_err() explicitly and truncate the orphan .idx row before returning the error, matching the existing handling for begin_write, open_table, and insert failures. --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
5d5ea18287 |
topology: fix fatal concurrent map read/write on VolumeLayout.crowded (#11216)
SetVolumeCrowded mutated the crowded map under accessLock.RLock(), while GetWritableVolumeCount reads the same map under RLock() on the Assign hot path. Two concurrent RLock holders with one writing and one reading the map triggers a fatal "concurrent map read and map write" that kills the master process (unrecoverable, bypasses recover). Take the write lock in SetVolumeCrowded instead. This event path is a low-frequency single consumer driven by the crowded-volume event loop, and every other mutation of crowded already holds Lock(); setVolumeCrowded takes no nested locks, so there is no deadlock path. The hot readers (GetWritableVolumeCount, CloneWritableVolumes) keep using RLock. Adds a -race regression test that fails (race detected) on the old RLock and passes with the write lock. Fixes #11211 |
||
|
dependabot[bot]
|
34936c610d |
build(deps): bump helm/kind-action from 1.14.0 to 1.15.0 (#11212)
Bumps [helm/kind-action](https://github.com/helm/kind-action) from 1.14.0 to 1.15.0. - [Release notes](https://github.com/helm/kind-action/releases) - [Commits](https://github.com/helm/kind-action/compare/v1.14.0...v1.15.0) --- updated-dependencies: - dependency-name: helm/kind-action dependency-version: 1.15.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
5e3af51d91 |
build(deps): bump docker/setup-qemu-action from 4.2.0 to 4.3.0 (#11213)
Bumps [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action) from 4.2.0 to 4.3.0. - [Release notes](https://github.com/docker/setup-qemu-action/releases) - [Commits](https://github.com/docker/setup-qemu-action/compare/v4.2.0...v4.3.0) --- updated-dependencies: - dependency-name: docker/setup-qemu-action dependency-version: 4.3.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Junker der ProvinzandJunker der Provinz
|
a4885b7975 |
log_buffer: stop closing a notification channel another reader still holds (#11177)
* fix(log_buffer): stop closing a notification channel another reader still holds - #10810 The report blames the polling loop for the busy spin, but that loop is not what burns the core. LogBuffer keeps one notification channel per subscriberID, and UnregisterSubscriber closes it. Two registrations that share a subscriberID share that channel, which happens whenever a client opens a second stream or an old stream has not yet noticed it was replaced, so the first unregister closes a channel the other reader is parked on. A closed channel makes every receive in awaitNotificationOrTimeoutFor return instantly, and that reader then spins at full speed for the rest of its life. Subscriptions are now reference counted. Registering an existing subscriberID hands back the same channel and raises the count; the channel is only closed when the last holder unregisters. * test: fail if the surviving reader stops instead of keeps reading Review caught that the iteration count alone proves nothing: had LoopProcessLogData returned when the duplicate reader unregistered, the counter would sit at 0 and the assertion would pass without a reader ever having been there to spin. Check the reader is still running before trusting its low count. --------- Co-authored-by: Junker der Provinz <jdp@braethoria.com> |
||
|
dependabot[bot]
|
e0f9e02761 |
build(deps): bump github.com/prometheus/client_model from 0.6.2 to 0.6.3 (#11208)
Bumps [github.com/prometheus/client_model](https://github.com/prometheus/client_model) from 0.6.2 to 0.6.3. - [Release notes](https://github.com/prometheus/client_model/releases) - [Commits](https://github.com/prometheus/client_model/compare/v0.6.2...v0.6.3) --- updated-dependencies: - dependency-name: github.com/prometheus/client_model dependency-version: 0.6.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
3f8cc380fb |
build(deps): bump cloud.google.com/go/storage from 1.64.0 to 1.67.0 (#11206)
Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.64.0 to 1.67.0. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md) - [Commits](https://github.com/googleapis/google-cloud-go/compare/compute/v1.64.0...compute/v1.67.0) --- updated-dependencies: - dependency-name: cloud.google.com/go/storage dependency-version: 1.67.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
3eba8ebb9b |
build(deps): bump github.com/go-ldap/ldap/v3 from 3.4.13 to 3.4.14 (#11209)
Bumps [github.com/go-ldap/ldap/v3](https://github.com/go-ldap/ldap) from 3.4.13 to 3.4.14. - [Release notes](https://github.com/go-ldap/ldap/releases) - [Commits](https://github.com/go-ldap/ldap/compare/v3.4.13...v3.4.14) --- updated-dependencies: - dependency-name: github.com/go-ldap/ldap/v3 dependency-version: 3.4.14 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Eliah Rusin
|
ed5b342f0c |
rust volume: optional redb insert_before bulk load (#11205)
* rust volume: quick-repair redb on durable checkpoints set_quick_repair(true) on the durable checkpoint transaction so an OOM-killed volume server opens without a full-file repair scan. * rust volume: optional redb insert_before bulk load Behind redb-experimental-cursor (default off). Production binary stays on sorted insert(). CI unit tests run both feature settings. * rust volume: exercise insert_before across leaf splits Replace the 5-key cfg clone with a 4000-key reverse-order rebuild so CursorMut::insert_before hits page splits. CI runs the feature only on storage::needle_map unit tests. |
||
|
dependabot[bot]
|
99b84eeb88 |
build(deps): bump github.com/getsentry/sentry-go from 0.48.0 to 0.49.0 (#11207)
Bumps [github.com/getsentry/sentry-go](https://github.com/getsentry/sentry-go) from 0.48.0 to 0.49.0. - [Release notes](https://github.com/getsentry/sentry-go/releases) - [Changelog](https://github.com/getsentry/sentry-go/blob/master/CHANGELOG.md) - [Commits](https://github.com/getsentry/sentry-go/compare/v0.48.0...v0.49.0) --- updated-dependencies: - dependency-name: github.com/getsentry/sentry-go dependency-version: 0.49.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Eliah RusinandChris Lu
|
b6690cfbc8 |
rust volume: drop the extra redb read transaction on put/delete (#11204)
* rust volume: insert redb rebuilds in needle-id order Unlink the .rdb before create (create does not truncate). Collapse last-write-wins, then insert live keys sorted so 4.2.0 packs leaves. * rust volume: rebuild redb from a BTreeMap and clear leftover keys Peak rebuild memory is one ordered map instead of HashMap + Vec + stable-sort scratch. Unlink stays best-effort: if it fails, retain clears the leftover table before sorted insert. Compute idx metrics before the write so a read error does not unlink a committed .rdb. * rust volume: drop the extra redb read transaction on put/delete put uses insert()'s previous value. delete gets then inserts the tombstone in the same write transaction. Truncate the .idx row on any failed redb write after the append. * rust volume: unpack redb blobs through packed_to_needle_value save_to_idx, ascending_visit, and collect_entries used the same length-check copy as get. Route them through the helper so a wrong-length value is absent everywhere, not a panic. --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Eliah Rusin
|
94b10c006d |
rust volume: insert redb rebuilds in needle-id order (#11202)
* rust volume: insert redb rebuilds in needle-id order Unlink the .rdb before create (create does not truncate). Collapse last-write-wins, then insert live keys sorted so 4.2.0 packs leaves. * rust volume: rebuild redb from a BTreeMap and clear leftover keys Peak rebuild memory is one ordered map instead of HashMap + Vec + stable-sort scratch. Unlink stays best-effort: if it fails, retain clears the leftover table before sorted insert. Compute idx metrics before the write so a read error does not unlink a committed .rdb. |
||
|
Chris Lu
|
15e4da65f7 |
volume: avoid read-only replica write targets (#11195)
* master: carry replica read-only state in volume lookups * volume: refresh writable replica targets * volume: preserve read-only replicas for deletes * master: propagate read-only delete capability * volume: target delete-capable replicas * volume: honor configured HTTPS for replica deletes * volume: reject insecure delete authorization forwarding * master: broadcast delete capability changes * volume: align Rust replica routing * http: protect credentialed replica redirects * master: preserve digest compatibility for delete capability * volume: propagate read-only state in short heartbeats * volume: report changed short volume state * http: guard TLS client redirects * master: announce mounted volume read-only state * volume: replace changed identity deltas * master: replace incremental volume layouts in order * master: keep moved volume lookup available * volume: announce read-only mounts |
||
|
Tim Olowandtimolow
|
331c6c3642 |
shell: volume.check.disk — actionable verdict for diverged vacuumed replicas (#11197)
* shell: volume.check.disk — actionable verdict for diverged vacuumed replicas
When -resurrectMissingNeedles is gated off because both replicas have been
vacuumed (compaction revision > 0) — the normal state of any production
cluster — check.disk previously stopped at 'cannot prove they are missing
writes vs vacuumed deletes' and did nothing, leaving a diverged replica with
no repair path. volume.fix.replication does not catch it either: it only
acts when the replica COUNT is below the expected replication, never when
two replicas are both present but hold different live data.
Classify the divergence instead of dead-ending:
liveDivergence() counts live (non-deleted) needles present on one replica's
index but entirely absent from the other, in both directions. Tombstones
are excluded, so vacuum asymmetry (a compacted replica that dropped deleted
entries) is not mistaken for divergence.
reportDivergenceVerdict() turns the count into an operator action:
- one-sided (one replica has all the live data, the other has no unique
live needles) -> print the exact safe repair:
volume.copy -source <complete> -target <lagging> -volumeId <id>
Re-copying the complete replica is safe precisely because the lagging
side holds no unique live data; VolumeCopy's verify-before-destroy gate
independently confirms the source holds the volume before deleting the
target.
- two-sided (split-brain, both sides have unique live data) -> warn and
do NOT emit an auto repair; point to volume.fsck -findMissingChunksInFiler
to confirm the 'missing' needles are orphans before converging.
Report-only: no data is modified and the resurrection safety gate is
untouched. This is what lets a 13-volume diverged cluster be diagnosed and
repaired in minutes instead of by hand-diffing every index.
Observed motivating case: home SeaweedFS 4.45 cluster, 13 010 cross-rack
volumes diverged after failed replicate writes (ReplicatedWrite MaxAttempts=1
fire-and-forget), 11 one-sided + 2 two-sided, all repaired via volume.copy.
* shell: volume.check.disk — address review nits on divergence verdict
- Use pb.NewServerAddressFromDataNode (dialable ip:port, Address with Id
fallback) for the advertised volume.copy -source/-target instead of the
logical node Id, which may not be dialable.
- Make the one-sided verdict tombstone-aware: when the lagging replica has
been vacuumed, absent live needles may be valid deletions whose tombstones
were dropped, so a whole-volume re-copy would resurrect them. The command
is only advertised as safe when the lagging side is proven never-vacuumed
(compaction revision 0 read under -resurrectMissingNeedles); otherwise a
caveat is printed pointing at fsck/needle-level repair.
- Fix reversed copy direction when the source replica is the lagging one
(must copy complete -> lagging in both cases).
- Test: real tombstone (negative size) with the correct 0/0 expectation and
|| assertion; verdict test now covers dialable address, corrected
direction, and caveat on/off.
* shell: volume.check.disk — per-replica revision knowledge, no copy command for vacuumed lagging side
- Track srcRevKnown/tgtRevKnown separately: in unidirectional mode the
target revision IS read, so a proven never-vacuumed target no longer
gets a false resurrection warning (regression: bidi=false, target rev 0).
- A one-sided verdict now only emits the volume.copy command when the
lagging side is proven never-vacuumed; when it is vacuumed (or unproven)
the verdict refuses to print the destructive command and points at
fsck/needle-level repair instead — an appended caveat next to a ready-to-
paste copy command was still inviting the resurrection.
- deletionCaveat now returns the boolean safety decision.
* shell: volume.check.disk — preserve gRPC port, proven two-sided is not split-brain
- Emit the raw ServerAddress string (host:port.grpcPort) instead of
String()/ToHttpAddress(), which drops the custom gRPC port and would
make the suggested volume.copy dial the default port and fail.
- Two-sided divergence with both replicas proven never-vacuumed under
-resurrectMissingNeedles is mutually missed writes, not split-brain:
recommend re-running with -apply (in-place resurrection both
directions) instead of the split-brain no-auto-repair warning.
- Tests: case F (proven two-sided -> -apply, no split-brain warning),
case G (custom gRPC ports preserved in emitted addresses).
---------
Co-authored-by: timolow <tim@timolow.dev>
|
||
|
Chris Lu
|
3225d2b0ce |
rust worker: install rustls CryptoProvider to fix TLS panic (#11194) (#11196)
* rust worker: add install_default_crypto_provider helper lance's aws backend pulls aws-lc-rs and reqwest's rustls-tls pulls ring, so rustls 0.23 cannot auto-select a CryptoProvider and tonic's client TLS panics on first use. Add install_default_crypto_provider, pinning the default to aws-lc-rs, mirroring the Rust volume server's helper of the same name. Includes a regression test that builds a TLS channel and panics without the install in this crate, where both providers link. * rust worker: install the crypto provider at startup Call install_default_crypto_provider before any TLS use, the way the Rust volume server does in its main. Without this a worker started with --tls-ca/--tls-cert/--tls-key panics on the first admin dial (#11194). |
||
|
github-actions[bot]
|
8537d5bc08 | docs: regenerate star history chart | ||
|
Chris Lu
|
70a26cb5d2 |
s3: gate IAM-cache gRPC RPCs behind admin Bearer auth (#11190)
* s3: gate IAM-cache gRPC RPCs behind admin Bearer auth The SeaweedS3IamCacheServer registered on the S3 gateway's internal gRPC port (default 0.0.0.0:18333) accepted PutIdentity/RemoveIdentity/PutPolicy/ DeletePolicy/GetPolicy/ListPolicies/PutGroup/RemoveGroup with no per-RPC authentication. An unauthenticated network peer could call PutIdentity with Actions:[Admin] and write straight into the live accessKeyIdent map that the SigV4 path reads, bypassing S3 authentication entirely. Mirror the filer's IamGrpcServer.checkAdminAuth: require a Bearer token signed with jwt.filer_signing.key (read from the existing s3a.filerGuard) at the top of every IAM-cache RPC. With no key configured the check is a no-op, matching the rest of SeaweedFS's gRPC surface. * credential: attach admin Bearer token to S3 IAM-cache propagation The filer's PropagatingCredentialStore fans IAM mutations out to peer S3 servers over the SeaweedS3IamCache gRPC service. Now that the S3 handlers require a Bearer token signed with jwt.filer_signing.key, attach one to the outgoing propagation context (mirroring shell/iamAdminAuthContext). With no key configured it is a no-op, so deployments that run without the signing key keep working. * credential: mint IAM-cache admin token after master discovery propagateChange attached the admin Bearer token before ListClusterNodes, so master-client retries could run down the (default 10s) token lifetime before the peer S3 fan-out began, leaving peers to reject an expired token and IAM caches stale. Move withIamCacheAdminAuth to after discovery succeeds, immediately before the propagation timeout is derived. * credential: cap IAM-cache propagation timeout below JWT lifetime The propagation fan-out used a fixed 10s timeout. If an operator configures jwt.filer_signing.expires_after_seconds below 10, the admin token can expire while slower S3 peers are still being contacted, leaving their IAM caches stale. Derive the propagation deadline as min(10s, tokenTTL) so it never outlives the token. withIamCacheAdminAuth now returns the token's lifetime (0 = no expiry) for this purpose. |
||
|
Eliah RusinandClaude Opus 5
|
ade4bdf9e6 |
rust volume: stop a tier move whose caller has gone (#11192)
Both tier-move handlers run in a detached tokio::spawn and report progress through a closure that returns (), with the send result discarded. Nothing observes the caller leaving, so an abandoned move uploads or downloads the whole .dat anyway and then commits the transition. Go aborts both. Its progress callback returns `stream.Send`'s error, which surfaces out of the reader in s3_upload.go:99 and the writer in s3_download.go:84 and fails the transfer, so the volume info is never rewritten. The Rust port dropped that by typing the callback as FnMut(i64, f32) with no result. Give the callback Go's signature -- FnMut(i64, f32) -> Result<(), String> -- and abort when the caller's channel is closed. Checked on every part rather than only where progress is reported, since the report is rate-limited to one a second and would miss a caller that left in between. A merely full channel is a slow reader, not a departed one, so only TrySendError::Closed counts as cancellation. Two consequences of aborting mid-transfer that the old code never had to handle: - upload_file now aborts the multipart upload when the transfer fails. An abandoned multipart upload does not show up in an ordinary object listing but still accrues storage charges until a lifecycle rule reaps it, and cancellation makes that a routine path rather than a rare one. - The tier-down handler removes the partial .dat. download_file pre-allocates the destination to the object's full size, so an aborted download leaves a file of the right length and the wrong content -- and this handler refuses to run at all when a local .dat exists, so leaving one wedges every retry on "already on local disk" and a restart would load the sparse file as the volume's data. There is deliberately no check between a finished transfer and the bookkeeping that follows. Once the object is in S3, or the .dat is on disk, that bookkeeping is what makes the state consistent; stopping there would leave an object paid for and referenced by nothing, or a complete local .dat the volume still calls remote. Go does not gate there either -- its callback only runs during the transfer. Claude-Session: https://claude.ai/code/session_0122W3eqt6gmLUMxmRoZdPAb Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
4b41329e12 |
rust volume: stop a VolumeCopy whose caller has gone (#11188)
* rust volume: stop a VolumeCopy whose caller has gone VolumeCopy runs its copy in a detached tokio::spawn and reports progress with the send error discarded, so nothing observes the client leaving. When the caller cancels the RPC -- which weed-admin's batch balance does routinely, starting far more copies than it finishes -- the server streamed the whole volume from the source, wrote it to disk, and mounted it. The destination is then left holding a volume nobody took delivery of: its index cache is never reclaimed, and under replication=000 one volume id ends up on two servers, both writable, which concurrent writes can diverge. Three checks now reach the task: - Every chunk in copy_file_from_source, via the sender's is_closed(). This is the one that matters in practice. The first progress report is 128MB in, so for a smaller volume -- the ordinary balance move -- no send ever happens and its result says nothing; only the closed channel does. The sender is passed for the .idx and .vif copies too, with reporting gated separately, so those phases notice as well. - The throttle sleep, which for a throttled copy runs for seconds at a time, now races the sender's closed() instead of being slept through. - Immediately before mount_volume, and once at the top of the task. Cancellation surfaces as an ordinary Err(Status::cancelled), so it lands in the existing error branch that already removes the partial .dat/.idx/ .vif and the .note. That branch also logs now: the error otherwise went to a channel nobody was reading, leaving the operator with the balancer's "delete that copy, then re-run the move" and no cause. This also clears the stranded read-only sources reported on the issue. They are downstream of the orphan mount, not a separate defect: LiveMoveVolume's cleanup probes the target before undoing the freeze (volume_move.go:95, "the server can finish the copy and mount the target even when the client loses the stream"), and when it finds a mounted copy it cannot attribute, or cannot delete, it deliberately keeps the source readonly rather than risk two writable replicas -- the messages at volume_move.go:110 and :123. With nothing mounted on the target the probe reports clean and the freeze is undone. On Go parity: the progress send result is honoured here too, matching `return false` in volume_grpc_copy.go. But that report is Go's only abort signal, and measured against a 120MiB volume -- above the throttler's activation threshold, below the 128MiB report interval -- a Go destination mounts an abandoned copy as well. The issue's premise that Go aborts holds only above the report interval. The Rust side now stops in both cases; the Go behaviour is worth its own issue. Tests: the integration test runs against both implementations and is green on Go, red on Rust before this change. Its 192MiB fixture is sized for two separate constraints, documented at the fixture: IoBytePerSecond is a no-op below ~100ms of wall clock (64MiB copies in ~110ms on a tmpfs loopback cluster), and the payload must exceed the 128MiB report interval for the Go leg to pass at all. The two Rust unit tests cover what the integration test cannot reach: cancellation detected with no progress report at all, and the cleanup of the partial files plus the .note. Fixes #11186 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0122W3eqt6gmLUMxmRoZdPAb * rust volume: surface VolumeCopy cancellation as Status::cancelled copy_file_from_source returned Result<_, String>, so the per-chunk cancellation path -- the one that matters in practice for volumes below the 128MB report interval -- was wrapped to Status::internal at the call sites. The spawn logging branch then classified it as a generic failure instead of the intended "abandoned by caller", defeating the logging change in the same PR for the case that occurs most often. Return Result<_, Status> from copy_file_from_source: Status::cancelled for caller-gone, Status::internal for the existing errors. Drop the .map_err(|e| Status::internal(e)) at the three call sites. The unit test now asserts the code is Cancelled, not just the message text, so the classification is locked in. * rust volume: close cancellation gaps in VolumeCopy Address two review findings on the same PR: 1. Roll back a mount that races a departing caller. The pre-mount is_closed() check cannot close the window between the check and mount_volume: if the receiver drops in that gap, the volume mounts and the final tx.send(Ok(...)) fails, but its error was discarded (let _ =), so the task returned Ok(()) and the error branch never ran. The destination then held an orphaned mounted replica — the exact defect this PR prevents. Fix: track a mounted flag. The final send now checks its result; on failure it returns Status::cancelled, and the error branch calls store.delete_volume (which unmounts AND removes the files) when mounted is true, instead of only unlinking. 2. Observe cancellation while awaiting the source stream. The per-chunk is_closed() check only runs after stream.message().await returns. A stalled source (slow disk, partition, GC pause) never delivers a chunk, so a caller that has already left cannot preempt the read: the task, the source connection, and the partial files (including the .note) all outlive the caller indefinitely. Fix: race stream.message() against progress_tx.closed() in a tokio::select!, so a departing caller preempts a stalled source. Adds test_volume_copy_after_mount_cancellation_rolls_back_mount to cover the after-mount rollback path. cargo test --release green (497 + 5 + 1 + 28). * rust volume: keep remote data on after-mount rollback, race RPC startup Two review findings on the after-mount rollback added in |
||
|
|
b82cb05d71 |
rust volume: checkpoint the redb index durably every 1000 writes (#11182)
* rust volume: checkpoint the redb index durably every 1000 writes
Every put and delete on a redb-backed volume committed with
Durability::None and nothing ever committed durably, on the theory that
the .idx file is the source of truth. redb, however, keeps an entry in
its transaction tracker for every non-durable commit and cannot recycle
pages that were on disk at the last durable commit until a durable one
happens. With no durable commit for the life of the process, both grew
with every write, and .rdb files could bloat toward double size after a
restart (#11179, the hash-table rehash stacks in the memleak output).
The needle map now counts non-durable commits and reports when a
checkpoint is due; the volume takes it, data first: flush the .dat, then
the map fsyncs the .idx and commits redb durably, recording in the same
transaction how much of the .idx the table reflects. A checkpoint makes
the index durable, so the bytes it points at must be down before it, or
after a power loss the index would reference past the end of the .dat
and the volume would load read-only. A failed .dat flush skips the
checkpoint; it is retried on the next write.
Volume::close() now closes the needle map instead of only syncing it,
and the redb map's close() takes the same checkpoint. Before, a clean
shutdown left the table durable (redb flushes on drop) but the recorded
.idx size stale at its load-time value, so the next load replayed every
entry written since load on top of the counters.
On load, the redb map's counters now come from the whole .idx history,
the way Go's LevelDB map rebuilds them (newest entry first, with a bloom
filter of seen keys), instead of from the table's final state. Both the
reuse and the full-rebuild path use it, so overwritten and deleted bytes
keep counting as garbage across restarts, and the incremental replay of
the .idx tail only touches the table, which makes it idempotent whether
or not the table is ahead of the recorded .idx size.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019x36FiSeyePh77YXao15kK
* rust volume: skip redundant .idx fsync on checkpoint after flush_idx
On the fsync=true write path, flush_idx() already fsyncs the .idx before
maybe_checkpoint_index() runs, so the checkpoint's own sync() fsyncs the
same file a second time for nothing. Thread an idx_already_synced flag
from the volume through maybe_checkpoint_index into checkpoint(sync_idx):
when it is true the checkpoint skips its .idx fsync and only does the
durable redb commit. The delete path and close() still sync (they have
not flushed the .idx beforehand).
Co-Authored-By: Chris Lu <chris.lu@gmail.com>
* rust volume: saturate writes_since_checkpoint to prevent u32 overflow
If checkpoints keep failing (e.g. a persistent .dat flush failure whose
error is not EIO and so does not mark the volume read-only), the counter
increments on every write with no upper bound and wraps at ~4.3 billion.
Use saturating_add so it pins at u32::MAX instead, which keeps
checkpoint_due() true and retries on every subsequent write.
Co-Authored-By: Chris Lu <chris.lu@gmail.com>
* rust volume: only update max_file_key on live entries in idx metric rebuild
metrics_from_idx called maybe_set_max_file_key on every entry including
tombstones, but the live on_put path only calls it for puts and on_delete
never does. A tombstone always has a preceding put for the same key that
already set max_file_key, so the result is the same today; restricting it
to live entries makes the parity with the live path exact and self-evident.
Co-Authored-By: Chris Lu <chris.lu@gmail.com>
* rust volume: advance idx_file_offset only after redb commit succeeds
put() and delete() appended to the .idx file and advanced idx_file_offset
before committing to redb. If the redb commit failed, the offset included
the orphan row that redb doesn't reflect. A later checkpoint would record
that offset as "the table reflects up to here," and the reload would skip
the orphan row entirely — the entry becomes permanently unindexed.
Move the idx_file_offset increment to after the successful redb commit.
The .idx file still has the orphan row (append-only), but idx_file_offset
stays behind it, so the next checkpoint records the smaller offset and
the reload replays the orphan row back into redb.
Co-Authored-By: Chris Lu <chris.lu@gmail.com>
* rust volume: skip index checkpoint on close when .dat sync fails
Volume::close() discarded the .dat sync_all() result and always
checkpointed the redb index. If the .dat sync failed, the checkpoint
made the index durable with entries that may point past the unflushed
.dat tail, and a power loss would leave the volume read-only on reload
(the max_needle_end check fires).
Check the .dat sync result: on success, checkpoint as before; on
failure, call close_without_checkpoint() — sync the .idx and drop the
writer without a durable redb commit. META_IDX_SIZE stays at the last
successful checkpoint, so the reload replays the uncheckpointed tail
(redb still flushes on drop, but without recording idx_size).
Co-Authored-By: Chris Lu <chris.lu@gmail.com>
* rust volume: schedule checkpoints on every index mutation path
maybe_checkpoint_index was only called from do_write_request and
do_delete_request. put_needle_index and write_needle_blob_and_index
also call NeedleMap::put, which increments writes_since_checkpoint,
but neither triggered the checkpoint. Through those paths the counter
could grow past the interval without ever being satisfied, leaving
non-durable redb transaction state until close().
Add maybe_checkpoint_index(false) after the successful nm.put in both
methods. The .dat flush inside maybe_checkpoint_index covers the blob
write in write_needle_blob_and_index; put_needle_index pairs with a
prior write_needle_blob, so the flush covers that too.
Co-Authored-By: Chris Lu <chris.lu@gmail.com>
* rust volume: truncate orphan .idx row on failed redb commit
Commit
|
||
|
Chris Lu
|
d8a40ef750 |
ci: pin actions/setup-python to v7 in star_history workflow (#11191)
The star_history workflow referenced actions/setup-python@v8, which does not exist, causing the workflow to fail at the "Set up job" step. Pin to v7, matching the version used across the other workflows. |
||
|
Chris Lu
|
e6f2386a0f |
admin: redact S3 secret keys for read-only sessions (#11189)
The Admin UI documents its read-only account as view-only and blocks its
write requests, but the authenticated read routes returned object-store
users with plaintext access and secret keys. A read-only admin user could
retrieve another user's live S3 credential pair from GET /api/users and
GET /api/users/{username} and use it directly against the S3 endpoint,
converting view-only access into the victim identity's object-store
authority.
Redact the reusable secret_key in GetUsers, GetUserDetails, and the
rendered users page whenever the requesting session has the read-only
role. The public access_key identifier is retained so identities remain
browsable; only the reusable secret is stripped. Admin and no-auth
sessions are unaffected.
|
||
|
Junker der Provinz
|
78f79a3919 |
master: honour -volume.fileSizeLimitMB on the master's /submit (#11176)
* fix(master): honour -volume.fileSizeLimitMB on the master's /submit - #6748 `weed server -volume.fileSizeLimitMB=2048` still refused anything over 256MB, and the reason is not the one the report assumes: the option does reach the volume server. The master does not use it. Uploads through the master's /submit are buffered by submitForClientHandler, which passed a hardcoded 256MB to needle.ParseUpload, so the master rejected what the volume server it started would have accepted. The limit is now passed in. `weed master` gains its own -fileSizeLimitMB with the same 256 default, so a standalone master behaves exactly as before, and `weed server` and `weed mini` hand it the value their volume server already got. * master.follower: take the same upload limit, and say which flag to match Review found the follower left behind. It serves /submit like the leader and buffers uploads under the same limit, but kept the fixed 256MB, so a cluster raised above that would accept an upload through the leader and refuse the identical one through a follower. Two smaller points from the same review: the master's flag description named only the standalone volume server's spelling, and now names the weed server and weed mini form too; and the under-limit test asserted on the error message alone, so it would have passed had the limit rejected that payload with different wording. It now requires the request to get past parsing. |
||
|
yanglongwei
|
eb717199d0 |
master: delete replica_placement_mismatch labels when volumes leave topology (#11062)
* master: delete replica_placement_mismatch labels when volumes leave topology Fixes #10804. Setting the gauge to 0 left stale Prometheus time series that grew unbounded with volume churn; remove the label set on unregister instead. * master: delete replica_placement_mismatch only after last placement leaves Unconditional DeleteLabelValues on UnRegisterVolumeLayout dropped the series while other data nodes still held the volume, hiding under-replication until the next collect cycle. Delete only when Lookup is empty, and cover the two-copy case in a regression test. |
||
|
Chris Lu
|
3e85d9ec8e |
admin: bind to loopback by default, guard public unauthenticated bind (#11185)
admin: bind to loopback by default, refuse public unauthenticated bind The admin HTTP server (port 23646) defaulted to binding 0.0.0.0 with authentication disabled when -adminPassword was not supplied, exposing the full admin REST API (user creation, credential issuance, bucket deletion, filer deletion) unauthenticated on the network. This is the footgun described in GHSA-m3m8-mrgq-hf9h. Keep the no-auth mode for local dev, but remove the network exposure: - Add -ip flag (default 127.0.0.1) so the server binds loopback only unless the operator explicitly chooses a public address. - Refuse to start when binding a non-loopback address with no -adminPassword and no [https.admin] mTLS. The operator must enable auth or use loopback. - weed mini sets -ip from its existing -ip.bind; the guard does not apply because mini calls startAdminServer directly, not runAdmin. Addresses GHSA-m3m8-mrgq-hf9h. |
||
|
Chris Lu
|
f35e2ccf21 |
s3: warn when a lifecycle change leaves fast-path-stamped objects on their old TTL (#11184)
* s3: warn when a lifecycle change leaves fast-path-stamped objects on their old TTL The per-write TTL fast path (opt-in via s3.bucket.lifecycle.fastpath) stamps a volume TTL at PutObject time that can't be taken back. When an operator lengthens or removes an Expiration.Days rule (or deletes the bucket lifecycle) on a fast-path-enabled bucket, objects already written keep their baked-in TTL and won't be rescued by the change — unlike the default worker-driven path, which re-evaluates the current rules each pass. This is the data-loss direction described in #11183. Surface it: Put/DeleteBucketLifecycle now emit a glog warning and set X-Seaweed-Lifecycle-Fastpath-Warning on the response when the change removes, disables, lengthens, or re-scopes a fast-path-eligible rule. Shortening a rule does not warn (old objects simply expire later, not data loss). Tag-only and overflow-day rules are never on the fast path and never warn. Addresses the warning half of option 2 in #11183. * s3: address review — emit warning after mutation succeeds, fix ID-rename false positive Two issues raised by CodeRabbit, Greptile, and Devin reviews: 1. Failed mutations retained the warning header. The warning was set on the ResponseWriter before storeBucketLifecycleConfiguration / clearStoredBucketLifecycleConfiguration was called; if that failed, the error response carried a warning for a change that was never applied. Now the reason is computed before the mutation but the log and header are emitted only after it succeeds. 2. Rule renames produced false "removed" warnings. fastpathRuleKey used Rule.ID as the sole identity when present, so renaming a rule (same prefix/size/days, different ID) treated the old rule as removed. Replaced with two-pass matching: first by ID, then by fast-path predicates (prefix + size). An ID-only rename with unchanged predicates and days no longer warns. Greedy matching ensures each new rule is consumed by at most one old rule. Added regression tests: ID-only rename (no warn), rename + lengthen (warn), rename + shorten (no warn). |
||
|
|
f99c4a1f14 |
s3: make RenameObject idempotent for a retried request (#11178)
* feat(s3): make RenameObject idempotent for a retried request - #10661 A rename that succeeds but whose response is lost leaves the client with no safe move: retrying returned 404, because the source is already gone, so a retry was indistinguishable from a rename that never happened. The destination now carries what the rename that created it was, under x-seaweedfs-rename-token: the client's token, the source key and the time. A retry that names the same token and the same source and destination is answered 200 without touching anything. The same token sent for a different rename is refused with 409 rather than silently answered, and a token older than 24 hours is treated as unrelated so a key cannot answer for a request indefinitely. Requests without the header behave exactly as before. * s3: answer a reused rename token with 409, not 400 The PR promised Conflict and the code returned Bad Request. 400 tells a client its request was malformed and invites it to give up; this request is well formed and resending it unchanged will not help, because what it collides with is a rename the same token already stands for. The status code is now asserted in a test, since it is the part of this behaviour a client actually acts on. * Update weed/s3api/s3err/s3api_errors.go Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: fix rename token review notes - ErrIdempotentParameterMismatch returns 409 Conflict, not 400. The comment and TestRenameTokenReuseAnswersConflict both expect 409; the code regressed to 400 in a later commit. - stampRenameToken: clarify that markRenameToken mutates srcEntry in place, so the token reaches the destination via the move regardless of whether the UpdateEntry succeeds. The precondition only guards the pre-move write, not the move itself. - Extract the handler retry branch into retryRenameDecision and add TestRetryRenameDecision, covering the source-still-exists fallthrough that was previously reasoned about but not tested. * s3: IdempotentParameterMismatch returns 400, matching AWS docs The AWS S3 RenameObject API documentation specifies HTTP Status Code: 400 for IdempotencyParameterMismatch. Revert the previous 409 change and align the comment and test with the documented behavior. --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Eliah RusinandClaude Fable 5.1
|
3f9b05946b |
rust volume: bound the redb index cache per volume by --index tier (#11180)
The Rust volume server opens one redb database per volume and built each with redb's defaults, which give every database a 1 GiB page cache (0.9 GiB read cache + 0.1 GiB write buffer). With hundreds of volumes behind one disk the process-wide ceiling was volumes x 1 GiB: memory grew in proportion to the pages traffic touched, never shrank when traffic stopped, and hosts running many instances were OOM-killed under bulk ingest. redb, redbMedium and redbLarge were also treated identically, so the "memory~performance" tiers did nothing. Size the cache per tier instead: 4, 8 and 16 MiB per volume, mirroring the Go server's 3/6/12 MiB LevelDB block cache + write buffer. Thread the budget through RedbNeedleMap::new/load_from_idx so every open path (create, reuse, full rebuild) uses Database::builder().set_cache_size. Fixes #11179 Claude-Session: https://claude.ai/code/session_019x36FiSeyePh77YXao15kK Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Eliah RusinandClaude Fable 5.1
|
adbee9452a |
rust volume: bump redb 3.1.3 -> 4.2.0 (#11181)
No source changes: the API surface the needle map uses (Database create/open/builder, set_cache_size, set_durability, tables, iterators) is unchanged and the on-disk format is still v3, so existing .rdb files open as-is. The 4.0.0 breaking changes (Drop on AccessGuardMut, removal of the Legacy type) do not touch this crate. Relevant to the redb-backed index (#11179): - 4.1.0: optimizes cache usage and memory usage; ~1.5x faster writes. - 4.2.0: Durability::None commits ~2x faster; pages freed by a durable transaction are reused by the very next one; a crash-recovery fix for a crash during repair of an earlier crash. Claude-Session: https://claude.ai/code/session_019x36FiSeyePh77YXao15kK Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Chris Lu
|
8a68337256 |
filer: pack SSE chunks into manifests (#11175)
* filer: pack SSE chunks into manifests * s3: resolve encrypted manifests before reads * s3: scope encrypted manifest resolution to ranges |
||
|
Chris Lu
|
97154802c5 |
docs(star-history): make the chart taller (#11173)
Change the matplotlib figure size from (10, 4) to (10, 6) so the star history chart renders vertically longer in the README. The regenerated note/star_history.svg reflects the new 5:3 aspect ratio (720x432pt) instead of the previous flat 2.5:1 (720x288pt). |
||
|
Chris Lu
|
ff0d5a9adf |
ci(mount-windows): clean up processes and don't let Logs step fail job (#11174)
The "Mount and exercise" step left the weed.exe mini server and the final WinFsp mount running when it exited. The next step's pwsh.exe then failed with STATUS_DLL_INIT_FAILED (0xC0000142), failing a job whose actual test step had passed. The same code passed on both the PR branch and the next master run, so this was a transient launch failure — but it was caused by an unclean environment and made fatal by a diagnostic step. Tear down all weed.exe processes at the end of the test step so subsequent steps launch into a clean environment, and mark the Logs step continue-on-error so a diagnostic step can never fail the job on its own. |
||
|
Chris Luanddevin-ai-integration[bot]
|
811b8b5734 |
make the remote-mount cache wait configurable per mount (#11168)
* add a per-mount cache_wait_ms to the remote storage mount mapping A read of an uncached remote-only object waits on a hardcoded size tier before it can fall back to the origin, so every ranged read of a large remote-only object pays that wait. Carry the wait in the mount mapping so it can be tuned, or set to zero, per mount. * resolve the cache wait of an uncached remote-only read from its mount The wait came only from the object size, so an operator could not trade cache hits for time to first byte. Both read paths now resolve the mount covering the object and let its cache_wait_ms replace the size tiers. * read straight from the remote when a mount waits zero for its cache A mount used as a streaming source pays the cache wait on every ranged read of an object too large to finish caching, and the caching itself is wasted work. A zero wait now skips the cache call, so both read paths go to the origin immediately. * let remote.mount set the cache wait of a mount remote.mount -cacheWait=0 turns a mount into a streaming source, and any other duration trades cache hits against time to first byte. * keep the size based wait for a version-specific read A read pinned to a version cannot fall back to the origin, since the mounted remote only holds the current key, so a mount that opts out of caching would leave it on the 503 retry loop forever. * let the operator allow a remote-only read to dial an internal endpoint The remote-mount read paths in the filer and the S3 gateway always refused an endpoint resolving to a loopback or private host, so a mount backed by an internal S3 could never be read from its origin, only through the local cache. Both now take the allowance the volume server already has, still off by default. * skip the background cache of a mount that waits zero for its cache GetObjectHandler kicks off caching for every remote-only read, so a mount serving as a streaming source kept downloading whole objects even though no read ever waited for them. * cover a zero cache wait end to end The read has to reach a real origin, so the harness also opts the filer and the S3 gateway into dialing the loopback remote it already allows for the volume server. * resolve the S3 cache wait once so the background cache follows it too The background cache that GetObjectHandler starts read the mount on its own, so it skipped a version-specific read that the foreground path still waits for. Both now ask the same resolver. * answer 404 when the origin of a zero-wait read is gone Metadata can outlive the object it points at, and with no cache to fill the read would sit on the 503 retry path forever. The remote backends already report a missing object as ErrRemoteObjectNotFound. * open the origin at write time for a multipart range Every part of a multipart Range is prepared before any is written, so opening eagerly would hold one origin connection per part and leak the ones already opened when a later part fails to open. * reject a cache wait shorter than a millisecond The mapping stores milliseconds, so -cacheWait=500us truncated to zero and silently turned caching off instead of waiting. * restore the doc comment of cacheRemoteObjectForStreamingWithShortTimeout Extracting the wait resolver left its comment on the new function. * stat the origin before committing a multipart range Opening at write time keeps no connection through the preparation, but it also moved a failure past the point where the multipart body picks the response status, so a gone origin truncated a 206 instead of answering 404. One stat up front puts the status back. * stat the origin once per request Every part of a multipart Range is prepared on its own, so the preflight ran once per range instead of once per read. * map Azure and GCS stream not-found to ErrRemoteObjectNotFound ReadFileAsStream on Azure and GCS returned provider-specific not-found errors instead of ErrRemoteObjectNotFound, so a zero-wait read of a deleted object was misclassified as a transient cache failure and retried indefinitely. Map BlobNotFound and ErrObjectNotExist the same way StatFile already does. * Update weed/remote_storage/gcs/gcs_storage_client.go Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
f79d83abf4 |
volume: expire TTL volumes whose only traffic is deletes (#11167)
* volume: count a TTL volume's age from its last write, not the .dat mtime A delete appends a tombstone needle and vacuum rewrites the .dat wholesale, so the file's mtime moves without any write ever landing. The loader read lastModifiedTsSeconds back from that mtime, so every restart of a volume taking delete traffic re-armed expired() for another full TTL: an overwrite-heavy collection kept growing until it hit the max-volume cap. Recover the clock from the newest .idx entry that is not a tombstone and read that needle's append timestamp, falling back to the mtime when no write is recoverable. Only TTL volumes pay for the scan. Fixes #11160 * volume: count the .vif destroy time from the last write too ExpireAtSec is what an EC volume is reclaimed on, and it was recomputed as now+TTL every time the .vif was written. A read-only mark, a tier upload or an EC encode therefore handed an already expiring volume another full TTL, the same way the .dat mtime did. Derive it from the volume's last write, falling back to now for a volume that has not taken one yet so a fresh volume is not born expired. * volume: mirror the last-write TTL clock in the Rust volume server Same recovery as the Go loader: scan the .idx backwards for the newest entry that is not a tombstone and take that needle's append timestamp, leaving the clock on the .dat mtime when no write is recoverable. * volume: mirror the last-write destroy time in the Rust volume server Both .vif writers and the EC encode computed ExpireAtSec as now+TTL, the same way Go did, so the destroy time moved every time the sidecar was rewritten. Route all three through the volume's last write. * volume: report the .dat mtime in the Rust heartbeat, like Go does The Rust server reported its TTL clock as ModifiedAtSecond while Go reports the .dat mtime. The shell's quiet-period gates (volume.tier.move, volume.delete_empty) read that field as "last touched", which a delete has to count towards even though the TTL clock deliberately ignores it -- and with the clock now recovered from the last write, the two drift further apart. * volume: take the newest write by timestamp on a vacuumed volume The reverse .idx scan trusted position, which holds only while the .dat is append ordered. Vacuum rewrites it in key order, and since an overwrite keeps its original key, the highest-key survivor is not necessarily the newest write -- the recovered clock could land up to a TTL early and take the volume with data still inside its TTL. A volume that has been vacuumed (CompactionRevision > 0) now takes the maximum append timestamp over a bounded window of write entries instead. An append-ordered volume still answers in one read. * volume: never guess a vacuumed volume's last write, and resolve wrapped offsets Two holes in the reverse scan, both from review: A vacuumed volume's writes are ordered by key, so any of them can hold the newest timestamp. Reading a capped window sampled the highest keys, which could still miss a recently overwritten low-key needle and expire data inside its TTL. The scan now covers every write a vacuumed volume indexes, and a volume too large to scan keeps the .dat mtime rather than report a partial maximum -- late is recoverable, early is not. A .dat past MaxPossibleVolumeSize wraps the offsets in its .idx, so reading a timestamp at the unwrapped offset picks up an unrelated needle. Resolve the entry against the needle header first and retry one volume size in, the way doCheckAndFixVolumeData already does. * volume: drop GitHub issue references from TTL comments |
||
|
Chris Lu
|
567578f08d |
docs(readme): replace star-history.com with self-generated chart (#11171)
* docs(readme): replace star-history.com with self-generated chart The star-history.com SVG is a third-party dependency that can rate limit or go down. Replace it with a GitHub Action that fetches stargazers via the REST API and renders an SVG with matplotlib, committing note/star_history.svg weekly. The README references the committed file directly, so the chart has no runtime dependency on any external service. * ci(star-history): run daily instead of weekly |
||
|
Chris Lu
|
070174726d |
docs(readme): replace rate-limited starchart with star-history (#11170)
starchart.cc is rate-limiting the SVG endpoint, so the Stargazers chart renders blank. Switch to star-history.com, which serves a live SVG for this repo and links to the interactive chart. |
||
|
Chris Lu
|
e4dc66c66b |
docs(readme): move sponsor section to the end (#11169)
The Patreon CTA and Gold Sponsors logos sat between the logo and the project intro, pushing the actual description below the fold. Move the whole block to a dedicated `# Sponsors #` section after `# License #`, add it to the TOC, and give it a real markdown heading so the anchor works on GitHub. |
||
|
Chris Lu
|
1ca19ea2e2 |
mount: add -volumeName to name the disk explicitly (#11165)
* mount: let volumeName take an explicit override volumeName only ever derived the disk's label from -filer.path, -dir, or the filer address, so a name that happened to collide with something else - e.g. a UNC share's own name - could not be changed without moving what was mounted. Give it an override parameter that wins over all three; nothing passes one yet. * mount: add -volumeName to name the disk explicitly Windows has no equivalent of the "weed fuse" -o passthrough that lets a Linux or macOS mount override its derived volname, so a name picked up from -dir - e.g. a UNC share's own name - could not be changed short of moving what was mounted. -volumeName overrides it on every platform. * mount: document -volumeName * mount: scope -volumeName's help text to macOS and Windows Linux has no volume-label mount option for -volumeName to feed, so the flag's own description says where it applies instead of leaving that unstated. * mount: forward -volumeName through the weed fuse option parser weed fuse (the /etc/fstab helper) turns -o key=value into the same MountOptions weed mount takes, but volumeName had no case, so it fell through to being forwarded as a literal, unrecognized FUSE option instead of ever reaching mountOptions.volumeName. * mount: apply -volumeName to FsName on Linux and FreeBSD FsName only ever took the filer address and -filer.path, so -volumeName had nothing to override there and silently did nothing; the skipAutofs case still forces "fuse", since that name is what util-linux/mount requires to recognize the pseudo filesystem. |
||
|
Eliah RusinandChris Lu
|
cfa8afec92 |
filer: guard FoundationDB 100KB value limit and pack earlier (#11161)
* filer: guard the FoundationDB value size limit, not the transaction limit An entry's whole chunk list is one FoundationDB value, and FDB caps a value at 100,000 bytes while a transaction may reach 10MB. UpdateEntry checked the transaction limit, so every entry between the two limits passed the guard and was rejected by FDB itself with error 2103 (Value length exceeds limit). The failure surfaced inside the store rather than at the guard, so the S3 layer dropped the connection and clients saw a network fault instead of an error. Check the value limit in UpdateEntry and KvPut instead, after gzip and before the transaction, with an error that names the limit it hit. The removed transaction-size constant guarded nothing else: DeleteFolderChildren batches by entry count. Refs #11158 * filer: fold at 500 chunks in the foundationdb build Manifest packing is what keeps a large file's entry small, but it only ran once a flat chunk list reached 10000 chunks. A FoundationDB value stops at 100,000 bytes and an entry's whole chunk list is one value, which at ~100 bytes per chunk record is about 1000 chunks -- so on FDB the write always failed before packing could help: a 3.3 GiB PutObject at the default -maxMB=4 was already past the limit. FoundationDB support is its own build (`go build -tags foundationdb`, shipped as its own image), so the batch is a build-time choice and needs no negotiation at run time. The tagged build folds at 500, every other build keeps 10000 and is untouched. 500 is not arbitrary: a single fold level leaves (chunks/batch) manifest pointers plus up to (batch-1) unfolded chunks in the entry, so the reachable chunk count is highest when the two terms are near equal. For a 100,000-byte budget that optimum is 500, which holds an entry inside the limit up to ~250,000 chunks -- ~1 TB at -maxMB=4, against ~4 GB before. Larger files need nested packing, which no batch size substitutes for. One binary serves every role in that image, so the filer and each client that folds -- S3, mount, WebDAV, weed shell, filer.copy -- agree on the batch by construction. A binary built with the tag but pointed at another store folds earlier than that store requires, costing one manifest blob per 500 chunks and one read to resolve it. Fixes #11158 * filer: fold with rollback inside MaybeManifestize, not beside it A fold that fails midway has already uploaded manifest blobs for its earlier batches, and returns only the data chunks -- dropping the manifests it had separated out of the caller's list. Both were wrong in ways that mattered: - AppendToEntry assigned that shortened list straight to entry.Chunks and created the entry, so an append to an already-folded file whose fold failed lost every previously folded chunk. weed mount had the same shape. - cleanupChunks logged the error as "not good, but should be ok" and then returned it through a named result, failing the whole CreateEntry or UpdateEntry, while the blobs it had written stayed behind referenced by nothing. The S3 path was alone in handling this, through a private helper beside MaybeManifestize. A second entry point next to the one everything else calls just means the wrong one gets used, so the behaviour moves inside MaybeManifestize: on failure it returns inputChunks as it received them, and hands the blobs it saved to a deleteChunks callback. The filer, S3 and filer.copy pass their existing deleters -- filer.copy already cleans up this way after a failed upload -- and mount, WebDAV and weed shell pass nil, which reports the blobs rather than collecting them, as before. Each caller keeps its own error policy: the filer HTTP PUT path and filer.copy still fail the request, the rest still continue with the flat list, which is a correct entry. --------- Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
5a515adab2 |
s3: HeadObject with partNumber returns the part's size and 206 (#11166)
* s3: HEAD with partNumber reports the part's size and range HeadObject set its headers from the total object size and then only validated the partNumber, so a client probing part 1 with HEAD got the whole object's Content-Length and a 200 while the same GET returned the part's size, a Content-Range and a 206. Resolve the part's byte range before the headers are written, through the range logic GetObject already used, and answer a partNumber HEAD as the ranged HEAD that AWS documents. * s3: answer an unsatisfiable partNumber with 416 InvalidPartNumber GET and HEAD rejected a partNumber past the number of parts with 400 InvalidPart, the code for a missing part in CompleteMultipartUpload. AWS answers a read of a part that does not exist with 416 InvalidPartNumber, which lets a client probing for the part count tell the two apart. The ceph suite pins RGW's 400 InvalidPart here, so the s3tests jobs patch that expectation the way they already patch prefix ordering. * s3: keep the whole-object checksum off a partNumber response The stored checksum covers the whole object, so it is already withheld from a ranged read. A partNumber HEAD now describes one part while the request carries no Range header, so exclude it there too rather than handing a client a checksum that does not match the bytes described. * s3: resolve a partNumber against the parts the object records Completion accepts ascending, not consecutive, part numbers, so the part count is not the highest part number. Comparing the two rejected an uploaded part 3 of a two-part object, and let a request for the absent part 2 fall through to the positional chunk lookup and serve part 3's bytes. Ask the recorded boundaries for the part instead, and keep the count comparison for objects written before boundaries were stored. * s3: apply a client Range within the part on HEAD too GET narrowed the part by a Range sent alongside partNumber; HEAD reported the whole part, so the two disagreed again for a request that carries both. Move the narrowing into the shared range lookup so either verb describes the same bytes. |
||
|
MorezMartinandAthena 🏛️
|
27b2411cdd |
fix(chart): add missing [grpc.s3] TLS section — S3 internal gRPC served plaintext while peers dial mTLS (#11157)
* fix(chart): serve S3 internal gRPC with mTLS when security enabled The security.toml generated by the chart has no [grpc.s3] section, so security.LoadServerTLS(viper, "grpc.s3") returns nil in weed/command/s3.go and the S3 server listens plaintext on its gRPC port (httpPort+10000 = 18333 by default). Workers dial that port with mTLS credentials (grpc.worker), producing: walker dispatch ...: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: first record does not look like a TLS handshake" This breaks the s3_lifecycle worker's LifecycleDelete RPC path (recovery walk, daily replay) and any S3->S3 IAM cache propagation would fail the same way if clients enforced TLS. Add [grpc.s3] reusing the client cert already mounted on s3 pods (or s3.tlsSecret when set, mirroring the seaweedfs.s3.tlsArgs helper for the HTTPS listener). Co-Authored-By: Athena 🏛️ <hermes-agent@local> (custom / Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf) * fix(chart): always use internal client cert for grpc.s3 identity s3.tlsSecret is the public HTTPS listener certificate (possibly issued by a public CA); internal gRPC peers only trust grpc.ca, so presenting it on the internal gRPC port would break lifecycle/IAM RPC verification. Keep the two trust domains separate. Co-Authored-By: Athena 🏛️ <hermes-agent@local> (custom / Qwen3.8-Flash-Next-ROCmFP4) |
||
|
Lucas Frederico Roeder de MelloandClaude Opus 5
|
98115ec2de |
deps: update golang.org/x/image to v0.45.0 for CVE-2026-46603 (#11164)
golang.org/x/image v0.44.0 is affected by CVE-2026-46603 (GO-2026-6222): a denial of service via excessive memory allocation when decoding malformed VP8L (lossless WebP) data. It is fixed in v0.45.0, released 2026-08-11. The decoder is reachable from SeaweedFS: weed/images/resizing.go blank-imports golang.org/x/image/webp, which registers the VP8L decoder with image.Decode, so the filer image resizing path decodes attacker supplied WebP data with the affected version. This is a go.mod/go.sum only change produced by `go get golang.org/x/image@v0.45.0 && go mod tidy`; no other dependency moved. `go build ./weed/`, `go vet ./weed/images/...`, `go test ./weed/images/...` and `go mod verify` all pass. Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
Chris Lu
|
a0b1272cc3 |
filer: authorize the chunk proxy and the root listing like the rest of the filer port (#11152)
* filer: require a read token for the root listing maybeCheckJwtAuthorization waved through every GET/HEAD on "/", so a filer with jwt.filer_signing.read.key set still served its root directory listing -- entry names, sizes and chunks[].file_id -- to a caller holding no token at all, and served the same listing to a token restricted by allowed_prefixes. The exemption was added for health checks before the filer had /healthz and /readyz. Both are registered on the default and read-only muxes ahead of the "/" handler and answer without a token, so drop it. Point the mTLS harness at /healthz, which is what it was probing for. * filer: keep the jwt query parameter out of a proxied chunk request The proxy stripped "jwt" from the forwarded query on reads only, on the grounds that a writer's own credential travels there. It does not: an uploader carries its AssignVolume token in the Authorization header, and the query parameter on this path holds a filer credential. Strip it for every method. A volume server has no business seeing a filer token, and because security.GetJwt reads the query before the header, relaying one would hide the writer's own token behind it. * filer: dispatch the chunk proxy after the JWT gate The ?proxyChunkId= branch returned before maybeCheckJwtAuthorization ran, so GET, PUT, POST and DELETE against any needle in the cluster were reachable on the filer's HTTP port with no filer credential, on a filer where every other request answered 401. An anonymous caller read a stored object, replaced its bytes, or deleted the needle, which the master's next vacuum makes permanent. #10434 stopped the filer from minting a volume write token for that caller, which closes the write half only where the volume server has a jwt.signing.key of its own -- not the shipped default, and not what scaffold/security.toml recommends for a filer deployment. The read half stayed open in every configuration, because the filer mints the read token itself. Move the dispatch below the gate. A file id carries no path, so a token restricted by allowed_prefixes cannot be scoped against one and is refused here; every consumer of this endpoint holds an unrestricted token. * filer: mint the volume credential for a proxied write too The proxy minted a volume token on reads and forwarded whatever the caller sent on writes. #10434 made it that way because the branch ran ahead of the JWT gate, so a token minted here would have been signed for an unauthenticated caller; the branch now runs behind the gate, and the credential the caller presents there is a filer one, which a volume server cannot validate and has no business seeing. Mint at the access level the request needs, and drop the caller's Authorization when there is no key to mint from. A proxied uploader then needs only the filer credential, instead of holding one for each hop with a single header to put them in. * mount, mq, filer.sync: send the filer credential for a proxied chunk Every in-tree consumer of ?proxyChunkId= reached the filer anonymously: mount and the broker put the AssignVolume token in the Authorization header, which is a volume credential, and filer.sync sent nothing at all. That was enough only while the branch ran ahead of the filer's JWT gate. Build the URL through one helper, and pick the credential from the URL it returns: a chunk proxied through a filer is a request to the filer, which authorizes it and attaches the volume credential itself, so the token there is a filer one at the access level the request needs. * filer: honor -exposeDirectoryData The flag was declared on all three commands that start a filer and read by none of them: FilerOption.ExposeDirectoryData was only ever assigned from filer.expose_directory_metadata in security.toml, so -exposeDirectoryData=false silently left the listing exposed. Only the TOML key had any effect. Plumb the flag through and let either switch turn the listing off. * filer: count a proxied chunk request once Moving the dispatch below the gate put it after the deferred request observation, so every proxied chunk now landed in FilerRequestHistogram twice, once under its HTTP method and once under chunkProxy. Name the deferred one after the proxy instead, the way the unsupported-method branch already does, which also gives the endpoint the status codes FilerRequestCounter records. |
||
|
Chris Lu
|
cda43f1976 |
filer: do not 404 a TUS session on a transient chunk-load failure (#11153)
* filer: do not 404 a TUS session on a transient chunk-load failure readTusSessionInfo already proved the session exists before loadTusSessionChunks is called, so a failure there is a read failure, not evidence the session is gone: a volume-server timeout or a canceled request context surfaces through ListDirectoryEntries the same way a missing session would. Every such error was mapped to writeTusSessionNotFound, answering 404 to HEAD/PATCH and 204 to DELETE. A spec-compliant TUS client trusts that and discards the session, orphaning every chunk it had committed until the 24h expiry sweep, or forever if it never issues a DELETE. Only an error matching filer_pb.ErrNotFound is now reported as not found; anything else answers 500 so the client retries against the same session instead of abandoning it. * test: cover a TUS session's transient chunk-load failure Adds a listErr hook to the in-memory test store, alongside the existing commitErr/deleteErr, to simulate a store or RPC failure from ListDirectoryEntries. HEAD, PATCH and DELETE against a live session all answer with a server error instead of a not-found status when the chunk listing fails transiently, and the session is left on disk untouched. A listing failure that genuinely means not found, filer_pb.ErrNotFound, still answers 404 (204 for DELETE). |
||
|
Chris Lu
|
ed9d58873e |
filer.remote.sync: skip an upload whose source entry was deleted or rewritten (#11149)
* filer.remote.sync: skip an upload whose source entry was deleted or rewritten A replay from an earlier offset (-timeAgo) re-emits create and update events for entries the filer has since deleted or rewritten. Their chunks are gone from the volume servers, so the upload can never succeed, and failing the event holds the sync offset before it: every restart of the subscription replays it into the same dead chunks, and progress on everything after it in the log is never persisted. One such entry stops replication for the whole mount. When the upload fails, look the entry up on the filer. Gone, or holding other content than the event described, the event is superseded and is skipped with an error log; the event that superseded it follows in the log and brings the remote to the current state. Otherwise the failure stands and the event is retried as before. Fixes #11148 * filer.remote.sync: compare chunks by file id when deciding an event is superseded filer.IsSameData compares chunk ETags, so a delete-and-recreate of identical bytes, which stores the same content under new file ids and drops the old ones, looked still as described and kept failing the event on its dead chunks. Compare by file id with DoMinusChunks, the way the filer itself decides which chunks an update leaves for deletion: the event is superseded when the current entry no longer references every chunk it named, and still as described when it does, including when more chunks were appended after it. * filer.remote.sync: ask the filer on the first failed upload attempt, not after the backoff The superseded check ran after util.Retry had given up, so every dead entry still cost the full retry cycle, about 13s, before it was skipped: the SDK reports a missing chunk as "RequestError", which IsTransientError takes as worth retrying. Move the check into the retry loop with util.RetryOnError. Any failed attempt asks the filer, and the loop stops at once when the entry is gone, surfacing errSuperseded for the caller to skip. An entry the filer still holds keeps the retry policy it had. filer.remote.gateway shares retriedWriteFile and the same offset-pinning processor, so its three call sites skip a superseded event the same way. |
||
|
Chris Lu
|
06838e28b2 |
filer: serve "//" paths at the cleaned path instead of redirecting (#11150)
* filer: serve "//" paths at the cleaned path instead of redirecting
http.ServeMux redirects a non-canonical path ("//", "..") to its cleaned
form, but since Go 1.22 it builds the Location from the already-escaped
path, so it is percent-encoded twice (golang/go#79897). A client that
follows the redirect re-posts "/负极全景" as "/%25E8%25B4%259F...", and
the filer stores a directory literally named "%E8%B4%9F...".
Wrap the filer muxes in CleanPathHandler, which rewrites the request to
the same cleaned path ServeMux would have redirected to and dispatches
directly. The decoded name reaches the handler, the round trip goes
away, and clients that do not follow redirects work too.
Fixes #11125
* filer: keep RequestURI in step with the cleaned path
PostHandler derives storage rules, the bucket and the read-only check from
r.RequestURI while writing the entry at r.URL.Path. After CleanPathHandler
rewrote only the URL, a "//" or ".." request would be placed by the raw
path and written to the cleaned one. Rewrite RequestURI too, as the
redirect-following client used to.
* filer: match storage rules on the decoded write path
PostHandler resolved the storage rule from r.RequestURI, the raw
request-target. Clients percent-encode non-ASCII segments on the wire, so
a read-only or TTL rule configured on "/data/只读/" never matched a POST
to "/data/%E5%8F%AA%E8%AF%BB/" and the write went through. Use r.URL.Path,
the decoded path the entry is actually written to, as the header-based
destination check already does. The query string no longer reaches the
rule lookup, so the "?" trimming in the read-only error is gone.
|