3 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
孙超	c80664ec21	s3: propagate storage rule fsync to volume server uploads (#10906) ... The storage rule's fsync decision was computed by the filer (detectStorageOption -> rule.Fsync) and applied on the filer's own HTTP write path, but was never carried onto the chunk uploads S3 issues: the AssignVolumeResponse had no fsync field, so the s3api client could not learn the decision, and the chunked upload URL was hardcoded without it. Every S3 write to a path with fsync configured went to the volume server as a non-fsync write. Carry the decision through the assign response: - filer.proto: AssignVolumeResponse gains bool fsync, filled from the storage option the assign resolved. - operation.AssignResult gains Fsync, so uploadChunk can append ?fsync=true to the volume server upload URL (single and replica fan-out paths). - The S3 PUT/UploadPart assignFunc, the S3 copy path, the admin file browser upload, and the Iceberg worker assign functions all forward the response field. Adds TestUploadReaderInChunksAppendsFsyncWhenAssigned.	2026-08-23 22:11:08 -07:00
3 people	f7c4636d22	topology: refresh oversized mark on every heartbeat (#10829) ... * topology: refresh oversized mark on every heartbeat The oversized flag on a volume location was only set when the volume was registered (RegisterVolume). A volume that later grew past the size limit kept its stale "not oversized" mark, so the heartbeat path (ensureCorrectWritables) kept re-adding it to the writable list while RecordAssign removed it on every assign - a writable/unwritable flip loop that let writes continue past the limit and made vacuum race in-flight writes. Refresh the mark from each heartbeat's reported size in both heartbeat paths (ApplyVolumeChanges and SyncDataNodeRegistration), mirroring what RegisterVolume already did at registration time. A volume that grew past the limit now stays unwritable, and one that shrank back clears the mark and can recover. * topology: order heartbeat writable correction after decay and honor cooldown Review feedback (Greptile, CodeRabbit) on the oversized-mark refresh: 1. Greptile: clearing the oversized mark before EnsureCorrectWritables let the delay-unaware helper re-add a just-compacted volume to writables, bypassing capacityRecoveryDelay. ensureCorrectWritables now checks fullSince and skips the re-add while the cooldown is pending, so a volume removed for capacity only recovers through UpdateVolumeSize's heartbeat recovery path. 2. CodeRabbit: in the full-heartbeat path the mark was refreshed after the writable correction, so a newly oversized volume stayed writable for an extra heartbeat cycle. The standalone changedVolumes loop is merged into the volumeInfos loop and EnsureCorrectWritables now runs after UpdateOversizedState + UpdateVolumeSize in both heartbeat paths, using the freshly refreshed mark. 3. TestHandlingVolumeServerHeartbeat used a size (254320) that is past the test's volumeSizeLimit (32768); it only passed because the stale mark hid the oversized state. Sized down to 30000 to keep testing the add/remove flow, and added TestEnsureCorrectWritablesHonorsRecoveryCooldown covering the cooldown window and the recovery after it. * topology: do not restore a still-crowded volume after the cooldown Greptile review: after capacityRecoveryDelay elapses, ensureCorrectWritables could restore a volume whose effective size is still past the crowded threshold. UpdateVolumeSize refuses the recovery (effectiveSize > crowded threshold -> setVolumeCrowded + return false), but the cooldown check in ensureCorrectWritables only looked at fullSince, so once the delay passed it re-added the volume even though capacity tracking still considers it crowded. Check the crowded mark before re-adding: a volume UpdateVolumeSize just marked crowded must not be restored here, otherwise assignments resume while the volume is still flagged for growth. Adds TestEnsureCorrectWritablesDoesNotRestoreCrowdedVolume: effectiveSize decays to 10500 (past the 9000 crowded threshold) after a report of 8000, and ensureCorrectWritables keeps the volume unwritable past the cooldown. * ci: trigger re-run of flaky FUSE jobs * topology: gate the writable restore on the limit, not on crowded A crowded volume is above the growth threshold, not full, and is normally writable. Refusing to restore one locks it out for good: nothing writes to a volume that is not writable, so its size can never fall back under the threshold. Gate on the same size the assign path uses to remove it. * topology: let only the heartbeat refresh set the oversized mark Registration also set it, from whatever VolumeInfo it was handed. The incremental path builds that from a short heartbeat message, which carries no size, so every arrival announcement cleared the mark and handed the volume back to the writable list until the next full report. * topology: use the re-resolved layout after a dropped one is replaced A layout dropped with its collection makes RegisterVolume refuse, and the full heartbeat then re-registered against a fresh layout but kept applying the size, oversized and writable updates to the dropped one. --------- Co-authored-by: hzsunchao <hzsunchao@corp.netease.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-08-20 09:56:53 -07:00
3 people	9575032b4c	volume: forward fsync=true to replicas in ReplicatedWrite (#10805) ... * volume: forward fsync=true to replicas in ReplicatedWrite When a write request carries fsync=true, only the primary volume server flushed to disk: the replica fan-out URL in ReplicatedWrite only carried type/ttl/ts/cm, so replicas always wrote without fsync even when the client explicitly requested a durable write. Forward the fsync request parameter to the replica volume servers so a durable write means every replica has flushed to disk, not just the primary. Replicas without fsync are untouched (zero behavior change). * storage: flush a durable write inline while stopping The fsync flag on the write path really selects the async batch worker, and it was switched off once the store is stopping. So a fsync=true write landing during the pre-stop drain got acked without ever being flushed - and now that ReplicatedWrite forwards fsync, that covers replicas too. Flush it inline instead of queueing it. The drain keeps accepting writes, which is the whole point of preStopSeconds, and the ack still means the .dat is on disk. If the fsync fails, the append comes back off the .dat and the needle map goes back to what it pointed at before, so nothing resolves to an offset past the truncated end. * storage: make the store's stopping flag atomic SetStopping runs on the signal handler goroutine while the write and vacuum paths read the flag, so every read of it was racy. Nothing about the shutdown ordering changes; only the flag itself is now safe to read. * topology: check the errors the replication test was dropping The mock replica ignored its response write and the mock master ignored whatever Serve returned, so a broken mock would have shown up as a confusing timeout rather than a failure. Also drops the explicit listener close: grpc.Server.Stop already closes the listener it was given. --------- Co-authored-by: hzsunchao <hzsunchao@corp.netease.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-08-18 17:17:50 -07:00