mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-18 09:46:24 +01:00
eb1dca808897090af06327cd05ac0b7e3c9ff12f
16280
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
eb1dca8088 |
filer: release the stamps of parent directories a rename creates (#12711)
* filer: release the stamps of parent directories a rename creates * filer: test that a suppressed create keeps the moved entry's stamp in flight Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
deacd575ea |
operation: prepare upload content before assigning the volume (#12714)
* operation: prepare upload content before assigning the volume * operation: keep the existing upload function names for prepared payloads UploadData and doUpload prepare once and hand the prepared payload to retriedUploadData, so the retry loop, doUploadData, uploadChunk and uploadChunkToHolders take a preparedUpload directly instead of growing a prepared twin of each and branching on nil. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * operation: send ciphertext whenever the prepared payload carries a cipher key doUploadData chose the cipher branch from the option, so a payload prepared under a different option could be sent as plaintext metadata or lose its key. Decide from the prepared payload itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * operation: build a chunk's preparation and upload options in one place The chunk loop prepared with a hand-written option while uploadChunk and uploadChunkToHolders built their own, so a setting that shapes the bytes could reach one and not the other. chunkUploadOption now feeds all three. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * operation: test that a reassigned chunk reuses its prepared payload Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
194e6ede9f |
seaweed-volume: reject FetchAndWriteNeedle reads that miss the requested size (#12717)
The Go volume server fails the RPC when the remote read returns a different length than req.Size; the Rust server cached whatever came back as a full-size chunk. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
97f49571a1 |
wdclient: let master-pushed adds replace the gateway's lookup-fed locations (#12705)
* wdclient: let master-pushed adds replace the gateway's lookup-fed locations After a master leader change the S3 gateway clears its FilerClient cache and refills it through filer lookups. The filer's own master client may still answer from the previous leader's generation until the new leader has heard from the volume servers, so a volume server lost together with the old leader is cached again. The new leader never knew that server and never pushes its removal, and the gateway keeps trying it until reads fail on it volume by volume, one 30 s dial timeout at a time. The gateway's master client relearns every volume from the new leader's adds. On each add, cache the volume where the master client now knows it to be, in place of what a lookup cached, and keep lookups already in flight from caching their answer over it. A provider answer for a volume is now cached whole or not at all. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * wdclient: take a lookup's fill epoch before its second cache check A master add that landed between the second cache check and the read of the fill epoch was invisible to both: the check had already missed, and the epoch the lookup then read already included the confirmation, so a filer answer that still named the lost server was merged back into the confirmed entry. Taking the epoch first means such an add either shows up in the check or stops the answer from being cached. The same order covers a reset or clear in that gap. TestConfirmDuringLookupIsNotUndone fires a confirmation from a test hook at that point and fails with the old order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * wdclient: gate the master lookup provider's cache writes on the map epoch The provider wrote each answer straight into the vid map, so a lookup sent before a master reset and answered after it landed in the new generation, which a gateway's FilerClient now reads as the current master's view. The answer now goes through fillLocations, which drops it once a reset passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * wdclient: keep the vidMapClient doc comment on its type Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
|
91eaabf371 |
s3: answer ClientDisconnected for a GET the client aborted before the first byte (#12707)
* s3: answer ClientDisconnected for a GET the client aborted before the first byte Fixes #12706 * s3: tell a gone client on TLS and HTTP/2 too, and not on a connection the server closed |
||
|
|
c91da3d1f8 |
filer: fail health check when absent from lock ring (#12709)
* filer: fail health check when absent from lock ring If the master broadcasts a lock ring that does not include this filer (e.g. the late-cleanup race in seaweedfs#11442), every lock acquisition fails with "no lock server found" and locked writes, including S3 PUTs, hang indefinitely while /healthz and /readyz still report OK. Make the filer health handler return 503 once this filer has been missing from its own lock ring for longer than a 30s grace period, so Kubernetes liveness probes restart it and it re-registers with the master. The grace period covers startup before the first ring update and the master's ring-change batching delay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: hold the lock-ring health clock until the filer announces AnnounceOnceCaughtUp can hold the master registration for up to a minute while HTTP already serves, so the filer cannot be in the ring yet. Report healthy until the announce gate opens and start the grace period after it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: keep a probe racing a ring rejoin from reading a reset clock A probe that loses the CompareAndSwap to a concurrent healthy probe could load the reset zero and report the filer absent since the epoch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: trim the lock-ring health comments Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: restart the lock-ring health grace when a ring update brings the filer back The clock only reset when a probe saw the filer in the ring, so a rejoin and a second removal between two probes reused the first absence and failed the next probe early. Reset it from the ring snapshot callback. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: test the announce gate leaves the lock-ring health clock unset Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: re-read lock-ring membership after starting the health clock A probe whose first read raced a rejoin could start the clock after the snapshot callback reset it, leaving a stale start for the next removal. The ring is installed before the callback runs, so a second read sees the rejoin whenever the reset came first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: treat a health clock changed during the probe as a rejoin A rejoin and a second removal can both land between the probe's two membership reads. The second read then shows the filer absent again, and the probe judged the new absence by the expired start it had read before the rejoin's reset. The snapshot callback resets the clock before the next ring update is installed, so a clock that no longer matches the probe's start means a rejoin happened; report healthy and let the next probe use the fresh clock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: test ring updates that land during a lock-ring health probe Move the absence clock into lockRingAbsence, which takes the membership read as a function, so the test can install ring updates between the probe's reads: a rejoin after the first read, and a rejoin plus removal between the two reads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: give every lock-ring health reset a new clock value A probe that saw the filer absent started the clock with a CAS from 0, so a rejoin that reset the clock to 0 again before that CAS was invisible, and the probe could install a start time from before the rejoin. Each reset now stores a new non-positive value and the probe swaps from the value it loaded before the membership read, so any rejoin during the probe makes the CAS fail. This drops the second membership read and never waits on the LockRing mutex. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: test lock-ring rejoins before and after the probe's membership read Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: keep the lock-ring version lookup out of the unhealthy health path LockRing.Version takes the ring mutex, which SetSnapshot holds while the snapshot callback transfers locks, so an expired-absence probe could wait on those transfers and time out instead of answering 503. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: test an expired-absence health probe while the ring mutex is held Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
|
c76fb36416 |
test(fuse): two mounts staging the same files without -dlm (#12704)
* test(fuse): two mounts staging the same files without -dlm Two jobs that extract the same archive into one shared directory, each through its own mount and without -dlm, both create many of the same files: a file that exists is skipped, otherwise it is created, written and closed, then chmod'ed and given its mtime. Before #12593, #12594 and #12596 the attribute update and the content flush sent the mount's whole cached entry. Sent after the other mount had rewritten the file, the update put back chunks that the rewrite had dropped and the filer had deleted, so the entry pointed at deleted needles and the file failed to read once its volume was vacuumed. Nothing told the writers. TestStageSameFilesWithoutDLM runs that on the fuse_dlm cluster with the mounts started without -dlm (startTestCluster takes a dlm flag; startDLMTestCluster is unchanged) and checks every staged file: each chunk is still on the volume server (HEAD on the fid, without readDeleted) and the filer serves the bytes written. What the losing writer is told depends on the version and is only logged. With a 4.53 weed binary the test fails: 66, 71 and 70 of the 400 files were broken in three runs. With master it passes in about 48 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(fuse): bound the staging test's filer and volume checks The checks used the default HTTP client and a background context, so a server that accepts a request and stops answering held the test until the package timeout instead of failing the check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(fuse): fail the staging test unless both mounts wrote the same files A mount whose MkdirAll failed returned early, and a mount that started late could skip every file, so the test passed with one writer. Start both loops together, fail on setup errors, and require files both mounts wrote. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(fuse): bound the staging test's checks with one deadline A per-call timeout still let 400 hung lookups and reads run for hours. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
|
0d3f5a220c |
s3: keep serving for -preStopSeconds after SIGTERM (#12703)
* s3: keep serving for -preStopSeconds after SIGTERM On SIGTERM the gateway closed its listeners at once, so requests a load balancer or Kubernetes Service still routed to it failed until the endpoint was removed. With -preStopSeconds set, the gateway makes /status, /healthz and /readyz return 503, keeps serving for that many seconds, then runs the existing graceful shutdown. 0 (the default) keeps the old behavior. The flag is registered for weed s3 and as -s3.preStopSeconds for weed filer, weed server and weed mini. A cancelled weed mini cluster context ends the wait early. The Helm chart passes s3.preStopSeconds and filer.s3.preStopSeconds through. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: assert the preStopSeconds chart render under set -e, drop timing from the probe test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: wait before weed mini cancels its clients; size grace periods in the chart weed mini's interrupt hook runs before the S3 one and cancels the S3 gateway's shutdownCtx, so the pre-stop wait never happened there. The wait is now also registered with beforeMiniClientsShutdown, which runs before that cancel, and a sync.OnceFunc keeps the interrupt hook from waiting a second time. Outside weed mini the registration is a no-op. Helm: the s3 deployment fails to render when terminationGracePeriodSeconds is below preStopSeconds + 15. The filer statefulset and the all-in-one deployment, whose grace period is fixed at 60s, add preStopSeconds to it. allInOne.s3.preStopSeconds is passed through, inheriting s3.preStopSeconds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: keep 64-bit counters aligned on 386; let allInOne.s3.preStopSeconds=0 disable the wait draining sat ahead of inFlightDataSize and inFlightUploads in S3ApiServer and shifted them off 8-byte alignment on 32-bit builds, so the atomic upload-limit counters panicked there. It now follows them. Helm: allInOne.s3.preStopSeconds inherits s3.preStopSeconds only when null; default treated an explicit 0 as unset. The s3 values and the grace period error now say a lifecycle preStop hook's time comes on top. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * grace: add OnInterruptFirst for hooks that must run before any shutdown Interrupt hooks run in reverse registration order, so a hook cannot run ahead of a service that registers later, such as a volume server that is still loading its volumes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: run the -preStopSeconds wait before every other interrupt hook Under weed server the volume server registers its hook once its volumes are loaded, often after the S3 gateway. Its hook then ran first and shut the volume server down, so the gateway spent the whole wait serving requests that could not read or write objects. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
|
c4e1802e3e |
s3: route ';' queries as authorized, and report an empty s3:prefix to policies (#12702)
* s3: escape raw semicolons before the router matches a route registerRouter applied EscapeSemicolonsInQuery with router.Use, but mux runs router middleware only after it has picked the route. Queries() matchers split pairs on ';', so GET /bucket?uploads;x=1 reached the multipart listing handler while the handler and the IAM resolver read uploads%3Bx=1 and saw no uploads subresource. Wrap the S3 router with the escape so routing and authorization read the same query. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: report a listing without prefix as s3:prefix "" in policy conditions The advanced IAM path already sets s3:prefix to "" when a bucket listing names no prefix, as AWS does, but bucket policies and attached policies read conditions from ExtractConditionValuesFromRequest, which leaves the key out. The AWS home-directory policy that grants s3:prefix ["", "home/"] then denied the root listing, and a Deny keyed on the empty prefix never applied. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: test the served handler composition for the semicolon escape Move the escape and the Vectors split into s3Handler so a command test can send ';' queries through the handler the server actually serves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: keep the isVectorsRequest doc comment on its function Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
1d30b0de21 |
fix(s3api): complete bucket-listing IAM condition handling (#12699)
* fix(s3api): complete bucket-listing IAM condition handling Populate s3:delimiter and s3:max-keys for bucket. Add regression tests for listing conditions and variants. Assisted-by: gpt-6.1-sol, deepseek-v4.1-flash * fix(s3api): ignore max-keys for multipart listing policies Assisted-by: GPT-6.1 Terra * s3api: drop max-keys from multipart-upload listing policy conditions Bucket policies and embedded IAM policies read condition values through ExtractConditionValuesFromRequest, which still exposed max-keys on ?uploads listings. Those page by max-uploads, so an unused max-keys parameter could satisfy an s3:max-keys condition. Match the advanced IAM path and leave the key absent there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
5681be241e |
topology: never report negative free volume slots (#12698)
* topology: never report negative free volume slots Free volume slots are computed as max + remote - volume - ecSlots. When phantom or over-registered volumes push the count past the configured max, the result is negative and rolls up into the cluster total, causing volume grow and assign to fail with a negative 'volumes left' count even while healthy servers have capacity. Clamp every free-volume count at zero so an over-committed server contributes no slots instead of subtracting from the cluster. * topology: drop the issue reference from the free-slot test comment Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * topology: test free slots next to an over-committed server A server with more volumes than slots must not hide the free slots of the other servers in its rack, data center or topology. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * topology: sum free slots per data node in racks, data centers and topology Clamping the rolled-up counters only clamped their sum, so one server with more volumes than slots still hid the free slots of every other server in its rack, data center and topology. Interior nodes now add up their children's free slots, so each data node counts as zero at worst. A data node in maintenance mode no longer adds slots to its parents either. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * topology: test that disk info keeps an over-committed deficit Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * topology: keep the signed free volume count in disk info EC encode retries and the balance planners add refunds and planned moves to the published FreeVolumeCount. Clamping it at zero dropped an over-committed disk's deficit, so a refund could make a full disk look free. Volume growth already reads the clamped AvailableSpaceFor. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * topology: test placement next to an over-committed server Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * topology: accept either data center order in the placement spread check Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
8ba33fae71 |
rust volume: bulk-load needle maps on redb 4.6 and compact the old floor (#12695)
* seaweed-volume: expect a fresh redb file to start at 72 KiB The assertion fails on redb 4.2.0, whose empty database is about 1 MiB. * seaweed-volume: bulk-load redb rebuilds with the stable cursor redb 4.6.0 has no experimental_cursor feature. Full rebuilds insert sorted needle ids through insert_before, and a new .rdb starts at 72 KiB. * seaweed-volume: test which redb files are still at the old floor The gate is missing, so the test does not compile yet. * seaweed-volume: decide when a reused redb is still at the old floor Compact only a file above 72 KiB and at most 1 MiB plus 8 KiB, and only when floor_compact has not been recorded. * seaweed-volume: test one-shot compact of an old-floor redb Reuse, shrink, and storage-error recovery are specified. The helpers they call are not implemented, so the tests do not compile yet. * seaweed-volume: compact a reused redb that is still at the old floor One compact runs when the file is above 72 KiB and at most 1 MiB plus 8 KiB. A storage error drops the handle and opens the same path again so the volume does not keep a PreviousIo latch. * seaweed-volume: publish a redb replay before compacting the old floor compact() is what would make a Durability::None tail durable, and a storage error then drops that tail on reopen. An in-band file is committed first. The floor_compact flag is read only inside the band, so a meta error cannot rebuild a larger index. * seaweed-volume: drop the redb experimental cursor CI step redb 4.6 has no redb-experimental-cursor feature. The needle-map tests that used it now run in the normal cargo test step. * seaweed-volume: drop the experimental-api-5 notes on redb reads redb's experimental-api-5 can no longer be enabled, so these reads always use the inherent ReadOnlyTable::get. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: drop the always-false flag from the old-floor band check Both callers pass false and read floor_compact on their own, so the band check only needs the file length. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d8a47042c5 |
master: unregister a volume server whose heartbeats stop (#12697)
* master: unregister a volume server whose heartbeats stop A volume server sends a heartbeat every 5 s. When its host hangs or its network drops packets, its SendHeartbeat stream stays open, and the master unregisters it only once gRPC keepalive closes the connection, 60 s plus 20 s after the last frame. Until then the master keeps assigning writes to volumes with a copy on it, and those writes hang. SendHeartbeat now ends the stream when no heartbeat arrives for three pulses (15 s), which unregisters the server through the existing deferred path and tells clients its locations are gone. The server reconnects and registers again once it can reach the master. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * master: test that clients learn a silent server's volumes are gone Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * master: fail the silent-stream receive test instead of hanging Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
|
dbb2fdb93b |
pb: recover when a local gRPC socket file is deleted (#12694)
* pb: fall back to TCP when the local gRPC socket cannot be dialed A temp cleaner can delete a local gRPC socket file while its server keeps running. Every later dial to that server then failed with "no such file or directory", so the master could no longer reach its own volume server and volume growth stopped. Dial the server's TCP address when the socket dial fails, as the S3 gateway already does for the filer's HTTP socket. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * pb: recreate a local gRPC socket whose file was removed Once its socket file was deleted, a server kept accepting on an unreachable listener until restart, so same-host RPCs stayed on the TCP fallback. Check the path every minute and, when the file is gone, close the old listener and bind the socket again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * pb: retry a failed local socket rebind on the next check A failed rebind ended the watcher, so the socket stayed missing after the cause cleared. Bind the replacement before closing the old listener: a failed bind keeps the old listener serving and is retried on the next check, and the old Serve still reports server shutdown. The old listener no longer unlinks the path on close, since the replacement now owns it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * pb: set the local socket unlink policy before serving the listener Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
b74971b6d4 |
server: add -grpc.localSocketDir for the local gRPC sockets (#12693)
* server: add -grpc.localSocketDir for the local gRPC sockets `weed server` and `weed mini` hard-coded each local gRPC Unix socket under /tmp. systemd-tmpfiles ages /tmp after 10 days and macOS after 3, so a long-running server loses its socket, and RegisterLocalGrpcSocket keeps routing local dials to the missing path: the master can no longer reach its volume server. Add -grpc.localSocketDir to both commands, defaulting to /tmp so an existing setup is unchanged, and build every socket path from it with filepath.Join. A folder that does not exist or is not a directory is refused at startup rather than at the first local RPC. A folder whose socket paths exceed the platform's sockaddr_un limit is refused too, because every bind then fails with EINVAL while the server keeps running and each local RPC dials a socket that does not exist. Refs #12692 * server: refuse a -grpc.localSocketDir the server cannot bind in An existing folder the server's user cannot write to passed the startup check, and then every socket bind failed: ListenOnLocalSocket only logs the error while GrpcDial keeps routing local dials to a socket that was never created. After the path-length check, create and remove a probe Unix socket in the folder, and refuse startup with the bind error if that fails. * server: give the -grpc.localSocketDir probe a random name A probe named only by the process id could be precreated by another user in a shared /tmp, or left by a crashed process whose pid was reused, and the server cannot remove such a file, so startup failed although the real socket paths were free. Name the probe with 8 random bytes, retry a name that is taken, and rely on the listener's close to unlink it. * server: keep the -grpc.localSocketDir probe name within the service socket lengths The probe was 5 bytes longer than the master and volume socket names, so a folder whose service sockets fit the sockaddr_un limit could still be refused. A 23-byte probe name is shorter than any service socket name, and a process with no local sockets skips the check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * server: trim the -grpc.localSocketDir comments and drop issue references Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Brent Pease <179639673+brentDSL@users.noreply.github.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
95f46c8fff |
admin: manage filer webhook targets from the UI with live reload (#12690)
* webhook: share the default target settings Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * webhook: managed webhook targets with a drainable queue Targets kept as JSON (name, endpoint, token, filters, retry and worker settings) build their own queue; Close waits for queued and in-flight deliveries before stopping it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: run webhook targets stored in /etc/seaweedfs when notification.webhook.managed is set The targets in /etc/seaweedfs/notification_webhooks.json start with the filer and are rebuilt from the store whenever the file changes on any filer, alongside notification.toml. The previous queue drains before it stops. Events for the file itself, which holds bearer tokens, are not sent to notification queues. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: manage filer webhook targets A Webhooks page under Management lists, adds, edits and deletes the targets kept in /etc/seaweedfs/notification_webhooks.json. Bearer tokens are write-only: the API reports whether one is set, and an empty token on edit keeps the stored one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: keep the webhook queue when the stored targets did not change Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: keep a replaced webhook queue open until its senders hand over Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: read chunk-backed webhook targets before editing them Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: keep the stored webhook targets out of fs.meta.notify Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * webhook: reject negative managed target settings Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * webhook: no signal handler on queues that reloads replace Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: swap the webhook queue without waiting behind blocked senders The read lock held across SendMessage made a reload wait for any sender blocked on a saturated target, and the waiting writer stalled every other filer write with it, which is when an operator disables that target. Close now refuses new sends and waits for the senders already inside SendMessage, and a send that hits the closing queue retries on its replacement. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * webhook: release blocked test handlers before the server shuts down Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: always run the webhook targets managed in the admin UI Drop the notification.webhook.managed switch: every filer loads and live reloads the stored targets. Without stored targets no queue is created. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * scaffold: place the admin-managed webhook note above enabled Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * scaffold: separate the admin-managed webhook note Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * scaffold: note that admin-managed webhooks follow the notification queue Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: retry loading webhook targets after a failed read Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: retire the admin-managed webhook queue on shutdown Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
386385101d | docs: regenerate star history chart | ||
|
|
6342876392 |
ci: run the Python 3.9 jobs on 3.10 (#12691)
* ci: run the Python 3.9 jobs on 3.10 ubuntu-latest now resolves to Ubuntu 26.04, which has no Python 3.9 build for actions/setup-python, so the Ceph S3 tests and the PyArrow Parquet 3.9 job fail during setup. 3.10 is the oldest version built for 26.04. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: list the Python versions the parquet CI runs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
40182a68bc |
admin: keep the dark theme from flashing light on reload (#12689)
* admin: apply the saved theme inline at the top of the page head The theme script was an external file, so a reload had to fetch or revalidate it before the root got its theme, and the page could paint with the light default in the meantime. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: set color-scheme on the root together with the theme Until the stylesheets load, the browser paints the page canvas from the root's color-scheme, which only the stylesheets set, so a dark theme showed a white canvas for the duration of every reload. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
eed00c3737 | docs: regenerate star history chart | ||
|
|
b449301a4e |
ec: mount shards through the encode's .vif when a source volume's .vif shares the disk (#12676)
* ec: mount shards against the index .vif when the local one has no EC config A worker encode can send shards to a disk that still holds the source volume's .vif while the encode's .vif lands with the index on a sibling disk. The shard disk mounted through the source's config-free .vif, so it reported encode time 0 and fell back to the sidecar layout, and reads carrying the encode time were rejected as a different encode run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rs): mount shards against the index .vif when the local one has no EC config Mirrors the Go mount: a data-dir .vif that records no EC config yields to the idx dir's when that one records it, and load_vif_info_across_dirs reports the directory the .vif actually came from. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * storage: do not mirror EC sidecars onto a disk whose .vif has no EC config Startup mirroring copied .ecx/.ecj next to a source volume's leftover config-free .vif and kept that .vif, so after a restart the shard disk mounted self-contained with encode time 0. Leave such a disk on the cross-disk mount, which reads the owner's .vif. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rs): do not mirror EC sidecars onto a disk whose .vif has no EC config Mirrors the Go startup mirror: a shard disk holding a source volume's config-free .vif stays on the cross-disk mount, which reads the owner's .vif, instead of mounting self-contained with encode time 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec: take the encode .vif from a sibling data dir when the index dir is shared With a separate -dir.idx the .ecx is shared by every disk, but the encode's .vif lands in the data dir of the disk that received it. A disk still holding a source volume's config-free .vif mounted through it at generation 0. NewEcVolume now accepts the server's data dirs and takes a sibling's EC .vif when the index is not local and every sibling EC .vif names the same run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * storage: give EC mounts the data dir of every disk on the server NewStore records every data dir on each DiskLocation before loading, so the startup scan, cross-disk reconcile and MountEcShards all pass them to NewEcVolume. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rs): take the encode .vif from a sibling data dir when the index dir is shared Mirrors the Go mount: EcVolume::new_with_vif_dirs takes a sibling's EC .vif when this disk's own records no EC config, the index is not local, and every sibling EC .vif names the same run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rs): give EC mounts the data dir of every disk on the server main sets Store.store_dirs before the first add_location, so a disk loaded before its siblings still sees them; each DiskLocation passes them to EcVolume::new_with_vif_dirs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * storage: size startup EC validation by the sibling encode .vif With a shared -dir.idx, a disk that still holds the source .dat and its config-free .vif read the default data-shard count, judged custom-ratio shards an interrupted encode, and removed them along with the shared .ecx before the mount could take the sibling's .vif. The startup lookup now falls back to the same sibling .vif the mount uses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rs): size startup EC validation by the sibling encode .vif Mirrors the Go change: validate_ec_volume falls back to the sibling data dir's encode .vif before judging shards beside a source .dat. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * storage: keep EC shards when sibling .vif files disagree on the encode SiblingEcConfigVif reports when a sibling .vif cannot be read or the siblings name different encode runs. The startup data-shard lookup then returns 0 instead of the default ratio, and validateEcVolume keeps the files rather than sizing custom-ratio shards by a ratio they do not use. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rs): keep EC shards when sibling .vif files disagree on the encode Mirrors the Go change: sibling_ec_config_vif returns Err for an unreadable sibling .vif or disagreeing runs, ec_data_shards_from_vif returns 0 then, and validate_ec_volume keeps the files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * storage: release the opened EC shard when its volume cannot be built Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
9e5357782e |
volume: keep reads over the download limit local when readMode is local (#12679)
* volume: keep reads over the download limit local when readMode is local With -readMode=local the server never sends a read for a volume it does not hold elsewhere, but a read over -concurrentDownloadLimitMB on a replicated volume was still redirected to another replica. Skip the replica fallback in local mode, so the read waits for a download slot and answers 429 when the wait times out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rs): keep reads over the download limit local when readMode is local Mirror the Go volume server: with readMode local, a read over the download limit waits for a slot instead of being redirected to a replica. proxy_or_redirect_to_target no longer needs the local-mode redirect switch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: check download capacity under the wait lock waitForDownloadSlot read the in-flight size outside the cond lock and the read handler broadcast without it, so a download finishing between the check and the wait was missed and the read waited out the timeout for a 429. Check capacity with util.WaitUntil under the lock and broadcast under it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume(rs): register for the download notification before checking capacity The wait loop created its Notified future only after the capacity check, so a body dropped in between had its notify_waiters() lost and the read waited out the deadline for a 429. Create the future first, then check, as Go now checks under the cond lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
f4e638fa1d |
Fix/admin cluster submenu (#12677)
* admin: fix cluster submenu toggle * admin: update embedded JavaScript * admin: drop the object store submenu handlers The sidebar no longer renders an object store submenu, so its expand, link and toggle handlers match nothing; the toggle handler repeated the manual collapse that fought Bootstrap on the cluster submenu. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: ivan <i.stepanov@navigine.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
1e31c6dbf5 |
test(pjdfstest): check the sudo'd mount with ps, not kill -0 (#12673)
sudo switches its real uid to root while it starts, so kill -0 from the invoking user fails with EPERM even though the mount process is alive, and the wait loop reported "weed mount exited early" a few milliseconds after starting it. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0f0f3811bf |
filer: serve caught-up aggregated subscribers from memory, not a held disk pass (#12674)
* filer: serve caught-up aggregated subscribers from memory, not a held disk pass An aggregated SubscribeMetadata stream that does not take metadata chunks (filer.sync, filer.backup, filer.meta.backup, filer.meta.tail, the filer.remote.* syncs, the S3 gateway's subscriptions, and mounts that predate chunk support) re-ran the persisted-log pass whenever its cursor moved: after every memory hold at the delivery watermark and on every 2 s re-probe. The pass holds at the peers' flush low-watermark. Peers flush on their own one-minute schedules, so on a cluster that keeps writing some peer has always flushed past a caught-up cursor while the slowest has not: the pass held at its first entry, the loop came back to it, and memory was never read. The stream got each change only once the slowest peer's flush covered it, in bursts up to a flush interval late, and every pass listed and re-read the newest log files. When the aggregated ring was already filling at the cursor and evicted nothing after it, memory holds everything the persisted logs could add, since every peer's log reaches the ring through the aggregator. Skip the pass and read memory, held at the delivery watermark as before. The skipped pass proves nothing past the cursor, and the next-day scan for an empty pass does not apply. An empty ring, a ring that starts past the cursor, and a cursor memory just refused still take the pass. Chunk subscribers already skip it for a caught-up cursor: their ref listing stops below the flush hold. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: keep a new peer's pre-start history on the disk pass The ring only receives a peer's events from the start of its subscription. A peer followed for the first time after this filer started (a filer added to the cluster, a restarted pod with a new address, or the late-arriving peers bootstrap) starts at its notice time, so what it logged before then is only in the persisted logs. A cursor below that start passed the ring check and skipped the disk pass, losing those events for subscribers that do not take metadata chunks. Record the newest start of a peer subscription that had no resume point and send cursors below it through the disk pass as before. Peers that rejoin with a resume point (a restart under the same address) do not move it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
|
53b234558e |
build(deps): bump actions/setup-go from 6 to 7 (#12685)
Bumps [actions/setup-go](https://github.com/actions/setup-go) from 6 to 7. - [Release notes](https://github.com/actions/setup-go/releases) - [Commits](https://github.com/actions/setup-go/compare/v6...v7) --- updated-dependencies: - dependency-name: actions/setup-go dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
|
d9c2d99f8f |
build(deps): bump gocloud.dev/pubsub/natspubsub from 0.46.0 to 0.49.0 (#12680)
Bumps gocloud.dev/pubsub/natspubsub from 0.46.0 to 0.49.0. --- updated-dependencies: - dependency-name: gocloud.dev/pubsub/natspubsub dependency-version: 0.49.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
|
e3769592a5 |
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob from 1.8.2 to 1.8.3 (#12684)
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/storage/azblob Bumps [github.com/Azure/azure-sdk-for-go/sdk/storage/azblob](https://github.com/Azure/azure-sdk-for-go) from 1.8.2 to 1.8.3. - [Release notes](https://github.com/Azure/azure-sdk-for-go/releases) - [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azidentity/v1.8.2...sdk/storage/azblob/v1.8.3) --- updated-dependencies: - dependency-name: github.com/Azure/azure-sdk-for-go/sdk/storage/azblob dependency-version: 1.8.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
|
c3f4b3e0eb |
build(deps): bump go.etcd.io/etcd/client/v3 from 3.7.1 to 3.7.3 (#12682)
Bumps [go.etcd.io/etcd/client/v3](https://github.com/etcd-io/etcd) from 3.7.1 to 3.7.3. - [Release notes](https://github.com/etcd-io/etcd/releases) - [Commits](https://github.com/etcd-io/etcd/compare/v3.7.1...v3.7.3) --- updated-dependencies: - dependency-name: go.etcd.io/etcd/client/v3 dependency-version: 3.7.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
|
438cc8562f |
build(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.114.1 to 1.114.10 (#12681)
build(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 Bumps [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) from 1.114.1 to 1.114.10. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.114.1...service/s3/v1.114.10) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/service/s3 dependency-version: 1.114.10 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
|
51ab451f37 |
build(deps): bump github.com/rabbitmq/amqp091-go from 1.15.0 to 1.16.0 (#12683)
Bumps [github.com/rabbitmq/amqp091-go](https://github.com/rabbitmq/amqp091-go) from 1.15.0 to 1.16.0. - [Release notes](https://github.com/rabbitmq/amqp091-go/releases) - [Changelog](https://github.com/rabbitmq/amqp091-go/blob/main/CHANGELOG.md) - [Commits](https://github.com/rabbitmq/amqp091-go/compare/v1.15.0...v1.16.0) --- updated-dependencies: - dependency-name: github.com/rabbitmq/amqp091-go dependency-version: 1.16.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
|
34927705fa |
build(deps): bump actions/setup-python from 6 to 7 (#12686)
Bumps [actions/setup-python](https://github.com/actions/setup-python) from 6 to 7. - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](https://github.com/actions/setup-python/compare/v6...v7) --- updated-dependencies: - dependency-name: actions/setup-python dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
|
d5b4e1352e |
helm: drop the 4.53 vector metrics render guard (#12672)
* helm: pin the 4.53 tag in the vector metrics rejection test The 4.54 release made 4.54 the chart's default image, and that image serves weed-vector metrics, so rendering vector.metricsPort with the default tag no longer fails. Pin image.tag=4.53 for the rejection case and check that the default image renders the metrics port. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * helm: describe vector metrics support from 4.54 on in the chart README Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * helm: say the vector metrics render guard skips image overrides Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * helm: drop the 4.53 vector metrics render guard The chart now defaults to 4.54, whose weed-vector serves --metrics-port, so the check keyed on one old tag only blocked valid installs. The test asserts the default image renders the metrics flag and HTTP probes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * helm: describe vector metrics without the render guard in the chart README Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
5ef99c1d8d | 4.54 4.54 | ||
|
|
8e15110b4f |
filer: wake metadata subscribers only for directories they can match (#12669)
* log_buffer: let a subscriber skip wake-ups for unwanted partition keys Every append woke every registered subscriber. SetSubscriberWakeFilter lets a reader sleep through appends whose partition key it would filter out; it still reads them on its next poll. Flushes and keyless appends wake everyone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer_pb: decide subscription matches from an event's directories MetadataDirectoryMayMatchSubscription rules out events whose directory no child path of can match the subscription, and ScanMetadataEventDirectories reads an event's directory and new parent path without allocating. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: wake subscribers of a move's destination directory Appends are keyed by the source directory, so a filtered subscriber of the destination would otherwise only see the move on its next poll. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: wake metadata subscribers only for directories they can match Every metadata event woke and was scanned by every subscriber, so filer CPU grew with the number of subscribers watching other paths. Readers now register a directory wake filter, and the per-entry prefilter rules entries out from their directories before the skeleton scan. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: test that moves into a subscribed prefix pass the prefilter Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
f13b763552 |
docs: explain the 3-filer recommendation in the production setup notes (#12671)
* docs: explain the 3-filer recommendation in the production setup notes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: note that a filer load balancer must forward the gRPC port Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
df30c883fe |
s3api: keep a multipart upload's state on the object's owner filer (#12668)
* s3api: create a multipart upload's directory on the object's owner filer CreateMultipartUpload wrote .uploads/<id> to the gateway's own filer while the upload's completion commits on the object's write owner. With independent filer stores, a part or completion reaching another gateway before peer metadata replication caught up could not see the upload. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: look up a multipart upload on the object's owner filer UploadPart, UploadPartCopy, ListParts, CompleteMultipartUpload, AbortMultipartUpload and the bucket policy SSE lookup read .uploads/<id> from the gateway's own filer, so a request on another gateway shortly after CreateMultipartUpload answered NoSuchUpload until replication caught up. Read it preferring the object's write owner, like object reads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: write multipart parts to the upload's owner filer UploadPart and UploadPartCopy stored part entries on the gateway's own filer, so ListParts and CompleteMultipartUpload on another gateway could miss a part that replication had not delivered yet. Store them next to the upload directory on the object's write owner. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: list and clean up a multipart upload on its owner filer ListParts, CompleteMultipartUpload and AbortMultipartUpload listed and removed part entries on the gateway's own filer. Do it where the upload and its parts now live, so completion sees every part and cleanup does not miss entries replication has not delivered yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: test a multipart upload against a lagging gateway filer Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: check the upload's owner before freeing a failed part's chunks A part write that the owner stored but answered with an error went to recovery with no object, so the owner was never among the filers checked. When the owner is outside the gateway's filer list, the others' not-found answers freed chunks the stored part still references. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: list an upload's parts on the prior owner after a ring change The upload record lookup already falls back to the key's prior owner while a ring change is in its cooling-off window, but the part listing read only the new owner. ListParts then reported no parts for an open upload. Fall back the same way when the new owner has nothing yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: find the upload record on the prior owner past a bare directory A part routed to a new owner that has not received the upload yet makes that filer create .uploads/<id> without the upload's key. The post-write check then read it as an aborted upload and removed the part. While the ring change is in its cooling-off window, a record on the prior owner now wins over such a directory. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: merge the prior owner's parts into an upload listing During a ring change's cooling-off window the new owner may have received only some of an upload's parts. Listing returned the new owner's parts alone once it had any, so ListParts dropped the rest and a completion naming them failed with InvalidPart. Both owners' pages are now merged, the owner's entry winning a shared name, and cut to the page limit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: fail an upload listing the prior owner could not answer When the new owner has none of an upload's parts yet and the prior owner's listing fails, the parts may all be on the prior owner. That listing read as empty, so CompleteMultipartUpload answered NoSuchUpload for an upload that still exists; it now returns the error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: fail an upload lookup the prior owner could not answer During a ring change's cooling-off window an upload the new owner has not received yet, or holds only as a bare directory, read as NoSuchUpload when the prior owner's lookup failed. That failure is now returned, so the client sees a retryable error instead of an upload that is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
5a1b0abc20 | docs: regenerate star history chart | ||
|
|
b2dd93eac8 |
admin: support [maintenance.s3_lifecycle] in admin.toml (#12661)
* admin: support [maintenance.s3_lifecycle] in admin.toml
The admin.toml overlay covers vacuum, volume_balance and erasure_coding but
not s3_lifecycle, so the bucket lifecycle job's settings can only be changed
from the web UI and are lost whenever the admin data directory does not
persist. That is the worst combination for this job type: its
meta_log_retention_days knob decides whether bucket rules ever apply to
objects written before the job was enabled, and silently reverting it to the
default makes the rules no-op without any error.
Wire the four admin-side knobs into the existing overlay:
meta_log_retention_days retention window the meta-log replay can reach
walker_interval_minutes minimum gap between full bucket walks
cluster_deletes_per_second cluster-wide delete budget
cluster_deletes_burst token-bucket burst for that budget
All four are admin config values, not worker values: the engine partition is
computed on the admin side and the delete budget is divided across
execute-capable workers at dispatch time. enabled/retry_limit/
retry_backoff_seconds come for free from the shared section handling.
Tested with TestApplyPluginConfigFromTomlS3Lifecycle, which also pins the
retention=0 case: 0 means unbounded and must survive as an explicit 0 rather
than being treated as unset.
* admin: fix meta_log_retention_days guidance in admin.toml scaffold
Both reviewers caught a real error. `0` is not "unbounded": runShard does
retentionWindow := cfg.RetentionWindow
if retentionWindow <= 0 {
retentionWindow = maxTTL
}
so zero falls back to the longest rule TTL, which keeps every replay rule in
replay and promotes nothing to the walk set. Recommending 0 to force full
scans was wrong, and so was the claim that pre-existing objects are only
reachable through the walk set -- the cold-start recovery walk covers them
regardless of the retention value.
Describe zero as "no retention info supplied" and tell operators to set the
window the filer meta-log actually retains. The code change is unaffected:
it only wires the four keys into the existing overlay.
* admin: tie s3_lifecycle walk-set scans to walker_interval_minutes in admin.toml
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: show the s3_lifecycle retry defaults in admin.toml
The s3_lifecycle descriptor sets no retry limit or backoff, so a failed
run is not retried and the scheduler's 5 s backoff applies; the scaffold
listed the vacuum values instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: shorten the s3_lifecycle toml section comment
The retention and walker knobs are read by the worker from the admin
values, not computed on the admin side.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: cover cluster_deletes_burst in the s3_lifecycle toml test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: check the s3_lifecycle retention zero is a stored int64
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
5f7fefadae |
s3 lifecycle: write the suspended expiration marker before deleting the null version (#12664)
* s3 test: evaluate mutation conditions and serve DeleteEntry in the fake filer The fake ObjectTransaction now checks each mutation's own condition before applying it, as the filer does, and reports the failed mutation. ETag clauses read the stored ETag attribute first, like the filer. DeleteEntry removes the entry so unrouted delete paths can run against the fake. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: write the suspended expiration marker before deleting the null version A suspended-versioning expiration deleted the null version first and then wrote the delete marker. When the pointer read or the marker write failed in between, the retry found no null version, resolved as NOT_FOUND or VERSION_NOT_LATEST, and never wrote the marker, so an older version became the latest. The marker now goes first. The latest pointer outranks the null object, so the key reads as deleted once the marker lands, and a null delete that fails after it leaves a noncurrent null version rather than exposing an older one. The null delete is pinned to the entry read before the marker: on the owner it carries IF_ENTRY_EQUAL on that read, and off the ring the entry is re-read and compared by identity first. A null object written after the read is a newer write and is no longer deleted. When the null version is the one that passed the expiration checks, a missing or changed entry at that read resolves the request as STALE_IDENTITY before anything is written. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: retry a failed routed null delete instead of deleting on the default filer A routed null delete that failed fell through to the unrouted path, which reads and deletes on the default filer. That filer may not have the owner's newer null object yet, so the fallback could delete it without the owner-side IF_ENTRY_EQUAL guard. The failure now returns RETRY_LATER; the marker is already written, so the key keeps reading as deleted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: recheck and delete an unrouted null version under the object write lock Without a ring owner the identity recheck and the null delete were separate calls, so a suspended PUT landing between them was deleted. Off the ring writers take the object write lock, so the recheck and the delete now run under it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 test: strip quotes from the stored ETag in the fake filer The filer normalizes both sides of an ETag comparison; the fake now strips quotes from the stored ETag attribute as well as from the candidate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: retry the null delete when the object lock lookup fails on a routed key Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: check the suspended marker against the validated version A suspended expiration that validated a named version wrote its marker without comparing the pointer, so a null PUT landing after the checks was hidden, and deleted too when it landed before the null read. The marker now compares the latest the way the enabled path does and resolves the request as VERSION_NOT_LATEST. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 test: count the marker's object write lock in the unrouted null delete Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: recheck the attempt deadline after reading the null version Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
c09009a74f |
s3: accept the governance bypass header case-insensitively (#12667)
* s3: accept the governance bypass header case-insensitively botocore and the AWS CLI send x-amz-bypass-governance-retention: True, so the exact "true" match ignored the bypass and governance-locked deletes and retention changes failed with AccessDenied. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: test governance bypass with SDK-style header values Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
c0509cf499 |
s3api: evaluate each policy condition against its own values (#12665)
* s3api: normalize a nil policy value to no values Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: evaluate each policy condition against its own values The process-wide normalized value cache keyed entries by fmt.Sprint of the value, so ["a", "b"] and ["a b"] (or [] and [""]) shared one entry and a condition in one policy could be compared against another policy's values. Normalizing is cheaper than building that key, so drop the cache. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: test that look-alike condition values stay distinct Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
63806b7073 |
s3 lifecycle: bound each LifecycleDelete attempt with its own deadline (#12646)
* s3 lifecycle: bound each LifecycleDelete attempt with its own deadline Replay and walker deletes used the job context with no request deadline, so one stalled unary RPC could hold a shard for the whole job. Each attempt now runs under DeleteTimeout (default 1m), set after the limiter wait. A timeout while the job is still live is retried like a transport error and then halts the shard or walk without advancing; parent cancellation still stops both the attempt and the backoff. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: admin setting for the delete RPC timeout Adds delete_timeout_seconds to the S3 Lifecycle admin form and passes it to both the replay drain and the walker. 0 keeps the 60s default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: describe replay and walker timeouts separately in the help text Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: fail the pass when a replay delete exhausts its attempts A transport error or attempt timeout that outlived the retries halted the shard with a nil error, so the job reported success. processMatches now returns the error (the cursor still holds at the event), and runShard treats a drain error as a clean stop only when its own context ended, since an attempt timeout also wraps DeadlineExceeded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: test that a replayed LifecycleDelete resolves after a lost reply Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: skip the expiration write once the attempt deadline passed A worker retries a LifecycleDelete whose attempt timed out, while the first attempt can still be running on the S3 server. An attempt that stalled before its marker write would then stack a second delete marker after the retry wrote one. The gRPC deadline reaches the handler context, so check it right before the expiration writes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: recheck the attempt deadline after the suspended null delete The null-version delete can block on the owner filer past the attempt deadline; the worker may have retried and written a marker by then. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: finish the suspended marker once the null version is deleted Skipping the marker after a successful null delete left an older version as the latest, and the retry then resolved as a no-op without hiding it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: let a routed delete marker commit only while its latest guard holds createDeleteMarkerIf passes an optional condition, evaluated on the .versions directory entry under the object's lock on the owner filer, and reports a refused commit as errLatestVersionChanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: write the expiration marker as a compare-and-set on the latest pointer A timed-out attempt can still be running on the S3 server while the worker's retry writes the marker. The marker commit is now conditioned on the .versions pointer as read just before it and skipped when the latest is already a delete marker, so whichever attempt commits second writes nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: resolve the marker as stale when another version became latest The marker's compare-and-set only covered changes after its own pointer read, so a version promoted after the expiration checks passed could be hidden by the marker. The enabled path now passes the validated version and a conflict retry that finds a different named latest resolves the request as VERSION_NOT_LATEST. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: resolve a null expiration as stale once a version takes the pointer Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: read the marker's latest version the way getLatestObjectVersion does A pointer names a version only with its file name; an id alone falls back to the null object, and the null-is-latest signal makes the null version current. An expiration that validated a named version no longer hides a null written by a suspended PUT, and a current null under an id-only pointer now expires instead of resolving as stale. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: hold the object write lock for an unrouted expiration marker Off the ring createDeleteMarkerIf cannot carry the pointer guard, so a PUT committing between the pointer read and the marker write was hidden. The read and the write now run under the object write lock that unrouted writers take. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: share transient-error retries between walker and replay deletes (#12649) * s3 lifecycle: retry only transient LifecycleDelete failures dispatchWithRetry retried every RPC error except parent cancellation, so an authentication, permission or argument error spent the whole retry budget before surfacing. Retry only attempt timeouts and the failures util.IsTransientError recognizes (Unavailable, ResourceExhausted, network resets); everything else returns on the first attempt. A nil response is now an error instead of a nil dereference. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: take a delete-rate token for every LifecycleDelete attempt processMatches waited on the shared limiter once per match, so the retries behind it ran outside the cluster-wide deletes/sec budget. The retry loop now waits on the limiter before each attempt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: give walker deletes the replay retry policy WalkerDispatcher sent each LifecycleDelete once, so one dropped connection aborted the bucket's daily walk while the same failure on the replay path was retried. The walker now goes through deleteWithRetry and shares its transient-error retries, per-attempt limiter wait and nil response handling. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: retry a DeadlineExceeded the server sends back A server or proxy can end the RPC with DeadlineExceeded while the attempt's own deadline is still running. util.IsTransientError rejects that status, so the delete halted with retries left. It is a timed-out attempt like the client-side one, so retry it the same way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: say walker deletes get 3 attempts in the timeout help Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
504de9b562 |
security: refuse to start when per-target gRPC server allowlists are not applied (#12662)
* security: refuse to start when per-target server allowlists are not applied A component that dials every server with one option ignored <target>_allowed_server_* keys, so a config that only set those accepted any CA-signed server. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * scaffold: say which components scope server allowlists per target Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * security: also catch per-target server allowlists set through WEED_ env vars Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d80e905fc7 |
security: optional dialed-host check for gRPC server certificates (#12663)
* security: optionally check the dialed host against gRPC server certificates verify_server_host under [grpc] or a [grpc.<component>] section switches the client to certificate-and-host verification. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * scaffold: document verify_server_host Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
799fd10342 |
s3 lifecycle: fail over between S3 fronts during a pass (#12647)
* s3 lifecycle: add a LifecycleDelete client that fails over between S3 fronts When the current front returns Unavailable, the client closes it, marks it failed for the rest of the pass, picks the next front that passes checkLifecycleReady and resends the same request there. Callers cut off by that switch closing their connection resend on the new front. When no front is left, every later call returns one error naming the fronts tried. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: serve the pass through the failover client Execute hands the replay drain and the walker one failover client built on the selected front, reports each switch as job progress, and closes the current connection when the pass ends. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: keep failing over until a front serves the resent delete The resend after a switch went straight to the new front, so a second lost front ended the call even with a healthy third one, and the walker has no retry of its own. Resend through the failover loop instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: do not recheck fronts skipped when the pass started The failover client started with no failed fronts, so losing the selected front checked the fronts selectS3Front had already skipped again, holding every delete for another readiness timeout per front. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a91b333658 |
s3 lifecycle: persist daily walk checkpoints and resume them safely (#12650)
* s3 lifecycle: walk checkpoints take each entry's resume position Versions of one object share a Path, so a checkpoint set from Path after the first version made a resume skip the rest. The ListFunc now gives each entry the start value that continues after it, and leaves it empty on all but an object's last version. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: resume a bucket listing where the walk stopped The filer listing skipped entries with key <= start, but its order is not key order: each directory emits its .versions groups before its files, and a directory's children come before a sibling like dir.txt. Positions are now one (pass, name) frame per directory level, so a resume seeds each listing and skips exactly what was finished. A version group is one unit, and a skipped group still keeps its bare null version out of the second pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: content hash of a walk view Hashes every non-disabled action of a view and the versioning of their buckets, so a saved walk position can be tied to the walk it came from: another rule set, another walk mode, or a bucket that turned versioned gives another hash. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: persist daily walk progress and resume it A walk that failed or was cut short started over on the next pass, listing again every bucket it had already finished. WalkBuckets now keeps the finished buckets and each unfinished bucket's position in a checkpoint file next to the shard cursors, one per shard set, and the next walk of the same view and shards resumes from it. The record is saved every 30s and when the walk stops, and cleared when a walk completes, so the next walk is full again and sees keys added behind the position; a clear that fails fails the walk rather than leave a record that would skip buckets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: a stuck resumed walk starts over A bucket that fails at the same place on every pass kept the walk from completing, so the buckets it had finished were never walked again and their due objects waited forever. A resumed walk that fails again without finishing anything clears its checkpoint, and the next walk is full. A canceled walk keeps it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: do not resume a walk checkpoint left for a week A checkpoint is cleared only by a later walk of the same shard set, so a set that stopped walking, such as a one-off per-shard walk, left its record behind, and a walk of that set much later would skip buckets finished long before. Records now carry their save time and are not resumed after seven days; a walk that keeps retrying saves on every attempt and stays fresh. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: a failed walk without progress leaves its checkpoint alone A pass canceled while it read the checkpoint failed the read, then saved its own empty progress over the record it never loaded, so the next pass started every bucket again. A failed walk that finished nothing has nothing new to record and no longer writes; the stuck resume reset still clears. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: space out walk checkpoint saves after a failed one The save time advanced only on success, so once the filer refused a periodic save every following entry tried again and logged another warning. The interval now counts from the last attempt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: a walk checkpoint save that errors may still be on disk A save whose reply was lost left onDisk false, so a walk that then completed skipped the clear and the next walk resumed from the stale record instead of walking everything. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: keep a stalled walk checkpoint for a day A resumed walk that failed again without progress dropped its record right away, so an outage spanning two attempts restarted the whole walk. Treat the walk as stuck only once its record has not advanced for a day. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: a resumed walk counts its earlier buckets as walked Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: replay a resumed recovery walk from the walk that began it A recovery or cold-start pass rewinds replay to runNow - maxTTL. When the walk resumed a checkpoint, its finished buckets were listed in an earlier attempt, so an object there that was not yet due then and whose write is now older than maxTTL was neither expired by the walk nor replayed. The checkpoint now keeps the start of the walk that began it, WalkBuckets returns that time (its own start for a fresh walk), and runShard floors the recovery rewind and the cold-start replay at it. A record without a start is not resumed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: a cold start that resumed a walk replays on the next pass The pass's shared subscription starts at runNow - maxTTL for a shard without a cursor, before its walk reports an earlier start, so flooring the cold-start drain at the walk start could not deliver the older events. Such a shard now saves its cursor at the walk start minus maxTTL and returns, like the recovery branch, and the next pass replays the window from that cursor. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: test that a cursor below the floor opens the next subscription Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3 lifecycle: test the cold-start replay across two Run passes A cold start whose walk resumed an earlier attempt sees an older event only on the next pass. The test runs Run twice against a filer that serves events past the requested floor, with a later in-window event that would otherwise move the cursor past the older one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
4f70ad015e |
mount: fail the write when the swap file chunk cannot be created (#12655)
* mount: fail the write when the swap file chunk cannot be created #7957 added a nil check after NewSwapFileChunk so a failed swap file creation returns an error instead of panicking. The result was assigned to pageChunk, a PageChunk interface, before the check, and an interface holding a nil *SwapFileChunk is not nil, so the check never fired: SaveDataAt stored the nil chunk and panicked in WriteDataAt. Check the *SwapFileChunk before converting it. The new test points the swap directory below a regular file; on master it panics in SwapFileChunk.WriteDataAt. Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com> * mount: grow the file only after its write is buffered Write set the new size, times, pending append offset and uncommitted bytes before AddPage. When AddPage failed (now that a swap file chunk that cannot be created returns an error), the write was reported as failed but the larger size stayed on the handle's entry, so the next flush published it with a hole where the data should be. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: report a partly buffered write as a short write PageWriter.AddPage fills one chunk at a time, so when a later chunk cannot be created the earlier chunks already hold their bytes. Return how much was buffered, and let Write grow the size and times over that prefix and report it, instead of failing the whole write while the prefix still reaches the next flush. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: report the bytes a failed copy_file_range already buffered A streaming copy whose page fails partway returned an error even though earlier pages, or the start of the failing one, were already buffered and would reach the next flush with the destination size left short. Count the buffered bytes, grow the destination over them, and return them as a short copy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: free the copy test's destination pages The buffered memory chunk counts toward the process-wide memory chunk ceiling, so repeated runs of the test pushed later ones onto the unusable swap directory. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: count only the bytes a swap chunk stored A swap chunk whose file write fails stores a prefix or nothing, but SaveDataAt returned no error and AddPage counted the whole span as buffered. SaveDataAt now reports the short write, and AddPage passes the stored count up so a write is never acknowledged past it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: fail a partly buffered writeback write Kernel writeback ignores the count in a WRITE reply, so a short write there drops the unbuffered rest without an error. A write flagged as page-cache writeback now keeps its buffered prefix but returns the error, which the kernel reports on fsync. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Signed-off-by: anant <170962632+antcybersec@users.noreply.github.com> Co-authored-by: anant <170962632+antcybersec@users.noreply.github.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
ddd3b3666f | docs: regenerate star history chart | ||
|
|
cdb01f8150 |
install.sh: add --enterprise (#12660)
* install.sh: offer the vector engine on linux only Releases build weed-vector for linux amd64/arm64 only, so --component vector on darwin or windows downloaded a missing asset. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * install.sh: add --enterprise Installs from seaweedfs/artifactory: weed-enterprise-<os>_<arch>[_large_disk], weed-volume-enterprise[_large_disk], weed-worker-enterprise and weed-vector-enterprise. The latest tag skips vfs-* releases, and the Rust volume server is linux-only there, so --component all skips it elsewhere. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * Revert "install.sh: offer the vector engine on linux only" Releases after 4.53 publish weed-vector for darwin and windows too (rust-vector-platforms.yml). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * install.sh: Enterprise vector engine is linux-only Enterprise releases publish weed-vector-enterprise for linux amd64/arm64; open-source releases keep their darwin and windows engines. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * test: installer --enterprise asset names, platforms and latest tag Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * install.sh: Enterprise latest is the newest plain version tag artifactory also releases previews and a dev build, and a prerelease such as kvcache-filer-native-v0.1.0-preview.1 sits in the release list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * README: show passing --enterprise through bash -s -- Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * install.sh: the open-source weed for windows is a .zip too Releases publish windows_amd64[_large_disk].zip, so asking for a .tar.gz 404s on every Windows host. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * install.sh: install the windows weed as weed.exe weed-volume and weed-vector keep their .exe on Windows; weed lost it, so cmd and PowerShell could not run the installed binary. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * install.sh: suggest weed-volume only when it was installed Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |