mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-11-12 06:48:45 +01:00
eb159f31816dead8fa8a34373f3f38624dffe885
10563
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris LuandClaude Opus 5.5
|
eb159f3181 |
lance: deregister and re-register need UpdateTable (#12552)
* lance: require UpdateTable to deregister a table Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: require UpdateTable to register over an existing table Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: test that GetTable alone cannot deregister or re-register a table Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: a register that finds a table the caller cannot read needs UpdateTable to bring it back Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: decide declare write access by whether the table existed, not by what the caller can read Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: test register and declare by a caller that cannot read the table Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: register clears the marker on the create branch only after checking it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance: keep a register whose table was created when the marker check fails Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
4cb52d7705 |
filer: reopen a leveldb store wedged by a transient write error (#12565)
* filer/leveldb_handle: reopen a database whose journal latched a write error goleveldb keeps the first journal write error and returns it for every later write until the database is reopened, so one transient ENOSPC or EIO fails all writes until a restart. The handle offers the leveldb.DB methods the stores use; a write that fails on an OS error probes the storage and, once it accepts writes and no call holds the database, closes and reopens it and retries the write once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: open the leveldb stores through leveldb_handle leveldb, leveldb2 and leveldb3 now recover from a transient storage write error without a restart. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: leveldb stores return iterator errors from listings and deletes A listing on a database that cannot be read reported an empty directory, and DeleteFolderChildren wrote an empty batch, because the iterator error was never checked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer/leveldb_handle: test a reopen that fails after the probe passes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer/leveldb_handle: probe storage before draining readers A probe that stalls in Create, Write or Sync held mu exclusively and blocked every read on the handle. Reopen attempts are now serialized by their own mutex, and the probe runs before the drain. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer/leveldb_handle: hold new calls back while a reopen drains TryLock succeeds only at an instant with no reader, so overlapping listings could keep a reopen from ever draining while each write spun for the full drain timeout. The drain now holds new calls back for at most drainTimeout; a call nested inside a running one waits that out instead of deadlocking. A drain that times out holds off the next attempt for busyReopenInterval, so new calls are not held back over and over. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer/leveldb_handle: bound the wait for a running reopen A write that hit the latched error waited on the reopen mutex for as long as another caller's storage probe ran, so a stalled probe queued every writer. Callers now wait at most reopenWait and return their error, and still retry once a concurrent reopen succeeds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer/leveldb_handle: tests wait for the drain and readers explicitly The nested-read test polled for a drain that could start and end between polls and then spun forever; it now syncs on the probe, widens the drain and fails if the put returns first. The overlapping-readers test waits until every reader holds an iterator before it writes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
5150168368 |
s3: authorize an empty versionId as the current-object action (#12566)
* s3: authorize an empty versionId as the current-object action GET, HEAD and DELETE treat an empty versionId as the current object, but the action resolver keyed on the parameter's presence and authorized it as GetObjectVersion/DeleteObjectVersion, so a Deny on GetObject or DeleteObject did not apply. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: test an empty versionId against a version-only policy Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: cover a bare versionId on HEAD Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
0c6c6868de |
volume: refuse a tier upload whose .dat was written during the copy (#12562)
* volume: refuse a tier upload whose .dat grew during the copy AttachRemoteDatFile only checked that the data backend was still the one copied, so a needle write acknowledged after the copy read the .dat was missing from the remote object, and the upload then switched reads to it and removed the local .dat. Compare the .dat size with the copy under the same data-file locks; on a mismatch the handler deletes the copy and keeps the local file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * seaweed-volume: refuse a tier upload whose .dat grew during the copy Mirror the Go AttachRemoteDatFile check. The tier-up attach took the write lease only inside each step, so a needle write acknowledged after the upload read the .dat was missing from the object the volume then switched to, and the local .dat holding it was removed. Hold one write lease across the instance, revision and size checks, the remote-file update, the .vif save and the backend swap, as tier-down does; a size mismatch aborts, deleting the object and keeping the local .dat. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: compare the physical .dat size before a tier attach DiskFile pads its size past an unaligned tail, which a read-only load keeps, while the upload copies the physical bytes. Stat the file itself so such a volume is not refused as written during the copy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
dc03c3b1f7 |
shell: parallel moves and a per-run limit for volumeServer.evacuate (#12561)
* shell: parallel moves and a per-run limit for volumeServer.evacuate -maxParallelization runs up to N EC shard moves at once (default 1), with the ec.balance scheduler: never two shards of one volume at a time, free workers go to the least busy targets. The moves are planned first with the same target choice as before, then applied through volume_move.Mover.MoveEcShards, which verifies the target registered a shard before deleting it on the source. -movesPerExec limits how many volumes and EC shards one run moves (default 0 for unlimited), like -volumesPerExec in volume.balance, so a big server can be emptied in several runs. With -skipNonMoveable a shard that fails to move (a read error on a failing disk, for example) no longer drops the other shards of its volume; they are moved and the failures are listed at the end. Without it the run stops at the first failure as before. Losing the admin lock stops the run in either mode. Refs #12501 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: plan evacuation again after failed moves; fail on a lost lock A failed shard move kept the target slot it was planned on, so a readable shard that found no destination in the plan stayed on the server although the room was never used. Failed moves now give their targets back to the model and the shards without a destination are planned again, up to maxEcEvacuationRounds times; the failed shards are not tried again. With -skipNonMoveable a lost admin lock was printed as a skipped failure and the run returned success. ec.ErrLockLost is now exported and always fails the run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: volumeServer.evacuate reports -movesPerExec after an applied EC run The round loop only checked the limit after a failed move, so an applied run that stopped at -movesPerExec without failures never said to run it again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: volumeServer.evacuate plans again for as long as failed moves leave room The ten-round cap left readable shards on the server once more than ten rounds of unreadable shards had taken the free room first, and a rerun fails the same shards first again. Every round moves or fails each shard it plans and a failed shard is not planned again, so the rounds end without it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: volumeServer.evacuate gives a failed move back every slot it took When the target already held the shard in another disk type, adding it to the planned disk type still took a slot, but the undo gave back only the explicit one. Record the slots the plan took and restore exactly those. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec: an evacuation move that landed on its target keeps the target slot A move can fail after the target registered the shard, when only the source unmount or delete failed. Evacuate then gave the target's slot back and could plan another shard onto a full server. The runner now checks the target after a failed move and leaves a shard the target has out of FailedMoves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: volumeServer.evacuate gives back the slot of a failed move onto a shard the target had A target that held the shard before the plan reports it after any failed move, so the runner took such a move for one that landed and evacuate kept the slot the plan had taken. The runner now lists those moves in LandedMoves, and evacuate undoes the ones whose target had the shard, then plans again with the room they gave back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Chris LuandClaude Opus 5.5
|
fcfca3fee3 |
s3, filer: bound the wait for an upload slot (#12559)
* util: add WaitUntil, a cond wait bounded by a context and a timeout The caller holds the cond lock. Cancellation and the timeout wake the waiter through a broadcast taken under the same lock, so no wakeup is lost. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: bound the wait for an upload slot A PUT that arrived while the concurrent upload limits were exhausted waited on the condition variable with no bound. The handler reads nothing from the body while it waits, so a client that gave up and reset its connection went unnoticed: the socket and goroutine stayed pinned until a slot freed, and a stalled volume server grew the gateway's fd count by one per aborted PUT. The wait now ends on InflightUploadDataTimeout with 503 SlowDown, which S3 SDKs retry, and on request cancellation. An admitted upload takes its slot under the lock, and a finished upload broadcasts, so freed bytes admit every waiter that fits instead of one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer: bound the wait for an upload slot The filer's PUT/POST upload limits queued requests the same unbounded way. Reuse the bounded wait: reject with 429 after InflightUploadDataTimeout, as the volume server does, and stop on request cancellation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * command: add -inflightUploadDataTimeout for the S3 gateway and the filer weed s3 -inflightUploadDataTimeout, weed filer -inflightUploadDataTimeout and -s3.inflightUploadDataTimeout, weed server and weed mini -filer.inflightUploadDataTimeout and -s3.inflightUploadDataTimeout. The default of 60s matches the volume server's flag of the same name; 0 keeps the old unbounded wait. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: reject a queued upload without reading its body WriteErrorResponse drains the request body before answering. Behind the stats recorder the drain has no read deadline, and reading the body sends an Expect: 100-continue client the go-ahead to upload what is about to be discarded. Answer the rejection with an empty-body copy of the request so the drain is skipped; net/http closes the connection when a large unread body remains. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3, filer: release an upload slot under the cond lock A waiter checks the counters under the cond lock and then waits. A release that decremented and broadcast without the lock could land between the two, so the waiter slept with a slot free until the next release or its timeout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3, filer: close the HTTP/1 connection when rejecting a queued upload net/http reads up to 256 KiB of an unread request body before sending a keep-alive reply, so a client that stalls its body held back the 503 or 429 until the connection idle timeout. Connection: close skips that read. HTTP/2 is left alone: there it would shut down the whole connection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
d659228338 |
shell: s3.clean.uploads frees an open upload under the object write lock (#12558)
* shell: s3.clean.uploads deletes uploads over filer gRPC Same DeleteEntry the HTTP handler runs, without minting a filer JWT, and the gRPC path is the one a locked, routed delete needs next. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: s3.clean.uploads frees an open upload under the object write lock The cleaner deleted a stale upload recursively with no lock, so a CompleteMultipartUpload could pass its upload-exists check after the part entries and their chunks were gone, and publish an object over freed chunks. Do what AbortMultipartUpload does: hold the object's distributed write lock, and free an open upload in an ObjectTransaction routed to the object's owner filer, conditioned on the object not carrying the upload; if it does, delete the metadata only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: s3.clean.uploads keeps the object write lock for the whole cleanup A large upload's recursive delete can outlast one 15 second lease, and a completion on the distributed-lock path could then take the lock and commit over chunks being freed. Renew the lock in the background and skip the delete once it is lost. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: s3.clean.uploads locks the path a completion commits under For a trailing-slash key a non-versioned completion writes and locks the entry inside the directory the key names, not the object path, so the cleaner's transaction did not serialize with it. Lock and check the same path the completion uses for the bucket's versioning state; the route key stays the object's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3api: abort locks the path a completion commits under A routed abort locked and checked the object path, but for a trailing-slash key a non-versioned completion commits under the entry inside the directory the key names. Use the same path the completion locks for the bucket's versioning state. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: end s3.clean.uploads bucket failure line with a newline Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * cluster: NewBlockingLongLivedLockWithin bounds the wait on a held lock Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: s3.clean.uploads skips an upload whose object lock stays held Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * cluster: share the contended lock filer setup across lock tests Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: s3.clean.uploads reads the bucket's versioning state per upload The commit path a routed completion locks depends on versioning for a trailing-slash key. Reading it once per bucket before the run left a window as long as the run; read it just before the delete, as the gateway's abort does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
b11af30cb1 |
mount: remove the local control socket when the mount exits (#12560)
* mount: remove the local control socket when the mount exits weed mount listened on /tmp/seaweedfs-mount-<hash>.sock and removed it only when a later mount of the same directory started, so every unmount or SIGTERM left one socket file behind. Remove it on the normal return path and in the interrupt hook, unless a later mount of the same directory has already replaced it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: remove the control socket before a failed mount exits glog.Fatalf and glog.Exitf call os.Exit, which skips the deferred removal, so a mount that failed after creating its socket (mount point not available, fusermount failure) still left it behind. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mount: register the socket cleanup hook before mount setup A SIGTERM that arrived while the mount was still setting up, for example waiting on the filer to create the mount root, exited through grace without running the deferred cleanup. Registering the hook right after the listener is created makes it run last, after the unmount and filesystem hooks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
7f18576798 |
filer/etcd: serve volume.fsck from one etcd revision (#12555)
* filer/etcd: scan entries from one revision volume.fsck walked etcd one directory at a time, so a file moved from a directory not yet listed into one already listed was seen under neither name and its chunks were reported as orphans. The etcd store now implements filer.EntryScanner: every page is read at the revision of the first read, and a compacted revision fails the scan instead of falling back to the walk. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer/etcd: read hard links at the scan revision Hard-link records resolved during a scan were read at the current revision, so a link updated meanwhile mixed two instants. Reads made with the scan's context now use its revision. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: list etcd among the stores volume.fsck reads from a snapshot Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer/etcd: skip hard-link records in a scan Hard-link records live beside the entries under their random ids, so a root scan could read one as an entry with a path made of random bytes, which the traversal stream cannot send. Skip hard-link ids and keys whose directory is not valid UTF-8. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer/etcd: recognize a hard-link record by the id it stores A hard-link id can be valid UTF-8, so the key shape alone let some records through a root scan as phantom files. The record's value is the shared entry, whose HardLinkId is the record's own key; no entry key can equal its own hard-link id. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer/etcd: scan entries under an invalid-UTF-8 directory The filer sanitizes names but stores directories as given, so a path written over HTTP can leave entries the listing never reaches. Skipping them let volume.fsck count their chunks as orphans. Send them under the sanitized directory instead, the substitution the filer uses for names. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer/etcd: fail a scan whose sanitized path names another entry An entry under an invalid-UTF-8 directory is reported under the sanitized directory. When that path is also another stored entry at the scan revision, or another sanitized entry in the same scan, acting on the reported path (volume.fsck deleting it) would reach the other file, so the scan fails and names the stored path instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
e23a952878 |
ec: don't index ev.Shards[0] when the last shard was unmounted (#12538)
* ec: don't index ev.Shards[0] when the last shard was unmounted * ec: skip a runtime emptied before the scrub merge reads its shards FindAllEcVolumes releases each location lock before MergeEcRuntimes runs, so an unmount can leave a merged runtime with no shards and the width scan indexed Shards[-1]. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec: treat a scrub merge with no shard left as a vanished volume Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec: read Shards once in ShardSize so a concurrent unmount cannot empty it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec: read each runtime's Shards once in the scrub merge The vanished check and the slot construction read Shards separately, so an unmount between them produced a non-nil view with no shards. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec: anchor the scrub merge on a generation that still holds shards An unmount can empty the newest runtime while an older one keeps its shards, which left a merged view anchored on the emptied generation with no slots and the holder reported as skipped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Mansur Akhmadi <10739460+mansur-akhmadi@users.noreply.github.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
641ab6204b |
shell: fs.mergeVolumes leaves in-progress multipart parts in place (#12556)
* shell: fs.mergeVolumes leaves in-progress multipart parts in place CompleteMultipartUpload reads the part entries and later commits the object with their fids, guarded only by the upload directory. A merge that moved a part chunk in between deleted the source needle the new object pointed at. Parts stay where they are until the upload completes or is cleaned, and their chunks still count as referenced sources. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: note in fs.mergeVolumes help that upload parts stay in place Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: match upload parts when the buckets folder is the root Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: count chunks behind a part's manifest as referenced sources A skipped part whose chunks on a merge source sit behind a manifest was not counted, so the merge printed no skip line and warned that nothing references the source. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: keep a part's direct source references when its manifest is unreadable Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: count chunks resolved before an unreadable part manifest ResolveChunkManifest returns the data chunks of earlier manifests along with a later manifest's error; record them before returning so the unreferenced-source warning does not fire for a source they reference. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: test the partial manifest result without a timing delay The unreadable manifest is now nested under a readable part manifest, so the first level resolves fully before the failing lookup runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
e9606c8ecc |
filer.meta.backup: resume an interrupted metadata copy and add -entriesPerSecond (#12549)
* filer_pb: add a depth-first traversal that can resume after a path TraverseDfs walks a tree in listing order and reports each finished path, so a caller can save one path and continue from it later. Pages are read whole before visiting, and the entries of a page are visited by up to five workers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.meta.backup: resume an interrupted metadata copy The walk saves the last finished path every few seconds, and a walk that continues an unfinished one starts after it. A fresh walk clears the saved path first, and a finished walk clears it with the walk start. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.meta.backup: add -entriesPerSecond to throttle the metadata copy Each entry the full copy reads waits on a rate limiter, and pages are listed only after the previous page is copied, so the flag also bounds how fast the copy lists the source filer. 0 keeps it unlimited. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer_pb: report each file of a depth-first walk once all before it are done A throttled walk only reported progress after a whole listing page, so a slow copy could go many minutes without a resumable path. Visit entries in runs that end at a directory and report each file as soon as every entry before it is done. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
0ef0260844 |
volume: fail a vacuum step that finds the volume already compacting (#12553)
* volume: fail a vacuum step that finds a compaction already running Compact, CompactByIndex and CommitCompact returned success when another compaction held the volume. A second, uncoordinated vacuum (an expired admin job still running, a shell volume.vacuum next to the worker) then went on to commit or clean up the first run's .cpd/.cpx. Return a "volume N is compacting" error instead, and refuse cleanup while a copy is in flight, as the Rust volume server already did. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: check the compaction files before a commit closes the volume CommitCompact closed the needle map and data file and wrote the .cpc marker before it found .cpd/.cpx missing. A duplicate commit, such as the second of two overlapping vacuums, then left the volume registered with no open files: it dropped out of the heartbeat and looked lost to volume.list and volume.fix.replication until a restart. Check for the files first so the refused commit leaves the volume as it was. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: run makeupDiff before a commit closes the volume CommitCompact closed the needle map and data file before makeupDiff, so a makeupDiff failure (a compaction revision another commit moved, an I/O error) returned with them closed: the volume stayed registered but missing from the heartbeat until a restart. makeupDiff reads the live files through its own handles while the commit holds the write lock, so close them only once it has succeeded, as the Rust volume server does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: log a volume the heartbeat skips because its files are closed A volume left without its needle map or data file vanished from the heartbeat with no log line, so the master saw a replica disappear while its files were still on disk. Warn on each skip, as the deleted-.dat case next to it already does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: claim the compaction flag while cleanup removes the temp files Cleanup only read the flag, so a compaction could start between the check and the removal and lose its .cpd/.cpx. Claim it for the cleanup instead, in Go and Rust. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
a4a68c885a |
filer.sync: move the watermark over jobs that finished out of order (#12554)
* filer.sync: move the watermark over jobs that finished ahead of an older one The watermark only took the finishing job's timestamp when it was the oldest in flight, so jobs that completed before an older one were never recorded and a quiet stream kept the offset behind finished events. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.sync: keep one finished timestamp per gap behind a running job A long job kept every later completion in the done heap. Only the newest one between two running or failed timestamps can become the watermark. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.sync: test done-heap compaction across several running and failed jobs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
000baf8dfa |
s3: return an error for GETs canceled before the first byte (#12502)
* s3: answer GETs canceled before the first byte A stream that fails before writing anything now always gets an S3 error response, canceled errors included. The idle timeout and a shared lookup canceled by another request both cancel the request context while the client is still connected, and returning silently sent that client a 200 with Content-Length: 0. A client that has gone away never sees the write. Refs #12500 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: mark inline GET write errors as written; bound the idle-cancel test An inline object's write error comes after the status line, so the caller must not write an S3 error on top of it. The idle-cancel test caps the client wait and gives the response write its own deadline, keeping only the read side on the 200 ms idle timeout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: test a GET whose request is canceled before the first byte Drives GetObjectHandler through a chunk lookup that cancels the request context, so a handler that returns before answering shows up as the empty 200 again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3: check the cancel test reached the chunk lookup Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Chris LuandClaude Opus 5.5
|
ea6e9d670a |
volume: forward no Content-MD5 for a JPEG the orientation fix rewrote (#12557)
* volume: forward no Content-MD5 for a JPEG the orientation fix rewrote With -images.fix.orientation the primary stores and forwards the rotated bytes, but sent replicas the client's digest of the original upload. The replica checks Content-MD5 against what it receives and rejects the write, so replicated JPEG uploads with a digest failed. The Rust volume server already drops the digest once the body is rewritten. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: test the forwarded Content-MD5 through PostHandler and a replica The test now posts each upload to a primary whose fake master names a real replica volume server, so it covers the JPEG gate, ReplicatedWrite and the replica's own Content-MD5 check, and reads the stored bytes back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: fail the replica forwarding test instead of hanging when nothing is forwarded Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: compare the primary and replica bytes in the replica forwarding test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
1202c5ba08 |
filer.copy: compute chunk md5 so files get a real ETag (#12497)
* filer.copy: compute chunk md5 so files get a real ETag Chunks uploaded by filer.copy had no ETag, so every multi-chunk file got the md5 of the empty string plus -N. An overwritten file kept its old ETag and a revalidating client got 304 for changed content. * filer.copy: store chunks in offset order Chunks are collected in upload completion order, and the entry ETag hashes the chunk ETags in stored order. Identical files copied with concurrent chunk uploads got different ETags, and a file with swapped equal-sized chunks could keep the old one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.copy: test chunk order and chunk md5 on chunked uploads Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
b9866bae4f |
weed/shell: support -noLock flag in volumeServer.evacuate (#12501) (#12516)
* weed/shell: support -noLock flag in volumeServer.evacuate (#12501) * shell: return flag parse errors from volumeServer.evacuate A typo such as -noLock=invalid printed the parse error but the command still succeeded, so a piped script treated the evacuation as done. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: volumeServer.evacuate -h prints usage without failing Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
f068fe7f09 |
fix(shell): fs.log.purge purges every filer, not just one (#12524)
* fix(shell): fs.log.purge purges every filer, not just one Each filer keeps its own metadata logs in its own store, so purging a single filer leaves the chunks referenced by the others and the volume vacuum cannot reclaim anything. The shell picked that single filer at random (commands.go:208) when no -filer flag was given, and the admin script worker pinned it to filers[0] (admin_script_handler.go:574), so the same command purged a different filer on every run and never the whole cluster. fs.log.purge now resolves the filer list from the master and purges each of them, continuing when one is unreachable and reporting the failures at the end. -filer=<addr>,... restricts the purge to a subset. Also stops mutating the shared command environment per filer, so a long-lived shell keeps its original filer. * fix(shell): bound filer discovery and cover the cluster path Address review feedback: - resolveLogPurgeFilers waited on the master with context.Background(): a shell whose master connection dropped would hang before purging anything, where it used to purge the connected filer. Bound the master round trip with a timeout and fall back to the connected filer when the master cannot be reached or lists no filer. - the tests only exercised the single-filer fallback. Rename that test to what it actually covers and add one asserting every filer of the list is attempted and the shared command environment keeps its original filer. * shell: reject an fs.log.purge -filer list with no addresses A comma or whitespace only -filer value fell through to master discovery and purged every filer instead of the intended subset. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: fall back to the shell's filer without a second master lookup When the master listed no filer, FilerAddress() ran filer discovery again and could wait out the shell's ConnectTimeout before failing. Also cover master discovery with the fake discovery master. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * shell: test the fs.log.purge filer loop against a fake filer The previous test looped over the addresses itself, so it passed no matter what Do did. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Martin Morez <martin@morez.org> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 4.8
|
fb5ffb815f |
filer: validate bucket names before the SQL DDL hooks (#12547)
* filer: validate bucket names before the SQL DDL hooks The SQL store builds CREATE TABLE and DROP TABLE statements by formatting the bucket name into the query. The DML paths already reject names that fail VerifyS3BucketName, but the bucket-creation and bucket-deletion hooks ran the DDL without that check, so a name reaching them through the filer or WebDAV directory path (which skips the check for dot-prefixed directories) could inject stacked statements. Apply the same validation the DML paths use. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * filer: check that a valid bucket runs both DDL statements Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
d350a0ad03 |
weed-db: end startup when password authentication fails (#12550)
* weed-db: end startup when password authentication fails The failure branches returned the result of writing the error response, which is nil on success, so the handshake carried on to ReadyForQuery. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-db: test startup with wrong and unknown credentials Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-db: check the auth error code and fail on stray read errors in the startup test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
f3150c3495 |
admin: escape file properties before rendering the modal (#12548)
* admin: escape file properties before rendering the modal The Properties modal concatenated the entry name, path, MIME type and extended attributes into HTML, so object metadata written over S3 could inject markup into the admin page. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * admin: keep every property key when escaping Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
8d96985131 |
security: scoped filer tokens are not admin credentials (#12551)
* security: do not accept a path- or method-scoped filer token as admin Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * security: test that scoped filer tokens fail the admin token check Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
2143761319 |
fix(s3tables): let a table renamed across namespaces keep committing (#12515)
* s3tables: let a table renamed across namespaces keep committing
* s3tables: keep the same-directory exception off a directory another table shares
A record written before metadata locations were confined to their namespace
(
|
||
|
Tsic
|
518009dcd4 |
filer: apply filerGroup prefix to bucket default collection (#12499)
filer: prefix bucket default collection, assign copy collection explicitly S3 server-side copy paths (CopyObject/UploadPartCopy) assign volumes with an empty collection and relied on the filer's bucket fallback. The fallback used the raw bucket name, while PutObject uses getCollectionName(bucket) = <filerGroup>_<bucket>, so copies landed in a second, unprefixed collection. Prefix the filer's bucket default with the filer group, and have the S3 copy path pass that collection explicitly, like PutObject. Direct filer writes keep path-rule precedence, so this only aligns S3 copy with S3 put. Adds regression tests for the fallback, explicit collection precedence, and the S3 copy collection resolver. |
||
|
Ryan Leung
|
7ad8419815 |
shell: preserve completed uploads for bucket-root objects (#12545)
Signed-off-by: rleungx <rleungx@gmail.com> |
||
|
Mansur AkhmadiandMansur Akhmadi
|
9886ac0bcf |
topology: take dn.Lock when a heartbeat links a new disk type (#12535)
Co-authored-by: Mansur Akhmadi <10739460+mansur-akhmadi@users.noreply.github.com> |
||
|
Mansur AkhmadiandMansur Akhmadi
|
9f4b652522 |
master: make admin lock lease and release atomic (#12532)
Co-authored-by: Mansur Akhmadi <10739460+mansur-akhmadi@users.noreply.github.com> |
||
|
Saarthak Gupta
|
017e42633f | shell: return flag parse errors from s3.bucket.create (#12544) | ||
|
Mohd Quamar TyagiandChris Lu
|
5863e32318 |
weed/s3api: avoid truncated marker rescan in ListObjects (#12540) (#12541)
* weed/s3api: avoid truncated marker rescan in ListObjects (#12540) Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com> * fix(s3): validate marker directory candidates Respect prefix filters and internal-directory exclusions when recovering marker candidates. Restrict lookups to direct children, propagate read failures other than not-found, and deduplicate directory marker echoes. Add regression coverage and model promoted objects consistently in the XML-unsafe prefix fixture. --------- Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
d485d2c9cd |
filer.sync: shorten retries for confirmed missing chunks (#12546)
* filer.sync: shorten retries for confirmed missing chunks * filer.sync: clarify missing chunk grace period names |
||
|
Chris Lu
|
6c473433c5 |
vector: bound request buffering before authentication and document installation (#12537)
* s3vectors: bound body buffering before authentication * docs: include vector engine installation and source builds * s3vectors: prevent MIME sniffing of JSON error responses * docs: load vector Cargo configuration when building from source * docs: note the native protoc requirement for Windows vector builds * docs: make vector source checkout and working directory explicit * ci: handle deb822 sources during Ubuntu mirror failover * ci: bound Ubuntu mirror attempts and reject partial index updates |
||
|
Chris Lu
|
8236c8d136 |
vector: integrate engine health, admin management, and local deployment (#12517)
* vector: expose request metrics and registration-aware health endpoints * vector: report immutable read cache and storage fetch metrics * docs: describe vector engine health, metrics, and maintenance * vector: count canceled requests when their handler is dropped * vector: account for uncached large immutable storage reads * vector: reliably release the stalled credential reader in tests * vector: observe large reads as their streamed body is consumed * vector: bound heartbeats on established master registration streams * test: exercise hardened vector maintenance on Go and Rust volumes * vector: register engines in their configured filer group * vector: make retried index creation preserve compatible datasets * vector: allow registration to advertise a separate address * test: release stalled credential readers without blocking FIFO cleanup * test: allow hardened vector coverage in a named filer group * vector: discover engines in the same filer group as their gateway * vector: fall back when the advertised address is empty * admin: expose vector readiness, maintenance, and management tools (#12520) * s3vectors: share vector discovery with trusted management clients * admin: show vector engine readiness and recent maintenance activity * admin: suggest S3 Vectors IAM actions from the supported operations * admin: manage vector indexes, policies, tags, encryption, and query data * admin: validate SSE-S3 support before creating encrypted vector buckets * admin: discover vector engines in their registration group * admin: probe vector engines concurrently with independent time budgets * admin: verify encryption capability on the remote S3 gateway * admin: defer discovery until after the HTTP server can start * admin: distinguish unknown vector health services from failed engines * admin: count workers that can execute vector index maintenance * admin: allow read-only users to query vectors and inspect tags * vector: scope gateway and admin discovery to the configured filer group * admin: refresh policy state after mutations and serialize policy controls * vector: drain active RPCs before closing replaced discovery connections * vector: respect request cancellation while waiting for discovery * vector: prefer another registered engine after a transport failure * admin: check all masters before reporting no registered vector engines * iam: honor attached vector policies without broadening legacy grants * vector: allow short-lived discovery clients to close their connections * vector: refresh cached master addresses for each discovery lookup * iam: enforce vector bucket tag conditions in attached policies * iam: authorize vector bucket creation with the requested tags * test: exercise vector IAM grants and tag denies on Go and Rust volumes * admin: reject authenticated vector mutations without an admin role * test: run vector IAM assertions before the shared engine is stopped * vector: add native mini startup, shell tools, and a runnable Compose example (#12522) * s3vectors: share vector discovery with trusted management clients * admin: show vector engine readiness and recent maintenance activity * admin: suggest S3 Vectors IAM actions from the supported operations * admin: manage vector indexes, policies, tags, encryption, and query data * admin: validate SSE-S3 support before creating encrypted vector buckets * admin: discover vector engines in their registration group * admin: probe vector engines concurrently with independent time budgets * admin: verify encryption capability on the remote S3 gateway * admin: defer discovery until after the HTTP server can start * admin: distinguish unknown vector health services from failed engines * admin: count workers that can execute vector index maintenance * admin: allow read-only users to query vectors and inspect tags * vector: scope gateway and admin discovery to the configured filer group * admin: refresh policy state after mutations and serialize policy controls * vector: drain active RPCs before closing replaced discovery connections * vector: respect request cancellation while waiting for discovery * vector: prefer another registered engine after a transport failure * admin: check all masters before reporting no registered vector engines * iam: honor attached vector policies without broadening legacy grants * vector: allow short-lived discovery clients to close their connections * vector: refresh cached master addresses for each discovery lookup * iam: enforce vector bucket tag conditions in attached policies * iam: authorize vector bucket creation with the requested tags * test: exercise vector IAM grants and tag denies on Go and Rust volumes * admin: reject authenticated vector mutations without an admin role * test: run vector IAM assertions before the shared engine is stopped * mini: optionally start and supervise the vector engine * mini: bootstrap vector buckets with existing Lance catalog helpers * shell: manage vector buckets, indexes, and vector data * docs: provide a runnable vector and maintenance worker Compose stack * shell: scope vector discovery to the session and close its connections * mini: honor the bind address for vector and metrics listeners * mini: reserve vector ports before assigning gRPC and compatibility listeners * mini: wait for vector storage setup and master registration * mini: create requested vector buckets without requiring the Lance HTTP listener * shell: refresh master addresses during vector discovery * mini: resolve sibling vector executables with platform extensions * vector: preserve uncertain creates and fix mini listener allocation (#12525) * vector: retain catalog records when index creation has an unknown outcome * mini: keep chosen listener ports reserved throughout allocation * mini: resolve hostname binds before starting vector listeners * vector: fix integration test imports and recover interrupted index creation (#12533) * shell: register vector operations from the command package * s3vectors: resume interrupted index initialization safely * mini: exclude disabled WebDAV from listener allocation * s3vectors: retain uncertain creates across refused retries * s3vectors: claim pending indexes before retrying initialization * s3vectors: compare resolved distance metrics during recovery * s3vectors: preserve metadata keys when retrying initialization * s3vectors: verify ownership before initializing new catalog records |
||
|
Chris Lu
|
84cd3c621b |
s3vectors: integrate bucket metrics, audit events, and request limits (#12514)
* s3vectors: attribute request and traffic metrics to authorized buckets * s3vectors: emit resource and identity audit events * s3vectors: apply existing S3 request limits and preserve JSON errors * s3vectors: select limit resources by the dispatched operation * s3vectors: count decoded chunked request bodies against byte limits |
||
|
Chris LuandClaude Opus 5.5
|
5c910ad546 |
topology: list collections without nesting the collection map read lock (#12495)
* topology: list collections without nesting the collection map read lock Items() already takes the read lock and copies the values. Holding it around the call made ListCollections take the read lock twice, so a DeleteCollection waiting for the write lock in between blocked the second read lock and wedged the master. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * topology: test ListCollections against a concurrent collection delete Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
github-actions[bot]
|
4ab66fb76d | 4.53 | ||
|
Chris LuandClaude Opus 5.5
|
76fddab5ef |
master: keep a re-registered filer in the lock ring when its old stream ends (#12494)
* master: move the KeepConnected stream cleanup into removeClusterNode Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * master: drop a filer from the lock ring only with its last registration A filer's replacement can register under the same address before the master ends the old stream. The cluster list counts both registrations, but the old stream's cleanup removed the address from the lock ring outright, leaving the live filer out of the ring until it re-registered. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * master: order a stream's cluster list and lock ring changes The cleanup of an ending stream could see its address removed from the cluster list, then a new registration of that address add it to the list and the ring, and only then remove it from the ring. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * master: take a leaving filer's reconnect out of the lock ring A filer that reconnects while leaving registers with LeaveLockRing set. Its broken stream can still be open on the master, and since that stream's end no longer drops an address that is still registered, the reconnect removes the address from the ring itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris Lu
|
35e98177d9 | shell: test collection.list keeps live statistics next to offline names (#12492) | ||
|
Ping Qiuandcodex03
|
0705e20fe9 |
shell: include collection names when topology statistics are unavailable (#12490)
fix(shell): include collections without live topology in list Co-authored-by: codex03 <codex03@local.invalid> |
||
|
Chris LuandClaude Opus 5.5
|
4e12c13416 |
Volume encryption follows the filer: weed s3, weed-vector, weed-worker, weed-vector-cli (#12487)
* s3: encrypt volume data when the filer is configured to weed s3 read the filer's configuration at startup (buckets dir, filer group, masters, metrics, maxMB) but not its -encryptVolumeData, so a cluster that turned encryption on at the filer still had S3 uploads stored plain, while uploads through the filer itself were encrypted. The gateway now encrypts when either its own -encryptVolumeData or the filer's is set; weed-vector, weed-worker and weed-vector-cli already follow the filer's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: weed-vector, weed-worker and the CLI take volume encryption from the filer They already encrypted chunks for the volume servers when the filer's configuration said so, or when their own --encryptVolumeData did. With weed s3 following the filer too, the filer's -encryptVolumeData is the one switch for every writer, so the three flags are removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * S3 Vectors tests: turn volume encryption on at the filer VECTOR_ENCRYPT_VOLUMES=1 set -encryptVolumeData on the gateway, the engine and the worker. It now sets it on the filer alone, so the leg checks that each of them follows the filer: the gateway's stored chunks carry a cipher key (with a gateway that ignores the filer, the leg fails on "the gateway stored a chunk without a cipher key"), and so do the engine's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker README: volume encryption follows the filer Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3: help text says the filer's -encryptVolumeData turns S3 encryption on Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * S3 Vectors tests: plain legs check the engine stored no cipher key Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Mansur AkhmadiandMansur Akhmadi
|
10ea662522 |
s3: don't report a version delete as done when the filer lookup failed (#12482)
* s3: don't report a version delete as done when the filer lookup failed deleteSpecificObjectVersion treated any getEntry error as "entry is gone": - the null-version precheck returned success without deleting anything, - the recheck after a failed rm returned success while the entry may still be there, - a failed .versions lookup was read as "not the latest version", so the latest blob was removed without repointing the pointer first. Only filer_pb.ErrNotFound now counts as a miss. Other errors fail the delete, so the S3 client or the lifecycle worker retries it. The lookups go through retryFilerOp, like the other .versions reads in this file, so a single transient filer error doesn't turn into a 500. * s3: include recheck error when a version delete fails * s3: recheck a failed version delete once instead of retrying --------- Co-authored-by: Mansur Akhmadi <10739460+mansur-akhmadi@users.noreply.github.com> |
||
|
|
f99cc6ca94 |
volume.fsck: batch and deduplicate filer entry deletions (#12480)
* volume.fsck: batch and deduplicate filer entry deletions In volume.fsck with -findMissingChunksInFiler and -reallyDeleteFilerEntries, deleting orphaned filer entries was slow and CPU intensive due to tiny worker channel buffers (128), lack of batch processing, redundant HTTP DELETE requests for multi-chunk files, and lock contention on parent directory tracking. Optimize by: - Adding startPurgeBatchWorkers with 4096-buffered channels to prevent producer stalls and batch opportunistic deletes up to 64 items. - Deduplicating missing chunks per file path during volume checks so that multi-chunk files are queued once. - Processing batches with in-batch path deduplication and aggregating parent directories before updating purgedDirs in a single lock acquisition. - Buffering temporary fid file writes (64KB) during the traversal to avoid millions of unbuffered syscalls. * volume.fsck: report a failed write of the filer fid files The fid files are written through a buffer now, so the last part of each one reaches the disk only in the deferred Flush. Its error was dropped: a full temp disk left a short fid file and the collection still succeeded, and the volume check would then take the chunks missing from it for orphans. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume.fsck: retry a batched path whose earlier delete failed A path was marked done before its delete ran, so a second copy of it in the same batch was skipped after a transient failure and the entry stayed. Only skip paths already deleted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume.fsck: keep reporting a failed DELETE response read The non-verbose path now discards the body without checking the copy, so a connection cut after the headers went unreported. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume.fsck: keep one purge worker starter and one delete function startPurgeWorkers and the single-path httpDelete were left only as wrappers the tests called. Give their names to the batch versions and adapt the tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
4f24b756ba |
s3tables, iceberg: give tables created without a location their own directory (#12475)
* s3tables: give a table created without a location its own directory An Iceberg table created without a location kept its files under its name path, so a rename left them where the next table of that name would land. Such a table now gets s3://<bucket>/<namespace>/<name>-<uuid>, stored as the record's warehouseLocation and returned by GetTable; its first commit writes there without moving the root. Records without one keep resolving to their name path, so existing tables need no rewrite. Lance datasets stay at their name path, where Lance clients and the vector engine look for them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iceberg: create tables and views in a directory of their own A table or view created without a location took its name path, or a unique one only when that path was already taken. It now always gets <name>-<uuid>, so a later rename never leaves its files under a name another table can take. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iceberg: load and commit a table at its warehouse location A table created through S3 Tables without a location has no metadata until its first commit; loading or committing it now uses the directory recorded for it, and only a record without one falls back to its name path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iceberg: fail a stage-create whose marker is not saved The marker is how the commit finds the location the stage-create handed out. Without it the commit fell back to the name path while the client's files sat under the returned location. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: expect tables created without a location in their own directory Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: rename a table that keeps its files elsewhere before reusing its name Enterprise keeps the storage marker on a renamed table's files, so the name of a table whose files sit under its name path stays taken there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * s3tables: remove the table directory when a create backs out Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iceberg: answer a create that lost the race with the registered table Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iceberg: answer a view create that lost the race with the registered view Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iceberg: fail a staged commit whose name another create registered Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iceberg: remove the directory of a create whose name another table took Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iceberg: keep the metadata of a staged commit whose name is taken Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: write the lakekeeper table's data under its warehouse location Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iceberg: commit a staged table at the location of its own stage Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iceberg: never continue another stage when a commit's own marker is gone Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * iceberg: test filer honors exclusive creates and token-guarded deletes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
55a4a5974c |
ec_balance: make the per-rack cap on global moves a worker setting (#12478)
* ec_balance: make the per-rack cap on global moves a worker setting The global load-balancing phase planned at most 10 moves per rack per detection run, hard-coded. On a cluster where one disk per machine holds several times its siblings' shards, that is about 20 moves an hour against tens of thousands needed, however many moves the worker can execute. Add global_max_moves_per_rack to the plugin worker config. The default stays 10, so nothing changes unless it is raised. It is clamped to 1..5000 because the planner reads 0 as unlimited and would plan a whole drain in one run. Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi * ec_balance: clamp global_max_moves_per_rack before narrowing it to int A value past the int32 range made ReadIntConfig return the default, so a request asking for more than the maximum got 10 moves instead of 5000. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec_balance: test the configured global cap through Detection The helper test still passed if Detection handed the planner a literal cap. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec_balance: plan no more global moves per rack than the detection keeps Each global move scans the source node's volumes, and the detection keeps at most max_results moves. With the cap at 5000, three skewed racks of 20,000 volumes planned 15,000 moves to keep 500, taking about 70 s of the 300 s detection budget; bounded by max_results it takes about 5 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec_balance: test the configured global cap with a result limit Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
78a3bc9dc3 |
filer.remote.sync: -allMounts writes back every mounted directory from one process (#12476)
* filer.remote.sync: -allMounts writes back every mounted directory from one process A filer with many remote mounts needs one filer.remote.sync process per mount. -allMounts runs the existing -dir write-back once per directory in the mount table, in one process: each with its own subscription, client id (the filer keeps one subscription per client id), offset, processor and retry loop. A supervisor reads the mount table every -mountsCheckInterval (5s) and starts a write-back for each mounted directory that has none running, 10 at a time, 500 ms apart. It never stops one because the live table changed: a write-back ends itself where the -dir process exits, at the first mount-table version it receives that ends its mount (#12472), and a directory still mounted is started again from its saved offset, after a pause that grows from 1s to 30s while it keeps ending within a minute. A directory mounted while the process runs, without a saved offset, waits up to 2 minutes for one (a mount followed by a pull sets it after the pull). A write-back that ends at an unmount 5 times in a row while the table still mounts it at the same location is logged as a warning. To run under a supervisor, a follow: - returns errMountGone at the end of its mount instead of exiting. admitUntilMountEnds takes the end action: the -dir process keeps draining and exiting as before; a -allMounts follow admits nothing more (events after the end can still arrive in the same batch or before the stream closes, including remote.unmount's purge and a remount), ends its stream, drains, and saves no offset. - drains its admitted jobs before it returns, however it ended, so a successor never races a job still running for the same path. - saves its offset only over the value it read at its start or last saved (offsetGuard), both at its checkpoints and at its end. Otherwise a write-back behind in the log can overwrite the offset remote.unmount writes after its purge (or a mount after its mapping write), and its successor replays the unmount and never writes the remounted directory back. Once it sees the offset moved it saves nothing more and keeps no resume position. A value whose write reported an error counts as its own when read back. - reads the stored offset once, before the mount table, and uses it as both its start and its guard's value: read after the table, an unmount completing in between would start a follow past the unmount's table. - keeps a resume position for its next attempt only with the mount location it was earned under and the offset stored at the time; it is dropped when the directory is mounted elsewhere or someone else moved the offset. - turns a panic in a job into that job's error, so one directory's failure does not end every directory's write-back. Fixes #12471 * filer.remote.sync -allMounts: retire a failed follow whose mount is gone A follow that failed before its stream reached the unmount (or before it started) was retried every 30s forever: DetectMountInfo reports the directory not mounted, which is not errMountGone. No stream runs on the retry path, so a directory the live table no longer maps is retired there and started again by reconcile if it is mounted again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync -allMounts: drop the restart pause of an unmounted directory ended() left notBefore and ends behind for every directory ever mounted. reconcile now drops them once the directory is neither mounted nor followed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: log a -dir drain held at the mount end The -dir exit drained with processor.Drain, so a remote call hung during an unmount or move held the process silently; the follow's own drain already warns every 5 minutes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: keep the resume position through a failed mount read A transient DetectMountInfo error dropped the only position behind the event a follow failed on. With -timeAgo and no checkpoint saved yet, the retry then started at a later now minus timeAgo and skipped that event. The next attempt that reads the table still checks the position against the mount location and the stored offset. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: judge a kept resume position by the directory's remote location resumeFor compared the mount's raw location, so a -dir below a mount kept its position when the same remote was mounted one level up (or down), although its paths now map to other keys; a replay from that position skips startAfterMountOffset and writes old events to the new keys. The position now carries dirRemoteLocation and is checked against it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync -allMounts: warn about a replayed unmount only from an unmoved offset The warning counted every end at an unmount while the directory stayed mounted at one location, so write-backs that each saved progress, or unmount and remount cycles between two reconciles, also tripped it, and it told the operator to set the offset to now, skipping events. An end saves no offset, so a run of ends is one replay only while the stored offset stays put: a moved offset now starts a new run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync -allMounts tests: wait for the supervisor before moving the frozen clock A reconcile or end recorded after the clock moved would start its wait or pause at the moved time, which the frozen clock never passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * remote.mount: save the sync offset right after the mount mapping A new mount had no saved offset, so filer.remote.sync -allMounts waited the full allMountsOffsetWait before following it and then started from the directory's creation time anyway. remote.mount and remote.mount.buckets now save the time taken just before the mapping write, keeping an offset remote.unmount saved for a remount. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * remote.mount: save the directory's creation time as the sync offset The mount time skipped local files already in a -nonempty directory, which a write-back with no offset replays from the directory's creation. Saving that creation time keeps the start position and only ends the -allMounts wait. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * remote.mount: warn instead of failing when the sync offset is not saved The mount mapping is already saved by then, and a mount without an offset still starts from the same creation time after the -allMounts wait. Failing stopped remote.mount.buckets before the remaining buckets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
554593796e |
ec_balance: drain a server holding far more shards than its siblings (#12477)
* ecbalancer: drain a full server through any target that can take a shard The global phase paired the fullest server with the single emptiest one and stopped for the whole rack when that pair had no legal move. Where every volume already has a shard on each machine, no cross-machine move keeps the spread, so on hosts running one volume server per disk the first server's disk kept its pile while its siblings on the same machine stayed nearly empty: the phase never planned a move. Try sources fullest first and targets emptiest first and take the first pair with a legal move, so such a server drains onto its own machine's emptier servers. A same-machine move goes only to a server holding none of the volume, so one disk never carries two of its shards. Per-machine counts are indexed once per rack and pairs without a move are remembered until a move can change them, so an uncapped drain stays fast. Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi * ec_balance: keep global moves ahead of within-rack moves under the detection cap Detection truncated the plan to the per-cycle job limit in planner order, which puts the global moves last. A long within-rack backlog pushed all of them past the limit, so even a planned global move never ran. Order dedup and cross-rack first, then the capped global phase, then within-rack. Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi * ec_balance: keep each volume's moves in plan order when promoting global moves The admin scheduler runs only the first proposal per volume each cycle. A global move promoted ahead of its volume's within-rack moves can take a shard that one of those moves placed, so it ran while the shard was still on the original server and failed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ecbalancer: order global targets by fullness after taking the shard The search stops at the first target that would end up fuller than the source, which only holds when targets are ordered by that same value. With unequal capacities an empty one-slot server sorted first and ended the search before a larger server that could take the shard. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ecbalancer: free the source disk slot on a global move The global phase cleared the moved shard's bits but left the source disk's free slots and shard count unchanged, unlike every other phase. Use releaseShard so a later pick of that server sees the freed disk. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ec_balance: test that a capped detection keeps a global move Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ecbalancer: test the same-machine move skips a sibling holding the volume Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nguyễn Lộc Phúc
|
195f8156de |
fix(iceberg): keep a losing table create from deleting the winner's metadata (#12474)
* iceberg: keep a losing table create from deleting the winner's metadata * iceberg: keep a staged file whose published record was renamed away A create answers a conflict when its record was already visible and another request changed it before the create could back it out. The cleanup then looked only at the original name, so a record renamed in between read as absent and the file it still points at was deleted. S3 Tables now reports that case with a conflict of its own, and both create paths keep the staged file on it instead of probing the name. A conflict over a still-hidden record, lost to another create, still removes the file. * iceberg: discard a losing create's metadata only while it is still its own The cleanup deleted the staged file by name. The path can be reused before it runs: the table is dropped and purged, and another create stages the same v1.metadata.json and registers it, which the losing request then deleted. Staged files now carry a per-request token in an internal extended attribute, and the cleanup deletes with that token as the filer's ExpectedExtended precondition, checked under the path lock the delete runs under. A file that took the path since is left in place. * iceberg: classify a typed S3 Tables error by its code, not its message isS3TablesAlreadyExists matched "already exists" in the message before looking at the code, so an InternalError whose text mentioned an existing entry read as the name being taken: CreateTable answered 409 instead of 500, and registrationRefused discarded a staged file the catalog may still point at. A typed error now decides by its code; only an untyped one is matched by message. * iceberg: pin stage-create on in the stage-create conflict test Stage-create is on by default, but a shell that sets ICEBERG_ENABLE_STAGE_CREATE=false turned the expected 409 into a 501. Setting it in the test keeps the result independent of the environment. * iceberg: test two creates overlapping at the canonical metadata name The concurrent-creators test only lines the creators up before they start, so whether two of them ever collide at v1.metadata.json is up to the scheduler. The new test fixes the order: the second create runs while the first has staged v1.metadata.json and not yet registered it. It checks that every staging attempt is exclusive, that the second falls back to v1-<uuid>, that both files hold their own metadata while both are staged, and that the loser removes only its own file. * s3tables: a create whose unmark write never went out stayed hidden confirmChild counted any failure to take a create's mark off as possibly published, the lookup that precedes the write included. That lookup failing leaves the mark on, so the record stayed hidden; if another create then adopted it, the back-out reported a published record that changed, and the Iceberg catalog kept a staged file no record names. The mark-removal calls now go through a wrapper that notes whether an UpdateEntry went out without the filer refusing it. Only then may the record have been seen. A refused compare, a missing entry, or a failure before any write leaves the create's record hidden. * s3tables: pass mayBeVisible into backOutChild Fixes the build after rebasing onto the backOutChild split. |
||
|
|
985706cc32 |
volume: check a needle's CRC the first time a range of it is read (#12467)
* volume: check a needle's CRC the first time a range of it is read A ranged read never checked the CRC, so a replica with a damaged needle served the damaged bytes with 206 and the reader never tried another copy. The first ranged read of a needle now streams the whole needle through the existing CRC check, with the usual pooled page buffers, before writing any byte. A mismatch fails the read, so the reader falls back to another replica. Needles that pass go into a bounded LRU, keyed by volume, compaction revision, id, append time and checksum, so later ranges of the same needle stream as before. Fixes #5959 for ranged reads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: key verified ranges by volume load, share concurrent checks From review: the cache key read v.SuperBlock.CompactionRevision without the data file lock, racing a compaction commit that reloads the super block, and a remounted copy of a volume inherited the old copy's verified entries. Key entries by a load generation instead, an atomic counter that changes on every load, including the reload in CommitCompact, and cache a result only if no reload happened during the check. Concurrent first ranges of a needle now share one check through singleflight instead of each reading the whole needle. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: keep hot needles verified, leave the io error streak to the range read From review: Contains does not refresh an entry's place in the LRU, so even a constantly read needle was evicted after 64k other needles were verified; look it up with Get instead. The verification pass also cleared the io error streak before the range read ran, so a range read failing with EIO after a passing check never built up a streak; the pass into io.Discard now leaves the streak to the range read, like the meta read before a streamed GET. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume (Rust): check a needle's CRC the first time a range of it is read Mirror of the Go change: a range served from the data file now checks the whole needle's CRC before any byte of it is sent, so a damaged replica answers 500 and the reader tries another copy. Needles that pass go into a bounded LRU keyed by the volume's load generation, needle id, append time and checksum; concurrent first ranges of a needle share one check, and a check that passes leaves the io error streak to the range read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume (Rust): a range over the whole needle checks its CRC Go reads a range that covers the whole needle through the checked full-read path. The Rust volume server read it from the data file unchecked, so "bytes=0-" served a damaged needle with 206. Check the slice's CRC before answering, as the full read does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: check the covered blocks of every range against CRCs recorded at the first check The first ranged read of a needle now records the CRC of each block (64 KiB to 1 MiB, at most 64 blocks below 64 MiB) while it checks the whole needle. Later ranges read the block-aligned span around them and write a block's bytes only once it matches, so damage that appears after the first check is not served. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume (Rust): check the covered blocks of every range against CRCs recorded at the first check verify_checksum now records a CRC per block (same sizes as Go's rangeCheckBlockSize) and returns them from the cache; read_checked_range reads the block-aligned span of a range and answers 500 if a block no longer matches, so damage after the first check is not served. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
Chris LuandClaude Opus 5.5
|
d0b5529a2d |
S3 Vectors: weed-vector, weed-worker and the CLI write tables to the filers and volume servers (#12470)
* sw-client: the filer and volume client library The client the Enterprise repository's VFS and vector engine use, with the same API: filer gRPC for entries, listings, volume locations, assignments and remote caching; volume HTTP, plain or TLS, for needle bytes; chunk resolution for sparse, overlapping and manifest chunks; a location cache; and the write side. Enterprise-only protocol fields (license, conditional mutations, hard links, KV listing) are left out. Compiled against weed/pb/filer.proto like the other worker crates. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-lance-worker: reads straight from the filers and volume servers direct_read::DirectReads wraps the object store Lance opens so that entries, listings and bytes come from the filers (gRPC) and the volume servers (HTTP, signed with the read JWT), whatever the entry holds: inline content, chunk manifests, sparse or overlapping chunks, the filer's volume encryption, the gateway's SSE-S3 (sse.rs, with the KEK from security.toml or the filer) and objects tiered to remote storage. Writes still go to the inner store. The code and its tests are those of the Enterprise repository's engine. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: read tables from the filers and volume servers The engine asks the masters for the filers (and follows the list), takes jwt.signing.read.key, grpc.client, https.client and the SSE-S3 keys from security.toml, and opens every table with DirectReads innermost and the read cache outside it. --masters is required: the gateway finds the engine through them and the engine finds the filers. Writes still go to the gateway. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * sse: chunks out of a manifest keep their own SSE-S3 keys A chunk expanded from a manifest carries its id as fid, so keying the per-chunk key cache by file_id gave every such chunk the first one's key and IV. Keyed by the chunk's id however it is carried. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct reads: a silent filer is passed over like an unreachable one Each filer attempt is bounded, so a filer that accepts the connection and never answers does not hold the read or the startup lookup. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * sw-client: carry the assignment's fsync to the volume upload The filer's storage rule may want a needle on disk before the volume answers; the assignment says so and the upload now asks for it, as the Go uploader does. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: argument tests pass the required masters; connect timeout for filers Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct reads: a failed read is tried once more against fresh locations A fetch that fails forgets the volume locations it was made against; the request now looks them up again and retries instead of failing on a volume that moved. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * sw-client: an exclusive create that finds its entry, and needle deletion create_entry marks ENTRY_ALREADY_EXISTS with EntryExists, so a caller can tell a lost conditional create from a failure; the transport deletes a needle whose entry was never created. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * sse: encode chunks as the gateway stores them EntryEncoder writes SSE-S3 under a fresh object key wrapped by the KEK, serialized with the IV, key id and commitment the gateway's reader expects, each chunk under the IV at its offset; and the filer's volume encryption under a per-chunk key. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct storage: writes, deletes and table removal without the gateway A put uploads its chunks to the volume servers the filer assigns, at the filer's chunk size, encoded as the bucket and cluster ask, and creates the entry the gateway would: md5, mime, size, the ETag attribute, the SSE-S3 attributes. A conditional create is the filer's exclusive create, so Lance's manifest commit conflicts the same way; the needles of a create that failed are discarded. A multipart upload turns parts into chunks and derives its ETag from them. Deletes remove the entry with its data, and remove_table a table directory. Copy and rename, which Lance does not use on S3, are not offered. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector.proto: Table says whether its bucket encrypts with SSE-S3 The engine writes the table's files itself, so it needs what the S3 write path knew: whether the bucket declares SSE-S3. storage_options go: the engine reaches no S3 store. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors: send the engine the bucket's SSE-S3 setting The rule the S3 write path applies to a Lance bucket's files, factored out of DeclaredBucketEncryption, decides it from the vector bucket's record. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: every table operation goes to the filers and volume servers Opens, index creation and drops all go through the direct store, so the engine has no S3 client of its own left to protect: --storage-option, the endpoint pin and the request-option allowlist go, and lance's own S3 store is built unsigned and never dialed. The bucket's SSE-S3 setting comes with the table; -encryptVolumeData encrypts chunks for the volume servers as the gateway's flag does. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors tests: the engine takes no credentials Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors e2e: the engine writes what the gateway would have Every leg checks that the entries the engine wrote carry the gateway's ETag attribute, the hardened leg that their chunks are encrypted for the volume servers, and a new test that an SSE-S3 bucket's files are written under keys the gateway unwraps when it serves them over S3. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * cluster plumbing moves to the worker crate for the engine, worker and CLI security.toml, the gRPC TLS connector and acceptor, the master dialer, the filer discovery and the storage connection built from them now live in weed-lance-worker (cluster, security, grpc_tls), where the worker and the CLI can use them too. The server crate re-exports security and tls under their old paths. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker, weed-vector-cli: tables through the filers and volume servers dataset::Storage replaces the S3 credentials every job and the CLI carried: a table is opened and written through the cluster's storage, encrypted as its bucket declares (read from the bucket's record on the filer), and dropped there. Both take --masters and security.toml like the engine; --access-key and --secret-key go, and with them the catalog's x-api-key credential, which the cluster token replaces. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * lance catalog: the cluster token alone authenticates a cluster process The engine, the maintenance worker and the CLI hold the cluster's filer signing key and reach a table's files themselves. A request presenting the cluster token and no other credential is theirs: it is authenticated by the token, runs through the trusted manager, and is vended no S3 credentials. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * worker chart, README and test design follow the worker's new arguments Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors tests: the worker and the CLI reach the cluster through the masters The cluster gets a filer signing key, which the catalog authenticates the worker and the CLI by; both take --masters in place of S3 keys. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct storage: encrypted chunks each start their own CTR stream The gateway encrypts a multipart part from the IV derived for its offset, from position zero; the encoder encrypted from the exact byte offset of the object's stream instead, which agreed only at block boundaries. A part of 3,000 bytes read back wrong; now tested. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct storage: keep what a lost reply saved, discard what a failed put left A create the filer saved but did not answer, or an exclusive create retried onto this very upload, keeps its chunks and counts as written; the needles of a put that failed midway are discarded. The report prints reads as reads. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-worker: the security.toml it is given also holds its signing key Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * worker chart: the Lance worker mounts grpc.client's certificate Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct reads: the filer deadline bounds a listing page, not the listing Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: security.toml booleans read as viper reads them Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: a filer the master lists badly does not hide the others Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * weed-vector: register with the masters only once ready to serve Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * sse: every chunk after the first is written under a key of its own Chunks of one object shared a key with IVs derived from their offsets, so two chunks meeting inside a cipher block shared keystream. The first chunk stays under the object's key and IV, which is how the gateway reads an object of one chunk; every other chunk gets a fresh key and IV, as the gateway's multipart parts have and as it reads an object of several. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * worker chart: the Lance worker is given the filer group Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3vectors e2e: the gateway serves engine-written SSE-S3 files of one chunk and of several Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * sw-client: the HTTP transport asks the volume to fsync when the assignment does write_synced fell back to the trait default, which refuses fsync, so every upload under a fsync storage rule failed before it was sent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: https.client.ca alone vouches for volume servers With a CA configured, reqwest still trusted the system roots next to it; weed processes trust only the configured pool. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: an empty volume range is read without a request A zero length underflowed the Range end, and a range past u64::MAX overflowed it; the first now returns no bytes, the second an error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: a whole-needle answer to a ranged read is cut to the range A server or proxy that ignores Range answers 200 with the whole needle; the read returned its first bytes in place of the ones asked for. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: a read size past u64::MAX reads to the end of the file offset + size overflowed before the clamp to the file size. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: an SSE-encrypted manifest is refused without a decoder Only filer-encrypted manifests were; an SSE one went to the protobuf decoder as ciphertext. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: an empty write leaves the staged file size alone A zero-byte write past the end grew the size to its offset, as a zero-byte pwrite does not. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: the chunk cache keeps no empty chunks They add no bytes, so nothing evicted them and the map grew with every distinct empty needle, even at capacity zero. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * sw-client: flushing with a zero chunk size is an error, not a panic Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: an empty master list fails at startup "" split into one empty address that passed the check, and the engine then waited two minutes for a filer before a generic error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance cluster: an empty master list fails at startup Cluster::new took "" as one master, so weed-vector, weed-worker and the CLI waited two minutes for a filer before a generic error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * direct storage: a failed create keeps its chunks when the filer cannot be asked The lookup after a failed create is what tells a lost reply from a refusal; when it fails too, the entry may be saved, so its chunks stay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector: a gateway-described table takes the shared unsigned store options Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * vector.proto: give sse_s3 a new field number and reserve storage_options Field 4 held the storage_options map on master; a gateway from that build would send a length-delimited value the engine cannot decode as a bool. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * weed-vector-cli: open s3:// URIs on the cluster's storage A table named by its s3:// location skipped the store wrapper and opened Lance's own S3 store, which has no credentials since the fallback flags were dropped. It now opens like a namespace table and needs --masters; local paths keep the plain builder. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * vector: an index id is plain path components, and drop trusts describe() `resolved()` now refuses an id whose part is empty, `.`, `..` or holds a `/`, since the colocated path would land elsewhere on the filer; and `drop_index` no longer repeats the colocation check for the location describe() already enforced for the configured schemes, which made a drop on a memory:// table fail in the scheme test configuration. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: a file delete takes only its entry; a table path is plain names The store's delete of one file no longer asks the filer for a recursive delete, so a path that names a directory does not take its content with it, and `remove_table` refuses a location whose bucket or path component is empty, `.`, `..` or holds a separator, which the filer would resolve outside the table. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: chunks are assigned to the bucket's collection The filer assigns a chunk's volume from the collection the caller names; the gateway names the bucket, prefixed by the filer group when the cluster has one, so a plain assign landed the engine's chunks in the default collection. The store now asks for `<bucket>` or `<group>_<bucket>`, with the group taken from the cluster connection. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: a put's content type becomes the entry's mime Lance sets no content type today, so every entry was written as application/octet-stream; a put that does carry one now creates the entry with it, as the gateway does. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: an upload whose create went unanswered is done with A create the filer took and never answered (a timeout, or a transport that reports the filer unavailable) may still be applied, so its chunks are kept, as they already were for a create whose reply was lost. The upload then records that it is done: a second completion is refused instead of creating an empty entry, and an abort discards nothing. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: an upload dropped unfinished discards its chunks A request cancelled mid-write drops its multipart upload with neither complete nor abort called, leaving its needles on the volume servers. Dropping an upload that still holds chunks now discards them on the runtime; a completed upload, or one whose completion is in doubt, holds none. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3: a cluster process registers and deregisters in the vector namespace The Lance catalog ran a cluster-token caller through the trusted manager for every operation but the vector-namespace write check of register and deregister, which still used the caller-checked manager and answered 403 for an index of a LANCE bucket. One `manager(r)` now picks the manager for all three. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: the worker's flags take two dashes The worker parses its flags with clap and, unlike the engine, does not accept Go-style single-dash long flags, so the harness's hardened worker leg and the README's `-encryptVolumeData` did not start it. The README's commands also gain the `--masters` the worker now requires. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: the CLI takes --encryptVolumeData An index build or optimize from the CLI wrote its chunks unencrypted for the volume servers unless the filer itself asked for the cipher; the CLI now takes the flag the engine and the worker have. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: a bucket's encryption is read when a table is opened The setting was cached for a minute, so a bucket recreated with SSE-S3 within that minute had its first tables written plain, and a record that could not be decoded was read as "no encryption". A table open now reads the bucket's record from the filer, once per open, and an unreadable record is an error. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: a removed table's files leave the object cache `remove_table` asked the cache to forget everything under the table, but the cache was built without invalidation closures, so the call was refused and its error dropped: a data file read before the drop was still served afterwards. The cache now supports the invalidation, and a refusal is an error. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: an abort keeps the chunks it could not discard `discard_all` stopped at the first chunk a volume server would not give up and dropped the rest from the upload's state, so a retried abort found nothing to do and the needles stayed. Every chunk is now attempted, the ones left are kept for the next abort or the drop, and the upload can no longer be completed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: a chunk write that fails gives its needle back When the volume server took the chunk but its answer was lost, the needle stayed under a file id nothing would ever reference. The assignment is known before the write, so a failed write now discards it, best effort, before the error is returned. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * vector: the live worker tests require the cluster's masters With WEED_LANCE_NAMESPACE set and WEED_MASTERS not, the tests seeded their tables through lance's own S3 store, which reaches nothing, and failed before a job ran. The missing variable is now named outright. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct storage: only the filer's missing-entry error means not found Matching any "not found" text read failures such as an unresolved remote storage as a missing entry, so a delete or a listing reported success over data that was still there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * direct storage test: compare the chunks' SSE-S3 IVs Each chunk's metadata wraps its key under a fresh nonce, so the metadata differed even if the key and IV were reused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * direct storage: a filer passed over unanswered keeps the outcome open When one filer timed out on a create and the next refused it, the refusal came back without the no-answer mark, so the put discarded chunks the first filer may still reference. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lance test: start every concurrent writer before awaiting them The lazy map spawned each writer only after the previous one finished, so the commits never overlapped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * direct storage: count failed uploads and deletes The storage report counted failed reads and lookups but not a volume upload, an object delete or a table removal that failed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * S3 Vectors tests: run weed-vector-cli as the cluster's processes run The CLI ran without the fixture's environment and outside its directory, so under VECTOR_READ_JWT it could not read the volumes and under VECTOR_TLS it found no security.toml. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * direct storage: a filer that fell away leaves the outcome open too An unavailable error (connection reset, deadline, transport failure) can arrive after the filer took the request, so a refusal from the next filer no longer settles a create the first one may still apply: the chunks are kept, as after a timeout. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct storage: a chunk write abandoned in flight gives its needle back The needle is assigned before the write, so a request cancelled while the write is in flight left it on the volume server with nothing to name it. `Source::upload` is now `assign` and `write`, and the upload holds the assignment across the write in a guard: a write that fails discards it and one abandoned discards it on the runtime, while a write that returns hands it to the entry. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * lance tests: sign for the namespace as the worker does The live tests reached the namespace anonymously once the S3 keys went, and an anonymous caller lists no tables, so detection proposed nothing. They now take the cluster key from where the worker reads it (WEED_JWT_FILER_SIGNING_KEY or security.toml) and log at info, so a table a handler skips says why in the harness output. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * s3tables lifecycle test: run the Lance worker against the cluster The worker leg of TestLanceTableLifecycle still handed the Rust tests S3 credentials and no masters. The mini cluster now has a cluster key, and the tests get WEED_MASTERS and that key; passes with WEED_LANCE_MAINTENANCE=worker (3 fragments became 1, 4 versions removed). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK * direct storage: name no collection outside a filer group, as the gateway Without a filer group the gateway leaves the collection to the filer, so a filer.conf rule for the bucket's path picks it; naming the bucket overrode that rule, and the chunks landed where a bucket delete does not look. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * direct storage: a read reports the entry's content type A put now stores its ContentType as the entry's mime, but get_opts returned no attributes, so the type did not come back on a read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Chris LuandClaude Opus 5.5
|
5cd38527ba |
filer.remote.sync: skip pre-mount mount-table versions, decide the unmount before admitting later events (#12472)
* filer.remote.sync: judge the mount table before admitting later events The unmount was decided inside a processor job, so events behind the mapping write, such as remote.unmount's purge deletes, were admitted and could delete remote objects before the exit landed. The stream callback now reads each mount-table version before the next event, admits nothing after the one that ends the mount, saves no offset, and drains the admitted jobs before exiting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: ignore mount-table versions from before the mount Without a saved offset the follow starts at the mounted directory's creation second, so it replays mount-table versions written before remote.mount added the directory. Each read as an unmount and the process exited on every start. A version now ends the write-back only when the version it replaced mapped the directory to its location. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: judge the mount table by the mount's own key -dir may lie below a mount, and DetectMountInfo resolves it to the enclosing mount. Looking the table up by -dir found no mapping in any version, so a replaced table never counted as mapped and the parent's unmount was missed, admitting its purge deletes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: a table renamed away ends the mount A rename of /etc/remote/mount.mapping to another name inside /etc/remote was ignored, although the filer no longer has a mount table. The table is now matched by its exact path on both sides of the event, so renaming it away counts as removing it, and a file of that name elsewhere under /etc/remote no longer counts as the table. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: a mount on another remote is not the same mount The mount-table check compared only bucket and path, so remounting the directory on another remote storage at the same bucket and path kept writing through the old client, and a replayed unmount from the other remote stopped the write-back. The remote storage name now counts too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: check the drain at the exit itself Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: resolve the mount covering -dir in each table version Judging the table by the startup mount key alone missed a mount added at -dir itself below that key: the key stayed mapped, so the nested mount and its later removal were both admitted, and the removal's purge deletes ran against the startup remote. Each version is now resolved the way DetectMountInfo resolves the live table, and a different covering mount ends the follow as a move. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: an earlier mount's unmount still ends the follow Skipping every version whose replaced table did not map the directory to this mount also skipped the unmount of an earlier mount of the directory, which a -timeAgo replay reaches, and admitted its purge deletes against this mount's remote. Only a version in which no mount covers the directory, replacing one in which none did, is now skipped as history from before the mount. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: map paths below -dir under its place in the mount With -dir below a mount, every remote key was taken relative to -dir and appended to the mount's location, so /mount/sub/x.txt uploaded to x.txt instead of sub/x.txt and could overwrite another object. The processor now gets the remote location of -dir itself; the mount's own location still judges mount-table changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * filer.remote.sync: resume below a mount no earlier than the mount's offset remote.unmount saves the offset of the directory it unmounts, after its purge, and leaves the offsets of directories below it alone. A write-back of -dir=/mount/sub resumed from its own offset after a remount, replayed the unmount and exited, on every start. Without -timeAgo it now starts no earlier than the mount's offset. Whatever that skips was purged, or was synced by a write-back of the whole mount. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |