mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-08 15:41:15 +02:00
9e06e1d0f9277cec126e83f75b3fe335dbe13f92
100
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris Lu
|
9e06e1d0f9 |
Report a delete the filer rejected instead of answering success (#11003)
* s3tables: report a delete the filer rejected deleteDirectory discarded DeleteEntryResponse and checked only the transport error, so DeleteTable, DeleteNamespace, DeleteView and DeleteTableBucket answered 200 for a delete the filer refused. Call filer_pb.DoRemove, which reads resp.Error and still treats a missing entry as success. Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU * admin: report a delete the filer rejected The bucket delete, the file browser handlers and the topic retention purger all discarded DeleteEntryResponse, so a delete the filer refused came back as success. Call filer_pb.DoRemove, which reads resp.Error. Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU * credential: report a delete the filer rejected DeleteUser, DeletePolicy and the full-sync cleanup loops discarded DeleteEntryResponse, so a rejected delete answered success and left the credential file in place. The service account path in the same store already read resp.Error; the rest now do too, via filer_pb.DoRemove where not-found is already tolerated. Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU * shell: report a delete the filer rejected remote.configure -delete, remote.cache and the remote metadata sync discarded DeleteEntryResponse, so a rejected delete printed as removed. Call filer_pb.DoRemove, which reads resp.Error. Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU * mq: report a delete the filer rejected The consumer offset group purge and the coordinator assignment delete discarded DeleteEntryResponse. Call filer_pb.DoRemove, which reads resp.Error. Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU * iam: count only the revocation entries the filer actually deleted The expiry sweep discarded DeleteEntryResponse, so a rejected delete was counted as purged and the entry stayed. Call filer_pb.DoRemove, which reads resp.Error, matching the role and provider stores beside it. Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU * mount: fail rmdir when the unary fallback delete was rejected The streaming branch turns DeleteEntryResponse.Error into an error, the unary fallback dropped it, so rmdir of a non-empty directory answered OK off the stream and ENOTEMPTY on it. Surface it in both. Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU * s3tables: fail DeleteTableBucket when the directory delete is refused The handler only failed when both the leaf entry and the directory delete failed, so a refused bucket directory delete still answered 200 with the bucket in place. The directory is the bucket, so it decides; the leaf entry stays best-effort. Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU |
||
|
Chris Lu
|
742b2f5896 |
s3: share one retry allowance across a batch delete (#11001)
Every key in a multi-object delete drives its own retryFilerOp, so a filer that is briefly unhealthy multiplied one op's ~3.1s of backoff by a key count the client picks. The batch now carries a single allowance in its context, sized to one op's worst case; once it is spent the remaining keys fail fast with a per-key error instead of holding the request goroutine. A single-object delete carries no allowance and keeps its full retries. Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU |
||
|
Chris Lu
|
902a12fd6f |
wdclient: bound the wait for a master leader by the caller's context (#11002)
* wdclient: bound the wait for a master leader by the caller's context WithClient waited on GetMaster with context.Background(), so a caller that arrived while no master leader was known parked in a 200ms poll loop until one appeared, whatever deadline it had already set on the RPC. Each retry above it then left another goroutine in the same wait. Take the context in WithClient and WithClientCustomGetMaster and hand it to GetMaster, and stop the retry loop once it is done. The dial keeps context.Background(): fn brings its own RPC context, so a cancellation seen here cannot be attributed to the shared connection. Call sites pass whatever they hold: the request context in the filer's CollectionList, DeleteCollection and Statistics handlers and in the credential store's propagation, the operation context in the shell's s3.bucket.delete and the kafka gateway's broker and filer discovery, and context.Background() where there is none - the shell commands, the admin dashboard wrapper, and the exclusive locker's initial lease. The locker's release keeps its own uncancelled context so a slow unlock cannot turn into a ghost lock. Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU * wdclient: test that WithClient gives up with the caller's context Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU * wdclient: cut the master retry backoff short when the caller gives up util.Retry sleeps unconditionally between attempts, so a transient error arriving just before the caller's deadline still cost it a full backoff step. Use the context-aware util.RetryWithBackoff, the same helper the volume lookup in this file already uses. Two call sites went with it: the shell's lock-holder lookup builds its three second bound before WithClient so it also covers finding the leader, as its comment already promised, and the filer's post-delete collection cleanup goes back to an uncancelled context - the entry is already gone, so a caller that hung up must not leave the collection behind. Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU * wdclient: test that a cancel during backoff ends the retry Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU |
||
|
Chris Lu
|
d850f36513 |
s3: distinguish a failed bucket lookup from a missing bucket on HEAD (#11000)
HeadBucket treated any lookup error as ErrNoSuchBucket, so a transient filer failure answered 404 instead of 500 and clients stopped retrying. Split the two cases the way the bucket policy handlers already do. Claude-Session: https://claude.ai/code/session_01BjDWtZsCoZY6x4pdDmGWxU |
||
|
Chris Lu
|
eed3c27d15 |
volume: cut the memory a server holding millions of volumes still uses (#10999)
* volume: stop the .vif guard depending on which entry the scan handed over A volume has both an .idx and a .vif, and loadExistingVolume skipped a .vif next to an .ecx as EC shard metadata. That was only ever correct because os.ReadDir sorted .idx ahead of .vif: an interrupted encode, where the .idx is still there, has to reach validateEcVolume to be reclaimed. Ask for the .idx instead of trusting the order. Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy * volume: walk volume directories in batches instead of listing them whole os.ReadDir builds, and sorts, a slice of every entry before the caller sees the first one. A disk holding millions of volumes has a .dat, .idx and .vif per volume, so each startup scan costs hundreds of MB of peak heap that the runtime is slow to hand back -- and there are several of them before the first volume loads. Walk in batches instead, and keep only the entries each scan acts on: loadAllEcShards now sorts and stats the shard and index files alone rather than every file on the disk. Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy * volume: skip the sibling-.dat scan when no EC volume is loaded pruneIncompleteEcWithSiblingDat only ever prunes EC volumes that are loaded, but it first walks every disk and keys a map by every .dat on the server. On a store with no EC volumes at all that is millions of map entries built to answer no question. Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy * volume: stop keeping a departure message for every volume The report state held a VolumeShortInformationMessage per volume copy so a departure could be named, but almost no volume ever departs. Hold a handle to the identity instead -- volumes share very few distinct ones -- and build the message on the way out. Measured over a populated report state: 195 -> 83 bytes per volume. Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy * rust volume: stop keeping a whole volume message per volume held The send loop kept a VolumeInformationMessage for every volume just to notice mounts and unmounts, and rebuilt the map from scratch on every beat. Keep the identity a delta names, which is what the Go report state keeps for the same reason. Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy * rust volume: keep only the EC files the shard scan acts on load_all_ec_shards named every file on the disk twice -- once in the dedup set and once in the sorted vector -- before deciding it only wanted .ec?? and .ecx. Filter while reading instead. Mirrors the same change in loadAllEcShards. Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy * volume: share the strings every .vif repeats A tiered volume's .vif names its replication and its backend, and every decode allocates a fresh copy, so a server holding millions of them holds millions of copies of the same handful of names. Route them through the interning table the volume info decode already uses. The remote key names one volume and is left alone. Claude-Session: https://claude.ai/code/session_01NWpFUwAJcR2KUENrhLc9Sy |
||
|
Chris Lu
|
fdd8bd9478 |
s3: reject a request that names two operations (#10987)
The router matches bucket subresource routes in registration order while the IAM action resolver matches its own list in a different order, so a request carrying two operation subresources is authorized as one operation and served as another. `PUT /bucket?policy&tagging` resolves to s3:PutBucketTagging and runs PutBucketPolicy, letting an identity delegated bucket tagging install an arbitrary bucket policy. The same mismatch reaches PutBucketCors, PutBucketLifecycle, PutBucketVersioning, PutObjectLockConfiguration, PutBucketRequestPayment and the policy and cors deletes. Reject the ambiguity where the other pre-routing checks live, so neither list has to stay in step with the other. Keys that modify an operation rather than select one -- versionId, partNumber, prefix -- still combine freely. |
||
|
Chris Lu
|
99cf7a66df |
shell: remove the directories emptied by volume.fsck's filer entry purge (#10992)
* shell: remove the directories emptied by volume.fsck's filer entry purge volume.fsck -findMissingChunksInFiler -reallyDeleteFilerEntries deleted the orphan entries but left their parent directories behind, so a namespace accumulated empty directories that had to be cleaned up by hand. Remember the parent of every purged entry and, once the purge is done, walk up from each one deleting the directories that are now empty. The delete is non-recursive, so the filer itself rejects a directory that still has children; a bucket and a directory that is an S3 object of its own are left alone. Claude-Session: https://claude.ai/code/session_01BncsNo2RVANCDtdbw96Kfc * shell: keep a directory volume.fsck saw change under it The empty-directory sweep read the entry to spot an S3 directory key object and then deleted unconditionally, so a directory promoted to an object in between was removed anyway. Delete with the mtime the lookup returned, leaving the filer to skip a directory that has changed since. Claude-Session: https://claude.ai/code/session_01BncsNo2RVANCDtdbw96Kfc * shell: leave a directory volume.fsck just saw written for the next run The mtime the delete is conditioned on has second resolution, so a write landing in the same second as the one already on the directory is indistinguishable from it and the directory would still be deleted. Skip a directory modified within the last few seconds. A write after the lookup then always carries a later second than the one the delete carries, and the sweep picks the directory up on the next run. Claude-Session: https://claude.ai/code/session_01BncsNo2RVANCDtdbw96Kfc * shell: skip a directory volume.fsck cannot condition a delete on A zero mtime disables the delete's condition at the filer, so a directory whose entry carries none was removed unconditionally and a concurrent promotion to an S3 object went with it. Leave such a directory alone. Claude-Session: https://claude.ai/code/session_01BncsNo2RVANCDtdbw96Kfc * shell: hold volume.fsck's quiet period to the cutoff second itself Mtime keeps whole seconds, so a directory whose mtime lands on the cutoff second was written up to a second after it. Skip that directory too, so the quiet period fails closed. Claude-Session: https://claude.ai/code/session_01BncsNo2RVANCDtdbw96Kfc |
||
|
Chris Lu
|
2a97e08caa |
s3: cover the directory marker key with object lock (#10988)
* s3: enforce object lock when deleting a directory marker The key "dir/" is deleted the unversioned way, ahead of the branches that enforce Object Lock, so a principal with plain delete permission could remove a key the gateway was reporting as COMPLIANCE-retained -- retention set through PutObjectRetention is stored on the directory entry and served back by GetObjectRetention, only the delete ignored it. The same path also takes any key ending in "/" regardless of size, while a PUT only makes a marker of one up to 1KiB. A larger one is a genuine versioned object, and deleting it here dropped its whole history after the versioned delete of the same key had been refused. Enforce in the marker delete itself, so the single, versioned and multi-object delete paths are all covered. * s3: apply object lock headers on a directory marker PUT The trailing-slash branch runs before the versioning and Object Lock handling, so it accepted x-amz-object-lock-* headers and stored none of them: a bucket owner could believe a key was retained while nothing recorded it, and an invalid mode or a past retention date that a regular key rejects came back 200 here. Validate the headers the way the regular path does, store what they ask for beside the owner the same callback already sets, and refuse to replace a key that is already retained. * s3: check every version a marker delete would remove The marker delete clears any history under the key in one recursive removal, while the lock check ahead of it resolves the latest version only. A version retained under an unretained one was taken with the rest, so enforce against each version the removal covers. * test: pin the marker lock refusals to AccessDenied A bare require.Error passes on any failure, including one that has nothing to do with the lock. Assert the code, the key the batch delete reports, and that the marker survives each refusal. * s3: check the history entries a version list leaves out The version list skips an entry without a version id, while the removal takes it with the rest, so an entry an older build left unnamed escaped the check. Walk the history directly instead, and refuse when an unnamed entry is still under a retention or a legal hold of its own. * s3: let a governance bypass reach an unnamed history entry The unnamed branch refused every active retention, so a caller allowed to bypass governance could not clear one, which the named path lets through. Refuse a legal hold and compliance mode as before, and take the bypass into account for governance. * s3: keep the object lock decision in one place The unnamed history entry had to repeat the retention and legal hold rules inline because the enforcement helper only takes a key to look up. Split the part that judges an entry out of it and call that from both. * s3: guard a marker PUT on the entry it replaces The overwrite check resolved the key's latest version, but mkdir builds a fresh entry for the marker itself, dropping the lock metadata the old one carried. Once the key had a history, an unlocked version answered for a retained marker and a plain PUT replaced it. Judge the entry the write is about to replace instead; a versioned write of the same key still adds a version, which is its own to allow. * s3: guard a marker delete on the entry it removes The check ran against the key rather than the entry, so once the key had a history it answered with a version and the retention recorded on the marker itself went unseen. Judge the entry that is about to be removed, the same way the PUT side now does; the versions under it are still covered by the walk that follows. * s3: take the object write lock for a marker PUT The overwrite check read the entry that the mkdir after it replaces, so two marker PUTs could both pass while one was still unlocked. The marker delete already runs under this lock; hold it across the check and the mkdir so the entry cannot change in between, and so the two paths are serialized against each other. |
||
|
Chris Lu
|
ab8b34720a |
s3tables: delete only the location the dropped table owns (#10986)
DeleteTable authorizes the named table, then recursively purges the data path derived from its stored MetadataLocation. That location is supplied by the caller at create/register time and never bound to the table, so a tenant allowed to drop one table could point it at a table in a sibling namespace and have the delete destroy that table's catalog entry and data files. A legitimately decoupled location -- a rename source, or a leftover the name was reused over -- has had its catalog attributes stripped, so a surviving metadata marker identifies a path that belongs to another entry. Refuse those, alongside the existing ancestor refusal. |
||
|
Chris Lu
|
0b5fff2ccd |
filer, s3: reuse the volume server's guarded remote-storage client builder (#10990)
* volume: build the guarded remote storage client through a shared helper Fold the endpoint validation, credential check and rebinding-safe dialer that FetchAndWriteNeedle applies before dialing a caller-supplied remote storage endpoint into a single BuildGuardedRemoteStorageClient helper, so other callers that dial the same endpoints can reuse it. No behavior change on this path. Claude-Session: https://claude.ai/code/session_01AiH1FU3rmshSbFFTbJpaZN * filer: build the remote-mount stream client through the guarded helper streamFromRemote serves a cold remote-only entry straight from its mounted origin. Build its client through BuildGuardedRemoteStorageClient so the same endpoint checks the volume server applies cover this read path too. Claude-Session: https://claude.ai/code/session_01AiH1FU3rmshSbFFTbJpaZN * s3: build the remote-mount stream client through the guarded helper openRemoteStream serves a remote-mounted object straight from its origin when the local read cannot. Build its client through the same guarded helper so the endpoint checks apply here as well. Claude-Session: https://claude.ai/code/session_01AiH1FU3rmshSbFFTbJpaZN |
||
|
Chris Lu
|
28862c866e |
Authorize an Iceberg table create before it writes (#10991)
* s3tables: share one CreateTable authorization gate CreateTable and RegisterTable each carried their own copy of the name validation, policy load and permission check. Fold them into authorizeCreateTable, and expose it on the Manager for callers that write into a table bucket before the table itself is registered. Claude-Session: https://claude.ai/code/session_01QiJkka1T2NAWDWq4JQ8Vuy * iceberg: authorize a table create before it writes Stage-create returns before the S3Tables registration that authorizes a create, and the plain create writes its metadata file before reaching it, so a caller who may not create the table could still leave a staged template, a marker and a v1.metadata.json in the target bucket - and get vended credentials for a location of their choosing. Run the CreateTable gate as soon as the table is known to be absent. Claude-Session: https://claude.ai/code/session_01QiJkka1T2NAWDWq4JQ8Vuy * iceberg: authorize a create-on-commit the same way A commit against a table that does not exist creates it, writing the metadata file first and only then reaching the registration that checks the caller may create it. Denied callers saw a 500 for what is a 403. Claude-Session: https://claude.ai/code/session_01QiJkka1T2NAWDWq4JQ8Vuy * iceberg: pin that identity actions reach the create gate The manager request is built from the caller's own context, so an identity whose actions carry the permission still passes. Worth a test: a fresh context here would silently deny every such caller. Claude-Session: https://claude.ai/code/session_01QiJkka1T2NAWDWq4JQ8Vuy |
||
|
Chris Lu
|
bc06505b40 |
mount: keep metadata operations working on an unlinked open file (#10989)
* mount: serve metadata ops from the open handle of an unlinked file ftruncate on a descriptor whose file was unlinked failed with ENOENT: maybeReadEntry resolved the inode to a path first, and unlink had already dropped it. GetAttr worked around that with its own handle fallback; SetAttr and the xattr handlers had none. Look the handle up first and let it answer whether or not a name still points at the inode. GetAttr keeps reporting nlink 0 there, now off the empty path. Claude-Session: https://claude.ai/code/session_01U1R8BM4bVT46KwPDEj2Ega * mount: read an open handle's attributes under the handle lock too GetAttr held only the LockedEntry lock, which covers the async uploader's chunk appends but not Write or the metadata flush: those rewrite size, times and the whole chunk slice under the handle lock, so FileSize could walk a slice mid-reassignment. The branch this replaced took both locks; take both here, outer handle lock first, as Read and Lseek do. Claude-Session: https://claude.ai/code/session_01U1R8BM4bVT46KwPDEj2Ega * mount: report nlink 0 from SetAttr for an unlinked open file The kernel caches the attributes a SETATTR reply carries, so an ftruncate on an unlinked file left fstat reporting nlink 1 until the cache expired, even though GetAttr had it right. Both replies go through the same rule. Claude-Session: https://claude.ai/code/session_01U1R8BM4bVT46KwPDEj2Ega |
||
|
Chris Lu
|
e9a464840c |
webdav: describe a listed entry the way clients expect (#10993)
* webdav: name the entry, not its path, in a listing DAV:displayname carried the full path of every entry. A client that takes displayname for the child's name - Windows Explorer does - then looks for /dir/name under /dir and finds nothing, so a folder shows up empty while the root, where the two spellings differ only by a leading slash, still lists. Readdir now builds its entries with toFileInfo like stat does, so a listing and a lookup describe a child the same way, and the wrapper that was trimming the sub-folder back off a name goes away with it. Claude-Session: https://claude.ai/code/session_01XCeuCWpF9xo9CfyHvCQE9c * webdav: derive an ETag when nothing hashed the entry Uploads through this gateway carry no content MD5, so filer.ETag comes back empty and every file in a PROPFIND answered with an empty DAV:getetag, which is not a valid entity-tag. Report it as unimplemented instead, the way the sub-folder wrapper already did, and webdav falls back to modification time and size. The wrapper's copy went with it - it swallowed the stat error a caller was meant to see. Claude-Session: https://claude.ai/code/session_01XCeuCWpF9xo9CfyHvCQE9c |
||
|
Chris Lu
|
d8a189f07f |
s3: keep a missing object a 404 under If-Match and If-Unmodified-Since (#10985)
* s3: keep a missing object a 404 under If-Match and If-Unmodified-Since GET and HEAD resolved the target before evaluating the conditional headers, and a missing target failed If-Match and If-Unmodified-Since outright, so absence surfaced as 412 PreconditionFailed. AWS reports the missing object instead: 404 for HeadObject, NoSuchKey for GetObject, and 412 only when a live object fails the condition. Clients cannot tell absence from a stale precondition without an extra racy HEAD, so OpenDAL disabled its four conditional stat/read capabilities against SeaweedFS. A precondition now only fails against an object that exists; a missing one -- including a latest version that is a delete marker -- returns NoSuchKey. Claude-Session: https://claude.ai/code/session_01X4kEbuwxd9DFsTnSXjfjgv * s3: evaluate a conditional read against the version the request names GET and HEAD resolved the latest version before evaluating the conditional headers, so a request carrying versionId had its If-Match compared against a different version than the one it was asking for: a live version whose ETag the client held failed once a newer version -- or a delete marker -- became the latest. resolveObjectEntry now resolves the named version on a versioned bucket, the way DELETE already does. A named version that resolves to nothing is left to the handler, which alone knows whether the bucket is versioned and so whether it owes NoSuchVersion. Claude-Session: https://claude.ai/code/session_01X4kEbuwxd9DFsTnSXjfjgv |
||
|
Chris Lu
|
2d25c39da4 |
volume: resolve the disk IO slow-latency threshold per disk (#10976)
* volume: resolve the disk IO slow-latency threshold per disk volume.toml keys [volume.disk.io.slow.latency] by disk type, but the threshold was chosen once per server by switching on the raw -disk flag. -disk is comma-separated, one entry per -dir, so a multi-disk server matched no case and silently took the hdd threshold. Carry the table on DiskIOProbeConfig and resolve it in CheckDiskSpace from the location's own DiskType. A type with no entry keeps falling back to the hdd threshold. * volume: run the disk IO probe on multi-directory volume servers The probe was disabled whenever more than one -dir was configured, because a single server-wide slow-latency threshold could not describe disks of different types. The threshold is per disk now, and the rest of the probe already is: diskRegistry is keyed by directory, each DiskLocation runs its own CheckDiskSpace, and Store consults isDiskUnavailable per location. * volume: reject duplicate -dir entries Nothing deduplicated -dir, so the same directory listed twice produced two DiskLocations that each loaded every volume in it, appending to the same .dat under two independent locks. Compare directory identity with os.SameFile rather than the path, so a symlink or bind mount aliasing an earlier entry is rejected as well. * volume: cover the per-disk slow-latency handoff SlowLatencyFor has a test, but nothing asserted that CheckDiskSpace feeds it the location's own disk type. Probe through a seam so the resolved threshold is observable, and check hdd, ssd, nvme, the empty type, and an unlisted tag. |
||
|
Chris Lu
|
8dcdb70594 |
mount: let a rename remove its source at the source's own version (#10973)
* mount: let a rename remove its source at the source's own version
A rename stamps the source and takes the name away at the same log position,
so the removal reaches the meta cache carrying exactly the version the source
already records. The version gate read that as a write already reflected and
dropped it, while the destination half of the same event still applied -- the
source stayed cached beside the destination, and readdir and stat went on
serving a name the filer no longer had:
gate dropped removal of /winfsp-test-TestRenameOverExisting/src
eventTs=1787761708173717200 record=1787761708173717200
floor=1787761708173717200 tombstone=false
A removal asks a different question from a write. An entry still present at
exactly that version has the write reflected but not its removal, so only a
strictly newer record fences one out; a tombstone is the removal already
reflected and goes on fencing as before.
* mount: sweep a section's vanished name recorded at the snapshot
The refresh deletes the names its listing did not return, but asked the gate
whether a write at the snapshot was reflected. A name recorded at exactly that
version has the write reflected and not its removal, so it survived the sweep
and stayed cached until some later event happened to touch it.
Same reading as the rename source a commit earlier: the call site removes, so
it asks about a removal.
|
||
|
Chris Lu
|
f5f1dcbd8c |
s3: keep verifying the request host when externalUrl is set (#10970)
* s3: keep verifying the request host when externalUrl is set externalUrl was the only host candidate once set, so a client that dialed the gateway directly instead of through the proxy always got SignatureDoesNotMatch. Make it lead the candidate walk instead: every candidate still needs a valid signature, and the request-derived hosts are already trusted when the flag is unset, so a mixed proxy plus in-cluster topology can now advertise a public endpoint and verify both planes. * s3: cover virtual-hosted addressing behind externalUrl The old pin also rejected an external client that signed bucket.api.example.com, since only the bare externalUrl host was ever tried. The candidate walk covers it; pin the case down. |
||
|
Chris Lu
|
3431bdcb74 |
s3: fix UploadPartCopy with volume-data encryption (#10971)
* operation: give an encrypted chunk the plaintext ETag With -encryptVolumeData the volume server stores ciphertext, so it cannot echo a Content-MD5 back and the chunk lands with an empty ETag. Every ETag derived from those chunks then comes out empty for a single chunk, or d41d8cd98f00b204e9800998ecf8427e-N for several. The caller already hashes the plaintext to send as Content-MD5, so keep that digest as the chunk ETag instead of dropping it, and compute it for a WantMd5 caller under cipher too. * s3: re-encrypt a part copy from a volume-encrypted source UploadPartCopy raw-copies source chunks when neither side uses SSE, which also caught -encryptVolumeData sources. Those chunks are ciphertext a whole-chunk cipher key decrypts, so copying a byte range out of one and keeping the key leaves a destination that fails authentication on GET, and the copied chunks carry no ETag for the part result to report. Route them through the re-encrypting path already used for SSE: it reads the source as plaintext, hashes the part, and writes the destination under the gateway's own encryption. * s3: fetch only the range a part copy asked for The re-encrypting UploadPartCopy path opened the source at offset 0 and threw the prefix away, so assembling an object part by part read the source once per part. Now that volume-encrypted sources take this path too, that is the common case rather than an SSE corner. The chunk stream already seeks, so hand it the range. * s3: reject an unsatisfiable copy-source-range A part copy has no way to report a short part, so a range reaching past the source cannot be clamped the way a GET clamps one. The fast path silently produced a part shorter than asked for, or an empty one; the re-encrypting path pads with zeros, so a 2 MiB source copied as bytes=1048576-9999999 came back as 1 MiB of data followed by 7.5 MiB of nothing. Answer InvalidRange instead, which is what s3-tests' test_multipart_copy_invalid_range expects. |
||
|
Chris Lu
|
12fd60f92e |
rust volume: stop racing the clock in torn_sdx_is_regenerated (#10966)
The test truncates a good .sdx and asserts the result still looks fresher than its .idx, on the reasoning that truncation bumps the mtime. That holds only at the filesystem's timestamp granularity: where both writes land in the same tick the precondition fails and the run reports a failure that says nothing about the code under test — as it did on CI. Backdate the .idx the way the sibling stale_sdx_is_regenerated already does. |
||
|
Chris Lu
|
da087f77b3 |
mount: stop a replaced rename destination from flushing over the rename (#10965)
* mount: stop a replaced rename destination from flushing over the rename
Rename replaces whatever the destination held, which deletes that entry, but
only the source handle was told. A handle still open on the replaced entry
went on flushing its metadata under that name, and on Windows -- where the
close carrying the flush runs after the application's CloseHandle has already
returned -- the flush landed after the rename and put the destination's old
content back:
dir Rename old_entry:{name:"src"} new_entry:{name:"dst" ... inode:...3416}
doFlush /dst fh 1521468582993181449
/dst saveToStorage 1,6872462993 [0,3)
flushMetadataToFiler /dst inode 11939747521756968515
InsertEntry /dst
The next read of the destination returned the content the rename was supposed
to replace. Unlink already handles this with markHandleDeleted, which raises
the flag under the handle's flush lock so a flush already writing finishes
first and any later one sees it; a rename that replaces an entry deletes it
just the same, so it now does likewise.
Verified on the Windows runner: TestRenameOverExisting 300/300, where the same
loop reproduced the corruption twice without this.
* test/winfsp: say which layer kept a renamed-away name
The failure only reported the stat. Which layer answered narrows the search a
lot: a listing reads no per-path cache, the mount's own forgets within a
second, and a name that survives both is still in the meta cache.
* mount: keep the destination barrier honest when the rename does not happen
Two gaps in the barrier the previous commit put in front of a replaced rename
destination:
The flag was raised before the filer rename, which can still fail. The
destination then stays exactly where it was, with its handle marked deleted
and its dirty metadata silently dropped from then on, so a rename that
returned an error has to put the flag back.
The handle was only found through the path mapping, which Forget drops while
the handle is still open. The source side already falls back to the inode the
entry carries; the destination now does the same, off the entry the sticky-bit
check had already loaded.
* mount: let only the caller that raised a delete mark lift it
Restoring the destination handle after a failed rename cleared isDeleted
outright, so an unlink that marked the same handle in between lost its mark and
a later flush could write the unlinked entry back.
Every raise of the flag already happens under the handle's flush lock, so
counting them there is enough to tell one caller's mark from another's: the
rename lifts only the mark it made itself.
* mount: drain the destination flush before marking it deleted
A flush already queued for the destination belongs to the entry as it stands.
Marking first meant the drain waited on a flush that then skipped its metadata
as deleted and released its handle, so a rename that failed afterwards had
nothing left to restore and the queued update was gone, its chunks orphaned.
Draining first lets that flush finish as itself, before the rename has taken
anything away.
|
||
|
Chris Lu
|
eb3bbfeb1f |
filer: apply the path's storage rule TTL on every write path (#10963)
* filer: cover the storage rule TTL on the object transaction write path An object written through ObjectTransaction used to land with ttlSec 0 even under an fs.configure TTL rule, while the same object written through CreateEntry got the rule's TTL. Guard the shared stamping so the two paths cannot drift apart again. * filer: apply the path's storage rule to an appended entry AppendToEntry resolved the storage option from the path - so its chunks land on a TTL volume under an fs.configure TTL rule - but never stamped the rule's TTL on the entry it creates, leaving an entry that outlives its data. Route it through applyStorageDefaultsToEntry, which now feeds the entry's own TTL into the option so the placement an existing entry's appended chunks get is unchanged. * filer: apply the path's storage rule to a completed TUS upload The PATCH path resolves the storage option from the target, so a TUS upload into an fs.configure TTL prefix writes its chunks to a TTL volume, but completion built the final entry with ttlSec 0 - the entry outlived the data it pointed at. Stamp it through applyStorageDefaultsToEntry, which also subsumes the hand-rolled read-only check and supplies the rule's name-length limit. * filer: apply the destination's storage option TTL to a copied entry The copy handler re-uploads the source's chunks under the destination's storage option, so a copy into an fs.configure TTL prefix already lands its data on a TTL volume. The entry, though, carried the source's ttlSec - 0 for a source outside the prefix, or the source's own TTL where the two rules differ - so it never expired with the data it pointed at. Take the TTL from the same option the chunks were placed with, after the data-only copy has restored the destination's metadata. |
||
|
Chris Lu
|
a02c0024e5 |
master: cap the reported capacity at what the disks hold (#10960)
* master: cap the reported capacity at what the disks hold Statistics reported max volume count times the volume size limit, which is how many volumes the cluster is allowed to place, not how much space it has. A cluster given far more slots than its disks can fill reported a capacity it could never reach -- 65536 slots at 30GB read as 1.9PB on a 460GB disk -- and the number never moved, since writing data changes neither the slot count nor the size limit. The volume servers already report each filesystem's total and free bytes in their heartbeats, so bound the answer by what they say is left. * mount: keep the last known sizes when filer statistics fails A failed Statistics call returned before df's answer was filled in, so a mount whose filer or master was briefly unreachable reported an empty filesystem rather than the sizes it already had. * master: drop the disk ceiling when a volume server does not report A cluster part way through an upgrade has volume servers that predate the disk bytes in the heartbeat. Summing only the ones that answered left the quiet server's free space out of the total, and the server holding the room is exactly the one that could make the cluster read as full. Answer with the disks only when every one of them reported. |
||
|
Chris Lu
|
b77d954f55 |
rust volume: fail closed on sorted-index failures and reconcile tier-up (#10956)
* rust volume: fail closed on sorted-index failures and reconcile tier-up Follow-ups to the .sdx sorted needle map (#10951): - get() folded open/read failures into None, so an EIO, a torn .sdx, or a failed pooled reopen answered reads with NotFound and let do_delete_request acknowledge the delete as Ok(0) without writing a tombstone. It now returns io::Result and every caller propagates; redb's get() had the same shape and is fixed with it. is_file_unchanged cannot propagate, so it reports unknown and logs rather than treating an unreadable index as proof of a change. - A delete whose .idx append landed but whose .sdx mark failed left the map still resolving the old live entry, so deleted content stayed readable until a reload. The map now records the tombstone before touching .sdx and only clears it once the mark lands; lookups consult that first and report the needle deleted, which is what the next reload concludes anyway. - Mode reconciliation ran one way. Entering remote mode made use_sorted_index() true, which returned early, so a volume tiered while the server runs kept its in-memory map and pinned .idx descriptor until restart — the RAM and fd win never applied. It now reconciles in both directions. - Tier-down dropped the remote reference before the fallible refresh, so a failure left volume_info local, the remote backend attached, the .vif still remote, and a retry reporting "already on local disk". The transition is snapshotted and rolled back. - The read-only fallback set no_write_or_delete but left no_write_can_delete, so metrics and mode checks called the volume delete-capable while every delete was refused. * rust volume: count a sorted-map delete against the durable .idx append The deletion counters sat after the in-place .sdx mark, so a mark that failed left them at their pre-delete values while the tombstone was already durable in .idx — and with retries now idempotent, nothing applied them later either. Heartbeats, status responses, and the garbage calculation would report the volume as free of that garbage until a reload. Move them to the append that makes the delete durable, which is also what a reload of .idx would count. Covered by a test that injects a mark failure through a cfg(test) seam: no portable filesystem trick reproduces it, since a read-only .sdx fails the borrow long before the mark. * rust volume: hide a pending tombstone from the sorted-map scans too The overlay that keeps a needle deleted after a failed .sdx mark was only consulted by get(). visit_live_entries still read the stale valid record straight off .sdx, so ascending_visit, iter_entries and save_to_idx all reported the needle live — and compaction takes iter_entries for the complete live set, so it would copy the deleted content forward and save_to_idx would write it back into the rebuilt .idx as live. Snapshot the overlay once per scan and skip its keys, which is the same conclusion the next reload reaches from the .idx tombstone. * rust volume: quarantine a durable write whose index lookup fails The prior-mapping lookup that decides whether to index a fresh append runs after the record is already down and flushed, so a failing lookup leaves exactly the state a failing put leaves: a durable .dat record nothing indexes. The put path marks the volume read only for it; this one returned the error and kept taking writes, and the next append would bury the orphan mid-file where the .dat tail check on reload cannot see it. Give it the same treatment. |
||
|
Chris Lu
|
7658305c76 |
mount: name the disk after the mounted path (#10958)
* mount: name the disk after the mounted path Finder and Explorer labelled every mount with the filer address, so two mounts from one filer were indistinguishable. Use the mounted path's last segment, the way df already shows it, and keep the filer address only for a whole-tree mount. * mount: let a given mount option override the default The options from -o were placed before the ones this mount derives, so a volname or iosize given on the command line lost to the derived value. Append them last, matching the Windows adapter. * mount: document what labels the disk |
||
|
Chris Lu
|
627b5e9d59 |
shell: parse every collection filter the same way (#10955)
* worker: move the collection filter parser into weed/util/wildcard
The parser sits beside the volume-list filtering it was written for, in
weed/plugin/worker, which imports weed/shell — so the shell commands that
parse the same filter three other ways can never call it. Move it down to
weed/util/wildcard, next to the comma-separated wildcard helper it already
replaced, leaving the behavior unchanged.
* shell: parse every collection filter the same way
The shell parsed a collection filter three ways: compileCollectionPattern
compiled one regex for ec.encode, ec.decode, volume.balance and the tier
commands; volume.list and volume.deleteEmpty matched a single wildcard; and
volume.tier.move, volume.fix.replication and volume.configure.replication
called filepath.Match on their own. None of them took a list, so
"ec.encode -collection=a,b" selected nothing, the same way the admin UI did.
They all go through the shared matcher now: a comma-separated list of names,
"*" and "?" wildcards, "_default" for the collection with no name, and regex
entries. The one thing that stays per-command is what an empty value means -
every collection for -collectionPattern, the unnamed collection for the ec
and tier -collection flag - so compileCollectionPattern keeps that mapping.
The matchers are compiled once per command instead of once per volume, and a
regex entry now has to match the whole name unless it anchors itself, so
-collection=bucket no longer picks up mybucket2.
* shell: keep dots in collection names, and commas inside a regex
A dot no longer marks an entry as a regex, so a collection named "my.bucket"
matches itself and not "my-bucket" - the difference decides which volumes
volume.deleteEmpty and volume.tier.move touch. A dot still counts when it is
quantified, so "bucket.*" stays a prefix regex.
The comma split also leaves alone the commas inside a character class or a
repetition count, so "bucket[0-9]{1,3}" stays one entry instead of becoming
two broken fragments.
* shell: let a regex entry match its own spelling
A collection named after regex syntax, say "logs(2024)", was unreachable:
the entry compiled to a pattern that matches "logs2024" instead. Match the
entry verbatim as well, so naming a collection always selects it, whatever
characters it holds.
* shell: reject a collection filter that names no collection
A value of "," parsed to no entries and then matched every collection, so a
typo widened ec.encode or volume.deleteEmpty to the whole cluster. Only a
genuinely empty filter means "all collections"; anything else has to name one.
* shell: keep commas inside a regex group out of the entry split
The split already left alone the commas inside a character class or a
repetition count, but not the ones inside a group, so "bucket(foo,bar)"
was cut into two fragments that no longer compile.
* shell: cover escaping a collection name that is not a regex
A name like "logs(2024" does not parse as a regex on its own; escaping it,
"logs\(2024", reaches it. Pin that so the escape hatch does not regress.
* shell: split entries only on commas inside a closed regex construct
An unmatched "{" or "[" made the splitter swallow every comma after it, so
"foo{bar,videos" became one entry that matches neither collection - the
silent no-op this filter work exists to remove. A construct now has to close
before its commas stop separating entries.
* shell: skip character classes while scanning a regex group
A ")" inside a class is a literal, so "(a[)],b)" ended its group early and
split into two fragments that no longer compile.
* shell: cover escaping a comma inside a collection name
A comma separates entries, so a name holding one is reached by escaping it.
* shell: follow the regexp parser when scanning a character class
A "]" leading a class is a member of it, and a POSIX class such as
"[:alpha:]" carries its own "]", so stopping at the first one cut a valid
filter like "(a[]),],b)" into fragments and rejected it.
|
||
|
Chris Lu
|
368b2035b2 |
s3: deny anonymous access when the identity config loads no identities (#10954)
* s3: deny anonymous requests when the identity config loads no identities Naming a config file is the operator asking for authentication. A file that yields no identity - an unpopulated secret mount, or a mistyped top-level key the proto parser silently drops - left the gateway open to every anonymous caller: ListBuckets returned 200, and anonymous PUT could create buckets and write objects. * s3: name the unknown top-level keys in an identity config The proto parser discards what it does not recognise, so a mistyped "identites" loads as an empty config. Naming the dropped keys at startup turns the resulting lockout into a one-line diagnosis. * s3: isolate the auth-enforcement tests from AWS environment credentials * s3: use a singular "identity" as the unrecognised-key example Codespell rejects the misspelling the example used. * s3: cover the empty identity config alongside the unrecognised key * s3: cover a config file whose body is an empty object |
||
|
Chris Lu
|
b58d52ac16 |
rust volume: search .sdx for read-only volumes instead of holding the index (#10951)
* rust volume: search .sdx for read-only volumes instead of holding the index The Go volume server loads every read-only volume through SortedFileNeedleMap: the index lives on disk as a sorted .sdx, a lookup is a binary search, and since #10950 no descriptor is held between lookups. The Rust server had no counterpart. Read-only volumes built a full in-memory CompactNeedleMap, and cloud-tiered ones — noWriteCanDelete, so not the read-only branch — went through the writable path and pinned an .idx append handle on top of it. At the hundreds of thousands of tiered volumes a real server carries, that is an index in RAM and a descriptor each, for volumes nobody reads. Port the sorted map and the bounded handle pool. A tiered volume now costs zero descriptors and zero index bytes when idle; the pool keeps the hot handles open so a busy volume does not pay an open() per needle. Handles are Arc<File>, so an eviction cannot close one a reader still holds. The generated .sdx is byte-identical to Go's — same sort, same last-write-wins, same dropped tombstones — so a volume moved between a Go and a Rust server reads whichever copy is already on disk. A test pins the bytes against a Go-generated fixture. * rust volume: fail compaction on an unreadable .sdx, and rebuild the map on tier-down Two ways the sorted map could lose data. iter_entries swallowed read errors and returned however many entries it managed to collect. Compaction takes that vector for the complete live set, so a truncated .sdx or a mid-scan I/O fault would commit a volume missing every needle past the failure. Return a Result instead and abort. redb's collect_entries dropped errors the same way on the same path, so it goes with it. Tier-down clears the remote mode and publishes the volume as writable, but the map it booted with is the read-only sorted one. Its put always fails, so the first write would append to the local .dat and then fail to index it, leaving bytes nothing references — and a non-fsync write repeats it. Fold the reopen_idx_for_write swap into refresh_remote_write_mode so the map always matches the mode it just published; a rebuild that fails pins the volume read-only rather than letting it take writes it cannot record. Go reaches neither: its tier-down leaves noWriteCanDelete set, so the volume stays read-only until a reload or an explicit mark-writable, which already goes through reopenIdxForWrite. * rust volume: keep read-only volumes mountable on a read-only index dir, and batch the .sdx scan Building .sdx writes to the index directory, and load_index_sorted_file also created a missing .idx there. A volume whose index sits on a read-only mount took both paths and failed to load, where before it mounted read-only off an in-memory index and served reads. Create the .idx only where deletes are allowed, and fall back to the in-memory map when the sorted one cannot be built, so a directory nobody can write costs memory rather than availability. The end-to-end scan behind iter_entries, ascending_visit and save_to_idx read one entry per syscall. Read 1024 at a time instead, the batch size idx::walk_index_file uses. Positional reads, not a cursor: the handle is shared with any other borrower. Also gate the Go byte-parity fixture on the 5bytes feature it describes, which is otherwise dead code in a 4-byte-offset build. * rust volume: roll back a failed writable mark, and rebuild a torn .sdx set_writable clears the read-only flags before it can know the rest will succeed, but only the map rebuild rolled them back. An .idx writer that fails to attach left the volume advertising writable over a needle map with no writer, so puts landed in memory and were gone after a restart — the exact failure the function exists to prevent. The read-only-mount fallback made it reachable: that path loads an in-memory map with no writer attached. All three steps now run behind one rollback point. A .sdx whose length is not a whole number of entries was accepted as long as it looked fresh, and truncation is what makes it look fresh. The entry count then floored, hiding the last needle from lookups and from compaction, which would commit the shorter set. Treat a torn file like a stale one and rebuild it from .idx. Go writes .sdx in place rather than through a temporary, so a crash mid-generation is a real way to produce one. Appends now start at the last whole .idx entry too, so a torn tail there is overwritten by the next tombstone instead of misaligning every row after it. * rust volume: trim a torn .idx before writing to it, keep delete-only volumes online, count sorted-map deletes Three from review. Flooring the sorted map's append offset only protected its own positional writes. Every writable path appends at EOF instead, so a partial row left by a short write pushed the next row off alignment and the following load parsed the rest of the file as garbage. Drop the partial row before attaching any writable index writer — it is unrecoverable anyway, and every loader already skips it. Go refuses to load such a volume at all; trimming keeps it mountable with the rows before the tear intact. The unwritable-index-dir fallback stopped one step short for volumes that allow deletes, which is every tiered one: the in-memory loader opens .idx read-write there and fails on the same directory that just refused the .sdx, so the volume stayed offline. Give up the deletes instead — without a writer no tombstone could be recorded anyway — and a remount on a writable directory restores them. Sorted-map deletes left the counters untouched, so a tiered volume reported itself garbage-free until it restarted. They now land where a reload would put them: the tombstone is another .idx row, and both it and the row it supersedes count as deletions under the rule the load-time metric applies. Go skips this too, and should not. |
||
|
Chris Lu
|
e482e67971 |
admin: accept a list of collections in the task collection filter (#10953)
The collection filter was parsed twice with two syntaxes: the master-side volume listing compiled the whole string as one regex, while EC encode and EC balance detection split it on commas and matched each entry as a wildcard. A volume had to pass both, so "collection-a,collection-b" matched nothing (no collection is named that), and the ALL_COLLECTIONS sentinel, which the master side skips, dropped every volume at the task side. Parse it once, in one place: a comma-separated list where an entry is a name with optional * and ? wildcards, or a regex when it carries regex syntax. A regex entry now has to match the whole name unless it anchors itself, so listing a collection no longer picks up its longer namesakes. |
||
|
Chris Lu
|
70c3adb983 |
volume: stop read-only volumes from pinning .idx and .sdx (#10950)
A read-only or cloud-tiered volume loads a SortedFileNeedleMap, which held both its .idx and its .sdx open for the life of the process. On a server with ~600K tiered volumes that is 1.2M descriptors before a single read, enough to exhaust the fd limit and take the listeners down. The .dat is not the problem: a tiered volume serves it from the remote backend. Neither index file is needed except while a lookup is in flight, so borrow them from a bounded process-wide pool instead. An idle volume now holds zero descriptors; a busy one keeps its handles hot rather than paying an open() per needle. Reads borrow O_RDONLY, so a volume on a read-only mount answers lookups that previously failed at load. Sync tracks whether a tombstone was appended, which also drops the fsync-per-volume storm at shutdown. |
||
|
Chris Lu
|
b77431c142 |
master: stop hintless small-file assigns from marking volumes full (#10944)
* master: estimate a hintless assign's size from the volume's average file size An assign that carries no dataSize hint charged a flat 1MB per file id against the volume's effective size. A small-file workload overpays by orders of magnitude: bulk-writing 4KB files marks volumes holding a few hundred MB of real data as crowded and then full, so the master grows unnecessary volumes and, once every volume is spuriously full, fails all assigns. Estimate from the volume's own average file size instead, and keep the 1MB fallback only for volumes with no history. * master: decay pending assign sizes for volumes gone quiet The decay that corrects pending assign estimates runs only when a heartbeat reports the volume, and a heartbeat only reports a volume whose content changed. A volume held out of the writable list takes no writes, so once inflated estimates mark every volume full, nothing is ever reported again, nothing decays, and the cluster refuses all writes until a restart. Run the decay from the master's periodic loop for volumes no heartbeat has reported within two pulses, feeding the last reported size back through the same path an unchanged heartbeat would take. * master: trim the comments on the assign size estimate * master: keep the periodic decay out of the replica-dedup window UpdateVolumeSize ignores a report arriving within two seconds of the last one, so replicas of the same volume do not each halve the pending estimate. The periodic decay went through the same path and stamped that window, so a real heartbeat landing right behind it was dropped along with its reported size and compact revision. Only a volume whose content changed is reported at all, so nothing would send that size again and the master kept a stale one. Let the dedup window belong to volume server reports alone. * master: let the decay read the size record under the lock it mutates The periodic decay picked its volumes under a read lock and replayed them under a write one, carrying the size it had read across the gap. A heartbeat landing in between was rolled back: the replay wrote the older size and compact revision over the fresh ones, and a compaction report lost that way is never resent, since only a volume whose content changed is reported. The decay has no size of its own to contribute, so it now reads the record under the same lock it mutates. * master: let a heartbeat that beat the decay stand for the cycle The decay chooses its volumes under a read lock and applies them under a write one. A heartbeat landing in that gap already did the halving the cycle owed, so applying the decay on top of it halved twice and forgot pending bytes the volume has not written yet - the double-halving the replica-dedup window exists to prevent. Both callers now give way to a report already handled for this cycle; only a real report still advances lastUpdateTime, so a quiet volume keeps decaying every pulse. * master: keep genuinely full volumes out of the decay pass A volume the disk really did fill keeps its fullSince set for good, so it was selected every pulse for a decay that cannot help it: UpdateVolumeSize refuses to recover a volume whose reported size is at the limit, and replaying a size that cannot move leaves the record as it found it. Full and quiet is the ordinary resting state of a cluster, so this was most of the pass, taking the layout write lock away from the heartbeats to do nothing. On a million tracked volumes with a hundredth of them phantom-full it costs ten thousand write locks a pulse instead of a million. * master: put the stale-replay test back on the path it guards Giving the decay the dedup window left this test short-circuiting there, so it no longer reached the locked read it was written for and passed with that read removed. Age the record past the window, which is the only case where reading it under the lock is what saves the report. |
||
|
Chris Lu
|
50b388771a |
s3: stop one abandoned request from cancelling every concurrent upload (#10948)
* grpc: a non-cancellable context is no evidence of a stale channel shouldInvalidateConnection only invalidates on Canceled/DeadlineExceeded while the context handed to WithGrpcClient is still live, so that an RPC timing out on its own does not close the shared cached ClientConn and cancel every other in-flight RPC on it. context.Background()/TODO never expire, so Err() stays nil forever and that guard always answered "invalidate" - and Background is what almost every caller passes, the S3 gateway included. One S3 request whose RPC rode an abandoned HTTP request context therefore closed the shared filer connection, and every multipart part in flight died with "the client connection is closing", surfacing to the client as 400 InvalidRequest. Only a cancellable context bounds an RPC attempt, so require one before reading it. A genuinely stale channel (a peer restart behind a stable L4 endpoint) surfaces as Unavailable, which invalidates on its own branch. * grpc: a bystander of a connection teardown is not a stale-channel witness gRPC raises ErrClientConnClosing locally, before an RPC reaches the wire, when this process has already closed the ClientConn. Every caller that touches a channel during another goroutine's teardown gets it, so reading it as a stale-channel signal lets one teardown re-arm itself across the whole herd of callers it just cancelled. The cached-connection version check keeps those callers from closing a replacement channel, but the streaming path invalidates by address alone and has no such guard. * grpc: end a stream without dropping the peer connection under it A streaming caller gets its own ClientConn, but on any error it also drops the cached non-streaming ClientConn every request handler shares with that peer, to recover a peer restart hidden behind a stable L4 endpoint. Any error includes the ordinary ones: a metadata subscription that reached its stop point, a follow callback that refused an event, a caller that gave up. The S3 gateway follows filer metadata on such a stream and reconnects forever, so each ordinary end of it cancelled every S3 request in flight against the filer. Drop the shared channel only for errors that say the peer went away, which is what invalidation is for. * test: close the connections the cascade tests leave cached Each test swaps in a fresh connection cache and restores the previous one, dropping its own entries without closing them, so the ClientConn's transport and reconnect goroutines outlive the fake filer they dialed. * grpc: say why ErrClientConnClosing's deprecation notice does not apply It points at codes.Canceled, which is the code this function exists to disambiguate. Only the message distinguishes a teardown a caller merely walked into, so the sentinel stays. |
||
|
Chris Lu
|
44115c1051 |
filer: stop TUS uploads from turning into garbage (#10945)
* filer: store TUS sub-chunks through the regular chunk writer A TUS sub-chunk was written with one assigned file id, retried up to three times against that same id, and abandoned on failure: an attempt that had landed on some replicas left a needle no session record and no entry ever references, unreclaimable by vacuum. dataToChunkWithSSE, which the regular write path uses per chunk, assigns a fresh file id per attempt and hands back the file ids of failed attempts, which are now freed the way the regular write path frees them. * filer: retry a chunk write on a fresh volume when the server 5xxs The filer's chunk writer assigns a fresh file id per attempt but only retried transient network errors, so a volume filling up and turning read-only mid-write failed the whole request even though the very next assignment would have landed elsewhere. Every other write client already routes this through ShouldReassignUpload; the filer's own write path now does the same, for regular uploads and TUS sub-chunks alike. * filer: export the chunk deletion queue The filer test harness in weed/server builds filer.Filer as a struct literal, so any code path reaching DeleteChunks dereferenced a nil queue. Exported like the neighboring DeletionRetryQueue so the harness can arm it. * filer: complete a TUS upload whose chunk records overlap A PATCH retried while its predecessor was still storing a sub-chunk - a proxy timeout with an immediate retry is enough - records the same range twice. HEAD computes Upload-Offset as the covered watermark and reported the upload fully received, but completion demanded exactly adjacent records and failed every attempt: the client concluded success from offset == length, no entry was created, and the session eventually expired, turning the entire upload into deleted needles for the vacuum to chew through. Completion now validates gapless coverage with the same watermark HEAD uses. A record extending coverage joins the entry - the read path resolves partial overlaps by ModifiedTsNs, and the raced copies carry identical bytes - while a fully covered duplicate is freed once the entry lands. * filer: allow one mutating TUS request per session at a time Nothing stopped two PATCHes from writing the same range concurrently: both loaded the same offset, both passed the conflict check, and both recorded their sub-chunks. A client whose request timed out in a proxy retries immediately while the server side is still storing the buffered sub-chunk, which is exactly that race. A session now accepts one PATCH or DELETE at a time, the way tusd locks uploads; a concurrent one is refused with 423 Locked, which TUS clients retry, and HEAD keeps answering so progress polling is unaffected. The chunk state is loaded under the claim, so a retried PATCH sees every record its predecessor left and conflicts cleanly instead of duplicating data. * test: cover a TUS PATCH raced by its own retry Stalls a PATCH mid-body over a raw connection, retries the same range while it is in flight, and expects the retry refused with 423 Locked; the upload then resumes from the reported offset and the final content must be intact. * filer: never free a TUS duplicate the entry still references Coverage is computed from ranges, so a record fully covered by another is treated as a duplicate no matter which needle it names. A malformed record naming a file id the entry keeps would have had that needle freed right after the entry landed - the corruption this change set exists to stop. The duplicates are now freed in one batch, skipping any file id the entry references; their records go with the session directory. * test: bound the raw TUS connection reads http.ReadResponse on the stalled PATCH's connection blocked until the whole go test timeout if the filer never answered. * filer: free the needles of chunk write attempts a retry replaced A volume server stores the needle locally and only then fans out to the replicas, so a replication failure 5xxs with the data already written. Each attempt assigns its own file id, so once a later attempt lands elsewhere nothing references the earlier ones: the caller only sees the chunk that succeeded, and the failed ids were dropped. They are now freed the way the caller frees them when the whole write fails. Retrying on a 5xx makes this reachable on every read-only or full volume, which is exactly the condition that filled the reporter's volumes. |
||
|
Chris Lu
|
68f0793b6f |
mount: register UNC mount points as WinFsp network file systems (#10943)
A \\server\share -dir was passed to WinFsp as a plain mount point, which treats it as a directory path on an actual remote server and fails. Turn it into the VolumePrefix option instead, so the mount registers with the WinFsp network provider: the UNC path is then reachable from every logon session, which a drive letter mounted from a service is not, and each user can map their own drive letter to it. |
||
|
Chris Lu
|
40f77503d0 |
helm: trim the Lance chart comments (#10940)
Comments only, no rendering change: the values paragraphs compress to the density of the file around them, the env-var note becomes a template comment instead of leaking into the rendered manifest, and the two spots that invite a wrong simplification - the unconditionally rendered -port.lance and the empty-placeholder platform guard - each get their one-line why. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm |
||
|
Chris Lu
|
d9d7d0be74 |
helm: serve the Lance catalog and deploy the Rust worker (#10936)
* helm: serve the S3 gateway's Lance Namespace, on by default Standalone `weed s3` serves the Lance Namespace API on 9101 unless told not to, so the chart defaulting s3.lancePort to 9101 matches weed's own posture instead of hiding the port behind a null. The flag is always rendered, so lancePort: 0 reaches weed as -port.lance=0 and genuinely disables the namespace rather than silently falling back to the binary default; 0 also drops the service port and the optional lanceIngress, which otherwise mirror the iceberg wiring. The NetworkPolicy admits the port the same way it admits icebergPort. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * helm: run the Lance maintenance worker beside the Go worker The Go and Rust workers have no overlapping jobs - Go serves vacuum, balance, EC and iceberg_maintenance, only /usr/bin/weed-worker serves the lance_* family - so a cluster serving Lance tables needs both, not an either/or switch. The worker deployment now adds a worker-lance container whenever the namespace is reachable: worker.namespaceUrl, or derived from the release's S3 service and s3.lancePort. Untouched Go container; admin address derived the same way; mTLS flags point at the already-mounted worker cert when security is on; metrics on their own worker.lanceMetricsPort (9328, next in the 932x convention) with the same health probes, service port and scrape endpoint the Go container gets, and the worker NetworkPolicy admits that port exactly when the container renders. The image carries an empty placeholder on armv7/386 where exec falls back to the shell and exits 0, so the command refuses those platforms by name; s3.lancePort: 0 is the escape hatch there. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm |
||
|
Chris Lu
|
b3be2f5449 |
filer.backup, filer.sync: stop sharing resume checkpoints across destinations (#10934)
* filer.backup: key the checkpoint by source path and sink destination The checkpoint id hashed only sink name + directory, so two backups to different buckets or endpoints sharing a directory layout advanced one checkpoint: whichever job was running pushed the shared offset forward, and a stopped or failing job later resumed from the other's position, silently skipping changes. Backups of different source paths to the same destination shared a checkpoint the same way. Each sink now reports a destination identity (endpoint or account, bucket or container, directory) and the checkpoint is keyed by the source path plus that identity. Reads fall back to the historical name+directory key when the new key has no value, so existing backups resume where they left off; writes go only to the new key. * filer.sync: include the target path in the offset key The offset stored on the target filer was keyed by source path and source filer signature only, so two syncs from the same source cluster and path to different directories on the same target cluster advanced one shared checkpoint, and the slower one could resume past events it never applied. The target path now participates in the key; "/" keeps the historical form, and a sync with a non-root target path falls back to the historical key once when its own key has no value yet. * join checkpoint key fields with NUL so they cannot alias A path or configuration value spelling out the separator could concatenate two different field tuples to the same checkpoint key. NUL cannot appear in a CLI path argument or any sane configuration value, making the encoding injective. |
||
|
Chris Lu
|
4a2879abad |
admin: show a copyable S3 object URL in the bucket file browser (#10933)
* admin: offer copyable S3 object URLs in the bucket file browser * admin: hide object urls when the bucket type lookup fails * admin: ignore an s3.public_endpoint that is not an absolute http url * mini: build the seeded s3 endpoint with JoinHostPort for ipv6 * admin: reject a query or fragment in s3.public_endpoint * mini: drop the seeded s3 endpoint when a later run disables s3 * admin: reject userinfo and bare delimiters in s3.public_endpoint, redact the warning * mini: pass its s3 endpoint as an admin option instead of mutating viper * admin: keep the rejected s3.public_endpoint value out of the log |
||
|
Chris Lu
|
2a70532d0d |
s3: log each request at -v=2 (#10931)
* s3: log each request at -v=2 * s3: quote requester and path in the access log line * s3: record the post-policy signing identity as the requester |
||
|
Chris Lu
|
d2c470af1b |
S3: commit SSE GET status only after the first read succeeds (#10935)
The SSE streaming path kept writing 200/206 from filer metadata before fetching or decrypting anything, so a missing needle or failed decrypt setup surfaced as a broken 200 body. Same deferral as the plain path: the status commits on the first body write, and every failure before that returns to the handler for a clean S3 error response. |
||
|
Chris Lu
|
115756dd41 | helm: expose loadBalancerClass, loadBalancerIP, loadBalancerSourceRanges on services (#10929) | ||
|
Chris Lu
|
d9d5fab35b |
S3: commit GET status only after the first read succeeds (#10930)
streamFromVolumeServers wrote the 200/206 status from filer metadata before any byte had been fetched from a volume server, so a missing or corrupted needle surfaced as a broken 200 body and the request metrics recorded a success. Defer the status commit to the first body write: a failed first read now returns a clean 500 before headers, while the wire timing of successful responses is unchanged since net/http buffers the status line until body bytes arrive anyway. |
||
|
Chris Lu
|
863fec6c3f |
S3: let a key that is a prefix of other keys be an object (#10912)
* filer: keep the sentinel when CreateEntry reports an update failure CreateEntry flattened the error UpdateEntry wraps, so errors.Is stopped matching and ErrExistingIsDirectory and ErrExistingIsFile never reached the S3 mapper, which answered a retryable 500 instead. * s3: let a key that is a prefix of other keys be an object S3 keys are flat, so "a/b" and "a/b/c" are independent objects that coexist in either write order. The filer stores a key as a path, so one of them has to live on the directory the other is nested under. Writing the nested key first refused the prefix key outright. Writing it second promoted the file to a directory, which kept its data but lost the key: an empty object left nothing to recognise it by and disappeared, and one with data listed under a trailing slash it never had. Mark the directory that carries such a key, and write the object onto it when the path is already a directory. The mark makes an empty prefix object visible to listings and readable by GET and HEAD, keeps the empty folder cleaner off it, and lists it under the key it was written with. Deleting the key strips the mark back off along with the data. * filer: keep a TTL off a directory that stands for an object An expired entry is deleted a row at a time, so expiring a directory removes it and leaves everything under it unreachable. Promoting a file to a directory carried its TTL across, and a promoted file is exactly the one that has keys nested under it. Drop the TTL on promotion, and leave one an older build wrote alone. The lifecycle worker still expires the object, through the delete that leaves the directory behind. * s3: delete the null version of a key other keys are nested under The routed delete cannot remove an entry that other keys live under, and answered a retryable 500 rather than falling back to the lock path the unversioned delete already falls back to. That path then looked the entry up under the bucket with the whole key as its name, so the demote wrote it back one directory too high and failed as not found. Fall back on any non-precondition error, and split the key before deleting it. Trailing-slash directory markers with children reach the same delete. * filer: keep the sentinel when MkFile and Mkdir report a create failure Same flattening one layer out: every mkFile caller lost the sentinel, so a CopyObject onto a key that other keys are nested under answered a retryable 500 where a PutObject of the same key answers 409. * s3: copy and rename a key that other keys are nested under Such a key is stored on the directory those keys live in, and copy and rename both refused it: the source lookup maps every directory entry to NoSuchKey, so a key a plain GET serves could not be copied or moved, and the destination side refused it as a directory conflict. The source is read through a view of the entry as the object it names. The destination is written the way a PutObject of that key writes it. A rename at either end copies the object's own data across and strips it off the source key rather than going through AtomicRenameEntry, which moves a directory by moving everything under it - the nested keys are not part of what is being renamed. |
||
|
Chris Lu
|
46ce2c45a2 |
mini: reserve the admin gRPC port instead of binding it late (#10928)
* mini: reserve the admin gRPC port instead of binding it late Port selection probes every port with a throwaway listener and closes it. Master, filer, volume and S3 bind a moment later, but the admin waits for all of them first and only then binds its worker gRPC port, roughly two seconds in. That port defaults to the admin http port + 10000, which lands inside the Linux ephemeral range, so one of the cluster's own outgoing gRPC dials can take it during the gap and the admin dies on bind, taking the worker with it. Keep the listener from the availability check and hand it to the admin. * mini: clear the admin gRPC reservation before retaking it A rerun inside one process would otherwise inherit the closed listener of the previous run whenever the reservation fails, and the admin would accept it and only find out inside Serve. * mini: snapshot the admin options for the startup goroutine The cleanup path read the package-level options long after the goroutine started, so a later in-process run could have its reserved listener closed by the previous run. |
||
|
Chris Lu
|
51eb5333d3 |
ec: read a needle's intervals in parallel (#10911)
* ec: read a needle's intervals in parallel A needle spanning more than one EC block gets one interval per block, and consecutive blocks live on different shards. We read those intervals in sequence, so a 4MB chunk landing in a volume's 1MB small-block region cost five round trips to five different servers. Read them concurrently into disjoint slices of a single buffer, at most 8 in flight. Same change in the Rust volume server's phase C. * ec test: seed the random payload instead of the deprecated rand.Read |
||
|
Chris Lu
|
69cc2869ad |
Fixes from the review of the admin bucket policy UI (#10907)
* admin: treat a missing S3 Tables policy as an empty load, not an error
The bucket/table policy GET relayed the backend's 404 NoSuchPolicy to the
dialog, whose loader treats any non-OK response as a load failure and
keeps Save and Delete blocked. A bucket or table without a policy could
never be given one. Return policy null instead, the same contract
ShowBucketPolicy uses for classic buckets.
* admin: reject policy documents the structured editor would misread
A top-level JSON array passed the object guard (typeof [] is 'object')
and loaded as a zero-statement policy, which the next commit would
rewrite to an empty document. Object elements in Action/Resource were
coerced to '[object Object]' and saved that way on the s3tables surface,
which stores policies verbatim. Both now throw, which routes the
document to the JSON tab like other unrepresentable shapes.
* admin: let the JSON tab save documents the structured editor can't model
Save with the JSON tab active required a round-trip through
policyDocToEditorState, so exactly the documents the dialogs shunt to
'JSON tab only' mode (unrepresentable Effect, Resource+NotResource, and
the like) could never be saved - Delete was the only mutation left.
Invalid JSON still blocks; an unrepresentable document now saves and the
editor state stays marked unparsed.
* admin: pin the policy editor to what each consumer's backend supports
The s3tables evaluator has no NotResource/NotPrincipal fields - it
silently drops them, turning Allow+NotResource into allow-everything and
making Deny+NotPrincipal inert - and it only matches s3tables: actions
against s3tables ARNs, while the editor suggested s3: actions and
arn:aws:s3::: resources. New registerPolicyEditor knobs: allowNegation
hides the Not* modes and routes documents using them to the JSON tab;
resourceSuggestions pins the Resource autocomplete to the open
resource's ARN; the S3 Tables dialogs get an s3tables-only action
datalist. requirePrincipal now also hides NotPrincipal, which
policy_engine.ValidateBucketPolicy always rejects, and the client-side
check requires Principal specifically to match that server rule.
* admin: save S3 Tables policies from a button, not form submission
The multi-input structured editor sits inside a form whose Save button
was type=submit, so Enter in any single-line editor input - accepting an
autocomplete suggestion, say - implicitly submitted whatever half-built
statement the editor held, and the backend stores the document verbatim.
A lone statement with no Principal matches nobody, locking out every
non-owner. Save is now an ordinary button and the form ignores
submission.
* admin: block zero-statement policy saves
Committing the active tab before the emptiness check made 'Policy JSON
is required' dead code: an empty editor serializes to {"Statement":[]},
which the s3tables backend stores verbatim - evaluated default-deny for
every non-owner, while the statement-count column keeps showing 'Not
configured'. All three policy dialogs now refuse a save with no
statements and point at Delete instead. The classic bucket modal only
gained a clearer message; the server already rejected the document.
* admin: guard S3 Tables policy mutations against stale and overlapping requests
The save/delete completions ran against whatever resource the shared
modal happened to show by then: a slow PUT for one bucket would hide the
modal mid-edit of another and misattribute its alerts, a late DELETE
cleared the shared textarea over the newly opened resource with its
loaded flag set, and nothing stopped a double-click from firing two
overlapping mutations. Ported the classic modal's pattern: capture the
target on start, flag the mutation in flight with the buttons disabled,
and only touch the UI when the completion still matches the open
resource. Success now reloads the page, which also keeps the Policy
column's statement count honest.
* admin: confirm before deleting an S3 Tables policy
Delete Policy sat next to Save and fired on a single click; with
default-allow enabled one stray click silently dropped the resource
policy and left the bucket open to every principal. Same confirmation
the classic bucket modal already has.
* admin: let a corrupt stored bucket policy be shown, fixed, and deleted
A stored document the decoder rejects made the policy GET 500, and with
the loaded flag never set the modal blocked both Save and Delete - the
one policy an operator most needs to remove was the one they couldn't,
even though the delete path never reads the document. The GET now
returns the raw bytes alongside a null policy; the dialog hands them to
the JSON tab and unblocks the buttons.
* admin: url-encode the bucket name in the policy API calls
The filer lists any directory under the buckets path, names S3 would
never allow included; one carrying '#' or '%' broke the fetch URL or
addressed a different name than the modal shows.
* admin: drop stale edit-policy responses on the IAM policies page
The same race the bucket and S3 Tables dialogs already guard against:
open one policy's editor while its GET stalls, open another, and the
late response populates the editor under the second policy's name -
Update then saves the first policy's statements over the second.
* admin: warn before a bucket policy save drops unsupported fields
The editor tracks unmodeled top-level keys precisely so
confirmPolicyFieldDiscard can warn before the server's Version+Statement
decode discards them, but only the IAM page called it; the bucket modal
saved a pasted document with e.g. a console-generated Id without a word
while the editor kept displaying the field.
* s3: enforce the bucket policy size cap on both surfaces
The 20KB cap lived only in the admin UI, so a larger policy stored via
the S3 API displayed there but could never be re-saved, desyncing the
two writers the cap comment claimed could not desync. The constant now
lives in policy_engine next to the shared validator and PutBucketPolicy
rejects oversized documents with PolicyTooLarge, matching AWS.
* admin: ship the policy editor's fieldset styles with the editor
The .policy-stmt-* rules that undo Bootstrap's full-width legend reset
stayed behind in policies.templ when the editor markup moved to the
shared script, so the bucket and S3 Tables dialogs rendered Actions/
Resource/Principal as full-width jumbo headings. PolicyDatalists is the
component every consumer already renders once; the styles live there
now.
* s3: mirror bucket policy changes into the IAM store from the metadata subscription
The advanced-IAM path appends the bucket-policy:<bucket> document to
every STS/session evaluation, but only this gateway's own PutBucketPolicy
maintained that mirror - a policy tightened or created through the admin
UI (or another gateway) never reached it, so revoked access stayed live
indefinitely, and the delete side was an unimplemented TODO in any case.
The metadata subscription now diffs the stored policy on every bucket
entry change and updates or removes the mirror, covering all writers and
deletion with one mechanism; IAMManager gains the missing
RemoveBucketPolicy.
* admin: deduplicate the bucket policy write path
Set and Delete carried line-for-line identical filer closures;
bucketPolicyMutation already treats nil as clear-the-key. The shared
helper sits below Set's validation, since ValidatePolicy cannot take the
nil document Delete passes.
* s3: drop ValidateBucketPolicy's re-checks of ValidatePolicy rules
Both callers run ValidatePolicy first, which already enforces the
version and at-least-one-statement rules; the duplicates were dead code
with drifted error text.
* admin: seed a new statement's Resource from the pinned suggestions
A fresh statement on the S3 Tables dialogs started with no resource row
at all; seed it with the broadest pinned ARN the same way cfg.bucket
already seeds the classic modal.
* admin: refuse to save Not* fields the backend would silently drop
Hiding the NotResource/NotPrincipal modes was not enough where negation
is disallowed: the JSON tab accepts any valid document (that is its
job), and a statement's Advanced-fields box can reintroduce the keys, so
an s3tables save could still store fields the evaluator drops - turning
Allow+NotResource into allow-everything. commitPolicyActiveTab now runs
a final document-level check over what would actually be saved; Delete
stays available for cleanup.
* s3: move the IAM bucket policy mirror on a bucket rename
A same-directory rename delivers one event carrying both entries, and
the byte-equality short-circuit skipped the new name's mirror when the
policy was unchanged - while the replayed delete for the old name
removed its mirror, leaving the renamed bucket unmirrored. The mirror
decision is now a pure function that removes the old name and writes the
new one regardless of byte equality, with the rename cases unit tested.
* s3: backfill the IAM bucket policy mirror on lazy bucket loads
The metadata subscription only mirrors changes, so a policy that
predates the IAM integration never reached the bucket-policy:<bucket>
mirror and its grants did not bind on the IAM path until the policy was
next modified. The gateway is deliberately lazy at startup (nothing
lists all buckets), so the backfill hooks the same place a bucket's
policy first becomes known: the cold bucket-config load. EnsureBucketPolicy
writes only when no mirror is stored, so repeat loads cost one cached
read.
* s3: reconcile the bucket policy backfill against concurrent changes
The backfill's check-then-write could race an event-driven mirror update
or removal and re-store bytes that were already stale, with no later
event to heal it. EnsureBucketPolicy now reports whether it wrote, and a
write is reconciled against a fresh authoritative entry read: a changed
policy is re-mirrored, a removed one is removed. Anything changing after
that read fires its own event, which finds the backfill's write already
present and supersedes it. The backfill also carries the entry's raw
bytes rather than a re-marshaled document, so the reconcile can
byte-compare.
* s3: prime the bucket policy mirror before advanced-IAM authorization
The backfill ran from the lazy bucket-config load, but IAM authorization
evaluates the bucket-policy:<bucket> mirror before any handler runs - a
grant carried only by a not-yet-mirrored policy denied forever, and the
denied request never reached the code that would have loaded the bucket.
authorizeWithIAM now primes the bucket config first (an in-memory cache
hit once warm), and the backfill runs synchronously on the cold load so
the very first authorization already sees the mirror.
|
||
|
Chris Lu
|
68ec8ca655 |
admin: honor a persisted or admin.toml maintenance enabled=false (#10909)
* admin: honor a persisted or admin.toml maintenance enabled=false The startup path discarded an operator's enabled=false twice over: ApplyDefaultsToProtobuf treated the bool zero value as unset and applied the schema default of true, and a force-enable migration block flipped any survivor. With the legacy /maintenance UI routes gone, nothing could write the config either, so the maintenance system ran unconditionally. Keep the persisted enabled flag across schema-default application in LoadMaintenanceConfig, drop the force-enable block, and add a top-level [maintenance] enabled key to admin.toml as the config surface, persisted through SaveMaintenanceConfig like the per-task settings. Absent config still defaults to enabled. * admin: track presence on the maintenance enabled flag A plain proto3 bool cannot distinguish an operator's persisted false from a legacy file that simply omits the field, so honoring false would have silently switched maintenance off for configs written before the toggle could be persisted. Make the field optional: files that predate presence tracking keep the enabled default, while a file that explicitly persists the toggle is honored either way. |
||
|
Chris Lu
|
71a8c77a36 |
telemetry: let the dashboard pick the confirmation window (#10904)
* telemetry: let the dashboard pick the confirmation window * telemetry: cover the serialized threshold map through the stats handler |
||
|
Chris Lu
|
9c8d3b6a81 |
ec: refund the cleared leftover shards' slots in the encode source health check (#10903)
* erasure_coding: one home for the shard-count to volume-slots conversion * ec: refund the cleared leftover shards' slots in the encode source health check |
||
|
Chris Lu
|
36c97344ef |
s3: confine a Lance catalog table location to the caller's own bucket (#10901)
The Lance namespace gateway took the request-body location field, trimmed a trailing slash, and passed it straight to the marker sink. That location feeds TableDataDirFromMetadataLocation, which joins it under /buckets and collapses any ../ segments, and writeMarker's CreateEntry then auto-creates every missing parent. A caller could point the location at another tenant's bucket, or escape /buckets entirely, and plant a fixed-name marker (recursively creating the parents) or hide a victim's live table with .lance-deregistered. Confine the declared location the way the Iceberg gateway already does: require an s3:// URI whose bucket is the caller's own and whose path carries no traversal segment, on both the declare and register handlers. |
||
|
Chris Lu
|
74038e1b14 |
master: don't let a dead KeepConnected handler close its successor's channel (#10900)
A client that reconnects before the old handler exits re-registers the same client name, and addClient overwrites the map entry. The old handler's deferred deleteClient then closed whatever channel the map held under that name: the new, live stream's. Receiving from a closed channel returns nil immediately and forever, so the new handler's send loop degenerated into sending empty responses at wire speed, pinning a core on each side until the client killed the connection. deleteClient now closes the channel its own handler registered and leaves the map entry alone unless it still points to that channel. This also closes the previously orphaned old channel, whose drain goroutine used to leak. The send loop treats a closed channel as an exit instead of a message stream. |
||
|
Chris Lu
|
cf0dba334c |
s3api: no filer failover after the callback has consumed part of a response (#10902)
s3api: no filer failover after fn has consumed part of a response withFilerClientFailover replays fn verbatim on the next filer, so a filer that died mid-stream followed by a healthy peer returned success with the callback's closure-captured accumulator holding the dead filer's prefix twice; the per-attempt accumulator in listWithRetry could not close this, because the replay happens inside a single attempt. Track delivery on the connection handed to fn: once a unary reply or streamed message has reached the callback, surface the transport error unwrapped instead of failing over, and let callers replay from a clean slate. A filer that fails before delivering anything fails over exactly as before. |
||
|
Chris Lu
|
c167af541e |
telemetry: confirm a cluster after a week of reports, not two days (#10899)
* telemetry: sync the server module to go 1.26 The root module moved to go 1.26 but the telemetry server module, which replaces seaweedfs with the repo root, stayed on 1.25.8, so go refuses to build or test it until the directive catches up. * telemetry: confirm a cluster after a week of reports, not two days Two days of history still lets recurring CI and demo clusters into the confirmed fleet: anything torn down and rebuilt across a UTC midnight counts. Requiring seven distinct UTC days keeps the fleet charts and the version/OS distributions to clusters that actually stay up; real clusters qualify after their first week, and the fallback to all active clusters while none is confirmed is unchanged. |
||
|
Chris Lu
|
0f85d005ad |
server: 416 only when no requested range overlaps, with Content-Range, and the Rust mirror (#10889)
* filer, volume server: return 416 when no requested range overlaps the content * seaweed-volume: return 416 when no requested range overlaps the content * server: check the range test error, use the request context, fix the no-overlap comment boundary |
||
|
Chris Lu
|
173adbc291 |
master: never re-seed a raft cluster over committed state under -raftBootstrap (#10883)
* master: never re-seed a raft cluster over committed state -raftBootstrap deleted logs.dat, stable.dat and snapshots on every start and then bootstrapped a fresh cluster. Since hashicorp raft only snapshots after 8192 log entries, the TopologyId lives in the log, not in a snapshot, so the pre-wipe snapshot recovery found nothing and each restart minted a new cluster identity. A master that came up while it could not reach its peers seeded a rival cluster; when the two logs met, SetTopologyId's split-brain guard fatally stopped every master holding the other id, and the master layer crash-looped with no quorum. Bootstrapping is genesis. Drop the wipe and the inline bootstrap. The first master in -peers already mints a cluster once it has confirmed no peer has a leader, so the flag has nothing left to do and is now ignored; keeping that one master the sole bootstrap authority is what stops a partition from minting two clusters, so the flag must not widen it either. A master with state rejoins its peers, and one whose data dir was reset is admitted by the sitting leader instead of forking again. * test: cover -raftBootstrap restarts in the multi-master suite Three masters start with -raftBootstrap, the way the helm chart renders it on every master on every roll, and the cluster has to hold one TopologyId after they all restart. /dir/status is proxied to the leader, so each master's own view of the identity is read out of its log, which is where a fork shows up. Before the fix the hashicorp case minted a new id on each restart. |
||
|
Chris Lu
|
3b10e43d5d | test: wait for volume server registration in the FUSE p2p harness (#10897) | ||
|
Chris Lu
|
9d06f2c378 | test: keep per-test log directories in the FUSE DLM harness (#10893) | ||
|
Chris Lu
|
9d4270f118 | test: wait for volume server registration in the FUSE DLM harness (#10891) | ||
|
Chris LuandJunker der Provinz
|
c58795354a |
s3api: retry a transient filer failure on metadata listings (#10890)
* s3api: retry a transient failure when listing multipart uploads/parts A blip on the way to the filer failed the whole ListMultipartUploads or ListParts request. Both reported failure points sit inside one streaming listing: the ListEntries call that opens the stream, and the stream.Recv calls that drain it. Neither retried, so a single Unavailable answer from a filer that was restarting turned into a 500 for the S3 client. Replay the listing instead, bounded to three attempts with a 100ms backoff that doubles. Only a transient failure is replayed. A not-found answer stays authoritative so the empty-list branch still works, and every other error still reaches the client on the first attempt. This is scoped to (*S3ApiServer).list rather than added inside DoSeaweedListWithSnapshot, which mount, the shell and the other object listings share, and where a retry after a partial stream would re-deliver entries the callback had already seen. Within one call to list, a replay is safe: it collects into a fresh slice each time, so it can neither duplicate nor drop entries. That guarantee does not extend past this function. withFilerClientFailover already re-runs its callback against the next filer on any non-NotFound error without resetting the caller's accumulator, so on a multi-filer gateway a mid-listing failover can itself produce a duplicated result with err == nil, independent of this change and not fixed by it. Noted in the PR rather than silently left for someone to rediscover. Fixes #7221 References #7235 * s3api: move the listing retry inside list itself --------- Co-authored-by: Junker der Provinz <jdp@braethoria.com> |
||
|
Chris Lu
|
228500fe37 |
install.sh: install the Rust maintenance worker (#10882)
* install.sh: install the Rust maintenance worker The release publishes weed-worker but the installer only knew weed and the Rust volume server, so the one binary that cannot be built without a Rust toolchain was the one you had to download by hand. --component all skips it on a platform it has no build for rather than failing an install that already put two binaries in place; asking for it by name there still says so. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * install.sh: clean each component's temp directory as it finishes The EXIT trap is per-process, so installing more than one component left every extraction but the last behind. Cleaning at the end of the function keeps the trap for the paths that exit early. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm |
||
|
Chris Lu
|
7ebf2ebac3 |
Build the Rust worker against the protoc that ships with the build (#10881)
* worker: compile plugin.proto with the protoc that ships with the build seaweed-volume already does this: protoc-bin-vendored carries the binary, so the build needs no package manager and every build sees the same version. An explicit PROTOC still wins, which is what lets the lance crates - whose own build scripts read the same variable - share it. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * ci: point the worker builds at the vendored protoc The jobs installed protobuf-compiler for lance's build scripts. They read PROTOC, so pointing it at the binary protoc-bin-vendored already puts in the registry serves them without a system package - one less apt call on the way to a release, and the same protoc a developer's build uses. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * docs: say what the worker build needs from protoc The lance crates' build scripts are the ones that need it, not ours, and they take the same vendored binary. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm |
||
|
Chris Lu
|
cc8364a03e |
Ship the Rust maintenance worker with the release (#10879)
* worker: name the binary weed-worker It is the Rust side of `weed worker`, the way weed-volume is the Rust side of `weed volume`, and lance is the first family of jobs it carries rather than the only one it ever will. The crate keeps its own name: when a second family arrives the bin target moves to a crate of its own, under this name. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * docker: ship the Rust maintenance worker in the image Lance table buckets need a worker that can read the format, and until now the only way to get one was a Rust toolchain and a cargo build. It now sits at /usr/bin/weed-worker beside the Rust volume server, reached as `docker run chrislusf/seaweedfs worker-rust --admin host:23646` — the verb mirrors volume-rust, so plain `worker` still runs the Go one. Taken pre-built or not at all: the lance jobs pull in arrow and datafusion, far too large a tree to compile inside the image build, so an architecture CI did not build for gets the empty placeholder the entrypoint refuses to exec, the way the Rust volume server already does. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * ci: build the Rust worker for the container images The same native cross-compile the volume server uses, so the release, latest and dev images all carry it on amd64 and arm64. The artifact holds both binaries now, so it is named for that rather than for the volume server. Only the release directory each job builds is cached: with a debug profile beside it the worker's target/ reaches 24GB, against a 10GB cache budget. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * ci: publish Rust worker binaries with the release Linux amd64 and arm64 only: the worker runs beside the cluster it maintains, and its dependency tree makes every extra target an expensive build. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * ci: build and test the Rust workers on change Nothing built seaweed-worker in CI, so the release and the container images would have been the first place a break showed up. Tests run in release too, rather than compiling lance, arrow and datafusion again in another profile. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * docs: say how to get a released worker Neither the image nor the release tarballs were mentioned; a toolchain and a cargo build read as the only way in. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * ci: install protoc for the Rust worker builds lance's crates compile their own protos, and unlike seaweed-volume they do not vendor a protoc to do it with, so every job that builds the worker failed at lance-encoding's build script. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * ci: do not persist credentials in the worker release checkout The upload step is handed a token explicitly; a cargo build script should not find another one sitting in the checkout's git config. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * docker: keep the worker's argument boundaries Unquoted $@ splits on whitespace and expands globs, so an argument carrying either arrived as something else. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm |
||
|
Chris Lu
|
c3f4d799b5 |
helm: install chart CI against an image tag that exists (#10877)
* helm: install chart CI against an image tag that exists The release bumps appVersion on master well before the container build publishes that tag, and the chart CI runs on the bump commit, so every release turns it red with ImagePullBackOff. Resolve the tag first and fall back to latest while the new one is still building. * helm: run the chart CI when the workflow itself changes * helm: bound the registry lookup in the chart CI An unbounded curl can hold the job, and the log did not say why the tag was rejected. Cap it and print the status. |
||
|
Chris Lu
|
c1a993bc3b |
filer: keep the TUS sub-chunks that already landed when a write fails (#10876)
* filer: keep the TUS sub-chunks that already landed when a write fails A PATCH is split into 4MB sub-chunks, and each one is recorded in the session as soon as it is stored. The session listing is what HEAD reports as Upload-Offset and what the final entry is assembled from, so a record is a promise that the data behind it exists. When a later sub-chunk failed - a read-only volume, or a client that hung up mid-body - the error path deleted the needles of every sub-chunk the same PATCH had written but left their records in place. The resuming client was then told to continue past bytes the filer had just queued for deletion, and the upload completed into a gapless manifest pointing at needles that were gone: HEAD returned the right size, GET died mid-body once a vacuum reclaimed them. Recorded sub-chunks now stay, which is what resumption expects: the client picks up at the offset the session reports, and an upload that is abandoned frees its chunks with the session. * filer: drop a TUS chunk's record before freeing its data filer.CreateEntry can return an error with the entry already inserted - the parent-directory pass runs after the insert and keeps the entry when it fails. A failed saveTusChunk therefore does not mean the record is absent, and deleting the needle outright left the same corruption the resume path used to cause: a session record pointing at data that is gone. Remove the record first and only free the needle once it is gone. A record lost with its data still stored merely leaks, which the vacuum and fsck paths already account for. * test: cover a TUS PATCH that is cut off mid-body Resets the connection after one 4MB sub-chunk has landed, resumes from the offset the session reports, and vacuums before reading the file back, so anything the filer deleted behind a kept record shows up as a short read. |
||
|
Chris Lu
|
34bb444f33 |
test: drive the Lance namespace with Spark (#10864)
* test: drive the Lance namespace with Spark
The counterpart of catalog_spark, which does this for the Iceberg REST
catalog. Spark is the engine most likely to be pointed at a lakehouse,
and it reaches the Lance catalog through the connector's DSV2 catalog -
org.lance.spark.LanceNamespaceSparkCatalog with impl=rest - over the same
routes every other client uses.
SHOW NAMESPACES -> ['`sparklance-lcephd80`.ml']
SHOW TABLES -> ['sparklance-lcephd80$ml$embeddings']
count -> 3
filtered -> [(2, 'two'), (3, 'three')]
count after a second commit -> 4
The second insert is there on purpose: a store that cannot order commits
fails on the second one, not the first.
Two things the run settled that were guesses beforehand. CREATE TABLE
works, because the connector declares through the namespace and writes the
data itself rather than pushing Arrow at the server. And SHOW TABLES
returns the namespace's own identifiers - bucket, namespace and name
joined by the delimiter - not bare Spark table names.
Credentials go under the catalog's storage.* prefix, which is handed to
lance as object_store options; a gateway without STS vends none, the same
trap the LanceDB suite documents.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: verify the Lance table bucket was actually created
weed shell prints a command's own failure and still exits 0, so the harness
would go on to blame Spark for a bucket that was never made.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: bound the Docker probe
An unhealthy daemon makes docker version hang, and the probe runs before the
test has a timeout of its own.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: keep the ivy cache under the user's cache directory
It is mounted into a container running as root, so a shared temp path lets
another local user pre-create it and choose what Spark loads.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: assert the vector column's type, not only its name
A column that came back as array<double> or array<string> would still be
called vector and still pass.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: read the dataset off its location for real
The catalog being optional is the property that lets duckdb and pandas read
these tables; it was asserted in a comment and printed, never exercised.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: do not persist credentials in the Spark Lance checkout
The job only uploads a log on failure; nothing in it pushes.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: say the hosts in the README are placeholders
The suite passes dynamically allocated host.docker.internal ports.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
|
||
|
Chris Lu
|
4af6798639 |
helm: render the mysql secret and env only for the mysql filer store (#10872)
The db credential secret and the filer's WEED_MYSQL_* env were gated on filer.enabled alone, so a filer on mongodb, redis, postgres or leveldb2 got a generated mysql secret it never reads - kept forever by resource-policy: keep - plus a mysql-db-host pointing nowhere. Gate all of it on WEED_MYSQL_ENABLED, which is how the store is selected, plain keys and secret-backed ones alike. An enable flag the chart cannot read - a valueFrom, or one in secretExtraEnvironmentVars - counts as selected, so nothing is dropped from a filer that is actually on mysql. |
||
|
Chris Lu
|
301d83cc7a |
test: wait for the master to register the volume servers before failover tests run (#10871)
The failover harness treated an open volume server port as readiness, but the master only learns of a volume server from its heartbeat. A lone master refuses heartbeats until its bootstrap check elects it, and the servers back off and retry, so registration lands seconds after the ports answer. Tests that started writing in that window assigned against an empty topology, which fails with "no free volumes left" and reaches the mount as ENOSPC. |
||
|
Chris Lu
|
c0a9b110dd |
volume: stop reporting read-only volumes that are no longer here (#10867)
* volume: clear per-collection metrics when a collection leaves a server The read-only and disk size gauges are only ever set for collections the heartbeat still finds here, and nothing zeroes the rest. volume.balance marks a volume read-only to move it, so the last heartbeat that saw it counts it read-only - and if it was the collection's last volume on that server, that count stands until the process restarts. The dashboard then shows read-only volumes that volume.list -readonly cannot find anywhere. Remember what each heartbeat set, and drop what is gone on the next one. * volume: stop the read-only volume count from wrapping at 256 The per-collection counters were uint8, so a server holding 256 read-only volumes of one collection reported zero of them. * volume: read the read-only flags once when counting them The heartbeat asked IsReadOnly for the verdict and then read noWriteOrDelete and noWriteCanDelete straight off the volume, unlocked, so the reasons could disagree with the verdict they were explaining. Take them together, under one lock. The location is now nil-checked rather than skipped by short-circuit evaluation, so a volume that has not joined a disk location yet stays safe. * volume: let only a surviving volume keep its collection reported A volume being deleted for expiry still made an entry in the read-only counts, which is what the cleanup reads as "this collection is still here". The collection's last volume could go and its series would stand for one more heartbeat. Count the survivors only. * volume: size a collection from the volumes it still has The size totals are rebuilt from scratch every heartbeat, so subtracting a volume that is about to be deleted took the surviving volumes' sizes down with it: a collection keeping a small volume and losing a larger one reported the difference, or lost its entry and kept the previous heartbeat's number. * volume: cover the deleted bytes total in the surviving volume test Deleted bytes are totalled the same way as sizes and were going unchecked, so the test now leaves deleted needles on both volumes and pins that gauge too. |
||
|
Chris Lu
|
96304b6870 |
S3: source config credentials from the environment, and let the chart point at an existing secret (#10868)
* s3: resolve ${VAR} in static config credentials from the environment
A deployment that keeps its S3 keys in a secret store had no way to hand
them to the gateway: -config takes a file, so the keys had to be written
into that file. Let a key in the static config name an environment
variable instead, and drop any credential whose reference stays unset so
the placeholder never becomes a usable key.
* helm: source the generated s3 identities from an existing secret
The only way to reuse credentials that already live in a Secret was to
hand-author the whole seaweedfs_s3_config JSON, since the literal keys in
values.yaml end up in git and a lookup-based keyRef renders empty under
helm template and Argo CD. Let s3.credentials.admin/read name a Secret and
its keys instead: the generated config references them as ${VAR} and the
gateway resolves them from the environment, so nothing is read from the
cluster at render time.
* s3: treat an empty environment value as an unresolved credential reference
A secret store can hand over a key that exists but is blank. Resolving it
would leave an access key whose signing secret is empty, so count it as
unresolved and drop the credential.
* helm: render the s3 secret when only the all-in-one auth flag is set
The all-in-one deployment mounts the s3 secret whenever any of the three
enableAuth flags is set, but the secret itself only rendered for the s3 and
filer flags, so allInOne.s3.enableAuth on its own left the pod waiting on a
secret nothing creates.
* helm ci: check the credential wiring on every workload that mounts it
The render check only looked at the standalone s3 deployment and only at
one of the four variables, so a helper that bound a variable to the wrong
secret key would still pass.
* helm: create the all-in-one s3 secret for every flag that mounts it
The all-in-one pod mounts the secret on any of the three enableAuth flags,
so keying its creation off allInOne.s3.enableAuth alone still left
filer.s3.enableAuth without filer.s3.enabled pointing at a secret nothing
creates. Mirror the deployment's own condition instead, and check each
flag renders both the mount and the secret.
* s3: reject a malformed credential reference instead of keying on it
A typo such as ${MY-VAR} matches no substitution, so it survived expansion
and the placeholder itself became the access key the gateway accepted.
Require every ${ in a static credential to open a well-formed reference.
|
||
|
Chris Lu
|
480795d40d |
release: cut the whole release from the version bump workflow (#10870)
* release: cut the whole release from the version bump workflow The bump workflow stopped after pushing the version commit, and the rest was manual: create the release, then run "Prepare release" in the csi-driver and the operator. It now pushes the tag itself, which is what starts the binary, container and helm workflows, creates the release with generated notes, and dispatches the other two repositories, waiting for both. Pushing the tag and reaching the other repositories both need RELEASE_PAT; GITHUB_TOKEN raises no events that start workflows. * release: tighten the release workflow after review Check out master explicitly: a dispatch can select any branch, and the tag, the commit and the release would then come off that branch while the downstream job dispatches master. Scope contents:write to the job that pushes; the downstream job talks to the other repositories with RELEASE_PAT and needs nothing here. Wait for the module proxy to serve the release commit as the tip before dispatching, instead of priming it and hoping. The dispatched workflows pin seaweedfs with `go get -u ...@latest`, so a stale tip means they release against a pre-release commit, silently. Identify the dispatched run by diffing the run list against the snapshot taken before dispatching, rather than assuming the newest run is ours. * release: wait on the downstream release, not on the run that makes it A dispatched run cannot be told apart from a concurrent one: the API does not report the inputs a run was dispatched with, so watching "the run that appeared after mine" can watch someone else's and report their result as ours. Wait for a release to appear in the downstream repository instead. That is the thing being waited for, and it holds however many runs are in flight. |
||
|
Chris Lu
|
5e7ab43ddd |
test: read Lance tables from DuckDB (#10866)
* test: read Lance tables from DuckDB
The LanceDB and Spark suites go through the catalog. DuckDB does not: its
lance extension reaches the data over S3 with no namespace involved, which
exercises the other half of the design - a table bucket's layout is a
valid Lance dataset directory, so a table stays readable when the catalog
is not in the path.
scan_rows=128
scan_columns=id,title,vector
filtered_rows=5
nearest=1,0,2
It also pins the one place the layout costs us. DuckDB's replacement scan
recognises a dataset by a .lance path suffix, and tables created through
this catalog deliberately have none: the catalog entry is the dataset
directory, a table name may not contain a dot, and a suffix would leak
into ARNs and policies. So __lance_scan is the way in, and the bare
SELECT ... FROM 's3://...' form does not see these tables.
The test asserts both halves - a suffixed path is read, a suffix-less one
is not - so if the extension ever recognises a bare directory, it fails
and says to update the documentation rather than leaving it wrong.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: require the catalog error from the suffix-less read
Any failure satisfied the old check - a missing extension, bad credentials,
an unreachable endpoint - so the assertion could pass without the
replacement scan ever classifying the path.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: verify the Lance table bucket was actually created
weed shell prints a command's own failure and still exits 0, so the harness
would go on to blame DuckDB for a bucket that was never made.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: bound the Docker probe
An unhealthy daemon makes docker version hang, and the probe runs before the
test has a timeout of its own.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: order the aggregates the assertions read
string_agg over an unordered relation may return the names, and the vector
search's ids, in any order, so the expectations could fail on a run where
nothing changed.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: do not persist credentials in the DuckDB Lance checkout
The job only uploads a log on failure; nothing in it pushes.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
|
||
|
Chris Lu
|
35d53a20f6 |
master: let the leader admit a master that starts with no raft state (#10865)
* master: answer with the leader raft already knows Topo.Leader() backs off for up to 20 seconds waiting for an election. Callers that a health probe or a client is blocked on cannot afford that: /cluster/status, /cluster/healthz and /readyz all sit past the probe timeout of both the helm chart and the operator, so a master that is still joining looks dead rather than joining, and the kubelet restarts it. informNewLeader and SendHeartbeat hold the client on a master that cannot serve it, exactly when it should move on to find the one that can. Answer these from MaybeLeader instead, which reports what raft knows right now. MaybeLeader takes over the "am I the leader myself" fallback that Leader() used to apply on top of it, so one non-blocking call is still correct; Leader() keeps the backoff for callers that must wait. * master: let the leader admit a master that starts with no raft state Neither raft implementation lets a server outside the configuration campaign: goraft's promotable() requires a non-empty log, and hashicorp rejects vote requests from a candidate that is not in its configuration. A master that comes up with fresh state therefore cannot elect itself in — the leader has to pull it in. Nothing did. The peer list is static, rendered from the replica count, so scaling it up leaves the sitting leader running the old list with no idea the new masters exist. Under goraft they wait forever. Under hashicorp they are worse off: each bootstraps a cluster of its own from the new list, and two of them form a quorum next to the live leader, with their own TopologyId. That is the split brain SetTopologyId kills a master over. Admit the peer where it registers instead. Only the leader gets past the IsLeader check in KeepConnected, and a joining master's client lands there, so that is the moment it joins. The broadcast OnPeerUpdate rides on is not enough on its own: it only reaches masters already connected, which is why a leader that came up first missed both newcomers. RaftAddServer grew a goraft branch on the way, so cluster.raft.add stops silently doing nothing on the default raft, and RaftRemoveServer with it. Bootstrapping is now one call for both implementations, made only after the peers confirm nobody has a leader, and retried until this master is in rather than checked once and dropped. * master: do not evict a peer that is still in -peers The hashicorp leader drops a master from the raft configuration as soon as it stops answering pings. A master that is merely restarting answers nothing, so an ordinary bounce shrinks the quorum behind the operator's back — and then races its own return: the master comes back, registers, gets re-admitted, and the eviction lands after it. A randomized start/stop walk lands on it. Two of three masters running, the leader evicts the one that just went down, the restart re-adds it, the removal commits late and takes the leader's own leadership with it. What is left is a two-server configuration whose other half is down, and a running master that nobody will ask for a vote — no quorum, no way back until the third master returns. -peers is what declares membership. updatePeers already reconciles the configuration against it on every leadership change, and an operator who really means to drop a master can say so with cluster.raft.remove, so keep the eviction for masters that are no longer listed at all. * test: bounce masters at random and hold the election to it Twelve rounds of stopping or starting a random master, on both raft implementations, checking the two things an election must never get wrong: two masters claiming leadership at once, and a quorum that comes back without agreeing on one. The cluster's identity has to survive the whole walk, since a master that re-mints a TopologyId is the split brain SetTopologyId kills its peers over. The seed is random and logged, so a failure names the walk that reproduces it. Below a quorum the walk moves straight on. A master that has lost its quorum cannot commit anything, and goraft only checks whether it still has one on an election-timeout ticker, after its peers have been quiet for a full timeout — measured taking over 30 seconds to step down. That direction belongs to TestTwoMastersDownAndRestart, which was giving it ten seconds and would have started failing on a slower machine; it now waits on that behaviour explicitly rather than sleeping twice and hoping. WaitForTopologyId returns the id it waited for. Reading it separately raced the leader applying the raft entry that carries it, which shows up as an empty id right after an election rather than as a wrong one. |
||
|
Chris Lu
|
0c95137528 |
filer: stop aggregated metadata subscribers from spinning on a peer watermark hold (#10863)
* fix(filer): stop logging a held aggregated read as an error An aggregated subscriber may not read past the peers' low-watermark, and it stops at the first entry beyond it by returning a sentinel from the read callback. LoopProcessLogData logs every callback error, so on a cluster that keeps writing - where there is almost always an entry newer than the watermark - every read wrote an ERROR line naming the entry it stopped at, thousands per minute per filer. Mark the stop as control flow: an error wrapping StopReadingError is handed back to the caller unlogged, and the held-read sentinel wraps it. * fix(filer): release an aggregated watermark hold on peer progress A held read waited on the aggregated buffer's data channel, which the next write signalled - but a write cannot release a hold, only a peer reporting further progress can. On a cluster that keeps writing the loop therefore re-ran a whole pass per arriving event, log file listing and all, and held again on the same entry every time. Signal held readers from the meta aggregator instead, whenever a low-watermark rises: a peer reporting, or one dropped past its removal grace. The retry interval stays as the backstop for what no watermark covers. Count the holds so a parked subscriber stays visible. * fix(filer): floor how often an aggregated watermark hold releases Peers advance their delivery watermark on every event they stream, so releasing a hold on every advance is the same pass-per-event storm as releasing on every write, just without the log lines - and each pass lists a day of log files. Floor the release at 20ms. Advances inside the floor collapse into one release, which then delivers everything they covered. * fix(filer): pace a peer's delivery claim by what its subscribers hold at A filer's local metadata stream carries an idle heartbeat to its peer aggregators, and each peer turns it into that filer's delivery low-watermark. Aggregated subscribers hold at the minimum across peers, so a filer quiet enough to fall back on the heartbeat parked every subscriber in the cluster up to a keepalive interval - 5 seconds - behind live writes. With nine filers, most of them quiet at any moment, the minimum sat there permanently. Pace that heartbeat at 200ms once the filer has peers. It stays a keepalive, at the keepalive interval, for a filer with none. * fix(filer): wake each aggregated hold on its own watermark A persisted-log read is held by what the peers have flushed, an in-memory read by what they have delivered, but both parked on one channel closed whenever either minimum rose. Peers advance their delivery watermark on every event they stream, so a flush-held reader woke at the coalescing floor to re-list a day of log files and park again on the same entry - the storm this set out to fix, in the one place asymmetric peer progress still reached. Signal the two separately and park each read on the one that bounds it. |
||
|
Chris Lu
|
0dfaa103d0 |
test: take a table through its whole life, for Iceberg and Lance (#10862)
* lance worker: share the integration tests' scaffolding The recorder that keeps what a handler sent, the config builder and the storage-option fallback all lived inside compaction.rs, so a second test binary would have had to copy them. They move to tests/common. The fallback now reads AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_ENDPOINT_URL from the environment, defaulting to what it used before. A harness can then point these tests at a gateway that checks what it is given rather than one that accepts anything. * lance worker: maintain one named table, for a harness to drive Compacts and cleans up whatever WEED_LANCE_TABLE names, through the handlers' own detect-then-execute path: a proposal the worker would not have made is not one worth running. The existing tests seed the tables they check. This one deliberately does not, so a harness that has already written a table and knows what is in it can have the real handlers maintain it and then read it back. * test: take a table through its whole life, for Iceberg and Lance Created in the catalog, filled by a real client, maintained by the worker, read again, dropped. The step nothing was checking is the read after maintenance: compaction once rewrote every dictionary-encoded column onto a single value and shipped, because the maintenance tests were thorough about sequence numbers, manifest entries and metadata versions and none of them opened the parquet file the worker had just written. So the assertion is a tally - row count, the cardinality of each dictionary-encoded column, and an md5 over whole rows - taken before maintenance and again after, required to be equal. The cardinalities name the failure that happened; the digest catches a rewrite that keeps every column's cardinality and hands the values to the wrong rows. A compaction that merged nothing fails rather than passes, or the read afterwards is checking a file the worker never wrote. The Iceberg half runs two clients. DuckDB is the one the corruption was reported against and the only one here that writes the deprecated PLAIN_DICTIONARY encoding, which parquet-go normalizes away on write, so a Go writer cannot produce it. PyIceberg writes the modern spelling. Pinning parquet-go back to v0.30.1 fails the DuckDB half and passes the PyIceberg one, which is why both are here. Lance maintenance lives in the Rust worker, so it runs there where cargo is installed and through the two lance calls those handlers wrap where it is not. WEED_LANCE_MAINTENANCE picks one instead of letting the test guess. * ci: run the table lifecycle tests CI maintains the Lance table through the lance library rather than the worker: a cold build of the lance crate costs more than the glue it would be checking, and the worker's own tests cover its handlers. The suite drives the Iceberg maintenance worker, so a change to it now triggers this workflow too. * test: let the lifecycle harness fail instead of skipping Setup failures all exited zero, so a cluster that would not come up, or a port allocation that lost, reported a green run for code nothing had executed. That is the failure mode this whole directory exists to close, and it was in the harness itself. Only a checkout without a weed binary skips now, and it runs the tests so each one says so rather than the package quietly passing. Everything else fails. The filer existence probe gets a deadline while I am here: it ran without one, so an unresponsive filer would hang the suite past every timeout the clients have. * test: make the lifecycle checks check what they claim to Three of them could pass without having looked. The DuckDB skip matched "syntax error", "not implemented" and "Failed to load" anywhere in the output, in any phase. A parse error in the SQL this test generates, or a refusal from our own catalog, would have taken the only coverage of the PLAIN_DICTIONARY encoding out of CI and left it green. It now matches the extension failing to install, and only in the phase that installs it. Everything past LOAD is ours and fails. The digests covered id, category and value. Compaction rewrites the whole row, so a defect confined to ts, or to a Lance vector, changed nothing either side of maintenance. Every persisted column goes in now, ts as microseconds so no timezone sits between the two runs. The Lance drop check caught every exception as proof the dataset was gone. pylance turns credential and transport failures into the same ValueError, so it only accepts the message that means not found. * docs: say up front which maintenance path the Lance half takes The opening summary said the worker maintains both tables. It maintains the Iceberg one always and the Lance one only where cargo is installed, which is not what CI does. |
||
|
Chris Lu
|
3bd218e030 |
volume: cut idle memory at high volume counts (#10861)
* volume: start a volume's batch write worker on first use Mounting a volume started a goroutine parked on a 128-slot channel, plus the 128-entry batch slice it had already allocated. That is around 6.7KB per volume the server pays whether or not the volume ever takes a write: 7231 bytes per mounted volume, of which 4101 is goroutine stack. Only a write that asks for fsync ever reaches the worker, and a remote-tiered or read-only volume never can. Create the channel and its goroutine on the first such request instead, and let a write arriving after Destroy fall back to the inline path rather than queue onto a worker that has gone. Measured over 20000 mounted volumes: 7231 -> 1269 bytes each. * volume: update the heartbeat report state in place Every heartbeat built a second map of what it was about to tell the master, holding a freshly allocated short information message per volume, then swapped it in over the old one -- and computed departures through a third map of the live volume ids. A server holding 2M volumes rebuilt all three every VolumePulsePeriod for a report that usually says nothing. Number the heartbeats instead and mark the entry already held with the pass that found the copy, so a quiet volume costs a map lookup and no allocation. Departures are the entries a pass did not mark; the live-id map is now built only when there are some, sized to them. Measured over 10000 mounted volumes: 436 -> 196 bytes allocated per volume per heartbeat. * volume: fill one volume information message per heartbeat, not per volume The heartbeat built a message for every volume held so it could hash it, then dropped all but the few it had something to say about. At 2M volumes that is 2M messages allocated every VolumePulsePeriod to send almost none of them. Fill a message the caller supplies instead, and replace it only when the heartbeat keeps it, so a server with nothing to report fills the same one all the way through. Measured over 10000 mounted volumes: 196 -> 4 bytes allocated per volume per heartbeat, and a heartbeat runs a third faster. * volume: drop the per-volume trace from the heartbeat's status read glog.V(4).Infof evaluates its arguments whether or not the verbosity is on, so every volume boxed its id into a fresh interface slice on every heartbeat: 759 of the 773 allocations a 1000-volume heartbeat made, for a line that at this scale would print millions of unreadable rows. Measured over 1000 mounted volumes: 4776 -> 1792 bytes and 759 -> 14 allocations per heartbeat, which no longer grows with the volume count. * seaweed-volume: mirror the in-place heartbeat report state Same change as the Go volume server: number the heartbeats and mark the entry already held with the pass that found the copy, instead of building a second map of hashes and swapping it in. The volume snapshot must leave the reporting state as it found it, so it keeps asking through changed() while a real heartbeat marks through record(). * volume: refuse writes to a closed volume instead of dereferencing nil Close and Destroy leave the needle map and data backend nil, but a caller that already holds the volume can still reach the write path, where both are used unguarded: a write racing a volume deletion took the server down. syncDelete has always checked; syncWrite and the batch worker had not. Reachable before this series and now also from the inline fallback a durable write takes when the worker has gone. * seaweed-volume: guard the report state with one mutex, as Go does The full-list flag and the generation that answers it have to move together. Split across separate atomics they cannot: a request landing between begin's two reads returns full == false with the generation it just raised, and one landing between commit's read and its clear is marked answered by a heartbeat that carried no list. Either way the resend is dropped. Neither is reachable today -- every caller reaches this through the store's RwLock, the flag setters under a read lock and the heartbeat build under a write lock, so they cannot interleave. The type should not depend on that being true two files away, and Go holds a single mutex over exactly these fields. * test: build the servers under test to match the harness's offset size The mixed Go/Rust suites run both servers against one dataset, so both have to agree on the offset width. They did not: the harness built Go with no tags, 4-byte offsets, while the Rust crate defaults to its 5bytes feature, and the Rust server then refused the .vif the Go server had just written -- "bytes_offset mismatch: found 4, expected 5". Build each side to match the offset size the test binary itself was compiled with, so a plain `go test` and one with -tags 5BytesOffset both get a matched pair. |
||
|
Chris Lu
|
6faa9d20e8 |
iceberg: stop compaction from corrupting dictionary-encoded columns (#10857)
* deps: upgrade parquet-go to v0.32.0 Iceberg compaction writes the merged file with the schema of its first input, encodings included. parquet-go before v0.31.0 took the deprecated PLAIN_DICTIONARY encoding that DuckDB writes at face value and encoded those pages as plain int32 indices, but the spec gives PLAIN_DICTIONARY the same bit-width-prefixed RLE layout as RLE_DICTIONARY. Every dictionary-encoded column in a compacted file then decoded onto a single dictionary entry, and anything past one page failed to decode at all. * iceberg: cover compaction of dictionary-encoded input The fixture is a DuckDB-written file, so it carries the PLAIN_DICTIONARY encoding a Go writer will not produce. * iceberg: tally whole rows in the dictionary merge test Counting each column on its own passes a merge that remaps names while leaving their cardinality intact. |
||
|
Chris Lu
|
813c439af6 |
admin: regenerate the gzipped static mirror (#10859)
The toast and modal changes edited static/js/ without rerunning gen_static_gz.go, so the embedded assets served to browsers still carry the old scripts and TestStaticGzMirror fails on master. |
||
|
Chris Lu
|
bd34565e56 |
admin: keep the copy confirmation in front of the access key modal (#10856)
* admin: raise nested modals above the ones already open Bootstrap gives every modal and every backdrop the same z-index, so a modal opened while another is showing paints behind it and its buttons cannot be clicked. Viewing an access key secret and then copying a field left the confirmation stuck behind the details modal with no way to dismiss it. Give each nested modal, and the backdrop Bootstrap creates for it, a z-index above what is already on screen, and put back the scroll lock that Bootstrap drops as soon as any one of them closes. * admin: confirm clipboard copies with a toast The access key details modal offers three copy buttons, and each one raised a modal that had to be dismissed before the next copy. Confirm with a toast instead, so the credentials stay in view and nothing has to be clicked away. |
||
|
Chris Lu
|
8a532cc0cf |
mini: state the format of a -tableBucket, do not infer it (#10851)
* mini: state the format of a -tableBucket, do not infer it
A table bucket holds one format and that format decides which catalog can
serve it, but the flag only took names. The format came from
miniTableBucketFormat(): Iceberg whenever its port was up, Lance only when
it was not. So -tableBucket=vectors on a default mini quietly made an
ICEBERG bucket that the Lance namespace then refused every table in, and
the only way to get a Lance one was -s3.port.iceberg=0, which buys it by
deleting the other catalog. One flag, two meanings, decided by an unrelated
port.
Each entry is now name[:FORMAT], unsuffixed meaning ICEBERG as before:
weed mini -tableBucket=warehouse,vectors:LANCE
Both catalogs stay up and both buckets are reachable. A name whose format
has no endpoint here is skipped with a warning rather than created out of
reach, and the Iceberg-only S3_TABLE_BUCKET default-routing hint gets the
Iceberg names alone, without their suffixes.
* mini: do not reuse a table bucket that holds another format
CreateTableBucket answers BucketAlreadyExists on the name alone, so
-tableBucket=vectors against a bucket created as LANCE logged "already
exists" and moved on, and the Iceberg default-warehouse hint then pointed
at it. Every table create against that catalog fails with "table bucket
vectors holds LANCE tables", far from the flag that chose it.
ensureMiniTableBuckets now reads the format of a bucket it did not create,
warns when it is not the one asked for, and returns only the buckets that
hold what was requested. S3_TABLE_BUCKET is seeded from that list, so an
unprefixed Iceberg request falls back to its own default rather than
committing into a Lance bucket. A bucket predating declared formats reports
an empty one and still accepts either.
* mini: normalize S3_TABLE_BUCKET whichever way the spec arrived
The rewrite that keeps Lance names out of the Iceberg default warehouse only
ran when the flag supplied the spec. Set the variable directly, as the docker
quickstart does, and it reached the catalog untouched: S3_TABLE_BUCKET=
vectors:LANCE,warehouse made the unprefixed default the literal string
"vectors:LANCE", a bucket no lookup finds, while warehouse sat behind it.
The variable is both mini's input and the catalog's routing hint, so it is
now always rewritten from the buckets that came back holding Iceberg tables,
and unset when there are none rather than left pointing somewhere stale.
* mini: reuse a table bucket only when its format reads back
An ordinary S3 bucket wearing the name answers CreateTableBucket with the
same BucketAlreadyExists as a table bucket does, and the format lookup that
follows returned "" for a failed read exactly as it does for a bucket
predating declared formats. So -bucket=data -tableBucket=data reported
nothing and published data as the Iceberg default warehouse, where every
unprefixed request 404s on a bucket that is not a catalog.
The lookup now returns its error, and only a bucket that reads back as the
format asked for is reused. Anything else is left alone with a warning
naming why, rather than routed to and discovered later.
|
||
|
Chris Lu
|
83753ccdad |
test: drive the Lance namespace with LanceDB (#10850)
* test: drive the Lance namespace with LanceDB
The Iceberg catalog is checked against Spark, Trino, ClickHouse, Doris,
Dremio and RisingWave. The Lance one had only its own reference client,
which is the same thing as checking it against ourselves.
LanceDB connects with connect_namespace("rest", ...), which speaks the
routes this catalog implements, so the suite exercises the protocol rather
than our idea of it: list the catalog, open a table through it, read the
schema, run a vector search and a filtered scan, create a table, and read
the same dataset straight off its URI with no catalog at all.
table_names -> ['lancedb-p0guidmm$ml$embeddings']
open_table -> 64 rows
search -> [1, 0, 2]
create_table -> 4 rows, listed by the catalog
direct read without the catalog -> 64 rows
Seeding is pylance, because the namespace records where a table lives and
does not carry its data. That split is the design rather than a limit of
the test.
One interop note the test encodes: a gateway without STS vends
storage_options carrying an endpoint and a region but no credentials, and
LanceDB uses what the namespace vends on some paths. The container gets
credentials in its environment as well, which is what a deployment without
STS would do.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: pin the LanceDB client, and index before searching
Three from review.
The client's dependencies were unpinned, so an unrelated upstream release
could change what an old commit reproduces. Pinned to the versions this
suite was verified against; the client is as much the thing under test as
the server.
The search was called ANN and was not: without an index LanceDB scans.
The test now builds an IVF_PQ index over 1024 rows first, which is worth
more than the wording fix - an index writes into a directory of the table
that the S3 door has to admit, and that guard has refused a Lance
directory before. It builds, covers all 1024 rows, and searches.
The assertion moved with it. Demanding the exact nearest neighbour was
right for a brute-force scan and wrong for a quantized index, which
answered 0 as readily as 1; both are correct, so the check is now the
neighbourhood.
And the pushdown check accepted any failure. It now requires the refusal
to be the catalog's Unsupported and requires that nothing was left behind,
or, when the client falls back, that the table is complete.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
|
||
|
Chris Lu
|
6f3b5a4f4d |
metrics: graph the plugin workers (#10849)
Nothing displayed the worker metrics, and the panels that look like they did are about something else: Workers Connected, Worker Slots and Worker Events in the Admin / Maintenance row read SeaweedFS_admin_*, which the older maintenance queue feeds. A cluster running plugin workers - Go or Rust - reads zero there while they are connected and busy. A Plugin Workers row graphs what the workers themselves publish: how many are connected, jobs and their failures, detection and proposal rates, job duration, slot usage, stream events, and what the Lance jobs reclaimed. The panel worth having is Objects Seen vs Skipped, since a sweep with nothing to do and a sweep that could read nothing report the same number of proposals. Also a commented scrape target in the sample Prometheus config. It is 9328 rather than 9327: the sample compose already gives 9327 to the S3 gateway, so the port the worker's own usage text suggests collides with it on a single host. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm |
||
|
Chris Lu
|
f56a7a1557 |
seaweed-worker: serve health, readiness and metrics (#10848)
* seaweed-worker: serve health, readiness and metrics A Rust worker had no surface of its own. If it wedged, the only signals were its stdout and whatever admin could infer from a stream that had gone quiet; nothing could be scraped and nothing could be alerted on. --metrics-port serves /health, /ready and /metrics, the same three the Go worker serves under -metricsPort, so one scrape config covers workers in either language. Off by default, loopback unless --metrics-ip says otherwise, since the endpoint is unauthenticated. Names follow the Go convention, SeaweedFS_worker_*. The counters live in core and are raised where the stream already knows what happened - connect, close, detection, execution, preview - so a worker for another format gets them without writing any of this. Slots are published from the heartbeat that already computes them, so a scrape and the admin UI cannot disagree. The pair worth having is objects_seen_total and objects_skipped_total. A sweep that proposed nothing because there was nothing to do and a sweep that proposed nothing because it could not read anything are the same number of proposals; they are not the same event, and until now only a log line told them apart. The Lance jobs add what they reclaimed - fragments, rows brought under an index, versions, bytes - on the same registry, so one endpoint serves both. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * seaweed-worker: fix the metrics address, the count, and a dead field Three from review. --metrics-ip ::1 failed at startup: the address was built by joining host and port with a colon, and "::1:9327" is not an address. It is parsed as a host and combined with SocketAddr::new now, so an IPv6 literal works, with or without the brackets an operator will reasonably type after seeing one in a URL. proposals_total counted before the send rather than after, so a stream that closed mid-sweep left the counter claiming proposals admin never received. And MeteredSender carried a Metrics clone and a job type it never read, kept alive by two statements that existed only to silence the warning about them. Everything is recorded by the caller, so both are gone. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm |
||
|
Chris Lu
|
fc97f8ea8f |
mount: index directory state by path (#10827)
Every directory-state lookup went through path2inode, the map that holds one full path per inode in the table, and then through dirStates. Directories now carry their own path and are indexed by it directly. There are orders of magnitude fewer directories than files, so this map stays small whatever the mount holds, and it is what a file needs before it can stop carrying a full path of its own: a child's path is its parent's plus its name. No behavior change - the two indexes are asserted to agree. |
||
|
Chris Lu
|
8c7d714d5e |
Lance catalog, and a Rust plugin worker to maintain it (#10841)
* iceberg: skip tables the maintenance worker does not own A Lance dataset registered through the Lance namespace's Iceberg REST adapter arrives as an Iceberg table with a placeholder schema and table_type=lance, and keeps its fragments under data/ - the same subdirectory the orphan cleaner walks. Every fragment is unreferenced by the Iceberg metadata, so a maintenance pass deletes the dataset. Views share the entry shape and were only skipped because parsing their metadata happened to fail first. Gate the scan and the execution path on the entry actually being an Iceberg table. Maintenance is off by default, so this was latent rather than live. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tables: let a table declare a format the catalog does not interpret CreateTable accepted ICEBERG and nothing else. A Lance table has no metadata file for the catalog to maintain - the entry records a name and the dataset root, and the client owns everything under it - so accept LANCE, and carry the declared format on the entry instead of hardcoding it back on the way out. ListTables now reports format and metadataLocation, so listing a catalog that holds both kinds takes one pass rather than a GetTable per row. AWS omits both fields; adding them is additive. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tables: move the in-memory filer into its own package The Lance namespace tests need the same harness, and copying it would leave two of them to keep in step. Extracted as it was, plus the two fidelity gaps that only surface once a paginating caller uses it: ListEntries ignored startFromFileName and limit, so a caller that paginates re-read the first page until it hit its own cap and reported the same entry over and over, and GetFilerConfiguration was missing, which CreateTableBucket needs to resolve the buckets directory. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: serve the Lance Namespace REST spec A second catalog surface beside the Iceberg one, over the same table buckets: the namespace and table metadata operations, the $-delimited identifier codec, the spec's numeric error model, the directory-catalog marker files, and storage_options vending through the STS path the Iceberg catalog already uses. Listens on -port.lance, 9101 by default, and inherits ARNs, policies and tags from the storage layer, so a Lance table needs no second permission model. Identifiers map bucket / namespace / table onto the three levels Lance clients already use, which is why there is no warehouse selector to invent. The data plane needs Lance format support that does not exist in Go and answers with the spec's Unsupported code rather than a bare 404. Two things it deliberately will not do: create a table bucket as a side effect of creating a namespace inside one, since a bucket carries its own policy and lifecycle, and resolve an Iceberg table's location for a Lance client, which would hand it a table another engine owns. The design note this follows is in design-lance-catalog.md, including the .lance directory suffix it proposed and this does not implement. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * mini: give the Lance port the same treatment as the Iceberg one The flag was registered but nothing else knew about it, so mini would start the server without reserving its port, waiting for it, or saying where it is. Adds it to the startup service list, the conflict resolver, the gRPC allocator's reserved set, the readiness wait, the stop reporting and the banner. The admin server still takes only the Iceberg port, because there is no Lance page for it to link to. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: stop deregister and repoint from deleting the dataset Deregistering preserves data by definition, and this did the opposite: the catalog entry is the dataset directory, so DeleteTable took the files with it. Registering over an existing name had the same shape, destroying the dataset the name used to hold. Found by driving the running server rather than the in-memory filer, where both looked like success because the table did stop being listed. Deregistering is now a state on the entry - the marker file hides it, and declaring or registering the name again brings it back. Repointing a name at another dataset is an UpdateTable against the version token, so neither dataset loses files. Drop is left alone; it is the operation that does remove data. The storage endpoint now falls back to the advertised -ip where the Iceberg derivation gives up. An Iceberg client brings its own s3.endpoint and advertising the wrong one hijacks it, but storage_options is the only place a Lance client learns where the store is, and without it object_store quietly talks to real AWS. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tables: refuse to create a table over one of another format Creating a table that already exists is idempotent, and that path returned the existing table without looking at its format. A Lance declare over an Iceberg table answered 200 and handed back a directory Iceberg owns, so the client would write its dataset on top. The view check immediately above it already guards the same class of collision. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tables: let a table bucket hold a format other than Iceberg The S3 door validated every object written into a table bucket against Iceberg's file layout, so a Lance client could not write its dataset at all: it got 403 on data/*.lance, on _versions/, and on the _transactions/ directory it turned out to write as well. Table buckets were only neutral containers by intention; in practice they were Iceberg-shaped and enforced as such. The allowed set is now the union of what the supported formats write, because the validator runs where the table's format is not in hand. Underscore-prefixed directories are treated as belonging to the format, since enumerating them means guessing at the next one - _transactions is exactly the one this missed - and their contents are checked only for traversal. Iceberg writes none of them, so it loses nothing. Marker files at the table root are admitted too, which the namespace/table/dir/file shape had rejected as too shallow. Describe also honours the request-body spellings of with_table_uri, load_detailed_metadata and check_declared. The spec puts them in the query string, but real clients send both. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: record what the implementation found The table bucket being an Iceberg-shaped container, enforced at the S3 door, was the premise this design never questioned and the one that had to change before anything worked end to end. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * iceberg: prove the data loss the foreign-format guard prevents The guard landed with a unit test for the predicate and nothing showing what it saves. These seed what the Lance namespace's Iceberg REST adapter actually leaves behind - an Iceberg table with a placeholder schema and table_type=lance whose directory holds a Lance dataset - and assert both halves: orphan collection does flag the dataset's fragments, because the Iceberg metadata beside them references nothing, and the scan never reaches the table. An ordinary Iceberg table in the same shape is still scanned, so the guard is not just skipping everything. Confirmed against a running gateway first: our Iceberg catalog accepts the adapter's registration, and a real Lance client then writes a dataset into that table's location. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tablestest: make the in-memory filer safe to race against Two gaps that only matter once a test drives concurrent writers, which is what an exclusive create has to be tested with: the entry map had no lock, and CreateEntry ignored O_EXCL entirely, so both writers of the same name would have won and the test would have passed while proving nothing. The BeforeUpdate hook runs before the lock is taken. Its whole purpose is to land a competing write in a handler's read-to-write window, and that write needs the lock the hook would otherwise be holding. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: make the namespace an external manifest store Lance commits a version by writing _versions/{v}.manifest with put-if-not-exists. The S3 layer in front of this same filer evaluates If-None-Match by looking the entry up and then writing without a precondition, so two writers can both pass the check and one commit is lost. The filer itself has the primitive: CreateEntry with o_excl. Adds the four version operations a Lance client actually calls - create, list, describe and batch-delete - recording one entry per version under _lance_versions/, and advertises managed_versioning so the client routes its commits here. Reserving a version is the exclusive create, so exactly one of several racing writers wins and the rest rebase. Off by default, behind -lance.managedVersioning. Turning it on moves where a table's version history lives, and a reader that does not come through this namespace no longer sees all of it; that is the operator's call, not a default. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: record what managed versioning does and does not reach The first commit through a namespace-backed store works and is recorded the way the protocol specifies. Later commits do not, because lance 4.0.0 refuses put_if_exists on that path in its own code, so the feature is capped upstream rather than here. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * test: integration tests for the Lance namespace Everything this surface got wrong so far - a deregister that deleted the dataset, an S3 door that refused every Lance file, a version reservation that could not actually be exclusive - passed against an in-memory filer first. So these run against a live gateway, and where the claim is about data they check storage rather than visibility. Five Go tests on the shared harness: namespace and table lifecycle including that deregister keeps the bytes and drop removes them, that a Lance client cannot resolve or declare over an Iceberg table, that a Lance dataset's files get past the table-bucket layout guard while junk still does not, and that eight writers racing for one version produce exactly one winner. One Docker-gated test drives the real Lance client, which is the only way to check that the location and storage_options the namespace vends are between them enough to write and read a dataset. It overrides the endpoint with the container's view of the same gateway, because the shared harness binds a wildcard address and so vends none. The harness gains a Lance port and turns managed versioning on; the flag touches nothing outside that surface. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3tables: a directory with no namespace metadata is a missing namespace Three callers resolved a namespace by reading its metadata attribute and each tested only for a missing entry, so a directory that carried no metadata came back as an internal error saying "attribute not found". Creating a table under a namespace that does not exist answered 500. Collapses the three copies into one helper that reports both conditions as absent, which is what they are: a directory without namespace metadata is not a namespace. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * iceberg: stop reporting storage-layer refusals as server faults writeManagerError recognised a missing table bucket and sent everything else to 500, so a missing namespace, a duplicate name and a commit conflict all reached the client as InternalServerError with nothing to act on. Creating a table in a namespace that does not exist is the case that turned up: 500 where the spec wants 404 NoSuchNamespaceException. Maps the storage error types onto the exception names this package already uses, and keeps the existing bucket message, which explains how to select a table bucket. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * iceberg: skip a foreign-format table by name, not by failing to parse it A table the namespace created as LANCE carries no Iceberg metadata, so the worker skipped it only because the parse failed, and logged that as damaged metadata. The catalog records the format on the entry and this never read it. Reading it turns an accident into a decision, and separates a mixed catalog from a corrupt one in the logs. The property check beside it still covers the other shape: a real Iceberg table wearing table_type=lance, which is what the Lance namespace's Iceberg REST adapter writes. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: answer whether a Lance table needs maintenance It does, and index optimization has no Iceberg equivalent: rows written after an index was built are not covered by it, so a vector search quietly misses them. None of the three jobs can run in the Go worker, and there is no useful subset, because deciding what an old version still references means parsing Lance manifests. Version cleanup at least has an answer that needs nothing from us - Lance can enable it on the dataset itself. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: the Lance maintenance worker is a plugin worker, in Rust Framing it as a sidecar was wrong. plugin.proto already defines a language-agnostic gRPC contract for external maintenance workers, and "weed worker -admin=..." is the Go reference implementation of it from outside the admin process. seaweed-volume already compiles protos out of weed/pb with tonic_build, so a Lance worker is that build plus plugin.proto and the lance crate. Scheduling, retries, dedupe, progress and the admin settings page all come from the protocol: a worker that answers RequestConfigSchema with a descriptor gets its configuration form rendered without a line of Go. The data plane is the part that genuinely does need a process answering HTTP, and this had the two conflated. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * seaweed-worker: Rust plugin worker workspace, with Lance as the first one plugin.proto is language-agnostic and the Rust toolchain was already in the tree, so a Lance maintenance worker needs no new integration surface: core is the contract and nothing else, and a worker crate beside it supplies handlers and a binary. A second worker is a new member here rather than a fork of the protocol, which is why this is seaweed-worker and not seaweed-lance-worker. Verified against a running admin: it connects, is accepted, and admin prefetches descriptors for lance_compact, lance_optimize_indices and lance_cleanup_versions, so their settings pages render from the Rust side without a line of Go. The stream stays up across heartbeats. The job bodies are stubs that report failure. Doing the work means adding the lance crate and opening the dataset, and claiming success before that would be worse than saying so. Two things running it caught that reading the proto did not: the admin address has to be converted to the gRPC port the way pb.ServerToGrpcAddress does, or the dial fails as an h2 frame error; and the generated field names differ from the Go ones in several places, so JobCompleted carries success rather than a state enum. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: implement compaction Detection lists tables from the namespace, opens each one, and proposes a job for any with more fragments than the policy allows; opening a dataset reads its manifest and not its data, so a sweep stays cheap. Execution re-resolves the table rather than trusting what detection saw - it may have been repointed, and the vended credentials expire - then compacts and reports the fragment counts either side. Verified against a live gateway: a twelve-fragment dataset became one fragment with all twelve rows intact. The test drives the handler directly and skips unless WEED_LANCE_NAMESPACE names a namespace, the way the Go integration tests skip without Docker. Running it turned up a gap the design had not: a gateway without STS vends no credentials at all, so the worker could not open anything and detection quietly proposed nothing. --access-key/--secret-key are the fallback, and whatever the namespace vends still wins over them. Two API assumptions did not survive contact either. Datasets open through DatasetBuilder::with_storage_options, not ReadParams, and lance 10's ObjectStoreParams has no storage_options field at all. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: implement index optimization and version cleanup Index optimization is the job with no Iceberg equivalent: rows appended after an index was built are invisible to a search of it until this runs. Detection reads num_unindexed_rows from each index's statistics and proposes a table once more rows sit outside its indices than the budget allows; a table with no indices is skipped, which is different from one whose indices have fallen behind. Cleanup applies a retention window, refusing rather than silently dropping a tagged version, and leaving unverified files alone because they may belong to a commit still in flight. Both verified against a live gateway: 512 uncovered rows became 0, and a fourteen-version table lost its old ones. Each test now seeds what it needs, including building an IVF_PQ index and appending rows outside it. The first version of these depended on state a script had left, so the second run found the work already done and asserted nothing - a test that passes by doing nothing is worse than no test. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: answer an empty catalog with an empty list, not null ListAllTables built its result from a nil slice, so a namespace holding no tables answered {"tables":null} on a field the spec marks required. A generated client may decode that differently from an empty list. Found running the namespace on a dev box, where the catalog was empty. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: give Lance maintenance its own scheduler lane Lane assignment is a hardcoded map, so the three lance_* job types fell through to the default lane. That lane serialises its work under the cluster admin lock because volume management shares global state, which would queue a table's compaction behind volume balancing for no reason - Iceberg has its own lock-free lane for exactly this. Adds the lane, maps the three job types to it, and puts it in the sidebar beside Iceberg and Lifecycle. The lane routes were already generic, so only the nav was hand-written. The lane-coverage test spelled out the three known lanes, so a fourth failed it. It now checks against AllLanes(), which is the property it was reaching for and does not need editing next time. Found by connecting the Rust worker to a real admin: it registered fine and its job types were known, but they were filed under "default" and had no page. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: log what detection saw "Detection proposed nothing" and "the worker could not read the table" look identical from the admin side, and the second is what a missing credential produces. One line per table separates them. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: fix a leaked heartbeat and a silent reconnect loop spawn_heartbeat returned a handle to an empty task rather than the ticker it had just spawned, so aborting it aborted nothing and every reconnect left another heartbeat running against a dead channel. A stream that admin closes cleanly is not an error, but reconnecting in silence hides why. Two workers sharing an id evict each other forever and the log shows nothing but a login every five seconds - which is exactly how this presented on a dev box, and it took a look at the admin's own log to see it. The message now names the id to check. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: a namespace cannot be created without its parent Storage keeps a namespace's parts flattened, so creating "a.b" with no "a" was accepted and left an intermediate that only existed inside a name. Listing derives child names by slicing those parts, so it reported "a", while describe and exists on "a" both answered 404 - a client walking the tree got a 404 on something the listing had just handed it. The spec asks for NamespaceNotFound when the parent is missing, which is also what keeps listing and describe telling the same story. Namespaces created through the S3 Tables API still bypass this, so listing keeps deriving intermediates rather than hiding whatever is already there. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: say why a non-Iceberg table shows no schema The table pages read Iceberg metadata for schema and snapshots, and a Lance table has none, so both panels rendered "No schema available" - which reads as an empty table rather than a table this page cannot describe. The dataset behind the one that prompted this holds 1024 rows. The format is already on the entry and shown two rows above, so the empty states now use it: the catalog records where a LANCE table lives, not what is in it. Reading the schema for real needs Lance format code, which is the same wall as the data plane. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * seaweed-worker: run rustfmt over the workspace Committed the crates unformatted, so `cargo fmt --all --check` failed on files nothing had touched since. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * plugin: let a worker report what it saw about an object Admin cannot read a Lance table: it knows where the dataset lives and nothing else, so the details page had a location and two empty panels. The worker already opens every dataset during detection to decide whether it needs compacting, so it knows the schema, the row count and the fragment count at that moment. It just had no way to say so. Add a WorkerObservations body to the worker stream. Admin caches the last observation per object and serves it back, timestamped, for display; nothing schedules from it. The Lance compaction sweep reports what it opened, and the S3 Tables details page fills its schema panel from the cache when it has no metadata of its own, badged with when the worker looked and which worker it was. Nothing about this is Lance-specific past the reporting side, which is the point: any format admin cannot parse can describe itself the same way. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: record the observation channel Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * plugin: ask a worker for sample rows of a table admin cannot read Browse Data reads an Iceberg table's Parquet files directly, so it shows real rows. For a Lance table it showed "Table has no Iceberg metadata" and an empty grid, because there is no Go Lance reader and never will be one worth maintaining. The worker has the reader. Add RequestObjectPreview / ObjectPreviewResponse to the stream, mirroring the config-schema round trip that already exists, and give the Rust worker a PreviewProvider that scans the dataset and formats the rows with Arrow's own formatter, so a vector column reads as a vector. Admin picks the worker from the observation store: whichever one last described this table is the one that can read it. Unlike an observation the rows are not cached. They are the table's data rather than a description of it, and a copy sitting in admin would be both stale and nobody's business. The page fetches on load, bounded at 200 rows and a 15 second round trip, and drops the snapshot and data-file panels that only mean something for Iceberg. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: record the preview channel Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * test: disable the lance listener when two gateways share a host * test: keep AllocatePorts away from the lance default port * s3tables: let a table bucket declare the format it holds A bucket is a catalog, and a catalog serves one protocol. Format was recorded per table, so nothing could answer "where do I point a client at this bucket" without opening a table first, and an empty bucket had no answer at all. CreateTableBucket takes an optional format, stored with the rest of the bucket metadata and returned by Get and List. Empty means ICEBERG, which is what AWS S3 Tables serves and therefore what an SDK that has never heard of the field means. CreateTable refuses a table of another format, and CreateView refuses outright in a bucket that is not Iceberg, since a view is Iceberg metadata. Buckets that already exist carry no declaration and keep accepting anything, so nothing is migrated and nothing that worked stops working. The Lance namespace declares LANCE for the buckets it creates, which is what stops one of them being described to a client as an Iceberg catalog. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: take the Lance port the way it takes the Iceberg one The UI cannot name the endpoint that serves a Lance bucket without it, and every format-aware page below needs to. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: show which format a table bucket holds The bucket list printed an Iceberg endpoint for every bucket, including ones holding Lance datasets, where that endpoint serves nothing. It was the most visible place the UI assumed one format. The list gains a Format column and its endpoint column follows the bucket's declaration. The banner names both endpoints rather than asserting everything is Iceberg, and says so only for the servers that are actually running. Create Bucket picks a format with two cards rather than a dropdown, since what matters is not the name but which clients can read the result, and the endpoint under them updates as you choose so the operator leaves the modal knowing where to point one. A bucket from before the declaration existed shows "unset" in an outline badge, explained on hover. It is a fact about the bucket's age, not a fault, so nothing nags about it. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: carry the bucket's format into the pages inside it Namespaces and tables are reached through a bucket, so both now say which catalog they belong to rather than making you go back up to find out. The tables list gains a Format column and a Rows column filled from what a worker last observed, since for a format admin cannot read that is the only row count there is; a table nothing has looked at shows a dash, not a zero. Create Table stops offering a choice the bucket has already made: in a declared bucket the format is fixed and says why, and only an undeclared one still offers both. Before this the select had exactly one option, hardcoded, which made a Lance table impossible to create from the UI at all. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: let the table page speak the table's own format Partitions and Snapshot History are Iceberg's shape. Rendering them empty for a Lance table reads as a fault; a Lance table has neither, and says so by not showing them. In their place is a Versions panel, which is what that format calls its history, carrying the worker's timestamp so it is clear the numbers are a cached look rather than something read live. The breadcrumb carries the format badge, so the page names what it is looking at before you read a panel and wonder why it is empty. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: show how to connect to either catalog, and group the two format workers The client examples on the buckets page were Iceberg's alone, so the one thing an operator wants after creating a Lance bucket - what to type to reach it - was not written down anywhere in the UI. Both formats now get a pair of snippets, and only for a server that is running. In the Workers menu, Iceberg moves below Lifecycle so it sits next to Lance: the two table-format workers together, the two cluster-wide ones above them. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * shell: create a table bucket of either format s3tables.bucket -create takes -format, so a Lance bucket can be made without going through the UI. The integration harness passes it too: its Lance tests were creating Iceberg buckets and getting away with it only because nothing checked. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * design: record that a bucket declares its format Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: drop managed versioning; the store already orders commits The namespace offered itself as an external manifest store, so that a commit could reserve a version through a real put-if-not-exists. That was designed around a gateway that no longer exists: If-None-Match: * is reduced to a filer WriteCondition and evaluated at the object's owner under its per-path lock, or under the object write lock on the fallback path. Sixteen writers racing one fresh key get a single 200 and fifteen 412s, every time. Lance needs nothing else. commit_handler_from_url hands every s3:// dataset a ConditionalPutCommitHandler, which puts with PutMode::Create, which object_store sends as If-None-Match: *. So the feature solved a problem this store does not have, while moving a table's version history out of the dataset and into the catalog - and lance could not use it past the first commit anyway, since its own namespace-backed store answers "put_if_not_exists is not supported" to the second. The version operations answer Unsupported with the rest, managed_versioning is false, and the flag is gone. In place of the reserve-once test there is one that races eight writers at the manifest key through S3, which is the path a commit actually takes. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: honour the version floor, the slot limits, and a shutdown Five findings from review, all of them things the worker claimed to do and did not. The version floor was checked when a cleanup job was proposed and ignored when it ran, so a table whose versions had aged past the retention window in between could be taken below the count the operator asked to keep. Execution now computes the floor itself and passes it as before_version; CleanupPolicy ANDs its clauses, so a version has to be both too old and below the floor to go. Both settings are clamped to the range the form offers, since Duration::hours panics on a large enough value and a negative min-versions wraps to a huge usize. Admin's shutdown was answered by returning from the stream, which the reconnect loop read as a healthy close and logged straight back in: the worker could not be stopped. serve_once now says which of the two happened. The advertised concurrency limits bounded nothing - every request spawned a task - and the heartbeat reported zero slots in use whatever was running. Both now go through semaphores sized from the limits, with the permits held for the life of the request and reported in the heartbeat. A namespace call had no timeout, so a gateway that accepted the connection and went quiet held a detection slot forever. And one table whose stats could not be read failed the whole sweep, losing the proposals for every table already scanned; it is now skipped and warned about, like a table that cannot be opened. The tests drove one shared catalog concurrently, which is why one of them asserted "no proposals at all" and passed by luck. They now take a lock and judge only their own tables. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * admin: fix the review findings on the format-aware pages The endpoint hint in Create Bucket built its HTML by concatenating the bucket name the operator is typing, so a name like <img onerror=...> ran in the admin origin as they typed it. It is built from DOM nodes now. A preview reply looked its channel up under the lock and then sent outside it, which Shutdown can close in between: a Gosched in that gap panics with "send on closed channel" every time. The send now happens under the lock. Observations were looked up by path alone, so a table dropped and remade in another format at the same path was described by the observation left behind. Lookups now have to agree on the format. Also: the Lance namespace caps a request body rather than reading whatever arrives; the details action no longer says "Iceberg" over a Lance table; mini stops advertising a catalog port when it is not running S3; a format whose server this cluster does not run cannot be picked in the modal or accepted by the API, since a bucket nothing can reach is not worth creating; and the unused catalogPortFor helper is gone. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: let the control stream use mTLS The channel was hardcoded to http://, so off loopback the stream carried preview rows and execution commands in the clear - and a cluster with grpc TLS turned on would refuse the worker outright. --tls-ca, --tls-cert and --tls-key take the same certificates the Go worker reads from the [grpc.worker] section of security.toml, and must be given together: a CA on its own would quietly mean one-way TLS, which a mutual setup rejects anyway. Without them the stream stays plaintext, which is what the Go worker also does when nothing is configured. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: answer null properties rather than an empty map The catalog does not keep a table's properties. Declare echoed the request's back and describe answered {}, both of which claim they were stored and are empty. Null says the catalog does not keep them, which is what the spec distinguishes and what is true here. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance worker: test the slot accounting The heartbeat reporting and the waiting are the two things the semaphores are for, and neither is observable from outside without catching a sweep mid-flight. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * test: fix the mixed-format catalog test, and name the binary it drives The integration suite passed locally and failed in CI on TestLanceRefusesIcebergTables. Both were right: CI builds the binary first, my tree had one from the day before, so locally the test drove a gateway with no format enforcement at all. The test itself no longer holds as written. It made a bucket, put an Iceberg table in it, and checked the Lance surface hid it - but a bucket that declares LANCE now refuses the Iceberg table outright. The invariant still matters from the other side, so it starts from an Iceberg bucket instead: Lance must not describe or list a table whose format it does not serve, and must refuse to declare one beside it. The harness now prints which weed binary it is about to run and when that was built. `make test` rebuilds first; a plain `go test` will happily drive a weeks-old binary and report a pass for code it never ran, which is exactly what happened here. Also make the row-limit conversion in the preview request explicitly bounded: CodeQL flagged the int-to-int32 conversion, and clamping by reassignment beforehand is not a form it recognises. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * lance: prove concurrent commits are kept, and preselect the only format on offer Two more from review. The commit test asserted that exactly one writer wins the conditional PUT, which is the mechanism, not the claim. The claim is that nothing is lost: the losers see the conflict, rebase and commit again. So there is now a test that has eight writers append to one dataset at once and counts the rows afterwards - all eight batches survive. That is also the sequence managed versioning could not finish, since its store refuses the second commit outright. And when Iceberg's endpoint is not running, the format picker offered two options with neither selected, so Create Bucket submitted no format at all, fell back to ICEBERG, and was refused by the guard added last round. Lance is preselected when it is the only format this cluster serves. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * Clamp the remaining worker settings, and bootstrap buckets in a served format Compaction and index optimization read their thresholds and cast straight to usize and u64, so a negative arrives as an enormous number and turns the threshold into "never": compaction and reindexing both go quiet with nothing to say. The cleanup job was fixed last round; these are the same bug. Clamped to the values that stay meaningful rather than to what the form offers - zero uncovered rows is a real setting, meaning reindex as soon as anything is not covered, so the floor there is zero and not the form's thousand. mini pre-creates the buckets named by -tableBucket, and did so without a format, which now means Iceberg. Started with the Iceberg endpoint off and the Lance one on, that left buckets nothing could reach and which refused every Lance table. It takes the format from the endpoint that is actually running, and creates nothing when neither is. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * s3: allow-unordered is a listing parameter, not an unimplemented subresource The guard that stops a bucket GET with an unknown subresource from being answered with a listing does not know about allow-unordered, so it answers 501 NotImplemented - to a parameter the listing handlers already read and already validate against delimiter. This is why test_bucket_list_unordered and test_bucket_listv2_unordered fail in the Ceph s3-tests suite. They fail on master too; this is not a Lance change and can be taken on its own. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm |
||
|
Chris Lu
|
814ee75af4 |
s3: allow-unordered is a listing parameter, not an unimplemented subresource (#10846)
The guard that stops a bucket GET with an unknown subresource from being answered with a listing does not know about allow-unordered, so it answers 501 NotImplemented - to a parameter the listing handlers already read and already validate against delimiter. This is why test_bucket_list_unordered and test_bucket_listv2_unordered fail in the Ceph s3-tests suite. They fail on master too; this is not a Lance change and can be taken on its own. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm |
||
|
Chris Lu
|
e5edd8be3c |
s3: place multipart part chunks by the destination object's storage rule (#10845)
Multipart parts stage under /buckets/<bucket>/.uploads/<id>/, so the filer resolved filer.conf storage rules against that path when the gateway assigned volumes for them. A rule scoped to a key prefix - fs.configure -locationPrefix=/buckets/b/data/ -ttl=30d - then matched a small object but not the parts of a large one, so an object whose entry carried the rule's TTL had its bytes spread over TTL-less volumes. Assign part chunks against the destination object's filer path instead, the way the x-seaweedfs-destination header made the filer resolve it before the S3 write path moved off the filer proxy. Covers PutObjectPart and both UploadPartCopy paths. The part entry itself is still written under .uploads, so a read-only rule there still rejects it. The lifecycle XML Expiration.Days TTL keeps passing 0 for parts: that rule targets the user-visible object key and would start its clock before CompleteMultipartUpload. |
||
|
Chris Lu
|
abd61de52c |
S3: stamp the gateway's own uid/gid on PutObject and copy entries (#10844)
* fix(s3): stamp the gateway's own ids on single-shot PutObject entries putToFiler builds the entry in the gateway now instead of proxying a PUT to the filer, and it hardcoded Uid/Gid 0 while every sibling write path stamps filer_pb.OS_UID/OS_GID. On a non-root deployment that leaves single-shot PUTs and multipart parts owned by root while directories and completed multipart objects keep the real ids, so a mount reader can list the tree but gets EACCES on every open once objects are not world-readable. * fix(s3): stamp mode and ownership on copy destinations CopyObject and UploadPartCopy build the destination attributes themselves and then assign them over the entry filer_pb.MkFile just stamped, so the copy landed with mode 0000 and uid/gid 0 - unreadable on a mount even by the filer's own user. Build the destination with the same mode PutObject resolves for the request and the gateway's own ids. |
||
|
Chris Lu
|
804111745a |
mount: discard a path-cache insert that raced a purge (#10842)
* mount: discard a path-cache insert that raced a purge The Windows adapter's walk resolves a component with a Lookup RPC and inserts the result holding no lock, so a purge can land in between - and what the walk just resolved is then the very name the purge removed. Anything opening the old path concurrently with a rename repopulates the cache with the vacated name, which the next stat is served from for up to a second. The release path already guards its equivalent insert; the walk had nothing. The cache counts purges now. A resolve snapshots the generation before its lookups and insert discards the entry when any purge ran in between, parking the reference in the graveyard so the in-flight caller keeps a valid inode either way. Seen once in CI as TestRenameOverExisting failing with 'source survived the rename': every SeaweedFS layer is synchronous with the rename, but a background open of the source - an antivirus scan of the just-written file fits - can requalify the stale name through this window. The assertion also reports what stat returned now, and whether it persisted, so a recurrence indicts a specific layer instead of reading as a mystery. * mount: cover the path-cache discard by key, and let a discard rest Review follow-ups. The generation was global, so any purge between a walk's snapshot and its insert discarded the entry whatever its name - and an open retries resolve-then-steal only four times before failing with EIO, so sustained unrelated churn could fail opens of untouched paths. Purges are remembered by key now and only one that covers the inserted name discards it; past the remembered window the insert is discarded without a check, which only costs a retry. A discard that itself tripped the sweep also handed its own reference straight to forget while the walker was still using the inode. The graveyard holds two generations now, so an appended reference always survives the sweep of the call that appended it - which the displaced-entry and purge paths needed too. Also restores the original path-cache test suite this branch had overwritten instead of extended, and rewords the semantics-test failure so it no longer claims the source survived when stat returned a transient error. * mount: take an open's reference directly instead of stealing it back resolveAndSteal cached the final component only to steal it back, so an open depended on that insert surviving whatever purges raced it - four attempts and then EIO. The keyed purge window narrowed how often an insert is discarded, but past the window the discard is blind again, so the cliff had only moved. A cached entry is still stolen; anything else is now looked up directly, with the caller owning the reference from the start. No retry loop, and no way for churn - covered, unrelated or overflowing the window - to fail an open. Also covers the whole-cache purge: purge of the root with prefix set clears every entry, but the covers check tested for a '/'-prefixed key that a normalised key never has, so it covered no in-flight insert at all. |
||
|
Chris Lu
|
da1e5e714f |
mount: move the inode table when a rename arrives from the cluster (#10822)
* mount: leave the target alone when a move has no source MovePath cleared whatever sat at the target before it checked that the source was still there, so a move it then declined to make had already taken the target's mapping apart. The same rename reaching the table twice - once for an open handle, once for the invalidation behind it - was enough to leave the moved inode with no path at all. * mount: move the inode table when a rename arrives from the cluster A rename made by another client reaches this mount only as a metadata event, and the only table update on that path sat inside the open-file-handle branch. Every other inode the kernel still addresses by nodeid kept resolving to its pre-rename path, so the next operation on it went to a path the filer no longer has. Move the entry for every rename invalidation. The filer sends one event per moved entry, so a renamed directory's children follow their parent without a descendant walk. * mount: move the inode table exactly once per rename event Making MovePath return early on a missing source was the wrong half to fix. The source is also missing when the rename came from a client that never visited it, and there the destination really was replaced and has to be unlinked - the early return kept the name resolving to a file the rename destroyed, so a dirty handle on it could still flush over what took its place. The two cases are indistinguishable from inside MovePath, so leave it alone and stop calling it twice: invalidateOpenFileHandle reports whether it moved, and the handler moves only when it did not. Both paths mark a replaced file's handle deleted, which the no-handle path previously did not do at all. * mount: leave a rename alone unless the source is still ours to move Two ways the fallback move could act on state it did not own. A handle whose version guard skipped the event never reached RememberPath, so moving the table under it left the handle flushing to the pre-rename path; the handle path owns its inode's rename, so the fallback now runs only for an inode without one. And the subscription can redeliver a rename once it falls out of the 4096-entry dedup ring. A replay found no source and a live target, unlinked the mapping the first delivery had just made, and marked the moved file's handle deleted - worse than the stale path this set out to fix. Only a source still in the table is moved now. That gives up unlinking a destination the rename replaced when the source was never visited here, which is where this started. It is what the mount already did before this branch, and it is the safer of the two: retaining a stale name costs a wrong lookup, while unlinking the wrong one costs a file's dirty data. * mount: decide a rename move inside the table's lock The source-presence check sat outside MovePath, so two invalidations for one rename could both see the source and the loser would unlink what the winner had just placed - the same damage the check was added to prevent. MovePath makes the decision under its own lock now and reports that nothing moved. The handle a rename destroyed is also marked from the caller rather than from inside invalidateOpenFileHandle, which was setting isDeleted bare on a second handle while holding the first one's lock. markHandleDeleted already takes the lock the flush reads that flag under, and marking from the caller keeps it to one handle lock at a time. The invalidation test harness wires onEntryInvalidation now, the way the mount does, rather than reaching past it. * mount: apply a rename ahead of the handle's version fence The fence exists so an old event cannot roll a handle's entry back to stale content. A rename carries no content: it says the name the inode answered to is gone. Skipping one on the strength of the fence left the inode and the handle both pointing at a name the filer had vacated, and no fallback ran either, since a handle owns its inode's rename. Applied before the fence now, and only when MovePath reports the source was still ours to move - which is what keeps a replayed rename from remembering a path the handle has already moved past. |
||
|
Chris Lu
|
cd3db76eed |
ci: stop installing FUSE headers nothing links against (#10840)
Four workflows ran apt to install libfuse3-dev before every FUSE job. Nothing needs it: go-fuse implements the protocol in pure Go, no cgo in the tree references fuse, and the package does not even provide the fusermount3 the mount actually execs - fuse3 does, and it is already on the runner image, which is why the setuid-repair step finds it. So the step downloaded a dev package to build against headers no compiler ever opened, and it is the step that has been hanging whenever the Ubuntu mirror goes slow. Configuring /etc/fuse.conf is all that is left. |
||
|
Chris Lu
|
e0c4732e5e |
rust: stop writing when a durable write's index flush fails (#10825)
* rust: stop writing when a durable write's index flush fails A durable write flushes the .dat, publishes the needle map row, then flushes the .idx. If that last flush failed we returned the error and carried on: the row stayed live, the volume stayed writable, and the handler answered 500 without replicating. The primary then served a needle its replicas never saw, for a write the client was told had failed - and if the unflushed row was lost on restart, the durable .dat tail took the volume read only anyway. Taking the row back out is not an option: it means undoing published state on a disk that is already failing, and a truncate afterwards would leave an .idx row pointing past the end. So the volume stops taking writes instead, the same as when the truncate after a failed .dat flush cannot be done. Nothing more gets appended past a record whose index is in doubt, and the master routes writes elsewhere once the volume heartbeats read only. The divergence against the replicas is still there, but it is bounded and it is visible. A failed nm.put after the .dat is down leaves the same durable but unindexed record, so it takes the same route. * rust: drop the import the rollback removal left behind NeedleValue came in with rollback_unflushed_write, which went away when the durable path moved to flushing before it publishes. Nothing has used the type since. * rust: mark the test-only heartbeat helper as such collect_heartbeat has only ever been called from the tests - the send loop uses collect_heartbeat_with_snapshot, which it wraps - so a lib build rightly called it dead code. * rust: flush the index on a durable write that dedups A durable write matching content already in the volume flushed the .dat and returned before reaching the index flush. So a fsync=true write that deduped against an earlier non-durable one was acked with the row that indexes it still in the page cache - the same false promise the index flush exists to rule out, and the same read-only volume on restart if the row is lost. The dedup path now flushes both files, and the quarantine on a failed index flush moved into flush_idx so it applies wherever the flush is reached rather than only at the one call site that had it inline. |
||
|
Chris Lu
|
3b18a635df |
ci: call the apt helper from the workflow's working directory (#10832)
The e2e workflow sets defaults.run.working-directory: docker, so the call I added resolved to docker/docker/apt-install and every FUSE Mount run has failed with 'sudo: docker/apt-install: command not found' since it merged. |
||
|
Chris Lu
|
baead6901c |
ci: build protoc into the crate instead of installing it per job (#10830)
Every workflow that builds the Rust volume server first installed protoc from a package manager - twelve steps across apt, brew and choco. That is 37s per job on a good day, and this week archive.ubuntu.com stalled long enough for four jobs to burn their whole timeout without reaching a build. protoc-bin-vendored ships the compiler as a build-dependency, so it now arrives through the cargo registry the workflows already cache and there is nothing left to install. cargo build works on a machine with no protoc at all, which is worth as much locally as it is in CI. It also pins the version. The apt protoc on ubuntu-22.04 is 3.12, old enough to reject proto3 optional, which is why build.rs passes --experimental_allow_proto3_optional; the vendored one is 31.1. The flag stays, since it costs nothing and keeps a build against an older PROTOC working, and an explicit PROTOC still overrides the vendored binary for packagers who supply their own. |
||
|
Chris Lu
|
1564244b1a |
ci: install the runner's own packages through the mirror fallback too (#10831)
The e2e job overwrote the runner's sources.list with two azure-only lines and installed fuse from it, so the same mirror outage that took out the image builds failed the step outright - this time on the runner rather than inside the container, where the image-side fallback cannot reach. Install through the same helper, and widen its rewrite to match any archive host so it works whether the pristine list came from the base image (archive.ubuntu.com) or from a CI runner (azure.archive.ubuntu.com). Keeping the runner's original list also restores the security and backports pockets, which the hand-written two-line replacement dropped. Verified against the outage itself: with the pristine list pointed at Azure, the build logged the skip after Azure timed out for real and installed from archive.ubuntu.com. |
||
|
Chris Lu
|
05013ad3da |
ci: fall through to another Ubuntu mirror when one is unreachable (#10828)
The e2e image pointed both archive and security at azure.archive.ubuntu.com and nothing else, and the samba and pjdfstest images inherit that list. When Azure is unreachable the build has nowhere to go: Acquire::Retries just retries a dead host, every package fails, and apt exits 100 before a single test runs. Two different workflows lost runs to it tonight. Install through a helper that starts from the pristine sources.list each time and walks a list of mirrors, so Azure stays the preferred one - the reason it was pinned in the first place - without being the only one. Verified both paths against a real build: the normal one installs from Azure, and with the first entry pointed at an unroutable host the fallback logs the skip and installs from archive.ubuntu.com. |
||
|
Chris Lu
|
da4f06ec12 |
Give the local Unix socket gRPC transport room to breathe (#10824)
* Give the local Unix socket gRPC transport room to breathe Unix socket buffers default small and never autotune: 208KB on Linux, 8KB on macOS. Once the buffer cannot absorb what gRPC's loopyWriter emits for the in-flight streams the writer blocks on Write, and since v1.82.1 grpc-go counts per-RPC bookkeeping toward its control-buffer throttle, so both peers stop reading and the connection deadlocks for good. weed mini wedged at roughly 320 concurrent S3 PUTs with every filer RPC parked in waitOnHeader and no handler running. Force 8MB on both ends of the sockets we open. Best effort, since a kernel may clamp it lower; that only lowers the concurrency this survives. TCP loopback never hit this because its buffers start large and grow. * Set the buffer on accepted connections too Linux does not carry the listener's SO_SNDBUF onto sockets returned by accept, so only the dialing half was getting the headroom: measured 8388608 on the dialed side against the 212992 default on the accepted side. Wrap the listener and re-apply per connection. macOS inherits either way, which is why this did not show up locally. |
||
|
Chris Lu
|
bb223967bd |
mount: fold an inode's single link into its entry (#10818)
InodeEntry held its one path in a slice, so every inode the kernel references cost a 16-byte backing array and a second heap object on top of the 32-byte entry. The extra links of a hard-linked file now hang off a pointer instead, which keeps the struct in the same 32-byte size class and leaves the ordinary single-link file with nothing to allocate. Populating the table with 1M children: 237.5 -> 221.5 B/inode at 85-character paths, 301.3 -> 285.6 at 148. |
||
|
Chris Lu
|
9f15e3935c |
mount: reuse the listed entry's path instead of rebuilding it (#10817)
readdir built dirPath.Child(name) for every child while entry.FullPath was already that exact string, from NewFullPath in the meta cache store or from FromPbEntry on the read-through path. One allocation per entry, and on a wide tree with long paths that is most of what a listing allocates. BenchmarkReadDirectory/kernel_readdirplus over 200k entries: 2,039,656 -> 1,839,318 allocs/op, 174.5 -> 167.8 MB/op, 152.6 -> 135.1 ms/op. |
||
|
Chris Lu
|
887910b377 |
rust: honor fsync on the volume server write path (#10816)
The Rust volume server ignored the fsync parameter completely: nothing parsed it, and write_volume_needle -> write_needle -> append_needle never flushed. So a ?fsync=true upload was acked out of the page cache, and since ReplicatedWrite forwards the parameter, a Go primary handing a durable write to a Rust replica got the same empty promise. The upload handler now reads fsync the way Go's r.FormValue does, off the decoded query fields, and threads it down to the volume. A durable write appends, flushes the .dat, publishes the needle map entry, then flushes the .idx, and only then is it acked. Nothing points at bytes that are not down yet, so a failed flush only has to take its own append back off the end - the index never moved and the volume's counters never saw the rejected write. If that truncate cannot be done the volume stops taking writes, rather than letting a later append bury the rejected record mid-file where the tail integrity check cannot see it. The .idx flush is what keeps the ack honest: load() rebuilds the map from .idx, so an acked write whose row was lost comes back as a .dat tail the integrity check cannot account for, and the volume loads read only. A dedup hit flushes too: there is nothing to append, but the write it matched may have been non-durable, and the caller is asking for the content to be on disk. Batched writes carry the flag per request rather than one flush per batch, so the write queue's module doc no longer claims otherwise. |
||
|
Chris Lu
|
358fd314ea |
test(s3/versioning): read the whole version body instead of one Read (#10815)
A single Read on the response body can return the last bytes together with io.EOF, so asserting NoError on it fails even though the body is complete. Use io.ReadAll, like every other test in this package. |
||
|
Chris Lu
|
1354b58675 |
s3: stop unrouted bucket subresources from being answered with a listing (#10814)
* s3: answer GetBucketReplication, GetBucketWebsite and GetBucketNotificationConfiguration None of the three had a route, so they reached the unconstrained ListObjectsV1 catch-all and a client asking for a bucket's replication config got 200 and a <ListBucketResult> back. Replication and website report their configuration as absent the way AWS does; notification returns the empty configuration AWS returns for a bucket with no events wired up. * s3: stop an unrouted bucket subresource from being answered with a listing ListObjectsV1 is the catch-all GET on a bucket, so every subresource without a route of its own - ?torrent today, whatever AWS adds next - came back 200 with a <ListBucketResult>. A client that asked for a configuration and got a listing either fails its XML decode in a way that reads like corruption, or worse, tolerantly parses it. Refuse the request instead. The allow-list is the ListObjects parameters rather than the subresources, so a new one fails closed. Presigned URLs sign their credentials into the query string, so X-Amz-* and the SigV2 trio have to stay listable. |