mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-30 17:00:48 +01:00
edf2b0f5eeb81ac188d0dd9169b8538c1efdd56c
10349
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Javier de la Torreanduser.email
|
edf2b0f5ee |
iceberg maintenance: an unreadable parquet file fails the job, not the process (#12188)
* iceberg maintenance: an unreadable parquet file fails the job, not the process parquet.NewReader panics on any file it cannot open (wrong magic, a truncated footer, zero bytes), and parquet-go's row decoder panics on a corrupt page under a valid footer. Maintenance runs in the worker's goroutines with no recover, so one bad delete or data file in any table ended the whole process. - Every parquet open in compaction goes through newParquetReader, which opens with parquet.OpenFile and returns the error. - A panic while decoding a delete file or a data file's rows is recovered into an error naming the file (recoverRead/guardRead). - As a backstop, detection recovers per table and Execute per operation: the table is skipped or the operation fails, and the worker keeps running. A failed delete file still fails the table's compaction (no commit), as a read error did before; a failed data file still skips only its bin. * iceberg maintenance: test a corrupt page inside a merge fails it naming the file * iceberg maintenance: assert a skipped bin's files stay in the snapshot --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com> |
||
|
Javier de la Torreanduser.email
|
55f2234026 |
iceberg maintenance: apply equality deletes only to older data files in their partition (#12187)
* iceberg maintenance: apply equality deletes only to older data files in their partition
Compaction applied every equality-delete key to every data file of every
bin. The Iceberg spec ("Scan Planning") applies an equality delete file only
to data files with a strictly lower data sequence number, and only to data
files in the delete file's own partition unless its spec is unpartitioned.
An upsert writer (Flink, the Iceberg Kafka Connect sink, CDC tools) commits
an update as the row's new version plus an equality delete of its key in the
same commit, so the compaction deleted both the old and the new version of
every updated row.
- equalityDeleteGroup is keyed by equality field ids and scope (unpartitioned,
or spec id plus partition) and keeps, per key, the highest data sequence
number of the delete files carrying it. A delete file without a data
sequence number is an error rather than a guess.
- Partition values are compared typed through the spec's partition type, so
one partition decoded in different Go shapes (a uuid as uuid.UUID or
[16]byte, a long as int32 or int64, a null present or missing) is one
partition. A spec whose fields are all void is unpartitioned. A delete whose
scope cannot be decided (unknown spec, no partition values under a
partitioned spec, a value that does not convert) stops the compaction; a bin
whose own partition cannot be typed while a delete of its spec exists is
not compacted.
- Each bin is merged with only the groups that apply to its partition, and a
row is deleted only by a key whose sequence number is higher than its data
file's.
- A where filter no longer drops unpartitioned delete files (the clause's
field is not in their spec), which let deleted rows return in a filtered
compaction.
Tests: the table fixture now commits its delete files after the data (data
sequence number 1, deletes 2), as writers do; hand-built entries in the unit
tests get sequence numbers. New eqdelete_sequence_test.go covers rows written
with and after their delete, partition scope, unpartitioned and all-void
specs, typed partition matching, missing sequence numbers and partition
values, where filters, and group-key collisions.
* iceberg maintenance: split the undecidable-scope branch in deleteInPredicateScope
---------
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
|
||
|
Chris Lu
|
ec43f4ca00 |
s3: accept bare composite checksum on CompleteMultipartUpload (#12194)
* s3: accept bare composite checksum on CompleteMultipartUpload Clients such as minio-go send the composite object checksum without the "-<parts>" suffix. Compare against the base64 part of the computed composite value as well as the full form. * s3: test object checksum matching with and without part suffix |
||
|
Chris LuandDevin
|
4ed9595a58 |
admin: explain detection cooldowns and skipped scans (#12185)
* plugin: describe detection cooldowns and report skipped scans * admin: expose cooldown eligibility and preserve skipped detection outcomes * admin: show cooldown eligibility alongside maintenance lock waits * plugin: retain cooldown activity for older admin servers * admin: keep manual detection skips out of scheduler run state * plugin: refresh descriptors on reconnect without downgrading saved schemas * plugin: allow valid schema refreshes to repair corrupt descriptors * admin: store eligible_after activity detail as RFC3339 * worker: report admin_script interval skips as skipped detection * worker: report admin_script interval skips after the last completed job * admin: show the selected job's cooldown while scheduling is disabled * admin: report an unsigned interval skip instead of a completed scan * worker: populate new DetectionComplete fields in Rust senders Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
cb993ab83b |
filer.redis3: accept username and keyPrefix options (#12192)
* filer.redis3: prefix every key via getKey, honoring keyPrefix Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.redis3: read username and keyPrefix options into client config Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.redis3: reject a username configured without a password go-redis only sends AUTH when a password is set, so a configured username with an empty password silently connects as the default user instead of the intended ACL user. Fail loudly at initialize instead. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.redis3: cover keyPrefix scoping in the store tests Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer.redis3: derive the test prefix and kv key from the per-run directory --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
e7c1713457 |
admin: show maintenance lock blockers in scheduler UI (#12184)
* admin: expose maintenance lock waiters and current owner * admin: display maintenance lock waits across scheduler pages * admin: preserve the active lock message while acquisitions wait * admin: synchronize lock fairness assertions with waiter registration * admin: display lock wait start times alongside elapsed duration |
||
|
|
6cd14f06c9 |
filer.remote.sync: upload a holed entry zero-filled instead of skipping it (#12178)
* filer.remote.sync: skip an event the filer has written past, not only rewritten
isSuperseded only looked for chunks the event named that the filer no
longer holds. A file copied in through a mount logs a create while the
copy is still running: the mount flushes metadata with chunk uploads in
flight, so that chunk list can have holes. ChunkStreamReader stops at
the first hole, the multipart upload sends a short part ("ContentLength=
8388608 with Body length 4194304"), and since every chunk the event
names is still live the event was never superseded. The daemon retried
it forever and the offset never moved past it.
Treat chunks the filer has added since the event the same way. The
event that added them follows in the log and uploads the finished file.
* filer.remote.sync: skip a holed snapshot even while the filer still holds it
A mid-write snapshot the writer never finished is neither deleted nor
written past: the filer holds it unchanged, so the superseded check never
fires and every retry reads short at the same gap, pinning the sync offset
forever. The upload can never succeed for that entry — skip it like a
superseded one. A finishing write logs its own event and uploads the full
content; a dead write has nothing readable to lose.
The check also covers a chunkless entry that is not a dropped local copy
(e.g. a truncated remote-only entry): its reader yields zero bytes against
a nonzero size, which can silently clobber the remote object with an empty
body where the backend does not verify the length.
* filer.remote.sync: upload a holed entry zero-filled instead of skipping it
Skipping the event left a completed sparse file absent or stale on the
remote, and the coverage check on the raw chunk list missed both gaps a
manifest's bounding extent hides and a FileSize smaller than the chunk
extent. ChunkReadAt already resolves manifests the way the upload reader
does and zero-fills every uncovered range up to the effective size, so
route the upload through it: a sparse file, a mid-write snapshot, and a
chunkless entry all send the bytes a read of the entry returns, and a
hole can neither wedge the event nor publish a truncated object.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* filer.remote.sync: skip a superseded entry on reader-construction error too
---------
Co-authored-by: Alex Russell <alex@sengine.cloud>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
||
|
Jay Hemnanianduser.email
|
6225e0a705 |
s3: return x-amz-copy-source-version-id from CopyObject and UploadPartCopy (#12181)
* s3: return x-amz-copy-source-version-id from CopyObject and UploadPartCopy * s3: cover x-amz-copy-source-version-id on latest copies and part copies The explicit-version case was already covered; add a .versions pointer fixture so the unversioned request resolves the latest version and asserts the header on CopyObject, plus the same assertion on UploadPartCopy. * s3: carry the resolved id on a version file that lacks it A .versions child written before the version-id attribute resolves to an empty source version even though the pointer or the rescan already knows it. Stamp the id on the returned entry so copy and get readers report it; the version copy tests now drop the attribute to cover that shape. * s3: report the null version in x-amz-copy-source-version-id A regular-path source in a versioning-configured bucket is the null version; echo it like every other version id, matching the GET path and AWS. Unversioned sources still report no header. * s3: report the version a self-copy retry actually read --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com> |
||
|
|
48782e2b30 |
s3: allow PutBucketAcl when auth is off (#12180)
* s3: allow PutBucketAcl when auth is off * s3: resolve the PutBucketAcl owner against the recorded owner The unauthenticated path now resolves the account as anonymous, but XML ACLs were still checked against it while GetBucketAcl reported an empty owner for ownerless buckets, so a read-modify-write ACL update failed with AccessDenied. GetBucketAcl and the XML check now both use the recorded owner when the bucket has one and the resolved account otherwise. The canned-ACL grantee moves to the same owner, and a request carrying no account no longer erases a stored owner. * s3: cover PutBucketAcl without auth for XML bodies and stored owners The XML case posts the anonymous owner GetBucketAcl reports for an ownerless bucket; the canned case asserts a recorded owner survives an unauthenticated update and receives the full-control grant. Ambient AWS credentials are cleared so the memory store never turns auth on. * s3: keep the recorded owner through a PutBucketAcl update The grantee and the stored owner were resolved differently: grants named the recorded owner while config.Owner still took the caller's account, so an admin updating another account's bucket rewrote the owner and broke an XML retry naming the original one. An ownerless bucket adopts the ACL's owner; an owned bucket keeps it. * s3: fall back to the admin owner for an ownerless bucket in ACL handlers An unauthenticated PutBucketAcl resolved the owner as anonymous and stored it on the bucket, but the registry reports an ownerless bucket as admin-owned: BucketOwnerEnforced uploads then stamped anonymous instead of admin, and ownership checks after enabling auth compared against the wrong owner. Both ACL handlers now resolve an anonymous fallback to the admin account, so GetBucketAcl reports the same effective owner the registry does and the stored owner matches it. * s3: resolve the admin owner for bucket grants when the caller is anonymous or untrusted Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: skip recording anonymous identity or account as bucket owner Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: report stored bucket ACL grants verbatim Explicitly saved anonymous full-control grants must survive a GET-to-PUT round trip unchanged, so the read path no longer rewrites them to the reported owner. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: expect the stored anonymous grant verbatim in GetBucketAcl Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Alex K
|
fffdb7cc75 |
filer.remote.sync: skip an event the filer has written past, not only rewritten (#12176)
isSuperseded only looked for chunks the event named that the filer no
longer holds. A file copied in through a mount logs a create while the
copy is still running: the mount flushes metadata with chunk uploads in
flight, so that chunk list can have holes. ChunkStreamReader stops at
the first hole, the multipart upload sends a short part ("ContentLength=
8388608 with Body length 4194304"), and since every chunk the event
names is still live the event was never superseded. The daemon retried
it forever and the offset never moved past it.
Treat chunks the filer has added since the event the same way. The
event that added them follows in the log and uploads the finished file.
|
||
|
Chris LuandDevin
|
74f196ceab |
s3: route regular copy source reads to the write owner (#12173)
* s3: read regular copy sources from their write owner * filer: report mutation sentinel codes in ObjectTransaction responses A failed mutation only surfaced resp.Error text, so routed writers could not distinguish an existing-directory conflict from a generic failure and could not apply the prefix-object retry the plain CreateEntry path has. * s3api: retry a routed mkfile as a prefix object over a directory mkFile retries ErrExistingIsDirectory by marking the entry a prefix object; the routed path returned the conflict instead, so a routed write to a key that is also a prefix failed where the lock path succeeds. The filer now reports the code on the transaction response, making the retry recognizable. * s3api: commit copy destinations on the key's write owner An owner-routed source read still left the destination on the gateway's default filer, so a second copy resolved the first copy's destination on the owner, found nothing, and returned NoSuchKey. Write unversioned and suspended destinations through routedMkFile, and write the version file on the owner the finalize recomputes on. * s3api: cache remote objects on the filer that served the entry An owner-routed lookup can return a remote-only entry that has not replicated to the gateway's default filer; the cache RPC then cannot find it and the copy or read degrades to a 503. Ask the entry's write owner to cache it first, falling back to the default filer. * s3: read versioned object state from the key's write owner CopyObject destinations now commit version files and the .versions pointer on the owner's object-scope ring key, so versioned reads through the default filer could miss them until replication. Generalize the routed read to dir/name lookups and paginated lists scoped to the object's owner, and point the versioned read paths at it: getSpecificObjectVersion, doGetLatestObjectVersion, the .versions quick-checks in Get/Head/attributes, the self-heal rescans, tagging, delete conditionals, directory-marker history checks, rename destination, multipart completion checks, and the version-id format probe. Reads fall back to the default filer when no ring owner is resolvable, preserving single-filer deployments. * test(s3api): fake RECOMPUTE_LATEST/DELETE and cover versioned copy reads The ambiguous-PUT fake filer now creates parent directories on PUT (mirroring InsertEntry), applies DELETE mutations, and re-derives the .versions pointer for RECOMPUTE_LATEST the way applyRecomputeLatest does, so owner-vs-replica tests can exercise versioned writes. TestVersionedCopyDestinationReadsOnOwner copies into a versioned bucket whose writes commit on the owner while the gateway's default filer is a replica with no replication, then asserts reads by versionId, latest-version resolution, and a copy chained off the new version all resolve on the owner. * test(s3api): gofmt the versioned-copy coverage * s3: only swap in a cache fill of the resolved remote generation A cache response can come back from a filer holding an older generation of the key. Replacing the authorized source or swapping its chunks under a size and range computed for the resolved entry serves the wrong object or truncates the response. Match RemoteEntry identity and size before using a cached entry, in both the copy and streaming paths. * filer: keep directory attributes when an object lands as prefix object The object-onto-directory retry replaced the existing directory's mode, owner, and extended attributes with the object's. Preserve the directory identity; merge directory-held extended keys only when the old entry never held an object, so a prior generation's object metadata is not carried forward. * s3: keep marker lock checks on the filer that replaces the marker The object-lock precondition read the key's write owner while mkdir still replaces the marker on the default filer, so a locked marker not yet replicated to the owner could be overwritten. Read the entry where the replacement lands, in both PutObjectHandler and zip extraction. Also hash a trailing-slash marker key to the same route key as the object it names, so marker and object share one owner and one write lock. * s3: route tag writes to the owner tag reads resolve GetObjectTagging reads the key's write owner, but PutObjectTagging and DeleteObjectTagging still wrote through the default filer, so a successful change returned the owner's stale tags until replication. setTags, rmTags, and the versioned tag updates now write on the owner first, falling back to the default filer when no owner resolves. * s3: keep rename lookups on the filer that performs the move AtomicRenameEntry and the retry token it carries both execute on the default filer, but the source and destination reads went to the write owner — a completed rename's token was invisible to its own retry, and a source that only exists on the default filer looked absent. Read both ends on the default filer. * s3: send the .versions pointer repair with the object's route key The heal reads the .versions entry on the object's owner, but the conditional UpdateEntry is routed by the .versions path's own ring key, which can hash to a different filer — leaving the pointer the routed reads consult unchanged. Ship the CAS repair as a PATCH_EXTENDED ObjectTransaction on the object's route key instead. * filer: preserve only directory identity on a prefix-object write Merging the old directory's extended attributes could carry a marker's encryption, version, or etag fields onto the new plaintext object, since a directory's own keys cannot be told apart from the object it held. Keep only mode, uid, gid, user name and group names. * s3: also require matching size when a cache fill replaces the source The remote-generation check covers identity; size equality is the cheap guard that keeps the streamed or copied range consistent with what was resolved. * s3: lock the object path for a routed .versions pointer repair conditionalUpdateEntryRouted held the lock on the .versions path while version writes and deletes serialize on the object path, so a repair could pass IF_ENTRY_EQUAL and then apply over a pointer a writer had advanced in between. Lock the object path like the writers and keep the condition on the .versions entry via condition_key. * s3: do not replay an owner-side tag mutation on another filer withWriteOwnerFilerClient re-ran fn on the default filer after any owner error, so an authoritative not-found could update a stale replica entry and still report success, and a post-apply transport error could apply the mutation twice. The failover walk now ends on any error fn returns; only a channel that never came up moves to another filer. * filer: keep a directory's xattrs when a prefix object replaces it The prefix-object branch preserves the directory's identity but dropped its Extended map wholesale; FUSE xattrs live there under the xattr- prefix and belong to the directory, not to the object the marker held. * s3: delete null and specific versions on the filer that serves them On the default filer an absent entry deletes as success, so a version that only reached the write owner survived a reported success while owner-routed reads kept serving it. The suspended-versioning and lifecycle deletes now go through routedDeleteNullVersion and routedDeleteSpecificVersion when an owner resolves; an unsettled attempt keeps the lock path. * s3: route a lifecycle null-version delete to the object entry The "null" version is the regular object entry, not a .versions file, so it goes through routedDeleteNullVersion; only a real version id takes the .versions-file delete. * s3: repoint .versions only while it still names the deleted version routedDeleteSpecificVersion recomputed the latest pointer on every version delete, so removing a noncurrent version re-derived the pointer to the newest remaining file — resurrecting an older version over a cleared pointer (a suspended bucket's null write keeps it clear while the null object stays current) or over a pointer that moved on. The pointer is now read on the owner and repointed under an IF_ENTRY_EQUAL guard only while it still names the deleted version; a contested pointer is left for the self-heal path rather than overwritten by a stale scan. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: let lifecycle expiry keep its metadata-only deletes on routed paths The routed lifecycle deletes hardcoded IsDeleteData, so expiring an entry with a volume TTL queued per-chunk deletions instead of leaving the data for the volume to reclaim — extra work on the owner for deletes recordMetadataOnlyIf still counts as metadata-only. Pass the caller's cleanup choice through routedDeleteNullVersion and routedDeleteSpecificVersion; the interactive DELETE path keeps deleting data. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: let RECOMPUTE_LATEST skip a pointer that moved on A version delete repoints .versions only while the pointer still names the deleted version, but the gateway cannot carry the pointer's current state into the transaction's condition — the condition slot is already spent on the version's WORM guards. only_if_current_name moves the check under the transaction lock: the filer compares the pointer's name_to_key value before scanning and skips the recompute when it moved (or was cleared by a suspended bucket's null write), so a noncurrent delete cannot resurrect an older version. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: repoint inside the version's locked delete repointLatestBeforeVersionDelete ran in its own transaction ahead of the delete, so a version delete denied by retention or legal hold still moved the latest pointer — GET and HEAD went on serving the older version it rolled to. The repoint and the delete now ride one transaction behind the version's WORM guards, atomically gated by the new only_if_current_name recompute guard instead of a stale-read IF_ENTRY_EQUAL. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
511727cf3a |
vector: DeleteIndex format guard, sticky discovery, advertise normalize (#12171)
* vector: DeleteIndex format guard, sticky discovery, advertise normalize - handleDeleteIndex verifies the table carries the Lance format before DeleteTable — an Iceberg table under the same namespace is not an index and must not be removed by it. - discovery keeps the current worker while it is still registered — membership-order churn can no longer cancel in-flight calls, and the resolver closes each master conn inside a 5s deadline. - advertised_addr strips scheme and .grpcPort so the UDP probe reaches every supported master form. * vector: empty-follower retry, format check without read grant - discoverVectorWorkers keeps looking past an empty master reply — followers list nothing when workers registered with the leader. - the DeleteIndex format check uses the internal catalog read, so DeleteTable-only callers don't gain a GetTable requirement, and the format must be Lance exactly — format-less tables no longer pass. |
||
|
Chris Lu
|
ae95b2c777 |
ec: rebuild a needle's suspect shard ranges when it fails to parse (#12170)
* ec: let shard-interval recovery exclude a set of suspect shards gatherEcShardIntervals and recoverOneRemoteEcShardInterval skipped only the shard being rebuilt. A needle-level rebuild wants every suspect shard kept out of the gather so a bad copy cannot poison the reconstruction it is being compared against. * ec: rebuild a needle's suspect shard ranges when it fails to parse A bit-rotted data shard serves a clean-looking but wrong interval, so the missing/unreadable-shard fallback never engages and the needle CRC check in ReadEcShardNeedle returns the corruption to the caller. Each interval's source shard is a suspect: rebuild the suspect ranges from the surviving shards — all of them at once while parity covers the suspect set, or one at a time when the needle spans more shards than that — and let a successful needle parse arbitrate which copy was corrupt. Shards whose range changed are dropped from the location cache so later reads go straight to reconstruction. * ec: mirror the corrupt-shard needle rebuild in the Rust volume server The snapshot now keeps every interval's shard placement so a parse failure can name its suspects. rebuild_ec_shard_needle rebuilds the suspect ranges through recover_one_remote_ec_shard_interval with the suspect set excluded from the gather, and the winning candidate's differing shards are dropped via forget_shard_location. * ec: recover suspect ranges under the requested needle id A failed ReadBytes re-reads the id out of the possibly corrupt header, so the rebuild passed whatever it decoded to peer recovery requests; carry the requested id through instead, and reject a candidate whose parsed id does not match it. * ec: retry a failed needle rebuild with each gathered input dropped The gather behind a suspect-range rebuild can itself serve a second corrupt shard: Reed-Solomon then reconstructs wrong bytes and the needle parse rejects the only candidate. Gather every surviving shard once, then re-decode with each of the first decode's inputs dropped in turn; the parse arbitrates the honest subset. * ec: mirror the dropped-input rebuild and id check in the Rust volume server Same two guards as the Go side: a candidate that parses under a different needle id is rejected, and a failed candidate whose gather fed it a corrupt input is retried by dropping one gathered shard at a time. The gather is shared through a want-bounded gather_ec_shard_intervals so the retry pays no extra fan-out. * ec: answer deleted when a rebuild's holder reports the needle deleted The rebuild discarded the deleted flag its interval recovery collected, so a needle deleted on its other holders but live in the local index came back as a corruption error. A deleted report is authoritative, so it now short-circuits the hypothesis search and answers ErrorDeleted. * ec: hydrate rebuild candidates into a scratch needle A failed ReadBytes leaves partially parsed fields on the needle, and optional sections do not reset when absent, so a corrupt first parse could leak stale name/mime/ttl into a successful candidate. Parse each candidate into a fresh needle and adopt it only on a matching id. * ec: corrupt parity+1 shards in the exceeds-parity read test The loop skipped the suspect's own shard when it fell inside the parity range, leaving only ParityShards corrupt shards — a case recovery can legitimately fix. Count the additional corruptions so the test always exercises the boundary it names. * ec: mirror the deleted-report short-circuit in the Rust volume server A holder's deleted answer during a rebuild propagates out as EcMiss::Deleted instead of surfacing as InvalidData, matching the Go path's authoritative-delete rule. |
||
|
Chris LuandDevin
|
eec2452e30 |
master: replicate the recovery-wait growth unfreeze through raft (#12165)
* master: replicate the recovery-wait growth unfreeze through raft state_loss_recovery_wait lifted the volume-growth freeze only in memory, while the freeze itself was replicated. A restart replayed the frozen flag and growth stayed refused until the wait expired again, on every restart, unless an operator ran volume.growth.resume. The lift now applies GrowthFrozen=false like ResumeVolumeGrowth, from a goroutine since IsGrowthFrozen is a read path that cannot block on raft. The apply runs under reportedTopologyIdsLock so a freeze landing after the lift keeps its command last in the log; the stale unfreeze skips itself when the flag is frozen again. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: cover raft replication of the recovery-wait unfreeze Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: retry and re-arm a growth unfreeze raft failed to record applyGrowthFreeze swallowed the raft error, so a rejected lift during a leader change left the flag clear while the log still held the freeze; nothing retried and a restart re-froze growth anyway. The apply now returns the error, replicateGrowthUnfreeze retries it, and on persistent failure re-arms the flag so the next IsGrowthFrozen attempts the lift again instead of diverging from the log. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: release the topology-id lock before the freeze raft write CheckReportedTopologyId and SetTopologyInitPending held reportedTopologyIdsLock across the blocking raft apply, so a slow apply stalled every later report — including the heartbeat that should trigger the next freeze. The lock now covers only the mismatch decision; the replicate step runs after it is released. * master: bind the recovery unfreeze to the freeze instant it lifts An unfreeze submitted while a newer freeze sat committed but not yet applied — e.g. after this node won an election — could land behind that freeze in the log and erase it. The command now carries the freeze instant cluster-wide (GrowthFrozenAt) and the lift is bound to the instant its submitter observed (GrowthUnfreezeFor): the state machine no-ops a lift while a strictly newer freeze exists, which also lets the unfreeze run without holding the topology-id lock at all. Concurrent freeze writes keep their own mutex so log order cannot regress the instant. * master: a superseded unfreeze retry leaves the resumed flag alone replicateGrowthUnfreeze broke out of its retry loop when a newer\nfreeze superseded the lift, but the break fell through to the\nre-arm: an operator resume in between was silently undone. Return\ninstead — the newer freeze generation owns the flag. * master: replayed freeze entries without an instant stay ageless Freeze commands written before the instant field existed carry zero.\nSubstituting the replay time made each restart stamp the freeze as\nnew, so a committed bound unfreeze always looked older and was\nrejected — the freeze came back on every restart. Keep the zero\ninstant: the recovery window reads as elapsed and bound lifts apply. * master: pack the growth freeze flag and generation into one atomic Separate flag and timestamp atomics left three holes: a resume's\nload-and-clear could lift a generation stamped in between, a bound\nlift compared stamps numerically although masters' clocks need not\nbe monotonic, and an ageless generation had no identity a bound lift\ncould match without colliding with the unconditional encoding.\n\nOne value now carries the state — 0 unfrozen, -1 an ageless freeze,\nelse the freeze instant — so every transition is a single atomic\nswap or compare-and-swap bound to an exact generation. * master: a repeat mismatch report keeps the armed freeze generation Re-stamping while already frozen never reaches the raft log, so the local generation diverges from the replicated one and every bound unfreeze becomes a no-op on replay — the freeze returns after restart or failover. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: snapshots read the freeze flag and generation in one load Save, Snapshot, and PersistMaxVolumeId read IsGrowthFrozen and the generation separately, so a resume landing between them persisted frozen=true with stamp 0 — restoring an ageless freeze a committed bound lift cannot clear. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Nguyễn Lộc Phúcandchrislusf
|
b62bbb99fc |
fix(iceberg): serve a registered table's metadata on load and commit (#12159)
* iceberg: serve a registered table's metadata on load and commit RegisterTable saved only the metadata location, and loads and commits never read the file there. They served a placeholder with no schema, no snapshots and a new random table UUID on every load, so a registered table read as empty, an INSERT failed against the empty schema, and every other commit failed its assert-table-uuid requirement. Metadata uploaded through the S3 gateway, as a migrated table's is, is stored in chunks that registration did not read, so it registered an empty table. Read the file following its chunks, reject one that is not Iceberg metadata with a 400, and keep its content and table UUID in the record the way CreateTable does. Records registered before this change hold only the location. Loads and commits now read the file there, and the first commit stores it in the record under the file's table UUID. * iceberg: transactions read a location-only registration's metadata file * iceberg: cover transactions on a location-only registration * iceberg: refresh the default name mapping on transaction commits * iceberg: cover name-mapping refresh in transactions * iceberg: roll a transaction back to a registration's metadata file A transaction that failed after flipping a table registered before the record kept its metadata could not undo the flip. The rollback restored the record's prior metadata, which for such a table is only a location, and UpdateTable keeps the full metadata a flip stored when the restore sends none. Loads went on serving the change the client was told had failed, and a client retrying the transaction could apply it twice. Restore the metadata loads served from the file instead, under the file's table UUID. * iceberg: register oversized metadata by its location Registration sends the metadata file's content to the S3 Tables manager, which refuses a request over its 10MB body limit, so a table whose metadata is larger could no longer be registered at all. Register such a file by its location and table UUID alone; loads read it from that location, as they do for a record that never kept its metadata. Export the limit as s3tables.MaxRequestBodySize so the catalog can check against it. A commit to such a table still sends the whole metadata to UpdateTable and stays over the limit. * iceberg: commit oversized metadata by its location * iceberg: let maintenance follow a location-only table record * admin: read a table's metadata file when the record keeps none * iceberg: bound the metadata file a location-only record reads * iceberg: bound only chunked metadata reads in the worker Inline content is what the catalog itself wrote and the REST load path accepts it at any size, so rejecting it in loadMetadata stranded tables over 64 MiB from every maintenance scan. Cap only the chunked read, where the recorded location can name any object in the bucket. --------- Co-authored-by: chrislusf <chrislusf@users.noreply.github.com> |
||
|
Jay Hemnanianduser.email
|
a816c6202c |
sftp: fail rm and rmdir of a missing path, and mkdir of an existing one (#12168)
* sftp: fail rm and rmdir of a missing path, and mkdir of an existing one * sftp: create directories exclusively --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com> |
||
|
Chris LuandDevin
|
3fd8d0cf09 |
master: let location-constrained assigns wait for pending growth (#12167)
* topology: add AvailableSpaceWithinLocation for location-scoped free slots AvailableSpaceFor answers for the whole topology; a location-constrained assign needs to know whether the requested data center, rack, or data node still has room before deciding growth can satisfy it. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: let location-constrained assigns wait for pending growth An assign carrying a data center or rack constraint failed on the first unsatisfied pick, even when the same request had just queued volume growth for that location — so the first write into a fresh collection errored out instead of picking up the volume growth landed a moment later. Keep the fail-fast path only when growth cannot help: growth disabled, no growth pending, or no free slots left inside the constraint. Otherwise the request takes the same wait/shed path as an unconstrained assign. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: sum leaf free slots in AvailableSpaceWithinLocation Rolled-up rack and data center counters keep a maintenance-mode server's capacity, and one overcommitted node can cancel a sibling's slots. Sum the matching data nodes directly so the location check agrees with what growth can actually use. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: let constrained growth initiators reach the shed path When a constrained assign's own growth concluded unfulfilled, the fast-fail check saw only that no grow was pending and returned the plain pick error — a code the assign client does not retry, so the first write died instead of re-triggering growth. Exclude the request's own initiated grow from the no-pending-growth condition so it reaches the existing re-pick and ResourceExhausted shed like an unconstrained assign. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: judge targeted grow requests by the location they name A grow request carrying a data center, rack, or data node constraint was discarded whenever the layout's writables elsewhere were healthy, so a location with free slots but no writable could never receive the volume its assign was waiting for. Add ShouldGrowVolumesWithinLocation and use it for constrained requests; the layout-wide check still governs untargeted ones. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: only positive leaf slots count in AvailableSpaceWithinLocation An over-committed node reports negative free slots; summed plainly it\ncancelled a sibling's room and a constrained assign could fail fast\nwhile the location still had space to grow into. * master: fail constrained assigns fast when growth cannot place A constrained assign with free slots in its location still waited out\ngrowth that could never satisfy the replica placement — a single-rack\ndata center asked for 010 sheds the same doomed grow on every client\nretry. Run the grow-time node search as a dry run and return its\nplacement error instead of a retryable shed. * master: judge constrained growth by layout, not free slots A slot-level feasibility probe races with the grow it judges: a\npending grow consuming the last slot in one rack reports the\nplacement impossible while it mints the request's own volume, and\nthe hard error skips the re-pick that would find it. GrowthBlocker\nnow counts racks and data nodes inside the named location — shape a\ngrow cannot consume — so only genuinely unplaceable layouts fail\nfast. * topology: GrowthBlocker mirrors the grow selector's feasibility The location filter bound every replica tier, so a 010 request naming one rack needed two racks inside it while the selector constrains only the first placement and counts the rest by room. And counting nodes without free slots let a full location queue a grow it could never serve, so the assign shed ResourceExhausted at timeout instead of failing fast. GrowthBlocker now applies the option's location to the primary placement chain only and leaf-sums free slots the way the selector does, so it reports a blocker exactly when the grow can never place. * master: repick once before surfacing a growth blocker An in-flight grow can spend the last free slot minting the request's own volume: the blocker's slot counts predate that registration by a heartbeat, so returning the placement error on the first failed pick could reject a write whose volume lands a moment later. A pending grow now earns one repick grace before the blocker error is returned. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
18691e7b5b |
server: keep the first write on a fresh weed server from stalling (#12149)
* master: end the post-mint allocation hold on the embedded reporter's clean inventory DeferAllocation armed a warmup-length hold whenever the leader minted a fresh topology id, and on weed server the hold ran the whole window: the embedded volume server's join kept refreshing the quiet evidence clock, so the two-pulse early release still cost the first write 10-15s on every fresh start. weed mini already releases on its single pinned reporter's first inventory, but pinning refuses outside volume servers, which weed server must keep admitting. Add a non-exclusive variant: the topology names one expected reporter whose first clean full inventory lifts the hold, while other reporters stay admitted and still feed prior-state evidence and the topology-id freeze. A single-master weed server points it at its embedded volume server — the one reporter guaranteed to arrive quickly; a veteran's heartbeat carries prior state and keeps the window through the regular checks, so a wiped -dir with surviving external volume servers still trips the freeze on first report. Restarted clusters are untouched: a recovered topology id means no mint and no hold. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: announce on the startup peer scan for embedded single-master setups A fresh filer whose startup peer scan came back empty re-polled for the full confirm window before registering on the master, so the lock ring had no member and routed traffic such as S3 PUTs spun on "no lock server found" for ~10s on every fresh weed server / weed mini start. For setups that start with exactly one filer the scan is already final: a single-master weed server and weed mini announce as soon as it completes. A late external filer still joins and replays through the normal path — the confirm window only ever caught peers arriving inside it, and the residual risk mirrors the accepted volume-side trade-off: an embedded filer announcing a few seconds earlier can own the lock ring while a surviving external filer's backlog is still on its way. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: retry a failed heartbeat promptly until first registration A heartbeat stream refused before the leader is known — the common race when a volume server starts while an election is still running — always fell back to sleeping out the full pulse, so a fresh volume's first registration could wait ~5s behind a leader elected a second later. The peer refresh already identifies the leader during the wait, so cap the retry pause at a second once a leader is named; while no leader is known the same short retry applies until the first stream is confirmed, since an unregistered server has nothing else to do. Once the server was registered the pulse backoff is unchanged, so a lost leader still paces reconnects the way it did before. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: retry a failed heartbeat promptly until first registration (rust) Mirror the Go heartbeat retry: cap the post-refresh wait at a second when a leader is named, and keep the short retry until the first stream is confirmed so a fresh volume registers promptly when it races an election. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * master: leave admitted reporters a confirm pulse when the embedded report ends the hold The single-master weed server releases its post-mint allocation hold on the embedded volume server's clean report, but outside volume servers stay admitted and one may still carry prior cluster state — an inventory from the embedded server alone cannot prove none is due. Clearing the hold outright let a rebuilt master re-issue a volume id a reconnecting veteran still owned. Collapse the released hold to one heartbeat interval instead: late reporters land their evidence or a topology-id freeze inside the pulse, and a re-mint re-arms that pulse rather than the full warmup. The pinned single-reporter release (weed mini) still drops the hold outright, since no other reporter can be admitted. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: keep the empty-membership confirm on embedded single-master weed server A single-master weed server still admits external filers, so an empty startup peer scan is not final: after a master restart the existing filers simply have not reconnected yet. Announcing on the scan let the embedded filer take lock-ring ownership and serve writes that a peer's replayed backlog could then overwrite. Only weed mini keeps the skip, where the embedded filer is the only member its dev cluster can have. The confirm window only delays the master listing for a fresh store; the filer serves while it runs. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: tolerate an expired heartbeat retry deadline (rust) refresh_leader_from_peers can consume the whole pulse budget, leaving retry_at in the past; subtracting a later Instant panics and kills the heartbeat task instead of retrying. Match the Go side, where time.Until of a past deadline just sleeps zero. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: retry the seed list without a second pulse sleep (rust) The error path already slept its retry delay; an extra full pulse after losing the round delayed the next attempt past what Go's heartbeat() does, which paces retries only inside the per-master attempt. * master: keep a configured allocation hold shorter than the confirm pulse Once the embedded report lands, the collapsed hold replaced the configured deferral with a full pulse even when the operator set a shorter one, including zero (disabled). Cap at the pulse instead. * filer: end the empty-membership confirm early on a fresh-minted embedded master On a fresh single-master weed server the announce gate still held the filer for the full confirm window, so the master's lock ring stayed empty and the first write retried 'no lock server found' for ~7s on top of the collapsed allocation pulse. A fresh TopologyId mint proves the master's persisted state was empty, making the empty peer scan final; skip the rest of the confirm only then, and keep it whenever the master recovers a prior id. * filer: keep a short reconnect window on a fresh-minted embedded master A fresh TopologyId mint proves the master's persisted state was empty, not that no external filer survived its loss — one can still be mid-reconnect on its keepalive cycle when the confirm starts. Ending the empty-scan confirm on the first poll let the fresh filer register and serve routed reads before that peer was found and backfilled, so existing files could read as missing. The fresh-mint path now shortens the confirm to a few keepalive cycles instead of skipping it: a reconnecting filer still lands in time, while a genuinely fresh cluster announces after ~3s rather than ~10s. * volume: skip the heartbeat loop when no seed masters are configured An empty seed list made the outer loop spin with no await between passes and no path to observe the stop flag. A masterless volume server has nothing to register with, so the heartbeat exits after a warning. * volume: skip the heartbeat loop when no master addresses are given (rust) Mirrors the Go heartbeat: an empty seed list spun the outer loop with no await and no shutdown checks. Exit after a warning so library callers get the same behavior as the binary path that validates the list up front. * filer: re-check the fresh-mint hint on each confirm pass The topology id is minted after raft election, which can land after the empty-scan confirm has already started — a one-time check at entry missed the mint and held the full window. Evaluate the hint on every poll and shorten the confirm from the first observed mint, still capped by the normal confirm deadline. * master: mark a fresh topology mint only when the volume bound is zero Recovery restores the max volume id unconditionally but the topology id only when present, so an upgraded or partially rebuilt master can mint a new id while still holding prior volume state. Marking fresh there would let the embedded filer shortcut its peer confirm on a cluster that is not empty; only mark the mint fresh when no bound survived. * filer: drop the fresh-mint shortcut once a peer has been observed --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Jay Hemnanianduser.email
|
9059cc2a07 |
s3: allow PutObjectAcl when auth is off (#12155)
* s3: allow PutObjectAcl when auth is off * s3: ownerless objects claim the admin account when auth is off An object written through the filer carries no owner metadata, so objectOwner fell back to an empty account id and ExtractAcl rejected the canned ACL's canonical grantee. Extend TestPutObjectAclWithoutAuth to cover the ownerless case. * s3: GetObjectAcl reports the admin owner for ownerless objects when auth is off The same empty-owner fallback produced an ACL response with a blank Owner ID and a blank canonical grantee. * s3: ignore the client-supplied account id when auth is off Auth does not strip AmzAccountId on the iam-disabled path, so the ownerless fallback could persist a forged owner. requestAccountId substitutes the admin account, matching the idiom already used in the upload ACL path. --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com> |
||
|
Chris Lu
|
d5433ae745 |
vector: cover the identity object in s3vectors auth (#12162)
vector: cover the identity object in the s3vectors auth context The auth middleware must carry the resolved identity object — only the name survives otherwise and authorization loses actions, account, and claims. |
||
|
Jay Hemnanianduser.email
|
9b3f09d0a5 |
sftp: refuse mkdir and uploads under a missing folder (#12156)
* sftp: refuse mkdir and uploads under a missing folder * sftp: re-check the parent directory when a buffered upload commits The open-time check cannot catch a parent removed while the upload buffered; the filer would recreate it when the entry lands. * sftp: cover mkdir and upload commit against a missing parent Extend the test filer with CreateEntry so makeDir can be exercised end to end. * sftp: refuse a rename into a missing folder The filer's rename path creates the target entry through CreateEntry, which recreates absent parents the same way uploads did. --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com> |
||
|
Chris Lu
|
afb8472c5e |
s3vectors: cover non-filterable key round-trip and nested filter rejection (#12160)
* s3vectors: cover non-filterable key round-trip and nested filter rejection The catalog writes the key list as a JSON string while older records may still carry the array form; indexFromRecord must decode both. The rejection path extracts field names through nested $and/$or filters, so a blocked key deep in the tree still fails the query. * s3vectors: factor the non-filterable rejection into a testable gate Pull the blocked-key check out of handleQueryVectors into nonFilterableField so the rejection itself is covered without standing up the catalog stack; the handler now 400s on whatever the gate names. * s3vectors: report the non-filterable match separately from the key An empty metadata key in nonFilterableMetadataKeys collided with the "no match" sentinel, so a filter on that key skipped rejection. Return a match flag and cover the empty-key case in the test. |
||
|
Chris Lu
|
7d823265e5 |
wdclient: regression test for the elapsed caller-deadline lookup path (#12154)
* wdclient: cover a lapsed caller deadline that Err() has not observed yet * wdclient: reuse isCanceledErr for the elapsed-deadline check * wdclient: reinterpret only deadline errors as caller-gave-up isCanceledErr also matches gRPC Canceled, which a filer can return for its own reasons; treating it as an elapsed caller deadline hid the failure from the filer's health record. The fallback now matches only context and gRPC DeadlineExceeded, so filer-side cancellations keep counting as filer failures. |
||
|
Chris Lu
|
237db29dca |
vector: round-trip and enforce non-filterable metadata keys (#12157)
vector: non-filterable metadata round-trips and filters reject it - GetIndex omitted nonFilterableMetadataKeys — the catalog stores the list as a JSON string but the reader only accepted an array; both forms now decode. - QueryVectors now rejects filters referencing non-filterable keys instead of silently scanning them. |
||
|
Chris Lu
|
190ab32cfb |
vector: weed-vector registers to master; s3 discovers it there (#12146)
* vector: weed-vector registers with the weed master weed-vector-server --masters holds a KeepConnected stream to the weed master, the same mechanism filer and s3 use via wdclient — the server shows up in ListClusterNodes(client_type=vector). Resends registration and rediscovers on failure. * vector: s3 discovers weed-vector through the master With -vectors.grpc unset the vectors gateway resolves one registered weed-vector via ListClusterNodes on the masters it already derives from filer config — discovery replaces the static dial for the single-worker OSS path. The conformance harness registers the engine to mini's master and drops -vectors.grpc. * vector: default-fill registration fields ..Default::default() keeps the source compiling against proto variants that carry extra fields, so the same file works in either tree. * vector: serve the Vectors API on the main s3 port The vectors router sits on the main -port listener behind a dispatch that checks the SigV4 credential scope for /s3vectors/ — no separate port needed. -port.vectors stays for explicit dedicated-port deployments; the conformance harness now hits vectors through :8334. * vector: drop -port.vectors entirely It was new — no compat to preserve. The main -port listener serves the API whenever a weed-vector backend is reachable. * vector: drop -vectors.grpc; discovery is the only locator weed-vector always registers to master, the gateway always resolves it through ListClusterNodes — a static address just forks membership. * vector: binary is weed-vector Matches the enterprise binary; package and lib names are unchanged. * vector: review fixes — cluster type, grpc port, lazy resolve, leader hint, advertise - cluster: register 'vector' as a node type so ListClusterNodes returns weed-vector members - s3: dial masters via ToGrpcAddress (HTTP port + 10000) - s3: backend resolves lazily through vectorDiscovery — workers joining after gateway startup work on first query, and address changes re-dial - master_client: convert master HTTP addr to grpc +10000, honor http(s) scheme, follow the leader hint in KeepConnectedResponse, and --advertise covers wildcard/unreachable listen addresses * vector: no --advertise flag; derive the outbound address A wildcard listen resolves to the source IP the master sees via a route lookup, so registration always advertises something reachable without another flag. * vector: review fixes — plaintext dispatch, conn leaks, discovery backoff The two plaintext http servers now serve the vectors dispatch too — signed s3vectors calls on the main port reach the right router. discoverVectorWorker closes each master conn and bounds lookups to 5s; a dead or absent worker no longer re-hits master on every RPC — the resolver backs off for 5s between rediscovery attempts. * vector: 1 GiB grpc cap, dead-worker invalidation, CLI rename Valid PutVectors batches exceed the tonic 4 MiB default — match the gateway's 1 GiB cap. A replaced worker no longer sticks: any RPC error resets the cached address and the next call rediscovers through the master. The core CLI bin renames to weed-vector-cli so the two crates do not collide on the same output path. * vector: retry-only-transport reset, periodic re-resolve, cli rename - reset() takes the failed client and runs only on transport errors — an InvalidArgument on a healthy conn no longer interrupts unrelated requests, and a concurrent re-resolve wins the race. - cli() re-resolves once the interval passes even with a live client, so a worker that moves addresses is picked up. - verify.rs spike docs use weed-vector-cli after the bin rename. * vector: only Unavailable counts as a transport failure A request deadline or an application error can fire on a healthy connection — resetting the shared conn on those churned reconnects. |
||
|
Chris Lu
|
a2863bf83b |
wdclient: count an elapsed caller deadline as caller-gave-up in LookupVolumeIds (#12151)
* wdclient: treat an elapsed caller deadline as caller-gave-up on lookup errors gRPC can surface its own DeadlineExceeded a tick before ctx.Err() becomes visible, so a lookup that died on the caller's deadline was sporadically recorded as a filer failure (seen in the linux/386 job). * wdclient: reinterpret only deadline errors against an elapsed caller deadline |
||
|
Chris Lu
|
10ba1b782f |
vector: carry the full identity through s3vectors auth (#12152)
vector: keep the full identity in the request context Auth discarded the resolved identity object — the s3tables manager request then re-resolved by name and denied authenticated users their actions, policies, account, and claims. Store the object via SetIdentityInContext so downstream authorization sees it. |
||
|
Chris Lu
|
5f26ef6ad1 |
filer,webdav,iam,sftp: add -idleTimeout flag for connection idle timeout (#12136)
* filer: make the HTTP connection idle timeout configurable The filer applied a hardcoded 10s activity timeout to its HTTP listener. A buffering proxy in front of the filer can stall upstream reads longer than that when the end client drains slowly, so large downloads through it were truncated (#8062). Expose the value as -idleTimeout (default 10, unchanged), matching the existing -idleTimeout flags on the volume and s3 servers, and wire filer.idleTimeout through 'weed server' and 'weed mini' too. A nil option keeps the 10s default so embedders constructing FilerOptions directly keep working. * filer: apply the idle timeout to the readonly port too The -port.readonly listener ran without any connection timeout while the main API port timed out. Give it the same -idleTimeout so both ports behave identically. * webdav: make the HTTP connection idle timeout configurable The WebDAV server used the same hardcoded 10s activity timeout. Expose it as -idleTimeout (default 10, unchanged), including -webdav.idleTimeout when embedded via 'weed filer -webdav', 'weed server -webdav', and 'weed mini'. * iam: make the HTTP connection idle timeout configurable The IAM API server used the same hardcoded 10s activity timeout. Expose it as -idleTimeout (default 10, unchanged), including -iam.idleTimeout when embedded via 'weed filer -iam' and 'weed server -iam'. * sftp: make the connection idle timeout configurable The SFTP server used the same hardcoded 10s activity timeout. Expose it as -idleTimeout (default 10, unchanged), including -sftp.idleTimeout when embedded via 'weed filer -sftp' and 'weed server -sftp'. * command: clamp idle timeout seconds before converting to duration A seconds value past time.Duration's range would overflow on multiplication and silently disable the timeout. Bound it in one shared helper, which also deduplicates the nil-default handling at the four call sites. * command: return zero for non-positive idle timeout seconds A large negative seconds value could wrap on multiplication and come out positive. Guard before converting so non-positive input always disables the timeout. * command: apply idle timeout to local unix socket listeners The unix-socket listeners in filer, s3, and sftp served accepted connections with no activity deadline while the TCP listeners already wrapped conns in util.Listener. Wrap the socket listeners the same way so a positive idleTimeout also bounds idle socket connections. * command: clamp s3 idle timeout seconds before multiplying s3 still multiplied the flag inline, so an out-of-range value could overflow time.Duration. Route it through idleTimeoutSeconds like the other commands; non-positive keeps meaning disabled. * volume: clamp idle connection timeout seconds before converting * command: cover idleTimeoutSeconds conversion semantics * command: gate idleTimeoutSeconds clamp check on 64-bit int |
||
|
Chris Lu
|
6536fac9bd |
s3: answer NoSuchBucket for encryption and policy writes to a missing bucket (#12133)
* s3: answer NoSuchBucket for encryption and policy writes to a missing bucket PutBucketEncryption, DeleteBucketEncryption, and PutBucketPolicy surfaced the filer's bucket-miss as InternalError; GetBucketEncryption masked it as ServerSideEncryptionConfigurationNotFoundError. Map the miss to NoSuchBucket instead, matching AWS. Internal readers still treat a missing bucket as "no encryption configured" so uploads keep their behavior. * s3: wrap ErrBucketNotFound only for verified misses in bucket policy ops The filer lookup helpers in get/set/deleteBucketPolicy marked every LookupDirectoryEntry failure as bucket-not-found, so a transient filer outage answered NoSuchBucket for a live bucket. Route them through filer_pb.LookupEntry, which normalizes only true misses to ErrNotFound. * s3: answer NoSuchBucket when the policy delete's own write loses the bucket * s3: classify a filer NotFound on the policy UpdateEntry as ErrBucketNotFound * s3: require the bucket entry to exist on config patch transactions * s3: answer NoSuchBucket when tagging and CORS writes lose the bucket mid-request * s3: keep the existence check on no-op bucket config patches An unchanged config write produced an empty patch that early-returned before the transaction, so a bucket deleted between the read and the write still reported success. Ship the condition-only transaction so IF_EXISTS runs on its own. * s3: guard the iam step and cover failure-vs-miss classification Nil-guard s3a.iam in the bucket policy handlers so the handlers do not panic when no IAM server is wired (unit tests, embedded servers), treat a "precondition failed" response string the same as the error code when mapping a failed patch to ErrNotFound, and add handler-level tests that pin the miss/failure distinction: missing buckets and vanished mid-write buckets answer 404 NoSuchBucket while transient filer failures stay 500 InternalError. |
||
|
Chris Lu
|
4c9e62177d |
s3: keep multipart part encryption on the initiate-time decision (#12134)
* s3: keep multipart part encryption on the initiate-time decision putToFiler applied the *current* bucket default encryption when writing a staged part, so a PutBucketEncryption landing between two UploadPart calls produced an object whose parts disagree: some chunks ciphertext, some plaintext. Completion then marks the object by its majority and GetObject fails decrypting the rest. Parts staged under .uploads now skip the bucket-default lookup; the decision snapshotted into the upload entry at CreateMultipartUpload is replayed through request headers instead, which matches AWS (default encryption applies at initiation). * s3: mark multipart part writes via request context, not the staged path Inferring "is a part" from the .uploads file path let an ordinary PutObject named .uploads/x.part skip bucket default encryption entirely. The part decision now travels on the request context from the two callers that stage parts (UploadPart, UploadPartCopy); a client-chosen key cannot reach it. |
||
|
Chris LuandDevin
|
660a826eab |
remote_storage/s3: send Content-MD5 on RemoveDirectory DeleteObjects (#12129)
S3DisableContentMD5Validation skips the request header that DeleteObjects requires, so AWS rejects every directory removal and the filer remote sync offset stalls. Issue the delete through DeleteObjectsRequest and re-enable the flag on that request's config copy so the SDK's own checksum handler adds the header without changing any other operation. Generated with [Devin](https://devin.ai) Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
7a47c63d54 |
volume: replicate FetchAndWriteNeedle writes over the configured TLS scheme (#12128)
* volume: replicate FetchAndWriteNeedle writes with the configured TLS scheme The guarded upload path hardcoded an http:// URL and used a bare *http.Client, so a peer fronted by https rejected the replica write and read-through of a remote-evicted object failed whenever the volume had replicas. Normalize the upload URL through the shared global client and let the guarded transport borrow its https.client TLS configuration, the same settings the regular replication path resolves. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: keep the global HTTPS downgrade guard on replica uploads Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: cover TLS replicate uploads and the redirect downgrade guard Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: pin the downgrade guard against a dialable plaintext target The previous redirect pointed at 169.254.169.254, which the guarded dialer already refuses, so the test passed even with CheckRedirect removed. Redirect to a live plaintext server on a dialable peer address and assert it never receives the resent body. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume-rs: keep the HTTPS downgrade redirect guard on outbound writes Mirrors the Go peer upload client: the shared outgoing HTTP client now stops redirects that would downgrade an https request to plaintext, so replicate uploads to TLS peers cannot leak data or auth headers via a 307 to http. * volume-rs: fail rejected redirects instead of returning the 30x response Go's rejectHttpsDowngrade uses ErrUseLastResponse, which surfaces to the caller as a non-2xx status it rejects; attempt.stop() returned the redirect response as Ok, so a refused downgrade could masquerade as a completed write to any caller that does not check the status. * volume-rs: fail replica writes on a non-2xx status fetch_and_write_needle discarded the replica response without checking its status, so a peer rejection — or a redirect the client refused to follow — was reported back as a successful copy while no replica was written. Go's doUploadData fails on any non-success status. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Hector Peeters
|
c46a42694b |
Fix s3 cancelled read relookup (#12130)
* filer: keep cached volume locations when the reader cancels * wdclient: do not count a caller's cancelled lookup as a filer failure |
||
|
|
a5036f568f |
webdav: refuse MKCOL and PUT under a missing folder with 409 (#12122)
* webdav: refuse MKCOL and PUT under a missing folder with 409 RFC 4918 9.3 and 9.7.1 say a MKCOL or PUT whose parent collection does not exist must fail with 409 Conflict. The filer creates missing parents, so both succeeded and made the intermediate folders. Check the parent in Mkdir and in OpenFile with O_CREATE and return os.ErrNotExist, which x/net/webdav turns into 409. * webdav: check the PUT parent before an overwrite deletes the target OpenFile deleted an existing destination before the new parent lookup, so a PUT whose parent check failed left the old file gone. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * webdav: report a missing rename source as not-exist Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * webdav: refuse MOVE to a missing destination folder AtomicRenameEntry grows any missing parents itself, so a MOVE to /missing/file created /missing; check the destination parent first. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * webdav: answer 409 for COPY and MOVE under a missing collection The webdav handler reports a failed directory Mkdir or Rename as 403, so the destination parent is checked at the request layer instead. Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * webdav: defer foreign-host MOVE and COPY destinations to the 502 handler * webdav: only check the destination parent for absolute paths * webdav: cover foreign-host and relative MOVE destinations in the guard test --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Jay Hemnani
|
5c14e566bc |
s3: allow bucket ownership controls when auth is off (#12121)
checkAccessByOwnership compared the caller's account with the bucket owner even with no identities configured, so Get/Put/Delete BucketOwnershipControls answered AccessDenied to every request. With auth off there is no owner to compare against, as the other auth-off checks already assume. |
||
|
|
80f3cc0613 |
topology: keep oversized volumes out of the writable list across re-registration (#12120)
* topology: keep the oversized verdict after the location list is gone The per-replica oversized marks live in the volume's location list, which is deleted when its last location leaves and recreated flagless on the next registration. Between that recreation and the next heartbeat that re-measures the volume, ensureCorrectWritables saw no mark and re-admitted an oversized volume to the writable list; two other paths (quarantine release, mark-writable) never consulted the marks at all. VolumeLayout now remembers the oversized verdict per vid, written only by heartbeat sizes and forgotten only when a report measures the volume under the limit. The writable admission gate itself refuses a marked volume, and the vacuum commit path drops a verdict its live size check has just disproved. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: mark oversized at registration when the report carries a size RegisterVolumeLayout ran EnsureCorrectWritables while the recreated location list still held no per-replica marks. When the registering report knows the volume's size, refresh the oversized state before the writable check instead of waiting for the next heartbeat pass. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: test that oversized volumes stay out of writables across re-registration Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: remember the oversized verdict per replica A shared per-volume verdict was cleared by any single replica's under-limit report, so a still-oversized replica whose location entry had died could re-register from a size-less report and put the volume back on the writable list. Keep one verdict per reporting replica: an under-limit report clears only its own, and a replica that never returns cannot pin its volume out of writables forever because the verdict ages out. The vacuum commit now feeds the node's fresh answer through the same per-replica bookkeeping instead of erasing the whole verdict. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: count a volume at the size limit as full on vacuum commit The layout's oversized test is >= the limit, but the vacuum commit and status checks used >, so a replica sitting exactly at the limit passed as not-full and the commit path could re-admit it. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: cover per-replica oversized verdicts in tests One replica shrinking must not clear a still-oversized replica's remembered verdict; cover that plus a size-less re-registration before the second replica is re-measured. The vacuum-commit case now shrinks the stored record first, since the commit re-derives the mark from the node's own answer. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * topology: judge the vacuum commit by the size it just measured SetVolumeAvailable rebuilt the oversized verdict from the cached VolumeInfo, which still carries the pre-compaction size until the next heartbeat. A commit that shrank a volume under the limit re-marked the replica oversized and never restored writes. The commit path already asks every replica for its fresh size; pass it through per replica and record it on the mark, the verdict, and the size tracking seed. * topology: hold an expired oversized verdict while its replica is back The TTL forgives a replica that never returns, but a remount announces the volume without a size and registration skips the oversized check. Deleting the verdict on expiry then re-admitted a still-oversized volume until the next size-bearing heartbeat. Keep counting an expired verdict while its replica sits in the live location list, and keep sweeping verdicts whose replicas stayed gone. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: chrislusf <chrislusf@gmail.com> |
||
|
Chris LuandDevin
|
d182be6a9e |
s3api: cap SOSAPI capacity by the reported physical disk bytes (#12118)
calculateClusterCapacity multiplies advertised volume slots by the volume size limit. Slot counts are an admission bound rather than a capacity measurement — an explicit -max can be set higher than the volumes a disk actually fills — so a small disk can report more total and available capacity than it physically has. Bound both numbers by the per-disk filesystem counters the volume server already reports in its heartbeat, falling back to the slot estimate only when they are absent. Generated with [Devin](https://devin.ai) Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
0ea818a44f |
mount: drain OFD locks when a file description is released (#12117)
* mount: drain OFD locks when a file description is released The kernel scopes OFD fcntl locks to the open file description but reports no lock_owner for them at RELEASE — it only sends one for flock holders — so an OFD write lock taken on a file handle outlived the holder and blocked every later open of that file (e.g. etcd failing to reopen its WAL after a restart). The mount now records the owner of each granted non-flock lock under its file handle and releases those owners on RELEASE, which also covers any POSIX locks left over when FLUSH could not run. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * mount: give every open its own fh value over the shared handle AcquireFileHandle reuses one FileHandle per inode, so every open reported the same fh to the kernel. A RELEASE for one description could then not be told apart from a sibling still holding the inode open — draining lock owners recorded under the shared fh dropped locks belonging to descriptions that stay open. Each open now registers its own fh value in fh2inode, all resolving to the shared FileHandle, and RELEASE removes exactly the id that is closing. This also stops a duplicated RELEASE on one id from double-decrementing the open count. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * mount: forget flushed lock owners so a later release cannot re-drop locks POSIX lock owners are process-scoped: Flush releases every lock the process holds on the inode, but the owner stayed recorded under each open's fh for the Release-time drain. If the process took a fresh lock through another open before an old description finished closing, the stale record made that RELEASE drop the new lock too. Flush now drops the released owner from every recorded open of the inode, so the Release drain only ever fires for owners whose locks were never flushed. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * mount: keep close-time lock owner records when the release fails A routed POSIX owner release that errors keeps the flush hint for a retry, but Flush also dropped the owner from every open's release records unconditionally — so the final Release skipped the retry and the owner filer's lock was stranded until lease reaping. Only forget the owner once its locks were actually released. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
b747e0d3b1 |
cluster: bound each DistributedLock attempt with a timeout (#12113)
* cluster: bound each DistributedLock attempt with a timeout An unreachable lock owner blocked the attempt on the connection timeout. Bound it so the write fails fast; a timed-out attempt keeps no renew token. * cluster: keep renewToken across failed lock attempts A request may have renewed the lease server-side before its response was lost, so the last token can still be valid on the next attempt. Clearing it on any transport failure strands a live lease until its TTL expires. A definitive rejection still clears the token through the response's empty RenewToken, and falling back to the seed filer no longer abandons the lease either. * cluster: bound each lock attempt by the remaining route window The 15s unavailability window started only after the first failure, so three 10s attempts plus sleeps outlasted it: the third failure selected the seed fallback at the same moment the loop gave up, and a reachable seed route was never tried. Start the clock before the first attempt and cap each attempt at what the window has left. When the window is spent on a non-seed target, fall back to the seed filer — it can forward to the real owner — and give that route its own window, so the fallback is always exercised before acquisition fails. * cluster: bound DistributedUnlock by the lock RPC timeout The unlock used context.Background(), so Stop against an unreachable lock server could wait on the TCP connect timeout far past the renewal goroutine's bounded shutdown wait. A timed-out unlock is harmless: the lease expires via its TTL regardless. * dlm: replay the current token on a lost renewal's retry A renewal rotates the token server-side before the response is sent. When that response is lost the client only knows the superseded token: retries mismatched, the token was cleared, and the holder conflicted with its own lease until it expired. The lock now remembers the superseded token, so a retry from the same owner replays the current token, and a superseded token can still release its own lock. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * cluster: give the seed lock fallback one fresh window per acquisition The seed can redirect back to a dead primary via LockHostMovedTo, so refilling the retry window on every return to the seed made the primary->seed->primary cycle unbounded. The first seed fallback keeps its fresh window; later returns to the seed keep the running deadline. * cluster: advance the lock seq when a lost-renewal replay extends the lease A retry of a renewal whose response was lost updates ExpiredAtNs in place but returned the existing Seq, so the replicated extension was rejected by InsertBackupLock as a stale mutation. A promoted backup would then grant the lock when the old expiry passed even though the holder still believes the lease is renewed. Bump Seq on the replay so the backup accepts the new expiry. * cluster: carry the superseded lock token through replication and transfer A renewal rotates the lock token and keeps the superseded one in PrevToken so a holder whose renewal response was lost can still recover. That recovery broke as soon as the lock moved: neither the replicated backup copy nor a transferred lock carried PrevToken, so a new primary rejected retries of the old token until the lease expired. Add prev_renew_token to the Lock and ReplicateLockRequest messages and thread it through ReplicateLock, TransferLocks, InsertBackupLock and InsertLock. * cluster: count clean lock redirects against the retry window A bare LockHostMovedTo response (no error, no token) returns nil from attemptToLockWithin, and retryUntilLocked reset the retry window on every nil result. Two ring-disagreeing filers forwarding to each other therefore bounced the client in an unbounded loop; the spent-window exit could also return a nil error with no token, handing callers an unacquired lock. * cluster: do not mark a lock held or hot-loop on tokenless attempts A bare LockHostMovedTo redirect returns nil with no renewToken; the attempt previously flipped isLocked to 1 anyway and retried immediately. Gate the flag on a granted token and pace the retry like other outcomes. * cluster: keep renewToken across bare lock redirects A LockHostMovedTo response carries no token and no error; overwriting renewToken with it dropped a held token, so the next renewal re-acquired fresh and could contend against its own still-live lease until TTL. * cluster: poll for replicated backup state in lock tests Replication runs in a goroutine, so a fixed sleep followed by an immediate backup read is racy. Wait for the expected backup state with require.Eventually instead. * cluster: mark a lock held only on a grant verdict A bare LockHostMovedTo redirect with a saved token left isLocked set without the redirected filer granting anything, and retryUntilLocked could exit on a stale token with no attempt made. doLock now reports whether the response was a grant, and both the retry loop and isLocked follow that verdict. --------- Co-authored-by: chrislusf <chrislusf@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
ee315527cb |
vector: address remaining s3vectors review findings (#12115)
* vector: address remaining s3vectors review findings Catalog and permission model: - Vector buckets are declared Lance table buckets end to end: create sets format=lance, get/list/delete reject or skip ICEBERG buckets, and the format lookup runs on an internal trusted catalog so callers need no GetTableBucket grant. - The implicit "vector" index namespace is created through the same internal manager, so a CreateTable grant alone authorizes CreateIndex; typed S3TablesError codes replace string matching on err.Error(). - CreateIndex pre-checks existence after the authorize, tags the record with a create nonce, and rolls back on engine failure under context.WithoutCancel with a versionToken CAS, so a retry never deletes a table it did not create. - DeleteIndex drops the catalog record directly; DropIndex is unimplemented on the shared VectorService. - DeleteVectorBucket removes the empty internal namespace on BucketNotEmpty and retries once. - PutVectors/DeleteVectors require the update grant, and engine gRPC errors map to typed API errors instead of generic 500s. API correctness: - CreateVectorBucket/CreateIndex return their resource ARNs. - ARNs reject empty bucket segments; vectorBucketArn must be an ARN. - ListIndexes returns summaries built only from catalog fields. - ListVectors honors maxResults/nextToken inside a segment via "seg:<pageStart>:<nextIndex>" resume tokens and validates segmentCount/segmentIndex. - QueryVectors requires positive topK; PutVectors rejects empty keys. - CreateVectorBucket rejects a supplied encryptionConfiguration rather than silently dropping it. - Metadata filters emit Lance's real JSONB UDFs: json_get_string/ json_get_float/json_get_bool picked by operand type, json_exists with a bracketed JSONPath, and per-member accessors for $in/$nin. Engine: - weed-vector-server stores the metadata column as Lance JSONB (lance.json extension) so the json_* UDFs apply; reads accept legacy Utf8 JSON columns. Vectors port TLS: - The TLS config is built before binding listeners, loads -cacert.file into ClientCAs when client-cert verification is on, fails startup on an empty/invalid CA file, and closes the gRPC conn on every failure path. * s3vectors: authorize index create before writing the implicit namespace A denied caller could still leave a persistent namespace behind because ensureIndexNamespace wrote through the trusted catalog before the caller's create grant ran. AuthorizeCreateTable now tolerates the not-yet-declared implicit namespace (bucket policy alone decides) and runs before any write; real table creates keep requiring the namespace. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3vectors: keep engine error details out of the API response Unavailable now maps to 503 ServiceUnavailableException and unclassified codes return a generic message; the real gRPC error stays in the log. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3vectors: exact-type metadata filters + NULL-safe $nin The jsonb extractors convert numeric-looking strings through json_get_float and 'true'/'false' through json_get_bool, so a stored '3' could satisfy a filter for the number 3. Numeric and boolean comparisons now also require the stored value not be a string, and each $in/$nin group coalesces NULL so NOT cannot invert NULL into NULL. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: validate the vectors TLS key pair before binding The reloading certificate provider reads the files asynchronously, so a missing or mismatched pair would only surface as failed handshakes on an announced HTTPS port. Load it once up front. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: make a token-checked table delete atomic The version token was read in one filer call and enforced in another, so a delete+recreate slipping between them let the delete erase the replacement. When a token is supplied the name path now carries the observed mtime as an if_not_modified_after precondition (evaluated under the path lock), and a decoupled delete strips the catalog marker under a compare of the captured attributes before touching the data path. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3vectors: compare filter operands against the stored JSON type jsonb's strict converters make json_get_string return text for numbers and booleans too, so a get_string IS NULL guard is never true for scalar metadata — numeric and boolean comparisons silently matched nothing. Numbers and booleans now compare through TRY_CAST of the serialized json_extract text, which yields NULL on any other stored type; string comparisons stay on json_get_string behind a CASE guard so the extractor only runs on rows storing a JSON string. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: allocate implicit namespace metadata before use A first table create in an undeclared namespace took the allowMissingNamespace branch with a nil record and panicked on the owner assignment. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: read the delete target's metadata and snapshot in one lookup Two separate lookups let a delete+recreate in between validate the caller token against the old record while the conditional delete aimed at the replacement. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: assert the captured record before a conditional table delete An xattr update keeps Mtime, so the IfNotModifiedAfterNs guard alone could not see a metadata write that had already rotated the version token, and a recreate preserving the mtime could slip the mtime check. The delete now first proves the entry still carries the captured attributes under the filer's path lock, then keeps the mtime precondition. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3vectors: match only JSON boolean literals in metadata filters * filer,s3tables: delete versioned tables under one attribute precondition * s3vectors: compare serialized JSON for string filter operands json_get_string converts numeric-looking strings and booleans, so it cannot serve exact-typed string comparisons even behind a guard. String operands now compare the serialized json_extract text directly, which is exact-typed and never errors; a Rust test pins the semantics against a mixed-type dataset. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3vectors: compare boolean filter operands as serialized text Arrow's boolean cast accepts '1' and '0', so a TRY_CAST column let a stored number satisfy a boolean filter. Booleans now compare the serialized json_extract text like strings do — a JSON boolean only ever serializes as 'true' or 'false'. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: claim a catalog name's markers in one compare-checked write A create wrote its markers attribute-by-attribute after checking the name was free. A delete's restore landing in the check-to-write gap put the old markers back first, and the create then overwrote only the ones it sets — leaving the old tags, policy, and maintenance settings on the new table. CreateTable, RegisterTable, and CreateView now write all their markers in a single update that refuses to run while any catalog marker is present, so the second writer loses with a conflict instead of inheriting them. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: test the attribute-preconditioned table delete and name claim Adds a BeforeDelete hook plus ExpectedExtended evaluation to the in-memory filer so tests can land a competing write in a handler's check-to-delete window, and covers two losing races: a token delete whose record changed since the read, and a create colliding with a delete's marker restore. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: reserve a catalog record under one compare-checked claim claimCatalogRecord writes a deleting marker onto the record while it still holds exactly the caller's captured attributes, so a delete or rename can hold the record through a multi-step operation and a losing writer backs out before any data or destination entry is written. The claim key joins the markers a name-claim asserts absent, so the name cannot be reused while the reservation stands. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: claim the record before purging or publishing past it A token delete of a decoupled table purged the data first and checked the marker afterwards, so a rename that finished in between left the moved record pointing at purged data while the delete reported a conflict. The delete now claims the record under the captured attributes before any data is removed and releases the claim if the purge fails, keeping the record intact for a retry. The rename claims the source before publishing the destination, so a losing rename backs out while no destination entry exists instead of leaving one behind pointing at data a delete went on to remove. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: test the record claim across delete and rename races A rename completing in the token delete's check-to-claim window turns the delete into a conflict with the data intact, and a delete's claim on the source turns the rename into a conflict before the destination entry is published. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
bfb6e40fb4 |
test: assert the resumed segment token in ListVectors paging (#12108)
test: assert the resumed segment token The second listSegmentVectors result was never checked (staticcheck SA4006); assert it resumes scanning at row 19 after matches 10..18. |
||
|
Chris LuandDevin
|
67240f538d |
filer.remote.gateway: stop zero-byte uploads of remote-only entries (#12110)
* filer.remote.gateway: skip uploading a dropped local copy on update The bucketed update path uploaded the chunkless entry whenever the update was not metadata-only. remote.uncache, eviction, and remote.meta.sync drop a synced entry's chunks and clear its sync stamp, which reads as dirty when the remote mtime shares the entry's second; the upload then replaced the object that is the entry's only content with zero bytes. Skip the upload the same way the mounted-directory sync does. * filer.remote.gateway: copy a remote-only entry's object on rename A rename deleted the source object and uploaded the chunkless entry, replacing the entry's only content with zero bytes — the same loss remote.uncache caused on plain updates. Complete the rename on the remote the way the mounted-directory sync does: copy the source object over (or take the destination when it already describes the entry), restamp so reads stop resolving through the pending-source marker, and remove the source only while it is still the object the stamp describes. A remote-only entry renamed out of the gateway's coverage keeps its object, and the plain rename's old-key delete only removes the object the event's stamp still describes. * filer.remote.gateway: scope the sync-stamp skip to same-path updates shouldSendToRemote reads the entry's sync stamp, which still covers a rename's unchanged mtime, so a synced or pending-stamped rename event was skipped before its copy or upload could run. Only updates in place may skip on the stamp. * filer.remote.gateway: keep the source a pending rename moved on from A rename event whose pending destination has since been renamed again still names the source a later event reads through its own marker; deleting the old key here loses the only copy. Mirror the remote.sync check: leave the source when the event's entry carries the pending-source marker and no entry holds the destination any more. * filer.remote.gateway: release a rename's pending source once complete A rename whose old entry read through a pending-source marker leaves that earlier source behind: the destination is released by its own key, never the marker's. Mirror releaseRenamedSource — once the renamed entry no longer reads through a marker, delete the pending source guarded by the old entry's stamp, keeping it while a remote-only entry holds its path. * filer.remote.gateway: finalize a complete rename before failing its source A rename source outside the gateway's coverage errored before the destination was checked, and the recorded failure held the event offset. Stat the destination first: when it already holds the content the entry describes, refresh the stamp and leave the unmanaged source untouched; fail only when nothing holds the content. * filer.remote.sync: document the source-of-truth protocol both processors share --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
e2d899ac25 |
test/vectors: fix the S3 Vectors e2e harness (#12112)
* test/vectors: give the vector server static storage credentials The e2e harness ran weed-vector-server without credentials, so lance's AWS object store fell back to IMDS and CreateIndex died on the runner's 411 from the token endpoint. Point it at the test credentials the s3 gateway now reads from s3.json. * vector: take the list item field from the schema when batching vectors Lance stores the fixed-size list item as nullable even when the write schema declared it non-null, so PutVectors built batches that no longer matched: expected FixedSizeList(4 x Float32) but found FixedSizeList(4 x non-null Float32). * vector: store index metadata as lance jsonb so filters resolve The metadata field was plain Utf8, but the filter functions lance's planner registers (json_get_string, json_get_float, json_exists) take a LargeBinary jsonb column, so QueryVectors with a metadata filter failed to plan. Declare the field with the arrow.json extension — lance encodes it to jsonb on write and back on read — and encode the batch's strings when the stored column is LargeBinary. Also rename the filter's json_get_str call to json_get_string, the name lance registers. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin
|
a9d6bfc51f |
test: boto3 s3vectors conformance harness (P1) (#12105)
* vector: boto3 s3vectors conformance harness test/vectors spins a mini cluster, an s3 gateway with -port.vectors and -vectors.grpc, and a weed-vector-server, then exercises the AWS client end to end: bucket and index lifecycle, put/get/list/delete vectors, and a filtered query asserting distances and metadata. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: run the s3vectors conformance subset An e2e job in rust-vector-tests: builds weed and weed-vector-server, then runs test/vectors against a mini cluster plus the vectors gateway. The workflow now also watches vector.proto, its stubs, and test/vectors. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: fix e2e protoc and codespell findings The e2e job needed the same vendored-protoc step as the unit-test job — lance's build scripts locate protoc themselves. Codespell flags the AWS `$nin` operator literal and the `lits` variable: exempt `nin` in .codespellrc like the existing word exemptions, rename the variable, and fold the negation into the IN/NOT IN symbol directly. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: classify catalog errors by code, not message text * s3: drop the implicit vector namespace on vector-bucket delete * test: point the vector engine at the gateway lance port * test: bind the vectors gateway listeners to loopback * test: make the filter and distance assertions load-bearing * test: rebuild the harness binaries on every run * ci: fix the vectors end-to-end job * ci: run the vector tests on gateway and command changes --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
c55b7c5a29 |
s3: fix CreateIndex metadata panic and ListVectors segmented paging (#12106)
* s3: nil-check metadataConfiguration before reading its keys * s3: honor maxResults/nextToken in segmented ListVectors * s3: exclude non-Lance buckets from ListVectorBuckets * s3: test segmented ListVectors paging token math * ci: silence codespell on the "$nin" operator and rename lits |
||
|
Chris LuandDevin
|
dcc94f818d |
vector: S3 Vectors API gateway (P1) (#12103)
* vector: S3 Vectors API gateway over VectorService weed/s3api/s3vectors serves the AWS S3 Vectors JSON API: bucket CRUD maps to S3 Tables table buckets, index CRUD to Lance table records plus the VectorService CreateIndex/DropIndex RPCs, and the data plane (Put/Get/List/Delete/QueryVectors) calls the shared service. The AWS filter document translates to a Lance SQL predicate over the metadata JSON column; unsupported operators fail loudly rather than overmatch. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: -port.vectors and -vectors.grpc on weed s3 Mirror the Lance server startup: the S3 Vectors API binds its own port and dials the weed-vector-server gRPC address; both disabled by default. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: resolve index ARNs, authorize index ops, fix catalog fields - vector buckets declare FormatLance so a new bucket accepts index tables instead of defaulting to ICEBERG. - ListIndexes decodes the `tables` response field; `tableSummaries` hid every index. - resolveIndexARNs honors indexArn alone; handlers no longer treat indexName as the only carrier. - data-plane RPCs fetch the index record first, so callers authorize against the index resource and a missing index 404s before touching the engine; QueryVectors reports the index's configured metric. - IndexArn added to the index-level request types. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: serve the vectors port over TLS when configured When -key.file/-cert.file are set, the vectors listener mirrors the main API and serves TLS rather than a second plaintext endpoint. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: harden the s3vectors gateway handlers - $ne no longer shadows the operator: exclusion filters emit <> instead of equality. - a null element in a PutVectors list fails the request instead of panicking. - requests are capped at 32 MiB; distances return via *float32 so an exact match still reports 0 when returnDistance is set. - CreateIndex declares the `vector` namespace first and validates dimension (1..=4096) before the catalog write; a failed VectorService.CreateIndex rolls the table record back. - non-filterable metadata keys persist as JSON rather than a comma-joined string, and index records carry their createdAt. - ListVectors honors segmentIndex/segmentCount by slicing the full listing in the gateway. - catalog errors map to AccessDeniedException/ConflictException instead of a blanket ValidationException, and malformed ARNs 400 instead of slicing. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: /{Operation} routes and vectors-port TLS, conn cleanup Path-dispatched clients (POST /QueryVectors) reach the same handlers as X-Amz-Target calls. A failed listener bind now closes the vector gRPC connection instead of leaving it open with the endpoint disabled. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Jay Hemnanianduser.email
|
1e7afbacf4 |
s3: list every bucket when auth is off (#12077)
* s3: list every bucket when auth is off With no identities configured, every request may create, read and write any bucket, but ListBuckets treated the caller as an unauthenticated user and returned an empty list, so `aws s3 ls` showed nothing. List all buckets in that mode, with prefix, max-buckets and continuation tokens as before. Listing with auth on is unchanged. * s3: cover the auth-on anonymous ListBuckets case The no-auth listing change also moves the nil-identity gate; pin the security-relevant direction that a request without an identity still sees no buckets once auth enforcement is on. * s3: verify ListBuckets continuation token reaches the next page --------- Co-authored-by: user.email <chrislusf@users.noreply.github.com> |
||
|
Chris LuandDevin
|
d771a91571 |
vector: VectorService gRPC contract (P1) (#12101)
* vector: VectorService protobuf contract The shared RPC boundary between the Go S3 gateway and the Rust vector engine from the vector design doc: CreateIndex/DropIndex, Put/Get/ List/Delete/Query vectors. A Table reference carries the Lance table id and an optional pinned version, matching the snapshot discipline queries are required to keep. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: generated Go stubs for vector.proto protoc output via weed/pb/Makefile. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * vector: carry return_data through QueryVectors S3 Vectors QueryVectors honours returnData, but the contract had no way to ask for or carry vector data in a query response, so the gateway would need a second GetVectors round-trip per query. Add return_data to the request and data to QueryResult now, while the contract is still new. * vector: pin metadata numbers to double precision in the contract Metadata arrives as JSON, where numbers carry double precision; say so on the field so neither side promises integer fidelity past 2^53. * vector: clarify the query filter is raw Lance SQL over one metadata column The doc said 'metadata columns' but the table stores all metadata in a single JSON column named metadata; the gateway translates S3 Vectors filter expressions onto it. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris Lu
|
6b379badbb |
filer: shut down test filers' meta-log buffers, fixing the 32-bit OOM (#12104)
* filer: shut down test filers' meta-log buffers, fixing the 32-bit OOM Each log_buffer.NewLogBuffer preallocates five 8 MiB windows and spawns loopFlush/loopInterval goroutines that keep them reachable. The weed/filer tests built ~70 filers and aggregators that almost never called ShutdownLogBuffer, pinning ~2.8 GB plus live churn, and the linux/386 job ran out of its ~4 GiB address space inside newSealedBuffers. - MetaAggregator.Shutdown now retires MetaLogBuffer too: the "aggr" buffer's loops previously outlived the aggregator. - newTestFiler and newTestFilerWithStubStore register t.Cleanup shutdowns; the one-off filer/aggregator literals defer theirs. * filer: wait for the aggregated meta-log buffer to stop in Shutdown ShutdownLogBuffer only signals the stop; loopFlush and its pooled windows exit asynchronously. Match Filer.Shutdown's local-buffer sequence so the aggregator returns with the buffer's memory already released. * filer: cancel test filers' flush context before buffer shutdown The stub-store helpers build real NewFiler instances, so the queued shutdown flush retries appendToFile every ~737ms under flushCtx, which nothing ever cancelled — the buffer and its pool stayed alive past each test. Cancelling first lets the flush drop the window and WaitForShutdown confirms loopFlush exited. * filer: wait for test log buffers to drain after shutdown The stub-flush buffers cannot hang, but without WaitForShutdown their loopFlush goroutines release the pooled windows asynchronously — the memory this PR means to free is still counted as live while the next test allocates. |
||
|
Chris LuandDevin
|
72ac424198 |
filer: deliver meta-log events under their stream position, not their stamp (#12084)
* filer: deliver meta-log events under their stream position, not their stamp A mutation's event timestamp is stamped before the filer store write so the entry and its event share a commit time, while the log-buffer append happens after the write. A write delayed behind later-stamped mutations keeps its early stamp in the payload while AddDataToBuffer bumps LogEntry.TsNs to keep the stream monotone - so subscribers could receive events whose payload TsNs arrives out of order. Consumers treat the delivered TsNs as a stream position (cursors, watermarks, replay windows). The s3 lifecycle daily-run drains events in order and pins its per-shard cursor behind the first not-yet-due match; an event delivered with a stale stamp after the cursor already passed it is filtered out forever and its pending match silently never fires - the flaky test_lifecyclev2_expiration signature of exactly one object of a same-rule pair surviving, with errors=0 on every pass. Normalize event.TsNs to logEntry.TsNs at delivery (and for the skeleton-filtered heartbeat) so delivered timestamps are monotone by construction. The commit-time stamp remains available on the entries themselves via CommitTsNs. * pb,filer: stamp metadata events with a mutation timestamp TsNs on the wire now carries the delivery stream position, so the mutation's commit stamp needs its own field. Add mutation_ts_ns to SubscribeMetadataResponse and set it when the event is created. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: carry the mutation stamp through event delivery Delivery rewrites TsNs to the stream position; the payload stamp survives in MutationTsNs, backfilled from TsNs for payloads written before the field existed. The notification callback forwards it to the wire message. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * pb: normalize chunk-replay timestamps like eachLogEntryFn ReadLogFileRefs bypasses the server decode path, so a delayed append still arrived under its stale payload stamp and could hide behind a subscriber cursor. Deliver the stream position and preserve the stamp in MutationTsNs. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: order replay and tombstones by the mutation stamp Replay treated the delivered TsNs as the commit time, so a peer write delayed in its own log could overwrite a newer local entry. Compare and stamp with MutationTsNs, falling back to TsNs for older peers. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * remote.sync: carry the mutation stamp through reshaped renames Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * filer: test replay ordering on the mutation stamp * filer: tombstone remote deletes by delivery position, order by commit stamp * filer: keep the peer commit stamp when restamping aggregated events * pb: keep the log cursor on the entry position past callback restamps --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |