* weed/shell: support -noLock flag in volumeServer.evacuate (#12501)
* shell: return flag parse errors from volumeServer.evacuate
A typo such as -noLock=invalid printed the parse error but the command
still succeeded, so a piped script treated the evacuation as done.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: volumeServer.evacuate -h prints usage without failing
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* helm: set replication and disk on buckets created by the chart
* helm: test bucket replication and disk without ttl
* helm: reject unquoted bucket replication values
* helm: treat a null bucket replication as unset
A blank replication key in a values file failed the render with the
quoted-string error, while blank ttl and disk are skipped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* helm: require a three-digit bucket replication
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* helm: note that removing bucket settings keeps the stored rule
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* helm: reject bucket replication above 255
fs.configure rejects an xyz value above 255, so the hook Job failed after
the release rendered.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(shell): fs.log.purge purges every filer, not just one
Each filer keeps its own metadata logs in its own store, so purging a
single filer leaves the chunks referenced by the others and the volume
vacuum cannot reclaim anything.
The shell picked that single filer at random (commands.go:208) when no
-filer flag was given, and the admin script worker pinned it to
filers[0] (admin_script_handler.go:574), so the same command purged a
different filer on every run and never the whole cluster.
fs.log.purge now resolves the filer list from the master and purges each
of them, continuing when one is unreachable and reporting the failures
at the end. -filer=<addr>,... restricts the purge to a subset.
Also stops mutating the shared command environment per filer, so a
long-lived shell keeps its original filer.
* fix(shell): bound filer discovery and cover the cluster path
Address review feedback:
- resolveLogPurgeFilers waited on the master with context.Background(): a
shell whose master connection dropped would hang before purging
anything, where it used to purge the connected filer. Bound the master
round trip with a timeout and fall back to the connected filer when the
master cannot be reached or lists no filer.
- the tests only exercised the single-filer fallback. Rename that test to
what it actually covers and add one asserting every filer of the list is
attempted and the shared command environment keeps its original filer.
* shell: reject an fs.log.purge -filer list with no addresses
A comma or whitespace only -filer value fell through to master
discovery and purged every filer instead of the intended subset.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: fall back to the shell's filer without a second master lookup
When the master listed no filer, FilerAddress() ran filer discovery again
and could wait out the shell's ConnectTimeout before failing. Also cover
master discovery with the fake discovery master.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* shell: test the fs.log.purge filer loop against a fake filer
The previous test looped over the addresses itself, so it passed no
matter what Do did.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Martin Morez <martin@morez.org>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* filer: validate bucket names before the SQL DDL hooks
The SQL store builds CREATE TABLE and DROP TABLE statements by formatting
the bucket name into the query. The DML paths already reject names that
fail VerifyS3BucketName, but the bucket-creation and bucket-deletion
hooks ran the DDL without that check, so a name reaching them through the
filer or WebDAV directory path (which skips the check for dot-prefixed
directories) could inject stacked statements. Apply the same validation
the DML paths use.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* filer: check that a valid bucket runs both DDL statements
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* weed-db: end startup when password authentication fails
The failure branches returned the result of writing the error response,
which is nil on success, so the handshake carried on to ReadyForQuery.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-db: test startup with wrong and unknown credentials
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* weed-db: check the auth error code and fail on stray read errors in the startup test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* admin: escape file properties before rendering the modal
The Properties modal concatenated the entry name, path, MIME type and
extended attributes into HTML, so object metadata written over S3 could
inject markup into the admin page.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* admin: keep every property key when escaping
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* security: do not accept a path- or method-scoped filer token as admin
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* security: test that scoped filer tokens fail the admin token check
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: let a table renamed across namespaces keep committing
* s3tables: keep the same-directory exception off a directory another table shares
A record written before metadata locations were confined to their namespace
(6f93f874) and claimed (#12473) can keep its files in another table's
directory, in another namespace. The exception that lets a table renamed
across namespaces keep committing to the directory it left behind accepted
such a record's commits too, where master refused them.
The namespace check now runs first, as on master. Only a location it refuses
is matched against the record's current directory, and the exception holds
only when no other record keeps its files there. Where other records' entries
sit is ignored, so a table created under the renamed table's old name, whose
files go to a directory of its own, does not take the exception away. The
scan runs only on that path, so commits inside the namespace cost what they
did.
Co-Authored-By: Claude <noreply@anthropic.com>
* s3tables: seed the table at the renamed table's old name
A rename may keep the old name taken, so creating a table there can be
refused; seeding the record keeps the test on the update path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
filer: prefix bucket default collection, assign copy collection explicitly
S3 server-side copy paths (CopyObject/UploadPartCopy) assign volumes with an empty collection and relied on the filer's bucket fallback. The fallback used the raw bucket name, while PutObject uses getCollectionName(bucket) = <filerGroup>_<bucket>, so copies landed in a second, unprefixed collection.
Prefix the filer's bucket default with the filer group, and have the S3 copy path pass that collection explicitly, like PutObject. Direct filer writes keep path-rule precedence, so this only aligns S3 copy with S3 put.
Adds regression tests for the fallback, explicit collection precedence, and the S3 copy collection resolver.
* weed/s3api: avoid truncated marker rescan in ListObjects (#12540)
Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com>
* fix(s3): validate marker directory candidates
Respect prefix filters and internal-directory exclusions when recovering marker candidates. Restrict lookups to direct children, propagate read failures other than not-found, and deduplicate directory marker echoes.
Add regression coverage and model promoted objects consistently in the XML-unsafe prefix fixture.
---------
Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* s3vectors: bound body buffering before authentication
* docs: include vector engine installation and source builds
* s3vectors: prevent MIME sniffing of JSON error responses
* docs: load vector Cargo configuration when building from source
* docs: note the native protoc requirement for Windows vector builds
* docs: make vector source checkout and working directory explicit
* ci: handle deb822 sources during Ubuntu mirror failover
* ci: bound Ubuntu mirror attempts and reject partial index updates
* vector: expose request metrics and registration-aware health endpoints
* vector: report immutable read cache and storage fetch metrics
* docs: describe vector engine health, metrics, and maintenance
* vector: count canceled requests when their handler is dropped
* vector: account for uncached large immutable storage reads
* vector: reliably release the stalled credential reader in tests
* vector: observe large reads as their streamed body is consumed
* vector: bound heartbeats on established master registration streams
* test: exercise hardened vector maintenance on Go and Rust volumes
* vector: register engines in their configured filer group
* vector: make retried index creation preserve compatible datasets
* vector: allow registration to advertise a separate address
* test: release stalled credential readers without blocking FIFO cleanup
* test: allow hardened vector coverage in a named filer group
* vector: discover engines in the same filer group as their gateway
* vector: fall back when the advertised address is empty
* admin: expose vector readiness, maintenance, and management tools (#12520)
* s3vectors: share vector discovery with trusted management clients
* admin: show vector engine readiness and recent maintenance activity
* admin: suggest S3 Vectors IAM actions from the supported operations
* admin: manage vector indexes, policies, tags, encryption, and query data
* admin: validate SSE-S3 support before creating encrypted vector buckets
* admin: discover vector engines in their registration group
* admin: probe vector engines concurrently with independent time budgets
* admin: verify encryption capability on the remote S3 gateway
* admin: defer discovery until after the HTTP server can start
* admin: distinguish unknown vector health services from failed engines
* admin: count workers that can execute vector index maintenance
* admin: allow read-only users to query vectors and inspect tags
* vector: scope gateway and admin discovery to the configured filer group
* admin: refresh policy state after mutations and serialize policy controls
* vector: drain active RPCs before closing replaced discovery connections
* vector: respect request cancellation while waiting for discovery
* vector: prefer another registered engine after a transport failure
* admin: check all masters before reporting no registered vector engines
* iam: honor attached vector policies without broadening legacy grants
* vector: allow short-lived discovery clients to close their connections
* vector: refresh cached master addresses for each discovery lookup
* iam: enforce vector bucket tag conditions in attached policies
* iam: authorize vector bucket creation with the requested tags
* test: exercise vector IAM grants and tag denies on Go and Rust volumes
* admin: reject authenticated vector mutations without an admin role
* test: run vector IAM assertions before the shared engine is stopped
* vector: add native mini startup, shell tools, and a runnable Compose example (#12522)
* s3vectors: share vector discovery with trusted management clients
* admin: show vector engine readiness and recent maintenance activity
* admin: suggest S3 Vectors IAM actions from the supported operations
* admin: manage vector indexes, policies, tags, encryption, and query data
* admin: validate SSE-S3 support before creating encrypted vector buckets
* admin: discover vector engines in their registration group
* admin: probe vector engines concurrently with independent time budgets
* admin: verify encryption capability on the remote S3 gateway
* admin: defer discovery until after the HTTP server can start
* admin: distinguish unknown vector health services from failed engines
* admin: count workers that can execute vector index maintenance
* admin: allow read-only users to query vectors and inspect tags
* vector: scope gateway and admin discovery to the configured filer group
* admin: refresh policy state after mutations and serialize policy controls
* vector: drain active RPCs before closing replaced discovery connections
* vector: respect request cancellation while waiting for discovery
* vector: prefer another registered engine after a transport failure
* admin: check all masters before reporting no registered vector engines
* iam: honor attached vector policies without broadening legacy grants
* vector: allow short-lived discovery clients to close their connections
* vector: refresh cached master addresses for each discovery lookup
* iam: enforce vector bucket tag conditions in attached policies
* iam: authorize vector bucket creation with the requested tags
* test: exercise vector IAM grants and tag denies on Go and Rust volumes
* admin: reject authenticated vector mutations without an admin role
* test: run vector IAM assertions before the shared engine is stopped
* mini: optionally start and supervise the vector engine
* mini: bootstrap vector buckets with existing Lance catalog helpers
* shell: manage vector buckets, indexes, and vector data
* docs: provide a runnable vector and maintenance worker Compose stack
* shell: scope vector discovery to the session and close its connections
* mini: honor the bind address for vector and metrics listeners
* mini: reserve vector ports before assigning gRPC and compatibility listeners
* mini: wait for vector storage setup and master registration
* mini: create requested vector buckets without requiring the Lance HTTP listener
* shell: refresh master addresses during vector discovery
* mini: resolve sibling vector executables with platform extensions
* vector: preserve uncertain creates and fix mini listener allocation (#12525)
* vector: retain catalog records when index creation has an unknown outcome
* mini: keep chosen listener ports reserved throughout allocation
* mini: resolve hostname binds before starting vector listeners
* vector: fix integration test imports and recover interrupted index creation (#12533)
* shell: register vector operations from the command package
* s3vectors: resume interrupted index initialization safely
* mini: exclude disabled WebDAV from listener allocation
* s3vectors: retain uncertain creates across refused retries
* s3vectors: claim pending indexes before retrying initialization
* s3vectors: compare resolved distance metrics during recovery
* s3vectors: preserve metadata keys when retrying initialization
* s3vectors: verify ownership before initializing new catalog records
* helm: deploy vector engines with TLS, probes, and monitoring
* helm: provision vector buckets through the S3 Vectors API
* helm: bracket IPv6 pod addresses for vector listeners
* helm: discover vector engines through the all-in-one master
* helm: accept vector component NetworkPolicy overrides
* helm: bind vector metrics to the pod address on both IP families
* helm: provision vector buckets through embedded and all-in-one gateways
* docs: use the full Helm name for external vector certificates
* helm: render vector instance labels as strings
* helm: allow vector bucket hooks to use restricted security contexts
* helm: honor vector security context enable switches
* helm: prevent duplicate vector monitoring labels
* helm: reject metrics flags with the default pre-metrics vector image
* test: skip vector chart compatibility checks when Helm is unavailable
* test: pass installer scripts through stdin on Windows
* ci: allow cold Intel macOS vector builds to finish
* ci: retain completed Rust builds when later platform checks fail
* ci: build, test, and publish vector engines for macOS and Windows
* installer: install native vector archives on macOS and Windows
* ci: run desktop vector checks for shared Rust and protobuf changes
* ci: publish vector releases from a separate job with write permissions
* ci: build tagged vector releases without restoring caches
* ci: propagate Windows bash path conversion failures
* grafana: add vector API, engine, cache, and maintenance panels
* helm: include vector panels with Kubernetes metric selectors
* grafana: preserve cluster and namespace identity when matching engine health
* grafana: retain engine identity when scrapes fail
* grafana: show zero cache hits before the first hit
* grafana: keep cluster and namespace identity in vector cache ratios
* topology: list collections without nesting the collection map read lock
Items() already takes the read lock and copies the values. Holding it
around the call made ListCollections take the read lock twice, so a
DeleteCollection waiting for the write lock in between blocked the
second read lock and wedged the master.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* topology: test ListCollections against a concurrent collection delete
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* master: move the KeepConnected stream cleanup into removeClusterNode
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* master: drop a filer from the lock ring only with its last registration
A filer's replacement can register under the same address before the
master ends the old stream. The cluster list counts both registrations,
but the old stream's cleanup removed the address from the lock ring
outright, leaving the live filer out of the ring until it re-registered.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* master: order a stream's cluster list and lock ring changes
The cleanup of an ending stream could see its address removed from the
cluster list, then a new registration of that address add it to the list
and the ring, and only then remove it from the ring.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* master: take a leaving filer's reconnect out of the lock ring
A filer that reconnects while leaving registers with LeaveLockRing set.
Its broken stream can still be open on the master, and since that
stream's end no longer drops an address that is still registered, the
reconnect removes the address from the ring itself.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* helm: say what the Lance worker needs from security.toml
Since #12470 the Lance namespace authenticates weed-worker by a token
derived from jwt.filer_signing.key, which the chart renders only with
securityConfig.jwtSigning.filerWrite. Without it, a gateway with
identities lists the worker no tables, and the lance_* jobs silently
find nothing to do.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* helm: warn that filerWrite also requires signed filer HTTP writes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3: encrypt volume data when the filer is configured to
weed s3 read the filer's configuration at startup (buckets dir, filer
group, masters, metrics, maxMB) but not its -encryptVolumeData, so a
cluster that turned encryption on at the filer still had S3 uploads
stored plain, while uploads through the filer itself were encrypted.
The gateway now encrypts when either its own -encryptVolumeData or the
filer's is set; weed-vector, weed-worker and weed-vector-cli already
follow the filer's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* vector: weed-vector, weed-worker and the CLI take volume encryption from the filer
They already encrypted chunks for the volume servers when the filer's
configuration said so, or when their own --encryptVolumeData did. With
weed s3 following the filer too, the filer's -encryptVolumeData is the
one switch for every writer, so the three flags are removed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* S3 Vectors tests: turn volume encryption on at the filer
VECTOR_ENCRYPT_VOLUMES=1 set -encryptVolumeData on the gateway, the
engine and the worker. It now sets it on the filer alone, so the leg
checks that each of them follows the filer: the gateway's stored chunks
carry a cipher key (with a gateway that ignores the filer, the leg fails
on "the gateway stored a chunk without a cipher key"), and so do the
engine's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* weed-worker README: volume encryption follows the filer
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
* s3: help text says the filer's -encryptVolumeData turns S3 encryption on
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* S3 Vectors tests: plain legs check the engine stored no cipher key
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
vector: the engine's, CLI's and worker's help say what they do now
weed-vector's --masters and --filer-group said "reads through" and
--security-toml named only gRPC TLS, though the engine keeps tables on the
filers and volume servers and takes https.client, the read JWT key and the
SSE-S3 keys from it too; weed-vector-cli called itself a "query spike";
weed-worker's --filer-signing-key spoke of write credentials the
namespace no longer vends.
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
vector: weed-vector drops the ignored --namespace and --filer-signing-key
Since #12468 the gateway resolves every table it sends, so the engine has
no catalog to name or authenticate to; the two flags were kept only so an
older command line still started, with a warning. The direct-storage
engine is new, and release 4.52's command lines need changing anyway
(--storage-option is refused), so they are removed.
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
vector: refuse an SSE-S3 index when the engine has no key for it
The engine writes and reads an SSE-S3 bucket's files itself, so without
s3.sse.kek or s3.sse.key it failed inside Lance and the client saw only
InternalFailure. weed mini makes this easy to hit: it generates its key
for its own S3 server only. A table the gateway marks sse_s3 is now
refused up front with FailedPrecondition, which the gateway answers as
ValidationException with the reason.
Claude-Session: https://claude.ai/code/session_011UpvafvK8JpagF94MHWqwK
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3: don't report a version delete as done when the filer lookup failed
deleteSpecificObjectVersion treated any getEntry error as "entry is gone":
- the null-version precheck returned success without deleting anything,
- the recheck after a failed rm returned success while the entry may
still be there,
- a failed .versions lookup was read as "not the latest version", so the
latest blob was removed without repointing the pointer first.
Only filer_pb.ErrNotFound now counts as a miss. Other errors fail the
delete, so the S3 client or the lifecycle worker retries it. The lookups
go through retryFilerOp, like the other .versions reads in this file, so
a single transient filer error doesn't turn into a 500.
* s3: include recheck error when a version delete fails
* s3: recheck a failed version delete once instead of retrying
---------
Co-authored-by: Mansur Akhmadi <10739460+mansur-akhmadi@users.noreply.github.com>
* volume.fsck: batch and deduplicate filer entry deletions
In volume.fsck with -findMissingChunksInFiler and -reallyDeleteFilerEntries, deleting orphaned filer entries was slow and CPU intensive due to tiny worker channel buffers (128), lack of batch processing, redundant HTTP DELETE requests for multi-chunk files, and lock contention on parent directory tracking.
Optimize by:
- Adding startPurgeBatchWorkers with 4096-buffered channels to prevent producer stalls and batch opportunistic deletes up to 64 items.
- Deduplicating missing chunks per file path during volume checks so that multi-chunk files are queued once.
- Processing batches with in-batch path deduplication and aggregating parent directories before updating purgedDirs in a single lock acquisition.
- Buffering temporary fid file writes (64KB) during the traversal to avoid millions of unbuffered syscalls.
* volume.fsck: report a failed write of the filer fid files
The fid files are written through a buffer now, so the last part of
each one reaches the disk only in the deferred Flush. Its error was
dropped: a full temp disk left a short fid file and the collection
still succeeded, and the volume check would then take the chunks
missing from it for orphans.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume.fsck: retry a batched path whose earlier delete failed
A path was marked done before its delete ran, so a second copy of it in
the same batch was skipped after a transient failure and the entry
stayed. Only skip paths already deleted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume.fsck: keep reporting a failed DELETE response read
The non-verbose path now discards the body without checking the copy,
so a connection cut after the headers went unreported.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* volume.fsck: keep one purge worker starter and one delete function
startPurgeWorkers and the single-path httpDelete were left only as
wrappers the tests called. Give their names to the batch versions and
adapt the tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: give a table created without a location its own directory
An Iceberg table created without a location kept its files under its name
path, so a rename left them where the next table of that name would land.
Such a table now gets s3://<bucket>/<namespace>/<name>-<uuid>, stored as the
record's warehouseLocation and returned by GetTable; its first commit writes
there without moving the root. Records without one keep resolving to their
name path, so existing tables need no rewrite. Lance datasets stay at their
name path, where Lance clients and the vector engine look for them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: create tables and views in a directory of their own
A table or view created without a location took its name path, or a
unique one only when that path was already taken. It now always gets
<name>-<uuid>, so a later rename never leaves its files under a name
another table can take.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: load and commit a table at its warehouse location
A table created through S3 Tables without a location has no metadata until
its first commit; loading or committing it now uses the directory recorded
for it, and only a record without one falls back to its name path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: fail a stage-create whose marker is not saved
The marker is how the commit finds the location the stage-create handed
out. Without it the commit fell back to the name path while the client's
files sat under the returned location.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: expect tables created without a location in their own directory
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: rename a table that keeps its files elsewhere before reusing its name
Enterprise keeps the storage marker on a renamed table's files, so the name
of a table whose files sit under its name path stays taken there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* s3tables: remove the table directory when a create backs out
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: answer a create that lost the race with the registered table
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: answer a view create that lost the race with the registered view
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: fail a staged commit whose name another create registered
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: remove the directory of a create whose name another table took
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: keep the metadata of a staged commit whose name is taken
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: write the lakekeeper table's data under its warehouse location
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: commit a staged table at the location of its own stage
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: never continue another stage when a commit's own marker is gone
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* iceberg: test filer honors exclusive creates and token-guarded deletes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* ec_balance: make the per-rack cap on global moves a worker setting
The global load-balancing phase planned at most 10 moves per rack per
detection run, hard-coded. On a cluster where one disk per machine holds
several times its siblings' shards, that is about 20 moves an hour against
tens of thousands needed, however many moves the worker can execute.
Add global_max_moves_per_rack to the plugin worker config. The default stays
10, so nothing changes unless it is raised. It is clamped to 1..5000 because
the planner reads 0 as unlimited and would plan a whole drain in one run.
Claude-Session: https://claude.ai/code/session_011UpvafcfUuC2mLxNAXLQJi
* ec_balance: clamp global_max_moves_per_rack before narrowing it to int
A value past the int32 range made ReadIntConfig return the default, so a
request asking for more than the maximum got 10 moves instead of 5000.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec_balance: test the configured global cap through Detection
The helper test still passed if Detection handed the planner a literal cap.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec_balance: plan no more global moves per rack than the detection keeps
Each global move scans the source node's volumes, and the detection keeps
at most max_results moves. With the cap at 5000, three skewed racks of
20,000 volumes planned 15,000 moves to keep 500, taking about 70 s of the
300 s detection budget; bounded by max_results it takes about 5 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ec_balance: test the configured global cap with a result limit
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
