Allow normal post-tag proxy propagation before dispatching downstream releases, while preserving the check that prevents them from pinning the previous commit.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* rust volume: quick-repair redb on durable checkpoints
set_quick_repair(true) on the durable checkpoint transaction so an
OOM-killed volume server opens without a full-file repair scan.
* rust volume: optional redb insert_before bulk load
Behind redb-experimental-cursor (default off). Production binary
stays on sorted insert(). CI unit tests run both feature settings.
* rust volume: exercise insert_before across leaf splits
Replace the 5-key cfg clone with a 4000-key reverse-order rebuild
so CursorMut::insert_before hits page splits. CI runs the feature
only on storage::needle_map unit tests.
The star_history workflow referenced actions/setup-python@v8, which
does not exist, causing the workflow to fail at the "Set up job" step.
Pin to v7, matching the version used across the other workflows.
Change the matplotlib figure size from (10, 4) to (10, 6) so the
star history chart renders vertically longer in the README. The
regenerated note/star_history.svg reflects the new 5:3 aspect ratio
(720x432pt) instead of the previous flat 2.5:1 (720x288pt).
The "Mount and exercise" step left the weed.exe mini server and the final
WinFsp mount running when it exited. The next step's pwsh.exe then failed
with STATUS_DLL_INIT_FAILED (0xC0000142), failing a job whose actual test
step had passed. The same code passed on both the PR branch and the next
master run, so this was a transient launch failure — but it was caused by
an unclean environment and made fatal by a diagnostic step.
Tear down all weed.exe processes at the end of the test step so subsequent
steps launch into a clean environment, and mark the Logs step
continue-on-error so a diagnostic step can never fail the job on its own.
* docs(readme): replace star-history.com with self-generated chart
The star-history.com SVG is a third-party dependency that can rate
limit or go down. Replace it with a GitHub Action that fetches
stargazers via the REST API and renders an SVG with matplotlib,
committing note/star_history.svg weekly. The README references the
committed file directly, so the chart has no runtime dependency on
any external service.
* ci(star-history): run daily instead of weekly
* s3: HEAD with partNumber reports the part's size and range
HeadObject set its headers from the total object size and then only
validated the partNumber, so a client probing part 1 with HEAD got the
whole object's Content-Length and a 200 while the same GET returned the
part's size, a Content-Range and a 206.
Resolve the part's byte range before the headers are written, through the
range logic GetObject already used, and answer a partNumber HEAD as the
ranged HEAD that AWS documents.
* s3: answer an unsatisfiable partNumber with 416 InvalidPartNumber
GET and HEAD rejected a partNumber past the number of parts with 400
InvalidPart, the code for a missing part in CompleteMultipartUpload. AWS
answers a read of a part that does not exist with 416 InvalidPartNumber,
which lets a client probing for the part count tell the two apart.
The ceph suite pins RGW's 400 InvalidPart here, so the s3tests jobs patch
that expectation the way they already patch prefix ordering.
* s3: keep the whole-object checksum off a partNumber response
The stored checksum covers the whole object, so it is already withheld
from a ranged read. A partNumber HEAD now describes one part while the
request carries no Range header, so exclude it there too rather than
handing a client a checksum that does not match the bytes described.
* s3: resolve a partNumber against the parts the object records
Completion accepts ascending, not consecutive, part numbers, so the part
count is not the highest part number. Comparing the two rejected an
uploaded part 3 of a two-part object, and let a request for the absent
part 2 fall through to the positional chunk lookup and serve part 3's
bytes. Ask the recorded boundaries for the part instead, and keep the
count comparison for objects written before boundaries were stored.
* s3: apply a client Range within the part on HEAD too
GET narrowed the part by a Range sent alongside partNumber; HEAD reported
the whole part, so the two disagreed again for a request that carries
both. Move the narrowing into the shared range lookup so either verb
describes the same bytes.
Every image-signing job has failed since signing was added (#11129):
must provide --new-bundle-format or --bundle where applicable with
--signing-config or --use-signing-config
Cosign 3 turned on two defaults, not one. The action only disabled
--new-bundle-format to keep the .sig tag layout, but --use-signing-config
is still on, and cosign refuses that pairing because the signing-config
path has nowhere to write its verification material without a bundle.
Disabling it too falls back to the default Fulcio and Rekor URLs, the
same services the .sig layout always used. The verify step needs no
change: cosign verify looks for a referrer bundle first and falls back
to the .sig tag when there is none.
Generated with [Devin](https://devin.ai)
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: composite action that signs and verifies an image with cosign
Keyless, by digest, with a verification pass against the calling workflow's
own identity right after signing. Signatures use the .sig tag layout rather
than the OCI-referrer bundle cosign 3 writes by default, since that is what
the verifiers people run today read. Dependabot is pointed at the action so
the cosign-installer pin keeps moving.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker release: sign every variant on both registries
The merge job signs each variant's multi-arch index on GHCR and Docker Hub
once the tag exists, recursively so the platform images are covered too.
latest re-tags the same manifest and inherits the signature.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker dev: sign the dev image
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker latest: sign a latest rebuilt by hand
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker release: sign the foundationdb image
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker: sign the per-version foundationdb and rocksdb builds
They push to the same repository as the releases, so an admission policy
that verifies chrislusf/seaweedfs would otherwise reject them.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker: document image signature verification
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* ci: pin the actions the signing jobs newly run by commit
These run with registry credentials and the OIDC token that signs under
the repository's identity, so a retargeted tag upstream must not be able
to reach them.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker latest: pass the dispatch tag through env, not the script
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker: complete Kyverno policy, digest note, identity scope
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker latest: keep the dispatch tag out of the manifest script too
The step predates signing, but the job now holds the OIDC identity.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* ci: pin every action in the jobs that sign
The jobs that hold the OIDC identity run these with registry credentials,
so a retargeted tag upstream must not reach them.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker release: copy and sign the digest the run created, pin the rest
crane copy and the signature both resolved the tag, which another
publisher could move between the two steps. The index digest is read once,
right after it is created, and the Docker Hub copy and both signatures use
it. The manual latest rebuild gets the same treatment. The actions in these
jobs are pinned to commits, crane to v0.22.0 by checksum, and the sparse
checkout no longer keeps the token.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker latest: the signing job checks out the workflow's own commit
The job only assembles and signs manifests, so nothing there needs the
source_ref checkout; the local signing action now comes from the same
revision as the workflow file that calls it.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker release: take the index digest from the create result
imagetools create writes the descriptor it pushed with --metadata-file
(buildx 0.32+, the runners ship 0.36), so the digest no longer comes from
re-resolving the tag even within the same step.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* helm: values-driven labels on every ingress
Each ingress already takes annotations from values, but its labels were
a fixed block, so tools that select ingresses by label (ExternalDNS
label filters, for one) had nothing to key on. Every ingress block now
has a labels map rendered after the standard app.kubernetes.io labels,
including the Traefik IngressRouteTCP that shares the filer gRPC values.
Claude-Session: https://claude.ai/code/session_01L6eJGXtYkwe1W9QjGeUgr1
* helm ci: render check for ingress labels
Claude-Session: https://claude.ai/code/session_01L6eJGXtYkwe1W9QjGeUgr1
* filer: keep the sentinel when CreateEntry reports an update failure
CreateEntry flattened the error UpdateEntry wraps, so errors.Is stopped
matching and ErrExistingIsDirectory and ErrExistingIsFile never reached
the S3 mapper, which answered a retryable 500 instead.
* s3: let a key that is a prefix of other keys be an object
S3 keys are flat, so "a/b" and "a/b/c" are independent objects that
coexist in either write order. The filer stores a key as a path, so one
of them has to live on the directory the other is nested under.
Writing the nested key first refused the prefix key outright. Writing it
second promoted the file to a directory, which kept its data but lost the
key: an empty object left nothing to recognise it by and disappeared, and
one with data listed under a trailing slash it never had.
Mark the directory that carries such a key, and write the object onto it
when the path is already a directory. The mark makes an empty prefix
object visible to listings and readable by GET and HEAD, keeps the empty
folder cleaner off it, and lists it under the key it was written with.
Deleting the key strips the mark back off along with the data.
* filer: keep a TTL off a directory that stands for an object
An expired entry is deleted a row at a time, so expiring a directory
removes it and leaves everything under it unreachable. Promoting a file
to a directory carried its TTL across, and a promoted file is exactly the
one that has keys nested under it.
Drop the TTL on promotion, and leave one an older build wrote alone. The
lifecycle worker still expires the object, through the delete that leaves
the directory behind.
* s3: delete the null version of a key other keys are nested under
The routed delete cannot remove an entry that other keys live under, and
answered a retryable 500 rather than falling back to the lock path the
unversioned delete already falls back to. That path then looked the entry
up under the bucket with the whole key as its name, so the demote wrote it
back one directory too high and failed as not found.
Fall back on any non-precondition error, and split the key before deleting
it. Trailing-slash directory markers with children reach the same delete.
* filer: keep the sentinel when MkFile and Mkdir report a create failure
Same flattening one layer out: every mkFile caller lost the sentinel, so
a CopyObject onto a key that other keys are nested under answered a
retryable 500 where a PutObject of the same key answers 409.
* s3: copy and rename a key that other keys are nested under
Such a key is stored on the directory those keys live in, and copy and
rename both refused it: the source lookup maps every directory entry to
NoSuchKey, so a key a plain GET serves could not be copied or moved, and
the destination side refused it as a directory conflict.
The source is read through a view of the entry as the object it names.
The destination is written the way a PutObject of that key writes it. A
rename at either end copies the object's own data across and strips it off
the source key rather than going through AtomicRenameEntry, which moves a
directory by moving everything under it - the nested keys are not part of
what is being renamed.
* worker: compile plugin.proto with the protoc that ships with the build
seaweed-volume already does this: protoc-bin-vendored carries the binary, so
the build needs no package manager and every build sees the same version. An
explicit PROTOC still wins, which is what lets the lance crates - whose own
build scripts read the same variable - share it.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* ci: point the worker builds at the vendored protoc
The jobs installed protobuf-compiler for lance's build scripts. They read
PROTOC, so pointing it at the binary protoc-bin-vendored already puts in the
registry serves them without a system package - one less apt call on the way
to a release, and the same protoc a developer's build uses.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* docs: say what the worker build needs from protoc
The lance crates' build scripts are the ones that need it, not ours, and they
take the same vendored binary.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* worker: name the binary weed-worker
It is the Rust side of `weed worker`, the way weed-volume is the Rust side of
`weed volume`, and lance is the first family of jobs it carries rather than
the only one it ever will. The crate keeps its own name: when a second family
arrives the bin target moves to a crate of its own, under this name.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* docker: ship the Rust maintenance worker in the image
Lance table buckets need a worker that can read the format, and until now the
only way to get one was a Rust toolchain and a cargo build. It now sits at
/usr/bin/weed-worker beside the Rust volume server, reached as
`docker run chrislusf/seaweedfs worker-rust --admin host:23646` — the verb
mirrors volume-rust, so plain `worker` still runs the Go one.
Taken pre-built or not at all: the lance jobs pull in arrow and datafusion, far
too large a tree to compile inside the image build, so an architecture CI did
not build for gets the empty placeholder the entrypoint refuses to exec, the
way the Rust volume server already does.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* ci: build the Rust worker for the container images
The same native cross-compile the volume server uses, so the release, latest
and dev images all carry it on amd64 and arm64. The artifact holds both
binaries now, so it is named for that rather than for the volume server.
Only the release directory each job builds is cached: with a debug profile
beside it the worker's target/ reaches 24GB, against a 10GB cache budget.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* ci: publish Rust worker binaries with the release
Linux amd64 and arm64 only: the worker runs beside the cluster it maintains,
and its dependency tree makes every extra target an expensive build.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* ci: build and test the Rust workers on change
Nothing built seaweed-worker in CI, so the release and the container images
would have been the first place a break showed up. Tests run in release too,
rather than compiling lance, arrow and datafusion again in another profile.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* docs: say how to get a released worker
Neither the image nor the release tarballs were mentioned; a toolchain and a
cargo build read as the only way in.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* ci: install protoc for the Rust worker builds
lance's crates compile their own protos, and unlike seaweed-volume they do not
vendor a protoc to do it with, so every job that builds the worker failed at
lance-encoding's build script.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* ci: do not persist credentials in the worker release checkout
The upload step is handed a token explicitly; a cargo build script should not
find another one sitting in the checkout's git config.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* docker: keep the worker's argument boundaries
Unquoted $@ splits on whitespace and expands globs, so an argument carrying
either arrived as something else.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* helm: install chart CI against an image tag that exists
The release bumps appVersion on master well before the container build
publishes that tag, and the chart CI runs on the bump commit, so every
release turns it red with ImagePullBackOff. Resolve the tag first and
fall back to latest while the new one is still building.
* helm: run the chart CI when the workflow itself changes
* helm: bound the registry lookup in the chart CI
An unbounded curl can hold the job, and the log did not say why the tag
was rejected. Cap it and print the status.
* test: drive the Lance namespace with Spark
The counterpart of catalog_spark, which does this for the Iceberg REST
catalog. Spark is the engine most likely to be pointed at a lakehouse,
and it reaches the Lance catalog through the connector's DSV2 catalog -
org.lance.spark.LanceNamespaceSparkCatalog with impl=rest - over the same
routes every other client uses.
SHOW NAMESPACES -> ['`sparklance-lcephd80`.ml']
SHOW TABLES -> ['sparklance-lcephd80$ml$embeddings']
count -> 3
filtered -> [(2, 'two'), (3, 'three')]
count after a second commit -> 4
The second insert is there on purpose: a store that cannot order commits
fails on the second one, not the first.
Two things the run settled that were guesses beforehand. CREATE TABLE
works, because the connector declares through the namespace and writes the
data itself rather than pushing Arrow at the server. And SHOW TABLES
returns the namespace's own identifiers - bucket, namespace and name
joined by the delimiter - not bare Spark table names.
Credentials go under the catalog's storage.* prefix, which is handed to
lance as object_store options; a gateway without STS vends none, the same
trap the LanceDB suite documents.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: verify the Lance table bucket was actually created
weed shell prints a command's own failure and still exits 0, so the harness
would go on to blame Spark for a bucket that was never made.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: bound the Docker probe
An unhealthy daemon makes docker version hang, and the probe runs before the
test has a timeout of its own.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: keep the ivy cache under the user's cache directory
It is mounted into a container running as root, so a shared temp path lets
another local user pre-create it and choose what Spark loads.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: assert the vector column's type, not only its name
A column that came back as array<double> or array<string> would still be
called vector and still pass.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: read the dataset off its location for real
The catalog being optional is the property that lets duckdb and pandas read
these tables; it was asserted in a comment and printed, never exercised.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: do not persist credentials in the Spark Lance checkout
The job only uploads a log on failure; nothing in it pushes.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: say the hosts in the README are placeholders
The suite passes dynamically allocated host.docker.internal ports.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
The db credential secret and the filer's WEED_MYSQL_* env were gated on
filer.enabled alone, so a filer on mongodb, redis, postgres or leveldb2 got a
generated mysql secret it never reads - kept forever by resource-policy: keep -
plus a mysql-db-host pointing nowhere.
Gate all of it on WEED_MYSQL_ENABLED, which is how the store is selected, plain
keys and secret-backed ones alike. An enable flag the chart cannot read - a
valueFrom, or one in secretExtraEnvironmentVars - counts as selected, so nothing
is dropped from a filer that is actually on mysql.
* s3: resolve ${VAR} in static config credentials from the environment
A deployment that keeps its S3 keys in a secret store had no way to hand
them to the gateway: -config takes a file, so the keys had to be written
into that file. Let a key in the static config name an environment
variable instead, and drop any credential whose reference stays unset so
the placeholder never becomes a usable key.
* helm: source the generated s3 identities from an existing secret
The only way to reuse credentials that already live in a Secret was to
hand-author the whole seaweedfs_s3_config JSON, since the literal keys in
values.yaml end up in git and a lookup-based keyRef renders empty under
helm template and Argo CD. Let s3.credentials.admin/read name a Secret and
its keys instead: the generated config references them as ${VAR} and the
gateway resolves them from the environment, so nothing is read from the
cluster at render time.
* s3: treat an empty environment value as an unresolved credential reference
A secret store can hand over a key that exists but is blank. Resolving it
would leave an access key whose signing secret is empty, so count it as
unresolved and drop the credential.
* helm: render the s3 secret when only the all-in-one auth flag is set
The all-in-one deployment mounts the s3 secret whenever any of the three
enableAuth flags is set, but the secret itself only rendered for the s3 and
filer flags, so allInOne.s3.enableAuth on its own left the pod waiting on a
secret nothing creates.
* helm ci: check the credential wiring on every workload that mounts it
The render check only looked at the standalone s3 deployment and only at
one of the four variables, so a helper that bound a variable to the wrong
secret key would still pass.
* helm: create the all-in-one s3 secret for every flag that mounts it
The all-in-one pod mounts the secret on any of the three enableAuth flags,
so keying its creation off allInOne.s3.enableAuth alone still left
filer.s3.enableAuth without filer.s3.enabled pointing at a secret nothing
creates. Mirror the deployment's own condition instead, and check each
flag renders both the mount and the secret.
* s3: reject a malformed credential reference instead of keying on it
A typo such as ${MY-VAR} matches no substitution, so it survived expansion
and the placeholder itself became the access key the gateway accepted.
Require every ${ in a static credential to open a well-formed reference.
* release: cut the whole release from the version bump workflow
The bump workflow stopped after pushing the version commit, and the rest was
manual: create the release, then run "Prepare release" in the csi-driver and
the operator. It now pushes the tag itself, which is what starts the binary,
container and helm workflows, creates the release with generated notes, and
dispatches the other two repositories, waiting for both.
Pushing the tag and reaching the other repositories both need RELEASE_PAT;
GITHUB_TOKEN raises no events that start workflows.
* release: tighten the release workflow after review
Check out master explicitly: a dispatch can select any branch, and the tag,
the commit and the release would then come off that branch while the
downstream job dispatches master.
Scope contents:write to the job that pushes; the downstream job talks to the
other repositories with RELEASE_PAT and needs nothing here.
Wait for the module proxy to serve the release commit as the tip before
dispatching, instead of priming it and hoping. The dispatched workflows pin
seaweedfs with `go get -u ...@latest`, so a stale tip means they release
against a pre-release commit, silently.
Identify the dispatched run by diffing the run list against the snapshot
taken before dispatching, rather than assuming the newest run is ours.
* release: wait on the downstream release, not on the run that makes it
A dispatched run cannot be told apart from a concurrent one: the API does not
report the inputs a run was dispatched with, so watching "the run that appeared
after mine" can watch someone else's and report their result as ours.
Wait for a release to appear in the downstream repository instead. That is the
thing being waited for, and it holds however many runs are in flight.
* test: read Lance tables from DuckDB
The LanceDB and Spark suites go through the catalog. DuckDB does not: its
lance extension reaches the data over S3 with no namespace involved, which
exercises the other half of the design - a table bucket's layout is a
valid Lance dataset directory, so a table stays readable when the catalog
is not in the path.
scan_rows=128
scan_columns=id,title,vector
filtered_rows=5
nearest=1,0,2
It also pins the one place the layout costs us. DuckDB's replacement scan
recognises a dataset by a .lance path suffix, and tables created through
this catalog deliberately have none: the catalog entry is the dataset
directory, a table name may not contain a dot, and a suffix would leak
into ARNs and policies. So __lance_scan is the way in, and the bare
SELECT ... FROM 's3://...' form does not see these tables.
The test asserts both halves - a suffixed path is read, a suffix-less one
is not - so if the extension ever recognises a bare directory, it fails
and says to update the documentation rather than leaving it wrong.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: require the catalog error from the suffix-less read
Any failure satisfied the old check - a missing extension, bad credentials,
an unreachable endpoint - so the assertion could pass without the
replacement scan ever classifying the path.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: verify the Lance table bucket was actually created
weed shell prints a command's own failure and still exits 0, so the harness
would go on to blame DuckDB for a bucket that was never made.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: bound the Docker probe
An unhealthy daemon makes docker version hang, and the probe runs before the
test has a timeout of its own.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: order the aggregates the assertions read
string_agg over an unordered relation may return the names, and the vector
search's ids, in any order, so the expectations could fail on a run where
nothing changed.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: do not persist credentials in the DuckDB Lance checkout
The job only uploads a log on failure; nothing in it pushes.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* lance worker: share the integration tests' scaffolding
The recorder that keeps what a handler sent, the config builder and the
storage-option fallback all lived inside compaction.rs, so a second test
binary would have had to copy them. They move to tests/common.
The fallback now reads AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and
AWS_ENDPOINT_URL from the environment, defaulting to what it used before.
A harness can then point these tests at a gateway that checks what it is
given rather than one that accepts anything.
* lance worker: maintain one named table, for a harness to drive
Compacts and cleans up whatever WEED_LANCE_TABLE names, through the
handlers' own detect-then-execute path: a proposal the worker would not
have made is not one worth running.
The existing tests seed the tables they check. This one deliberately does
not, so a harness that has already written a table and knows what is in it
can have the real handlers maintain it and then read it back.
* test: take a table through its whole life, for Iceberg and Lance
Created in the catalog, filled by a real client, maintained by the worker,
read again, dropped. The step nothing was checking is the read after
maintenance: compaction once rewrote every dictionary-encoded column onto
a single value and shipped, because the maintenance tests were thorough
about sequence numbers, manifest entries and metadata versions and none of
them opened the parquet file the worker had just written.
So the assertion is a tally - row count, the cardinality of each
dictionary-encoded column, and an md5 over whole rows - taken before
maintenance and again after, required to be equal. The cardinalities name
the failure that happened; the digest catches a rewrite that keeps every
column's cardinality and hands the values to the wrong rows. A compaction
that merged nothing fails rather than passes, or the read afterwards is
checking a file the worker never wrote.
The Iceberg half runs two clients. DuckDB is the one the corruption was
reported against and the only one here that writes the deprecated
PLAIN_DICTIONARY encoding, which parquet-go normalizes away on write, so a
Go writer cannot produce it. PyIceberg writes the modern spelling. Pinning
parquet-go back to v0.30.1 fails the DuckDB half and passes the PyIceberg
one, which is why both are here.
Lance maintenance lives in the Rust worker, so it runs there where cargo
is installed and through the two lance calls those handlers wrap where it
is not. WEED_LANCE_MAINTENANCE picks one instead of letting the test guess.
* ci: run the table lifecycle tests
CI maintains the Lance table through the lance library rather than the
worker: a cold build of the lance crate costs more than the glue it would
be checking, and the worker's own tests cover its handlers.
The suite drives the Iceberg maintenance worker, so a change to it now
triggers this workflow too.
* test: let the lifecycle harness fail instead of skipping
Setup failures all exited zero, so a cluster that would not come up, or a
port allocation that lost, reported a green run for code nothing had
executed. That is the failure mode this whole directory exists to close,
and it was in the harness itself.
Only a checkout without a weed binary skips now, and it runs the tests so
each one says so rather than the package quietly passing. Everything else
fails.
The filer existence probe gets a deadline while I am here: it ran without
one, so an unresponsive filer would hang the suite past every timeout the
clients have.
* test: make the lifecycle checks check what they claim to
Three of them could pass without having looked.
The DuckDB skip matched "syntax error", "not implemented" and "Failed to
load" anywhere in the output, in any phase. A parse error in the SQL this
test generates, or a refusal from our own catalog, would have taken the
only coverage of the PLAIN_DICTIONARY encoding out of CI and left it
green. It now matches the extension failing to install, and only in the
phase that installs it. Everything past LOAD is ours and fails.
The digests covered id, category and value. Compaction rewrites the whole
row, so a defect confined to ts, or to a Lance vector, changed nothing
either side of maintenance. Every persisted column goes in now, ts as
microseconds so no timezone sits between the two runs.
The Lance drop check caught every exception as proof the dataset was
gone. pylance turns credential and transport failures into the same
ValueError, so it only accepts the message that means not found.
* docs: say up front which maintenance path the Lance half takes
The opening summary said the worker maintains both tables. It maintains
the Iceberg one always and the Lance one only where cargo is installed,
which is not what CI does.
* test: drive the Lance namespace with LanceDB
The Iceberg catalog is checked against Spark, Trino, ClickHouse, Doris,
Dremio and RisingWave. The Lance one had only its own reference client,
which is the same thing as checking it against ourselves.
LanceDB connects with connect_namespace("rest", ...), which speaks the
routes this catalog implements, so the suite exercises the protocol rather
than our idea of it: list the catalog, open a table through it, read the
schema, run a vector search and a filtered scan, create a table, and read
the same dataset straight off its URI with no catalog at all.
table_names -> ['lancedb-p0guidmm$ml$embeddings']
open_table -> 64 rows
search -> [1, 0, 2]
create_table -> 4 rows, listed by the catalog
direct read without the catalog -> 64 rows
Seeding is pylance, because the namespace records where a table lives and
does not carry its data. That split is the design rather than a limit of
the test.
One interop note the test encodes: a gateway without STS vends
storage_options carrying an endpoint and a region but no credentials, and
LanceDB uses what the namespace vends on some paths. The container gets
credentials in its environment as well, which is what a deployment without
STS would do.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: pin the LanceDB client, and index before searching
Three from review.
The client's dependencies were unpinned, so an unrelated upstream release
could change what an old commit reproduces. Pinned to the versions this
suite was verified against; the client is as much the thing under test as
the server.
The search was called ANN and was not: without an index LanceDB scans.
The test now builds an IVF_PQ index over 1024 rows first, which is worth
more than the wording fix - an index writes into a directory of the table
that the S3 door has to admit, and that guard has refused a Lance
directory before. It builds, covers all 1024 rows, and searches.
The assertion moved with it. Demanding the exact nearest neighbour was
right for a brute-force scan and wrong for a quantized index, which
answered 0 as readily as 1; both are correct, so the check is now the
neighbourhood.
And the pushdown check accepted any failure. It now requires the refusal
to be the catalog's Unsupported and requires that nothing was left behind,
or, when the client falls back, that the table is complete.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
Four workflows ran apt to install libfuse3-dev before every FUSE job. Nothing
needs it: go-fuse implements the protocol in pure Go, no cgo in the tree
references fuse, and the package does not even provide the fusermount3 the
mount actually execs - fuse3 does, and it is already on the runner image, which
is why the setuid-repair step finds it.
So the step downloaded a dev package to build against headers no compiler ever
opened, and it is the step that has been hanging whenever the Ubuntu mirror
goes slow. Configuring /etc/fuse.conf is all that is left.
The e2e workflow sets defaults.run.working-directory: docker, so the call I
added resolved to docker/docker/apt-install and every FUSE Mount run has
failed with 'sudo: docker/apt-install: command not found' since it merged.
Every workflow that builds the Rust volume server first installed protoc
from a package manager - twelve steps across apt, brew and choco. That is
37s per job on a good day, and this week archive.ubuntu.com stalled long
enough for four jobs to burn their whole timeout without reaching a build.
protoc-bin-vendored ships the compiler as a build-dependency, so it now
arrives through the cargo registry the workflows already cache and there
is nothing left to install. cargo build works on a machine with no protoc
at all, which is worth as much locally as it is in CI.
It also pins the version. The apt protoc on ubuntu-22.04 is 3.12, old
enough to reject proto3 optional, which is why build.rs passes
--experimental_allow_proto3_optional; the vendored one is 31.1. The flag
stays, since it costs nothing and keeps a build against an older PROTOC
working, and an explicit PROTOC still overrides the vendored binary for
packagers who supply their own.
The e2e job overwrote the runner's sources.list with two azure-only lines and
installed fuse from it, so the same mirror outage that took out the image
builds failed the step outright - this time on the runner rather than inside
the container, where the image-side fallback cannot reach.
Install through the same helper, and widen its rewrite to match any archive
host so it works whether the pristine list came from the base image
(archive.ubuntu.com) or from a CI runner (azure.archive.ubuntu.com). Keeping
the runner's original list also restores the security and backports pockets,
which the hand-written two-line replacement dropped.
Verified against the outage itself: with the pristine list pointed at Azure,
the build logged the skip after Azure timed out for real and installed from
archive.ubuntu.com.
* mount: re-resolve volume locations after a failed chunk read
NewChunkGroup passed nil as the ReaderCache's CacheInvalidator, so
retryFetchAfterCacheInvalidation was dead code on the FUSE read path. A
mount that cached a volume's locations while one server was down kept
retrying that server after it died, then returned EIO, even though the
master and filer both resolved the live replica. The S3 gateway already
passes its filerClient; do the same for the mount.
* test: FUSE integration tests for volume server failover
One mount appends while a second tails, and a volume server is killed,
started or restarted mid-stream against a 001-replicated cluster of three
volume servers. Automates the scenario matrix reported for Docker Swarm
mounts, including the large-file variant and a no-chaos control.
* test: report the filer's own view when append content mismatches
A mismatch between what the writer wrote and what the reader sees can come
from either side's cache. Read the file back through the filer's HTTP
handler as well, and let the mount verbosity be raised from the
environment, so a failing run says which layer lost the data.
* test: wait for the reader mount to converge before comparing
A mount caches metadata for about a second, so reading the file the instant
the writer's last close returned can legitimately come back short. Poll the
reader until it matches or the timeout expires; content that is wrong rather
than merely late never converges and still fails, now with the writer's
mount and the filer's own view alongside it.
* test: detect a failover cluster child that exited at startup
Signal(0) succeeds for a zombie and nothing reaped these children until
shutdown, so a process that died on startup looked alive until the readiness
timeout expired. Reap each child as it is started and consult the result.
* test: read a file the killed volume server actually holds
Placement decides which two of three servers back each volume, so killing
volume N and reading readfile-N could pass without the victim ever holding a
replica of it. Resolve each file's volumes through the filer and the master,
and pick one the victim backs, preferring a file the reader has not cached.
* ci: stop persisting checkout credentials in the failover workflow
The job does not use the token after cloning. Also tag the README's command
block as bash and match the timeout the workflow actually uses.
* test: discard the ignored errors errcheck flags in the failover harness
* test: resolve manifests when mapping a file to its volumes
A manifest chunk's own fid names the volume holding the manifest, not the
volumes holding the data, so a large enough file would point the failover
victim at the wrong server.
* test: pin the stale-location recovery path with a primed reader
Reading a file for the first time after a server dies proves nothing: the
lookup is fresh and returns the survivor. Kill one holder and wait for the
master to drop it, read a file on that volume so the reader caches the lone
survivor, restart the first server, then kill the survivor. The reader's only
cached location is now dead while the data is live elsewhere, which is the
case the invalidator exists for: EIO without it, recovery with it.
* ec: bounded-exhaustive model check of the volume lifecycle
The randomized chaos harness samples the state space; this enumerates
it. The lifecycle is a state machine whose steps mirror the pipelines in
this package, and the checker explores every schedule within the bound:
a crash at every step boundary, an error return running the rollback
(itself crashable at every step), a volume-server restart applying the
startup reconciliation rules in every quiescent state, and the
prescribed restart-based recovery from every crashed state.
Checked in every reachable state: durability (a readable copy always
exists), at most one generation mounted, and — a property the sweep
discipline turns out to guarantee — at most one generation's files on
disk. From every quiescent state the recovery must converge to a clean
volume. Runs in well under a second.
* test: deterministic EC interruption matrix
Enumerate every phase of every interruptible EC operation and kill a
real weed shell exactly when the phase announces itself on the command
output, instead of at a random moment: four encode phases, four decode
phases, and the balance's move phase (set up with -rebalance=false so a
move is guaranteed). Each scenario prepares its precondition, kills at
the marker, runs the prescribed recovery, and verifies every stored byte
still reads back identical.
The interruption recoveries move out of the randomized ops into shared
chaosRun helpers both drivers use.
* test: make the randomized EC chaos walk opt-in
The systematic layers — the interruption matrix and the lifecycle model
check — carry the CI coverage deterministically; the randomized walk
stays for exploratory runs, behind EC_CHAOS_SEED.
* ci: bound the EC integration suite by the job budget, not go test's default
The suite with the interruption matrix runs close to the default 10m
binary timeout on slower runners.
* test: require every interruption-matrix marker to appear
A marker that never prints means a pipeline refactor renamed or dropped
the progress line; silently degenerating into a no-interruption run
would let CI pass without exercising the boundary the scenario names.
Also recheck the marker channel after the wait: a shell that prints and
exits at once makes both channels ready, and select picking the exit
case must not report a printed marker as missed.
* redis2: route the orphan cleanup existence checks to the master
* scaffold: the redis_cluster2 read routing key is useReadOnly
* ci: run the gated redis store tests
* redis2: poll for the redis expiry instead of a fixed sleep
* redis2: assert the value key exists before testing its expiry
* benchmark tool for mounted filesystems
* ci: on-demand mount benchmark, native WinFsp vs rclone plus a Linux reference
* windows mount: let the Windows cache manager cache file data
WinFsp only turns the cache manager on for a file when FileInfoTimeout
is infinite; at any finite value every application read and write is a
synchronous trip into the mount process at whatever size the application
issued. Metadata events already reach FspFileSystemNotify, which purges
a changed file's cached pages and attributes, so an infinite timeout
stays coherent. The dir listing, volume info and EA timeouts are pinned
to one second so they do not silently inherit the infinity.
* windows mount: cache resolved paths and attributes in the adapter
WinFsp addresses every operation by path and has no FORGET, so the
adapter walked the whole path through Lookup on each one, and in a
directory the filer has not listed yet every walk was a filer round
trip; nothing played the part of the kernel's dentry and attribute
caches. The path cache owns one lookup reference per entry the way the
kernel holds one until FORGET, serves attribute reads for files without
an open handle, and is purged by the mount's own mutations and by
metadata events, with the timeout as backstop.
* windows mount: keep a closed file's attributes cached
Open steals the path's cache entry for its handle and Release returned
the reference with a purge, so the stat that follows every copied file
walked to the filer again. Reading the handle's final attributes before
it goes away and moving the reference back into the cache serves that
stat locally, the way the kernel's attribute cache does after a close.
Only if the path still names that inode, though: WinFsp reports the
path the handle opened with, and after a delete-on-close or a rename
caching it would resurrect an entry that is gone.
* windows mount: persist entries at create, and let the flush stay at close
WinFsp posts the cleanup and close that carry the flush after
CloseHandle has returned, so deferring the filer entry to the flush let
everything that reads through the filer race an unflushed close: a
listing missed just-written files, and a directory rename moved a
directory on the filer before its newest child existed there, leaving
the straggler flush to recreate the child under the dead path.
Flush-at-cleanup is not the answer either: it makes every handle's
cleanup flush, and those flushes race the unlinks of delete-on-close,
re-inserting the entry the unlink just removed. Persisting the entry at
create takes the ordering question away.
* mount: flush written pages before a truncate shrinks past them
The shrink trims chunks, but written pages that have not become chunks
yet are invisible to it, so the next flush wrote them back and the file
grew again, resurrecting the truncated bytes. Windows hits this on
every write-then-shrink because its flush runs after CloseHandle, but
the gap is platform-neutral.
* mount: order a file's unlink against its in-flight flush
Unlink set the handle's deleted flag bare, so a flush already past its
own check of that flag wrote the entry back right after the delete
removed it, and a delete-on-close file outlived its last handle. The
flag is now set under the handle's flush lock and re-checked under it,
so a flush either completes before the delete or sees the flag and
skips. An eagerly created handle also starts clean: the dirty mark
existed to make the deferred filer create happen at flush, and eager
creates have nothing to flush.
* filer: keep the existing peer subscription on a repeated add
A cluster node add for a peer that is already followed restarted the
subscription, dropping the metadata events between the two runs.
* master: tell a connecting client the current cluster membership
Cluster node updates are only broadcast to the clients connected at that
moment. A filer that lost its master stream while a peer came back never
learned about the peer, and stopped replicating its metadata for good.
* test: a filer joining the master learns about the filers already there
* test: a filer resubscribes to a peer that registered while it was disconnected
Runs the reported sequence against real processes: filer2 leaves, filer1
is paused and its master stream is broken, filer2 registers again, and
filer1 has to replicate from it after reconnecting.
* s3tables: add ClickHouse iceberg catalog integration test
* ci: run the ClickHouse iceberg catalog test
* s3tables: bound setup HTTP calls in the ClickHouse test
* s3tables: pin the ClickHouse writer image dependencies
* helm: mount an enterprise license Secret into every component
Running the enterprise image under this chart meant hand-rolling
extraVolumes and extraVolumeMounts on every component. Missing one is
easy and quiet: a component without the license silently drops to
community mode, and on the admin that surfaces only as Data Recovery and
Point-in-Time Recovery refusing to enable, with the master looking fine.
Add global.seaweedfs.license.existingSecret. The Secret is mounted
read-only into master, volume, filer, s3, sftp, admin, worker and
all-in-one, and SEAWEED_LICENSE points every process at the file rather
than relying on the binary's search paths, which depend on the working
directory.
The mount is a directory, never a subPath: kubelet refreshes Secret
contents in place, but a subPath is resolved once at container start and
never updates, which would break license renewal. There is deliberately
no checksum annotation on the pod template either — that would roll every
pod on renewal, the opposite of what is wanted. Verified on kind: the
renewed file reached a running master ~70s after the Secret was patched,
same pod UID, restartCount 0.
Also documents that only the master re-reads the license on a timer
today; the other components pick a renewal up on their next restart.
* helm: keep master data on a claim by default
The master's -mdir holds its Raft log and snapshots, and with them the
cluster's topology UUID — the identity an enterprise license is issued
against. It defaulted to a hostPath under /ssd, which does not follow a
rescheduled pod: the master came back with an empty data directory, a
freshly generated cluster UUID, and a license that no longer matched.
With the chart's default of a single master replica there is no peer to
recover the identity from either.
Default master.data.type to persistentVolumeClaim, sized 1Gi (Raft state
is small). hostPath stays available for anyone who wants it.
This is breaking for existing releases: volumeClaimTemplates is immutable,
so helm upgrade on a release installed with the old default fails with
"updates to statefulset spec for fields other than ... are forbidden".
Verified on kind, along with both ways out — pinning
master.data.type=hostPath upgrades cleanly, and the documented migration
(stop the master, pre-seed a claim named after the StatefulSet, upgrade)
preserves the cluster UUID. Seeding has to happen while the master is
stopped; copying into a live pod loses the state, since the running
master rewrites its Raft files before the restart.
* helm: mount the license on masters only
The master is what reads the license file: it validates it, enforces the
capacity limit and binds it to the cluster UUID. Mounting the Secret on
volume, filer, s3, sftp, admin and worker put it in six more containers
that never look at it, so drop it there and keep master plus all-in-one,
which runs `weed server -master`.
Two fixes from review while here:
- project only the configured key out of the Secret, so an unrelated
key in the same Secret is not exposed to the container. Verified the
key-scoped projection still updates in place: patching the Secret
reached the running master in ~50s, same pod UID, restartCount 0.
- drop SEAWEED_LICENSE from merged extraEnvironmentVars while a
license Secret is configured. It used to be possible to render the
key twice in one container, with the user's value winning over the
path the chart actually mounts.
CI now pins the scope (master only, all-in-one separately), the
key-scoped projection, readOnly, and that SEAWEED_LICENSE renders once.
* helm: fix the documented master-data migration
The seed pod in the migration never mounted the claim it was supposed to
seed, so following the steps verbatim copied the Raft state onto the
pod's ephemeral filesystem and threw it away with the pod — landing the
reader in exactly the empty-claim, new-cluster-UUID state the section
exists to avoid. Give the pod the volume.
The names were assembled as <release>-seaweedfs-*, which is wrong
whenever the release name already contains the chart name or an override
is set; read the StatefulSet name from the cluster instead and derive the
claim from it. Also scope the procedure to the chart's single-master
default, and create the Secret in the release namespace.
Trims the enterprise prose this section had accumulated: this is the OSS
chart, and the master-data default is a durability fix that stands on its
own.
* helm: quote the projected license key, reserve it on the secret env path
A secretKey that YAML reads as a non-string (123, yes, no) rendered
unquoted into the volume's items, so the projection would not match the
Secret's key. Quote both key and path.
all-in-one renders secretExtraEnvironmentVars itself, outside the merge
helper that already drops SEAWEED_LICENSE, so an entry there could still
render the variable twice. Skip it there too while a license Secret is
configured. The master template has no such block, so this is the only
remaining path.
* helm: correct the license helper comments after scoping to masters
* helm: keep hostPath as the master data default
Defaulting master.data.type to a claim broke every existing release:
volumeClaimTemplates is immutable on a StatefulSet, so helm upgrade
failed with "updates to statefulset spec for fields other than ... are
forbidden" before it changed anything.
Keep hostPath as the default and document the claim as the option to
choose — for a new install, or for an existing one via the migration
already in the README. The chart supported both types all along; only
the default moves back.
Every immutable field of every rendered StatefulSet is now identical to
upstream under default values, so an in-place upgrade cannot trip the
API. Verified on kind: install with the unmodified upstream chart,
upgrade to this branch (ok), upgrade again turning the license Secret on
(ok, volume added in place). A fresh install with
master.data.type=persistentVolumeClaim binds its claim as before.
The whole PR is additive now: nothing renders differently until a value
is set.
* helm: scope the migration's StatefulSet lookup to the release
* helm: trim the comments added by this change
* s3: give a versioned metadata-only copy its own chunks
A self-copy that only rewrites metadata clones the source entry, chunk fids
and all, and writes the clone back. With no versioning that is exactly right:
the clone replaces the entry it came from, so one entry owns the needles the
whole time. Under versioning the clone lands in a new .versions/ file and the
source stays live, and nothing refcounts a plain shared chunk list -- deleting
either version (a NoncurrentVersionExpiration rule, say) frees needles the
other still points at, and the next vacuum makes that permanent. rclone hits
this on every upload, since it stamps mtime with exactly this copy.
Take the metadata-only path only where the write replaces the entry it read:
the bare key of a bucket without versioning. Versioned, suspended, and
versionId-pinned copies fall through to the regular copy path, which gives the
destination its own chunks.
* s3: reencrypt a versioned SSE-KMS key rotation instead of reusing the chunks
A same-object copy that changes the KMS key id hands the source chunks straight
back, on the assumption that the copy overwrites the entry they came from. A
versioned bucket writes a new version beside the source instead, so the two end
up sharing needles that nothing refcounts, and deleting either one frees the
other's data. Reuse the chunks only when the destination really is the source
entry; otherwise fall through to the reencrypt path, which also gives the new
version the key it asked for rather than leaving it on the old one.
* s3: make one predicate decide whether a copy replaces its source
The metadata-only branch and the key-rotation strategy both answer the same
question -- does this copy write back to the entry it read -- so let them share
one predicate instead of pairing a same-destination check with it separately at
each site.
* test(s3): fail the copy regression tests when the vacuum does not run
The helper swallowed a failed or non-200 request to the master, so a vacuum
that never ran turned both chunk-ownership assertions into no-ops: the
tombstoned needles were still readable and the surviving version looked fine
either way. Require the endpoint, the request, and a 200.
* ci(s3): run every versioning test in the regression gate
The gate named the tests it wanted, so a new regression test sat there
uncovered until someone remembered this file -- it fooled me into thinking two
tests added in this PR never ran anywhere, when the comprehensive job had them
all along. Invert it: run everything, and name a test only to keep it out. The
delete job beside this one already works that way, and the suite costs about
two minutes.
Only the pagination stress tests are excluded; they build 1500+ versions, skip
themselves without ENABLE_STRESS_TESTS, and have their own make target.
Go's regexp has no negation, so the pattern is still assembled from a listing,
the way the volume-server integration workflow does it. Note the trailing $$:
make eats a lone trailing $ and takes the anchor with it.
Every test bucket is its own collection and each grows 7 volumes, so the
suite asks for 140 while the job caps the volume server at 100. Slots come
back only when the volume server's next full heartbeat tells the master the
deleted collections are gone, and the suite finishes inside one 5s pulse:
the run survives on whichever buckets happened to be dropped before that
single tick. The last run cleared by two slots, this one wedged the final
PutObject with 'No writable volumes and no free volumes left'.
Both workflows split the suite with ^Test[A-H] / ^Test[I-S] / ^Test[T-Z].
Test names cluster, so shard 2 drew 50 of the 114 grpc tests and 30 of
the 64 http ones, and spent 13m51s against 7m48s and 9m09s for its peers.
Listing the tests and dealing them out one at a time splits them 38/38/38
and 21/22/21, and keeps splitting evenly as tests are added. The pattern
is computed once into the environment rather than repeated in the summary
step, where the two copies had to be kept in agreement by hand.
The four targets ran in a shell loop at ~3m13s each, so the job took
13m29s and gated the whole workflow by itself: everything else finished
within 8m13s. A matrix runs them concurrently, ~3m45s wall clock.
fail-fast is off so one broken target still reports the other three,
instead of one target per push.
* volume: skip directory fsync on Windows
* ci: run the windows jobs for the whole vacuum path
Both windows jobs start the same weed mini cluster, so both exercise the
volume server's vacuum path, but only one of them watched a single file
in it. Cover the compact, reconcile and load files in both.
* volume: report a failed makeupDiff instead of discarding it
The cleanup removes assigned to the same err the makeupDiff failure was
held in, so an aborted compaction returned nil once both removes
succeeded. The master then recorded the vacuum as committed and the
volume reloaded against the discarded generation.
* volume: correct the fsyncDir comments after the windows skip
Both comments described the old shape, where windows fell through to a
sync whose error was swallowed.
* volume: keep the makeupDiff failure ahead of its cleanup errors
A failed remove of .cpd/.cpx outranked the failure that abandoned the
compaction, so the caller saw the cleanup error instead of the cause.
Log it and return the original, matching the Rust do_commit_compact. A
leftover temp file is rolled back by reconcile on the next start.
Key the Rust cargo cache on the toolchain that built it
The cache key was rust-<Cargo.lock hash> with a bare rust- restore prefix, so
one seaweed-volume/target survived across runner images. cargo tracks its own
inputs but not the runner's C toolchain, so build-script output for C
dependencies is reused even when the system libc underneath it changed.
That is how the Rust jobs got wedged: the cached aws-lc-sys objects reference
__isoc23_sscanf and __isoc23_strtol, symbols glibc only grew in 2.38, while the
jobs link on ubuntu-22.04 with glibc 2.35. Every job died at
rust-lld: error: undefined symbol: __isoc23_sscanf
>>> referenced by bcm.c in archive libaws_lc_sys-*.rlib
with nothing in the tree to explain it, and no amount of re-running helped
because the poisoned entry was hit every time.
Fold the glibc and rustc versions into the key so a toolchain change misses the
cache and rebuilds instead of producing an unlinkable target/.
* mount: report . and .. from windows directories
WinFsp strips the dot entries for the root itself and expects every other
directory to report them, the way a real NTFS enumeration does: its
dirctl test asserts a subdirectory's first two entries are "." and ".."
and that a hundred files enumerate as 102 entries. Dropping them
unconditionally is what fails querydir_test.
The Go test that guarded the old behaviour went with it: os.File.Readdir
filters dot entries itself, so it could never have observed either way.
* mount: give the windows dot entries their directory type
The readdir fills an attribute block only for real children, so "." and
".." arrived with a zeroed one and were reported with mode 0. Windows
refuses to enumerate a directory whose first entry is not marked as a
directory, which is the assertion querydir_test fails on with
STATUS_OBJECT_NAME_NOT_FOUND.
They now carry the type the readdir already knew. The explorer walk also
names any unexpected entry rather than only counting, so a dot entry
leaking through reads differently from a missing file.
