mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-08 23:50:43 +02:00
Chris Lu
cc8364a03e
* worker: name the binary weed-worker It is the Rust side of `weed worker`, the way weed-volume is the Rust side of `weed volume`, and lance is the first family of jobs it carries rather than the only one it ever will. The crate keeps its own name: when a second family arrives the bin target moves to a crate of its own, under this name. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * docker: ship the Rust maintenance worker in the image Lance table buckets need a worker that can read the format, and until now the only way to get one was a Rust toolchain and a cargo build. It now sits at /usr/bin/weed-worker beside the Rust volume server, reached as `docker run chrislusf/seaweedfs worker-rust --admin host:23646` — the verb mirrors volume-rust, so plain `worker` still runs the Go one. Taken pre-built or not at all: the lance jobs pull in arrow and datafusion, far too large a tree to compile inside the image build, so an architecture CI did not build for gets the empty placeholder the entrypoint refuses to exec, the way the Rust volume server already does. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * ci: build the Rust worker for the container images The same native cross-compile the volume server uses, so the release, latest and dev images all carry it on amd64 and arm64. The artifact holds both binaries now, so it is named for that rather than for the volume server. Only the release directory each job builds is cached: with a debug profile beside it the worker's target/ reaches 24GB, against a 10GB cache budget. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * ci: publish Rust worker binaries with the release Linux amd64 and arm64 only: the worker runs beside the cluster it maintains, and its dependency tree makes every extra target an expensive build. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * ci: build and test the Rust workers on change Nothing built seaweed-worker in CI, so the release and the container images would have been the first place a break showed up. Tests run in release too, rather than compiling lance, arrow and datafusion again in another profile. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * docs: say how to get a released worker Neither the image nor the release tarballs were mentioned; a toolchain and a cargo build read as the only way in. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * ci: install protoc for the Rust worker builds lance's crates compile their own protos, and unlike seaweed-volume they do not vendor a protoc to do it with, so every job that builds the worker failed at lance-encoding's build script. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * ci: do not persist credentials in the worker release checkout The upload step is handed a token explicitly; a cargo build script should not find another one sitting in the checkout's git config. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm * docker: keep the worker's argument boundaries Unquoted $@ splits on whitespace and expands globs, so an argument carrying either arrived as something else. Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
362 lines
13 KiB
YAML
362 lines
13 KiB
YAML
name: "rust: build versioned binaries"
|
|
|
|
on:
|
|
push:
|
|
tags:
|
|
- '*'
|
|
|
|
workflow_dispatch:
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
jobs:
|
|
|
|
build-rust-volume-linux:
|
|
permissions:
|
|
contents: write
|
|
runs-on: ubuntu-22.04
|
|
strategy:
|
|
matrix:
|
|
include:
|
|
- target: x86_64-unknown-linux-gnu
|
|
asset_suffix: linux_amd64
|
|
- target: aarch64-unknown-linux-gnu
|
|
asset_suffix: linux_arm64
|
|
cross: true
|
|
|
|
steps:
|
|
- uses: actions/checkout@v7
|
|
|
|
- name: Install Rust toolchain
|
|
uses: dtolnay/rust-toolchain@stable
|
|
with:
|
|
targets: ${{ matrix.target }}
|
|
|
|
- name: Install cross-compilation tools
|
|
if: matrix.cross
|
|
run: |
|
|
sudo dpkg --add-architecture arm64
|
|
sudo sed -i 's/^deb /deb [arch=amd64] /' /etc/apt/sources.list
|
|
echo "deb [arch=arm64] http://ports.ubuntu.com/ jammy main restricted universe multiverse" | sudo tee /etc/apt/sources.list.d/arm64.list
|
|
echo "deb [arch=arm64] http://ports.ubuntu.com/ jammy-updates main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list.d/arm64.list
|
|
sudo apt-get update
|
|
sudo apt-get install -y gcc-aarch64-linux-gnu libssl-dev:arm64
|
|
echo "CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER=aarch64-linux-gnu-gcc" >> "$GITHUB_ENV"
|
|
echo "OPENSSL_DIR=/usr" >> "$GITHUB_ENV"
|
|
echo "OPENSSL_INCLUDE_DIR=/usr/include" >> "$GITHUB_ENV"
|
|
echo "OPENSSL_LIB_DIR=/usr/lib/aarch64-linux-gnu" >> "$GITHUB_ENV"
|
|
|
|
- name: Cache cargo registry and target
|
|
uses: actions/cache@v6
|
|
with:
|
|
path: |
|
|
~/.cargo/registry
|
|
~/.cargo/git
|
|
seaweed-volume/target
|
|
key: rust-release-${{ matrix.target }}-${{ hashFiles('seaweed-volume/Cargo.lock') }}
|
|
restore-keys: |
|
|
rust-release-${{ matrix.target }}-
|
|
|
|
- name: Build Rust volume server (large disk)
|
|
env:
|
|
SEAWEEDFS_COMMIT: ${{ github.sha }}
|
|
run: |
|
|
cd seaweed-volume
|
|
cargo build --release --target ${{ matrix.target }} --target-dir target/large-disk
|
|
|
|
- name: Build Rust volume server (normal)
|
|
env:
|
|
SEAWEEDFS_COMMIT: ${{ github.sha }}
|
|
run: |
|
|
cd seaweed-volume
|
|
cargo build --release --target ${{ matrix.target }} --no-default-features --target-dir target/normal
|
|
|
|
- name: Package binaries
|
|
run: |
|
|
# Large disk (default, 5bytes feature)
|
|
cp seaweed-volume/target/large-disk/${{ matrix.target }}/release/weed-volume weed-volume-large-disk
|
|
tar czf weed-volume_large_disk_${{ matrix.asset_suffix }}.tar.gz weed-volume-large-disk
|
|
rm weed-volume-large-disk
|
|
|
|
# Normal volume size
|
|
cp seaweed-volume/target/normal/${{ matrix.target }}/release/weed-volume weed-volume-normal
|
|
tar czf weed-volume_${{ matrix.asset_suffix }}.tar.gz weed-volume-normal
|
|
rm weed-volume-normal
|
|
|
|
- name: Generate md5 checksums
|
|
run: |
|
|
for f in weed-volume_large_disk_${{ matrix.asset_suffix }}.tar.gz weed-volume_${{ matrix.asset_suffix }}.tar.gz; do
|
|
md5sum "$f" > "$f.md5"
|
|
done
|
|
|
|
- name: Upload release assets
|
|
if: startsWith(github.ref, 'refs/tags/')
|
|
uses: softprops/action-gh-release@v3
|
|
with:
|
|
files: |
|
|
weed-volume_large_disk_${{ matrix.asset_suffix }}.tar.gz
|
|
weed-volume_large_disk_${{ matrix.asset_suffix }}.tar.gz.md5
|
|
weed-volume_${{ matrix.asset_suffix }}.tar.gz
|
|
weed-volume_${{ matrix.asset_suffix }}.tar.gz.md5
|
|
env:
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
|
|
- name: Upload artifacts
|
|
if: ${{ !startsWith(github.ref, 'refs/tags/') }}
|
|
uses: actions/upload-artifact@v7
|
|
with:
|
|
name: rust-volume-${{ matrix.asset_suffix }}
|
|
path: |
|
|
weed-volume_large_disk_${{ matrix.asset_suffix }}.tar.gz
|
|
weed-volume_large_disk_${{ matrix.asset_suffix }}.tar.gz.md5
|
|
weed-volume_${{ matrix.asset_suffix }}.tar.gz
|
|
weed-volume_${{ matrix.asset_suffix }}.tar.gz.md5
|
|
|
|
# The Rust maintenance worker: Linux only, because it runs beside the cluster
|
|
# it maintains rather than on a laptop, and its dependency tree (lance, arrow,
|
|
# datafusion) makes every extra target an expensive build.
|
|
build-rust-worker-linux:
|
|
permissions:
|
|
contents: write
|
|
runs-on: ubuntu-22.04
|
|
strategy:
|
|
matrix:
|
|
include:
|
|
- target: x86_64-unknown-linux-gnu
|
|
asset_suffix: linux_amd64
|
|
- target: aarch64-unknown-linux-gnu
|
|
asset_suffix: linux_arm64
|
|
cross: true
|
|
|
|
steps:
|
|
- uses: actions/checkout@v7
|
|
with:
|
|
# The upload step is handed a token explicitly; a cargo build script
|
|
# should not find another one sitting in the checkout's git config.
|
|
persist-credentials: false
|
|
|
|
- name: Install Rust toolchain
|
|
uses: dtolnay/rust-toolchain@stable
|
|
with:
|
|
targets: ${{ matrix.target }}
|
|
|
|
- name: Install cross-compilation tools
|
|
if: matrix.cross
|
|
run: |
|
|
sudo dpkg --add-architecture arm64
|
|
sudo sed -i 's/^deb /deb [arch=amd64] /' /etc/apt/sources.list
|
|
echo "deb [arch=arm64] http://ports.ubuntu.com/ jammy main restricted universe multiverse" | sudo tee /etc/apt/sources.list.d/arm64.list
|
|
echo "deb [arch=arm64] http://ports.ubuntu.com/ jammy-updates main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list.d/arm64.list
|
|
sudo apt-get update
|
|
sudo apt-get install -y gcc-aarch64-linux-gnu
|
|
echo "CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER=aarch64-linux-gnu-gcc" >> "$GITHUB_ENV"
|
|
|
|
# lance's build scripts compile their own protos, and unlike seaweed-volume
|
|
# they do not vendor a protoc to do it with.
|
|
- name: Install protoc
|
|
run: sudo apt-get install -y protobuf-compiler
|
|
|
|
- name: Cache cargo registry and target
|
|
uses: actions/cache@v6
|
|
with:
|
|
path: |
|
|
~/.cargo/registry
|
|
~/.cargo/git
|
|
seaweed-worker/target/${{ matrix.target }}/release
|
|
key: rust-worker-release-${{ matrix.target }}-${{ hashFiles('seaweed-worker/Cargo.lock') }}
|
|
restore-keys: |
|
|
rust-worker-release-${{ matrix.target }}-
|
|
|
|
- name: Build the Rust maintenance worker
|
|
run: |
|
|
cd seaweed-worker
|
|
cargo build --release -p weed-lance-worker --target ${{ matrix.target }}
|
|
|
|
- name: Package binary
|
|
run: |
|
|
cp seaweed-worker/target/${{ matrix.target }}/release/weed-worker weed-worker
|
|
tar czf weed-worker_${{ matrix.asset_suffix }}.tar.gz weed-worker
|
|
rm weed-worker
|
|
md5sum weed-worker_${{ matrix.asset_suffix }}.tar.gz > weed-worker_${{ matrix.asset_suffix }}.tar.gz.md5
|
|
|
|
- name: Upload release assets
|
|
if: startsWith(github.ref, 'refs/tags/')
|
|
uses: softprops/action-gh-release@v3
|
|
with:
|
|
files: |
|
|
weed-worker_${{ matrix.asset_suffix }}.tar.gz
|
|
weed-worker_${{ matrix.asset_suffix }}.tar.gz.md5
|
|
env:
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
|
|
- name: Upload artifacts
|
|
if: ${{ !startsWith(github.ref, 'refs/tags/') }}
|
|
uses: actions/upload-artifact@v7
|
|
with:
|
|
name: rust-worker-${{ matrix.asset_suffix }}
|
|
path: |
|
|
weed-worker_${{ matrix.asset_suffix }}.tar.gz
|
|
weed-worker_${{ matrix.asset_suffix }}.tar.gz.md5
|
|
|
|
build-rust-volume-darwin:
|
|
permissions:
|
|
contents: write
|
|
runs-on: macos-latest
|
|
strategy:
|
|
matrix:
|
|
include:
|
|
- target: x86_64-apple-darwin
|
|
asset_suffix: darwin_amd64
|
|
- target: aarch64-apple-darwin
|
|
asset_suffix: darwin_arm64
|
|
|
|
steps:
|
|
- uses: actions/checkout@v7
|
|
|
|
- name: Install Rust toolchain
|
|
uses: dtolnay/rust-toolchain@stable
|
|
with:
|
|
targets: ${{ matrix.target }}
|
|
|
|
- name: Cache cargo registry and target
|
|
uses: actions/cache@v6
|
|
with:
|
|
path: |
|
|
~/.cargo/registry
|
|
~/.cargo/git
|
|
seaweed-volume/target
|
|
key: rust-release-${{ matrix.target }}-${{ hashFiles('seaweed-volume/Cargo.lock') }}
|
|
restore-keys: |
|
|
rust-release-${{ matrix.target }}-
|
|
|
|
- name: Build Rust volume server (large disk)
|
|
env:
|
|
SEAWEEDFS_COMMIT: ${{ github.sha }}
|
|
run: |
|
|
cd seaweed-volume
|
|
cargo build --release --target ${{ matrix.target }} --target-dir target/large-disk
|
|
|
|
- name: Build Rust volume server (normal)
|
|
env:
|
|
SEAWEEDFS_COMMIT: ${{ github.sha }}
|
|
run: |
|
|
cd seaweed-volume
|
|
cargo build --release --target ${{ matrix.target }} --no-default-features --target-dir target/normal
|
|
|
|
- name: Package binaries
|
|
run: |
|
|
cp seaweed-volume/target/large-disk/${{ matrix.target }}/release/weed-volume weed-volume-large-disk
|
|
tar czf weed-volume_large_disk_${{ matrix.asset_suffix }}.tar.gz weed-volume-large-disk
|
|
rm weed-volume-large-disk
|
|
|
|
cp seaweed-volume/target/normal/${{ matrix.target }}/release/weed-volume weed-volume-normal
|
|
tar czf weed-volume_${{ matrix.asset_suffix }}.tar.gz weed-volume-normal
|
|
rm weed-volume-normal
|
|
|
|
- name: Generate md5 checksums
|
|
run: |
|
|
for f in weed-volume_large_disk_${{ matrix.asset_suffix }}.tar.gz weed-volume_${{ matrix.asset_suffix }}.tar.gz; do
|
|
md5 -r "$f" > "$f.md5"
|
|
done
|
|
|
|
- name: Upload release assets
|
|
if: startsWith(github.ref, 'refs/tags/')
|
|
uses: softprops/action-gh-release@v3
|
|
with:
|
|
files: |
|
|
weed-volume_large_disk_${{ matrix.asset_suffix }}.tar.gz
|
|
weed-volume_large_disk_${{ matrix.asset_suffix }}.tar.gz.md5
|
|
weed-volume_${{ matrix.asset_suffix }}.tar.gz
|
|
weed-volume_${{ matrix.asset_suffix }}.tar.gz.md5
|
|
env:
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
|
|
- name: Upload artifacts
|
|
if: ${{ !startsWith(github.ref, 'refs/tags/') }}
|
|
uses: actions/upload-artifact@v7
|
|
with:
|
|
name: rust-volume-${{ matrix.asset_suffix }}
|
|
path: |
|
|
weed-volume_large_disk_${{ matrix.asset_suffix }}.tar.gz
|
|
weed-volume_large_disk_${{ matrix.asset_suffix }}.tar.gz.md5
|
|
weed-volume_${{ matrix.asset_suffix }}.tar.gz
|
|
weed-volume_${{ matrix.asset_suffix }}.tar.gz.md5
|
|
|
|
build-rust-volume-windows:
|
|
permissions:
|
|
contents: write
|
|
runs-on: windows-latest
|
|
|
|
steps:
|
|
- uses: actions/checkout@v7
|
|
|
|
- name: Install Rust toolchain
|
|
uses: dtolnay/rust-toolchain@stable
|
|
|
|
- name: Cache cargo registry and target
|
|
uses: actions/cache@v6
|
|
with:
|
|
path: |
|
|
~/.cargo/registry
|
|
~/.cargo/git
|
|
seaweed-volume/target
|
|
key: rust-release-windows-${{ hashFiles('seaweed-volume/Cargo.lock') }}
|
|
restore-keys: |
|
|
rust-release-windows-
|
|
|
|
- name: Build Rust volume server (large disk)
|
|
env:
|
|
SEAWEEDFS_COMMIT: ${{ github.sha }}
|
|
run: |
|
|
cd seaweed-volume
|
|
cargo build --release --target-dir target/large-disk
|
|
|
|
- name: Build Rust volume server (normal)
|
|
env:
|
|
SEAWEEDFS_COMMIT: ${{ github.sha }}
|
|
run: |
|
|
cd seaweed-volume
|
|
cargo build --release --no-default-features --target-dir target/normal
|
|
|
|
- name: Package binaries
|
|
shell: bash
|
|
run: |
|
|
cp seaweed-volume/target/large-disk/release/weed-volume.exe weed-volume-large-disk.exe
|
|
7z a weed-volume_large_disk_windows_amd64.zip weed-volume-large-disk.exe
|
|
rm weed-volume-large-disk.exe
|
|
|
|
cp seaweed-volume/target/normal/release/weed-volume.exe weed-volume-normal.exe
|
|
7z a weed-volume_windows_amd64.zip weed-volume-normal.exe
|
|
rm weed-volume-normal.exe
|
|
|
|
- name: Generate md5 checksums
|
|
shell: bash
|
|
run: |
|
|
for f in weed-volume_large_disk_windows_amd64.zip weed-volume_windows_amd64.zip; do
|
|
md5sum "$f" > "$f.md5"
|
|
done
|
|
|
|
- name: Upload release assets
|
|
if: startsWith(github.ref, 'refs/tags/')
|
|
uses: softprops/action-gh-release@v3
|
|
with:
|
|
files: |
|
|
weed-volume_large_disk_windows_amd64.zip
|
|
weed-volume_large_disk_windows_amd64.zip.md5
|
|
weed-volume_windows_amd64.zip
|
|
weed-volume_windows_amd64.zip.md5
|
|
env:
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
|
|
- name: Upload artifacts
|
|
if: ${{ !startsWith(github.ref, 'refs/tags/') }}
|
|
uses: actions/upload-artifact@v7
|
|
with:
|
|
name: rust-volume-windows_amd64
|
|
path: |
|
|
weed-volume_large_disk_windows_amd64.zip
|
|
weed-volume_large_disk_windows_amd64.zip.md5
|
|
weed-volume_windows_amd64.zip
|
|
weed-volume_windows_amd64.zip.md5
|