Compare commits

...
Author SHA1 Message Date
user.email 25a715028b test(volume_server/grpc): expect admission rejection for volume-server ping target
Ping admission (added by #9445) rejects non-master targets on volume
servers, so the volumeServer case now asserts InvalidArgument like the
filer case instead of the old dial-time error.
2026-10-24 23:16:54 -07:00
user.email 94f200ce52 fix(util): parse PROXY v1 header lazily inside the connection read path
Reading the header in Accept let a silent peer stall the listener for
everyone and let a malformed header surface as an accept error. The
header now parses on first Read/RemoteAddr under a once, inside the
connection's own goroutine lifecycle.
2026-10-24 23:01:31 -07:00
user.email ee5955e6f7 fix(volume): guard proxyProtocol flag for embedded volume starts 2026-10-24 22:45:58 -07:00
user.email b571ced535 fix(rust-volume-server): build backend binary into a private cache dir
A preexisting executable in a world-writable temp path could have been
planted by another local user; only reuse the cache when the directory
is verifiably ours, and otherwise build in a per-process directory.
2026-10-24 22:45:25 -07:00
user.email 76597324e6 feat(volume): carry client address through PROXY v1 to supervised backends
The proxy/native frontends terminate the client TCP connection, so the
Go backend saw loopback peers and -whiteList could not be enforced.
The launcher now passes -proxyProtocol and prepends a PROXY v1 line to
each proxied connection; opted-in listeners unwrap it and expose the
real client address to HTTP and gRPC checks.
2026-10-24 22:45:21 -07:00
user.email 2bb43f4772 fix(util): skip localhost side-listener for loopback-alias binds
Binding a non-default loopback address already confines the server to
local clients; an extra localhost listener only risks grabbing ports a
colocated frontend expects to own.
2026-10-24 22:45:19 -07:00
user.email d8b725d607 fix(util): ignore loopback source for outbound TCP dials
A loopback bind address can never originate connections to non-loopback
targets, so recording it pinned every outbound dial (e.g. to a remote
master) to an unreachable source.
2026-10-24 22:45:19 -07:00
user.email 717966a656 test(volume_server): expect 200 for HEAD with mismatched If-None-Match
The read handler evaluates If-None-Match before If-Modified-Since for both
GET and HEAD, so a mismatched etag returns the full response, not 304.
2026-10-24 22:45:19 -07:00
user.email 65b92b654d fix(rust-volume-server): skip native size pre-check for multipart uploads
The limit applies to the file part, not the multipart envelope; multipart
uploads are delegated so the backend measures the right bytes.
2026-10-24 22:45:14 -07:00
user.email bedb60a8a8 fix(rust-volume-server): delegate Content-MD5 verification to the Go backend
The native pre-check rejected every upload carrying Content-MD5, even when
the checksum was correct; the backend accepts matching checksums and only
rejects mismatches.
2026-10-24 22:45:11 -07:00
user.emailandDevin 18607b5760 ci: read go version from go.mod in rust smoke job
Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 22:27:41 -07:00
user.emailandDevin e7753a5978 docs(volume_server): match rust-mode env docs to harness resolver
Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 22:25:10 -07:00
user.emailandDevin a3d343178a fix(rust-volume-server): delegate placeholder native responses to the Go backend
The native handlers returned fabricated empty content for /status,
/healthz, /favicon.ico, /seaweedfsstatic, and /ui/index.html, hiding the
real server state behind 200 responses. Let those paths fall through to
the supervised backend so it serves the real data; keep the parity-only
native checks (OPTIONS, method gating, request validation).

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 22:24:51 -07:00
user.emailandDevin 7baa9c357f fix(rust-volume-server): advertise frontend address through the supervised backend
Rewriting the backend's -ip and ports made its heartbeat register the
private loopback backend instead of the listener clients should use.
When the frontend ports can also be bound on a non-conflicting address
(loopback alias for loopback frontends, 127.0.0.1 for remote ones), run
the backend there with the frontend identity so master advertises the
proxy; otherwise keep the old loopback-plus-random-ports fallback.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 22:24:29 -07:00
user.emailandDevin b4532b7083 fix(rust-volume-server): bound proxied connections and time out idle header peeks
Each accepted connection previously spawned an unbounded worker thread
that could block forever in peek, letting idle clients exhaust threads.
Cap in-flight connections per listener and give the header peek a read
deadline, cleared again before traffic is proxied.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 22:24:17 -07:00
user.emailandDevin 3b3042099f fix(rust-volume-server): accept --flag style and securityFile for the weed backend
The integration harness passes rust-native flags (--port, --ip.bind,
--securityFile). Translate --securityFile into the weed-level -config_dir
flag placed before the volume subcommand, and teach flag parsing and
rewriting to match single- and double-dash forms.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 22:23:53 -07:00
user.email 9436099d55 Merge remote-tracking branch 'origin/master' into codex-rust-volume-server-bootstrap
# Conflicts:
#	.github/workflows/volume-server-integration-tests.yml
#	test/volume_server/framework/cluster.go
#	test/volume_server/framework/cluster_dual.go
#	test/volume_server/framework/volume_fixture.go
#	test/volume_server/grpc/data_stream_success_test.go
#	test/volume_server/http/public_cors_methods_test.go
#	test/volume_server/matrix/config_profiles.go
2026-10-24 22:23:33 -07:00
Chris Lu 73e4173789 ci: collect and archive logs when rust smoke job fails 2026-10-24 21:41:37 -07:00
Chris Lu 07954a03c0 fix(rust-volume-server): check executable extension on non-unix instead of always true 2026-10-24 21:41:37 -07:00
Chris Lu 661d8df4d9 fix(rust-volume-server): read argv via args_os so non-UTF8 args cannot panic 2026-10-24 21:41:33 -07:00
github-actions[bot] ec8f979350 4.50 2026-10-25 03:09:42 +00:00
Chris LuandDevin 1a0485a43d test: bound trino container memory + dump logs when it exits before ready (#12044)
* test: dump trino container state and logs when it exits before ready

When the Trino container dies during startup, waitForTrino only
reported the docker exec error ("container is not running"), which
gives no hint whether the JVM was OOM-killed, the image broke, or the
config is bad. Capture docker inspect exit state and the container
logs in the failure message so runner-side flakes are diagnosable
from the test output alone.

* test: cap trino container memory at 4g in catalog tests

Trino's jvm.config sizes the heap at 80% of host RAM. On a loaded
self-hosted runner the JVM attempts to claim most of system memory
at boot and the container is killed ~15s after docker run — the
flake seen in CI today. Bounding the container to 4g gives the JVM
a fixed ~3.2g ceiling (observed usage is ~750mb), immune to host
memory pressure.

* test: keep docker errors in trino container diagnostics

Report docker inspect/logs failures instead of dropping them, and
include the last docker exec output in the container-dead fatal so a
missing container still explains what happened.

* test: detect trino container exit in readiness fallback

When the polling loop times out while the container still appears
alive, the trino --version fallback may itself fail because the
container has since exited. Re-check its output for the
exited-container signature and route through the diagnostic dump.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test: dump trino container diagnostics on readiness timeout too

The alive-but-never-ready timeout path reported only the last docker
exec error, losing the container state and logs that the dead-container
path already captures.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 20:00:06 -07:00
Chris LuandDevin 8c1a0a0cc2 filer: fix prefixed listing when the resume point sorts below the prefix (#12049)
* test: prefixed directory listing with start below the prefix

A listing constrained to prefix "item-b" and resumed from "item" must
return item-b.versions. The seek-based stores return an empty page
because the iterator lands before the prefix range and breaks on the
first non-matching name.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: clamp prefixed-listing seek to the prefix range

Seek-based stores (leveldb, leveldb2, leveldb3, rocksdb, tikv, etcd)
started the scan at startFileName and stopped at the first key outside
the prefix. A startFileName below the prefix landed the iterator on a
non-matching sibling first, so the listing returned an empty page even
though matching entries followed. etcd had the mirror problem: a range
beginning below the prefix end leaked non-matching entries. Begin the
seek at max(startFileName, prefix), which is what foundationdb and the
sql stores already do.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer(hbase): honor startFileName lower bound in prefixed listings

The scan starts at the prefix but only skipped the exact startFileName,
so entries sorting before it were returned again on resume.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: keep paging a prefixed listing while matches can still follow

The fallback for stores without native prefix support stopped paging as
soon as the last scanned name reached the prefix, dropping prefix
matches that continue on the next page.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer: return the last delivered name from the prefixed-listing fallback

The fallback for stores without native prefix support pages raw entries
and filters client-side. Keep the scan cursor and the returned cursor
separate: resuming must continue after the last delivered match, not the
last scanned name, or matches still inside the current raw page are
skipped on the next page. Also keep paging while the scan cursor is
inside or below the prefix range instead of stopping at the first name
that reaches the prefix.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* filer(hbase): start prefixed-listing scans at the resume key

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 19:59:19 -07:00
github-actions[bot] 0c32eab48e docs: regenerate star history chart 2026-10-25 00:48:29 +00:00
Chris Lu 85470416cb volume: overlap the local needle write with the replica fan-out (#10074)
* volume: overlap the local needle write with the replica fan-out

ReplicatedWrite wrote the needle to local disk and then fanned it out to
the replicas serially, so a replicated write cost local + replicas. The
needle is already fully buffered, so the two can run at once. Snapshot
the fields the fan-out reads first (the local write stamps the volume
TTL and updates needle flags, so the replica goroutine must not touch
the shared needle), then run the local write and the replica fan-out
concurrently and ack only after both. Where local disk and replica
network are separate resources the write latency drops to max(local,
replicas) instead of their sum; co-located on one disk the two writes
contend, so this helps distributed placements, not single-host tests.

* volume: return the local isUnchanged from the write goroutine

* volume: trim overlap-write comments

* volume-rs: overlap the replica fan-out with the local needle write

* volume: test the overlapped local write and replica fan-out

* volume: decode replica pairs only when replicas exist

A single-copy or incoming replicate write never uses the decoded pair map, so a malformed Pairs blob only burned an error counter and a log line. Gate the decode on replicaCount.

* volume: release the stalled replica on test failure paths

An early t.Fatal left the httptest handler blocked on releaseReplica, so the deferred Close hung instead of reporting the failure. Release idempotently ahead of the server close.

* test: mismatched If-None-Match serves the object over If-Modified-Since

The precedence case asserted the old If-Modified-Since-first ordering and
expected 304. The volume servers now evaluate If-None-Match first, so a
non-matching tag must return 200 with the body even when If-Modified-Since
would allow 304 (RFC 9110 13.1.3).
2026-10-24 15:16:16 -07:00
Chris LuandDevin e442eda88e s3: authorize POST policy uploads through the shared object check (#12047)
* s3: authorize POST policy uploads through the shared object check

doesPolicySignatureMatch gated a verified POST Object signature on the
identity's legacy Write action, which cannot represent attached IAM
policies, STS session policies, or key-scoped grants. Users authorized
only by an attached policy (e.g. s3:PutObject on bucket/*) were denied
with AccessDenied even though the equivalent PUT succeeds.

The handler already runs AuthorizeObjectWrite against the real object
key after the form's x-amz-* fields are promoted to headers, so the
coarse check only shadowed the full evaluation. Signature verification
now just returns the identity; authorization stays in one place.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test: POST Object authorizes attached-policy identities, still denies others

Cover the signature check and AuthorizeObjectWrite for an identity whose
only write grant comes from an attached IAM policy, plus the deny leg
for an identity whose attached policy omits s3:PutObject.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 15:09:34 -07:00
Chris Lu 216c1032e6 test: mismatched If-None-Match serves the object over If-Modified-Since (#12045)
The precedence case asserted the old If-Modified-Since-first ordering and
expected 304. The volume servers now evaluate If-None-Match first, so a
non-matching tag must return 200 with the body even when If-Modified-Since
would allow 304 (RFC 9110 13.1.3).
2026-10-24 15:06:19 -07:00
03f773f087 redis2: pipeline entry reads when listing a directory (#12042)
ListDirectoryEntries fetched every member with its own GET, so a page of
1024 entries cost 1024 round trips. Fetch the values in pipelined batches
of 128 as the loop reaches them and walk the replies in order. Each reply
keeps its own error, so missing and expired members are cleaned up and
skipped exactly as before, and an unreadable member still ends the
listing at that name. Batching keeps a caller that lists with a huge
limit and stops after a few entries from reading the whole directory.

Co-authored-by: Patrick Schmidt <1310072+Woellchen@users.noreply.github.com>
Co-authored-by: user.email <chrislusf@users.noreply.github.com>
2026-10-24 15:04:37 -07:00
Junker der Provinz f9dac64374 helm: remove filer.redirectOnRead, which weed filer does not accept (#12040)
weed filer lost -redirectOnRead in 9b3109a5d (1.59), which also removed
the chart value, but 2bed15f1d added it back. Setting it makes the filer
exit with "flag provided but not defined".
2026-10-24 10:45:57 -07:00
7b43dec665 shell: run volume.configure.replication in parallel per volume server (#12041)
* shell: run volume.configure.replication in parallel per volume server

Volume servers were configured one after another, and each server's
result overwrote the previous error, so a failure on one server was
lost when a later server succeeded. Run one job per volume server,
bounded by -maxParallelization like the other shell commands, and
return the errors from all servers.

VolumeConfigure unmounts and remounts the volume, so replicas of the
same volume on different servers take turns instead of all going
offline at once.

Co-authored-by: NyaMisty <5344431+NyaMisty@users.noreply.github.com>

* shell: test that volume.configure.replication overlaps servers up to the limit

* shell: assert every server ran in the parallel configure-replication test

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: NyaMisty <5344431+NyaMisty@users.noreply.github.com>
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 10:29:38 -07:00
Chris Lu 95af3f3649 docs(volume_server): use repo-relative paths in plan/readme 2026-05-07 22:29:14 -07:00
Chris Lu 7e6db15a88 docs(volume-server): record slash fid-path native validation progress 2026-02-16 05:06:57 -08:00
Chris Lu e66983e9bd feat(rust-volume-server): broaden native slash fid-path validation 2026-02-16 05:06:21 -08:00
Chris Lu b010af88fb docs(volume-server): log native write prevalidation parity progress 2026-02-16 04:37:41 -08:00
Chris Lu 94cefd6f4c feat(rust-volume-server): add native write prevalidation for multipart md5 and size limits 2026-02-16 04:37:11 -08:00
Chris Lu 47156eb8ce docs(volume-server): record native malformed-fid parity milestone 2026-02-16 04:07:51 -08:00
Chris Lu 1ce0174b23 feat(rust-volume-server): validate malformed fid routes in native http path 2026-02-16 04:07:22 -08:00
Chris Lu 120aa956a0 docs(volume-server): update native parity progress logs 2026-02-16 03:36:34 -08:00
Chris Lu d6ff6ed6d4 feat(rust-volume-server): mirror backend healthz and normalize absolute request targets 2026-02-16 03:36:06 -08:00
Chris Lu 532bf863ad docs(volume-server): log native ui static method parity progress 2026-02-16 03:00:10 -08:00
Chris Lu 23e4497b21 feat(rust-volume-server): add native ui static and public no-op handlers 2026-02-16 02:44:14 -08:00
Chris Lu b0447d2479 docs(volume-server): track native rust options parity milestone 2026-02-16 01:52:46 -08:00
Chris Lu fbff2cb39a feat(rust-volume-server): serve native OPTIONS on admin and public listeners 2026-02-16 01:52:02 -08:00
Chris Lu d2a6066181 docs(volume-server): log native rust status healthz milestone 2026-02-16 01:22:40 -08:00
Chris Lu 7e6e0261ab feat(rust-volume-server): serve native /status and /healthz in native mode 2026-02-16 01:22:06 -08:00
Chris Lu c7c7be42ed docs(volume-server): track native launcher bootstrap progress 2026-02-16 00:44:31 -08:00
Chris Lu 2e65966c06 feat(rust-volume-server): default rust launcher mode to native 2026-02-16 00:43:35 -08:00
Chris Lu 61befd10fc ci(volume-server): include native rust mode in smoke matrix 2026-02-16 00:15:45 -08:00
Chris Lu 70ddbee370 feat(rust-volume-server): add native mode bootstrap entrypoint 2026-02-16 00:15:33 -08:00
Chris Lu 14c863dbff docs(volume-server): refocus plan on native rust parity 2026-02-16 00:09:31 -08:00
Chris Lu 6bb9d8bac2 docs(volume_server): log head readDeleted parity coverage 2026-02-15 23:51:50 -08:00
Chris Lu cc80ad3643 test(volume_server/http): add head readDeleted parity coverage 2026-02-15 23:51:37 -08:00
Chris Lu 9009e38f7b docs(volume_server): log ping volume-server unreachable coverage 2026-02-15 23:50:08 -08:00
Chris Lu b9fbb85af2 test(volume_server/grpc): add ping unreachable volume-server target case 2026-02-15 23:49:53 -08:00
Chris Lu 47d3001572 docs(volume_server): log csv query payload parity coverage 2026-02-15 23:48:35 -08:00
Chris Lu a12dd5f8d3 test(volume_server/grpc): cover csv-query payload no-output parity 2026-02-15 23:48:22 -08:00
Chris Lu 8e614486a3 docs(volume_server): log tail-receiver interruption coverage 2026-02-15 23:47:06 -08:00
Chris Lu a5864c3eb6 test(volume_server/grpc): cover tail-receiver source-unavailable branch 2026-02-15 23:46:55 -08:00
Chris Lu 6302809442 docs(volume_server): log tail sender cancellation coverage 2026-02-15 18:45:38 -08:00
Chris Lu 27a80f7607 test(volume_server/grpc): add tail-sender cancellation interruption coverage 2026-02-15 18:45:23 -08:00
Chris Lu ec429e0361 docs(volume_server): log framework port-range hardening and rerun 2026-02-15 18:44:13 -08:00
Chris Lu 90e82b15ce test(volume_server/framework): allocate volume ports within safe grpc-offset range 2026-02-15 18:43:57 -08:00
Chris Lu a3e1ee1653 docs(volume_server): log mkcol method parity coverage 2026-02-15 18:13:51 -08:00
Chris Lu 2ab30900d4 test(volume_server/http): add mkcol unsupported-method parity 2026-02-15 18:13:40 -08:00
Chris Lu 62ee14fa61 docs(volume_server): log read-all-needles multi-volume coverage 2026-02-15 17:58:44 -08:00
Chris Lu ab95a6ef15 test(volume_server/grpc): cover read-all-needles multi-volume success 2026-02-15 17:58:34 -08:00
Chris Lu 24965fd489 docs(volume_server): log head conditional precedence coverage 2026-02-15 17:56:47 -08:00
Chris Lu ed23e290fc test(volume_server/http): expand head conditional precedence coverage 2026-02-15 17:56:36 -08:00
Chris Lu 9b57fb6961 docs(volume_server): log ec batch delete success coverage 2026-02-15 17:55:43 -08:00
Chris Lu 1bb40b6bc5 test(volume_server/grpc): add ec batch delete success coverage 2026-02-15 17:55:29 -08:00
Chris Lu 34e342da63 docs(volume_server): log replicated write failure coverage 2026-02-15 17:45:36 -08:00
Chris Lu 4835d34438 test(volume_server/http): cover replicated write failure when replication unmet 2026-02-15 17:45:25 -08:00
Chris Lu 5814729def docs(volume_server): log ec-only read meta coverage 2026-02-15 14:49:31 -08:00
Chris Lu 37bf9b5ebf test(volume_server/grpc): cover ec-only read needle meta unsupported path 2026-02-15 14:49:18 -08:00
Chris Lu 19201df6d7 docs(volume_server): log oversized upload limit coverage 2026-02-15 14:47:18 -08:00
Chris Lu 4d61cbdeed test(volume_server/http): cover oversized upload file-size limit rejection 2026-02-15 14:47:09 -08:00
Chris Lu 3ce883624e docs(volume_server): log jwt ui access override coverage 2026-02-15 14:45:40 -08:00
Chris Lu de974c05d5 test(volume_server/http): cover jwt ui access override behavior 2026-02-15 14:45:28 -08:00
Chris Lu 7768fda023 docs(volume_server): record proxy-mode validation and CI matrix 2026-02-15 14:29:28 -08:00
Chris Lu 548b3d9a38 ci(volume_server): run rust smoke tests in exec and proxy modes 2026-02-15 14:28:35 -08:00
Chris Lu a7f50d23b5 feat(rust/volume_server): add proxy supervision mode for integration parity 2026-02-15 14:28:12 -08:00
Chris Lu 6ce4d7eded docs(volume_server): record rust-mode full-suite validation 2026-02-15 12:11:31 -08:00
Chris Lu 3bd20e6a10 chore(rust/volume_server): add Cargo.lock 2026-02-15 11:55:42 -08:00
Chris Lu d402573ea8 docs(volume_server): document rust-mode harness and tracking 2026-02-15 11:55:33 -08:00
Chris Lu 63d08e8a91 ci(volume_server): add rust-mode integration smoke job 2026-02-15 11:55:28 -08:00
Chris Lu 880c2e1dab feat(rust/volume_server): add compatibility launcher and migration plan 2026-02-15 11:55:25 -08:00
Chris Lu 7beab85c21 test(volume_server/framework): support selectable volume server binary 2026-02-15 11:55:15 -08:00
58 changed files with 4780 additions and 1340 deletions

No files matched your search

@@ -9,6 +9,7 @@ on:
- 'weed/storage/**'
- 'weed/pb/volume_server.proto'
- 'weed/pb/volume_server_pb/**'
- 'rust/volume_server/**'
- '.github/workflows/volume-server-integration-tests.yml'
push:
branches: [ master, main ]
@@ -18,6 +19,7 @@ on:
- 'weed/storage/**'
- 'weed/pb/volume_server.proto'
- 'weed/pb/volume_server_pb/**'
- 'rust/volume_server/**'
- '.github/workflows/volume-server-integration-tests.yml'
concurrency:
@@ -141,3 +143,77 @@ jobs:
echo "## Volume Server Integration Test Summary (${{ matrix.test-type }} - Shard ${{ matrix.shard }})" >> "$GITHUB_STEP_SUMMARY"
echo "- Suite: test/volume_server/${{ matrix.test-type }} (shard ${{ matrix.shard }} of ${SHARD_COUNT}, see 'Select this shard's tests' for the split)" >> "$GITHUB_STEP_SUMMARY"
echo "- Command: go test -v -count=1 -timeout=${{ env.TEST_TIMEOUT }} ./test/volume_server/${{ matrix.test-type }}/... -run \"\${TEST_PATTERN}\"" >> "$GITHUB_STEP_SUMMARY"
volume-server-rust-smoke:
name: Volume Server Rust Smoke (${{ matrix.mode }})
runs-on: ubuntu-22.04
timeout-minutes: 40
strategy:
fail-fast: false
matrix:
mode: [exec, proxy, native]
steps:
- name: Checkout code
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
- name: Set up Rust toolchain
uses: dtolnay/rust-toolchain@stable
- name: Build SeaweedFS binary
run: |
cd weed
go build -o weed .
chmod +x weed
./weed version
- name: Build Rust volume launcher
run: |
cd rust/volume_server
cargo build --release
- name: Run Rust-mode HTTP smoke tests
env:
WEED_BINARY: ${{ github.workspace }}/weed/weed
VOLUME_SERVER_IMPL: rust
RUST_VOLUME_BINARY: ${{ github.workspace }}/rust/volume_server/target/release/weed-volume-rs
VOLUME_SERVER_RUST_MODE: ${{ matrix.mode }}
run: |
go test -v -count=1 -timeout=25m ./test/volume_server/http/... -run "TestAdminStatusAndHealthz|TestUploadReadRangeHeadDeleteRoundTrip"
- name: Run Rust-mode gRPC smoke tests
env:
WEED_BINARY: ${{ github.workspace }}/weed/weed
VOLUME_SERVER_IMPL: rust
RUST_VOLUME_BINARY: ${{ github.workspace }}/rust/volume_server/target/release/weed-volume-rs
VOLUME_SERVER_RUST_MODE: ${{ matrix.mode }}
run: |
go test -v -count=1 -timeout=25m ./test/volume_server/grpc/... -run "TestStateAndStatusRPCs|TestVolumeSyncStatusAndReadVolumeFileStatus"
- name: Collect logs on failure
if: failure()
run: |
mkdir -p /tmp/volume-server-rust-smoke-logs
find /tmp -maxdepth 1 -type d -name "seaweedfs_volume_server_it_*" -print -exec cp -r {} /tmp/volume-server-rust-smoke-logs/ \; || true
- name: Archive logs on failure
if: failure()
uses: actions/upload-artifact@v7
with:
name: volume-server-rust-smoke-logs-${{ matrix.mode }}
path: /tmp/volume-server-rust-smoke-logs/
if-no-files-found: warn
retention-days: 7
- name: Rust smoke summary
if: always()
run: |
echo "## Volume Server Rust Smoke Summary (${{ matrix.mode }})" >> "$GITHUB_STEP_SUMMARY"
echo "- Mode: Go master + Rust volume server launcher (VOLUME_SERVER_IMPL=rust, VOLUME_SERVER_RUST_MODE=${{ matrix.mode }})" >> "$GITHUB_STEP_SUMMARY"
echo "- HTTP Command: go test -v -count=1 -timeout=25m ./test/volume_server/http/... -run \"TestAdminStatusAndHealthz|TestUploadReadRangeHeadDeleteRoundTrip\"" >> "$GITHUB_STEP_SUMMARY"
echo "- gRPC Command: go test -v -count=1 -timeout=25m ./test/volume_server/grpc/... -run \"TestStateAndStatusRPCs|TestVolumeSyncStatusAndReadVolumeFileStatus\"" >> "$GITHUB_STEP_SUMMARY"
+2 -2
View File
@@ -1,6 +1,6 @@
apiVersion: v1
description: SeaweedFS
name: seaweedfs
appVersion: "4.49"
appVersion: "4.50"
# Dev note: Trigger a helm chart release by `git tag -a helm-<version>`
version: 4.49.0
version: 4.50.0
@@ -170,9 +170,6 @@ spec:
{{- if .Values.filer.metricsIp }}
-metricsIp={{ .Values.filer.metricsIp }} \
{{- end }}
{{- if .Values.filer.redirectOnRead }}
-redirectOnRead \
{{- end }}
{{- if .Values.filer.disableHttp }}
-disableHttp \
{{- end }}
-3
View File
@@ -740,9 +740,6 @@ filer:
# encrypt data on volume servers
encryptVolumeData: false
# Whether proxy or redirect to volume server during file GET request
redirectOnRead: false
# Limit sub dir listing size (default 100000)
dirListLimit: 100000
+1034 -1080
View File
File diff suppressed because it is too large. Load diff

Before

Width:  |  Height:  |  Size: 55 KiB

After

Width:  |  Height:  |  Size: 53 KiB

+7
View File
@@ -0,0 +1,7 @@
# This file is automatically @generated by Cargo.
# It is not intended for manual editing.
version = 4
[[package]]
name = "weed-volume-rs"
version = "0.1.0"
+6
View File
@@ -0,0 +1,6 @@
[package]
name = "weed-volume-rs"
version = "0.1.0"
edition = "2021"
[dependencies]
+245
View File
@@ -0,0 +1,245 @@
# Rust Volume Server Parity Implementation Plan
## Objective
Implement a native Rust volume server that replicates Go volume-server behavior for HTTP and gRPC APIs, so it can become a drop-in replacement validated by existing integration suites.
## Current Focus (2026-02-16)
- Program focus is now Rust implementation parity, not broad test expansion.
- `test/volume_server` is treated as the parity gate.
- Existing Rust launcher modes (`exec`, `proxy`) are transition tools; they are not the final target.
## Current Status
- Rust crate and launcher are in place.
- Integration harness can run:
- Go master + Go volume (default)
- Go master + Rust launcher (`VOLUME_SERVER_IMPL=rust`)
- Rust launcher `proxy` mode has full-suite integration pass while delegating backend handlers to Go.
- Rust launcher `native` mode is wired as the default Rust entrypoint and currently bootstraps via supervised Go backend delegation.
- Native Rust HTTP handlers now serve control/surface paths in `native` mode:
- `/status`, `/healthz`
- `OPTIONS` admin/public behavior (method allow-list + CORS preflight headers)
- `/ui/index.html` with config-driven JWT/access-ui gating
- `/favicon.ico` and `/seaweedfsstatic/*`
- public non-read methods (`POST`/`PUT`/`DELETE`/unsupported verbs) as `200` no-op passthrough parity
- admin unsupported verbs as `400` parity
- Native Rust control-surface parity now also includes:
- `/healthz` status mirroring from backend state transitions (e.g. leave/stopping => `503`)
- absolute-form HTTP request-target normalization before native route matching
- Native Rust HTTP data-path prevalidation now includes:
- early malformed vid/fid rejection (`400`) for GET/HEAD/POST/PUT fid-route shapes before delegation
- slash-form fid-route parsing for `/{vid}/{fid}` and `/{vid}/{fid}/{filename}` with reserved-path exclusions (`/status`, `/healthz`, `/ui/index.html`, `/stats/*`, static assets)
- write-path error prevalidation for fid routes:
- malformed multipart form-data requests without a boundary => `400`
- requests containing `Content-MD5` header => `400` mismatch parity branch
- request bodies exceeding configured `-fileSizeLimitMB` (by `Content-Length`) => `400`
- Native Rust API/storage handlers are not implemented yet.
## Parity Exit Criteria
1. Native mode passes:
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/http`
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/grpc`
2. CI runs native Rust mode integration coverage (at least smoke, then expanded shards).
3. Rust mode defaults to native behavior for integration harness.
4. Go-backend delegation is removed (or retained only as explicit fallback mode).
## Architecture Workstreams
### A. Runtime and Configuration Parity
- [x] Add `native` runtime mode in `weed-volume-rs` (bootstrap delegation path).
- [ ] Parse and honor volume-server CLI/config flags used by integration harness:
- [ ] network/bind ports (`-ip`, `-port`, `-port.grpc`, `-port.public`)
- [ ] master target/config dir/read mode/throttling/JWT-related config
- [ ] size/timeout controls and maintenance state defaults
- [ ] Implement graceful lifecycle behavior (signals, shutdown, readiness).
### B. Native HTTP Surface
- [ ] Admin/control endpoints:
- [x] `GET /status` (native Rust in `native` mode)
- [x] `GET /healthz` (native Rust in `native` mode)
- [x] `OPTIONS` admin/public method+CORS control behavior (native Rust in `native` mode)
- [x] static/UI endpoints currently exercised
- [ ] Data read path parity:
- [ ] fid parsing/path variants
- [ ] conditional headers (`If-Modified-Since`, `If-None-Match`)
- [ ] range handling (single/multi/invalid)
- [ ] deleted reads, auth checks, read-mode branches
- [ ] chunk-manifest and compression/image transformation branches
- [ ] Data write/delete parity:
- [ ] write success/unchanged/error paths
- [ ] replication and file-size-limit paths
- [ ] delete and chunk-manifest delete branches
- [ ] Method/CORS/public-port parity for split admin/public behavior.
### C. Native gRPC Surface
- [ ] Control-plane RPCs:
- [ ] `GetState`, `SetState`, `VolumeServerStatus`, `Ping`, `VolumeServerLeave`
- [ ] admin lifecycle: allocate/mount/unmount/delete/configure/readonly/writable
- [ ] Data RPCs:
- [ ] `ReadNeedleBlob`, `ReadNeedleMeta`, `WriteNeedleBlob`
- [ ] `BatchDelete`, `ReadAllNeedles`
- [ ] sync/copy/receive and status endpoints
- [ ] Stream RPCs:
- [ ] tail sender/receiver
- [ ] vacuum streams
- [ ] query streams
- [ ] Advanced families:
- [ ] erasure coding RPC set
- [ ] tiering/remote fetch
- [ ] scrub/query mode matrix
### D. Storage Compatibility Layer
- [ ] Implement volume data/index handling compatible with Go on-disk format.
- [ ] Preserve cookie/checksum/timestamp semantics used by tests.
- [ ] Match read/write/delete consistency and error mapping behavior.
- [ ] Ensure EC metadata/data-path compatibility with existing files.
### E. Operational Hardening
- [ ] Deterministic startup/readiness and shutdown semantics.
- [ ] Log/error parity sufficient for debugging and CI triage.
- [ ] Concurrency/timeout behavior alignment for throttling and streams.
- [ ] Performance baseline checks vs Go for key flows.
## Milestone Plan
### M0 (Completed): Harness + Launcher Transition
- [x] Rust launcher integrated into harness.
- [x] Proxy mode full-suite validation with Go backend delegation.
### M1: Native Skeleton (Control Plane First)
- [ ] `native` mode boots and serves:
- [x] `/status`, `/healthz`
- [ ] `GetState`, `SetState`, `VolumeServerStatus`, `Ping`, `VolumeServerLeave`
- Gate:
- [x] targeted HTTP/grpc control tests pass in `native` mode (delegated backend path).
### M2: Native Core Data Paths
- [ ] Native HTTP read/write/delete baseline parity.
- [ ] Native gRPC data baseline parity (`Read/WriteNeedle*`, `BatchDelete`, `ReadAllNeedles`).
- Gate:
- core HTTP and gRPC data suites pass in `native` mode.
### M3: Native Stream + Copy/Sync
- [ ] Tail/copy/receive/sync paths in native mode.
- Gate:
- stream/copy families pass in `native` mode.
### M4: Native Advanced Feature Families
- [ ] EC, tiering, scrub/query advanced branches.
- Gate:
- full `/test/volume_server/http` and `/test/volume_server/grpc` pass in `native` mode.
### M5: CI/Cutover
- [x] Add/expand native-mode CI jobs (smoke matrix includes `native`).
- [x] Make native mode default for Rust integration runs.
- [ ] Keep `exec`/`proxy` only as explicit fallback modes during rollout.
## Immediate Next Steps
1. Implement minimal native gRPC state/ping RPC handlers (`GetState`, `SetState`, `VolumeServerStatus`, `Ping`, `VolumeServerLeave`).
2. Expand native HTTP parity from control/static surface into data-path handlers (read/write/delete variants) while preserving current native control behavior.
3. Keep rerunning native-mode integration suites as each delegated branch is replaced.
4. Add mismatch triage notes for each API moved from delegation to native implementation.
## Risk Register
- On-disk format mismatch risk:
- Mitigation: implement format-level compatibility tests early (idx/dat/needle encoding).
- Behavioral drift in edge branches:
- Mitigation: use integration suite failures as primary truth; only add tests for newly discovered untracked branches.
- Stream/concurrency semantic mismatch:
- Mitigation: stabilize with focused interruption/timeout parity tests.
## Progress Log
- Date: 2026-02-15
- Change: Added Rust launcher integration (`exec`) and harness wiring.
- Validation: Rust launcher mode passed smoke and full integration suites while delegating to Go backend.
- Commits: `7beab85c2`, `880c2e1da`, `63d08e8a9`, `d402573ea`, `3bd20e6a1`, `6ce4d7ede`
- Date: 2026-02-15
- Change: Added Rust proxy supervisor mode and validated full integration suite.
- Validation:
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=proxy go test -count=1 ./test/volume_server/http`
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=proxy go test -count=1 ./test/volume_server/grpc`
- Commits: `a7f50d23b`, `548b3d9a3`
- Date: 2026-02-16
- Change: Re-focused plan from test expansion to native Rust implementation parity.
- Validation basis: latest Rust proxy full-suite pass keeps regression baseline stable while native implementation starts.
- Commits: `14c863dbf`
- Date: 2026-02-16
- Change: Added native Rust launcher mode bootstrap, set Rust launcher default mode to `native`, and expanded CI Rust smoke matrix to include `native`.
- Validation:
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/http`
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/grpc`
- `env VOLUME_SERVER_IMPL=rust go test -count=1 ./test/volume_server/http -run '^TestAdminStatusAndHealthz$'`
- `env VOLUME_SERVER_IMPL=rust go test -count=1 ./test/volume_server/grpc -run '^TestStateAndStatusRPCs$'`
- Commits: `70ddbee37`, `61befd10f`, `2e65966c0`
- Date: 2026-02-16
- Change: Implemented first native Rust HTTP handlers in `native` mode for `/status` and `/healthz` on the admin listener while preserving proxy delegation for other APIs.
- Validation:
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/http -run '^TestAdminStatusAndHealthz$'`
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/http`
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/grpc`
- Commits: `7e6e0261a`
- Date: 2026-02-16
- Change: Implemented native Rust `OPTIONS` handling for admin and public listeners in `native` mode, including method allow-list and origin-driven CORS response parity used by integration tests.
- Validation:
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/http -run '^TestOptionsMethodsByPort$|^TestOptionsWithOriginIncludesCorsHeaders$'`
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/http`
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/grpc`
- Commits: `fbff2cb39`
- Date: 2026-02-16
- Change: Added native Rust UI/static/public-surface handlers in `native` mode:
- `/ui/index.html` with config-driven JWT/access-ui gating parity
- `/favicon.ico` and `/seaweedfsstatic/*` static endpoint parity
- public non-read methods as `200` no-op passthrough parity
- admin unsupported methods as `400` parity
- Validation:
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/http -run '^TestAdminStatusAndHealthz$|^TestUiIndexNotExposedWhenJwtSigningEnabled$|^TestUiIndexExposedWhenJwtSigningEnabledAndAccessUITrue$|^TestStaticAssetEndpoints$|^TestStaticAssetEndpointsOnPublicPort$'`
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/http -run '^TestOptionsMethodsByPort$|^TestOptionsWithOriginIncludesCorsHeaders$|^TestPublicPortReadOnlyMethodBehavior$|^TestCorsAndUnsupportedMethodBehavior$|^TestUnsupportedMethodTraceParity$|^TestUnsupportedMethodPropfindParity$|^TestUnsupportedMethodConnectParity$|^TestUnsupportedMethodMkcolParity$'`
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/http`
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/grpc`
- Commits: `23e4497b2`
- Date: 2026-02-16
- Change: Hardened native Rust HTTP control-path parity:
- `/healthz` now mirrors backend health status (`200`/`503`) in native mode
- absolute-form request-target parsing (e.g. `GET http://host/path HTTP/1.1`) now normalizes to route path parity
- Validation:
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/http -run '^TestAdminStatusAndHealthz$'`
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/http/...`
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/grpc/...`
- Commits: `d6ff6ed6d`
- Date: 2026-02-16
- Change: Added native malformed fid-route validation in `native` mode:
- GET/HEAD/POST/PUT requests matching fid URL shapes now return Rust-native `400 Bad Request` for invalid vid/fid tokens
- valid fid-route requests still delegate to backend handlers for storage/data execution paths
- Validation:
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/http -run '^TestInvalidReadPathReturnsBadRequest$|^TestWriteInvalidVidAndFidReturnBadRequest$|^TestMalformedVidFidPathReturnsBadRequest$|^TestDownloadLimitInvalidVidWhileOverLimitReturnsBadRequest$'`
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/http/...`
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/grpc/...`
- Commits: `1ce0174b2`
- Date: 2026-02-16
- Change: Added native Rust write-path prevalidation branches for fid routes in `native` mode:
- multipart form uploads without boundary now return native `400`
- `Content-MD5` requests now return native `400` mismatch branch
- `Content-Length` over configured `-fileSizeLimitMB` now returns native `400` limit branch
- Validation:
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/http -run '^TestUploadReadRangeHeadDeleteRoundTrip$|^TestWriteMalformedMultipartAndMD5Mismatch$|^TestWriteRejectsPayloadOverFileSizeLimit$'`
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/http/...`
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/grpc/...`
- Commits: `94cefd6f4`
- Date: 2026-02-16
- Change: Broadened native fid-route parser to cover slash-form URLs:
- `/{vid}/{fid}` and `/{vid}/{fid}/{filename}` now flow through native malformed vid/fid validation
- added explicit non-fid exclusions for control/stats/static paths to preserve route parity
- Validation:
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/http -run '^TestAdminStatusAndHealthz$|^TestReadPathShapesAndIfModifiedSince$|^TestMalformedVidFidPathReturnsBadRequest$'`
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/http/...`
- `env VOLUME_SERVER_IMPL=rust VOLUME_SERVER_RUST_MODE=native go test -count=1 ./test/volume_server/grpc/...`
- Commits: `e66983e9b`
File diff suppressed because it is too large. Load diff
+52 -40
View File
@@ -3039,50 +3039,62 @@ pub async fn post_handler(
let needle_etag = n.etag();
let needle_has_name = n.has_name();
let write_result = write_upload_needle(&state, vid, n, fsync).await;
// Replicate to remote volume servers if this volume has replicas.
// Matches Go's GetWritableRemoteReplications: skip if copy_count == 1.
let needs_replication = !is_replicate && {
let store = state.store.read().unwrap();
store
.find_volume(vid)
.is_some_and(|(_, v)| v.super_block().replica_placement.get_copy_count() > 1)
};
// The fan-out runs concurrently with the local needle write, like Go's
// ReplicatedWrite: it reads the already-buffered request body, so the two
// sides share no mutable state. The ack still waits for both.
let replication = if needs_replication {
let state_clone = state.clone();
let path_clone = path.clone();
let query_clone = query.clone();
let headers_clone = headers.clone();
let body_clone = body.clone();
Some(tokio::spawn(async move {
do_replicated_request(
&state_clone,
vid.0,
Method::POST,
&path_clone,
&query_clone,
&headers_clone,
Some(body_clone),
)
.await
}))
} else {
None
};
let write_result = write_upload_needle(&state, vid, n, fsync).await;
// No leader-known gate: when the lookup cannot run, the write must fail
// rather than be acknowledged on fewer copies than required.
if !is_replicate && write_result.is_ok() {
let needs_replication = {
let store = state.store.read().unwrap();
store
.find_volume(vid)
.is_some_and(|(_, v)| v.super_block().replica_placement.get_copy_count() > 1)
};
if needs_replication {
let state_clone = state.clone();
let path_clone = path.clone();
let query_clone = query.clone();
let headers_clone = headers.clone();
let body_clone = body.clone();
let replication = tokio::spawn(async move {
do_replicated_request(
&state_clone,
vid.0,
Method::POST,
&path_clone,
&query_clone,
&headers_clone,
Some(body_clone),
)
.await
});
let replication_result = replication
.await
// rather than be acknowledged on fewer copies than required. The local
// write result takes precedence, matching Go's error ordering.
let replication_result = match replication {
Some(task) => Some(
task.await
.map_err(|e| format!("replication task failed: {}", e))
.flatten();
if let Err(e) = replication_result {
tracing::error!("replicated write failed: {}", e);
return json_error_with_query(
StatusCode::INTERNAL_SERVER_ERROR,
format!("replication failed: {}", e),
Some(&query),
);
}
}
.flatten(),
),
None => None,
};
if write_result.is_ok()
&& let Some(Err(e)) = replication_result
{
tracing::error!("replicated write failed: {}", e);
return json_error_with_query(
StatusCode::INTERNAL_SERVER_ERROR,
format!("replication failed: {}", e),
Some(&query),
);
}
let resp = match write_result {
@@ -466,6 +466,7 @@ func (env *TestEnvironment) startTrinoContainer(t *testing.T, configDir string)
cmd := exec.Command("docker", "run", "-d",
"--name", containerName,
"--memory", "4g",
"--add-host", "host.docker.internal:host-gateway",
"-v", fmt.Sprintf("%s:/etc/trino/catalog", configDir),
"-v", fmt.Sprintf("%s:/test", env.dataDir),
@@ -484,7 +485,7 @@ func waitForTrino(t *testing.T, containerName string, timeout time.Duration) {
deadline := time.Now().Add(timeout)
var lastOutput []byte
retryCount := 0
containerDead := false
for time.Now().Before(deadline) {
// Try system catalog query as a readiness check
cmd := exec.Command("docker", "exec", containerName,
@@ -495,26 +496,51 @@ func waitForTrino(t *testing.T, containerName string, timeout time.Duration) {
return
} else {
lastOutput = output
outputStr := string(output)
if strings.Contains(outputStr, "No such container") ||
strings.Contains(outputStr, "is not running") {
if trinoContainerExited(output) {
containerDead = true
break
}
retryCount++
}
time.Sleep(1 * time.Second)
}
// If we can't connect to system catalog, try to at least connect to Trino server
cmd := exec.Command("docker", "exec", containerName, "trino", "--version")
if err := cmd.Run(); err == nil {
if output, err := exec.Command("docker", "exec", containerName, "trino", "--version").CombinedOutput(); err == nil {
// Trino process is running, even if catalog isn't ready yet
// Give it a bit more time
time.Sleep(5 * time.Second)
return
} else {
lastOutput = output
containerDead = containerDead || trinoContainerExited(output)
}
t.Fatalf("Timed out waiting for Trino to be ready\nLast output:\n%s", string(lastOutput))
if containerDead {
t.Fatalf("Trino container %s exited before becoming ready\nLast output:\n%s\n%s", containerName, string(lastOutput), trinoContainerDiagnostics(containerName))
}
t.Fatalf("Timed out waiting for Trino to be ready\nLast output:\n%s\n%s", string(lastOutput), trinoContainerDiagnostics(containerName))
}
func trinoContainerExited(output []byte) bool {
s := string(output)
return strings.Contains(s, "No such container") || strings.Contains(s, "is not running")
}
func trinoContainerDiagnostics(containerName string) string {
var b strings.Builder
if out, err := exec.Command("docker", "inspect", "-f",
"status={{.State.Status}} exitCode={{.State.ExitCode}} oomKilled={{.State.OOMKilled}} error={{.State.Error}}",
containerName).CombinedOutput(); err == nil {
fmt.Fprintf(&b, "docker inspect: %s\n", strings.TrimSpace(string(out)))
} else {
fmt.Fprintf(&b, "docker inspect failed: %v (%s)\n", err, strings.TrimSpace(string(out)))
}
if out, err := exec.Command("docker", "logs", "--tail", "80", containerName).CombinedOutput(); err == nil {
fmt.Fprintf(&b, "docker logs:\n%s", string(out))
} else {
fmt.Fprintf(&b, "docker logs failed: %v (%s)", err, strings.TrimSpace(string(out)))
}
return b.String()
}
func runTrinoSQL(t *testing.T, containerName, sql string) string {
+205
View File
@@ -3,6 +3,12 @@
## Goal
Create a Go integration test suite under `test/volume_server` that validates **drop-in behavior parity** for the Volume Server HTTP and gRPC APIs, so a Rust rewrite can be verified against the current Go behavior.
## Current Program Focus (2026-02-16)
- Primary execution focus has shifted to implementing native Rust volume-server parity.
- This integration suite is now the parity gate for Rust implementation work.
- New tests should be added only when native Rust implementation reveals uncovered Go behavior that is not yet captured.
- Rust implementation roadmap lives in `rust/volume_server/DEV_PLAN.md`.
## Hard Requirements
- Tests live under `test/volume_server`.
- Tests are written in Go.
@@ -1127,3 +1133,202 @@ Update this section during implementation:
- Profiles covered: P1.
- Gaps introduced/remaining: `499` cancellation and transport-interruption branches remain pending.
- Commit: `d1e5f390a`
- Date: 2026-02-15
- Change: Added Rust volume-server migration bootstrap wiring.
- APIs covered: Harness now supports Go master + selectable volume binary (`VOLUME_SERVER_IMPL=rust` or `VOLUME_SERVER_BINARY`), with Rust-mode smoke coverage for representative HTTP/gRPC integration tests.
- Profiles covered: P1 smoke and default Go matrix unchanged.
- Gaps introduced/remaining: native Rust handler implementations are tracked in `rust/volume_server/DEV_PLAN.md`; current Rust mode is a compatibility launcher phase.
- Commit: `7beab85c2`, `880c2e1da`, `63d08e8a9`, `d402573ea`
- Date: 2026-02-15
- Change: Validated Rust-mode compatibility path with full integration suites.
- APIs covered: full `/test/volume_server/http` and `/test/volume_server/grpc` suites executed successfully with `VOLUME_SERVER_IMPL=rust`.
- Profiles covered: existing integration matrix in Rust launcher mode.
- Gaps introduced/remaining: native Rust endpoint and storage/RPC implementation remains pending; current mode delegates execution to Go volume server.
- Commit: `6ce4d7ede`
- Date: 2026-02-15
- Change: Added Rust proxy-supervision launcher mode and validated all volume-server integration suites in proxy mode.
- APIs covered: full `/test/volume_server/http` and `/test/volume_server/grpc` suites executed successfully with `VOLUME_SERVER_IMPL=rust` and `VOLUME_SERVER_RUST_MODE=proxy`.
- Profiles covered: existing integration matrix in Rust proxy launcher mode.
- Gaps introduced/remaining: native Rust endpoint/storage/RPC logic remains pending; current proxy mode still delegates backend handlers to Go.
- Commit: `a7f50d23b`
- Date: 2026-02-15
- Change: Expanded GitHub CI Rust smoke job to run both launcher modes (`exec` and `proxy`).
- APIs covered: representative HTTP/gRPC smoke flows now validated in CI for both Rust launcher execution paths.
- Profiles covered: CI smoke profile for P1 representative calls.
- Gaps introduced/remaining: CI still runs Rust mode as smoke scope; full-suite Rust-mode CI remains optional due runtime cost.
- Commit: `548b3d9a3`
- Date: 2026-02-15
- Change: Added JWT UI override integration coverage with explicit `access.ui=true` profile support.
- APIs covered: `/ui/index.html` behavior under JWT profile now verifies both default auth-gated path (`401`) and explicit UI-enable override path (`200` with rendered UI).
- Profiles covered: P3 and P3+`access.ui=true`.
- Gaps introduced/remaining: UI exposure behavior under JWT profiles is now covered for both secured and override branches.
- Commit: `de974c05d`
- Date: 2026-02-15
- Change: Added oversized upload integration coverage using explicit `-fileSizeLimitMB` profile configuration.
- APIs covered: HTTP write path now verifies payloads larger than configured file-size limit are rejected (`400`) with limit-related error context.
- Profiles covered: P1-derived profile with `FileSizeLimitMB=1`.
- Gaps introduced/remaining: file-size limit rejection branch is now covered; replicate-write failure and cancellation (`499`) branches remain pending.
- Commit: `4d61cbdee`
- Date: 2026-02-15
- Change: Added gRPC EC-only metadata-read unsupported-path coverage.
- APIs covered: `ReadNeedleMeta` now verifies the explicit unsupported branch when a volume is unmounted and only EC shards are mounted.
- Profiles covered: P1 with EC shard generation/mount lifecycle setup.
- Gaps introduced/remaining: low-level read metadata fault-injection and transport interruption branches remain pending.
- Commit: `37bf9b5eb`
- Date: 2026-02-15
- Change: Added deterministic replicated-write failure coverage for unmet replication requirements.
- APIs covered: HTTP write path now verifies replication error handling (`500`) and no-local-commit outcome (`404` on follow-up read) when volume replication is configured (`001`) but replica set cannot be satisfied.
- Profiles covered: P1-derived single-node profile with per-volume replication override.
- Gaps introduced/remaining: replicated-write failure branch is now covered; cancellation (`499`) and deeper transport fault-injection branches remain pending.
- Commit: `4835d3443`
- Date: 2026-02-15
- Change: Added EC-backed `BatchDelete` positive-path integration coverage.
- APIs covered: `BatchDelete` now validates successful EC-needle deletion (`202`) with post-delete `VolumeEcShardRead` deleted-marker verification (`IsDeleted=true`).
- Profiles covered: P1 with EC shard generate/mount lifecycle; includes test-local gRPC offset proxy for EC internal dial-path parity.
- Gaps introduced/remaining: EC `BatchDelete` success branch is now covered; deeper distributed failure-injection permutations remain pending.
- Commit: `1bb40b6bc`
- Date: 2026-02-15
- Change: Expanded HTTP conditional-header matrix for `HEAD` requests with combined header scenarios.
- APIs covered: `HEAD` now verifies `If-Modified-Since` precedence over mismatched `If-None-Match`, and `If-None-Match` precedence when IMS is stale but ETag matches.
- Profiles covered: P1.
- Gaps introduced/remaining: combined conditional-header precedence paths for `HEAD`/`GET` are now explicitly covered.
- Commit: `ed23e290f`
- Date: 2026-02-15
- Change: Added multi-volume success-stream coverage for `ReadAllNeedles`.
- APIs covered: `ReadAllNeedles` now verifies happy-path streaming across two existing volumes in one request, with per-volume payload validation.
- Profiles covered: P1.
- Gaps introduced/remaining: multi-volume stream boundary coverage is now present for successful reads; transport interruption branches remain pending.
- Commit: `ab95a6ef1`
- Date: 2026-02-15
- Change: Expanded split-port unsupported HTTP method matrix with `MKCOL` coverage.
- APIs covered: admin/public parity for `MKCOL` (`400` on admin, passthrough `200` on public) with post-call data-integrity verification.
- Profiles covered: P2.
- Gaps introduced/remaining: unsupported-method sampling now covers `PATCH`, `TRACE`, `PROPFIND`, `CONNECT`, and `MKCOL`; cancellation/transport fault paths remain the main unaddressed area.
- Commit: `2ab30900d`
- Date: 2026-02-15
- Change: Hardened framework port allocation to keep volume ports within `10000..55535`.
- APIs covered: test harness now guarantees valid `admin+10000` gRPC offset ranges, eliminating EC batch-delete flake scenarios that rely on offset dialing.
- Profiles covered: all profiles using shared framework port allocation.
- Gaps introduced/remaining: no new API gaps; transport cancellation/fault-injection branches remain the primary uncovered area.
- Commit: `90e82b15c`
- Date: 2026-02-15
- Change: Re-validated full Rust proxy-mode integration suite after latest HTTP/gRPC coverage additions and framework hardening.
- APIs covered: full `/test/volume_server/http` and `/test/volume_server/grpc` packages.
- Profiles covered: existing matrix in Rust proxy launcher mode.
- Gaps introduced/remaining: native Rust endpoint/storage/RPC implementation remains pending (current Rust mode still delegates to Go backend handlers).
- Commit: `6bb9d8bac`
- Date: 2026-02-15
- Change: Added tail sender stream-cancellation interruption coverage.
- APIs covered: `VolumeTailSender` now verifies client-side context cancellation behavior (`codes.Canceled`) after stream start/heartbeat.
- Profiles covered: P1.
- Gaps introduced/remaining: tail transport interruption coverage is now partially closed; receiver-side interruption and deeper network fault injection remain pending.
- Commit: `27a80f760`
- Date: 2026-02-15
- Change: Added receiver-side tail stream interruption coverage for unavailable source node.
- APIs covered: `VolumeTailReceiver` now verifies source connect/dial failure branch when `SourceVolumeServer` is unreachable.
- Profiles covered: P1.
- Gaps introduced/remaining: both sender and receiver transport interruption branches are now covered at integration level; deeper injected network faults remain pending.
- Commit: `a5864c3eb`
- Date: 2026-02-15
- Change: Expanded query CSV matrix with explicit CSV-payload parity coverage.
- APIs covered: `Query` now verifies current CSV-input behavior (no streamed rows / immediate EOF) even when source needle content is valid CSV text.
- Profiles covered: P1.
- Gaps introduced/remaining: CSV parsing/selection implementation still absent in current server behavior; integration parity now locks the current no-output semantics for both JSON and CSV payload shapes.
- Commit: `a12dd5f8d`
- Date: 2026-02-15
- Change: Expanded ping unreachable-target matrix with volume-server target coverage.
- APIs covered: `Ping` now explicitly verifies unreachable-target error wrapping for `target_type=volumeServer`, alongside existing master/filer variants.
- Profiles covered: P1.
- Gaps introduced/remaining: ping target-type matrix is covered for success and unreachable branches across master/filer/volume-server.
- Commit: `b9fbb85af`
- Date: 2026-02-15
- Change: Expanded deleted-read HTTP matrix with `HEAD` parity coverage.
- APIs covered: `HEAD` with `readDeleted=true` now validates current behavior (`200`, empty body, payload-size `Content-Length`) after delete.
- Profiles covered: P1.
- Gaps introduced/remaining: deleted-read parity now covers both `GET` and `HEAD` semantics on local-volume path.
- Commit: `cc80ad364`
- Date: 2026-02-16
- Change: Shifted planning priority to native Rust implementation parity.
- APIs covered: no new API additions in this plan entry; integration suite remains the validation gate.
- Profiles covered: unchanged.
- Gaps introduced/remaining: primary remaining gap is native Rust handler/storage/RPC implementation replacing Go backend delegation.
- Commit: `14c863dbf`
- Date: 2026-02-16
- Change: Added Rust launcher `native` mode bootstrap, made it the default Rust launcher mode, and expanded CI Rust smoke matrix coverage to include `native`.
- APIs covered: full `/test/volume_server/http` and `/test/volume_server/grpc` packages re-validated in `VOLUME_SERVER_RUST_MODE=native`; default Rust launcher path (`VOLUME_SERVER_IMPL=rust`) smoke-validated for HTTP and gRPC control tests.
- Profiles covered: existing HTTP/gRPC matrix in native launcher mode; default-mode smoke checks on P1 control flows.
- Gaps introduced/remaining: native Rust API/storage/RPC handlers still pending; current `native` mode remains a delegation bootstrap while parity replacement proceeds.
- Commit: `70ddbee37`, `61befd10f`, `2e65966c0`
- Date: 2026-02-16
- Change: Implemented first native Rust HTTP control handlers in `native` mode for `/status` and `/healthz` (admin listener path), with proxy fallback retained for remaining endpoints.
- APIs covered: `/status` and `/healthz` now served by Rust-native code path in `VOLUME_SERVER_RUST_MODE=native`; full HTTP and gRPC integration suites re-validated.
- Profiles covered: full existing HTTP/gRPC matrix under native mode.
- Gaps introduced/remaining: gRPC control/data APIs and most HTTP data/static/auth paths remain delegated and require incremental native replacement.
- Commit: `7e6e0261a`
- Date: 2026-02-16
- Change: Implemented native Rust HTTP `OPTIONS` handling for admin/public listeners in `VOLUME_SERVER_RUST_MODE=native`.
- APIs covered: admin/public `OPTIONS` method allow-list and CORS origin handling now served from Rust-native path; full HTTP and gRPC packages re-validated.
- Profiles covered: P2 split-port CORS/OPTIONS profiles plus full existing matrix in native mode.
- Gaps introduced/remaining: static/UI/auth and data-path HTTP handlers, plus gRPC control/data/stream handlers, remain delegated and pending native replacement.
- Commit: `fbff2cb39`
- Date: 2026-02-16
- Change: Implemented broader native Rust HTTP surface in `native` mode:
- `/ui/index.html` auth-gating parity (JWT + access.ui handling)
- static asset paths (`/favicon.ico`, `/seaweedfsstatic/*`)
- public non-read method no-op parity (`200`) and admin unsupported-method rejection parity (`400`)
- APIs covered: UI/static/admin/public control-surface behavior now served by Rust-native path; full `/test/volume_server/http` and `/test/volume_server/grpc` packages re-validated.
- Profiles covered: P1/P2/P3 variants touched by UI/static/CORS/method behavior, plus full matrix in native mode.
- Gaps introduced/remaining: HTTP data-path handlers and all gRPC handlers remain delegated and still require native replacement.
- Commit: `23e4497b2`
- Date: 2026-02-16
- Change: Improved native Rust control-path parity for health/status and request-target parsing.
- APIs covered: native `/healthz` now mirrors backend service health transitions (`200`/`503`) and native route matching now handles absolute-form HTTP request targets before path dispatch.
- Profiles covered: full existing HTTP/gRPC integration matrix in native mode (`VOLUME_SERVER_IMPL=rust`, `VOLUME_SERVER_RUST_MODE=native`).
- Gaps introduced/remaining: core HTTP data handlers and all gRPC RPC handlers are still delegated and remain the primary native implementation gap.
- Commit: `d6ff6ed6d`
- Date: 2026-02-16
- Change: Added Rust-native malformed fid-route validation ahead of delegated data handlers.
- APIs covered: GET/HEAD/POST/PUT fid-shaped paths now return native `400` for invalid vid/fid tokens (including invalid-read and invalid-write path variants) while valid routes continue through delegated storage handlers.
- Profiles covered: full existing HTTP/gRPC integration matrix in native mode, plus targeted invalid-path parity runs.
- Gaps introduced/remaining: data-path success branches and all gRPC handler bodies remain delegated; native replacement work continues on those core execution paths.
- Commit: `1ce0174b2`
- Date: 2026-02-16
- Change: Added Rust-native write error prevalidation for delegated fid routes.
- APIs covered: native path now rejects malformed multipart boundary requests, `Content-MD5` writes, and over-limit `Content-Length` writes (using `-fileSizeLimitMB`) with `400` parity behavior before delegation.
- Profiles covered: P1 default and P1 with explicit file-size limit profile, plus full existing HTTP/gRPC native-mode matrix revalidation.
- Gaps introduced/remaining: positive write/read/delete data handlers and all gRPC handler bodies are still delegated; native implementation continues incrementally.
- Commit: `94cefd6f4`
- Date: 2026-02-16
- Change: Broadened Rust-native malformed-fid handling to slash-form read/write URL shapes with reserved-route exclusions.
- APIs covered: `/{vid}/{fid}` and `/{vid}/{fid}/{filename}` malformed cases now hit native `400` validation path, while `/status`, `/healthz`, `/ui/index.html`, `/stats/*`, and static asset routes remain excluded from fid parsing.
- Profiles covered: full existing HTTP/gRPC native-mode matrix, plus targeted admin/read-path malformed-slash validation run.
- Gaps introduced/remaining: successful data-path read/write/delete handlers and all gRPC method bodies remain delegated and are still the primary native implementation gap.
- Commit: `e66983e9b`
+4 -1
View File
@@ -1,7 +1,10 @@
.PHONY: test-volume-server test-volume-server-short
.PHONY: test-volume-server test-volume-server-short test-volume-server-rust-smoke
test-volume-server:
go test ./test/volume_server/... -v
test-volume-server-short:
go test ./test/volume_server/... -short -v
test-volume-server-rust-smoke:
VOLUME_SERVER_IMPL=rust go test ./test/volume_server/http ./test/volume_server/grpc -run 'TestAdminStatusAndHealthz|TestUploadReadRangeHeadDeleteRoundTrip|TestStateAndStatusRPCs|TestVolumeSyncStatusAndReadVolumeFileStatus' -v
+4 -1
View File
@@ -15,6 +15,9 @@ If a `weed` binary is not found, the harness will build one automatically.
## Optional environment variables
- `WEED_BINARY`: explicit path to the `weed` executable (disables auto-build).
- `VOLUME_SERVER_IMPL`: select volume-server implementation (`go` default, `rust` for a Rust volume server).
- `RUST_VOLUME_BINARY`: explicit path to the Rust volume-server executable (overrides crate auto-detect).
- `VOLUME_SERVER_RUST_MODE`: `weed-volume-rs` launcher mode (`native` default, `exec` for direct Go delegation, `proxy` for Rust front proxy + Go backend).
- `VOLUME_SERVER_IT_KEEP_LOGS=1`: keep temporary test directories and process logs.
## Current scope (Phase 0)
@@ -24,4 +27,4 @@ If a `weed` binary is not found, the harness will build one automatically.
- Initial HTTP admin endpoint checks
- Initial gRPC state/status checks
More API coverage is tracked in `/Users/chris/dev/seaweedfs2/test/volume_server/DEV_PLAN.md`.
More API coverage is tracked in `test/volume_server/DEV_PLAN.md`.
+3
View File
@@ -287,6 +287,9 @@ func (c *Cluster) startVolume(dataDirs []string) error {
if c.profile.InflightDownloadTimeout > 0 {
args = append(args, "-inflightDownloadDataTimeout="+c.profile.InflightDownloadTimeout.String())
}
if c.profile.FileSizeLimitMB > 0 {
args = append(args, "-fileSizeLimitMB="+strconv.Itoa(c.profile.FileSizeLimitMB))
}
c.volumeCmd = exec.Command(c.weedBinary, args...)
c.volumeCmd.Dir = c.baseDir
@@ -265,6 +265,9 @@ func (c *MixedVolumeCluster) startGoVolume(index int, dataDir string) error {
if c.profile.InflightDownloadTimeout > 0 {
args = append(args, "-inflightDownloadDataTimeout="+c.profile.InflightDownloadTimeout.String())
}
if c.profile.FileSizeLimitMB > 0 {
args = append(args, "-fileSizeLimitMB="+strconv.Itoa(c.profile.FileSizeLimitMB))
}
cmd := exec.Command(c.weedBinary, args...)
cmd.Dir = c.baseDir
@@ -267,6 +267,9 @@ func (c *MultiVolumeCluster) startVolume(index int, dataDirs []string) error {
if c.profile.InflightDownloadTimeout > 0 {
args = append(args, "-inflightDownloadDataTimeout="+c.profile.InflightDownloadTimeout.String())
}
if c.profile.FileSizeLimitMB > 0 {
args = append(args, "-fileSizeLimitMB="+strconv.Itoa(c.profile.FileSizeLimitMB))
}
cmd := exec.Command(c.weedBinary, args...)
cmd.Dir = c.baseDir
+14 -4
View File
@@ -214,6 +214,9 @@ func rustVolumeArgs(
if profile.InflightDownloadTimeout > 0 {
args = append(args, "--inflightDownloadDataTimeout", profile.InflightDownloadTimeout.String())
}
if profile.FileSizeLimitMB > 0 {
args = append(args, "--fileSizeLimitMB", strconv.Itoa(profile.FileSizeLimitMB))
}
return args
}
@@ -252,12 +255,21 @@ func FindOrBuildRustBinary() (string, error) {
rustBinaryOnce.Do(func() {
// Derive the Rust volume crate directory from this source file's location.
rustCrateDir := ""
releaseBin := ""
if _, file, _, ok := runtime.Caller(0); ok {
repoRoot := filepath.Clean(filepath.Join(filepath.Dir(file), "..", "..", ".."))
for _, candidate := range []string{"seaweed-volume", "weed-volume"} {
dir := filepath.Join(repoRoot, candidate)
for _, candidate := range []struct {
dir string
bin string
}{
{"seaweed-volume", "weed-volume"},
{"weed-volume", "weed-volume"},
{filepath.Join("rust", "volume_server"), "weed-volume-rs"},
} {
dir := filepath.Join(repoRoot, candidate.dir)
if isDir(dir) && isFile(filepath.Join(dir, "Cargo.toml")) {
rustCrateDir = dir
releaseBin = filepath.Join(dir, "target", "release", candidate.bin)
break
}
}
@@ -267,8 +279,6 @@ func FindOrBuildRustBinary() (string, error) {
return
}
releaseBin := filepath.Join(rustCrateDir, "target", "release", "weed-volume")
// Always rebuild once per test process so the harness uses current source
// and features. The crate defaults to 5bytes, so a test binary built
// without 5BytesOffset has to turn it off or the Rust server refuses the
+13 -5
View File
@@ -16,31 +16,39 @@ import (
func AllocateVolume(t testing.TB, client volume_server_pb.VolumeServerClient, volumeID uint32, collection string) {
t.Helper()
allocateVolume(t, client, volumeID, collection, "")
allocateVolume(t, client, volumeID, collection, "", "")
}
func AllocateVolumeWithReplication(t testing.TB, client volume_server_pb.VolumeServerClient, volumeID uint32, collection, replication string) {
t.Helper()
allocateVolume(t, client, volumeID, collection, replication, "")
}
// AllocateVolumeWithTTL allocates a volume whose needles expire after ttl
// (e.g. "10m").
func AllocateVolumeWithTTL(t testing.TB, client volume_server_pb.VolumeServerClient, volumeID uint32, collection string, ttl string) {
t.Helper()
allocateVolume(t, client, volumeID, collection, ttl)
allocateVolume(t, client, volumeID, collection, "", ttl)
}
func allocateVolume(t testing.TB, client volume_server_pb.VolumeServerClient, volumeID uint32, collection string, ttl string) {
func allocateVolume(t testing.TB, client volume_server_pb.VolumeServerClient, volumeID uint32, collection string, replication, ttl string) {
t.Helper()
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
if replication == "" {
replication = "000"
}
_, err := client.AllocateVolume(ctx, &volume_server_pb.AllocateVolumeRequest{
VolumeId: volumeID,
Collection: collection,
Replication: "000",
Replication: replication,
Ttl: ttl,
Version: uint32(needle.GetCurrentVersion()),
})
if err != nil {
t.Fatalf("allocate volume %d: %v", volumeID, err)
t.Fatalf("allocate volume %d (replication=%s): %v", volumeID, replication, err)
}
}
@@ -456,6 +456,19 @@ func TestPingUnknownAndUnreachableTargetPaths(t *testing.T) {
if got := status.Code(err); got != codes.InvalidArgument {
t.Fatalf("Ping filer expected InvalidArgument, got %s: %v", got, err)
}
// Volume-server targets are not tracked peers either, so they are
// rejected at admission the same way filer targets are.
_, err = grpcClient.Ping(ctx, &volume_server_pb.PingRequest{
TargetType: cluster.VolumeServerType,
Target: "127.0.0.1:1.1",
})
if err == nil {
t.Fatalf("Ping volume server target should be rejected by admission")
}
if got := status.Code(err); got != codes.InvalidArgument {
t.Fatalf("Ping volume server expected InvalidArgument, got %s: %v", got, err)
}
}
func TestPingMasterTargetSuccess(t *testing.T) {
@@ -3,7 +3,10 @@ package volume_server_grpc_test
import (
"bytes"
"context"
"io"
"net"
"net/http"
"strconv"
"strings"
"testing"
"time"
@@ -265,3 +268,141 @@ func TestBatchDeleteRejectsChunkManifestNeedles(t *testing.T) {
t.Fatalf("chunk manifest should not be deleted by BatchDelete reject path, got %d", readResp.StatusCode)
}
}
func TestBatchDeleteEcNeedleSuccess(t *testing.T) {
if testing.Short() {
t.Skip("skipping integration test in short mode")
}
cluster := framework.StartSingleVolumeCluster(t, matrix.P1())
conn, client := framework.DialVolumeServer(t, cluster.VolumeGRPCAddress())
defer conn.Close()
if stopProxy := maybeStartGrpcOffsetProxy(t, cluster.VolumeAdminAddress(), cluster.VolumeGRPCAddress()); stopProxy != nil {
t.Cleanup(stopProxy)
}
const volumeID = uint32(34)
const needleID = uint64(930001)
const cookie = uint32(0x6677BBCC)
framework.AllocateVolume(t, client, volumeID, "")
httpClient := framework.NewHTTPClient()
fid := framework.NewFileID(volumeID, needleID, cookie)
uploadResp := framework.UploadBytes(t, httpClient, cluster.VolumeAdminURL(), fid, []byte("batch-delete-ec-success"))
_ = framework.ReadAllAndClose(t, uploadResp)
if uploadResp.StatusCode != http.StatusCreated {
t.Fatalf("upload expected 201, got %d", uploadResp.StatusCode)
}
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
_, err := client.VolumeEcShardsGenerate(ctx, &volume_server_pb.VolumeEcShardsGenerateRequest{
VolumeId: volumeID,
Collection: "",
})
if err != nil {
t.Fatalf("VolumeEcShardsGenerate failed: %v", err)
}
_, err = client.VolumeEcShardsMount(ctx, &volume_server_pb.VolumeEcShardsMountRequest{
VolumeId: volumeID,
Collection: "",
ShardIds: []uint32{0, 1, 2, 3, 4, 5, 6, 7, 8, 9},
})
if err != nil {
t.Fatalf("VolumeEcShardsMount failed: %v", err)
}
deleteResp, err := client.BatchDelete(ctx, &volume_server_pb.BatchDeleteRequest{FileIds: []string{fid}})
if err != nil {
t.Fatalf("BatchDelete EC needle should return response, got grpc error: %v", err)
}
if len(deleteResp.GetResults()) != 1 {
t.Fatalf("BatchDelete EC needle expected one result, got %d", len(deleteResp.GetResults()))
}
if deleteResp.GetResults()[0].GetStatus() != http.StatusAccepted {
t.Fatalf("BatchDelete EC needle expected status 202, got %d error=%q", deleteResp.GetResults()[0].GetStatus(), deleteResp.GetResults()[0].GetError())
}
if deleteResp.GetResults()[0].GetSize() == 0 {
t.Fatalf("BatchDelete EC needle expected non-zero deleted size")
}
deletedStream, err := client.VolumeEcShardRead(ctx, &volume_server_pb.VolumeEcShardReadRequest{
VolumeId: volumeID,
ShardId: 0,
FileKey: needleID,
Offset: 0,
Size: 1,
})
if err != nil {
t.Fatalf("VolumeEcShardRead deleted-check start failed: %v", err)
}
deletedMsg, err := deletedStream.Recv()
if err != nil {
t.Fatalf("VolumeEcShardRead deleted-check recv failed: %v", err)
}
if !deletedMsg.GetIsDeleted() {
t.Fatalf("VolumeEcShardRead expected IsDeleted=true after EC batch delete")
}
}
func maybeStartGrpcOffsetProxy(t testing.TB, adminAddr, actualGrpcAddr string) func() {
t.Helper()
host, portText, err := net.SplitHostPort(adminAddr)
if err != nil {
t.Fatalf("split admin address %q: %v", adminAddr, err)
}
port, err := strconv.Atoi(portText)
if err != nil {
t.Fatalf("parse admin port %q: %v", portText, err)
}
expectedGrpcAddr := net.JoinHostPort(host, strconv.Itoa(port+10000))
if expectedGrpcAddr == actualGrpcAddr {
return nil
}
listener, err := net.Listen("tcp", expectedGrpcAddr)
if err != nil {
t.Fatalf("listen grpc-offset proxy %s -> %s: %v", expectedGrpcAddr, actualGrpcAddr, err)
}
stopped := make(chan struct{})
go func() {
for {
conn, acceptErr := listener.Accept()
if acceptErr != nil {
select {
case <-stopped:
return
default:
continue
}
}
go proxyBidirectional(conn, actualGrpcAddr)
}
}()
return func() {
close(stopped)
_ = listener.Close()
}
}
func proxyBidirectional(inbound net.Conn, targetAddr string) {
outbound, err := net.Dial("tcp", targetAddr)
if err != nil {
_ = inbound.Close()
return
}
go func() {
_, _ = io.Copy(outbound, inbound)
_ = outbound.Close()
}()
_, _ = io.Copy(inbound, outbound)
_ = inbound.Close()
}
+64
View File
@@ -2,6 +2,7 @@ package volume_server_grpc_test
import (
"context"
"net/http"
"strings"
"testing"
"time"
@@ -144,3 +145,66 @@ func TestReadNeedleBlobAndMetaInvalidOffsets(t *testing.T) {
t.Fatalf("ReadNeedleMeta should fail for invalid offset")
}
}
func TestReadNeedleMetaUnsupportedForEcMountedVolume(t *testing.T) {
if testing.Short() {
t.Skip("skipping integration test in short mode")
}
clusterHarness := framework.StartSingleVolumeCluster(t, matrix.P1())
conn, grpcClient := framework.DialVolumeServer(t, clusterHarness.VolumeGRPCAddress())
defer conn.Close()
const volumeID = uint32(93)
const needleID = uint64(880002)
const cookie = uint32(0x11AA22BB)
framework.AllocateVolume(t, grpcClient, volumeID, "")
httpClient := framework.NewHTTPClient()
fid := framework.NewFileID(volumeID, needleID, cookie)
uploadResp := framework.UploadBytes(t, httpClient, clusterHarness.VolumeAdminURL(), fid, []byte("ec-meta-unsupported"))
_ = framework.ReadAllAndClose(t, uploadResp)
if uploadResp.StatusCode != http.StatusCreated {
t.Fatalf("upload expected 201, got %d", uploadResp.StatusCode)
}
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
_, err := grpcClient.VolumeEcShardsGenerate(ctx, &volume_server_pb.VolumeEcShardsGenerateRequest{
VolumeId: volumeID,
Collection: "",
})
if err != nil {
t.Fatalf("VolumeEcShardsGenerate failed: %v", err)
}
_, err = grpcClient.VolumeEcShardsMount(ctx, &volume_server_pb.VolumeEcShardsMountRequest{
VolumeId: volumeID,
Collection: "",
ShardIds: []uint32{0, 1, 2, 3, 4, 5, 6, 7, 8, 9},
})
if err != nil {
t.Fatalf("VolumeEcShardsMount failed: %v", err)
}
_, err = grpcClient.VolumeUnmount(ctx, &volume_server_pb.VolumeUnmountRequest{
VolumeId: volumeID,
})
if err != nil {
t.Fatalf("VolumeUnmount failed: %v", err)
}
_, err = grpcClient.ReadNeedleMeta(ctx, &volume_server_pb.ReadNeedleMetaRequest{
VolumeId: volumeID,
NeedleId: needleID,
Offset: 0,
Size: 128,
})
if err == nil {
t.Fatalf("ReadNeedleMeta should fail when only EC shards are mounted")
}
if !strings.Contains(strings.ToLower(err.Error()), "ec shards is not supported") {
t.Fatalf("ReadNeedleMeta EC-only error mismatch: %v", err)
}
}
@@ -426,3 +426,67 @@ func findLastNeedleOffsetAndSize(t testing.TB, idxBytes []byte, needleID uint64)
}
return offset, size
}
func TestReadAllNeedlesStreamsAcrossMultipleVolumes(t *testing.T) {
if testing.Short() {
t.Skip("skipping integration test in short mode")
}
clusterHarness := framework.StartVolumeCluster(t, matrix.P1())
conn, grpcClient := framework.DialVolumeServer(t, clusterHarness.VolumeGRPCAddress())
defer conn.Close()
const volumeIDA = uint32(86)
const volumeIDB = uint32(87)
const needleIDA = uint64(445561)
const needleIDB = uint64(445562)
framework.AllocateVolume(t, grpcClient, volumeIDA, "")
framework.AllocateVolume(t, grpcClient, volumeIDB, "")
client := framework.NewHTTPClient()
fidA := framework.NewFileID(volumeIDA, needleIDA, 0xAA11CC22)
fidB := framework.NewFileID(volumeIDB, needleIDB, 0xBB22DD33)
payloadA := "read-all-multi-volume-a"
payloadB := "read-all-multi-volume-b"
uploadA := framework.UploadBytes(t, client, clusterHarness.VolumeAdminURL(), fidA, []byte(payloadA))
_ = framework.ReadAllAndClose(t, uploadA)
if uploadA.StatusCode != http.StatusCreated {
t.Fatalf("upload A expected 201, got %d", uploadA.StatusCode)
}
uploadB := framework.UploadBytes(t, client, clusterHarness.VolumeAdminURL(), fidB, []byte(payloadB))
_ = framework.ReadAllAndClose(t, uploadB)
if uploadB.StatusCode != http.StatusCreated {
t.Fatalf("upload B expected 201, got %d", uploadB.StatusCode)
}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
stream, err := grpcClient.ReadAllNeedles(ctx, &volume_server_pb.ReadAllNeedlesRequest{
VolumeIds: []uint32{volumeIDA, volumeIDB},
})
if err != nil {
t.Fatalf("ReadAllNeedles start failed: %v", err)
}
seen := map[uint64]string{}
for {
msg, recvErr := stream.Recv()
if recvErr == io.EOF {
break
}
if recvErr != nil {
t.Fatalf("ReadAllNeedles recv failed: %v", recvErr)
}
if msg.GetNeedleId() == needleIDA || msg.GetNeedleId() == needleIDB {
seen[msg.GetNeedleId()] = string(msg.GetNeedleBlob())
}
}
if got := seen[needleIDA]; got != payloadA {
t.Fatalf("ReadAllNeedles missing/mismatched payload for volume A needle: got %q want %q", got, payloadA)
}
if got := seen[needleIDB]; got != payloadB {
t.Fatalf("ReadAllNeedles missing/mismatched payload for volume B needle: got %q want %q", got, payloadB)
}
}
@@ -280,6 +280,54 @@ func TestQueryJsonSuccessAndCsvNoOutput(t *testing.T) {
}
}
func TestQueryCsvInputWithCsvPayloadStillReturnsNoRows(t *testing.T) {
if testing.Short() {
t.Skip("skipping integration test in short mode")
}
clusterHarness := framework.StartSingleVolumeCluster(t, matrix.P1())
conn, grpcClient := framework.DialVolumeServer(t, clusterHarness.VolumeGRPCAddress())
defer conn.Close()
const volumeID = uint32(67)
const needleID = uint64(777004)
const cookie = uint32(0xDCDCADAD)
framework.AllocateVolume(t, grpcClient, volumeID, "")
csvPayload := []byte("name,score\nalice,12\nbob,3\n")
httpClient := framework.NewHTTPClient()
fid := framework.NewFileID(volumeID, needleID, cookie)
uploadResp := framework.UploadBytes(t, httpClient, clusterHarness.VolumeAdminURL(), fid, csvPayload)
_ = framework.ReadAllAndClose(t, uploadResp)
if uploadResp.StatusCode != 201 {
t.Fatalf("upload expected 201, got %d", uploadResp.StatusCode)
}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
csvStream, err := grpcClient.Query(ctx, &volume_server_pb.QueryRequest{
FromFileIds: []string{fid},
Selections: []string{"score"},
Filter: &volume_server_pb.QueryRequest_Filter{
Field: "score",
Operand: ">",
Value: "10",
},
InputSerialization: &volume_server_pb.QueryRequest_InputSerialization{
CsvInput: &volume_server_pb.QueryRequest_InputSerialization_CSVInput{},
},
})
if err != nil {
t.Fatalf("Query csv start failed: %v", err)
}
_, err = csvStream.Recv()
if err != io.EOF {
t.Fatalf("Query csv with csv payload expected EOF with no rows, got: %v", err)
}
}
func TestQueryJsonNoMatchReturnsEmptyStripe(t *testing.T) {
if testing.Short() {
t.Skip("skipping integration test in short mode")
+75
View File
@@ -12,6 +12,8 @@ import (
"github.com/seaweedfs/seaweedfs/test/volume_server/framework"
"github.com/seaweedfs/seaweedfs/test/volume_server/matrix"
"github.com/seaweedfs/seaweedfs/weed/pb/volume_server_pb"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"
)
func TestVolumeTailSenderMissingVolume(t *testing.T) {
@@ -91,6 +93,36 @@ func TestVolumeTailReceiverMissingVolume(t *testing.T) {
}
}
func TestVolumeTailReceiverSourceUnavailable(t *testing.T) {
if testing.Short() {
t.Skip("skipping integration test in short mode")
}
clusterHarness := framework.StartSingleVolumeCluster(t, matrix.P1())
conn, grpcClient := framework.DialVolumeServer(t, clusterHarness.VolumeGRPCAddress())
defer conn.Close()
const volumeID = uint32(89)
framework.AllocateVolume(t, grpcClient, volumeID, "")
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
_, err := grpcClient.VolumeTailReceiver(ctx, &volume_server_pb.VolumeTailReceiverRequest{
VolumeId: volumeID,
SourceVolumeServer: "127.0.0.1:19999.29999",
SinceNs: 0,
IdleTimeoutSeconds: 1,
})
if err == nil {
t.Fatalf("VolumeTailReceiver should fail when source volume server is unavailable")
}
lowered := strings.ToLower(err.Error())
if !strings.Contains(lowered, "dial") && !strings.Contains(lowered, "unavailable") && !strings.Contains(lowered, "connection refused") {
t.Fatalf("VolumeTailReceiver source-unavailable error mismatch: %v", err)
}
}
func TestVolumeTailReceiverReplicatesSourceUpdates(t *testing.T) {
if testing.Short() {
t.Skip("skipping integration test in short mode")
@@ -204,3 +236,46 @@ func TestVolumeTailSenderLargeNeedleChunking(t *testing.T) {
t.Fatalf("VolumeTailSender expected a final data chunk marked IsLastChunk=true")
}
}
func TestVolumeTailSenderStreamCancellation(t *testing.T) {
if testing.Short() {
t.Skip("skipping integration test in short mode")
}
clusterHarness := framework.StartSingleVolumeCluster(t, matrix.P1())
conn, grpcClient := framework.DialVolumeServer(t, clusterHarness.VolumeGRPCAddress())
defer conn.Close()
const volumeID = uint32(74)
framework.AllocateVolume(t, grpcClient, volumeID, "")
ctx, cancel := context.WithCancel(context.Background())
stream, err := grpcClient.VolumeTailSender(ctx, &volume_server_pb.VolumeTailSenderRequest{
VolumeId: volumeID,
SinceNs: 0,
IdleTimeoutSeconds: 30,
})
if err != nil {
cancel()
t.Fatalf("VolumeTailSender start failed: %v", err)
}
firstMsg, err := stream.Recv()
if err != nil {
cancel()
t.Fatalf("VolumeTailSender first recv failed: %v", err)
}
if !firstMsg.GetIsLastChunk() {
cancel()
t.Fatalf("expected heartbeat first message before cancellation")
}
cancel()
_, err = stream.Recv()
if err == nil {
t.Fatalf("VolumeTailSender recv after cancellation expected error")
}
if status.Code(err) != codes.Canceled {
t.Fatalf("VolumeTailSender cancellation expected grpc code Canceled, got %v (%v)", status.Code(err), err)
}
}
+20
View File
@@ -174,6 +174,26 @@ func TestUiIndexNotExposedWhenJwtSigningEnabled(t *testing.T) {
}
}
func TestUiIndexExposedWhenJwtSigningEnabledAndAccessUITrue(t *testing.T) {
if testing.Short() {
t.Skip("skipping integration test in short mode")
}
profile := matrix.P3()
profile.EnableUIAccess = true
cluster := framework.StartSingleVolumeCluster(t, profile)
client := framework.NewHTTPClient()
resp := framework.DoRequest(t, client, mustNewRequest(t, http.MethodGet, cluster.VolumeAdminURL()+"/ui/index.html"))
body := framework.ReadAllAndClose(t, resp)
if resp.StatusCode != http.StatusOK {
t.Fatalf("expected /ui/index.html to be exposed when access.ui=true under JWT profile, got %d body=%s", resp.StatusCode, string(body))
}
if !strings.Contains(strings.ToLower(string(body)), "volume") {
t.Fatalf("ui page does not look like volume status page")
}
}
func mustNewRequest(t testing.TB, method, url string) *http.Request {
t.Helper()
req, err := http.NewRequest(method, url, nil)
@@ -241,3 +241,90 @@ func TestPublicPortHeadReadParity(t *testing.T) {
t.Fatalf("public HEAD body should be empty, got %d bytes", len(headBody))
}
}
func TestUnsupportedMethodConnectParity(t *testing.T) {
if testing.Short() {
t.Skip("skipping integration test in short mode")
}
clusterHarness := framework.StartVolumeCluster(t, matrix.P2())
conn, grpcClient := framework.DialVolumeServer(t, clusterHarness.VolumeGRPCAddress())
defer conn.Close()
const volumeID = uint32(85)
framework.AllocateVolume(t, grpcClient, volumeID, "")
fid := framework.NewFileID(volumeID, 124001, 0x03030303)
client := framework.NewHTTPClient()
uploadResp := framework.UploadBytes(t, client, clusterHarness.VolumeAdminURL(), fid, []byte("connect-method-check"))
_ = framework.ReadAllAndClose(t, uploadResp)
if uploadResp.StatusCode != http.StatusCreated {
t.Fatalf("upload expected 201, got %d", uploadResp.StatusCode)
}
adminReq := mustNewRequest(t, "CONNECT", clusterHarness.VolumeAdminURL()+"/"+fid)
adminResp := framework.DoRequest(t, client, adminReq)
_ = framework.ReadAllAndClose(t, adminResp)
if adminResp.StatusCode != http.StatusBadRequest {
t.Fatalf("admin CONNECT expected 400, got %d", adminResp.StatusCode)
}
publicReq := mustNewRequest(t, "CONNECT", clusterHarness.VolumePublicURL()+"/"+fid)
publicResp := framework.DoRequest(t, client, publicReq)
_ = framework.ReadAllAndClose(t, publicResp)
if publicResp.StatusCode != http.StatusOK {
t.Fatalf("public CONNECT expected passthrough 200, got %d", publicResp.StatusCode)
}
verifyResp := framework.ReadBytes(t, client, clusterHarness.VolumeAdminURL(), fid)
verifyBody := framework.ReadAllAndClose(t, verifyResp)
if verifyResp.StatusCode != http.StatusOK {
t.Fatalf("verify GET expected 200, got %d", verifyResp.StatusCode)
}
if string(verifyBody) != "connect-method-check" {
t.Fatalf("CONNECT should not mutate data, got %q", string(verifyBody))
}
}
func TestUnsupportedMethodMkcolParity(t *testing.T) {
if testing.Short() {
t.Skip("skipping integration test in short mode")
}
clusterHarness := framework.StartVolumeCluster(t, matrix.P2())
conn, grpcClient := framework.DialVolumeServer(t, clusterHarness.VolumeGRPCAddress())
defer conn.Close()
const volumeID = uint32(88)
framework.AllocateVolume(t, grpcClient, volumeID, "")
fid := framework.NewFileID(volumeID, 124004, 0x06060606)
client := framework.NewHTTPClient()
uploadResp := framework.UploadBytes(t, client, clusterHarness.VolumeAdminURL(), fid, []byte("mkcol-method-check"))
_ = framework.ReadAllAndClose(t, uploadResp)
if uploadResp.StatusCode != http.StatusCreated {
t.Fatalf("upload expected 201, got %d", uploadResp.StatusCode)
}
adminReq := mustNewRequest(t, "MKCOL", clusterHarness.VolumeAdminURL()+"/"+fid)
adminResp := framework.DoRequest(t, client, adminReq)
_ = framework.ReadAllAndClose(t, adminResp)
if adminResp.StatusCode != http.StatusBadRequest {
t.Fatalf("admin MKCOL expected 400, got %d", adminResp.StatusCode)
}
publicReq := mustNewRequest(t, "MKCOL", clusterHarness.VolumePublicURL()+"/"+fid)
publicResp := framework.DoRequest(t, client, publicReq)
_ = framework.ReadAllAndClose(t, publicResp)
if publicResp.StatusCode != http.StatusOK {
t.Fatalf("public MKCOL expected passthrough 200, got %d", publicResp.StatusCode)
}
verifyResp := framework.ReadBytes(t, client, clusterHarness.VolumeAdminURL(), fid)
verifyBody := framework.ReadAllAndClose(t, verifyResp)
if verifyResp.StatusCode != http.StatusOK {
t.Fatalf("verify GET expected 200, got %d", verifyResp.StatusCode)
}
if string(verifyBody) != "mkcol-method-check" {
t.Fatalf("MKCOL should not mutate data, got %q", string(verifyBody))
}
}
@@ -2,6 +2,7 @@ package volume_server_http_test
import (
"net/http"
"strconv"
"testing"
"github.com/seaweedfs/seaweedfs/test/volume_server/framework"
@@ -51,4 +52,17 @@ func TestReadDeletedQueryReturnsDeletedNeedleData(t *testing.T) {
if string(readDeletedBody) != string(payload) {
t.Fatalf("readDeleted body mismatch: got %q want %q", string(readDeletedBody), string(payload))
}
headReadDeletedReq := mustNewRequest(t, http.MethodHead, clusterHarness.VolumeAdminURL()+"/"+fid+"?readDeleted=true")
headReadDeletedResp := framework.DoRequest(t, client, headReadDeletedReq)
headReadDeletedBody := framework.ReadAllAndClose(t, headReadDeletedResp)
if headReadDeletedResp.StatusCode != http.StatusOK {
t.Fatalf("HEAD with readDeleted=true expected 200, got %d", headReadDeletedResp.StatusCode)
}
if len(headReadDeletedBody) != 0 {
t.Fatalf("HEAD with readDeleted=true expected empty body, got %d bytes", len(headReadDeletedBody))
}
if got := headReadDeletedResp.Header.Get("Content-Length"); got != strconv.Itoa(len(payload)) {
t.Fatalf("HEAD with readDeleted=true content-length mismatch: got %q want %d", got, len(payload))
}
}
@@ -164,17 +164,21 @@ func TestConditionalHeaderPrecedenceAndInvalidIfModifiedSince(t *testing.T) {
if lastModified == "" {
t.Fatalf("baseline read expected Last-Modified header")
}
etag := baselineResp.Header.Get("ETag")
if etag == "" {
t.Fatalf("baseline read expected ETag header")
}
precedenceReq := mustNewRequest(t, http.MethodGet, clusterHarness.VolumeAdminURL()+"/"+fid)
precedenceReq.Header.Set("If-Modified-Since", lastModified)
precedenceReq.Header.Set("If-None-Match", "\"definitely-different-etag\"")
precedenceResp := framework.DoRequest(t, client, precedenceReq)
precedenceBody := framework.ReadAllAndClose(t, precedenceResp)
if precedenceResp.StatusCode != http.StatusNotModified {
t.Fatalf("conditional precedence expected 304, got %d", precedenceResp.StatusCode)
if precedenceResp.StatusCode != http.StatusOK {
t.Fatalf("mismatched If-None-Match overrides If-Modified-Since: expected 200, got %d", precedenceResp.StatusCode)
}
if len(precedenceBody) != 0 {
t.Fatalf("conditional precedence expected empty body, got %d bytes", len(precedenceBody))
if string(precedenceBody) != string(payload) {
t.Fatalf("mismatched If-None-Match body mismatch: got %q want %q", string(precedenceBody), string(payload))
}
invalidIMSReq := mustNewRequest(t, http.MethodGet, clusterHarness.VolumeAdminURL()+"/"+fid)
@@ -188,4 +192,28 @@ func TestConditionalHeaderPrecedenceAndInvalidIfModifiedSince(t *testing.T) {
if string(invalidIMSBody) != string(payload) {
t.Fatalf("invalid If-Modified-Since fallback body mismatch: got %q want %q", string(invalidIMSBody), string(payload))
}
headIMSPrecedenceReq := mustNewRequest(t, http.MethodHead, clusterHarness.VolumeAdminURL()+"/"+fid)
headIMSPrecedenceReq.Header.Set("If-Modified-Since", lastModified)
headIMSPrecedenceReq.Header.Set("If-None-Match", "\"definitely-different-etag\"")
headIMSPrecedenceResp := framework.DoRequest(t, client, headIMSPrecedenceReq)
headIMSPrecedenceBody := framework.ReadAllAndClose(t, headIMSPrecedenceResp)
if headIMSPrecedenceResp.StatusCode != http.StatusOK {
t.Fatalf("HEAD mismatched If-None-Match overrides If-Modified-Since: expected 200, got %d", headIMSPrecedenceResp.StatusCode)
}
if len(headIMSPrecedenceBody) != 0 {
t.Fatalf("HEAD conditional precedence expected empty body, got %d bytes", len(headIMSPrecedenceBody))
}
headETagPrecedenceReq := mustNewRequest(t, http.MethodHead, clusterHarness.VolumeAdminURL()+"/"+fid)
headETagPrecedenceReq.Header.Set("If-Modified-Since", "Thu, 01 Jan 1970 00:00:00 GMT")
headETagPrecedenceReq.Header.Set("If-None-Match", etag)
headETagPrecedenceResp := framework.DoRequest(t, client, headETagPrecedenceReq)
headETagPrecedenceBody := framework.ReadAllAndClose(t, headETagPrecedenceResp)
if headETagPrecedenceResp.StatusCode != http.StatusNotModified {
t.Fatalf("HEAD conditional precedence expected 304 when If-None-Match matches, got %d", headETagPrecedenceResp.StatusCode)
}
if len(headETagPrecedenceBody) != 0 {
t.Fatalf("HEAD conditional etag precedence expected empty body, got %d bytes", len(headETagPrecedenceBody))
}
}
@@ -1,6 +1,7 @@
package volume_server_http_test
import (
"bytes"
"net/http"
"strings"
"testing"
@@ -72,3 +73,64 @@ func TestWriteMalformedMultipartAndMD5Mismatch(t *testing.T) {
t.Fatalf("content-md5 mismatch response should mention Content-MD5, got %q", string(md5MismatchBody))
}
}
func TestWriteRejectsPayloadOverFileSizeLimit(t *testing.T) {
if testing.Short() {
t.Skip("skipping integration test in short mode")
}
profile := matrix.P1()
profile.FileSizeLimitMB = 1
clusterHarness := framework.StartSingleVolumeCluster(t, profile)
conn, grpcClient := framework.DialVolumeServer(t, clusterHarness.VolumeGRPCAddress())
defer conn.Close()
const volumeID = uint32(99)
framework.AllocateVolume(t, grpcClient, volumeID, "")
client := framework.NewHTTPClient()
fid := framework.NewFileID(volumeID, 772002, 0x2A3B4C5D)
oversizedPayload := bytes.Repeat([]byte("z"), 1024*1024+1)
oversizedReq := newUploadRequest(t, clusterHarness.VolumeAdminURL()+"/"+fid, oversizedPayload)
oversizedResp := framework.DoRequest(t, client, oversizedReq)
oversizedBody := framework.ReadAllAndClose(t, oversizedResp)
if oversizedResp.StatusCode != http.StatusBadRequest {
t.Fatalf("oversized write expected 400, got %d", oversizedResp.StatusCode)
}
if !strings.Contains(strings.ToLower(string(oversizedBody)), "limited") {
t.Fatalf("oversized write response should mention limit, got %q", string(oversizedBody))
}
}
func TestReplicatedWriteFailsWhenReplicaRequirementsNotMet(t *testing.T) {
if testing.Short() {
t.Skip("skipping integration test in short mode")
}
clusterHarness := framework.StartSingleVolumeCluster(t, matrix.P1())
conn, grpcClient := framework.DialVolumeServer(t, clusterHarness.VolumeGRPCAddress())
defer conn.Close()
const volumeID = uint32(109)
framework.AllocateVolumeWithReplication(t, grpcClient, volumeID, "", "001")
client := framework.NewHTTPClient()
fid := framework.NewFileID(volumeID, 772003, 0x3A4B5C6D)
payload := []byte("replicated-write-failure-path")
writeResp := framework.UploadBytes(t, client, clusterHarness.VolumeAdminURL(), fid, payload)
writeBody := framework.ReadAllAndClose(t, writeResp)
if writeResp.StatusCode != http.StatusInternalServerError {
t.Fatalf("replicated write with unmet replication requirements expected 500, got %d body=%s", writeResp.StatusCode, string(writeBody))
}
if !strings.Contains(strings.ToLower(string(writeBody)), "replica") {
t.Fatalf("replicated write failure response should mention replica write failure, got %q", string(writeBody))
}
readResp := framework.ReadBytes(t, client, clusterHarness.VolumeAdminURL(), fid)
_ = framework.ReadAllAndClose(t, readResp)
if readResp.StatusCode != http.StatusNotFound {
t.Fatalf("local read after failed replicate write expected 404 (not committed locally), got %d", readResp.StatusCode)
}
}
@@ -19,6 +19,7 @@ type Profile struct {
ConcurrentDownloadLimitMB int
InflightUploadTimeout time.Duration
InflightDownloadTimeout time.Duration
FileSizeLimitMB int
ReplicatedLayout bool
HasErasureCoding bool
+25 -3
View File
@@ -3,6 +3,7 @@ package command
import (
"context"
"crypto/tls"
"net"
"net/http"
httppprof "net/http/pprof"
"os"
@@ -69,6 +70,7 @@ type VolumeServerOptions struct {
preStopSeconds *int
metricsHttpPort *int
metricsHttpIp *string
proxyProtocol *bool
// pulseSeconds *int
inflightUploadDataTimeout *time.Duration
inflightDownloadDataTimeout *time.Duration
@@ -219,6 +221,7 @@ func init() {
v.id = cmdVolume.Flag.String("id", "", "volume server id. If empty, default to ip:port")
v.publicUrl = cmdVolume.Flag.String("publicUrl", "", "Publicly accessible address")
v.bindIp = cmdVolume.Flag.String("ip.bind", "", "ip address to bind to. If empty, default to same as -ip option.")
v.proxyProtocol = cmdVolume.Flag.Bool("proxyProtocol", false, "accept PROXY protocol v1 headers on volume listeners; enable only behind a trusted proxy")
v.mastersString = cmdVolume.Flag.String("master", "localhost:9333", "comma-separated master servers")
v.mserverString = cmdVolume.Flag.String("mserver", "", "comma-separated master servers (deprecated, use -master instead)")
v.preStopSeconds = cmdVolume.Flag.Int("preStopSeconds", 10, "number of seconds between stop send heartbeats and stop volume server")
@@ -632,12 +635,23 @@ func (m multiHttpdownServer) Stop() error {
return firstErr
}
// acceptProxyProtocol reports whether volume listeners should unwrap PROXY
// v1 headers; the flag pointer is nil for embedded starts (server/mini)
// that do not register it.
func (v VolumeServerOptions) acceptProxyProtocol() bool {
return v.proxyProtocol != nil && *v.proxyProtocol
}
func (v VolumeServerOptions) startGrpcService(vs volume_server_pb.VolumeServerServer) *grpc.Server {
grpcPort := *v.portGrpc
grpcL, grpcLocalL, err := util.NewIpAndLocalListeners(*v.bindIp, grpcPort, 0)
if err != nil {
glog.Fatalf("failed to listen on grpc port %d: %v", grpcPort, err)
}
var grpcListener net.Listener = grpcL
if v.acceptProxyProtocol() {
grpcListener = util.WrapProxyProtocolListener(grpcListener)
}
grpcS := pb.NewGrpcServer(security.LoadServerTLS(util.GetViper(), "grpc.volume"))
volume_server_pb.RegisterVolumeServerServer(grpcS, vs)
reflection.Register(grpcS)
@@ -645,7 +659,7 @@ func (v VolumeServerOptions) startGrpcService(vs volume_server_pb.VolumeServerSe
go grpcS.Serve(grpcLocalL)
}
go func() {
if err := grpcS.Serve(grpcL); err != nil {
if err := grpcS.Serve(grpcListener); err != nil {
glog.Fatalf("start gRPC service failed, %s", err)
}
}()
@@ -660,9 +674,13 @@ func (v VolumeServerOptions) startPublicHttpService(handler http.Handler) httpdo
if e != nil {
glog.Fatalf("Volume server listener error:%v", e)
}
var publicHttpListener net.Listener = publicListener
if v.acceptProxyProtocol() {
publicHttpListener = util.WrapProxyProtocolListener(publicHttpListener)
}
pubHttp := httpdown.HTTP{StopTimeout: 5 * time.Minute, KillTimeout: 5 * time.Minute}
publicHttpDown := multiHttpdownServer{pubHttp.Serve(&http.Server{Handler: handler}, publicListener)}
publicHttpDown := multiHttpdownServer{pubHttp.Serve(&http.Server{Handler: handler}, publicHttpListener)}
if publicLocalListener != nil {
publicHttpDown = append(publicHttpDown, pubHttp.Serve(&http.Server{Handler: handler}, publicLocalListener))
}
@@ -724,7 +742,11 @@ func (v VolumeServerOptions) startClusterHttpService(handler http.Handler) (http
httpS.TLSConfig.GetCertificate = getCert
}
clusterHttpServers := multiHttpdownServer{httpDown.Serve(httpS, listener)}
var clusterHttpListener net.Listener = listener
if v.acceptProxyProtocol() {
clusterHttpListener = util.WrapProxyProtocolListener(clusterHttpListener)
}
clusterHttpServers := multiHttpdownServer{httpDown.Serve(httpS, clusterHttpListener)}
if localListener != nil {
localHttpS := &http.Server{Handler: handler, TLSConfig: httpS.TLSConfig}
clusterHttpServers = append(clusterHttpServers, httpDown.Serve(localHttpS, localListener))
+1 -1
View File
@@ -181,7 +181,7 @@ func (store *EtcdStore) DeleteFolderChildren(ctx context.Context, fullpath weed_
func (store *EtcdStore) ListDirectoryPrefixedEntries(ctx context.Context, dirPath weed_util.FullPath, startFileName string, includeStartFile bool, limit int64, prefix string, eachEntryFunc filer.ListEachEntryFunc) (lastFileName string, err error) {
directoryPrefix := genDirectoryKeyPrefix(dirPath, prefix)
lastFileStart := directoryPrefix
if startFileName != "" {
if startFileName > prefix {
lastFileStart = genDirectoryKeyPrefix(dirPath, startFileName)
}
+29 -36
View File
@@ -400,46 +400,39 @@ func (fsw *FilerStoreWrapper) prefixFilterEntries(ctx context.Context, dirPath u
return actualStore.ListDirectoryEntries(ctx, dirPath, startFileName, includeStartFile, limit, eachEntryFunc)
}
var notPrefixed []*Entry
lastFileName, err = actualStore.ListDirectoryEntries(ctx, dirPath, startFileName, includeStartFile, limit, func(entry *Entry) (bool, error) {
notPrefixed = append(notPrefixed, entry)
return true, nil
})
if err != nil {
return
}
// lastFileName must stay at the last name delivered to eachEntryFunc; the
// raw scan cursor runs ahead over non-matching names and cannot be returned.
count := int64(0)
for count < limit && len(notPrefixed) > 0 {
for _, entry := range notPrefixed {
if strings.HasPrefix(entry.Name(), prefix) {
count++
res, resErr := eachEntryFunc(entry)
if resErr != nil {
err = fmt.Errorf("failed to process eachEntryFunc for entry %q: %w", entry.Name(), resErr)
return
}
if !res {
return
}
if count >= limit {
break
}
}
scanName, scanInclusive := startFileName, includeStartFile
for count < limit {
var page []*Entry
if scanName, err = actualStore.ListDirectoryEntries(ctx, dirPath, scanName, scanInclusive, limit, func(entry *Entry) (bool, error) {
page = append(page, entry)
return true, nil
}); err != nil {
return
}
if count < limit && lastFileName < prefix {
notPrefixed = notPrefixed[:0]
lastFileName, err = actualStore.ListDirectoryEntries(ctx, dirPath, lastFileName, false, limit, func(entry *Entry) (bool, error) {
notPrefixed = append(notPrefixed, entry)
return true, nil
})
if err != nil {
scanInclusive = false
if len(page) == 0 {
return
}
for _, entry := range page {
if !strings.HasPrefix(entry.Name(), prefix) {
continue
}
count++
res, resErr := eachEntryFunc(entry)
if resErr != nil {
err = fmt.Errorf("failed to process eachEntryFunc for entry %q: %w", entry.Name(), resErr)
return
}
} else {
break
lastFileName = entry.Name()
if !res || count >= limit {
return
}
}
if scanName >= prefix && !strings.HasPrefix(scanName, prefix) {
return
}
}
return
+38
View File
@@ -205,3 +205,41 @@ func TestFilerStoreWrapperRollbackSucceedsWithCancelledContext(t *testing.T) {
assert.NoError(t, wrapper.RollbackTransaction(cancelledCtx()))
assert.NoError(t, wrapper.RollbackTransaction(expiredCtx()))
}
type noPrefixListStore struct {
*stubFilerStore
}
func (s *noPrefixListStore) ListDirectoryPrefixedEntries(context.Context, util.FullPath, string, bool, int64, string, ListEachEntryFunc) (string, error) {
return "", ErrUnsupportedListDirectoryPrefixed
}
// The wrapper fallback must return the last delivered name as the cursor, not
// the end of the raw page, and must keep paging while prefix matches can still
// follow the raw cursor.
func TestPrefixFilterEntriesCursorStaysAtLastDelivered(t *testing.T) {
store := &noPrefixListStore{newStubFilerStore()}
wrapper := NewFilerStoreWrapper(store)
ctx := context.Background()
for _, name := range []string{"a", "b1", "b2", "b3", "b4"} {
require.NoError(t, store.InsertEntry(ctx, &Entry{FullPath: util.FullPath("/d/" + name)}))
}
var delivered []string
lastFileName, err := wrapper.ListDirectoryPrefixedEntries(ctx, util.FullPath("/d"), "", false, 2, "b", func(entry *Entry) (bool, error) {
delivered = append(delivered, entry.Name())
return true, nil
})
require.NoError(t, err)
assert.Equal(t, []string{"b1", "b2"}, delivered)
assert.Equal(t, "b2", lastFileName)
delivered = delivered[:0]
lastFileName, err = wrapper.ListDirectoryPrefixedEntries(ctx, util.FullPath("/d"), lastFileName, false, 10, "b", func(entry *Entry) (bool, error) {
delivered = append(delivered, entry.Name())
return true, nil
})
require.NoError(t, err)
assert.Equal(t, []string{"b3", "b4"}, delivered)
assert.Equal(t, "b4", lastFileName)
}
+6 -2
View File
@@ -156,7 +156,11 @@ func (store *HbaseStore) ListDirectoryEntries(ctx context.Context, dirPath util.
func (store *HbaseStore) ListDirectoryPrefixedEntries(ctx context.Context, dirPath util.FullPath, startFileName string, includeStartFile bool, limit int64, prefix string, eachEntryFunc filer.ListEachEntryFunc) (lastFileName string, err error) {
family := map[string][]string{store.cfMetaDir: {COLUMN_NAME}}
expectedPrefix := []byte(dirPath.Child(prefix))
scan, err := hrpc.NewScanRange(ctx, store.table, expectedPrefix, nil, hrpc.Families(family))
scanStart := expectedPrefix
if resumeKey := []byte(dirPath.Child(startFileName)); bytes.Compare(resumeKey, scanStart) > 0 {
scanStart = resumeKey
}
scan, err := hrpc.NewScanRange(ctx, store.table, scanStart, nil, hrpc.Families(family))
if err != nil {
return lastFileName, err
}
@@ -188,7 +192,7 @@ func (store *HbaseStore) ListDirectoryPrefixedEntries(ctx context.Context, dirPa
value := cell.Value
if fileName == startFileName && !includeStartFile {
if fileName < startFileName || (fileName == startFileName && !includeStartFile) {
continue
}
+1 -1
View File
@@ -221,7 +221,7 @@ func (store *LevelDBStore) ListDirectoryPrefixedEntries(ctx context.Context, dir
directoryPrefix := genDirectoryKeyPrefix(dirPath, prefix)
lastFileStart := directoryPrefix
if startFileName != "" {
if startFileName > prefix {
lastFileStart = genDirectoryKeyPrefix(dirPath, startFileName)
}
+25
View File
@@ -110,3 +110,28 @@ func BenchmarkInsertEntry(b *testing.B) {
store.InsertEntry(ctx, entry)
}
}
func TestListDirectoryPrefixedEntries_StartBelowPrefix(t *testing.T) {
store := &LevelDBStore{}
store.initialize(t.TempDir())
defer store.Shutdown()
ctx := context.Background()
for _, name := range []string{"item-a", "item-a.versions", "item-b.versions"} {
if err := store.InsertEntry(ctx, &filer.Entry{FullPath: util.FullPath("/d/" + name)}); err != nil {
t.Fatalf("insert %s: %v", name, err)
}
}
var names []string
_, err := store.ListDirectoryPrefixedEntries(ctx, util.FullPath("/d"), "item", true, 100, "item-b", func(entry *filer.Entry) (bool, error) {
names = append(names, entry.Name())
return true, nil
})
if err != nil {
t.Fatalf("list: %v", err)
}
if len(names) != 1 || names[0] != "item-b.versions" {
t.Fatalf("want [item-b.versions], got %v", names)
}
}
+1 -1
View File
@@ -239,7 +239,7 @@ func (store *LevelDB2Store) ListDirectoryPrefixedEntries(ctx context.Context, di
directoryPrefix, partitionId := genDirectoryKeyPrefix(dirPath, prefix, store.dbCount)
lastFileStart := directoryPrefix
if startFileName != "" {
if startFileName > prefix {
lastFileStart, _ = genDirectoryKeyPrefix(dirPath, startFileName, store.dbCount)
}
@@ -151,6 +151,31 @@ func TestInitializeRejectsOversizedOption(t *testing.T) {
}
}
func TestListDirectoryPrefixedEntries_StartBelowPrefix(t *testing.T) {
store := &LevelDB2Store{}
store.initialize(t.TempDir(), 2)
defer store.Shutdown()
ctx := context.Background()
for _, name := range []string{"item-a", "item-a.versions", "item-b.versions"} {
if err := store.InsertEntry(ctx, &filer.Entry{FullPath: util.FullPath("/d/" + name)}); err != nil {
t.Fatalf("insert %s: %v", name, err)
}
}
var names []string
_, err := store.ListDirectoryPrefixedEntries(ctx, util.FullPath("/d"), "item", true, 100, "item-b", func(entry *filer.Entry) (bool, error) {
names = append(names, entry.Name())
return true, nil
})
if err != nil {
t.Fatalf("list: %v", err)
}
if len(names) != 1 || names[0] != "item-b.versions" {
t.Fatalf("want [item-b.versions], got %v", names)
}
}
func TestEmptyRoot(t *testing.T) {
testFiler := filer.NewFiler(pb.ServerDiscovery{}, nil, "", "", "", "", "", 255, nil)
dir := t.TempDir()
+1 -1
View File
@@ -370,7 +370,7 @@ func (store *LevelDB3Store) ListDirectoryPrefixedEntries(ctx context.Context, di
directoryPrefix := genDirectoryKeyPrefix(shortPath, prefix)
lastFileStart := directoryPrefix
if startFileName != "" {
if startFileName > prefix {
lastFileStart = genDirectoryKeyPrefix(shortPath, startFileName)
}
@@ -182,3 +182,28 @@ func TestEmptyRoot(t *testing.T) {
}
}
func TestListDirectoryPrefixedEntries_StartBelowPrefix(t *testing.T) {
store := &LevelDB3Store{}
store.initialize(t.TempDir())
defer store.Shutdown()
ctx := context.Background()
for _, name := range []string{"item-a", "item-a.versions", "item-b.versions"} {
if err := store.InsertEntry(ctx, &filer.Entry{FullPath: util.FullPath("/d/" + name)}); err != nil {
t.Fatalf("insert %s: %v", name, err)
}
}
var names []string
_, err := store.ListDirectoryPrefixedEntries(ctx, util.FullPath("/d"), "item", true, 100, "item-b", func(entry *filer.Entry) (bool, error) {
names = append(names, entry.Name())
return true, nil
})
if err != nil {
t.Fatalf("list: %v", err)
}
if len(names) != 1 || names[0] != "item-b.versions" {
t.Fatalf("want [item-b.versions], got %v", names)
}
}
+33 -3
View File
@@ -15,6 +15,10 @@ import (
const (
DIR_LIST_MARKER = "\x00"
// a listing whose callback stops early reads at most one batch more than
// it uses, and no single pipeline grows with the listing limit
listFetchBatchSize = 128
)
type UniversalRedis2Store struct {
@@ -96,7 +100,12 @@ func (store *UniversalRedis2Store) UpdateEntry(ctx context.Context, entry *filer
func (store *UniversalRedis2Store) FindEntry(ctx context.Context, fullpath util.FullPath) (entry *filer.Entry, err error) {
data, err := store.Client.Get(ctx, store.getKey(string(fullpath))).Result()
return decodeEntry(fullpath, store.Client.Get(ctx, store.getKey(string(fullpath))))
}
func decodeEntry(fullpath util.FullPath, get *redis.StringCmd) (entry *filer.Entry, err error) {
data, err := get.Result()
if err == redis.Nil {
return nil, filer_pb.ErrNotFound
}
@@ -202,9 +211,13 @@ func (store *UniversalRedis2Store) ListDirectoryEntries(ctx context.Context, dir
// fetch entry meta
var entry *filer.Entry
for _, fileName := range members {
var gets []*redis.StringCmd
for i, fileName := range members {
if i%listFetchBatchSize == 0 {
gets = store.getEntries(ctx, dirPath, members[i:])
}
path := util.NewFullPath(string(dirPath), fileName)
entry, err = store.FindEntry(ctx, path)
entry, err = decodeEntry(path, gets[i%listFetchBatchSize])
lastFileName = fileName
if err != nil {
glog.V(0).InfofCtx(ctx, "list %s : %v", path, err)
@@ -235,6 +248,23 @@ func (store *UniversalRedis2Store) ListDirectoryEntries(ctx context.Context, dir
return lastFileName, err
}
// getEntries pipelines the GETs for up to listFetchBatchSize of members. Each
// command keeps its own error, so a missing value still reads as redis.Nil and
// fails only that member.
func (store *UniversalRedis2Store) getEntries(ctx context.Context, dirPath util.FullPath, members []string) []*redis.StringCmd {
if len(members) > listFetchBatchSize {
members = members[:listFetchBatchSize]
}
gets := make([]*redis.StringCmd, len(members))
store.Client.Pipelined(ctx, func(pipe redis.Pipeliner) error {
for i, fileName := range members {
gets[i] = pipe.Get(ctx, store.getKey(string(util.NewFullPath(string(dirPath), fileName))))
}
return nil
})
return gets
}
func (store *UniversalRedis2Store) removeOrphanedDirectoryListMember(ctx context.Context, dirPath util.FullPath, fileName string) {
// a directory converted to super large after accumulating members still has a legacy index
if store.isSuperLargeDirectory(string(dirPath)) {
@@ -256,6 +256,95 @@ func TestListDirectoryEntriesDeletesLogicallyExpiredEntries(t *testing.T) {
}
}
func TestListDirectoryEntriesSkipsMissingAndExpiredMembersInOrder(t *testing.T) {
for _, keyPrefix := range []string{"", "sw:"} {
t.Run("keyPrefix="+keyPrefix, func(t *testing.T) {
store, dir := newTestStore(t, keyPrefix)
insertTestEntry(t, store, dir.Child("a"), 0)
insertTestEntry(t, store, dir.Child("b"), 0)
insertLogicallyExpiredTestEntry(t, store, dir.Child("c"))
insertTestEntry(t, store, dir.Child("d"), 0)
if err := store.Client.Del(context.Background(), store.getKey(string(dir.Child("b")))).Err(); err != nil {
t.Fatalf("drop value key: %v", err)
}
if names := listNames(t, store, dir); fmt.Sprint(names) != "[a d]" {
t.Fatalf("listed %v, want [a d]", names)
}
if members := indexMembers(t, store, dir); fmt.Sprint(members) != "[a d]" {
t.Fatalf("directory index holds %v, want [a d]", members)
}
})
}
}
func TestListDirectoryEntriesAcrossFetchBatches(t *testing.T) {
store, dir := newTestStore(t, "")
var want []string
for i := 0; i < listFetchBatchSize+2; i++ {
name := fmt.Sprintf("f%04d", i)
insertTestEntry(t, store, dir.Child(name), 0)
if i != listFetchBatchSize {
want = append(want, name)
}
}
// the first member of the second batch
missing := dir.Child(fmt.Sprintf("f%04d", listFetchBatchSize))
if err := store.Client.Del(context.Background(), store.getKey(string(missing))).Err(); err != nil {
t.Fatalf("drop value key: %v", err)
}
var names []string
if _, err := store.ListDirectoryEntries(context.Background(), dir, "", true, 1000, func(entry *filer.Entry) (bool, error) {
names = append(names, entry.Name())
return true, nil
}); err != nil {
t.Fatalf("ListDirectoryEntries %s: %v", dir, err)
}
if fmt.Sprint(names) != fmt.Sprint(want) {
t.Fatalf("listed %v, want %v", names, want)
}
if members := indexMembers(t, store, dir); fmt.Sprint(members) != fmt.Sprint(want) {
t.Fatalf("directory index holds %v, want %v", members, want)
}
}
func TestListDirectoryEntriesStopsAtUnreadableMember(t *testing.T) {
store, dir := newTestStore(t, "")
ctx := context.Background()
insertTestEntry(t, store, dir.Child("a"), 0)
insertTestEntry(t, store, dir.Child("b"), 0)
insertTestEntry(t, store, dir.Child("c"), 0)
// a value of the wrong type makes its GET fail with something other than redis.Nil
key := store.getKey(string(dir.Child("b")))
if err := store.Client.Del(ctx, key).Err(); err != nil {
t.Fatalf("drop value key: %v", err)
}
if err := store.Client.HSet(ctx, key, "f", "v").Err(); err != nil {
t.Fatalf("plant hash: %v", err)
}
var names []string
lastFileName, err := store.ListDirectoryEntries(ctx, dir, "", true, 100, func(entry *filer.Entry) (bool, error) {
names = append(names, entry.Name())
return true, nil
})
if err == nil {
t.Fatal("ListDirectoryEntries returned no error for an unreadable member")
}
if fmt.Sprint(names) != "[a]" || lastFileName != "b" {
t.Fatalf("listed %v up to %q, want [a] up to \"b\"", names, lastFileName)
}
}
func TestDeleteExpiredEntryKeepsRecreatedValue(t *testing.T) {
store, dir := newTestStore(t, "")
+1 -1
View File
@@ -243,7 +243,7 @@ func (store *RocksDBStore) ListDirectoryPrefixedEntries(ctx context.Context, dir
directoryPrefix := genDirectoryKeyPrefix(dirPath, prefix)
lastFileStart := directoryPrefix
if startFileName != "" {
if startFileName > prefix {
lastFileStart = genDirectoryKeyPrefix(dirPath, startFileName)
}
+1 -1
View File
@@ -246,7 +246,7 @@ func (store *TikvStore) ListDirectoryPrefixedEntries(ctx context.Context, dirPat
lastFileName := ""
directoryPrefix := store.genDirectoryKeyPrefix(dirPath, prefix)
lastFileStart := directoryPrefix
if startFileName != "" {
if startFileName > prefix {
lastFileStart = store.genDirectoryKeyPrefix(dirPath, startFileName)
}
+8 -18
View File
@@ -139,7 +139,7 @@ func (s3a *S3ApiServer) PostPolicyBucketHandler(w http.ResponseWriter, r *http.R
// Forward validated POST form fields to the underlying PUT as headers.
applyPostPolicyFormHeaders(r, formValues)
// Authorize the object like the PUT path; the coarse Write check above does not consult the bucket policy.
// Authorize the object like the PUT path.
if errCode := s3a.iam.AuthorizeObjectWrite(r, identity, bucket, object); errCode != s3err.ErrNone {
s3err.WriteErrorResponse(w, r, refineDenialForBucketName(r, identity, errCode))
return
@@ -382,25 +382,15 @@ func getRedirectPostRawQuery(bucket, key, etag string) string {
}
// Check to see if Policy is signed correctly, returning the signing identity.
// The verified form's access key is recorded in ctx's identity holder. The
// identity's Write permission on the bucket is checked only once the signature
// verifies, as on the header-signed path, so a form that fails verification
// learns nothing about the key's permissions. A verified identity that may not
// write is returned with ErrAccessDenied.
// The verified form's access key is recorded in ctx's identity holder.
// Write authorization happens later in the handler via AuthorizeObjectWrite,
// which evaluates the identity's attached IAM policies, session policies, and
// the bucket policy against the real object key; gating here on the legacy
// Actions projection would deny identities those mechanisms allow.
func (iam *IdentityAccessManagement) doesPolicySignatureMatch(ctx context.Context, formValues http.Header) (*Identity, s3err.ErrorCode) {
var identity *Identity
var errCode s3err.ErrorCode
// For SignV2 - Signature field will be valid
if _, ok := formValues["Signature"]; ok {
identity, errCode = iam.doesPolicySignatureV2Match(ctx, formValues)
} else {
identity, errCode = iam.doesPolicySignatureV4Match(ctx, formValues)
return iam.doesPolicySignatureV2Match(ctx, formValues)
}
if errCode != s3err.ErrNone {
return nil, errCode
}
if !identity.CanDo(s3_constants.ACTION_WRITE, formValues.Get("bucket"), "") {
return identity, s3err.ErrAccessDenied
}
return identity, s3err.ErrNone
return iam.doesPolicySignatureV4Match(ctx, formValues)
}
@@ -32,6 +32,11 @@ const (
postAclAccessKey = "AKIAACLTESTTEST"
postAclSecretKey = "secret-key-for-acl-tests"
postAclPrincipal = "arn:aws:iam::000000000000:user/aclwriter"
postPolicyAccessKey = "AKIAPOLICYONLY0000"
postPolicySecretKey = "secret-key-for-policy-only"
postReadAccessKey = "AKIAPOLICYREAD0000"
postReadSecretKey = "secret-key-for-policy-read"
)
// newPostAuthzServer builds an S3ApiServer whose non-admin "tester" identity
@@ -274,3 +279,118 @@ func TestAuthorizeObjectAclWrite(t *testing.T) {
s3a.iam.AuthorizeObjectAclWrite(req, admin, postAuthzBucket, "acl-denied/secret.txt"),
"an admin is not confined by the bucket policy")
}
// newPostPolicyOnlyServer builds an S3ApiServer with identities that hold no
// legacy Write action: "poster" carries an attached IAM policy allowing
// s3:PutObject on the test bucket, and "reader" one allowing only s3:GetObject.
func newPostPolicyOnlyServer(t *testing.T) *S3ApiServer {
t.Helper()
iam := &IdentityAccessManagement{
hashes: make(map[string]*sync.Pool),
hashCounters: make(map[string]*int32),
}
err := iam.loadS3ApiConfiguration(&iam_pb.S3ApiConfiguration{
Identities: []*iam_pb.Identity{{
Name: "poster",
Account: &iam_pb.Account{Id: "000000000000", DisplayName: "poster"},
Credentials: []*iam_pb.Credential{{AccessKey: postPolicyAccessKey, SecretKey: postPolicySecretKey}},
Actions: []string{"List:" + postAuthzBucket},
PolicyNames: []string{"post-write"},
}, {
Name: "reader",
Account: &iam_pb.Account{Id: "000000000000", DisplayName: "reader"},
Credentials: []*iam_pb.Credential{{AccessKey: postReadAccessKey, SecretKey: postReadSecretKey}},
Actions: []string{"Read:" + postAuthzBucket, "List:" + postAuthzBucket},
PolicyNames: []string{"post-read"},
}},
Policies: []*iam_pb.Policy{{
Name: "post-write",
Content: fmt.Sprintf(`{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": ["s3:PutObject", "s3:GetObject", "s3:DeleteObject"],
"Resource": ["arn:aws:s3:::%s/*"]
}, {
"Effect": "Allow",
"Action": ["s3:ListBucket", "s3:GetBucketLocation"],
"Resource": ["arn:aws:s3:::%s"]
}]
}`, postAuthzBucket, postAuthzBucket),
}, {
Name: "post-read",
Content: fmt.Sprintf(`{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": ["s3:GetObject"],
"Resource": ["arn:aws:s3:::%s/*"]
}]
}`, postAuthzBucket),
}},
})
require.NoError(t, err)
s3a := &S3ApiServer{
option: &S3ApiServerOption{BucketsPath: "/buckets"},
iam: iam,
}
s3a.bucketRegistry = NewBucketRegistry(s3a)
s3a.bucketRegistry.setMetadataCache(&BucketMetaData{Name: postAuthzBucket, IsTableBucket: false})
return s3a
}
// postFormValues canonicalizes the signed request's multipart fields the way
// extractPostPolicyFormValues does, for driving doesPolicySignatureMatch.
func postFormValues(t *testing.T, r *http.Request) http.Header {
t.Helper()
require.NoError(t, r.ParseMultipartForm(1<<20))
formValues := make(http.Header)
for k, v := range r.MultipartForm.Value {
formValues[http.CanonicalHeaderKey(k)] = v
}
return formValues
}
// TestPostPolicySignatureAllowsAttachedPolicyIdentity signs a POST Object form
// with a key whose holder has no legacy Write action - only an attached IAM
// policy granting s3:PutObject - and asserts the signature check accepts it
// and the shared object-write authorization allows the key. Gating the
// signature check on the Actions projection used to fail this with
// AccessDenied before authorization ever ran.
func TestPostPolicySignatureAllowsAttachedPolicyIdentity(t *testing.T) {
s3a := newPostPolicyOnlyServer(t)
req := newSignedPostRequestFor(t, "uploads/pic.png", postPolicyAccessKey, postPolicySecretKey, nil)
identity, errCode := s3a.iam.doesPolicySignatureMatch(req.Context(), postFormValues(t, req))
require.Equal(t, s3err.ErrNone, errCode,
"a verified POST signature must not require the legacy Write action")
require.NotNil(t, identity)
assert.Equal(t, s3err.ErrNone,
s3a.iam.AuthorizeObjectWrite(req, identity, postAuthzBucket, "uploads/pic.png"),
"an attached IAM policy granting s3:PutObject must authorize the key like the PUT path")
}
// TestPostPolicyAttachedPolicyDeny covers the deny leg: a verified identity
// whose attached policy does not grant s3:PutObject is still refused by
// AuthorizeObjectWrite and by the full handler.
func TestPostPolicyAttachedPolicyDeny(t *testing.T) {
s3a := newPostPolicyOnlyServer(t)
req := newSignedPostRequestFor(t, "uploads/pic.png", postReadAccessKey, postReadSecretKey, nil)
identity, errCode := s3a.iam.doesPolicySignatureMatch(req.Context(), postFormValues(t, req))
require.Equal(t, s3err.ErrNone, errCode)
require.NotNil(t, identity)
assert.Equal(t, s3err.ErrAccessDenied,
s3a.iam.AuthorizeObjectWrite(req, identity, postAuthzBucket, "uploads/pic.png"),
"an attached policy without s3:PutObject must still deny the upload")
rec := httptest.NewRecorder()
s3a.PostPolicyBucketHandler(rec, newSignedPostRequestFor(t, "uploads/pic.png", postReadAccessKey, postReadSecretKey, nil))
assert.Equal(t, http.StatusForbidden, rec.Code,
"POST from a read-only identity must stay AccessDenied, body: %s", rec.Body.String())
assert.Contains(t, rec.Body.String(), "AccessDenied")
}
@@ -6,6 +6,7 @@ import (
"flag"
"fmt"
"io"
"sync"
"github.com/seaweedfs/seaweedfs/weed/pb"
@@ -33,6 +34,9 @@ func (c *commandVolumeConfigureReplication) Help() string {
This command changes a volume replication value. It should be followed by "volume.fix.replication".
Up to -maxParallelization volume servers are configured at a time, one volume at a time on each,
and replicas of the same volume on different servers take turns.
`
}
@@ -46,6 +50,7 @@ func (c *commandVolumeConfigureReplication) Do(args []string, commandEnv *Comman
volumeIdInt := configureReplicationCommand.Int("volumeId", 0, "the volume id")
replicationString := configureReplicationCommand.String("replication", "", "the intended replication value")
collectionPattern := configureReplicationCommand.String("collectionPattern", "", "comma-separated collection names, with '*' and '?' wildcards; empty matches all")
maxParallelization := configureReplicationCommand.Int("maxParallelization", DefaultMaxParallelization, "run up to X tasks in parallel, whenever possible")
if err = configureReplicationCommand.Parse(args); err != nil {
return nil
}
@@ -75,6 +80,19 @@ func (c *commandVolumeConfigureReplication) Do(args []string, commandEnv *Comman
return err
}
return configureReplicationOnVolumeServers(commandEnv, topologyInfo, volumeFilter, replicaPlacement.String(), *maxParallelization)
}
func configureReplicationOnVolumeServers(commandEnv *CommandEnv, topologyInfo *master_pb.TopologyInfo, volumeFilter func(*master_pb.VolumeInformationMessage) bool, replication string, maxParallelization int) error {
type serverVolumes struct {
addr pb.ServerAddress
volumeIds []uint32
}
var servers []serverVolumes
// VolumeConfigure unmounts and remounts the volume, so replicas take turns
// to keep at least one of them serving.
volumeLocks := make(map[uint32]*sync.Mutex)
// find all data nodes with volumes that needs replication change
eachDataNode(topologyInfo, func(dc DataCenterId, rack RackId, dn *master_pb.DataNodeInfo) {
var targetVolumeIds []uint32
@@ -82,33 +100,45 @@ func (c *commandVolumeConfigureReplication) Do(args []string, commandEnv *Comman
for _, v := range diskInfo.VolumeInfos {
if volumeFilter(v) {
targetVolumeIds = append(targetVolumeIds, v.Id)
if volumeLocks[v.Id] == nil {
volumeLocks[v.Id] = &sync.Mutex{}
}
}
}
}
if len(targetVolumeIds) == 0 {
return
}
err = operation.WithVolumeServerClient(false, pb.NewServerAddressFromDataNode(dn), commandEnv.option.GrpcDialOption, func(volumeServerClient volume_server_pb.VolumeServerClient) error {
for _, targetVolumeId := range targetVolumeIds {
resp, configureErr := volumeServerClient.VolumeConfigure(context.Background(), &volume_server_pb.VolumeConfigureRequest{
VolumeId: targetVolumeId,
Replication: replicaPlacement.String(),
})
if configureErr != nil {
return configureErr
}
if resp.Error != "" {
return errors.New(resp.Error)
}
}
return nil
})
if err != nil {
return
if len(targetVolumeIds) > 0 {
servers = append(servers, serverVolumes{pb.NewServerAddressFromDataNode(dn), targetVolumeIds})
}
})
return err
ewg := NewErrorWaitGroup(maxParallelization)
for _, server := range servers {
ewg.Add(func() error {
err := operation.WithVolumeServerClient(false, server.addr, commandEnv.option.GrpcDialOption, func(volumeServerClient volume_server_pb.VolumeServerClient) error {
for _, targetVolumeId := range server.volumeIds {
volumeLocks[targetVolumeId].Lock()
resp, configureErr := volumeServerClient.VolumeConfigure(context.Background(), &volume_server_pb.VolumeConfigureRequest{
VolumeId: targetVolumeId,
Replication: replication,
})
volumeLocks[targetVolumeId].Unlock()
if configureErr != nil {
return configureErr
}
if resp.Error != "" {
return errors.New(resp.Error)
}
}
return nil
})
if err != nil {
return fmt.Errorf("volume server %s: %w", server.addr, err)
}
return nil
})
}
return ewg.Wait()
}
func getVolumeFilter(replicaPlacement *super_block.ReplicaPlacement, volumeId uint32, collectionPattern string) (func(message *master_pb.VolumeInformationMessage) bool, error) {
@@ -0,0 +1,226 @@
package shell
import (
"context"
"fmt"
"maps"
"net"
"strings"
"sync"
"testing"
"time"
"github.com/seaweedfs/seaweedfs/weed/pb/master_pb"
"github.com/seaweedfs/seaweedfs/weed/pb/volume_server_pb"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
)
type configureReplicationTestVolumeServer struct {
volume_server_pb.UnimplementedVolumeServerServer
failure string
mu sync.Mutex
configured map[uint32]string
}
func (s *configureReplicationTestVolumeServer) VolumeConfigure(_ context.Context, req *volume_server_pb.VolumeConfigureRequest) (*volume_server_pb.VolumeConfigureResponse, error) {
s.mu.Lock()
defer s.mu.Unlock()
if s.configured == nil {
s.configured = make(map[uint32]string)
}
s.configured[req.VolumeId] = req.Replication
return &volume_server_pb.VolumeConfigureResponse{Error: s.failure}, nil
}
// configureReplicationOverlapServer can back several volume servers at once and
// records whether two calls for the same volume were ever in flight together.
type configureReplicationOverlapServer struct {
volume_server_pb.UnimplementedVolumeServerServer
mu sync.Mutex
inFlight map[uint32]int
overlapped bool
}
func (s *configureReplicationOverlapServer) VolumeConfigure(_ context.Context, req *volume_server_pb.VolumeConfigureRequest) (*volume_server_pb.VolumeConfigureResponse, error) {
s.mu.Lock()
s.inFlight[req.VolumeId]++
if s.inFlight[req.VolumeId] > 1 {
s.overlapped = true
}
s.mu.Unlock()
time.Sleep(20 * time.Millisecond)
s.mu.Lock()
s.inFlight[req.VolumeId]--
s.mu.Unlock()
return &volume_server_pb.VolumeConfigureResponse{}, nil
}
func startConfigureReplicationTestVolumeServer(t *testing.T, server volume_server_pb.VolumeServerServer, volumeIds ...uint32) *master_pb.DataNodeInfo {
t.Helper()
lis, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatalf("listen: %v", err)
}
srv := grpc.NewServer()
volume_server_pb.RegisterVolumeServerServer(srv, server)
go srv.Serve(lis)
t.Cleanup(srv.Stop)
port := lis.Addr().(*net.TCPAddr).Port
disk := &master_pb.DiskInfo{}
for _, vid := range volumeIds {
disk.VolumeInfos = append(disk.VolumeInfos, &master_pb.VolumeInformationMessage{Id: vid})
}
return &master_pb.DataNodeInfo{
Id: fmt.Sprintf("127.0.0.1:%d", port),
GrpcPort: uint32(port),
DiskInfos: map[string]*master_pb.DiskInfo{"": disk},
}
}
func configureReplicationTestEnv(dataNodes ...*master_pb.DataNodeInfo) (*CommandEnv, *master_pb.TopologyInfo) {
commandEnv := &CommandEnv{option: &ShellOptions{
GrpcDialOption: grpc.WithTransportCredentials(insecure.NewCredentials()),
}}
topologyInfo := &master_pb.TopologyInfo{DataCenterInfos: []*master_pb.DataCenterInfo{{
RackInfos: []*master_pb.RackInfo{{DataNodeInfos: dataNodes}},
}}}
return commandEnv, topologyInfo
}
func allVolumes(*master_pb.VolumeInformationMessage) bool { return true }
func TestConfigureReplicationOnVolumeServersReportsEveryFailingServer(t *testing.T) {
for _, maxParallelization := range []int{1, DefaultMaxParallelization} {
t.Run(fmt.Sprintf("maxParallelization=%d", maxParallelization), func(t *testing.T) {
first := &configureReplicationTestVolumeServer{failure: "first server failed"}
healthy := &configureReplicationTestVolumeServer{}
last := &configureReplicationTestVolumeServer{failure: "last server failed"}
commandEnv, topologyInfo := configureReplicationTestEnv(
startConfigureReplicationTestVolumeServer(t, first, 1),
startConfigureReplicationTestVolumeServer(t, healthy, 2, 3),
startConfigureReplicationTestVolumeServer(t, last, 4),
)
err := configureReplicationOnVolumeServers(commandEnv, topologyInfo, allVolumes, "001", maxParallelization)
if err == nil {
t.Fatal("expected an error")
}
for _, failure := range []string{first.failure, last.failure} {
if !strings.Contains(err.Error(), failure) {
t.Errorf("error %q does not mention %q", err, failure)
}
}
healthy.mu.Lock()
defer healthy.mu.Unlock()
if expected := map[uint32]string{2: "001", 3: "001"}; !maps.Equal(healthy.configured, expected) {
t.Errorf("healthy server configured %v, expected %v", healthy.configured, expected)
}
})
}
}
func TestConfigureReplicationOnVolumeServersConfiguresReplicasOneAtATime(t *testing.T) {
server := &configureReplicationOverlapServer{inFlight: make(map[uint32]int)}
commandEnv, topologyInfo := configureReplicationTestEnv(
startConfigureReplicationTestVolumeServer(t, server, 1, 2),
startConfigureReplicationTestVolumeServer(t, server, 1, 2),
startConfigureReplicationTestVolumeServer(t, server, 1, 2),
)
if err := configureReplicationOnVolumeServers(commandEnv, topologyInfo, allVolumes, "002", DefaultMaxParallelization); err != nil {
t.Fatal(err)
}
server.mu.Lock()
defer server.mu.Unlock()
if server.overlapped {
t.Error("replicas of one volume were configured at the same time")
}
}
// configureReplicationGateServer blocks inside VolumeConfigure until the test
// releases it, so the test can observe how many servers run at once.
type configureReplicationGateServer struct {
volume_server_pb.UnimplementedVolumeServerServer
gate *configureReplicationGate
}
type configureReplicationGate struct {
entered chan struct{}
release chan struct{}
mu sync.Mutex
active int
maxActive int
calls int
}
func (s *configureReplicationGateServer) VolumeConfigure(_ context.Context, _ *volume_server_pb.VolumeConfigureRequest) (*volume_server_pb.VolumeConfigureResponse, error) {
s.gate.mu.Lock()
s.gate.calls++
s.gate.active++
if s.gate.active > s.gate.maxActive {
s.gate.maxActive = s.gate.active
}
s.gate.mu.Unlock()
s.gate.entered <- struct{}{}
<-s.gate.release
s.gate.mu.Lock()
s.gate.active--
s.gate.mu.Unlock()
return &volume_server_pb.VolumeConfigureResponse{}, nil
}
func TestConfigureReplicationOnVolumeServersRunsServersInParallelUpToTheLimit(t *testing.T) {
const servers = 4
const maxParallelization = 2
gate := &configureReplicationGate{
entered: make(chan struct{}, servers),
release: make(chan struct{}),
}
var dataNodes []*master_pb.DataNodeInfo
for i := 0; i < servers; i++ {
dataNodes = append(dataNodes, startConfigureReplicationTestVolumeServer(t,
&configureReplicationGateServer{gate: gate}, uint32(i+1)))
}
commandEnv, topologyInfo := configureReplicationTestEnv(dataNodes...)
done := make(chan error, 1)
go func() {
done <- configureReplicationOnVolumeServers(commandEnv, topologyInfo, allVolumes, "001", maxParallelization)
}()
for i := 0; i < maxParallelization; i++ {
select {
case <-gate.entered:
case <-time.After(10 * time.Second):
t.Fatalf("only %d servers started; serial execution would hold the second call", i)
}
}
select {
case <-gate.entered:
t.Fatalf("more than %d servers ran while calls were held open", maxParallelization)
case <-time.After(100 * time.Millisecond):
}
close(gate.release)
if err := <-done; err != nil {
t.Fatal(err)
}
gate.mu.Lock()
defer gate.mu.Unlock()
if gate.maxActive != maxParallelization {
t.Errorf("at most %d servers ran together, expected %d", gate.maxActive, maxParallelization)
}
if gate.calls != servers {
t.Errorf("%d servers ran, expected %d", gate.calls, servers)
}
}
+128 -95
View File
@@ -9,6 +9,7 @@ import (
"net/url"
"strconv"
"strings"
"sync"
"time"
"github.com/seaweedfs/seaweedfs/weed/glog"
@@ -46,8 +47,8 @@ func ReplicatedWrite(ctx context.Context, masterFn operation.GetMasterFn, grpcDi
}
// with no local volume and no replica to forward to, the write would be acknowledged without being stored
hasLocalVolume := s.HasVolume(volumeId)
if !hasLocalVolume && len(remoteLocations) == 0 {
vol := s.GetVolume(volumeId)
if vol == nil && len(remoteLocations) == 0 {
err = fmt.Errorf("volume %d not found on %s:%d", volumeId, s.Ip, s.Port)
glog.V(0).Infoln(err)
return
@@ -68,109 +69,141 @@ func ReplicatedWrite(ctx context.Context, masterFn operation.GetMasterFn, grpcDi
}(time.Now())
}
if hasLocalVolume {
start := time.Now()
inFlightGauge := stats.VolumeServerInFlightRequestsGauge.WithLabelValues(stats.WriteToLocalDisk)
inFlightGauge.Inc()
defer inFlightGauge.Dec()
isUnchanged, err = s.WriteVolumeNeedle(volumeId, n, true, fsync)
stats.VolumeServerRequestHistogram.WithLabelValues(stats.WriteToLocalDisk).Observe(time.Since(start).Seconds())
if err != nil {
stats.VolumeServerHandlerCounter.WithLabelValues(stats.ErrorWriteToLocalDisk).Inc()
err = fmt.Errorf("failed to write to local disk: %w", err)
glog.V(0).Infoln(err)
return
// Snapshot the fields the fan-out reads before the concurrent local write
// mutates the needle (it stamps the volume TTL and flags), so the two cannot
// race; the fan-out then touches only these locals plus read-only n.Data.
replicaTtl := n.Ttl
if vol != nil && replicaTtl == needle.EMPTY_TTL && vol.Ttl != needle.EMPTY_TTL {
replicaTtl = vol.Ttl
}
replicaLastModified := n.LastModified
replicaIsChunked := n.IsChunkedManifest()
replicaIsCompressed := n.IsCompressed()
replicaName := string(n.Name)
replicaMime := string(n.Mime)
replicaData := n.Data
replicaPairMap := make(map[string]string)
if replicaCount > 0 && n.HasPairs() {
tmpMap := make(map[string]string)
if pe := json.Unmarshal(n.Pairs, &tmpMap); pe != nil {
stats.VolumeServerHandlerCounter.WithLabelValues(stats.ErrorUnmarshalPairs).Inc()
glog.V(0).Infoln("Unmarshal pairs error:", pe)
}
for k, v := range tmpMap {
replicaPairMap[needle.PairNamePrefix+k] = v
}
}
// Observe replication targets histogram for all operations (including zero)
stats.VolumeServerReplicationTargets.Observe(float64(replicaCount))
if replicaCount > 0 { //send to other replica locations
start := time.Now()
// Run the local disk write and the replica fan-out concurrently against the
// buffered needle (latency max(local, replicas), not the sum); ack after both.
var wg sync.WaitGroup
var localErr, remoteErr error
var localIsUnchanged bool
inFlightGauge := stats.VolumeServerInFlightRequestsGauge.WithLabelValues(stats.WriteToReplicas)
inFlightGauge.Inc()
defer inFlightGauge.Dec()
if vol != nil {
wg.Add(1)
go func() {
defer wg.Done()
start := time.Now()
inFlightGauge := stats.VolumeServerInFlightRequestsGauge.WithLabelValues(stats.WriteToLocalDisk)
inFlightGauge.Inc()
defer inFlightGauge.Dec()
err = DistributedOperation(ctx, remoteLocations, func(ctx context.Context, location operation.Location) error {
u := url.URL{
Scheme: "http",
Host: location.Url,
Path: r.URL.Path,
localIsUnchanged, localErr = s.WriteVolumeNeedle(volumeId, n, true, fsync)
stats.VolumeServerRequestHistogram.WithLabelValues(stats.WriteToLocalDisk).Observe(time.Since(start).Seconds())
if localErr != nil {
stats.VolumeServerHandlerCounter.WithLabelValues(stats.ErrorWriteToLocalDisk).Inc()
localErr = fmt.Errorf("failed to write to local disk: %w", localErr)
glog.V(0).Infoln(localErr)
}
q := url.Values{
"type": {"replicate"},
"ttl": {n.Ttl.String()},
}
if fsync {
q.Set("fsync", "true")
}
if n.LastModified > 0 {
q.Set("ts", strconv.FormatUint(n.LastModified, 10))
}
if n.IsChunkedManifest() {
q.Set("cm", "true")
}
u.RawQuery = q.Encode()
pairMap := make(map[string]string)
if n.HasPairs() {
tmpMap := make(map[string]string)
err := json.Unmarshal(n.Pairs, &tmpMap)
if err != nil {
stats.VolumeServerHandlerCounter.WithLabelValues(stats.ErrorUnmarshalPairs).Inc()
glog.V(0).Infoln("Unmarshal pairs error:", err)
}
for k, v := range tmpMap {
pairMap[needle.PairNamePrefix+k] = v
}
}
bytesBuffer := buffer_pool.SyncPoolGetBuffer()
defer buffer_pool.SyncPoolPutBuffer(bytesBuffer)
// volume server do not know about encryption
// TODO optimize here to compress data only once
uploadOption := &operation.UploadOption{
UploadUrl: u.String(),
Filename: string(n.Name),
Cipher: false,
IsInputCompressed: n.IsCompressed(),
IsReplication: true,
MimeType: string(n.Mime),
PairMap: pairMap,
Jwt: jwt,
Md5: contentMd5,
BytesBuffer: bytesBuffer,
MaxAttempts: 1, // fail fast on a dead replica; the client write retries
}
uploader, err := operation.NewUploader()
if err != nil {
glog.Errorf("replication-UploadData, err:%v, url:%s", err, u.String())
return err
}
_, err = uploader.UploadData(ctx, n.Data, uploadOption)
if err != nil {
glog.Errorf("replication-UploadData, err:%v, url:%s", err, u.String())
}
return err
})
stats.VolumeServerRequestHistogram.WithLabelValues(stats.WriteToReplicas).Observe(time.Since(start).Seconds())
if err != nil {
stats.VolumeServerHandlerCounter.WithLabelValues(stats.ErrorWriteToReplicas).Inc()
stats.VolumeServerReplicationCounter.WithLabelValues(stats.ReplicationOpWrite, stats.ReplicationFailure).Inc()
reason := classifyReplicationError(err)
stats.VolumeServerReplicationFailures.WithLabelValues(stats.ReplicationOpWrite, reason).Inc()
err = fmt.Errorf("failed to write to replicas for volume %d: %v", volumeId, err)
glog.V(0).Infoln(err)
return false, err
}
stats.VolumeServerReplicationCounter.WithLabelValues(stats.ReplicationOpWrite, stats.ReplicationSuccess).Inc()
}()
}
return
if replicaCount > 0 {
wg.Add(1)
go func() {
defer wg.Done()
start := time.Now()
inFlightGauge := stats.VolumeServerInFlightRequestsGauge.WithLabelValues(stats.WriteToReplicas)
inFlightGauge.Inc()
defer inFlightGauge.Dec()
remoteErr = DistributedOperation(ctx, remoteLocations, func(ctx context.Context, location operation.Location) error {
u := url.URL{
Scheme: "http",
Host: location.Url,
Path: r.URL.Path,
}
q := url.Values{
"type": {"replicate"},
"ttl": {replicaTtl.String()},
}
if fsync {
q.Set("fsync", "true")
}
if replicaLastModified > 0 {
q.Set("ts", strconv.FormatUint(replicaLastModified, 10))
}
if replicaIsChunked {
q.Set("cm", "true")
}
u.RawQuery = q.Encode()
bytesBuffer := buffer_pool.SyncPoolGetBuffer()
defer buffer_pool.SyncPoolPutBuffer(bytesBuffer)
// volume server do not know about encryption
uploadOption := &operation.UploadOption{
UploadUrl: u.String(),
Filename: replicaName,
Cipher: false,
IsInputCompressed: replicaIsCompressed,
IsReplication: true,
MimeType: replicaMime,
PairMap: replicaPairMap,
Jwt: jwt,
Md5: contentMd5,
BytesBuffer: bytesBuffer,
MaxAttempts: 1, // fail fast on a dead replica; the client write retries
}
uploader, uploaderErr := operation.NewUploader()
if uploaderErr != nil {
glog.Errorf("replication-UploadData, err:%v, url:%s", uploaderErr, u.String())
return uploaderErr
}
_, uploadErr := uploader.UploadData(ctx, replicaData, uploadOption)
if uploadErr != nil {
glog.Errorf("replication-UploadData, err:%v, url:%s", uploadErr, u.String())
}
return uploadErr
})
stats.VolumeServerRequestHistogram.WithLabelValues(stats.WriteToReplicas).Observe(time.Since(start).Seconds())
if remoteErr != nil {
stats.VolumeServerHandlerCounter.WithLabelValues(stats.ErrorWriteToReplicas).Inc()
stats.VolumeServerReplicationCounter.WithLabelValues(stats.ReplicationOpWrite, stats.ReplicationFailure).Inc()
reason := classifyReplicationError(remoteErr)
stats.VolumeServerReplicationFailures.WithLabelValues(stats.ReplicationOpWrite, reason).Inc()
remoteErr = fmt.Errorf("failed to write to replicas for volume %d: %v", volumeId, remoteErr)
glog.V(0).Infoln(remoteErr)
} else {
stats.VolumeServerReplicationCounter.WithLabelValues(stats.ReplicationOpWrite, stats.ReplicationSuccess).Inc()
}
}()
}
wg.Wait()
if localErr != nil {
return false, localErr
}
if remoteErr != nil {
return false, remoteErr
}
return localIsUnchanged, nil
}
// ReplicatedDelete deletes a needle from the local volume and sends delete
+173
View File
@@ -16,8 +16,11 @@ import (
"github.com/seaweedfs/seaweedfs/weed/operation"
"github.com/seaweedfs/seaweedfs/weed/pb"
"github.com/seaweedfs/seaweedfs/weed/pb/master_pb"
"github.com/seaweedfs/seaweedfs/weed/stats"
"github.com/seaweedfs/seaweedfs/weed/storage"
"github.com/seaweedfs/seaweedfs/weed/storage/needle"
"github.com/seaweedfs/seaweedfs/weed/storage/types"
"github.com/seaweedfs/seaweedfs/weed/util"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
)
@@ -230,6 +233,176 @@ func TestReplicatedWriteForwardsFsyncToReplicas(t *testing.T) {
}
}
// newReplicateTestStore builds a single-disk store in dir, draining the notify
// channels so volume mounts never block on a heartbeat reader.
func newReplicateTestStore(t *testing.T, dir string) *storage.Store {
t.Helper()
store := storage.NewStore(nil, "127.0.0.1", 8080, 18080, "http://127.0.0.1:8080", "store-id",
[]string{dir}, []int32{100}, []util.MinFreeSpace{{}}, "",
storage.NeedleMapInMemory, []types.DiskType{types.HardDriveType}, nil, 3,
stats.DefaultDiskIOProbeConfig())
done := make(chan struct{})
go func() {
for {
select {
case <-store.NewVolumesChan:
case <-store.DeletedVolumesChan:
case <-store.NewEcShardsChan:
case <-store.DeletedEcShardsChan:
case <-store.StateUpdateChan:
case <-done:
return
}
}
}()
t.Cleanup(func() {
close(done)
store.Close()
})
return store
}
// TestReplicatedWriteOverlapsLocalAndReplicaWrites runs the write path with a
// real local volume and a replica that stalls mid-request: the fan-out starts
// while the local write is still free to run, the local write lands before the
// replica unblocks, and the acknowledged write waits for both legs.
func TestReplicatedWriteOverlapsLocalAndReplicaWrites(t *testing.T) {
replicaStarted := make(chan url.Values, 1)
releaseReplica := make(chan struct{})
var releaseOnce sync.Once
release := func() { releaseOnce.Do(func() { close(releaseReplica) }) }
replica := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
replicaStarted <- r.URL.Query()
<-releaseReplica
w.WriteHeader(http.StatusCreated)
if _, err := w.Write([]byte(`{"size":21}`)); err != nil {
t.Errorf("replica write response: %v", err)
}
}))
defer replica.Close()
defer release()
replicaHost := strings.TrimPrefix(replica.URL, "http://")
masterFn, dialOption := startMockMasterServer(t, &mockMasterServer{
locations: []*master_pb.Location{{Url: "127.0.0.1:8080"}, {Url: replicaHost}},
})
store := newReplicateTestStore(t, t.TempDir())
volumeId := needle.VolumeId(7)
operation.InvalidateVolumeIdLocationCache(volumeId.String())
if err := store.AddVolume(volumeId, "", storage.NeedleMapInMemory, "001", "15d", 0, needle.GetCurrentVersion(), 0, types.HardDriveType, 0); err != nil {
t.Fatal(err)
}
data := []byte("overlapped write data")
n := &needle.Needle{
Id: 1,
Data: data,
Checksum: needle.NewCRC(data),
Ttl: needle.EMPTY_TTL,
}
r := httptest.NewRequest(http.MethodPost, "http://127.0.0.1:8080/7,01637037d6", nil)
if err := r.ParseForm(); err != nil {
t.Fatal(err)
}
done := make(chan error, 1)
go func() {
_, err := ReplicatedWrite(context.Background(), masterFn, dialOption, store, volumeId, n, r, "")
done <- err
}()
var q url.Values
select {
case q = <-replicaStarted:
case <-time.After(5 * time.Second):
t.Fatal("replica never received the fan-out request")
}
if got := q.Get("type"); got != "replicate" {
t.Errorf("replica type = %q, want replicate", got)
}
if got := q.Get("ttl"); got != "15d" {
t.Errorf("replica ttl = %q, want the inherited volume ttl 15d", got)
}
select {
case err := <-done:
t.Fatalf("write returned before the replica responded: %v", err)
case <-time.After(50 * time.Millisecond):
}
deadline := time.Now().Add(5 * time.Second)
for {
readBack := &needle.Needle{Id: n.Id}
if _, err := store.ReadVolumeNeedle(volumeId, readBack, nil, nil); err == nil && string(readBack.Data) == string(data) {
break
}
if time.Now().After(deadline) {
t.Fatal("local write did not land while the replica was still blocked")
}
time.Sleep(5 * time.Millisecond)
}
release()
if err := <-done; err != nil {
t.Fatalf("ReplicatedWrite: %v", err)
}
}
// TestReplicatedWriteFailsWhenEitherLegFails checks that a failed local write
// or a failed replica fan-out fails the whole request even when the other leg
// succeeded.
func TestReplicatedWriteFailsWhenEitherLegFails(t *testing.T) {
for _, tc := range []struct {
name string
volumeId needle.VolumeId
readonlyLocal bool
replicaStatus int
}{
{name: "local write fails", volumeId: 8, readonlyLocal: true, replicaStatus: http.StatusCreated},
{name: "replica write fails", volumeId: 9, replicaStatus: http.StatusInternalServerError},
} {
t.Run(tc.name, func(t *testing.T) {
replica := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(tc.replicaStatus)
}))
defer replica.Close()
replicaHost := strings.TrimPrefix(replica.URL, "http://")
masterFn, dialOption := startMockMasterServer(t, &mockMasterServer{
locations: []*master_pb.Location{{Url: "127.0.0.1:8080"}, {Url: replicaHost}},
})
store := newReplicateTestStore(t, t.TempDir())
operation.InvalidateVolumeIdLocationCache(tc.volumeId.String())
if err := store.AddVolume(tc.volumeId, "", storage.NeedleMapInMemory, "001", "", 0, needle.GetCurrentVersion(), 0, types.HardDriveType, 0); err != nil {
t.Fatal(err)
}
if tc.readonlyLocal {
if err := store.MarkVolumeReadonly(tc.volumeId, false, false); err != nil {
t.Fatal(err)
}
}
data := []byte("write data")
n := &needle.Needle{
Id: 1,
Data: data,
Checksum: needle.NewCRC(data),
Ttl: needle.EMPTY_TTL,
}
r := httptest.NewRequest(http.MethodPost, fmt.Sprintf("http://127.0.0.1:8080/%d,01637037d6", tc.volumeId), nil)
if err := r.ParseForm(); err != nil {
t.Fatal(err)
}
if _, err := ReplicatedWrite(context.Background(), masterFn, dialOption, store, tc.volumeId, n, r, ""); err == nil {
t.Fatal("ReplicatedWrite succeeded despite a failed leg")
}
})
}
}
// TestReplicatedWriteRejectsWriteWithNoTarget verifies that a write this server
// cannot store and has no replica to forward to fails instead of being
// acknowledged with nothing written.
+1 -1
View File
@@ -165,7 +165,7 @@ func NewIpAndLocalListeners(host string, port int, timeout time.Duration) (ipLis
Timeout: timeout,
}
if host != "localhost" && host != "" && host != "0.0.0.0" && host != "127.0.0.1" && host != "[::]" && host != "[::1]" {
if host != "localhost" && host != "" && host != "0.0.0.0" && host != "127.0.0.1" && host != "[::]" && host != "[::1]" && !isLoopbackHost(host) {
listener, err = net.Listen("tcp", JoinHostPort("localhost", port))
if err != nil {
glog.V(0).Infof("skip starting on %s:%d: %v", host, port, err)
+4 -3
View File
@@ -20,8 +20,9 @@ var (
)
// SetOutboundLocalIP records ip as the source address for outbound TCP
// connections. An empty, wildcard (0.0.0.0 / ::), or unparseable value leaves
// the source address to the OS.
// connections. An empty, wildcard (0.0.0.0 / ::), loopback, or unparseable
// value leaves the source address to the OS. A loopback source can only
// originate connections to loopback targets, which already skip binding.
//
// The first call wins. `weed server` applies the top-level -ip.bind before its
// embedded master/volume/filer/s3/... start, so a component's own (possibly
@@ -32,7 +33,7 @@ func SetOutboundLocalIP(ip string) {
return
}
parsed := net.ParseIP(ip)
if parsed == nil || parsed.IsUnspecified() {
if parsed == nil || parsed.IsUnspecified() || parsed.IsLoopback() {
return
}
outboundLocalAddr.Store(&net.TCPAddr{IP: parsed})
+2
View File
@@ -24,6 +24,8 @@ func TestSetOutboundLocalIP(t *testing.T) {
{"empty", "", ""},
{"wildcard v4", "0.0.0.0", ""},
{"wildcard v6", "::", ""},
{"loopback v4", "127.0.0.2", ""},
{"loopback v6", "::1", ""},
{"garbage", "not-an-ip", ""},
}
for _, tc := range cases {
+102
View File
@@ -0,0 +1,102 @@
package util
import (
"bufio"
"io"
"net"
"strconv"
"strings"
"sync"
)
// WrapProxyProtocolListener returns a listener whose accepted connections
// read and strip a HAProxy PROXY protocol v1 header line, exposing the
// advertised source address as the connection's RemoteAddr. Connections
// that do not start with a PROXY header keep their real remote address,
// so direct clients continue to work (and fail any address-based checks
// against their actual address).
//
// The header is parsed lazily on the first Read or RemoteAddr call, inside
// the connection's own read path; a silent or malformed peer occupies only
// its own connection, never the accept loop.
//
// Only enable this on listeners reachable solely through a trusted proxy:
// any direct client could otherwise spoof its reported address.
func WrapProxyProtocolListener(listener net.Listener) net.Listener {
return &proxyProtocolListener{Listener: listener}
}
type proxyProtocolListener struct {
net.Listener
}
func (l *proxyProtocolListener) Accept() (net.Conn, error) {
conn, err := l.Listener.Accept()
if err != nil {
return nil, err
}
return &proxyProtocolConn{Conn: conn}, nil
}
type proxyProtocolConn struct {
net.Conn
parseOnce sync.Once
reader *bufio.Reader
remote net.Addr
headerErr error
}
func (c *proxyProtocolConn) Read(b []byte) (int, error) {
c.parse()
if c.headerErr != nil {
return 0, c.headerErr
}
return c.reader.Read(b)
}
func (c *proxyProtocolConn) RemoteAddr() net.Addr {
c.parse()
if c.remote != nil {
return c.remote
}
return c.Conn.RemoteAddr()
}
func (c *proxyProtocolConn) parse() {
c.parseOnce.Do(func() {
c.reader = bufio.NewReader(c.Conn)
head, err := c.reader.Peek(len("PROXY "))
if err != nil {
if err != io.EOF {
c.headerErr = err
}
return
}
if string(head) != "PROXY " {
return
}
line, err := c.reader.ReadSlice('\n')
if err != nil {
// A truncated or overlong header means the peer never sent a
// parseable line; keep the real address and let the buffered
// bytes fail the downstream protocol on their own.
c.discardHeaderLine()
return
}
fields := strings.Fields(strings.TrimRight(string(line), "\r\n"))
if len(fields) == 6 && fields[0] == "PROXY" && (fields[1] == "TCP4" || fields[1] == "TCP6") {
if srcIP := net.ParseIP(fields[2]); srcIP != nil {
if srcPort, err := strconv.Atoi(fields[4]); err == nil {
c.remote = &net.TCPAddr{IP: srcIP, Port: srcPort}
}
}
}
})
}
// discardHeaderLine drops the remainder of a malformed PROXY line so the
// following bytes still reach the wrapped connection.
func (c *proxyProtocolConn) discardHeaderLine() {
discard := make([]byte, c.reader.Buffered())
_, _ = io.ReadFull(c.reader, discard)
}
+106
View File
@@ -0,0 +1,106 @@
package util
import (
"bufio"
"io"
"net"
"testing"
"time"
)
func acceptOne(t *testing.T, ln net.Listener) net.Conn {
t.Helper()
accept := make(chan net.Conn, 1)
go func() {
conn, err := ln.Accept()
if err == nil {
accept <- conn
}
}()
select {
case conn := <-accept:
return conn
case <-time.After(2 * time.Second):
t.Fatal("Accept blocked waiting for the peer to send data")
return nil
}
}
func newPipePair(t *testing.T) (client, server net.Conn) {
t.Helper()
serverConn, client := net.Pipe()
ln := WrapProxyProtocolListener(&oneShotListener{conn: serverConn})
return client, acceptOne(t, ln)
}
type oneShotListener struct {
conn net.Conn
done bool
}
func (l *oneShotListener) Accept() (net.Conn, error) {
if l.done {
return nil, io.EOF
}
l.done = true
return l.conn, nil
}
func (l *oneShotListener) Close() error { return l.conn.Close() }
func (l *oneShotListener) Addr() net.Addr { return l.conn.LocalAddr() }
func TestProxyProtocolHeaderRewritesRemoteAddr(t *testing.T) {
client, server := newPipePair(t)
defer client.Close()
defer server.Close()
go func() {
_, _ = client.Write([]byte("PROXY TCP4 10.99.0.1 192.0.2.5 45678 8080\r\nGET / HTTP/1.0\r\n\r\n"))
}()
got, err := bufio.NewReader(server).ReadString('\n')
if err != nil {
t.Fatalf("read request line: %v", err)
}
if got != "GET / HTTP/1.0\r\n" {
t.Fatalf("request line = %q, want GET / HTTP/1.0", got)
}
if addr, ok := server.RemoteAddr().(*net.TCPAddr); !ok || addr.IP.String() != "10.99.0.1" || addr.Port != 45678 {
t.Fatalf("RemoteAddr = %v, want 10.99.0.1:45678", server.RemoteAddr())
}
}
func TestProxyProtocolAbsentKeepsRemoteAddr(t *testing.T) {
client, server := newPipePair(t)
defer client.Close()
defer server.Close()
go func() {
_, _ = client.Write([]byte("GET / HTTP/1.0\r\n\r\n"))
}()
got, err := bufio.NewReader(server).ReadString('\n')
if err != nil {
t.Fatalf("read request line: %v", err)
}
if got != "GET / HTTP/1.0\r\n" {
t.Fatalf("request line = %q, want GET / HTTP/1.0", got)
}
if server.RemoteAddr().String() != "pipe" {
t.Fatalf("RemoteAddr = %v, want underlying conn address", server.RemoteAddr())
}
}
func TestProxyProtocolTruncatedHeaderKeepsRemoteAddr(t *testing.T) {
client, server := newPipePair(t)
defer server.Close()
go func() {
_, _ = client.Write([]byte("PROXY TCP4 10.99.0.1"))
_ = client.Close()
}()
_, _ = server.Read(make([]byte, 64))
if server.RemoteAddr().String() != "pipe" {
t.Fatalf("RemoteAddr = %v, want underlying conn address", server.RemoteAddr())
}
}
+1 -1
View File
@@ -9,7 +9,7 @@ import (
var (
MAJOR_VERSION = int32(4)
MINOR_VERSION = int32(49)
MINOR_VERSION = int32(50)
VERSION_NUMBER = fmt.Sprintf("%d.%02d", MAJOR_VERSION, MINOR_VERSION)
VERSION = util.SizeLimit + " " + VERSION_NUMBER
COMMIT = ""