security.GetJwt reads the "jwt" query parameter before the Authorization
header, and the proxy forwarded the caller's whole query apart from
proxyChunkId. So on a read, where the filer mints a volume token and sets the
header itself, a caller-supplied ?jwt= silently outranked it: the volume
server validated a credential the caller chose rather than the one the filer
attached, and the read failed with a 401 the filer could not explain.
Drop it on the read path, where the filer owns the credential. Writes keep
theirs -- the proxy forwards a writer's own AssignVolume token either way, so
the query parameter is just a second channel for the same credential and
stripping it would break a caller that presents it that way.
Nothing in the tree passes a jwt by query; maybeAddAuth always sets the
header.
* filer: claim the base fid when minting a volume read token
GenJwtForVolumeServer stamped the fid verbatim, but the volume server strips a
trailing _N delta suffix before comparing the claim, so a token minted for a
batch-assigned fid like 3,01637037d6_1 was checked against 3,01637037d6 and
never matched. Reading such a chunk through the filer returned 401 wherever
jwt.signing.read.key was configured.
Strip the suffix before minting, via a helper shared with the proxyChunkId
validation that was already doing the same thing inline.
* filer: don't mint a volume write token for an anonymous proxy caller
The ?proxyChunkId= branch dispatches and returns before the JWT gate, so
whatever credential the proxy attaches is reachable without authentication.
It attached a token from maybeGetVolumeReadJwtAuthorizationToken, which fell
back to the write signing key when jwt.signing.read.key was unset -- the
configuration scaffold/security.toml recommends for a filer, since read JWTs
are only supported in a master+volume setup. An anonymous
DELETE /?proxyChunkId=<fid> therefore arrived at the volume server holding a
write-key token scoped to that fid, and the volume server honored it.
Sign read tokens with the read key only. The fallback bought nothing on a
read anyway: a volume server enforces read JWTs solely when that same key is
set, so when the fallback fired the read was unchecked regardless.
Mint only for reads. Writers proxied through the filer carry their own volume
JWT from AssignVolume, forwarded with the rest of the caller's headers, so
weed mount -filerProxy uploads are unaffected. Moving the dispatch below the
JWT gate instead would have broken them, since that token is signed with
jwt.signing rather than jwt.filer_signing.
On a read with nothing to mint, drop the caller's Authorization rather than
relaying it: there it is a filer credential, and forwarding it would hand a
volume server a token it never used to see.
* filer: keep proxied writes out of the read concurrency semaphore
The semaphore is named and documented for reads -- it exists so replication
bursts can't open hundreds of connections to one volume server -- but it was
applied to every proxied method. A write queued behind sixteen in-flight reads
can wait past the 10s default expiry of the AssignVolume token it carries, and
the volume server then answers 401. shouldReassignUpload treats a 4xx as final,
so the uploader replays the same expired token instead of re-assigning and the
write fails up to the caller.
This only became reachable once the filer stopped re-minting a fresh token
after the wait.
LookupFileId only requires the fid to contain a single comma, and the value
is pasted straight into the volume server URL path, so ?proxyChunkId=3,x/../../status
resolves to volume 3 and then addresses an endpoint the caller never named:
Go sends the dot segments verbatim, the volume server's mux cleans the path
and redirects to /status, and the filer follows the redirect and relays the
body. That reaches any handler on the volume server -- status, stats, the UI --
past a filer that operators expect to be the only exposed surface.
Parse the fid before the lookup and answer 400 when it doesn't parse.
A trailing _N delta suffix from batch assigns is legal, so it is stripped
first, but only when it is a non-empty run of digits. The volume server strips
at the last "_" unconditionally, which is safe there because its fid came out
of a path the mux parsed and so cannot hold a "/"; here the value is raw query
input, and an unguarded strip would reduce "3,01637037d6_1/../../status" to a
valid fid and wave the traversal through.
Introduce security.BearerPrefix ("Bearer ", RFC 6750) and use it
everywhere an "Authorization: Bearer <token>" header is constructed,
replacing the scattered "BEARER "/"Bearer " string literals. SeaweedFS
matches the scheme case-insensitively when parsing (security.GetJwt), so
behavior is unchanged; this removes the magic string and settles the
casing on the standard form. The parser's upper-case comparison stays as
is on purpose.
The /?proxyChunkId= endpoint forwards the caller's headers to the volume
server but never mints a read token, so proxied chunk reads return 401
once jwt.signing.read.key is configured. Generate a fileId-scoped volume
token the same way the direct filer read path does, which fixes
filer.sync, filer.backup, filerProxy mounts, the MQ broker and the upload
gateway in one place.
* fix(filer): propagate proxyChunkId query params to volume server
When weed mount reads via filer proxy mode (-volumeServerAccess=filerProxy),
the mount adds query params like readDeleted=true to chunk read requests.
Two bugs prevented these from working:
1. filer_server_handlers.go extracted fileId from the raw RequestURI, which
includes query params, corrupting the fileId (e.g. '6,abc&readDeleted=true').
Fix: use r.URL.Query().Get("proxyChunkId") for clean extraction.
2. filer_server_handlers_proxy.go didn't forward query params to the volume
server. The urlStrings from LookupFileId already contain the fileId in the
path, so just append the original query string.
* filer: match chunk proxy by query param, not URI prefix order
Order-dependent prefix slicing missed proxyChunkId when it wasn't the
first query param. Gate on root path and read the parsed query value.
* filer: drop internal proxyChunkId from proxied volume query
Lookup URLs already carry the fileId in the path, so forwarding the raw
query duplicated proxyChunkId onto the volume server. Strip it and only
append the remaining caller params (e.g. readDeleted).
---------
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* chore: remove unreachable dead code across the codebase
Remove ~50,000 lines of unreachable code identified by static analysis.
Major removals:
- weed/filer/redis_lua: entire unused Redis Lua filer store implementation
- weed/wdclient/net2, resource_pool: unused connection/resource pool packages
- weed/plugin/worker/lifecycle: unused lifecycle plugin worker
- weed/s3api: unused S3 policy templates, presigned URL IAM, streaming copy,
multipart IAM, key rotation, and various SSE helper functions
- weed/mq/kafka: unused partition mapping, compression, schema, and protocol functions
- weed/mq/offset: unused SQL storage and migration code
- weed/worker: unused registry, task, and monitoring functions
- weed/query: unused SQL engine, parquet scanner, and type functions
- weed/shell: unused EC proportional rebalance functions
- weed/storage/erasure_coding/distribution: unused distribution analysis functions
- Individual unreachable functions removed from 150+ files across admin,
credential, filer, iam, kms, mount, mq, operation, pb, s3api, server,
shell, storage, topology, and util packages
* fix(s3): reset shared memory store in IAM test to prevent flaky failure
TestLoadIAMManagerFromConfig_EmptyConfigWithFallbackKey was flaky because
the MemoryStore credential backend is a singleton registered via init().
Earlier tests that create anonymous identities pollute the shared store,
causing LookupAnonymous() to unexpectedly return true.
Fix by calling Reset() on the memory store before the test runs.
* style: run gofmt on changed files
* fix: restore KMS functions used by integration tests
* fix(plugin): prevent panic on send to closed worker session channel
The Plugin.sendToWorker method could panic with "send on closed channel"
when a worker disconnected while a message was being sent. The race was
between streamSession.close() closing the outgoing channel and sendToWorker
writing to it concurrently.
Add a done channel to streamSession that is closed before the outgoing
channel, and check it in sendToWorker's select to safely detect closed
sessions without panicking.
* Add iceberg_maintenance plugin worker handler (Phase 1)
Implement automated Iceberg table maintenance as a new plugin worker job
type. The handler scans S3 table buckets for tables needing maintenance
and executes operations in the correct Iceberg order: expire snapshots,
remove orphan files, and rewrite manifests.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add data file compaction to iceberg maintenance handler (Phase 2)
Implement bin-packing compaction for small Parquet data files:
- Enumerate data files from manifests, group by partition
- Merge small files using parquet-go (read rows, write merged output)
- Create new manifest with ADDED/DELETED/EXISTING entries
- Commit new snapshot with compaction metadata
Add 'compact' operation to maintenance order (runs before expire_snapshots),
configurable via target_file_size_bytes and min_input_files thresholds.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix memory exhaustion in mergeParquetFiles by processing files sequentially
Previously all source Parquet files were loaded into memory simultaneously,
risking OOM when a compaction bin contained many small files. Now each file
is loaded, its rows are streamed into the output writer, and its data is
released before the next file is loaded — keeping peak memory proportional
to one input file plus the output buffer.
* Validate bucket/namespace/table names against path traversal
Reject names containing '..', '/', or '\' in Execute to prevent
directory traversal via crafted job parameters.
* Add filer address failover in iceberg maintenance handler
Try each filer address from cluster context in order instead of only
using the first one. This improves resilience when the primary filer
is temporarily unreachable.
* Add separate MinManifestsToRewrite config for manifest rewrite threshold
The rewrite_manifests operation was reusing MinInputFiles (meant for
compaction bin file counts) as its manifest count threshold. Add a
dedicated MinManifestsToRewrite field with its own config UI section
and default value (5) so the two thresholds can be tuned independently.
* Fix risky mtime fallback in orphan removal that could delete new files
When entry.Attributes is nil, mtime defaulted to Unix epoch (1970),
which would always be older than the safety threshold, causing the
file to be treated as eligible for deletion. Skip entries with nil
Attributes instead, matching the safer logic in operations.go.
* Fix undefined function references in iceberg_maintenance_handler.go
Use the exported function names (ShouldSkipDetectionByInterval,
BuildDetectorActivity, BuildExecutorActivity) matching their
definitions in vacuum_handler.go.
* Remove duplicated iceberg maintenance handler in favor of iceberg/ subpackage
The IcebergMaintenanceHandler and its compaction code in the parent
pluginworker package duplicated the logic already present in the
iceberg/ subpackage (which self-registers via init()). The old code
lacked stale-plan guards, proper path normalization, CAS-based xattr
updates, and error-returning parseOperations.
Since the registry pattern (default "all") makes the old handler
unreachable, remove it entirely. All functionality is provided by
iceberg.Handler with the reviewed improvements.
* Fix MinManifestsToRewrite clamping to match UI minimum of 2
The clamp reset values below 2 to the default of 5, contradicting the
UI's advertised MinValue of 2. Clamp to 2 instead.
* Sort entries by size descending in splitOversizedBin for better packing
Entries were processed in insertion order which is non-deterministic
from map iteration. Sorting largest-first before the splitting loop
improves bin packing efficiency by filling bins more evenly.
* Add context cancellation check to drainReader loop
The row-streaming loop in drainReader did not check ctx between
iterations, making long compaction merges uncancellable. Check
ctx.Done() at the top of each iteration.
* Fix splitOversizedBin to always respect targetSize limit
The minFiles check in the split condition allowed bins to grow past
targetSize when they had fewer than minFiles entries, defeating the
OOM protection. Now bins always split at targetSize, and a trailing
runt with fewer than minFiles entries is merged into the previous bin.
* Add integration tests for iceberg table maintenance plugin worker
Tests start a real weed mini cluster, create S3 buckets and Iceberg
table metadata via filer gRPC, then exercise the iceberg.Handler
operations (ExpireSnapshots, RemoveOrphans, RewriteManifests) against
the live filer. A full maintenance cycle test runs all operations in
sequence and verifies metadata consistency.
Also adds exported method wrappers (testing_api.go) so the integration
test package can call the unexported handler methods.
* Fix splitOversizedBin dropping files and add source path to drainReader errors
The runt-merge step could leave leading bins with fewer than minFiles
entries (e.g. [80,80,10,10] with targetSize=100, minFiles=2 would drop
the first 80-byte file). Replace the filter-based approach with an
iterative merge that folds any sub-minFiles bin into its smallest
neighbor, preserving all eligible files.
Also add the source file path to drainReader error messages so callers
can identify which Parquet file caused a read/write failure.
* Harden integration test error handling
- s3put: fail immediately on HTTP 4xx/5xx instead of logging and
continuing
- lookupEntry: distinguish NotFound (return nil) from unexpected RPC
errors (fail the test)
- writeOrphan and orphan creation in FullMaintenanceCycle: check
CreateEntryResponse.Error in addition to the RPC error
* go fmt
---------
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix(filer): limit concurrent proxy reads per volume server
Add a per-volume-server semaphore (default 16) to proxyToVolumeServer
to prevent replication bursts from overwhelming individual volume
servers with hundreds of concurrent connections, which causes them
to drop connections with "unexpected EOF".
Excess requests queue up and respect the client's context, returning
503 if the client disconnects while waiting.
Also log io.CopyBuffer errors that were previously silently discarded.
* Apply suggestion from @gemini-code-assist[bot]
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* fix(filer): use non-blocking release for proxy semaphore
Prevents a goroutine from blocking forever if releaseProxySemaphore
is ever called without a matching acquire.
* test(filer): clean up proxySemaphores entries in all proxy tests
---------
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Added global http client
* Added Do func for global http client
* Changed the code to use the global http client
* Fix http client in volume uploader
* Fixed pkg name
* Fixed http util funcs
* Fixed http client for bench_filer_upload
* Fixed http client for stress_filer_upload
* Fixed http client for filer_server_handlers_proxy
* Fixed http client for command_fs_merge_volumes
* Fixed http client for command_fs_merge_volumes and command_volume_fsck
* Fixed http client for s3api_server
* Added init global client for main funcs
* Rename global_client to client
* Changed:
- fixed NewHttpClient;
- added CheckIsHttpsClientEnabled func
- updated security.toml in scaffold
* Reduce the visibility of some functions in the util/http/client pkg
* Added the loadSecurityConfig function
* Use util.LoadSecurityConfiguration() in NewHttpClient func
Chris Lu