mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-15 19:10:48 +02:00
cd7c8ae784ce0394dfd19f8f4d28cd41d853e8cd
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris Lu
|
fa549e9c83 |
filer: harden TUS session authorization against cross-prefix access (#10315)
* filer: authorize TUS existing-session verbs against the validated stored target Scope-check on TUS HEAD/PATCH/DELETE only populated a resource path for POST, so a prefix-restricted token that learned another tenant's session id could act on that session and land content at a TargetPath its own AllowedPrefixes forbid. Split the filer JWT check into authenticateFilerJwt (signature and method) and authorizeFilerJwtPaths (resource scope), and make the scope check fail closed: a prefix-restricted token with no resolved resource path is denied instead of authorized on signature alone. The TUS handler now authenticates first, reads and validates the session once, authorizes the stored TargetPath, then operates on that single pinned snapshot. readTusSessionInfo rejects a session whose id, target or size is unusable, and getTusSession is split so the authorization lookup no longer lists chunks. * filer: reject non-canonical TUS upload ids The uploads route took the first path component as the session id, so a trailing path or other non-canonical spelling aliased one session under several URLs. Require the id to be a canonical UUID, the only form the server mints, both at routing and when reading a session's metadata, so one URL maps to one resource. * filer: revalidate the pinned TUS session before completing an upload Completion re-read chunks but not the session identity, so a PATCH finishing after a concurrent DELETE or metadata replacement could still land at the id's stored path. Before completing, confirm the session still exists and its target, size and creation time are unchanged from the authorized snapshot; otherwise the completion fails instead of writing to a path the request never authorized. * filer: log TUS session lookup failures before returning not-found readTusSessionInfo and loadTusSessionChunks failures answered "not found" with no log line, so a transient filer or listing error was indistinguishable from a genuinely missing session. Log the lookup at V(1) (a missing session is common and benign) and the chunk-load error at Errorf (the session already resolved). |
||
|
Chris Lu
|
ce82e3a057 |
filer: scope TUS HEAD/PATCH/DELETE against the session target path (#10309)
* filer: scope TUS HEAD/PATCH/DELETE against the session target path checkTusJwtAuthorization only populated the scoped-path list for POST, so a prefix-restricted token was scope-checked at session creation but not on HEAD/PATCH/DELETE, which act on an existing session addressed by id. A low-privilege tenant holding a valid write token who learns another upload's session id could PATCH attacker bytes into that session, or DELETE/HEAD it, landing content at a target path its own AllowedPrefixes forbids. The session id is unguessable to an unauthenticated attacker but is not an authorization boundary against a legitimate tenant. Resolve the effective target for HEAD/PATCH/DELETE from the session's stored TargetPath and scope the prefix check against it, the same way POST scopes the create target. The scoped paths are resolved lazily so the session read runs only for a prefix-restricted token; unrestricted tokens and deployments without a signing key touch no extra state. An unknown session stays unscoped so the handler still answers 404. * filer: fail closed when a TUS session target cannot be resolved The lazy scope resolver swallowed every readTusSessionInfo error and left the request unscoped, so a filer read error or a corrupt session .info would authorize a prefix-restricted token against a target the server never resolved. Only a genuinely missing session should stay unscoped (so the handler answers 404); any other failure now propagates and denies the request. checkJwtAuthorizationScoped's resolver returns an error and a failed resolution is treated as not-authorized. * filer: fold the scoped-path resolver into checkJwtAuthorization checkJwtAuthorization now takes the lazy resolver directly instead of a separate checkJwtAuthorizationScoped wrapper, and the surrounding comments are trimmed to the load-bearing why. |
||
|
Chris Lu
|
4f1f0dcb17 |
filer: require JWT authorization on TUS upload endpoints (#10249)
* filer: require JWT authorization on TUS upload endpoints filerHandler and readonlyFilerHandler run every request through maybeCheckJwtAuthorization, but the TUS routes were registered only behind filerGuard.WhiteList, which is a pass-through for the filer (its guard is built with an empty whitelist), and no TUS handler called the JWT check. So with jwt.filer_signing.key set, normal PUT/POST/DELETE were authorized while the TUS endpoints were not. Run the same check in tusHandler before routing: HEAD uses the read key, POST/PATCH/DELETE use the write key, and creation scopes a prefix-restricted token against the resolved target path. OPTIONS stays open for capability discovery. * filer: enforce read-only and WORM rules on TUS completion completeTusUpload wrote the final entry with CreateEntry directly, skipping the read-only and WORM checks the normal write path applies. Reject completion when the target prefix is read-only or the existing entry at the target is WORM-enforced. * filer: align TUS write path with the normal write path Resolve the create target with a guaranteed leading slash so a prefix-restricted token and the stored path stay absolute even if TusBasePath were set with a trailing slash. Reject read-only prefixes at session creation before any chunk is written, and map read-only and WORM rejections at completion to 507 and 403 instead of a generic 500. * filer: cover method-restricted TUS tokens in the auth test |
||
|
Chris Lu
|
83195fc111 |
filer: reuse the caller's fetched entry in CreateEntry (#9645)
CreateEntry starts with a FindEntry to load the current entry. A conditional CreateEntry already fetched that entry to evaluate the precondition under the per-path lock, so the create repeated the lookup. Add an existing *Entry parameter: when non-nil it is used as the current entry and the internal lookup is skipped; nil keeps the lookup. The gRPC CreateEntry handler passes the entry it fetched for the precondition, removing the redundant read while the lock is held. All other callers pass nil. |
||
|
Chris Lu
|
1b1e5f69a2 |
Add TUS protocol support for resumable uploads (#7592)
* Add TUS protocol integration tests
This commit adds integration tests for the TUS (resumable upload) protocol
in preparation for implementing TUS support in the filer.
Test coverage includes:
- OPTIONS handler for capability discovery
- Basic single-request upload
- Chunked/resumable uploads
- HEAD requests for offset tracking
- DELETE for upload cancellation
- Error handling (invalid offsets, missing uploads)
- Creation-with-upload extension
- Resume after interruption simulation
Tests are skipped in short mode and require a running SeaweedFS cluster.
* Add TUS session storage types and utilities
Implements TUS upload session management:
- TusSession struct for tracking upload state
- Session creation with directory-based storage
- Session persistence using filer entries
- Session retrieval and offset updates
- Session deletion with chunk cleanup
- Upload completion with chunk assembly into final file
Session data is stored in /.uploads.tus/{upload-id}/ directory,
following the pattern used by S3 multipart uploads.
* Add TUS HTTP handlers
Implements TUS protocol HTTP handlers:
- tusHandler: Main entry point routing requests
- tusOptionsHandler: Capability discovery (OPTIONS)
- tusCreateHandler: Create new upload (POST)
- tusHeadHandler: Get upload offset (HEAD)
- tusPatchHandler: Upload data at offset (PATCH)
- tusDeleteHandler: Cancel upload (DELETE)
- tusWriteData: Upload data to volume servers
Features:
- Supports creation-with-upload extension
- Validates TUS protocol headers
- Offset conflict detection
- Automatic upload completion when size is reached
- Metadata parsing from Upload-Metadata header
* Wire up TUS protocol routes in filer server
Add TUS handler route (/.tus/) to the filer HTTP server.
The TUS route is registered before the catch-all route to ensure
proper routing of TUS protocol requests.
TUS protocol is now accessible at:
- OPTIONS /.tus/ - Capability discovery
- POST /.tus/{path} - Create upload
- HEAD /.tus/.uploads/{id} - Get offset
- PATCH /.tus/.uploads/{id} - Upload data
- DELETE /.tus/.uploads/{id} - Cancel upload
* Improve TUS integration test setup
Add comprehensive Makefile for TUS tests with targets:
- test-with-server: Run tests with automatic server management
- test-basic/chunked/resume/errors: Specific test categories
- manual-start/stop: For development testing
- debug-logs/status: For debugging
- ci-test: For CI/CD pipelines
Update README.md with:
- Detailed TUS protocol documentation
- All endpoint descriptions with headers
- Usage examples with curl commands
- Architecture diagram
- Comparison with S3 multipart uploads
Follows the pattern established by other tests in test/ folder.
* Fix TUS integration tests and creation-with-upload
- Fix test URLs to use full URLs instead of relative paths
- Fix creation-with-upload to refresh session before completing
- Fix Makefile to properly handle test cleanup
- Add FullURL helper function to TestCluster
* Add TUS protocol tests to GitHub Actions CI
- Add tus-tests.yml workflow that runs on PRs and pushes
- Runs when TUS-related files are modified
- Automatic server management for integration testing
- Upload logs on failure for debugging
* Make TUS base path configurable via CLI
- Add -tus.path CLI flag to filer command
- TUS is disabled by default (empty path)
- Example: -tus.path=/.tus to enable at /.tus endpoint
- Update test Makefile to use -tus.path flag
- Update README with TUS enabling instructions
* Rename -tus.path to -tusBasePath with default .tus
- Rename CLI flag from -tus.path to -tusBasePath
- Default to .tus (TUS enabled by default)
- Add -filer.tusBasePath option to weed server command
- Properly handle path prefix (prepend / if missing)
* Address code review comments
- Sort chunks by offset before assembling final file
- Use chunk.Offset directly instead of recalculating
- Return error on invalid file ID instead of skipping
- Require Content-Length header for PATCH requests
- Use fs.option.Cipher for encryption setting
- Detect MIME type from data using http.DetectContentType
- Fix concurrency group for push events in workflow
- Use os.Interrupt instead of Kill for graceful shutdown in tests
* fmt
* Address remaining code review comments
- Fix potential open redirect vulnerability by sanitizing uploadLocation path
- Add language specifier to README code block
- Handle os.Create errors in test setup
- Use waitForHTTPServer instead of time.Sleep for master/volume readiness
- Improve test reliability and debugging
* Address critical and high-priority review comments
- Add per-session locking to prevent race conditions in updateTusSessionOffset
- Stream data directly to volume server instead of buffering entire chunk
- Only buffer 512 bytes for MIME type detection, then stream remaining data
- Clean up session locks when session is deleted
* Fix race condition to work across multiple filer instances
- Store each chunk as a separate file entry instead of updating session JSON
- Chunk file names encode offset, size, and fileId for atomic storage
- getTusSession loads chunks from directory listing (atomic read)
- Eliminates read-modify-write race condition across multiple filers
- Remove in-memory mutex that only worked for single filer instance
* Address code review comments: fix variable shadowing, sniff size, and test stability
- Rename path variable to reqPath to avoid shadowing path package
- Make sniff buffer size respect contentLength (read at most contentLength bytes)
- Handle Content-Length < 0 in creation-with-upload (return error for chunked encoding)
- Fix test cluster: use temp directory for filer store, add startup delay
* Fix test stability: increase cluster stabilization delay to 5 seconds
The tests were intermittently failing because the volume server needed more
time to create volumes and register with the master. Increasing the delay
from 2 to 5 seconds fixes the flaky test behavior.
* Address PR review comments for TUS protocol support
- Fix strconv.Atoi error handling in test file (lines 386, 747)
- Fix lossy fileId encoding: use base64 instead of underscore replacement
- Add pagination support for ListDirectoryEntries in getTusSession
- Batch delete chunks instead of one-by-one in deleteTusSession
* Address additional PR review comments for TUS protocol
- Fix UploadAt timestamp: use entry.Crtime instead of time.Now()
- Remove redundant JSON content in chunk entry (metadata in filename)
- Refactor tusWriteData to stream in 4MB chunks to avoid OOM on large uploads
- Pass filer.Entry to parseTusChunkPath to preserve actual upload time
* Address more PR review comments for TUS protocol
- Normalize TUS path once in filer_server.go, store in option.TusPath
- Remove redundant path normalization from TUS handlers
- Remove goto statement in tusCreateHandler, simplify control flow
* Remove unnecessary mutexes in tusWriteData
The upload loop is sequential, so uploadErrLock and chunksLock are not needed.
* Rename updateTusSessionOffset to saveTusChunk
Remove unused newOffset parameter and rename function to better reflect its purpose.
* Improve TUS upload performance and add path validation
- Reuse operation.Uploader across sub-chunks for better connection reuse
- Guard against TusPath='/' to prevent hijacking all filer routes
* Address PR review comments for TUS protocol
- Fix critical chunk filename parsing: use strings.Cut instead of SplitN
to correctly handle base64-encoded fileIds that may contain underscores
- Rename tusPath to tusBasePath for naming consistency across codebase
- Add background garbage collection for expired TUS sessions (runs hourly)
- Improve error messages with %w wrapping for better debuggability
* Address additional TUS PR review comments
- Fix tusBasePath default to use leading slash (/.tus) for consistency
- Add chunk contiguity validation in completeTusUpload to detect gaps/overlaps
- Fix offset calculation to find maximum contiguous range from 0, not just last chunk
- Return 413 Request Entity Too Large instead of silently truncating content
- Document tusChunkSize rationale (4MB balances memory vs request overhead)
- Fix Makefile xargs portability by removing GNU-specific -r flag
- Add explicit -tusBasePath flag to integration test for robustness
- Fix README example to use /.uploads/tus path format
* Revert log_buffer changes (moved to separate PR)
* Minor style fixes from PR review
- Simplify tusBasePath flag description to use example format
- Add 'TUS upload' prefix to session not found error message
- Remove duplicate tusChunkSize comment
- Capitalize warning message for consistency
- Add grep filter to Makefile xargs for better empty input handling
|