Enable discovery of SOSAPI capabilities without requiring credentials.
- Modify AuthWithPublicRead to bypass auth for SOSAPI objects if bucket exists
- Supports Veeam's initial discovery phase before full IAM setup
- Validates bucket existence to prevent information disclosure
Add early interception for SOSAPI virtual objects in GetObjectHandler and HeadObjectHandler.
- Check for SOSAPI objects (.system-*/system.xml, .system-*/capacity.xml) before normal processing
- Delegate to handleSOSAPIGetObject and handleSOSAPIHeadObject when detected
- Ensures virtual objects are served without hitting storage layer
Implement Smart Object Storage API (SOSAPI) support for Veeam integration.
- Add s3api_sosapi.go with XML structures and handlers for system.xml and capacity.xml
- Implement virtual object detection and dynamic XML generation
- Add capacity retrieval via gRPC (to be optimized in follow-up)
- Include comprehensive unit tests covering detection, XML generation, and edge cases
This enables Veeam Backup & Replication to discover SeaweedFS capabilities and capacity.
When paginating with keyMarker, we must collect all versions first because
filtering happens after sorting. Previously, we limited collection to maxKeys+1
which caused us to miss versions beyond the marker when there were many versions
before it.
The ListObjectVersions API was receiving key-marker and version-id-marker
parameters but not using them to filter results. This caused infinite
pagination loops when clients tried to paginate through results.
Fix by adding filtering logic after sorting:
- Skip versions with key < keyMarker (already returned in previous pages)
- For key == keyMarker, skip versions with versionId >= versionIdMarker
- Include versions with key > keyMarker or (key == keyMarker and versionId < versionIdMarker)
This respects the S3 sort order (key ascending, versionId descending for same key)
and correctly returns only versions that come AFTER the marker position.
Add pagination stress tests (>1000 versions) to the S3 versioning stress
test job in GitHub CI. These tests run on master branch pushes to validate
that ListObjectVersions correctly handles objects with more than 1000
versions using pagination.
This resolves GitHub issue #7747 by extracting duplicated IAM code into
a shared package that both the embedded S3 IAM and standalone IAM use.
New shared package (weed/iam/):
- constants.go: Common constants (charsets, action strings, error messages)
- helpers.go: Shared helper functions (Hash, GenerateRandomString,
GenerateAccessKeyId, GenerateSecretAccessKey, StringSlicesEqual,
MapToStatementAction, MapToIdentitiesAction, MaskAccessKey)
- responses.go: Common IAM response structs (CommonResponse, ListUsersResponse,
CreateUserResponse, etc.)
- helpers_test.go: Unit tests for shared helpers
Updated files:
- weed/s3api/s3api_embedded_iam.go: Use type aliases and function wrappers
to the shared package, removing ~200 lines of duplicated code
- weed/iamapi/iamapi_management_handlers.go: Use shared package for constants
and helper functions, removing ~100 lines of duplicated code
- weed/iamapi/iamapi_response.go: Re-export types from shared package for
backwards compatibility
Benefits:
- Single source of truth for IAM constants and helpers
- Easier maintenance - changes only need to be made in one place
- Reduced risk of inconsistencies between embedded and standalone IAM
- Better test coverage through shared test suite
The allowEmptyFolder option is no longer functional because:
1. The code that used it was already commented out
2. Empty folder cleanup is now handled asynchronously by EmptyFolderCleaner
The CLI flags are kept for backward compatibility but marked as deprecated
and ignored. This removes:
- S3ApiServerOption.AllowEmptyFolder field
- The actual usage in s3api_object_handlers_list.go
- Helm chart values and template references
- References in test Makefiles and docker-compose files
- Modified test/s3/tagging/s3_tagging_test.go to use environment variables for configurable endpoint and credentials
- Added s3-tagging-tests job to .github/workflows/s3-go-tests.yml to run tagging tests in CI
- Tests will now run automatically on pull requests
- Add X-Amz-Tagging header parsing in putToFiler function for PUT object operations
- Store tags with X-Amz-Tagging- prefix in entry.Extended metadata
- Add comprehensive test suite for S3 object tagging functionality
- Tests cover upload tagging, API operations, special characters, and edge cases
- Update matchesCollection to recognize 'default' as filter for empty collection
- Remove incorrect conversion of 'default' to empty string in handlers
- Fixes issue where ?collection=default would show all collections instead of just default collection
Chris Lu