When paginating with keyMarker, we must collect all versions first because
filtering happens after sorting. Previously, we limited collection to maxKeys+1
which caused us to miss versions beyond the marker when there were many versions
before it.
The ListObjectVersions API was receiving key-marker and version-id-marker
parameters but not using them to filter results. This caused infinite
pagination loops when clients tried to paginate through results.
Fix by adding filtering logic after sorting:
- Skip versions with key < keyMarker (already returned in previous pages)
- For key == keyMarker, skip versions with versionId >= versionIdMarker
- Include versions with key > keyMarker or (key == keyMarker and versionId < versionIdMarker)
This respects the S3 sort order (key ascending, versionId descending for same key)
and correctly returns only versions that come AFTER the marker position.
Add pagination stress tests (>1000 versions) to the S3 versioning stress
test job in GitHub CI. These tests run on master branch pushes to validate
that ListObjectVersions correctly handles objects with more than 1000
versions using pagination.
This resolves GitHub issue #7747 by extracting duplicated IAM code into
a shared package that both the embedded S3 IAM and standalone IAM use.
New shared package (weed/iam/):
- constants.go: Common constants (charsets, action strings, error messages)
- helpers.go: Shared helper functions (Hash, GenerateRandomString,
GenerateAccessKeyId, GenerateSecretAccessKey, StringSlicesEqual,
MapToStatementAction, MapToIdentitiesAction, MaskAccessKey)
- responses.go: Common IAM response structs (CommonResponse, ListUsersResponse,
CreateUserResponse, etc.)
- helpers_test.go: Unit tests for shared helpers
Updated files:
- weed/s3api/s3api_embedded_iam.go: Use type aliases and function wrappers
to the shared package, removing ~200 lines of duplicated code
- weed/iamapi/iamapi_management_handlers.go: Use shared package for constants
and helper functions, removing ~100 lines of duplicated code
- weed/iamapi/iamapi_response.go: Re-export types from shared package for
backwards compatibility
Benefits:
- Single source of truth for IAM constants and helpers
- Easier maintenance - changes only need to be made in one place
- Reduced risk of inconsistencies between embedded and standalone IAM
- Better test coverage through shared test suite
The allowEmptyFolder option is no longer functional because:
1. The code that used it was already commented out
2. Empty folder cleanup is now handled asynchronously by EmptyFolderCleaner
The CLI flags are kept for backward compatibility but marked as deprecated
and ignored. This removes:
- S3ApiServerOption.AllowEmptyFolder field
- The actual usage in s3api_object_handlers_list.go
- Helm chart values and template references
- References in test Makefiles and docker-compose files
- Modified test/s3/tagging/s3_tagging_test.go to use environment variables for configurable endpoint and credentials
- Added s3-tagging-tests job to .github/workflows/s3-go-tests.yml to run tagging tests in CI
- Tests will now run automatically on pull requests
- Add X-Amz-Tagging header parsing in putToFiler function for PUT object operations
- Store tags with X-Amz-Tagging- prefix in entry.Extended metadata
- Add comprehensive test suite for S3 object tagging functionality
- Tests cover upload tagging, API operations, special characters, and edge cases
- Update matchesCollection to recognize 'default' as filter for empty collection
- Remove incorrect conversion of 'default' to empty string in handlers
- Fixes issue where ?collection=default would show all collections instead of just default collection
Gap detection and skipping to earliest memory time
Time-based reads that include events at boundary times for first reads (offset ≤ 0)
Aggregated subscriber wake-up via ListenersWaits signaling
Removed the replace directive and updated all references to use
github.com/seaweedfs/cockroachdb-parser directly instead of redirecting
from github.com/cockroachdb/cockroachdb-parser.
Changes:
- Updated import in weed/query/engine/cockroach_parser.go
- Removed replace directive from go.mod
- Now using seaweedfs/cockroachdb-parser@909763b171
which includes all 32-bit architecture fixes and uses
seaweedfs namespace throughout
Build verified successfully for GOOS=openbsd GOARCH=arm.
Use seaweedfs/cockroachdb-parser v0.0.0-20251021182748-d0c58c67297e
via replace directive to fix building on 32-bit architectures like OpenBSD ARM.
The replace directive ensures all imports of github.com/cockroachdb/cockroachdb-parser
use the seaweedfs fork which includes fixes for:
- Integer overflow in tsearch evaluation (int -> int64)
- Flag type overflow in lexbase and tree packages (int -> int64)
Updated to seaweedfs/cockroachdb-parser v0.0.0-20251021182748-d0c58c67297e
which includes fixes for building on 32-bit architectures like OpenBSD ARM.
Fixes:
- Integer overflow in tsearch evaluation (int -> int64)
- Flag type overflow in lexbase and tree packages (int -> int64)
Chris Lu