* fix filer range read
Only return true if we're reading the ENTIRE chunk from the beginning.
// This prevents bandwidth amplification when range requests happen to align
// with chunk boundaries but don't actually want the full chunk.
* Update weed/filer/filechunks.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* refactoring
* add ec shard size
* address comments
* passing task id
There seems to be a disconnect between the pending tasks created in ActiveTopology and the TaskDetectionResult returned by this function. A taskID is generated locally and used to create pending tasks via AddPendingECShardTask, but this taskID is not stored in the TaskDetectionResult or passed along in any way.
This makes it impossible for the worker that eventually executes the task to know which pending task in ActiveTopology it corresponds to. Without the correct taskID, the worker cannot call AssignTask or CompleteTask on the master, breaking the entire task lifecycle and capacity management feature.
A potential solution is to add a TaskID field to TaskDetectionResult and worker_pb.TaskParams, ensuring the ID is propagated from detection to execution.
* 1 source multiple destinations
* task supports multi source and destination
* ec needs to clean up previous shards
* use erasure coding constants
* getPlanningCapacityUnsafe getEffectiveAvailableCapacityUnsafe should return StorageSlotChange for calculation
* use CanAccommodate to calculate
* remove dead code
* address comments
* fix Mutex Copying in Protobuf Structs
* use constants
* fix estimatedSize
The calculation for estimatedSize only considers source.EstimatedSize and dest.StorageChange, but omits dest.EstimatedSize. The TaskDestination struct has an EstimatedSize field, which seems to be ignored here. This could lead to an incorrect estimation of the total size of data involved in tasks on a disk. The loop should probably also include estimatedSize += dest.EstimatedSize.
* at.assignTaskToDisk(task)
* refactoring
* Update weed/admin/topology/internal.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* fail fast
* fix compilation
* Update weed/worker/tasks/erasure_coding/detection.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* indexes for volume and shard locations
* dedup with ToVolumeSlots
* return an additional boolean to indicate success, or an error
* Update abstract_sql_store.go
* fix
* Update weed/worker/tasks/erasure_coding/detection.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update weed/admin/topology/task_management.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* faster findVolumeDisk
* Update weed/worker/tasks/erasure_coding/detection.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update weed/admin/topology/storage_slot_test.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* refactor
* simplify
* remove unused GetDiskStorageImpact function
* refactor
* add comments
* Update weed/admin/topology/storage_impact.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update weed/admin/topology/storage_slot_test.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update storage_impact.go
* AddPendingTask
The unified AddPendingTask function now serves as the single entry point for all task creation, successfully consolidating the previously separate functions while maintaining full functionality and improving code organization.
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* refactor planning into task detection
* refactoring worker tasks
* refactor
* compiles, but only balance task is registered
* compiles, but has nil exception
* avoid nil logger
* add back ec task
* setting ec log directory
* implement balance and vacuum tasks
* EC tasks will no longer fail with "file not found" errors
* Use ReceiveFile API to send locally generated shards
* distributing shard files and ecx,ecj,vif files
* generate .ecx files correctly
* do not mount all possible EC shards (0-13) on every destination
* use constants
* delete all replicas
* rename files
* pass in volume size to tasks
* initial design
* added simulation as tests
* reorganized the codebase to move the simulation framework and tests into their own dedicated package
* integration test. ec worker task
* remove "enhanced" reference
* start master, volume servers, filer
Current Status
✅ Master: Healthy and running (port 9333)
✅ Filer: Healthy and running (port 8888)
✅ Volume Servers: All 6 servers running (ports 8080-8085)
🔄 Admin/Workers: Will start when dependencies are ready
* generate write load
* tasks are assigned
* admin start wtih grpc port. worker has its own working directory
* Update .gitignore
* working worker and admin. Task detection is not working yet.
* compiles, detection uses volumeSizeLimitMB from master
* compiles
* worker retries connecting to admin
* build and restart
* rendering pending tasks
* skip task ID column
* sticky worker id
* test canScheduleTaskNow
* worker reconnect to admin
* clean up logs
* worker register itself first
* worker can run ec work and report status
but:
1. one volume should not be repeatedly worked on.
2. ec shards needs to be distributed and source data should be deleted.
* move ec task logic
* listing ec shards
* local copy, ec. Need to distribute.
* ec is mostly working now
* distribution of ec shards needs improvement
* need configuration to enable ec
* show ec volumes
* interval field UI component
* rename
* integration test with vauuming
* garbage percentage threshold
* fix warning
* display ec shard sizes
* fix ec volumes list
* Update ui.go
* show default values
* ensure correct default value
* MaintenanceConfig use ConfigField
* use schema defined defaults
* config
* reduce duplication
* refactor to use BaseUIProvider
* each task register its schema
* checkECEncodingCandidate use ecDetector
* use vacuumDetector
* use volumeSizeLimitMB
* remove
remove
* remove unused
* refactor
* use new framework
* remove v2 reference
* refactor
* left menu can scroll now
* The maintenance manager was not being initialized when no data directory was configured for persistent storage.
* saving config
* Update task_config_schema_templ.go
* enable/disable tasks
* protobuf encoded task configurations
* fix system settings
* use ui component
* remove logs
* interface{} Reduction
* reduce interface{}
* reduce interface{}
* avoid from/to map
* reduce interface{}
* refactor
* keep it DRY
* added logging
* debug messages
* debug level
* debug
* show the log caller line
* use configured task policy
* log level
* handle admin heartbeat response
* Update worker.go
* fix EC rack and dc count
* Report task status to admin server
* fix task logging, simplify interface checking, use erasure_coding constants
* factor in empty volume server during task planning
* volume.list adds disk id
* track disk id also
* fix locking scheduled and manual scanning
* add active topology
* simplify task detector
* ec task completed, but shards are not showing up
* implement ec in ec_typed.go
* adjust log level
* dedup
* implementing ec copying shards and only ecx files
* use disk id when distributing ec shards
🎯 Planning: ActiveTopology creates DestinationPlan with specific TargetDisk
📦 Task Creation: maintenance_integration.go creates ECDestination with DiskId
🚀 Task Execution: EC task passes DiskId in VolumeEcShardsCopyRequest
💾 Volume Server: Receives disk_id and stores shards on specific disk (vs.store.Locations[req.DiskId])
📂 File System: EC shards and metadata land in the exact disk directory planned
* Delete original volume from all locations
* clean up existing shard locations
* local encoding and distributing
* Update docker/admin_integration/EC-TESTING-README.md
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* check volume id range
* simplify
* fix tests
* fix types
* clean up logs and tests
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* fix listing objects
* add more list testing
* address comments
* fix next marker
* fix isTruncated in listing
* fix tests
* address tests
* Update s3api_object_handlers_multipart.go
* fixes
* store json into bucket content, for tagging and cors
* switch bucket metadata from json to proto
* fix
* Update s3api_bucket_config.go
* fix test issue
* fix test_bucket_listv2_delimiter_prefix
* Update cors.go
* skip special characters
* passing listing
* fix test_bucket_list_delimiter_prefix
* ok. fix the xsd generated go code now
* fix cors tests
* fix test
* fix test_bucket_list_unordered and test_bucket_listv2_unordered
do not accept the allow-unordered and delimiter parameter combination
* fix test_bucket_list_objects_anonymous and test_bucket_listv2_objects_anonymous
The tests test_bucket_list_objects_anonymous and test_bucket_listv2_objects_anonymous were failing because they try to set bucket ACL to public-read, but SeaweedFS only supported private ACL.
Updated PutBucketAclHandler to use the existing ExtractAcl function which already supports all standard S3 canned ACLs
Replaced the hardcoded check for only private ACL with proper ACL parsing that handles public-read, public-read-write, authenticated-read, bucket-owner-read, bucket-owner-full-control, etc.
Added unit tests to verify all standard canned ACLs are accepted
* fix list unordered
The test is expecting the error code to be InvalidArgument instead of InvalidRequest
* allow anonymous listing( and head, get)
* fix test_bucket_list_maxkeys_invalid
Invalid values: max-keys=blah → Returns ErrInvalidMaxKeys (HTTP 400)
* updating IsPublicRead when parsing acl
* more logs
* CORS Test Fix
* fix test_bucket_list_return_data
* default to private
* fix test_bucket_list_delimiter_not_skip_special
* default no acl
* add debug logging
* more logs
* use basic http client
remove logs also
* fixes
* debug
* Update stats.go
* debugging
* fix anonymous test expectation
anonymous user can read, as configured in s3 json.
* add back tests
* get put object acl
* check permission to put object acl
* rename file
* object list versions now contains owners
* set object owner
* refactoring
* Revert "add back tests"
This reverts commit 9adc507c45.
* test versioning also
* fix some versioning tests
* fall back
* fixes
Never-versioned buckets: No VersionId headers, no Status field
Pre-versioning objects: Regular files, VersionId="null", included in all operations
Post-versioning objects: Stored in .versions directories with real version IDs
Suspended versioning: Proper status handling and null version IDs
* fixes
Bucket Versioning Status Compliance
Fixed: New buckets now return no Status field (AWS S3 compliant)
Before: Always returned "Suspended" ❌
After: Returns empty VersioningConfiguration for unconfigured buckets ✅
2. Multi-Object Delete Versioning Support
Fixed: DeleteMultipleObjectsHandler now fully versioning-aware
Before: Always deleted physical files, breaking versioning ❌
After: Creates delete markers or deletes specific versions properly ✅
Added: DeleteMarker field in response structure for AWS compatibility
3. Copy Operations Versioning Support
Fixed: CopyObjectHandler and CopyObjectPartHandler now versioning-aware
Before: Only copied regular files, couldn't handle versioned sources ❌
After: Parses version IDs from copy source, creates versions in destination ✅
Added: pathToBucketObjectAndVersion() function for version ID parsing
4. Pre-versioning Object Handling
Fixed: getLatestObjectVersion() now has proper fallback logic
Before: Failed when .versions directory didn't exist ❌
After: Falls back to regular objects for pre-versioning scenarios ✅
5. Enhanced Object Version Listings
Fixed: listObjectVersions() includes both versioned AND pre-versioning objects
Before: Only showed .versions directories, ignored pre-versioning objects ❌
After: Shows complete version history with VersionId="null" for pre-versioning ✅
6. Null Version ID Handling
Fixed: getSpecificObjectVersion() properly handles versionId="null"
Before: Couldn't retrieve pre-versioning objects by version ID ❌
After: Returns regular object files for "null" version requests ✅
7. Version ID Response Headers
Fixed: PUT operations only return x-amz-version-id when appropriate
Before: Returned version IDs for non-versioned buckets ❌
After: Only returns version IDs for explicitly configured versioning ✅
* more fixes
* fix copying with versioning, multipart upload
* more fixes
* reduce volume size for easier dev test
* fix
* fix version id
* fix versioning
* Update filer_multipart.go
* fix multipart versioned upload
* more fixes
* more fixes
* fix versioning on suspended
* fixes
* fixing test_versioning_obj_suspended_copy
* Update s3api_object_versioning.go
* fix versions
* skipping test_versioning_obj_suspend_versions
* > If the versioning state has never been set on a bucket, it has no versioning state; a GetBucketVersioning request does not return a versioning state value.
* fix tests, avoid duplicated bucket creation, skip tests
* only run s3tests_boto3/functional/test_s3.py
* fix checking filer_pb.ErrNotFound
* Update weed/s3api/s3api_object_versioning.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update weed/s3api/s3api_object_handlers_copy.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update weed/s3api/s3api_bucket_config.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update test/s3/versioning/s3_versioning_test.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* fix GetObjectLockConfigurationHandler
* cache and use bucket object lock config
* subscribe to bucket configuration changes
* increase bucket config cache TTL
* refactor
* Update weed/s3api/s3api_server.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* avoid duplidated work
* rename variable
* Update s3api_object_handlers_put.go
* fix routing
* admin ui and api handler are consistent now
* use fields instead of xml
* fix test
* address comments
* Update weed/s3api/s3api_object_handlers_put.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update test/s3/retention/s3_retention_test.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update weed/s3api/object_lock_utils.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* change error style
* errorf
* read entry once
* add s3 tests for object lock and retention
* use marker
* install s3 tests
* Update s3tests.yml
* Update s3tests.yml
* Update s3tests.conf
* Update s3tests.conf
* address test errors
* address test errors
With these fixes, the s3-tests should now:
✅ Return InvalidBucketState (409 Conflict) for object lock operations on invalid buckets
✅ Return MalformedXML for invalid retention configurations
✅ Include VersionId in response headers when available
✅ Return proper HTTP status codes (403 Forbidden for retention mode changes)
✅ Handle all object lock validation errors consistently
* fixes
With these comprehensive fixes, the s3-tests should now:
✅ Return InvalidBucketState (409 Conflict) for object lock operations on invalid buckets
✅ Return InvalidRetentionPeriod for invalid retention periods
✅ Return MalformedXML for malformed retention configurations
✅ Include VersionId in response headers when available
✅ Return proper HTTP status codes for all error conditions
✅ Handle all object lock validation errors consistently
The workflow should now pass significantly more object lock tests, bringing SeaweedFS's S3 object lock implementation much closer to AWS S3 compatibility standards.
* fixes
With these final fixes, the s3-tests should now:
✅ Return MalformedXML for ObjectLockEnabled: 'Disabled'
✅ Return MalformedXML when both Days and Years are specified in retention configuration
✅ Return InvalidBucketState (409 Conflict) when trying to suspend versioning on buckets with object lock enabled
✅ Handle all object lock validation errors consistently with proper error codes
* constants and fixes
✅ Return InvalidRetentionPeriod for invalid retention values (0 days, negative years)
✅ Return ObjectLockConfigurationNotFoundError when object lock configuration doesn't exist
✅ Handle all object lock validation errors consistently with proper error codes
* fixes
✅ Return MalformedXML when both Days and Years are specified in the same retention configuration
✅ Return 400 (Bad Request) with InvalidRequest when object lock operations are attempted on buckets without object lock enabled
✅ Handle all object lock validation errors consistently with proper error codes
* fixes
✅ Return 409 (Conflict) with InvalidBucketState for bucket-level object lock configuration operations on buckets without object lock enabled
✅ Allow increasing retention periods and overriding retention with same/later dates
✅ Only block decreasing retention periods without proper bypass permissions
✅ Handle all object lock validation errors consistently with proper error codes
* fixes
✅ Include VersionId in multipart upload completion responses when versioning is enabled
✅ Block retention mode changes (GOVERNANCE ↔ COMPLIANCE) without bypass permissions
✅ Handle all object lock validation errors consistently with proper error codes
✅ Pass the remaining object lock tests
* fix tests
* fixes
* pass tests
* fix tests
* fixes
* add error mapping
* Update s3tests.conf
* fix test_object_lock_put_obj_lock_invalid_days
* fixes
* fix many issues
* fix test_object_lock_delete_multipart_object_with_legal_hold_on
* fix tests
* refactor
* fix test_object_lock_delete_object_with_retention_and_marker
* fix tests
* fix tests
* fix tests
* fix test itself
* fix tests
* fix test
* Update weed/s3api/s3api_object_retention.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* reduce logs
* address comments
* refactor
* rename
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* fix GetObjectLockConfigurationHandler
* cache and use bucket object lock config
* subscribe to bucket configuration changes
* increase bucket config cache TTL
* refactor
* Update weed/s3api/s3api_server.go
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* avoid duplidated work
* rename variable
* Update s3api_object_handlers_put.go
* fix routing
* admin ui and api handler are consistent now
* use fields instead of xml
* fix test
* address comments
* Update weed/s3api/s3api_object_handlers_put.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update test/s3/retention/s3_retention_test.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update weed/s3api/object_lock_utils.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* change error style
* errorf
* read entry once
* add s3 tests for object lock and retention
* use marker
* install s3 tests
* Update s3tests.yml
* Update s3tests.yml
* Update s3tests.conf
* Update s3tests.conf
* address test errors
* address test errors
With these fixes, the s3-tests should now:
✅ Return InvalidBucketState (409 Conflict) for object lock operations on invalid buckets
✅ Return MalformedXML for invalid retention configurations
✅ Include VersionId in response headers when available
✅ Return proper HTTP status codes (403 Forbidden for retention mode changes)
✅ Handle all object lock validation errors consistently
* fixes
With these comprehensive fixes, the s3-tests should now:
✅ Return InvalidBucketState (409 Conflict) for object lock operations on invalid buckets
✅ Return InvalidRetentionPeriod for invalid retention periods
✅ Return MalformedXML for malformed retention configurations
✅ Include VersionId in response headers when available
✅ Return proper HTTP status codes for all error conditions
✅ Handle all object lock validation errors consistently
The workflow should now pass significantly more object lock tests, bringing SeaweedFS's S3 object lock implementation much closer to AWS S3 compatibility standards.
* fixes
With these final fixes, the s3-tests should now:
✅ Return MalformedXML for ObjectLockEnabled: 'Disabled'
✅ Return MalformedXML when both Days and Years are specified in retention configuration
✅ Return InvalidBucketState (409 Conflict) when trying to suspend versioning on buckets with object lock enabled
✅ Handle all object lock validation errors consistently with proper error codes
* constants and fixes
✅ Return InvalidRetentionPeriod for invalid retention values (0 days, negative years)
✅ Return ObjectLockConfigurationNotFoundError when object lock configuration doesn't exist
✅ Handle all object lock validation errors consistently with proper error codes
* fixes
✅ Return MalformedXML when both Days and Years are specified in the same retention configuration
✅ Return 400 (Bad Request) with InvalidRequest when object lock operations are attempted on buckets without object lock enabled
✅ Handle all object lock validation errors consistently with proper error codes
* fixes
✅ Return 409 (Conflict) with InvalidBucketState for bucket-level object lock configuration operations on buckets without object lock enabled
✅ Allow increasing retention periods and overriding retention with same/later dates
✅ Only block decreasing retention periods without proper bypass permissions
✅ Handle all object lock validation errors consistently with proper error codes
* fixes
✅ Include VersionId in multipart upload completion responses when versioning is enabled
✅ Block retention mode changes (GOVERNANCE ↔ COMPLIANCE) without bypass permissions
✅ Handle all object lock validation errors consistently with proper error codes
✅ Pass the remaining object lock tests
* fix tests
* fixes
* pass tests
* fix tests
* fixes
* add error mapping
* Update s3tests.conf
* fix test_object_lock_put_obj_lock_invalid_days
* fixes
* fix many issues
* fix test_object_lock_delete_multipart_object_with_legal_hold_on
* fix tests
* refactor
* fix test_object_lock_delete_object_with_retention_and_marker
* fix tests
* fix tests
* fix tests
* fix test itself
* fix tests
* fix test
* Update weed/s3api/s3api_object_retention.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* reduce logs
* address comments
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* it compiles
* refactored
* reduce to 4 concurrent chunk upload
* CopyObjectPartHandler
* copy a range of the chunk data, fix offset size in copied chunks
* Update s3api_object_handlers_copy.go
What the PR Accomplishes:
CopyObjectHandler - Now copies entire objects by copying chunks individually instead of downloading/uploading the entire file
CopyObjectPartHandler - Handles copying parts of objects for multipart uploads by copying only the relevant chunk portions
Efficient Chunk Copying - Uses direct chunk-to-chunk copying with proper volume assignment and concurrent processing (limited to 4 concurrent operations)
Range Support - Properly handles range-based copying for partial object copies
* fix compilation
* fix part destination
* handling small objects
* use mkFile
* copy to existing file or part
* add testing tools
* adjust tests
* fix chunk lookup
* refactoring
* fix TestObjectCopyRetainingMetadata
* ensure bucket name not conflicting
* fix conditional copying tests
* remove debug messages
* add custom s3 copy tests
* add a menu item "Message Queue"
* add a menu item "Message Queue"
* move the "brokers" link under it.
* add "topics", "subscribers". Add pages for them.
* refactor
* show topic details
* admin display publisher and subscriber info
* remove publisher and subscribers from the topic row pull down
* collecting more stats from publishers and subscribers
* fix layout
* fix publisher name
* add local listeners for mq broker and agent
* render consumer group offsets
* remove subscribers from left menu
* topic with retention
* support editing topic retention
* show retention when listing topics
* create bucket
* Update s3_buckets_templ.go
* embed the static assets into the binary
fix https://github.com/seaweedfs/seaweedfs/issues/6964
* add object versioning
* add missing file
* Update weed/s3api/s3api_object_versioning.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update weed/s3api/s3api_object_versioning.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update weed/s3api/s3api_object_versioning.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* ListObjectVersionsResult is better to show multiple version entries
* fix test
* Update weed/s3api/s3api_object_handlers_put.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update weed/s3api/s3api_object_versioning.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* multiple improvements
* move PutBucketVersioningHandler into weed/s3api/s3api_bucket_handlers.go file
* duplicated code for reading bucket config, versioningEnabled, etc. try to use functions
* opportunity to cache bucket config
* error handling if bucket is not found
* in case bucket is not found
* fix build
* add object versioning tests
* remove non-existent tests
* add tests
* add versioning tests
* skip a new test
* ensure .versions directory exists before saving info into it
* fix creating version entry
* logging on creating version directory
* Update s3api_object_versioning_test.go
* retry and wait for directory creation
* revert add more logging
* Update s3api_object_versioning.go
* more debug messages
* clean up logs, and touch directory correctly
* log the .versions creation and then parent directory listing
* use mkFile instead of touch
touch is for update
* clean up data
* add versioning test in go
* change location
* if modified, latest version is moved to .versions directory, and create a new latest version
Core versioning functionality: WORKING
TestVersioningBasicWorkflow - PASS
TestVersioningDeleteMarkers - PASS
TestVersioningMultipleVersionsSameObject - PASS
TestVersioningDeleteAndRecreate - PASS
TestVersioningListWithPagination - PASS
❌ Some advanced features still failing:
ETag calculation issues (using mtime instead of proper MD5)
Specific version retrieval (EOF error)
Version deletion (internal errors)
Concurrent operations (race conditions)
* calculate multi chunk md5
Test Results - All Passing:
✅ TestBucketListReturnDataVersioning - PASS
✅ TestVersioningCreateObjectsInOrder - PASS
✅ TestVersioningBasicWorkflow - PASS
✅ TestVersioningMultipleVersionsSameObject - PASS
✅ TestVersioningDeleteMarkers - PASS
* dedupe
* fix TestVersioningErrorCases
* fix eof error of reading old versions
* get specific version also check current version
* enable integration tests for versioning
* trigger action to work for now
* Fix GitHub Actions S3 versioning tests workflow
- Fix syntax error (incorrect indentation)
- Update directory paths from weed/s3api/versioning_tests/ to test/s3/versioning/
- Add push trigger for add-object-versioning branch to enable CI during development
- Update artifact paths to match correct directory structure
* Improve CI robustness for S3 versioning tests
Makefile improvements:
- Increase server startup timeout from 30s to 90s for CI environments
- Add progressive timeout reporting (logs at 30s, full logs at 90s)
- Better error handling with server logs on failure
- Add server PID tracking for debugging
- Improved test failure reporting
GitHub Actions workflow improvements:
- Increase job timeouts to account for CI environment delays
- Add system information logging (memory, disk space)
- Add detailed failure reporting with server logs
- Add process and network diagnostics on failure
- Better error messaging and log collection
These changes should resolve the 'Server failed to start within 30 seconds' issue
that was causing the CI tests to fail.
* adjust testing volume size
* Update Makefile
* Update Makefile
* Update Makefile
* Update Makefile
* Update s3-versioning-tests.yml
* Update s3api_object_versioning.go
* Update Makefile
* do not clean up
* log received version id
* more logs
* printout response
* print out list version response
* use tmp files when put versioned object
* change to versions folder layout
* Delete weed-test.log
* test with mixed versioned and unversioned objects
* remove versionDirCache
* remove unused functions
* remove unused function
* remove fallback checking
* minor
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* add ui for maintenance
* valid config loading. fix workers page.
* refactor
* grpc between admin and workers
* add a long-running bidirectional grpc call between admin and worker
* use the grpc call to heartbeat
* use the grpc call to communicate
* worker can remove the http client
* admin uses http port + 10000 as its default grpc port
* one task one package
* handles connection failures gracefully with exponential backoff
* grpc with insecure tls
* grpc with optional tls
* fix detecting tls
* change time config from nano seconds to seconds
* add tasks with 3 interfaces
* compiles reducing hard coded
* remove a couple of tasks
* remove hard coded references
* reduce hard coded values
* remove hard coded values
* remove hard coded from templ
* refactor maintenance package
* fix import cycle
* simplify
* simplify
* auto register
* auto register factory
* auto register task types
* self register types
* refactor
* simplify
* remove one task
* register ui
* lazy init executor factories
* use registered task types
* DefaultWorkerConfig remove hard coded task types
* remove more hard coded
* implement get maintenance task
* dynamic task configuration
* "System Settings" should only have system level settings
* adjust menu for tasks
* ensure menu not collapsed
* render job configuration well
* use templ for ui of task configuration
* fix ordering
* fix bugs
* saving duration in seconds
* use value and unit for duration
* Delete WORKER_REFACTORING_PLAN.md
* Delete maintenance.json
* Delete custom_worker_example.go
* remove address from workers
* remove old code from ec task
* remove creating collection button
* reconnect with exponential backoff
* worker use security.toml
* start admin server with tls info from security.toml
* fix "weed admin" cli description
* rename
* set agent address
* refactor
* add agent sub
* pub messages
* grpc new client
* can publish records via agent
* send init message with session id
* fmt
* check cancelled request while waiting
* use sessionId
* handle possible nil stream
* subscriber process messages
* separate debug port
* use atomic int64
* less logs
* minor
* skip io.EOF
* rename
* remove unused
* use saved offsets
* do not reuse session, since always session id is new after restart
remove last active ts from SessionEntry
* simplify printing
* purge unused
* just proxy the subscription, skipping the session step
* adjust offset types
* subscribe offset type and possible value
* start after the known tsns
* avoid wrongly set startPosition
* move
* remove
* refactor
* typo
* fix
* fix changed path
* listing files to convert to parquet
* write parquet files
* save logs into parquet files
* pass by value
* compact logs into parquet format
* can skip existing files
* refactor
* refactor
* fix compilation
* when no partition found
* refactor
* add untested parquet file read
* rename package
* refactor
* rename files
* remove unused
* add merged log read func
* parquet wants to know the file size
* rewind by time
* pass in stop ts
* add stop ts
* adjust log
* minor
* adjust log
* skip .parquet files when reading message logs
* skip non message files
* Update subscriber_record.go
* send messages
* skip message data with only ts
* skip non log files
* update parquet-go package
* ensure a valid record type
* add new field to a record type
* Update read_parquet_to_log.go
* fix parquet file name generation
* separating reading parquet and logs
* add key field
* add skipped logs
* use in memory cache
* refactor
* refactor
* refactor
* refactor, and change compact log
* refactor
* rename
* refactor
* fix format
* prefix v to version directory
* adjust for better logs
* if shared lock, still need to wait for exclusive lock to release
* fix compilation
* fix waiting condition
* avoid deleting the entry too early
* compare chunks by timestamp
* fix slab clearing error
* fix test compilation
* move oldest chunk to sealed, instead of by fullness
* lock on fh.entryViewCache
* remove verbose logs
* revert slat clearing
* less logs
* less logs
* track write and read by timestamp
* remove useless logic
* add entry lock on file handle release
* use mem chunk only, swap file chunk has problems
* comment out code that maybe used later
* add debug mode to compare data read and write
* more efficient readResolvedChunks with linked list
* small optimization
* fix test compilation
* minor fix on writer
* add SeparateGarbageChunks
* group chunks into sections
* turn off debug mode
* fix tests
* fix tests
* tmp enable swap file chunk
* Revert "tmp enable swap file chunk"
This reverts commit 985137ec47.
* simple refactoring
* simple refactoring
* do not re-use swap file chunk. Sealed chunks should not be re-used.
* comment out debugging facilities
* either mem chunk or swap file chunk is fine now
* remove orderedMutex as *semaphore.Weighted
not found impactful
* optimize size calculation for changing large files
* optimize performance to avoid going through the long list of chunks
* still problems with swap file chunk
* rename
* tiny optimization
* swap file chunk save only successfully read data
* fix
* enable both mem and swap file chunk
* resolve chunks with range
* rename
* fix chunk interval list
* also change file handle chunk group when adding chunks
* pick in-active chunk with time-decayed counter
* fix compilation
* avoid nil with empty fh.entry
* refactoring
* rename
* rename
* refactor visible intervals to *list.List
* refactor chunkViews to *list.List
* add IntervalList for generic interval list
* change visible interval to use IntervalList in generics
* cahnge chunkViews to *IntervalList[*ChunkView]
* use NewFileChunkSection to create
* rename variables
* refactor
* fix renaming leftover
* renaming
* renaming
* add insert interval
* interval list adds lock
* incrementally add chunks to readers
Fixes:
1. set start and stop offset for the value object
2. clone the value object
3. use pointer instead of copy-by-value when passing to interval.Value
4. use insert interval since adding chunk could be out of order
* fix tests compilation
* fix tests compilation
Temporarily enable cgo following findings from https://github.com/seaweedfs/seaweedfs/discussions/3152 by https://github.com/dyus
Resolve build issue:
```
# github.com/mattn/go-ieproxy
/go/pkg/mod/github.com/mattn/go-ieproxy@v0.0.3/ieproxy.go:36:9: undefined: getConf
/go/pkg/mod/github.com/mattn/go-ieproxy@v0.0.3/ieproxy.go:41:9: undefined: reloadConf
/go/pkg/mod/github.com/mattn/go-ieproxy@v0.0.3/ieproxy.go:48:2: undefined: overrideEnvWithStaticProxy
/go/pkg/mod/github.com/mattn/go-ieproxy@v0.0.3/ieproxy.go:53:13: psc.findProxyForURL undefined (type *ProxyScriptConf has no field or method findProxyForURL, but does have FindProxyForURL)
```
* Revert previous changes
* s3: use cursor to track tree traversal
fix https://github.com/seaweedfs/seaweedfs/issues/3166
* special cases for empty prefix and empty directory
* use constants
* address empty folder
* undo local changes
* fix IsTruncated
* adjust counting directories
* fix cases when prefix is a directory
* s3: handle directory object
works for
aws --endpoint-url http://127.0.0.1:8333/ s3api list-objects-v2 --bucket test --prefix "fakedir"
