4 Commits

Include renames

Author	SHA1	Message	Date
Chris Lu	fa9bf58c86	test(s3): make the whole versioning suite pass and gate it in CI (#9844) ... * test(s3): correct bucket-recreate expectations and cover the different-owner case A same-owner CreateBucket on an existing bucket returns BucketAlreadyOwnedByYou (idempotent recreate); the suite expected BucketAlreadyExists, which only applies when the name is owned by someone else. Fix the same-owner cases (plain and Object-Lock) and implement the previously-skipped different-owner test, which now exercises the BucketAlreadyExists path via a second identity. * test(s3): assert the deletion invariant for suspended-versioning delete A suspended-versioning DELETE removes the null version and records a delete marker so the object reads as deleted; the test expected no marker, which would let an older version resurface. Assert that a marker is recorded (and read DeleteMarker through aws.ToBool) rather than an exact count, so it holds whether or not the suspended-marker id/dedup is later collapsed to AWS's single null marker. * test(s3): run the whole versioning suite by default TEST_PATTERN was TestVersioning, which left bucket-creation, suspended-delete and directory/version-listing tests ungated. Default to '.' so every test runs; opt-in stress tests self-skip without ENABLE_STRESS_TESTS and keep their own targets.	2026-06-06 18:38:28 -07:00
Chris Lu	02574314f6	test(s3): force-drop collection after deleteBucket in tagging/versioning/cors/copying (#9270) ... * test(s3): force-drop collection after deleteBucket across tagging/versioning/cors/copying Each test creates a unique bucket (= new SeaweedFS collection) and the master's warm-create issues a 7-volume grow batch. The S3 DeleteBucket-driven collection sweep snapshots the layout once, but in-flight `volume_grow` requests keep registering volumes after the snapshot, leaking 1-3 volumes per bucket. On a single `weed mini` data node with the auto-derived volume cap, those leaks pile up fast and every subsequent PutObject 500s with "Not enough data nodes found". Mirror the retention-suite fix (commits ac3a756d, 363d5caa) into the four other suites that share the same shape: defer a master-side /col/delete after each bucket teardown, and sweep stale prefix-matching buckets before every new createBucket. Each suite gets its own MASTER_ENDPOINT default plus the allTestBucketPrefixes / cleanupAllTestBuckets / cleanupLeftoverTestBuckets / forceDeleteCollection helpers. Skipped: iam (separate framework + v1 SDK; already passes -master.volumeSizeLimitMB=100), sse (different cleanup signature; docker-compose already passes -volumeSizeLimitMB=50), policy (its TestCluster already uses -master.volumeSizeLimitMB=32 and tears the whole cluster down). Suites under 5 tests cannot exhaust the cap and were left untouched. * test(s3): default master endpoint to 127.0.0.1 to avoid IPv6 resolution Mirror the explicit IPv4 default already used by test/s3/copying. On hosts where `localhost` resolves to ::1 first the master HTTP listener (bound on 0.0.0.0) is unreachable, so the new force-delete-collection helper would silently skip cleanup. Pin the default to 127.0.0.1 in tagging/cors/versioning; the MASTER_ENDPOINT env var still wins when set. * test(s3): skip buckets newer than this process in leftover-bucket sweep cleanupAllTestBuckets/cleanupTestBuckets used to delete every bucket whose name matched the suite's prefix. With shared S3/master endpoints, a second concurrent `go test` run could have its live buckets torn down mid-test by the first run's sweep. Capture testRunStart at package init (with a 1-minute backdate for clock skew) and skip any bucket whose CreationDate is newer than that. Stale buckets from panicked or interrupted prior runs (the original target of the sweep) still get collected because they were created before this process started. * test(s3-copying): sweep stale prefix buckets from createBucket, not just from a few callers The s3-copying suite already had cleanupTestBuckets that walks every bucket and drops the test-copying-* prefix matches, but only four tests in the file invoke it; the rest go straight to createBucket. So a panicked or interrupted prior run could leak buckets that survive into the next run and exhaust the data node's volume slots before any of the prefix-sweeping tests get a chance to run. Hoist the sweep into createBucket so every test that creates a bucket starts on a clean slate. The per-process CreationDate filter from the prior commit keeps this safe under concurrent runs. * test(s3): scope leftover-bucket sweep to this run via runID marker The CreationDate filter only protected against runs that started later than this process. A different `go test` against the same S3/master endpoints that started *earlier* and is still active has buckets older than testRunStart, so the sweep would still tear them down mid-test. Replace the time window with a per-process runID baked into bucket names: every bucket this run creates gets `r{runID}-` after the suite's BucketPrefix, and the sweep only matches that owned subset. Concurrent runs each carry their own runID and never see each other's buckets. Trade-off: buckets left behind by a crashed prior run carry a different runID and won't be cleaned up by this sweep. That recovery path now belongs to `make clean` / data-dir wipe, which is what CI already does between jobs. Fixed-name versioning buckets (e.g. test-versioning-directories) bypass getNewBucketName and so also bypass this sweep — they are short-lived and handled by their own deferred deleteBucket. * test(s3): drop cleanupLeftoverTestBuckets wrapper, call cleanupAllTestBuckets directly cleanupLeftoverTestBuckets was a one-line forwarder kept only as a semantic name at the createBucket call site. Inline the call and update the doc comments to point at the actual implementation. tagging/cors/versioning all had the wrapper; copying never did.	2026-04-28 22:13:42 -07:00
Chris LuandCopilot	12f50d37fa	test versioning also (#7000) ... * test versioning also * fix some versioning tests * fall back * fixes Never-versioned buckets: No VersionId headers, no Status field Pre-versioning objects: Regular files, VersionId="null", included in all operations Post-versioning objects: Stored in .versions directories with real version IDs Suspended versioning: Proper status handling and null version IDs * fixes Bucket Versioning Status Compliance Fixed: New buckets now return no Status field (AWS S3 compliant) Before: Always returned "Suspended" ❌ After: Returns empty VersioningConfiguration for unconfigured buckets ✅ 2. Multi-Object Delete Versioning Support Fixed: DeleteMultipleObjectsHandler now fully versioning-aware Before: Always deleted physical files, breaking versioning ❌ After: Creates delete markers or deletes specific versions properly ✅ Added: DeleteMarker field in response structure for AWS compatibility 3. Copy Operations Versioning Support Fixed: CopyObjectHandler and CopyObjectPartHandler now versioning-aware Before: Only copied regular files, couldn't handle versioned sources ❌ After: Parses version IDs from copy source, creates versions in destination ✅ Added: pathToBucketObjectAndVersion() function for version ID parsing 4. Pre-versioning Object Handling Fixed: getLatestObjectVersion() now has proper fallback logic Before: Failed when .versions directory didn't exist ❌ After: Falls back to regular objects for pre-versioning scenarios ✅ 5. Enhanced Object Version Listings Fixed: listObjectVersions() includes both versioned AND pre-versioning objects Before: Only showed .versions directories, ignored pre-versioning objects ❌ After: Shows complete version history with VersionId="null" for pre-versioning ✅ 6. Null Version ID Handling Fixed: getSpecificObjectVersion() properly handles versionId="null" Before: Couldn't retrieve pre-versioning objects by version ID ❌ After: Returns regular object files for "null" version requests ✅ 7. Version ID Response Headers Fixed: PUT operations only return x-amz-version-id when appropriate Before: Returned version IDs for non-versioned buckets ❌ After: Only returns version IDs for explicitly configured versioning ✅ * more fixes * fix copying with versioning, multipart upload * more fixes * reduce volume size for easier dev test * fix * fix version id * fix versioning * Update filer_multipart.go * fix multipart versioned upload * more fixes * more fixes * fix versioning on suspended * fixes * fixing test_versioning_obj_suspended_copy * Update s3api_object_versioning.go * fix versions * skipping test_versioning_obj_suspend_versions * > If the versioning state has never been set on a bucket, it has no versioning state; a GetBucketVersioning request does not return a versioning state value. * fix tests, avoid duplicated bucket creation, skip tests * only run s3tests_boto3/functional/test_s3.py * fix checking filer_pb.ErrNotFound * Update weed/s3api/s3api_object_versioning.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_object_handlers_copy.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_bucket_config.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update test/s3/versioning/s3_versioning_test.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	2025-07-19 21:43:34 -07:00
Chris LuandCopilot	cf5a24983a	S3: add object versioning (#6945) ... * add object versioning * add missing file * Update weed/s3api/s3api_object_versioning.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_object_versioning.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_object_versioning.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * ListObjectVersionsResult is better to show multiple version entries * fix test * Update weed/s3api/s3api_object_handlers_put.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_object_versioning.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * multiple improvements * move PutBucketVersioningHandler into weed/s3api/s3api_bucket_handlers.go file * duplicated code for reading bucket config, versioningEnabled, etc. try to use functions * opportunity to cache bucket config * error handling if bucket is not found * in case bucket is not found * fix build * add object versioning tests * remove non-existent tests * add tests * add versioning tests * skip a new test * ensure .versions directory exists before saving info into it * fix creating version entry * logging on creating version directory * Update s3api_object_versioning_test.go * retry and wait for directory creation * revert add more logging * Update s3api_object_versioning.go * more debug messages * clean up logs, and touch directory correctly * log the .versions creation and then parent directory listing * use mkFile instead of touch touch is for update * clean up data * add versioning test in go * change location * if modified, latest version is moved to .versions directory, and create a new latest version Core versioning functionality: WORKING TestVersioningBasicWorkflow - PASS TestVersioningDeleteMarkers - PASS TestVersioningMultipleVersionsSameObject - PASS TestVersioningDeleteAndRecreate - PASS TestVersioningListWithPagination - PASS ❌ Some advanced features still failing: ETag calculation issues (using mtime instead of proper MD5) Specific version retrieval (EOF error) Version deletion (internal errors) Concurrent operations (race conditions) * calculate multi chunk md5 Test Results - All Passing: ✅ TestBucketListReturnDataVersioning - PASS ✅ TestVersioningCreateObjectsInOrder - PASS ✅ TestVersioningBasicWorkflow - PASS ✅ TestVersioningMultipleVersionsSameObject - PASS ✅ TestVersioningDeleteMarkers - PASS * dedupe * fix TestVersioningErrorCases * fix eof error of reading old versions * get specific version also check current version * enable integration tests for versioning * trigger action to work for now * Fix GitHub Actions S3 versioning tests workflow - Fix syntax error (incorrect indentation) - Update directory paths from weed/s3api/versioning_tests/ to test/s3/versioning/ - Add push trigger for add-object-versioning branch to enable CI during development - Update artifact paths to match correct directory structure * Improve CI robustness for S3 versioning tests Makefile improvements: - Increase server startup timeout from 30s to 90s for CI environments - Add progressive timeout reporting (logs at 30s, full logs at 90s) - Better error handling with server logs on failure - Add server PID tracking for debugging - Improved test failure reporting GitHub Actions workflow improvements: - Increase job timeouts to account for CI environment delays - Add system information logging (memory, disk space) - Add detailed failure reporting with server logs - Add process and network diagnostics on failure - Better error messaging and log collection These changes should resolve the 'Server failed to start within 30 seconds' issue that was causing the CI tests to fail. * adjust testing volume size * Update Makefile * Update Makefile * Update Makefile * Update Makefile * Update s3-versioning-tests.yml * Update s3api_object_versioning.go * Update Makefile * do not clean up * log received version id * more logs * printout response * print out list version response * use tmp files when put versioned object * change to versions folder layout * Delete weed-test.log * test with mixed versioned and unversioned objects * remove versionDirCache * remove unused functions * remove unused function * remove fallback checking * minor --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	2025-07-09 01:51:45 -07:00