Commit Graph
12114 Commits
Author SHA1 Message Date
chrislu d2e7b39e4a fmt 2025-09-14 23:00:35 -07:00
chrislu 70d53ce91a s3: further improve filer consistency retry logic for CI environment
- Increase retry attempts from 5 to 8 for both updateLatestVersionInDirectory
  and getLatestObjectVersion functions
- Increase base delay from 50ms to 100ms with exponential backoff up to 6.4s
- Add specific retry logic for 'no Extended metadata' race condition where
  .versions directory exists but metadata is not yet written
- Add detailed timing logs to track retry delays and total wait times
- Addresses persistent CI failures where even 5 retries with 400ms max delay
  were insufficient for filer store consistency in GitHub Actions environment
2025-09-14 22:49:03 -07:00
chrislu b0afb12b16 s3: improve retry logic for filer consistency with exponential backoff
- Increase retry attempts from 3 to 5 for both updateLatestVersionInDirectory
  and getLatestObjectVersion functions
- Implement exponential backoff: 50ms, 100ms, 200ms, 400ms delays
- Addresses persistent CI failures where .versions directories are not
  immediately visible after creation in filer store
- Based on CI log analysis showing 50ms fixed delays were insufficient
- Maintains CI debug logging to track improved retry behavior
2025-09-14 22:40:12 -07:00
chrislu 581392638b s3: add retry logic for filer consistency in versioned object operations
- Add retry logic to updateLatestVersionInDirectory to handle cases where
  .versions directory creation succeeds but is not immediately visible
- Add retry logic to getLatestObjectVersion for the same consistency issue
- Use 3 retries with 50ms delays to handle filer store consistency timing
- Addresses CI failures where 'filer: no entry is found in filer store'
  occurs after successful directory creation
- Maintains CI debug logging to track retry attempts and outcomes
2025-09-14 22:24:53 -07:00
chrislu 1f275afc8f Add CI debug logging for versioning race condition investigation
- Add detailed logging for .versions metadata updates in putVersionedObject
- Add logging for latest version resolution in getLatestObjectVersion
- Add logging for HeadObject latest version requests
- All logs use glog.V(0) with CI-DEBUG prefix for easy filtering
- Will help diagnose timing issues between object creation and retrieval in CI

Debug logs will show:
- When .versions metadata updates start and complete
- When HeadObject tries to read latest version metadata
- Race conditions if HeadObject runs before metadata update completes
- Missing metadata if .versions directory exists but metadata keys are missing
- File access issues if version files exist but can't be accessed
2025-09-14 22:07:18 -07:00
chrislu f24a50e001 Update s3api_object_handlers_put.go 2025-09-14 21:45:15 -07:00
chrislu 20ab0960a3 less debug 2025-09-14 21:39:28 -07:00
chrislu e5ef704cdd add debug messages 2025-09-14 21:35:09 -07:00
chrislu 19daf8371a fix Error: s3api/s3api_object_handlers_put.go:595:12: no new variables on left side of := 2025-09-14 21:17:10 -07:00
chrislu a22b63927e Ensure the .versions directory exists before uploading 2025-09-14 21:12:48 -07:00
chrislu 206191a2d3 ensure .versions directory exists 2025-09-14 21:04:07 -07:00
chrislu b91160aae9 Fix S3 tests bucket creation conflicts 2025-09-14 20:52:21 -07:00
chrislu a395ef16fa ensure Kafka Consumer Group Tests (Highly Isolated) 2025-09-14 20:37:37 -07:00
chrislu ea1978e2af Bucket already exists - always return BucketAlreadyExists per S3 specification 2025-09-14 20:27:25 -07:00
chrislu b0392731b7 bucket re-creation error handling
Different Owner: Always fails with 409 BucketAlreadyExists
Checks the s3-identity-id header against the stored bucket owner
Returns error immediately if owners don't match
Same Owner + Conflicting Settings: Fails with 409 BucketAlreadyExists
Compares requested Object Lock settings with existing bucket configuration
Returns error if settings are incompatible (e.g., trying to enable Object Lock on a bucket that doesn't have it)
Same Owner + Compatible Settings: Returns 200 OK (idempotent)
If the bucket already exists with the same owner and compatible settings
Returns success response without recreating the bucket
2025-09-14 20:10:24 -07:00
chrislu 3f58eb3763 Update s3tests.yml 2025-09-14 19:10:31 -07:00
chrislu b2dab1d3af Update s3tests.yml 2025-09-14 19:01:29 -07:00
chrislu a5b264d582 6fd748b0e3a0088e0bf32c849865039355326327 2025-09-14 18:54:50 -07:00
chrislu 99f0176286 pin to commit 95c232b for https://github.com/ceph/s3-tests.git 2025-09-14 18:51:51 -07:00
chrislu cee8fa3f99 randomize 2025-09-14 14:53:23 -07:00
chrislu 40ecd6ad6e Update s3tests.yml 2025-09-14 14:45:41 -07:00
chrislu 033237a732 fix path for -s3.config=../docker/compose/s3.json 2025-09-14 14:36:00 -07:00
chrislu ec7adc5a36 Update fetch_multibatch.go 2025-09-14 14:10:44 -07:00
chrislu d0059cb65b update plans 2025-09-14 13:36:20 -07:00
chrislu ef6a1f3798 Delete size.txt 2025-09-14 13:11:38 -07:00
chrislu 61bfe73029 Revert "Merge branch 'master' into feature/mq-kafka-gateway-m1"
This reverts commit 78382b380e, reversing
changes made to 56aa5278af.
2025-09-14 12:50:19 -07:00
chrislu 7c82d5e7a0 Update metadata_version_test.go 2025-09-14 12:05:11 -07:00
chrislu 75847eacfb update 2025-09-14 11:41:22 -07:00
chrislu 515a03a377 Update kafka-tests.yml 2025-09-14 11:29:30 -07:00
chrislu 6faa0e86a7 fux ulimit 2025-09-14 11:26:27 -07:00
chrislu 76a5cfbf1f fixes 2025-09-14 11:21:53 -07:00
chrislu 0437957db5 Update offset_management.go 2025-09-14 11:15:33 -07:00
chrislu 4e11a378fd fix tests 2025-09-14 10:52:23 -07:00
chrislu 881a418b95 Update kafka-tests.yml 2025-09-14 10:14:01 -07:00
chrislu 12d890d45c only run one test 2025-09-14 09:04:54 -07:00
chrislu 3b551fcaf7 Update kafka-tests.yml 2025-09-13 21:50:49 -07:00
chrislu 579de7381d Update kafka-tests.yml 2025-09-13 21:44:13 -07:00
chrislu 8b1962bd3a Update kafka-tests.yml 2025-09-13 21:43:55 -07:00
chrislu 8ff78983e3 separate containers 2025-09-13 21:41:32 -07:00
chrislu 3224e71e87 Update kafka-tests.yml 2025-09-13 21:32:15 -07:00
chrislu 58eb40db01 ctx.Done() 2025-09-13 21:29:27 -07:00
chrislu e501abcde6 timeout 2025-09-13 21:24:32 -07:00
chrislu 4a2642e54d more timeout 2025-09-13 21:20:54 -07:00
chrislu 1c21527179 timout 2025-09-13 21:15:05 -07:00
chrislu 5d5c820c88 timeouts 2025-09-13 21:07:24 -07:00
chrislu 75ffd30a03 goroutine-based context-cancellable I/O + aggressive goroutine timeouts 2025-09-13 20:58:51 -07:00
chrislu 48803d9a68 server shutdown timeout 2025-09-13 20:53:29 -07:00
chrislu 031c67dd5d Both io.ReadFull and time.Sleep are now context-aware 2025-09-13 20:45:34 -07:00
chrislu 015f6f2910 cancel ctx 2025-09-13 20:30:31 -07:00
chrislu 5a2fd1413f context-cancellable read 2025-09-13 20:23:58 -07:00