mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-20 13:30:46 +02:00
c2cdefd06d89f4510dfe06508eee7ea5aa49d607
21
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris Lu
|
86761cc7d5 |
filer: keep empty folders that are s3tables catalog entries (#11102)
* s3tables: build the catalog attribute keys from one shared prefix Every attribute the catalog stores on a bucket, namespace, table or view entry is spelled out with the same literal prefix. Name it once in s3_constants so code outside the package can recognize a catalog entry without repeating the string. Claude-Session: https://claude.ai/code/session_01GfZsc4cyNB2yr6KYLRv9q1 * filer: keep empty folders that are s3tables catalog entries A namespace, table or view is a directory whose extended attributes are the catalog record. Its files can live elsewhere - a rename moves only the catalog pointer and leaves the data at the old path, and a view has no files at all - so an empty one is still a live entry. Drop a table, then rename another table onto that name: the drop queues the old table's folders, the rename recreates the name path, and two minutes later the cleaner deletes it and cascades into the namespace, losing a table the catalog still lists. Claude-Session: https://claude.ai/code/session_01GfZsc4cyNB2yr6KYLRv9q1 * filer: drop a queued cleanup when the folder is created again A cleanup is queued against the folder that was found empty. If that folder is deleted and a new one takes its name, the queue entry outlives the folder it was about and the next pass deletes the replacement. A drop followed by a rename onto the dropped name does exactly this: the name path comes back as a live catalog entry two minutes before the queue is read. Claude-Session: https://claude.ai/code/session_01GfZsc4cyNB2yr6KYLRv9q1 |
||
|
Chris Lu
|
7c87d78ea2 |
s3: a key deleted after enabling versioning must leave the listing (#10684)
* s3: a null object wins over a rescan when the latest-version pointer is absent The read path already resolves an absent pointer this way; the listing-path counterpart scanned .versions/ first and could surface an old version or delete marker over the current suspended-versioning null object. * s3: dedup a key against its .versions sibling in suspended buckets too A suspended bucket keeps its .versions directories, so a suspended-versioning null object and its .versions sibling emitted the same key twice. * s3: retract a null object from the listing when a delete marker shadows it Deleting a key whose null version predates versioning leaves the base-path entry in place and records the delete marker under <key>.versions. The listing appended the base-path entry and relied on the .versions sibling to replace it, but a delete-marker current version emitted nothing, so the deleted key stayed visible to ListObjects while GET and HEAD returned 404. * s3: keep a key's .versions sibling on the same page as the key When the page quota ran out between a base-path entry and its .versions directory, the page ended with the stale entry and the next page skipped the directory as a marker echo, so the replacement or retraction never happened. * s3: the null version is not latest when the .versions pointer names a newer one ListObjectVersions stamped IsLatest on every base-path null object, so a key deleted after enabling versioning reported IsLatest on both the delete marker and the null version. * s3: test listing after a pre-versioning null object is delete-marked * s3: find a key's earlier page entry by scan, not by adjacency A key such as k.bak sorts between k and k.versions, so the entry a .versions sibling replaces or retracts is not always the last one on the page. Scan back through the page for the key, and insert a late resolution in sorted position instead of at the end. * s3: settle trailing null objects by lookup when a page fills The quota can run out while keys still sit between a null object and its .versions sibling, and the sibling-adjacent page-boundary exception never fires for those. Track the trailing null objects whose sibling has not been ruled out and look each one up before declaring the page full; a retraction reopens the quota. * s3: do not resolve a .versions sibling its page has already moved past A page resuming from a marker inside the base key's extension region has already listed and settled the base null object on an earlier page, so resolving the .versions directory again re-emitted the key. * s3: test listing with keys between a null object and its .versions sibling * s3: pick the newer of the null object and the scanned versions Making the null object win outright whenever the pointer is absent misread multi-filer pointer lag: version files replicate ahead of the pointer, and a key overwritten or delete-marked after pre-versioning days would list its stale null again. The suspended-versioning write that legitimately makes the null current is also the newer entry, so mtime tells the two apart. * s3: a delete-marked null object no longer keeps its prefix alive The hidden-entries probe took any plain file as proof of a listable key, but a null object shadowed by its .versions sibling's delete marker is not one. Hold plain files pending until the sibling settles them either way. * s3: settle an evicted pending null instead of dropping it Nested keys like k, k!, k!! can hold more pending nulls than the cap. A silently evicted one could close the page unsettled, and the resume skip would then keep the stale entry for good. * s3: test deleted-prefix hiding and the pending-null cap * s3: cover the reported '!' intervening key with a live version * s3: an unstamped same-second version outranks the null object Second-resolution mtimes cannot order same-second writes, so the tie went to the stale null when the pointer lagged. The suspended write that makes a null current stamps the version it displaces before clearing the pointer, so the stamp is the authoritative signal and a tie without it goes to the version. * s3: a pointer-less versions listing still checks what replicated ListObjectVersions took a missing pointer as proof the null object is latest, but under pointer lag the sibling can already hold newer replicated versions or markers. Apply the same nullObjectWins rule as the listing recovery. * s3: a failed null-object settlement fails the listing Every getEntry error read as a missing sibling, so a transient filer error at a page boundary committed the unsettled null and the next page skipped its sibling for good. Only a definitive not-found means the null is live; other failures are retained on eviction and fail the request at page close. * s3: retract a CommonPrefix whose only backers were delete-marked nulls The directory probe settles this for the / delimiter, but any other delimiter derives prefixes from base-path keys directly, and a prefix built solely from null objects survived their delete markers. Count the unsettled null backers behind the newest prefix and retract it when the last one settles as a marker; a live resolution or any listable contributor confirms the prefix instead. * s3: test custom-delimiter prefix retraction * s3: an explicit signal marks the null object current, not the demotion stamp The NoncurrentSinceNs stamp survives promotion: delete the version that demoted another and the promoted one is current yet still stamped, so a lagging replica would resurrect the stale null. A suspended-versioning write now records Seaweed-X-Amz-Null-Version-Is-Latest on the .versions directory when it clears the pointer, every pointer update removes it, and the recovery paths trust the signal instead of the stamp. * s3: a filer failover retry rebuilds the listing page from scratch The failover wrapper reruns the callback on another filer after a transport error, and the partially built page, spent quota, and advanced marker leaked into the retry, which could then return a stale or duplicated page as success. * s3: only a prefix's own backers can debit it A delete marker for a version-only key (no base object) derived the same prefix as its neighbors and decremented backing it never contributed, retracting a prefix that a live null object still backed. Track backers by key so settlement is idempotent and only debits what was counted. * s3: test a version-only marker against a null-backed prefix * s3: a pointer recompute clears the null-current signal The routed finalize for delete markers, COPY, and multipart rewrites the .versions pointer through RECOMPUTE_LATEST, which left a suspended-era null-current signal in place. Version files never carry the signal, so mapping it in CopyExtended deletes it whenever the pointer recomputes. * s3: the pointer outranks the null-current signal in the versions listing The signal check guarded the pointer check, so a stale signal a recompute had not cleared yet would have let the null claim IsLatest alongside the pointed-at version. |
||
|
|
d1f503181b |
[s3] force filer apply s3 expiry metadata (#10469)
* fix: apply S3 Expiry Metadata * add test Header X-Seaweedfs-Expires-S3 * resolve comments * test entry lookup by mtime * filer: skip s3 expiry stamp on versioned entries The s3 expiry path skips entries carrying a version id, so stamping one takes away its expiry rather than moving it onto mtime. Files under .versions/ are written once, so crtime already tracks their needles. --------- Co-authored-by: Konstantin Lebedev <whitefox@mayflower.work> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
d8d29c4ede |
s3: carry storage class in the cached listing metadata (#10516)
A listing on a versioned bucket is served from metadata cached on the .versions directory entry so the whole listing is a single scan. The cache carried size, mtime, ETag, owner and the delete-marker flag but not the storage class, so newListEntry found none and fell back to STANDARD. The result was that HEAD and the listings disagreed about the same object: HEAD reported the class the object was stored with, while ListObjectsV2 and ListObjectVersions reported STANDARD for every object. Clients that filter or tier on storage class act on the listing. Caches the class alongside the other listing fields, clears it with them, and copies it in the routed RECOMPUTE_LATEST path so both finalize paths agree. |
||
|
Cole Helbling
|
eefe634962 |
s3api: FULL_OBJECT checksums for CRC multipart uploads (#10236)
CRC64NVME multipart objects now emit a full-object checksum instead of a composite base64-N value, matching AWS (CRC64NVME supports full-object only). Adds x-amz-checksum-type handling (COMPOSITE/FULL_OBJECT) for CRC32/CRC32C/CRC64NVME via CRC combination, resolved at CreateMultipartUpload and applied at completion. |
||
|
Chris Lu
|
6e16994615 |
s3: make lifecycle TTL fast path per-bucket opt-in (#9825)
Stamping an Expiration.Days rule as a volume TTL at write time bakes an irreversible TTL into the object: removing or lengthening the rule later can't un-expire it, unlike worker-driven expiration. The metadata-only delete it enables also skips per-chunk DeleteFile, so dead bytes linger in a not-yet-expired TTL volume with no deleted-byte accounting until the whole volume ages out. Gate the resolver on a per-bucket flag, off by default; toggle with the s3.bucket.lifecycle.fastpath shell command. Default writes take the worker path: real deletes that honor current policy and let vacuum reclaim space. |
||
|
Chris Lu
|
46bb70d93e |
feat(s3): stamp noncurrent_since on versioned demotions (#9431)
* feat(s3): stamp noncurrent_since on versioned demotions A version's noncurrent TTL clock starts when the next version is written, not at its own mtime. Today the lifecycle engine derives that moment from the next-newer sibling's mtime — a heuristic that drifts if the sibling is later modified and is unavailable when the demoting event sits outside meta-log retention. Stamp Seaweed-X-Amz-Noncurrent-Since-Ns on the demoted entry at the two places where a PUT flips the latest pointer: updateLatestVersionInDirectory and updateIsLatestFlagsForSuspendedVersioning. Timestamp source is time.Now().UnixNano() captured once per demotion — the documented Phase 1 fallback until the filer write API surfaces its own TsNs. Engine reads the stamp on both the bootstrap walker path and the event-driven router; missing/zero falls back to the legacy sibling-mtime derivation, so pre-stamp entries keep working. Prerequisite for the daily-replay lifecycle worker (Phase 2+). * fix(s3): address CI failure and PR review feedback - Backdating tests must move both clocks: the lifecycle integration tests backdate version mtimes to simulate aging, but my earlier commit made the engine prefer the explicit demotion stamp over sibling mtime, so a real-now stamp dominated a backdated mtime and the rule never fired. Update backdateVersionedMtime to also rewrite Seaweed-X-Amz-Noncurrent-Since-Ns when the entry already carries it. This is a test simplification — production stamps record when the successor was written, not the demoted version's own mtime — but the resulting clock is correctly old enough. - Refactor stamp parsing into one shared helper. Per gemini-code-assist: the parsing logic for ExtNoncurrentSinceNsKey was duplicated in router/router.go and scheduler/bootstrap.go. Move it to a new weed/s3api/s3lifecycle/noncurrent_since.go as exported SuccessorFromEntryStamp; both call sites now go through it. - Make the parser ordering test deterministic. Per coderabbitai: time.Now().UnixNano() drops the monotonic clock component, so two back-to-back calls can decrease if the wall clock steps backward — the prior test was exercising OS clock behavior rather than the parser. Replace with fixed nanosecond values. - Close a suspended-versioning race. Per coderabbitai: the prior putSuspendedVersioningObject called updateIsLatestFlagsForSuspendedVersioning after putToFiler returned, i.e. after the object write lock released. A concurrent PUT could promote a newer latest version, which we'd then wipe — leaving the older "null" object incorrectly current. Move the cleanup into the afterCreate callback so the null write and the .versions pointer clear (including the new demotion stamp) run atomically under the same lock. Best-effort logging is preserved. * fix(s3/lifecycle): clear noncurrent_since stamp on test backdate Backdating a version's mtime in tests is not a coherent claim about when it became noncurrent — production stamps record the successor's PUT time, which the test doesn't manipulate. The prior commit rewrote the stamp to the backdated instant, but for TestLifecycleNewerNoncurrent that creates an inconsistent state: v3's stamp says "demoted 30 days ago" while v4's mtime (the supposed demoter) is real-now. With both NewerNoncurrentVersions and NoncurrentDays in the same rule, the NoncurrentDays floor passes against the backdated stamp and the rank-based check then deletes v3 via the meta-log historical replay that misranks against current state. Clearing the stamp instead lets the lifecycle engine fall back to the sibling-mtime derivation the tests were originally written against: the legacy code path is preserved end-to-end while the new explicit- stamp path is exercised by the unit tests in s3lifecycle/noncurrent_since_test.go and the bootstrap-walker integration in scheduler/bootstrap_test.go. The deeper interaction — historical meta-log replay ranking against current state inside routePointerTransitionExpand — is pre-existing and is no longer masked by the freshly-PUT successor's mtime once the stamp is read. Tracked separately; not blocking this PR. * fix(s3): stamp noncurrent_since before the .versions/ pointer flip The pointer-flip on the .versions/ directory emits a meta-log event that the lifecycle router consumes via routePointerTransition. The router then calls LookupVersion on the demoted version's id. With the prior ordering — pointer flip first, stamp second — the router could read the demoted entry before markVersionNoncurrent landed and fall back to the legacy sibling-mtime derivation. Versioned COPY is the clean break: the new latest version keeps the source object's mtime instead of recording the moment v_old was demoted, so the fallback's successor clock can be arbitrarily wrong. Reorder both updateLatestVersionInDirectory and updateIsLatestFlagsForSuspendedVersioning so the stamp is written first; the pointer flip then emits an event into a state where the stamp is already present. Failure of the stamp write remains non-fatal — lifecycle still falls back to the legacy derivation in that case, with the same caveats as before the PR but no race window. |
||
|
Chris Lu
|
3efe88c718 |
feat(s3): store and return checksum headers for additional checksum algorithms (#8914)
* feat(s3): store and return checksum headers for additional checksum algorithms When clients upload with --checksum-algorithm (SHA256, CRC32, etc.), SeaweedFS validated the checksum but discarded it. The checksum was never stored in metadata or returned in PUT/HEAD/GET responses. Now the checksum is computed alongside MD5 during upload, stored in entry extended attributes, and returned as the appropriate x-amz-checksum-* header in all responses. Fixes #8911 * fix(s3): address review feedback and CI failures for checksum support - Gate GET/HEAD checksum response headers on x-amz-checksum-mode: ENABLED per AWS S3 spec, fixing FlexibleChecksumError on ranged GETs and multipart copies - Verify computed checksum against client-provided header value for non-chunked uploads, returning BadDigest on mismatch - Add nil check for getCheckSumWriter to prevent panic - Handle comma-separated values in X-Amz-Trailer header - Use ordered slice instead of map for deterministic checksum header selection; extract shared mappings into package-level vars * fix(s3): skip checksum header for ranged GET responses The stored checksum covers the full object. Returning it for ranged (partial) responses causes SDK checksum validation failures because the SDK validates the header value against the partial content received. Skip emitting x-amz-checksum-* headers when a Range request header is present, fixing PyArrow large file read failures. * fix(s3): reject unsupported checksum algorithm with 400 detectRequestedChecksumAlgorithm now returns an error code when x-amz-sdk-checksum-algorithm or x-amz-checksum-algorithm contains an unsupported value, instead of silently ignoring it. * feat(s3): compute composite checksum for multipart uploads Store the checksum algorithm during CreateMultipartUpload, then during CompleteMultipartUpload compute a composite checksum from per-part checksums following the AWS S3 spec: concatenate raw per-part checksums, hash with the same algorithm, format as "base64-N" where N is part count. The composite checksum is persisted on the final object entry and returned in HEAD/GET responses (gated on x-amz-checksum-mode: ENABLED). Reuses existing per-part checksum storage from putToFiler and the getCheckSumWriter/checksumHeaders infrastructure. * fix(s3): validate checksum algorithm in CreateMultipartUpload, error on missing part checksums - Move detectRequestedChecksumAlgorithm call before mkdir callback so an unsupported algorithm returns 400 before the upload is created - Change computeCompositeChecksum to return an error when a part is missing its checksum (the upload was initiated with a checksum algorithm, so all parts must have checksums) - Propagate the error as ErrInvalidPart in CompleteMultipartUpload * fix(s3): return checksum header in CompleteMultipartUpload response, validate per-part algorithm - Add ChecksumHeaderName/ChecksumValue fields to CompleteMultipartUploadResult and set the x-amz-checksum-* HTTP response header in the handler, matching the AWS S3 CompleteMultipartUpload response spec - Validate that each part's stored checksum algorithm matches the upload's expected algorithm before assembling the composite checksum; return an error if a part was uploaded with a different algorithm |
||
|
Chris Lu
|
f7c27cc81f | go fmt | ||
|
Chris Lu
|
b57429ef2e |
Switch empty-folder cleanup to bucket policy (#8292)
* Fix Spark _temporary cleanup and add issue #8285 regression test * Generalize empty folder cleanup for Spark temp artifacts * Revert synchronous folder pruning and add cleanup diagnostics * Add actionable empty-folder cleanup diagnostics * Fix Spark temp marker cleanup in async folder cleaner * Fix Spark temp cleanup with implicit directory markers * Keep explicit directory markers non-implicit * logging * more logs * Switch empty-folder cleanup to bucket policy * Seaweed-X-Amz-Allow-Empty-Folders * less logs * go vet * less logs * refactoring |
||
|
Chris Lu
|
8880f9932f |
filer: auto clean empty implicit s3 folders (#8051)
* filer: auto clean empty s3 implicit folders Explicitly tag implicitly created S3 folders (parent directories from object uploads) with 'Seaweed-X-Amz-Implicit-Dir'. Update EmptyFolderCleaner to check for this attribute and cache the result efficiently. * filer: correctly handle nil attributes in empty folder cleaner cache * filer: refine implicit tagging logic Prevent tagging buckets as implicit directories. Reduce code duplication. * filer: safeguard GetEntryAttributes against nil entry and not found error * filer: move ErrNotFound handling to EmptyFolderCleaner * filer: add comment to explain level > 3 check for implicit directories |
||
|
promalertandChris Lu
|
9012069bd7 |
chore: execute goimports to format the code (#7983)
* chore: execute goimports to format the code Signed-off-by: promalert <promalert@outlook.com> * goimports -w . --------- Signed-off-by: promalert <promalert@outlook.com> Co-authored-by: Chris Lu <chris.lu@gmail.com> |
||
|
Chris Lu
|
bccef78082 |
fix: reduce N+1 queries in S3 versioned object list operations (#7814)
* fix: achieve single-scan efficiency for S3 versioned object listing When listing objects in a versioning-enabled bucket, the original code triggered multiple getEntry calls per versioned object (up to 12 with retries), causing excessive 'find' operations visible in Grafana and leading to high memory usage. This fix achieves single-scan efficiency by caching list metadata (size, ETag, mtime, owner) directly in the .versions directory: 1. Add new Extended keys for caching list metadata in .versions dir 2. Update upload/copy/multipart paths to cache metadata when creating versions 3. Update getLatestVersionEntryFromDirectoryEntry to use cached metadata (zero getEntry calls when cache is available) 4. Update updateLatestVersionAfterDeletion to maintain cache consistency Performance improvement for N versioned objects: - Before: N×1 to N×12 find operations per list request - After: 0 extra find operations (all metadata from single scan) This matches the efficiency of normal (non-versioned) object listing. * Update s3api_object_versioning.go * s3api: fix ETag handling for versioned objects and simplify delete marker creation - Add Md5 attribute to synthetic logicalEntry for single-part uploads to ensure filer.ETag() returns correct value in ListObjects response - Simplify delete marker creation by initializing entry directly in mkFile callback - Add bytes and encoding/hex imports for ETag parsing * s3api: preserve default attributes in delete marker mkFile callback Only modify Mtime field instead of replacing the entire Attributes struct, preserving default values like Crtime, FileMode, Uid, and Gid that mkFile initializes. * s3api: fix ETag handling in newListEntry for multipart uploads Prioritize ExtETagKey from Extended attributes before falling back to filer.ETag(). This properly handles multipart upload ETags (format: md5-parts) for versioned objects, where the synthetic entry has cached ETag metadata but no chunks to calculate from. * s3api: reduce code duplication in delete marker creation Extract deleteMarkerExtended map to be reused in both mkFile callback and deleteMarkerEntry construction. * test: add multipart upload versioning tests for ETag verification Add tests to verify that multipart uploaded objects in versioned buckets have correct ETags when listed: - TestMultipartUploadVersioningListETag: Basic multipart upload with 2 parts - TestMultipartUploadMultipleVersionsListETag: Multiple multipart versions - TestMixedSingleAndMultipartVersionsListETag: Mix of single-part and multipart These tests cover a bug where synthetic entries for versioned objects didn't include proper ETag handling for multipart uploads. * test: add delete marker test for multipart uploaded versioned objects TestMultipartUploadDeleteMarkerListBehavior verifies: - Delete marker creation hides object from ListObjectsV2 - ListObjectVersions shows both version and delete marker - Version ETag (multipart format) is preserved after delete marker - Object can be accessed by version ID after delete marker - Removing delete marker restores object visibility * refactor: address code review feedback - test: use assert.ElementsMatch for ETag verification (more idiomatic) - s3api: optimize newListEntry ETag logic (check ExtETagKey first) - s3api: fix edge case in ETag parsing (>= 2 instead of > 2) * s3api: prevent stale cached metadata and preserve existing extended attrs - setCachedListMetadata: clear old cached keys before setting new values to prevent stale data when new version lacks certain fields (e.g., owner) - createDeleteMarker: merge extended attributes instead of overwriting to preserve any existing metadata on the entry * s3api: extract clearCachedVersionMetadata to reduce code duplication - clearCachedVersionMetadata: clears only metadata fields (size, mtime, etag, owner, deleteMarker) - clearCachedListMetadata: now reuses clearCachedVersionMetadata + clears ID/filename - setCachedListMetadata: uses clearCachedVersionMetadata (not clearCachedListMetadata because caller has already set ID/filename) * s3api: share timestamp between version entry and cache entry Capture versionMtime once before mkFile and reuse for both: - versionEntry.Attributes.Mtime in the mkFile callback - versionEntryForCache.Attributes.Mtime for list caching This keeps list vs. HEAD LastModified timestamps aligned. * s3api: remove amzAccountId variable shadowing in multipart upload Extract amzAccountId before mkFile callback and reuse in both places, similar to how versionMtime is handled. Avoids confusion from redeclaring the same variable. |
||
|
Konstantin Lebedevandchrislu
|
084b377f87 |
do delete expired entries on s3 list request (#7426)
* do delete expired entries on s3 list request
https://github.com/seaweedfs/seaweedfs/issues/6837
* disable delete expires s3 entry in filer
* pass opt allowDeleteObjectsByTTL to all servers
* delete on get and head
* add lifecycle expiration s3 tests
* fix opt allowDeleteObjectsByTTL for server
* fix test lifecycle expiration
* fix IsExpired
* fix locationPrefix for updateEntriesTTL
* fix s3tests
* resolv coderabbitai
* GetS3ExpireTime on filer
* go mod
* clear TtlSeconds for volume
* move s3 delete expired entry to filer
* filer delete meta and data
* del unusing func removeExpiredObject
* test s3 put
* test s3 put multipart
* allowDeleteObjectsByTTL by default
* fix pipline tests
* rm dublicate SeaweedFSExpiresS3
* revert expiration tests
* fix updateTTL
* rm log
* resolv comment
* fix delete version object
* fix S3Versioning
* fix delete on FindEntry
* fix delete chunks
* fix sqlite not support concurrent writes/reads
* move deletion out of listing transaction; delete entries and empty folders
* Revert "fix sqlite not support concurrent writes/reads"
This reverts commit
|
||
|
|
c6a22ce43a |
Fix get object lock configuration handler (#6996)
* fix GetObjectLockConfigurationHandler * cache and use bucket object lock config * subscribe to bucket configuration changes * increase bucket config cache TTL * refactor * Update weed/s3api/s3api_server.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * avoid duplidated work * rename variable * Update s3api_object_handlers_put.go * fix routing * admin ui and api handler are consistent now * use fields instead of xml * fix test * address comments * Update weed/s3api/s3api_object_handlers_put.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update test/s3/retention/s3_retention_test.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/object_lock_utils.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * change error style * errorf --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Chris LuandCopilot
|
dde1cf63c2 |
S3 Object Lock: ensure x-amz-bucket-object-lock-enabled header (#6990)
* ensure x-amz-bucket-object-lock-enabled header * fix tests * combine 2 metadata changes into one * address comments * Update s3api_bucket_handlers.go * Update weed/s3api/s3api_bucket_handlers.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update test/s3/retention/object_lock_reproduce_test.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update test/s3/retention/object_lock_validation_test.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update test/s3/retention/s3_bucket_object_lock_test.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_bucket_handlers.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_bucket_handlers.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update test/s3/retention/s3_bucket_object_lock_test.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_bucket_handlers.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * package name --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Chris Lu
|
7cb1ca1308 | Add policy engine (#6970) | ||
|
Chris LuandCopilot
|
1549ee2e15 |
implement PubObjectRetention and WORM (#6969)
* implement PubObjectRetention and WORM
* Update s3_worm_integration_test.go
* avoid previous buckets
* Update s3-versioning-tests.yml
* address comments
* address comments
* rename to ExtObjectLockModeKey
* only checkObjectLockPermissions if versioningEnabled
* address comments
* comments
* Revert "comments"
This reverts commit
|
||
|
Chris LuandCopilot
|
cf5a24983a |
S3: add object versioning (#6945)
* add object versioning * add missing file * Update weed/s3api/s3api_object_versioning.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_object_versioning.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_object_versioning.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * ListObjectVersionsResult is better to show multiple version entries * fix test * Update weed/s3api/s3api_object_handlers_put.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_object_versioning.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * multiple improvements * move PutBucketVersioningHandler into weed/s3api/s3api_bucket_handlers.go file * duplicated code for reading bucket config, versioningEnabled, etc. try to use functions * opportunity to cache bucket config * error handling if bucket is not found * in case bucket is not found * fix build * add object versioning tests * remove non-existent tests * add tests * add versioning tests * skip a new test * ensure .versions directory exists before saving info into it * fix creating version entry * logging on creating version directory * Update s3api_object_versioning_test.go * retry and wait for directory creation * revert add more logging * Update s3api_object_versioning.go * more debug messages * clean up logs, and touch directory correctly * log the .versions creation and then parent directory listing * use mkFile instead of touch touch is for update * clean up data * add versioning test in go * change location * if modified, latest version is moved to .versions directory, and create a new latest version Core versioning functionality: WORKING TestVersioningBasicWorkflow - PASS TestVersioningDeleteMarkers - PASS TestVersioningMultipleVersionsSameObject - PASS TestVersioningDeleteAndRecreate - PASS TestVersioningListWithPagination - PASS ❌ Some advanced features still failing: ETag calculation issues (using mtime instead of proper MD5) Specific version retrieval (EOF error) Version deletion (internal errors) Concurrent operations (race conditions) * calculate multi chunk md5 Test Results - All Passing: ✅ TestBucketListReturnDataVersioning - PASS ✅ TestVersioningCreateObjectsInOrder - PASS ✅ TestVersioningBasicWorkflow - PASS ✅ TestVersioningMultipleVersionsSameObject - PASS ✅ TestVersioningDeleteMarkers - PASS * dedupe * fix TestVersioningErrorCases * fix eof error of reading old versions * get specific version also check current version * enable integration tests for versioning * trigger action to work for now * Fix GitHub Actions S3 versioning tests workflow - Fix syntax error (incorrect indentation) - Update directory paths from weed/s3api/versioning_tests/ to test/s3/versioning/ - Add push trigger for add-object-versioning branch to enable CI during development - Update artifact paths to match correct directory structure * Improve CI robustness for S3 versioning tests Makefile improvements: - Increase server startup timeout from 30s to 90s for CI environments - Add progressive timeout reporting (logs at 30s, full logs at 90s) - Better error handling with server logs on failure - Add server PID tracking for debugging - Improved test failure reporting GitHub Actions workflow improvements: - Increase job timeouts to account for CI environment delays - Add system information logging (memory, disk space) - Add detailed failure reporting with server logs - Add process and network diagnostics on failure - Better error messaging and log collection These changes should resolve the 'Server failed to start within 30 seconds' issue that was causing the CI tests to fail. * adjust testing volume size * Update Makefile * Update Makefile * Update Makefile * Update Makefile * Update s3-versioning-tests.yml * Update s3api_object_versioning.go * Update Makefile * do not clean up * log received version id * more logs * printout response * print out list version response * use tmp files when put versioned object * change to versions folder layout * Delete weed-test.log * test with mixed versioned and unversioned objects * remove versionDirCache * remove unused functions * remove unused function * remove fallback checking * minor --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
LHHDZ
|
d21e2f523d |
split ExtAcpKey to ExtAmzOwnerKey and ExtAmzAclKey to avoid unn… (#3824)
split `ExtAcpKey` to `ExtAmzOwnerKey` and `ExtAmzAclKey` to avoid unnecessary `json.Unmarshal()` call Signed-off-by: changlin.shi <changlin.shi@ly.com> Signed-off-by: changlin.shi <changlin.shi@ly.com> |
||
|
LHHDZ
|
3de1e19780 | s3: sync bucket info from filer (#3759) |