s3: a key deleted after enabling versioning must leave the listing (#10684)

* s3: a null object wins over a rescan when the latest-version pointer is absent

The read path already resolves an absent pointer this way; the listing-path
counterpart scanned .versions/ first and could surface an old version or
delete marker over the current suspended-versioning null object.

* s3: dedup a key against its .versions sibling in suspended buckets too

A suspended bucket keeps its .versions directories, so a suspended-versioning
null object and its .versions sibling emitted the same key twice.

* s3: retract a null object from the listing when a delete marker shadows it

Deleting a key whose null version predates versioning leaves the base-path
entry in place and records the delete marker under <key>.versions. The
listing appended the base-path entry and relied on the .versions sibling to
replace it, but a delete-marker current version emitted nothing, so the
deleted key stayed visible to ListObjects while GET and HEAD returned 404.

* s3: keep a key's .versions sibling on the same page as the key

When the page quota ran out between a base-path entry and its .versions
directory, the page ended with the stale entry and the next page skipped the
directory as a marker echo, so the replacement or retraction never happened.

* s3: the null version is not latest when the .versions pointer names a newer one

ListObjectVersions stamped IsLatest on every base-path null object, so a key
deleted after enabling versioning reported IsLatest on both the delete marker
and the null version.

* s3: test listing after a pre-versioning null object is delete-marked

* s3: find a key's earlier page entry by scan, not by adjacency

A key such as k.bak sorts between k and k.versions, so the entry a .versions
sibling replaces or retracts is not always the last one on the page. Scan
back through the page for the key, and insert a late resolution in sorted
position instead of at the end.

* s3: settle trailing null objects by lookup when a page fills

The quota can run out while keys still sit between a null object and its
.versions sibling, and the sibling-adjacent page-boundary exception never
fires for those. Track the trailing null objects whose sibling has not been
ruled out and look each one up before declaring the page full; a retraction
reopens the quota.

* s3: do not resolve a .versions sibling its page has already moved past

A page resuming from a marker inside the base key's extension region has
already listed and settled the base null object on an earlier page, so
resolving the .versions directory again re-emitted the key.

* s3: test listing with keys between a null object and its .versions sibling

* s3: pick the newer of the null object and the scanned versions

Making the null object win outright whenever the pointer is absent misread
multi-filer pointer lag: version files replicate ahead of the pointer, and a
key overwritten or delete-marked after pre-versioning days would list its
stale null again. The suspended-versioning write that legitimately makes the
null current is also the newer entry, so mtime tells the two apart.

* s3: a delete-marked null object no longer keeps its prefix alive

The hidden-entries probe took any plain file as proof of a listable key, but
a null object shadowed by its .versions sibling's delete marker is not one.
Hold plain files pending until the sibling settles them either way.

* s3: settle an evicted pending null instead of dropping it

Nested keys like k, k!, k!! can hold more pending nulls than the cap. A
silently evicted one could close the page unsettled, and the resume skip
would then keep the stale entry for good.

* s3: test deleted-prefix hiding and the pending-null cap

* s3: cover the reported '!' intervening key with a live version

* s3: an unstamped same-second version outranks the null object

Second-resolution mtimes cannot order same-second writes, so the tie went to
the stale null when the pointer lagged. The suspended write that makes a null
current stamps the version it displaces before clearing the pointer, so the
stamp is the authoritative signal and a tie without it goes to the version.

* s3: a pointer-less versions listing still checks what replicated

ListObjectVersions took a missing pointer as proof the null object is latest,
but under pointer lag the sibling can already hold newer replicated versions
or markers. Apply the same nullObjectWins rule as the listing recovery.

* s3: a failed null-object settlement fails the listing

Every getEntry error read as a missing sibling, so a transient filer error at
a page boundary committed the unsettled null and the next page skipped its
sibling for good. Only a definitive not-found means the null is live; other
failures are retained on eviction and fail the request at page close.

* s3: retract a CommonPrefix whose only backers were delete-marked nulls

The directory probe settles this for the / delimiter, but any other delimiter
derives prefixes from base-path keys directly, and a prefix built solely from
null objects survived their delete markers. Count the unsettled null backers
behind the newest prefix and retract it when the last one settles as a marker;
a live resolution or any listable contributor confirms the prefix instead.

* s3: test custom-delimiter prefix retraction

* s3: an explicit signal marks the null object current, not the demotion stamp

The NoncurrentSinceNs stamp survives promotion: delete the version that
demoted another and the promoted one is current yet still stamped, so a
lagging replica would resurrect the stale null. A suspended-versioning write
now records Seaweed-X-Amz-Null-Version-Is-Latest on the .versions directory
when it clears the pointer, every pointer update removes it, and the
recovery paths trust the signal instead of the stamp.

* s3: a filer failover retry rebuilds the listing page from scratch

The failover wrapper reruns the callback on another filer after a transport
error, and the partially built page, spent quota, and advanced marker leaked
into the retry, which could then return a stale or duplicated page as
success.

* s3: only a prefix's own backers can debit it

A delete marker for a version-only key (no base object) derived the same
prefix as its neighbors and decremented backing it never contributed,
retracting a prefix that a live null object still backed. Track backers by
key so settlement is idempotent and only debits what was counted.

* s3: test a version-only marker against a null-backed prefix

* s3: a pointer recompute clears the null-current signal

The routed finalize for delete markers, COPY, and multipart rewrites the
.versions pointer through RECOMPUTE_LATEST, which left a suspended-era
null-current signal in place. Version files never carry the signal, so
mapping it in CopyExtended deletes it whenever the pointer recomputes.

* s3: the pointer outranks the null-current signal in the versions listing

The signal check guarded the pointer check, so a stale signal a recompute
had not cleared yet would have let the null claim IsLatest alongside the
pointed-at version.

weed/s3api/s3api_object_versioning.go

@@ -708,7 +708,7 @@ func (vc *versionCollector) processRegularFile(currentPath, entryPath string, en
 
 	// Check if a .versions directory exists for this object
 	versionsEntryName := entry.Name + s3_constants.VersionsFolder
-	_, versionsErr := vc.s3a.getEntry(currentPath, versionsEntryName)
+	versionsDirEntry, versionsErr := vc.s3a.getEntry(currentPath, versionsEntryName)
 	if versionsErr == nil && !hasVersionMeta {
 		// .versions exists but file has no version metadata - check for null version in .versions
 		versions, err := vc.s3a.getObjectVersionList(vc.bucket, normalizedObjectKey)
@@ -731,10 +731,26 @@ func (vc *versionCollector) processRegularFile(currentPath, entryPath string, en
 	}
 	vc.seenVersionIds[versionKey] = true
 
+	// A latest-version pointer on the .versions sibling names the current version
+	// and outranks a stale null-current signal a recompute may not have cleared
+	// yet. With no pointer, the explicit signal decides; with neither, the
+	// sibling may still hold replicated versions the lagging pointer has not
+	// caught up with, and the nullObjectWins rule decides.
+	isLatest := true
+	if versionsErr == nil {
+		if len(versionsDirEntry.Extended[s3_constants.ExtLatestVersionIdKey]) > 0 {
+			isLatest = false
+		} else if !nullVersionIsLatest(versionsDirEntry) {
+			if latestVersion, _, _, _, scanErr := vc.s3a.scanLatestVersionEntry(currentPath + "/" + versionsEntryName); scanErr == nil && latestVersion != nil && !nullObjectWins(entry, latestVersion) {
+				isLatest = false
+			}
+		}
+	}
+
 	versionEntry := &VersionEntry{
 		Key:          normalizedObjectKey,
 		VersionId:    "null",
-		IsLatest:     true,
+		IsLatest:     isLatest,
 		LastModified: time.Unix(entry.Attributes.Mtime, 0),
 		ETag:         vc.s3a.calculateETagFromChunks(entry.Chunks),
 		Size:         int64(entry.Attributes.FileSize),
@@ -2029,7 +2045,7 @@ func (s3a *S3ApiServer) getLatestVersionEntryFromDirectoryEntry(bucket, object s
 	// here). Indexing a nil Extended map is safe and yields !ok.
 	latestVersionIdBytes, hasLatestVersionId := versionsDirEntry.Extended[s3_constants.ExtLatestVersionIdKey]
 	if !hasLatestVersionId {
-		return s3a.recoverLatestListEntryByScan(bucket, normalizedObject)
+		return s3a.recoverLatestListEntryByScan(bucket, normalizedObject, nullVersionIsLatest(versionsDirEntry))
 	}
 
 	// Check if this is a delete marker (should not be shown in regular list)
@@ -2095,7 +2111,7 @@ func (s3a *S3ApiServer) getLatestVersionEntryFromDirectoryEntry(bucket, object s
 	// Fallback: fetch version file if cached metadata not available (for older versions)
 	latestVersionFileBytes, hasLatestVersionFile := versionsDirEntry.Extended[s3_constants.ExtLatestVersionFileNameKey]
 	if !hasLatestVersionFile {
-		return s3a.recoverLatestListEntryByScan(bucket, normalizedObject)
+		return s3a.recoverLatestListEntryByScan(bucket, normalizedObject, nullVersionIsLatest(versionsDirEntry))
 	}
 	latestVersionFile := string(latestVersionFileBytes)
 
@@ -2128,6 +2144,26 @@ func (s3a *S3ApiServer) getLatestVersionEntryFromDirectoryEntry(bucket, object s
 	return logicalEntry, nil
 }
 
+// nullObjectWins decides, with no latest-version pointer and no null-is-latest
+// signal to consult, whether the base-path null object or the newest scanned
+// version is the current version: the newer mtime wins, and a tie goes to the
+// version, since second-resolution mtimes cannot order same-second writes and
+// an intentional null carries the ExtNullVersionIsLatestKey signal. The
+// NoncurrentSinceNs demotion stamp is deliberately not consulted: promotions
+// do not clear it, so it does not prove the null displaced the version.
+func nullObjectWins(regular, latest *filer_pb.Entry) bool {
+	if latest == nil {
+		return true
+	}
+	return regular.GetAttributes().GetMtime() > latest.GetAttributes().GetMtime()
+}
+
+// nullVersionIsLatest reports the explicit signal a suspended-versioning write
+// leaves on the .versions directory when the null object is current.
+func nullVersionIsLatest(versionsDirEntry *filer_pb.Entry) bool {
+	return versionsDirEntry != nil && string(versionsDirEntry.Extended[s3_constants.ExtNullVersionIsLatestKey]) == "true"
+}
+
 // recoverLatestListEntryByScan rebuilds an object's current-version list entry by
 // rescanning .versions/ when the cached latest-version pointer is missing on the
 // filer serving the list. This is the listing-path counterpart to the read path's
@@ -2139,20 +2175,27 @@ func (s3a *S3ApiServer) getLatestVersionEntryFromDirectoryEntry(bucket, object s
 // a write per diverged object); convergence is handled on the write/replication
 // side. Returns ErrDeleteMarker when the current version is a delete marker
 // (excluded from a regular listing) and filer_pb.ErrNotFound when nothing remains.
-func (s3a *S3ApiServer) recoverLatestListEntryByScan(bucket, normalizedObject string) (*filer_pb.Entry, error) {
+func (s3a *S3ApiServer) recoverLatestListEntryByScan(bucket, normalizedObject string, nullIsLatest bool) (*filer_pb.Entry, error) {
 	bucketDir := s3a.bucketDir(bucket)
 	versionsDir := bucketDir + "/" + normalizedObject + s3_constants.VersionsFolder
 
+	// An absent pointer can mean a suspended-versioning write made the null
+	// object current (the write clears the pointer and leaves the explicit
+	// nullIsLatest signal), or that the pointer has not replicated to this
+	// filer while the version files have.
+	regularEntry, regularErr := s3a.getEntry(bucketDir, normalizedObject)
+	if regularErr == nil && nullIsLatest {
+		return regularEntry, nil
+	}
+
 	latestEntry, latestVersionId, _, isDeleteMarker, err := s3a.scanLatestVersionEntry(versionsDir)
 	if err != nil {
 		return nil, err
 	}
+	if regularErr == nil && nullObjectWins(regularEntry, latestEntry) {
+		return regularEntry, nil
+	}
 	if latestEntry == nil {
-		// No version files remain. A pre-versioning / suspended "null" object at the
-		// base path is the current version if one exists.
-		if regularEntry, regularErr := s3a.getEntry(bucketDir, normalizedObject); regularErr == nil {
-			return regularEntry, nil
-		}
 		return nil, fmt.Errorf("%w: no current version for %s/%s", filer_pb.ErrNotFound, bucket, normalizedObject)
 	}
 	if isDeleteMarker {