s3: keep transient object-entry errors out of ErrObjectNotFound (#12083)

* s3: keep transient object-entry errors out of ErrObjectNotFound

getObjectEntry wrapped every fetch error as ErrObjectNotFound, so a
transient filer failure (busy filer, ctx deadline, transport blip) was
indistinguishable from a real miss. LifecycleDelete resolves
ErrObjectNotFound permanently as NOOP_RESOLVED, so one flaky lookup at
dispatch time silently skipped the object's expiration - errors=0, no
retry, no log. This matches the flaky s3tests
test_lifecyclev2_expiration signature: exactly one object of a
same-rule pair surviving past its window.

Only collapse genuine misses (filer not-found, version/latest-version
sentinels); everything else propagates so the worker retries it as
RETRY_LATER instead of resolving it.

Also seed bucketConfigCache in the retention test server and teach the
fake filer to inject per-path lookup errors.

* s3: keep invalid version ids on the not-found path in getObjectEntry

A malformed versionId used to collapse into ErrObjectNotFound like every
other fetch error, so the handlers answered NoSuchKey. With transient
errors now propagated, an invalid id would surface as InternalError;
fold it back into the miss classification.

* filer_pb: keep Unavailable out of LookupEntry's not-found normalization

* s3: reject Unavailable in isObjectEntryMiss before message matching

* s3: keep transient fallback errors out of latest-version resolution

* filer_pb: limit LookupEntry's not-found text match to miss statuses

* s3: limit isFilerNotFound text matching to miss statuses

weed/s3api/s3api_object_retention.go

@@ -13,6 +13,7 @@ import (
 	"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
 	"github.com/seaweedfs/seaweedfs/weed/s3api/s3err"
 	"github.com/seaweedfs/seaweedfs/weed/util"
+	"google.golang.org/grpc/codes"
 )
 
 // ====================================================================
@@ -208,12 +209,35 @@ func (s3a *S3ApiServer) getObjectEntry(bucket, object, versionId string) (*filer
 	}
 
 	if err != nil {
-		return nil, fmt.Errorf("failed to retrieve object %s/%s: %w", bucket, object, ErrObjectNotFound)
+		// Only a genuine miss becomes ErrObjectNotFound. Transport and
+		// internal failures must stay distinguishable: callers like
+		// LifecycleDelete resolve a miss permanently, so collapsing a
+		// flaky read into "not found" silently drops the object.
+		if isObjectEntryMiss(err) {
+			return nil, fmt.Errorf("failed to retrieve object %s/%s: %w", bucket, object, ErrObjectNotFound)
+		}
+		return nil, fmt.Errorf("failed to retrieve object %s/%s: %w", bucket, object, err)
 	}
 
 	return entry, nil
 }
 
+// isObjectEntryMiss reports whether an object-entry fetch error is a genuine
+// miss (no object, version, or latest version) rather than a transient or
+// internal failure the caller may want to retry.
+func isObjectEntryMiss(err error) bool {
+	// Unavailable is a failed read, not a miss - a wrapped not-found
+	// message in its status text must not classify it as one.
+	if st, ok := util.ServerStatus(err); ok && st.Code() == codes.Unavailable {
+		return false
+	}
+	return isFilerNotFound(err) ||
+		errors.Is(err, ErrObjectNotFound) ||
+		errors.Is(err, ErrVersionNotFound) ||
+		errors.Is(err, ErrLatestVersionNotFound) ||
+		errors.Is(err, errInvalidVersionID)
+}
+
 // getObjectEntryForUpdate resolves the entry a metadata update targets and the
 // filer path it is written under. It applies the same version resolution as
 // getSpecificObjectVersion - "null" names the plain pre-versioning entry, not a