volume: recover .idx rows overwritten by tiered deletes (#10474)

* volume: recover .idx rows overwritten by tiered deletes

A delete on a read-only volume backed by a remote tier used to write its
tombstone row at .idx offset 0 rather than appending it, so each delete
overwrote one more row at the front and lost the Put rows indexing the
first needles in .dat. Those needles 404 even though .dat still holds
them, and rebuilding .idx with weed fix means stopping the server and
pulling the whole .dat back from the tier.

The damage has a fingerprint -- .idx opening with a run of offset-0
tombstones, which a healthy .idx never does -- and .idx and .dat grow in
lockstep, so the lost rows indexed exactly the first N .dat records.
Detect it at load and re-derive them from a header-only walk over the
head of .dat, cheap even against a remote tier, appending only the keys
the .idx no longer names.

* rust volume: mirror the .idx head tombstone recovery

Port the Go detection and repair: an .idx opening with a run of offset-0
tombstones lost the Put rows indexing the first needles in .dat, so
re-derive them at load from a header-only walk over the head of .dat and
append the keys the .idx no longer names.

* volume: put recovered .idx rows back in front instead of appending

Appending left the offset-0 tombstone run at the head, so every later
load re-walked .idx to the tail to notice the volume was already
recovered, and the rows for the head of .dat sat past the .dat-tail row
-- costing CheckVolumeDataIntegrity its O(1) path and breaking the
ascending append order BinarySearchByAppendAtNs assumes.

Rewrite .idx as the recovered rows followed by its current contents,
through a temp file and a rename. .idx is back in .dat append order, so
a later load stops after reading one row.

* volume: keep the .idx mode when the repair replaces it

The recovery renames a fresh temp file over .idx, so a fixed 0644 (Go)
or whatever the umask allows (Rust) would silently widen an index an
operator had locked down. Carry the mode off the file being replaced.

weed/storage/volume_loading.go

@@ -252,6 +252,14 @@ func (v *Volume) load(alsoLoadIndex bool, createDatIfMissing bool, needleMapKind
 			}
 			glog.Fatalf("check volume idx file %s: %v", v.FileName(".idx"), err)
 		}
+		// Recover rows that deletes on a tiered read-only volume overwrote at
+		// the front of .idx. Best effort: a volume that cannot be repaired is
+		// still servable for everything the surviving rows index.
+		if restored, repairErr := v.repairIdxHeadTombstones(); repairErr != nil {
+			glog.Warningf("volume %d: recover overwritten %s rows: %v", v.Id, v.FileName(".idx"), repairErr)
+		} else if restored > 0 {
+			glog.V(0).Infof("volume %d: recovered %d overwritten %s rows from %s", v.Id, restored, v.FileName(".idx"), v.FileName(".dat"))
+		}
 		var indexFile *os.File
 		if v.noWriteOrDelete {
 			glog.V(0).Infoln("open to read file", v.FileName(".idx"))