* filer: recover leveldb without rebuilding broken tables
leveldb.Recover rebuilds a table with damaged blocks using the user
comparer and filter on internal keys. The shortened index keys it writes
are not valid internal keys, so a Get on the rebuilt table panics, and
the bloom filter it writes misses every lookup. Recover with a comparer
that keeps full index keys (same name, so the manifest still matches)
and no filter.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: recover a leveldb store when a compaction finds a corrupted table
When a compaction reads a damaged table block, as a power loss can
leave one, goleveldb latches the corruption error and returns it for
every later write until the database is reopened. A plain reopen loads
the same table and the next compaction fails again, so the store kept
rejecting inserts even across restarts.
Treat a corruption error from a write like a storage error, and reopen
through leveldb.Recover, which rebuilds the table without the damaged
blocks, then retry the write.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: rebuild damaged leveldb tables before Recover
leveldb.Recover's own table rebuild orders internal keys with the user
comparer, so a damaged table holding several versions of one key fails
with "keys are not in increasing order" and the recovery gives up.
Overriding the comparer cannot fix that, because the same comparer
also serves the recovered database's user keys.
Rebuild every table that has corrupted blocks first, using an internal
key comparer and full index keys, then let Recover rebuild the manifest
from tables it no longer needs to rewrite. Seed the test tables from a
journal flush so they keep several versions of a key, and check keys
the write test does not rewrite.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: leave unreadable leveldb tables for Recover to drop
table.NewReader reports a truncated table or a damaged footer through
the reader, not its error, so rebuildDamagedTables never stopped on one,
but it rewrote it as an empty table first. Skip tables with nothing
readable and let Recover drop them, and test an unreadable table next
to a healthy one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: let StrictRecovery drop damaged leveldb tables
With StrictRecovery set, leveldb.Recover drops a table that has corrupted
blocks. Salvaging the readable blocks first would hide the damage from it,
so skip the rebuild under that option.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* filer: hold the leveldb storage lock while rebuilding tables
The table rebuild ran after the failed open released the storage lock
and before Recover took it, so it could rewrite the tables of a database
another caller had open on the same storage. Take the lock for the
rebuild; Recover takes it again afterwards.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>