cmd/geth: add prune history command #31384
Conversation
|
This is basically done now. One thing we could still add is a recovery path when tx index pruning fails or when it is interrupted. Right now, if indexing is interrupted, the index is in an invalid state, but this state is not detectable. We could fix this by adding a field to the database that marks the tx index as invalid. On next startup, we would find this field, and set the index tail to the current head block. |
| @@ -361,3 +363,50 @@ func UnindexTransactions(db ethdb.Database, from uint64, to uint64, interrupt ch | |||
| func unindexTransactionsForTesting(db ethdb.Database, from uint64, to uint64, interrupt chan struct{}, hook func(uint64) bool) { | |||
| unindexTransactions(db, from, to, interrupt, hook, false) | |||
| } | |||
|
|
|||
| // PruneTransactionIndex removes all tx index entries below a certain block number. | |||
| func PruneTransactionIndex(db ethdb.Database, pruneBlock uint64) { | |||
There was a problem hiding this comment.
Any particular reason to unindex the transactions by traversing the entire tx lookups in db? Why not reuse the UnindexTransaction with block range?
e.g.
// PruneTransactionIndex removes all tx index entries below a certain block number.
func PruneTransactionIndex(db ethdb.Database, pruneBlock uint64) {
tail := ReadTxIndexTail(db)
if tail == nil || *tail > pruneBlock {
return // no index, or index ends above pruneBlock
}
UnindexTransactions(db, *tail, pruneBlock, nil, true)
}
There was a problem hiding this comment.
We decided to implement it like this because unindexing everything takes 45 minutes, while this scan takes only 1.5min (on Sepolia).
|
One missing part is that: txindexer has no knowledge about the chain cutoff now, and it will reindex the missing part by default. (e.g. with txlookupLimit = 0) It could be problematic as block bodies decoder will fail with an empty blob. |
|
@rjl493456442 on that note can you please check my PR? :) |
Working on it. It's actually pretty complicated |
| log.Error("Tx index pruning iterator error", "err", iter.Error()) | ||
| } | ||
|
|
||
| WriteTxIndexTail(db, pruneBlock) |
There was a problem hiding this comment.
It's dangerous to delete the transaction indexes first and then update the tail.
As it can lead to a weird situation that corresponding transaction index is not found while the tail indicates it should exist.
Anyway, I think we can reuse the one here https://github.com/s1na/go-ethereum/blob/txlookup-cutoff/core/rawdb/accessors_indexes.go#L106
|
The whole idea is pretty straightforward. Let's rebase it on top of txindexer PR, and get it merged |
In #31384 we unindex TXes prior to the merge block. However when the node starts up it will try to re-index those back if the config is to index the whole chain. This change makes the indexer aware of the history cutoff block, avoiding reindexing in that segment. --------- Co-authored-by: Gary Rong <garyrong0905@gmail.com> Co-authored-by: Felix Lange <fjl@twurst.com>
Implements #31135