tortoise, sync: faster requests for unavailable blocks

[dshulyak]

in the implementation block is requested if it crosses local threshold. however in verifying tortoise we will never know if block crosses local threshold, so node has to wait until it falls into full tortoise.

the reason why we don't know is that we count votes only for opinions that match our own, and as we don't have data our opinion definitely will not support such block.

we need a faster way to recognize if block is needed and request a fetch. several options:

• count votes not only for our own opinion but for several distinct opinions, based on the opinion hash in the ballot. so that if we see that one of the opinions starts winning we request blocks that it supports.
• run full mode in parallel with verifying. it will be a bit more expensive, but generally not a problem

[dshulyak]

copying here observation, which is likely the root cause of slow block downloading

i think i found the reason why it takes so long to request blocks sometimes.
there is this code https://github.com/spacemeshos/go-spacemesh/blob/develop/fetch/fetch.go#L486-L490
it gets random peers that registered as hash providers, but it doesn't check if we are connected with that peer.
so for example on my computer i loop with for almost 20 minutes.
peer was connected, then went away but this code thinks that it should use it, which is clearly not ok

2023-08-31T08:41:17.708+0200    WARN    fd68b.fetcher   failed to fetch blocks  {"node_id": "fd68b9397572556c2f329f3e5af2faf23aef85dbbbb7e38447fae2f4ef38899f", "module": "fetcher", "error": "hint: blocksDB, hash: 0xa7fda17e3b073d66aa94e251207faca321e39e7f000000000000000000000000, err: batched request failed w retries: peer is not connected: 12D3KooWCdy5ELnzUoA8kDzG26444DqjSjbNgGMKT3fHPJo7oLmP"}
...
2023-08-31T08:58:01.793+0200	WARN	fd68b.fetcher	failed to fetch blocks	{"node_id": "fd68b9397572556c2f329f3e5af2faf23aef85dbbbb7e38447fae2f4ef38899f", "module": "fetcher", "error": "hint: blocksDB, hash: 0xa7fda17e3b073d66aa94e251207faca321e39e7f000000000000000000000000, err: batched request failed w retries: peer is not connected: 12D3KooWCdy5ELnzUoA8kDzG26444DqjSjbNgGMKT3fHPJo7oLmP"}

[dshulyak]

should've been solved with #4939