Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Merged by Bors] - hare3 implementation #4765

Closed
wants to merge 70 commits into from
Closed
Show file tree
Hide file tree
Changes from 62 commits
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
de3d42a
squash hare3
dshulyak Aug 2, 2023
ca90c2e
integrate into codebase
dshulyak Aug 2, 2023
42538d3
validation
dshulyak Aug 2, 2023
c575e1d
linter
dshulyak Aug 2, 2023
8ea1178
progress
dshulyak Aug 3, 2023
2ec5578
add more references
dshulyak Aug 3, 2023
d85de6d
add fuzzer for decode
dshulyak Aug 5, 2023
0f4e364
cleanup and more tests
dshulyak Aug 5, 2023
68ff558
add sufficient metrics for debug
dshulyak Aug 5, 2023
a41d87c
cleanup
dshulyak Aug 5, 2023
d12a2ae
Merge branch 'develop' into hare3-fresh
dshulyak Aug 5, 2023
6710ae7
linter
dshulyak Aug 5, 2023
8ebdd81
move message key back
dshulyak Aug 5, 2023
1fd9531
limit iterations in test
dshulyak Aug 5, 2023
cbd9373
remove redundant checks
dshulyak Aug 5, 2023
3ffa3df
adjust to fit 2 iterations into one layer
dshulyak Aug 5, 2023
7a3a95b
remove redundant grade
dshulyak Aug 6, 2023
323f7e5
move malfe creation to types
dshulyak Aug 8, 2023
4e08da8
better synchronization faster integration test
dshulyak Aug 8, 2023
0b7bd19
test handler
dshulyak Aug 8, 2023
f888a9e
proposals unit tests
dshulyak Aug 8, 2023
ee6f4c2
agree on empty layer
dshulyak Aug 9, 2023
e5d795e
Merge branch 'develop' into hare3-fresh
dshulyak Aug 9, 2023
ba8db06
linter
dshulyak Aug 9, 2023
9783cb1
remove custom panics
dshulyak Aug 9, 2023
df575ea
typo
dshulyak Aug 10, 2023
e7b700a
change receive to invoke
dshulyak Aug 10, 2023
300974d
drop struct for valid values
dshulyak Aug 10, 2023
4417679
improve reference
dshulyak Aug 10, 2023
f9669b0
Single -> Absolute
dshulyak Aug 10, 2023
c6ba677
fix iteration limit
dshulyak Aug 10, 2023
46beb86
changes after call
dshulyak Aug 10, 2023
ce19cf8
Merge branch 'develop' into hare3-fresh
dshulyak Aug 22, 2023
d4679d0
run test with passive
dshulyak Aug 22, 2023
0fe991a
use mustencode
dshulyak Aug 22, 2023
6453986
rename instance to session
dshulyak Aug 23, 2023
1d198a1
add a test for softlock
dshulyak Aug 23, 2023
8e20b8e
refactor test setup
dshulyak Aug 24, 2023
c83b5ba
setup equivocators
dshulyak Aug 24, 2023
26ce13e
add randomization to proposals selection and weight
dshulyak Aug 24, 2023
d2ea761
Merge branch 'develop' into hare3-fresh
dshulyak Aug 24, 2023
378bdab
linter
dshulyak Aug 24, 2023
3545ccd
change short string for hash20
dshulyak Aug 27, 2023
84e9565
foudn a reason why i skipped hardlock
dshulyak Aug 27, 2023
ab4056c
reuse same state for all protocolls
dshulyak Aug 27, 2023
13e21b4
change round 6 to notify
dshulyak Aug 27, 2023
f39b6bf
misc review fixes
dshulyak Aug 27, 2023
f2664ad
assume proposals are called by active nodes
dshulyak Aug 27, 2023
4092350
better enabling
dshulyak Aug 27, 2023
929ab8c
add disabled layer
dshulyak Aug 27, 2023
62a3d80
fix flakiness due to nodeclock
dshulyak Aug 28, 2023
0ec655e
Merge branch 'develop' into hare3-fresh
dshulyak Aug 28, 2023
302713a
switch to grade5
dshulyak Aug 28, 2023
cb1b6c8
add patrol and other changes
dshulyak Aug 29, 2023
064c514
misc
dshulyak Aug 29, 2023
e756b8e
process messages without channel
dshulyak Aug 29, 2023
badafc3
deflake equivocation test
dshulyak Aug 29, 2023
afdc292
add patrol to node
dshulyak Aug 29, 2023
77c3767
better latency in histograms
dshulyak Aug 29, 2023
655fd7b
race on startup
dshulyak Aug 29, 2023
ca1b0b2
didn't change grade5 in test protocol for commit
dshulyak Aug 29, 2023
4fea19d
use generic for two threshold gossip function
dshulyak Aug 29, 2023
0b77921
switch to count from min atx grade
dshulyak Aug 29, 2023
5d669d2
add parallel execution
dshulyak Aug 30, 2023
a389f66
Merge branch 'develop' into hare3-fresh
dshulyak Aug 30, 2023
40cc565
set disabled layer in config
dshulyak Aug 30, 2023
ef7ed9a
Merge branch 'develop' into hare3-fresh
dshulyak Aug 30, 2023
7618036
run with debug logs in system tests
dshulyak Aug 30, 2023
666f447
add a line about hare3
dshulyak Aug 30, 2023
2c86042
remove debug log
dshulyak Aug 30, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
4 changes: 4 additions & 0 deletions cmd/base.go
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,10 @@ func EnsureCLIFlags(cmd *cobra.Command, appCFG *config.Config) error {
elem = reflect.ValueOf(&appCFG.HareEligibility).Elem()
assignFields(ff, elem, name)

ff = reflect.TypeOf(appCFG.HARE3)
elem = reflect.ValueOf(&appCFG.HARE3).Elem()
assignFields(ff, elem, name)

ff = reflect.TypeOf(appCFG.Beacon)
elem = reflect.ValueOf(&appCFG.Beacon).Elem()
assignFields(ff, elem, name)
Expand Down
7 changes: 7 additions & 0 deletions codec/codec.go
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,13 @@
encoderPool.Put(b)
}

func MustEncodeTo(w io.Writer, value Encodable) {
_, err := EncodeTo(w, value)
if err != nil {
panic(err)

Check warning on line 55 in codec/codec.go

View check run for this annotation

Codecov / codecov/patch

codec/codec.go#L55

Added line #L55 was not covered by tests
}
}

func MustEncode(value Encodable) []byte {
buf, err := Encode(value)
if err != nil {
Expand Down
2 changes: 1 addition & 1 deletion common/types/hashes.go
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ func (h Hash20) String() string {
// ShortString returns a the first 5 characters of the hash, for logging purposes.
func (h Hash20) ShortString() string {
l := len(h.Hex())
return h.Hex()[util.Min(2, l):util.Min(7, l)]
return Shorten(h.Hex()[util.Min(2, l):], 10)
}

// Format implements fmt.Formatter, forcing the byte slice to be formatted as is,
Expand Down
24 changes: 19 additions & 5 deletions common/types/malfeasance.go
Original file line number Diff line number Diff line change
Expand Up @@ -182,6 +182,16 @@
return nil
}

func (hp *HareProof) ToMalfeasenceProof() *MalfeasanceProof {
return &MalfeasanceProof{
Layer: hp.Messages[0].InnerMsg.Layer,
Proof: Proof{
Type: HareEquivocation,
Data: hp,
},
}
}

type AtxProofMsg struct {
InnerMsg ATXMetadata

Expand Down Expand Up @@ -235,6 +245,14 @@
return hm.Layer == other.Layer && hm.Round == other.Round && hm.MsgHash != other.MsgHash
}

func (hm HareMetadata) ToBytes() []byte {
buf, err := codec.Encode(&hm)
if err != nil {
panic(err.Error())

Check warning on line 251 in common/types/malfeasance.go

View check run for this annotation

Codecov / codecov/patch

common/types/malfeasance.go#L251

Added line #L251 was not covered by tests
}
return buf
}

type HareProofMsg struct {
InnerMsg HareMetadata

Expand All @@ -244,11 +262,7 @@

// SignedBytes returns the actual data being signed in a HareProofMsg.
func (m *HareProofMsg) SignedBytes() []byte {
data, err := codec.Encode(&m.InnerMsg)
if err != nil {
log.With().Fatal("failed to serialize MultiBlockProposalsMsg", log.Err(err))
}
return data
return m.InnerMsg.ToBytes()
}

func MalfeasanceInfo(smesher NodeID, mp *MalfeasanceProof) string {
Expand Down
3 changes: 3 additions & 0 deletions config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ import (
vm "github.com/spacemeshos/go-spacemesh/genvm"
hareConfig "github.com/spacemeshos/go-spacemesh/hare/config"
eligConfig "github.com/spacemeshos/go-spacemesh/hare/eligibility/config"
"github.com/spacemeshos/go-spacemesh/hare3"
"github.com/spacemeshos/go-spacemesh/p2p"
"github.com/spacemeshos/go-spacemesh/syncer"
timeConfig "github.com/spacemeshos/go-spacemesh/timesync/config"
Expand Down Expand Up @@ -50,6 +51,7 @@ type Config struct {
P2P p2p.Config `mapstructure:"p2p"`
API grpcserver.Config `mapstructure:"api"`
HARE hareConfig.Config `mapstructure:"hare"`
HARE3 hare3.Config `mapstructure:"hare3"`
HareEligibility eligConfig.Config `mapstructure:"hare-eligibility"`
Beacon beacon.Config `mapstructure:"beacon"`
TIME timeConfig.TimeConfig `mapstructure:"time"`
Expand Down Expand Up @@ -140,6 +142,7 @@ func DefaultConfig() Config {
P2P: p2p.DefaultConfig(),
API: grpcserver.DefaultConfig(),
HARE: hareConfig.DefaultConfig(),
HARE3: hare3.DefaultConfig(),
HareEligibility: eligConfig.DefaultConfig(),
Beacon: beacon.DefaultConfig(),
TIME: timeConfig.DefaultConfig(),
Expand Down
8 changes: 8 additions & 0 deletions config/presets/fastnet.go
Original file line number Diff line number Diff line change
Expand Up @@ -24,13 +24,21 @@ func fastnet() config.Config {

conf.BaseConfig.TestConfig.MinerGoodAtxPct = 50

conf.HARE.Disable = 1 // non-zero low layer will prevent hare1 from running
conf.HARE.N = 800
conf.HARE.ExpectedLeaders = 10
conf.HARE.LimitConcurrent = 5
conf.HARE.LimitIterations = 3
conf.HARE.RoundDuration = 2 * time.Second
conf.HARE.WakeupDelta = 3 * time.Second

conf.HARE3.Enable = true
conf.HARE3.Committee = 800
conf.HARE3.Leaders = 10
conf.HARE3.PreroundDelay = 3 * time.Second
conf.HARE3.RoundDuration = 500 * time.Millisecond
conf.HARE3.IterationsLimit = 2

conf.P2P.MinPeers = 10

conf.Genesis = &config.GenesisConfig{
Expand Down
9 changes: 8 additions & 1 deletion hare/config/config.go
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
package config

import "time"
import (
"time"

"github.com/spacemeshos/go-spacemesh/common/types"
)

// Config is the configuration of the Hare.
type Config struct {
Expand All @@ -13,6 +17,9 @@ type Config struct {
StopAtxGrading uint32 `mapstructure:"stop-atx-grading"`

Hdist uint32

// If set to non-zero value will disable hare starting at that layer.
Disable types.LayerID `mapstructure:"disable"`
}

// DefaultConfig returns the default configuration for the hare.
Expand Down
4 changes: 4 additions & 0 deletions hare/hare.go
Original file line number Diff line number Diff line change
Expand Up @@ -321,6 +321,10 @@
h.With().Debug("hare exiting", log.Context(ctx), lid)
return false, nil
}
if h.config.Disable != 0 && lid >= h.config.Disable {
h.With().Debug("hare is disabled at this layer", log.Context(ctx), lid)
return false, nil
}

Check warning on line 327 in hare/hare.go

View check run for this annotation

Codecov / codecov/patch

hare/hare.go#L325-L327

Added lines #L325 - L327 were not covered by tests

h.setLastLayer(lid)

Expand Down
29 changes: 29 additions & 0 deletions hare3/compat/result.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
package compat

import (
"context"

"go.uber.org/zap"

"github.com/spacemeshos/go-spacemesh/hare"
"github.com/spacemeshos/go-spacemesh/hare3"
)

func ReportResult(ctx context.Context, logger *zap.Logger, from <-chan hare3.ConsensusOutput, to chan<- hare.LayerOutput) {
for {
select {
case <-ctx.Done():
logger.Info("hare3 results reporter exited")
return
case out := <-from:
select {

Check warning on line 19 in hare3/compat/result.go

View check run for this annotation

Codecov / codecov/patch

hare3/compat/result.go#L12-L19

Added lines #L12 - L19 were not covered by tests
case to <- hare.LayerOutput{
Ctx: ctx,
Layer: out.Layer,
Proposals: out.Proposals,
}:
case <-ctx.Done():

Check warning on line 25 in hare3/compat/result.go

View check run for this annotation

Codecov / codecov/patch

hare3/compat/result.go#L24-L25

Added lines #L24 - L25 were not covered by tests
}
}
}
}
31 changes: 31 additions & 0 deletions hare3/compat/weakcoin.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
package compat

import (
"context"

"go.uber.org/zap"

"github.com/spacemeshos/go-spacemesh/common/types"
"github.com/spacemeshos/go-spacemesh/hare3"
)

type weakCoin interface {
Set(types.LayerID, bool) error
}

func ReportWeakcoin(ctx context.Context, logger *zap.Logger, from <-chan hare3.WeakCoinOutput, to weakCoin) {
for {
select {
case <-ctx.Done():
logger.Info("weak coin reporter exited")
return
case out := <-from:
if err := to.Set(out.Layer, out.Coin); err != nil {
logger.Error("failed to update weakcoin",
zap.Uint32("lid", out.Layer.Uint32()),
zap.Error(err),
)
}

Check warning on line 28 in hare3/compat/weakcoin.go

View check run for this annotation

Codecov / codecov/patch

hare3/compat/weakcoin.go#L16-L28

Added lines #L16 - L28 were not covered by tests
}
}
}