Skip to content
This repository has been archived by the owner on Nov 15, 2023. It is now read-only.

Add benchmarks for for import/validation/production #2579

Merged
merged 25 commits into from
Jun 5, 2023

Conversation

skunert
Copy link
Contributor

@skunert skunert commented May 15, 2023

Closes #2048

This PR adds some benchmarks to work on. The requirements are specified in #2048.
The goal is to capture the current performance of block import/block production/block validation.

Transfers

First, we have a benchmark that fills a block with transfer_keep_alive transactions until the block is full.
This block is then built/imported/validated.

1313 transfer transactions per block.

Triecache Enabled:

Benchmark Time (ms) Throughput (Kelem/s)
Import 164.59 7.9772
Production w/o proof 192.13 6.8339
Production w/ proof 220.21 5.9624
Validation - -

Triecache Disabled:

Benchmark Time (ms) Throughput (Kelem/s)
Import 220.6 5.9638
Production w/o proof 261.81 5.0151
Production w/ proof 270.22 4.8580
Validation 263.20 4.9886

Compute/IO

In addition to the benchmarking of blocks with transfer extrinsics, I used the glutton pallet
to fully use the weight of a block with configurable compute/IO part. The glutton pallet will just "burn" the leftover weight
of the current block in the on_idle hook of the pallet. The weight consumed for PoV size and time
can be chosen using the glutton pallets extrinsics.

Note: Throughput is not available for this benchmark since we add no extrinsics.

Triecache Enabled:

Benchmark compute = 100%, storage = 0% (ms) compute = 100%, storage = 100% (ms)
Import 264.09 259.51
Production w/o proof 271.31 267.88
Production w/ proof 272.64 275.72
Validation - -

Triecache Disabled:

Benchmark compute = 100%, storage = 0% (ms) compute = 100%, storage = 100% (ms)
Import 270.0 274.73
Production w/o proof 267.13 268.32
Production w/ proof 264.57 276.60
Validation 271.59 284.26

Overall, these benchmark results raise some questions. For the benchmarks that use the glutton pallet, I was expecting the PoV size to have a much larger impact
on the execution time.

Edit: Switched out client block import with execute_block for a more fair comparison, results make more sense now. Block import is fastest, followed by block production. Validate block is slowest due to the constant decoding of storage values.

@skunert skunert added B0-silent Changes should not be mentioned in any release notes A0-please_review Pull request needs code review. C1-low PR touches the given topic and has a low impact on builders. D3-trivial 🧸 PR contains trivial changes in a runtime directory that do not require an audit labels May 15, 2023
@skunert skunert requested review from koute, bkchr and a team May 15, 2023 18:11
Copy link
Contributor

@melekes melekes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

test/client/src/lib.rs Show resolved Hide resolved
test/client/src/lib.rs Show resolved Hide resolved
test/client/src/lib.rs Show resolved Hide resolved
test/service/src/chain_spec.rs Show resolved Hide resolved
test/service/src/lib.rs Outdated Show resolved Hide resolved
@skunert skunert requested a review from a team May 23, 2023 08:58
test/service/src/bench_utils.rs Show resolved Hide resolved
test/service/src/bench_utils.rs Outdated Show resolved Hide resolved
test/service/src/bench_utils.rs Outdated Show resolved Hide resolved
test/service/src/bench_utils.rs Outdated Show resolved Hide resolved
test/service/src/chain_spec.rs Outdated Show resolved Hide resolved
test/service/benches/block_import.rs Outdated Show resolved Hide resolved
Copy link
Member

@bkchr bkchr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally looks good, I mainly left some nitpicks.

test/service/benches/block_import_compute.rs Outdated Show resolved Hide resolved
test/service/benches/block_production_compute.rs Outdated Show resolved Hide resolved
test/service/benches/block_production_compute.rs Outdated Show resolved Hide resolved
transaction_validity::{InvalidTransaction, TransactionValidityError},
};

fn create_extrinsics(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why don't you use this from the utils?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah this is a bit of a compromise. I am using the test client for the validate block benchmarks and test service for block production and block import since I wanted to reuse the nice convenience features of the test client block builder.

test/service/src/bench_utils.rs Show resolved Hide resolved
test/service/benches/validate_block_compute.rs Outdated Show resolved Hide resolved
test/service/src/bench_utils.rs Outdated Show resolved Hide resolved
@skunert skunert merged commit 5b0db49 into paritytech:master Jun 5, 2023
31 checks passed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
A0-please_review Pull request needs code review. B0-silent Changes should not be mentioned in any release notes C1-low PR touches the given topic and has a low impact on builders. D3-trivial 🧸 PR contains trivial changes in a runtime directory that do not require an audit
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Benchmark building vs importing vs validating
5 participants