[Merged by Bors] - prune data and compact state #4998

countvonzero · 2023-09-12T02:42:24Z

Motivation

Closes #3049
Closes #3588

Changes

prune data consistently with tortoise hdist distance.
- proposals
- certificates
- proposal id<->tid mapping
certificate is needed for consensus within hdist.
the same distance is used for all for simplicity.
extract active set data from ballots and save them in activesets table.
vacuum and checkpoint database after migration 4 is complete

this PR concludes the first update described #4984 (comment)

codecov · 2023-09-12T02:51:24Z

Codecov Report

Merging #4998 (9bb6af0) into develop (dcd501d) will decrease coverage by 0.1%.
The diff coverage is 63.9%.

@@            Coverage Diff            @@
##           develop   #4998     +/-   ##
=========================================
- Coverage     77.1%   77.1%   -0.1%     
=========================================
  Files          254     257      +3     
  Lines        30113   30242    +129     
=========================================
+ Hits         23241   23318     +77     
- Misses        5363    5398     +35     
- Partials      1509    1526     +17

Files Changed	Coverage Δ
blocks/generator.go	`91.0% <ø> (+0.8%)`	⬆️
blocks/metrics.go	`65.2% <ø> (ø)`
sql/database.go	`70.7% <42.8%> (-3.7%)`	⬇️
sql/vacuum.go	`45.4% <45.4%> (ø)`
prune/prune.go	`48.5% <48.5%> (ø)`
sql/migrations.go	`60.0% <58.3%> (-1.9%)`	⬇️
sql/certificates/certs.go	`75.7% <62.5%> (-0.9%)`	⬇️
sql/ballots/ballots.go	`68.5% <66.6%> (-0.1%)`	⬇️
sql/transactions/transactions.go	`77.2% <66.6%> (-0.4%)`	⬇️
sql/ballots/util/extract.go	`68.5% <68.5%> (ø)`
... and 4 more

... and 4 files with indirect coverage changes

sql/vacuum/vacuum.go

mesh/janitor.go

sql/vacuum/vacuum.go

cmd/root.go

node/node.go

cmd/root.go

countvonzero · 2023-09-14T07:24:59Z

sql/migrations/0004_next.sql

+DELETE FROM proposals WHERE layer < 19000;
+DELETE FROM proposal_transactions WHERE layer < 19000;
+UPDATE certificates SET cert = NULL WHERE layer < 19000;
+UPDATE ballots SET ballot = prune_actives(ballot);


considering #5002 and the fact that right now the ref ballot is still saved with activeset.
ActiveSetFromEpochFirstBlock also still retrieve activeset directly from ref ballot.
is there a safe range i should apply to remove the activeset from ballots?

at layer 17800, the state size changed from 8.2GB to 470MB with this migration. the migration took 1.5 min.

my proposal was to save active sets into activeset table while pruning. and then we either have an option to sql-join them on activesethash field (requires extending ballots table so maybe not optimal) or do another query to get activeset hash when we need it.

#4984 (comment)

we should also stop writing active sets encoded together with ballot, after this migration is done. not necessarily within same change, but would be better to release them together

changed to extract active set from ballots and save the prune blob.

please consider #4984 (comment)
i'll continue with the other code change accordingly.

sql/vacuum/vacuum.go

node/node.go

sql/database.go

sql/migrations/0004_next.sql

dshulyak

please see notes about pruner/extract code

mesh/janitor.go

countvonzero · 2023-09-17T23:01:14Z

bors try

bors · 2023-09-17T23:51:48Z

try

Build succeeded!

The publicly hosted instance of bors-ng is deprecated and will go away soon.

If you want to self-host your own instance, instructions are here.
For more help, visit the forum.

If you want to switch to GitHub's built-in merge queue, visit their help page.

countvonzero · 2023-09-19T00:25:22Z

bors try

bors · 2023-09-19T01:11:32Z

try

Build succeeded!

The publicly hosted instance of bors-ng is deprecated and will go away soon.

If you want to self-host your own instance, instructions are here.
For more help, visit the forum.

If you want to switch to GitHub's built-in merge queue, visit their help page.

countvonzero · 2023-09-19T01:28:28Z

bors merge

## Motivation Closes #3049 Closes #3588 ## Changes - prune data consistently with tortoise hdist distance. - proposals - certificates - proposal id<->tid mapping certificate is needed for consensus within hdist. the same distance is used for all for simplicity. - extract active set data from ballots and save them in activesets table. - vacuum and checkpoint database after migration 4 is complete this PR concludes the first update described #4984 (comment)

bors · 2023-09-19T01:52:43Z

Build failed:

ci-status

countvonzero · 2023-09-19T02:23:46Z

bors merge

## Motivation Closes #3049 Closes #3588 ## Changes - prune data consistently with tortoise hdist distance. - proposals - certificates - proposal id<->tid mapping certificate is needed for consensus within hdist. the same distance is used for all for simplicity. - extract active set data from ballots and save them in activesets table. - vacuum and checkpoint database after migration 4 is complete this PR concludes the first update described #4984 (comment)

bors · 2023-09-19T03:32:12Z

Build failed:

systest-status

countvonzero · 2023-09-19T03:32:21Z

bors merge

## Motivation Closes #3049 Closes #3588 ## Changes - prune data consistently with tortoise hdist distance. - proposals - certificates - proposal id<->tid mapping certificate is needed for consensus within hdist. the same distance is used for all for simplicity. - extract active set data from ballots and save them in activesets table. - vacuum and checkpoint database after migration 4 is complete this PR concludes the first update described #4984 (comment)

bors · 2023-09-19T03:45:32Z

Build failed:

ci-status

countvonzero · 2023-09-19T04:23:58Z

bors merge

## Motivation Closes #3049 Closes #3588 ## Changes - prune data consistently with tortoise hdist distance. - proposals - certificates - proposal id<->tid mapping certificate is needed for consensus within hdist. the same distance is used for all for simplicity. - extract active set data from ballots and save them in activesets table. - vacuum and checkpoint database after migration 4 is complete this PR concludes the first update described #4984 (comment)

bors · 2023-09-19T04:39:00Z

Build failed:

ci-status

countvonzero · 2023-09-19T04:47:18Z

bors merge

## Motivation Closes #3049 Closes #3588 ## Changes - prune data consistently with tortoise hdist distance. - proposals - certificates - proposal id<->tid mapping certificate is needed for consensus within hdist. the same distance is used for all for simplicity. - extract active set data from ballots and save them in activesets table. - vacuum and checkpoint database after migration 4 is complete this PR concludes the first update described #4984 (comment)

bors · 2023-09-19T05:33:43Z

Pull request successfully merged into develop.

Build succeeded!

The publicly hosted instance of bors-ng is deprecated and will go away soon.

If you want to self-host your own instance, instructions are here.
For more help, visit the forum.

If you want to switch to GitHub's built-in merge queue, visit their help page.

countvonzero changed the title ~~prune data and vacuum~~ prune data and compact state Sep 12, 2023

countvonzero marked this pull request as ready for review September 13, 2023 03:16

countvonzero requested review from dshulyak, fasmat and poszu as code owners September 13, 2023 03:16

dshulyak reviewed Sep 13, 2023

View reviewed changes

sql/vacuum/vacuum.go Outdated Show resolved Hide resolved

dshulyak reviewed Sep 13, 2023

View reviewed changes

mesh/janitor.go Outdated Show resolved Hide resolved

mesh/janitor.go Outdated Show resolved Hide resolved

sql/vacuum/vacuum.go Outdated Show resolved Hide resolved

cmd/root.go Show resolved Hide resolved

node/node.go Outdated Show resolved Hide resolved

countvonzero commented Sep 14, 2023

View reviewed changes

sql/database.go Show resolved Hide resolved

dshulyak reviewed Sep 14, 2023

View reviewed changes

sql/migrations/0004_next.sql Outdated Show resolved Hide resolved

dshulyak approved these changes Sep 15, 2023

View reviewed changes

mesh/janitor.go Outdated Show resolved Hide resolved

mesh/janitor.go Outdated Show resolved Hide resolved

countvonzero force-pushed the prune-for-real branch from 5031a73 to 5a20c04 Compare September 17, 2023 22:33

bors bot added a commit that referenced this pull request Sep 17, 2023

Try #4998:

5e7ca7c

countvonzero added 7 commits September 18, 2023 16:42

prune data and vacuum

b04cda0

set mainnet config

e09bee2

use free page count as vacuum criteria

7f6d153

code review feedback

1be2038

add db migrations

98b24de

extract active set and save

6f21bfc

code review feedback and delay pruning of activeset to next update

35b94e0

countvonzero force-pushed the prune-for-real branch from 5a20c04 to 35b94e0 Compare September 18, 2023 23:47

bors bot added a commit that referenced this pull request Sep 19, 2023

Try #4998:

46a6402

update changelog

ffe37de

small cleanup

9bb6af0

bors bot changed the title ~~prune data and compact state~~ [Merged by Bors] - prune data and compact state Sep 19, 2023

bors bot closed this Sep 19, 2023

bors bot deleted the prune-for-real branch September 19, 2023 05:33

countvonzero mentioned this pull request Sep 19, 2023

[Merged by Bors] - prune active set from ballot blob #5032

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Merged by Bors] - prune data and compact state #4998

[Merged by Bors] - prune data and compact state #4998

countvonzero commented Sep 12, 2023 •

edited

codecov bot commented Sep 12, 2023 •

edited

countvonzero Sep 14, 2023 •

edited

dshulyak Sep 14, 2023 •

edited

dshulyak Sep 14, 2023 •

edited

countvonzero Sep 15, 2023

dshulyak left a comment

countvonzero commented Sep 17, 2023

bors bot commented Sep 17, 2023

countvonzero commented Sep 19, 2023

bors bot commented Sep 19, 2023

countvonzero commented Sep 19, 2023

bors bot commented Sep 19, 2023

countvonzero commented Sep 19, 2023

bors bot commented Sep 19, 2023

countvonzero commented Sep 19, 2023

bors bot commented Sep 19, 2023

countvonzero commented Sep 19, 2023

bors bot commented Sep 19, 2023

countvonzero commented Sep 19, 2023

bors bot commented Sep 19, 2023

[Merged by Bors] - prune data and compact state #4998

[Merged by Bors] - prune data and compact state #4998

Conversation

countvonzero commented Sep 12, 2023 • edited

Motivation

Changes

codecov bot commented Sep 12, 2023 • edited

Codecov Report

countvonzero Sep 14, 2023 • edited

Choose a reason for hiding this comment

dshulyak Sep 14, 2023 • edited

Choose a reason for hiding this comment

dshulyak Sep 14, 2023 • edited

Choose a reason for hiding this comment

countvonzero Sep 15, 2023

Choose a reason for hiding this comment

dshulyak left a comment

Choose a reason for hiding this comment

countvonzero commented Sep 17, 2023

bors bot commented Sep 17, 2023

try

countvonzero commented Sep 19, 2023

bors bot commented Sep 19, 2023

try

countvonzero commented Sep 19, 2023

bors bot commented Sep 19, 2023

countvonzero commented Sep 19, 2023

bors bot commented Sep 19, 2023

countvonzero commented Sep 19, 2023

bors bot commented Sep 19, 2023

countvonzero commented Sep 19, 2023

bors bot commented Sep 19, 2023

countvonzero commented Sep 19, 2023

bors bot commented Sep 19, 2023

countvonzero commented Sep 12, 2023 •

edited

codecov bot commented Sep 12, 2023 •

edited

countvonzero Sep 14, 2023 •

edited

dshulyak Sep 14, 2023 •

edited

dshulyak Sep 14, 2023 •

edited