Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cache on successful file processing #3614

Merged

Conversation

yguedidi
Copy link
Contributor

@yguedidi yguedidi commented Apr 13, 2023

New version of #3604
Closes rectorphp/rector#7770
Now a file will be cached only when not in dry run mode

needs first:


if ($errorAndFileDiffs[Bridge::SYSTEM_ERRORS] !== []) {
$this->changedFilesDetector->invalidateFile($file->getFilePath());
} elseif (! $configuration->isDryRun()) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here the added condition

@@ -128,6 +130,12 @@ public function processFiles(array $files, Configuration $configuration): array
$systemErrorsAndFileDiffs = $this->arrayParametersMerger->merge($systemErrorsAndFileDiffs, $result);
}

if ($systemErrorsAndFileDiffs[Bridge::SYSTEM_ERRORS] !== []) {
$this->changedFilesDetector->invalidateFile($file->getFilePath());
} elseif (! $configuration->isDryRun()) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here the added condition

@samsonasik
Copy link
Member

I think this one need e2e test to avoid regression and back and forth check, eg:

  • multiple --dry-run check should ok.
  • multiple succesfull keep changing, eg, the following code got twice changed should ok.
if ($a && $b) {
    return true;
}

on config:

<?php

declare(strict_types=1);

use Rector\CodingStyle\Rector\Stmt\NewlineAfterStatementRector;
use Rector\Config\RectorConfig;
use Rector\EarlyReturn\Rector\If_\ChangeAndIfToEarlyReturnRector;

return static function (RectorConfig $rectorConfig): void {
    $rectorConfig->rule(ChangeAndIfToEarlyReturnRector::class);
    $rectorConfig->rule(NewlineAfterStatementRector::class);
};

First: https://getrector.com/demo/55b308a8-af17-48cb-b2be-9c0d26ab60c4
Second: https://getrector.com/demo/53cd71fb-1851-44d0-aa1e-ba7e5329db77

@yguedidi
Copy link
Contributor Author

Thanks for guidance! Will work on separate PRs this week-end to add those e2e tests

@yguedidi yguedidi marked this pull request as draft April 14, 2023 22:17
@yguedidi yguedidi force-pushed the cache-on-successful-file-processing branch 2 times, most recently from f8f8588 to ceac622 Compare April 15, 2023 00:04
@staabm
Copy link
Contributor

staabm commented Apr 22, 2023

Is this still WIP ?

@samsonasik
Copy link
Member

It needs one more e2e test for consecutive succesfull.

for consecutive dry-run, there was already PR #3616

@yguedidi
Copy link
Contributor Author

I planned to work on that e2e test this week-end

@yguedidi
Copy link
Contributor Author

@samsonasik @staabm here it is: #3666

@@ -32,7 +32,7 @@ public function testHasFileChanged(): void
$filePath = __DIR__ . '/Source/file.php';
Copy link
Contributor

@staabm staabm Apr 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are these test-changes required?

existing tests shouldn't be changed with new PRs

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

because previous logic is splitted in 2 methods now, I renamed one so maybe it's clearer

@yguedidi yguedidi force-pushed the cache-on-successful-file-processing branch from 092b701 to 58e34f2 Compare April 28, 2023 09:14
@staabm
Copy link
Contributor

staabm commented Apr 28, 2023

I think the PR itself needs a new test-case reproducing the issue it is trying to fix

@yguedidi yguedidi force-pushed the cache-on-successful-file-processing branch 2 times, most recently from c323a5e to a6d89fc Compare April 28, 2023 09:43
@yguedidi
Copy link
Contributor Author

@staabm good point! I'll try to come with a test in following days, but the final code is ready if you want to have a look, see that all tests pass now, even the consecutive runs test.
@samsonasik Idea is to cache files only when Rector didn't changed them, so when they are in their final state. So following runs will still go through previously changed files to attempt to change them again.
Maybe not the best way, but at least it preserve actual Rector vision on multiple runs and makes the cache reliable

Credits to @stof for the idea! Thank you

@yguedidi yguedidi force-pushed the cache-on-successful-file-processing branch from a6d89fc to 7635179 Compare May 1, 2023 11:44
@samsonasik
Copy link
Member

Please rename commit message with double quote "" , that will cause failure build downgrade

@yguedidi yguedidi force-pushed the cache-on-successful-file-processing branch from 7635179 to 2e10efa Compare May 1, 2023 12:01
@yguedidi
Copy link
Contributor Author

yguedidi commented May 1, 2023

@samsonasik that revert commit is a temporary one just to generate a failing build, to show the issue.
will remove it from the final PR, showing that the PR solves the issue

@yguedidi yguedidi force-pushed the cache-on-successful-file-processing branch from 2e10efa to 489b03b Compare May 1, 2023 12:04
@yguedidi
Copy link
Contributor Author

yguedidi commented May 1, 2023

Here the failing test without the fix: https://github.com/rectorphp/rector-src/actions/runs/4850942011/jobs/8644319975?pr=3614
See that with the fix by caching at the end instead of caching at the beginning, that previously failing test is now green: https://github.com/rectorphp/rector-src/actions/runs/4850961546/jobs/8644357227?pr=3614

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we expect the timeout error to stay even with cache, and now the timouted files are not cached because we cache on success instead of caching at the beginning of the processing


return static function (RectorConfig $rectorConfig): void {
$rectorConfig->cacheClass(FileCacheStorage::class);
$rectorConfig->parallel(1);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

used 1 second timeout to maximize chances of timeout

__DIR__ . '/src',
]);

$rectorConfig->sets([LevelSetList::UP_TO_PHP_82]);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

used a big rule set to maximize chances of timeout

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

used a big file from src to maximize chances of timeout

@yguedidi yguedidi marked this pull request as ready for review May 1, 2023 12:11
@yguedidi
Copy link
Contributor Author

yguedidi commented May 1, 2023

@TomasVotruba @samsonasik @staabm this is now ready for review!

@yguedidi yguedidi force-pushed the cache-on-successful-file-processing branch from 489b03b to b7bc6c4 Compare May 1, 2023 12:17
@yguedidi yguedidi force-pushed the cache-on-successful-file-processing branch from b7bc6c4 to 53cfe73 Compare May 1, 2023 12:30
@yguedidi yguedidi force-pushed the cache-on-successful-file-processing branch from 53cfe73 to cba16d3 Compare May 1, 2023 12:40
@yguedidi yguedidi requested a review from samsonasik May 1, 2023 12:41
@yguedidi yguedidi force-pushed the cache-on-successful-file-processing branch from cba16d3 to cc2811b Compare May 1, 2023 13:05
Copy link
Member

@samsonasik samsonasik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good for me 👍

@TomasVotruba
Copy link
Member

Thank you 👍

@TomasVotruba TomasVotruba merged commit f6972de into rectorphp:main May 1, 2023
40 checks passed
@yguedidi yguedidi deleted the cache-on-successful-file-processing branch May 1, 2023 21:55
@staabm
Copy link
Contributor

staabm commented May 5, 2023

I think we need the same fix in https://github.com/easy-coding-standard/easy-coding-standard. @yguedidi could you check that?

@yguedidi
Copy link
Contributor Author

yguedidi commented May 5, 2023

@staabm I'm not using ECS, so I can't guarantee that I can come with a PR any time soon there sorry.
It's fine if some else inspire from this PR to implement it in ECS :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants