Bugfix/pickle errors #629
Merged
Bugfix/pickle errors #629
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This fixes an issue where the PapermillExecutionError can be pickled but will not be unpicklable. Pythons BaseException class has a handling to make sure that exceptions are able to be loaded once they are pickled. This works by ensuring the args used to generate the instance are saved on the exception. However by only passing the message to the superclass, the pickle module was unable to load a new instance of PapermillExecutionError from a file. A new __str__ method was added to preserve the functionality of calling str() on the exception and only getting the error message back, while ensuring the superclass had access to all instantiating arguments.
MSeal
approved these changes
Aug 31, 2021
Codecov Report
@@ Coverage Diff @@
## main #629 +/- ##
==========================================
+ Coverage 91.63% 91.65% +0.01%
==========================================
Files 17 17
Lines 1555 1558 +3
==========================================
+ Hits 1425 1428 +3
Misses 130 130
Continue to review full report at Codecov.
|
|
Thanks for the improvement @Calder-Ty ! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds the ability to load PapermillExecutionErrors from pickle files.
Airflow 2.0.0 changed the way it handles error reporting by temporarily pickling the exception. Prior to this commit, papermill exceptions could be pickled by airflow, but then not loaded back, causing cryptic error messages.
This pull request addresses this issue, by modifying the call to the superclass, and sending up all the arguments used to create the exception. It also implements a
__str__method, to maintain compatibility with the behavior of callingstr()on the exception returning the formatted message.Fixes #628