Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle backup errors more consistently #133522

Merged
merged 16 commits into from
Jan 2, 2025

Conversation

MartinHjelmare
Copy link
Member

Breaking change

Proposed change

  • I've reviewed the backup manager code for possible errors that can be raised.
  • I'm mostly targeting the create backup API across the project in this PR. Some manager APIs, eg receive backup will be reviewed later and updated to handle exceptions like done in this PR.
  • The backup manager should now always raise BackupManagerError.
  • The BackupReaderWriter should now always raise BackupReaderWriterError.

Type of change

  • Dependency upgrade
  • Bugfix (non-breaking change which fixes an issue)
  • New integration (thank you!)
  • New feature (which adds functionality to an existing integration)
  • Deprecation (breaking change to happen in the future)
  • Breaking change (fix/feature causing existing functionality to break)
  • Code quality improvements to existing code or addition of tests

Additional information

  • This PR fixes or closes issue: fixes #
  • This PR is related to issue:
  • Link to documentation pull request:

Checklist

  • The code change is tested and works locally.
  • Local tests pass. Your PR cannot be merged unless tests pass
  • There is no commented out code in this PR.
  • I have followed the development checklist
  • I have followed the perfect PR recommendations
  • The code has been formatted using Ruff (ruff format homeassistant tests)
  • Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

If the code communicates with devices, web services, or third-party tools:

  • The manifest file has all fields filled out correctly.
    Updated and included derived files by running: python3 -m script.hassfest.
  • New or updated dependencies have been added to requirements_all.txt.
    Updated by running python3 -m script.gen_requirements_all.
  • For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.

To help with the load of incoming pull requests:

Sorry, something went wrong.

@home-assistant
Copy link

Hey there @home-assistant/core, mind taking a look at this pull request as it has been labeled with an integration (backup) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of backup can trigger bot actions by commenting:

  • @home-assistant close Closes the pull request.
  • @home-assistant rename Awesome new title Renames the pull request.
  • @home-assistant reopen Reopen the pull request.
  • @home-assistant unassign backup Removes the current integration label and assignees on the pull request, add the integration domain after the command.
  • @home-assistant add-label needs-more-information Add a label (needs-more-information, problem in dependency, problem in custom component) to the pull request.
  • @home-assistant remove-label needs-more-information Remove a label (needs-more-information, problem in dependency, problem in custom component) on the pull request.

@home-assistant
Copy link

Hey there @home-assistant/supervisor, mind taking a look at this pull request as it has been labeled with an integration (hassio) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of hassio can trigger bot actions by commenting:

  • @home-assistant close Closes the pull request.
  • @home-assistant rename Awesome new title Renames the pull request.
  • @home-assistant reopen Reopen the pull request.
  • @home-assistant unassign hassio Removes the current integration label and assignees on the pull request, add the integration domain after the command.
  • @home-assistant add-label needs-more-information Add a label (needs-more-information, problem in dependency, problem in custom component) to the pull request.
  • @home-assistant remove-label needs-more-information Remove a label (needs-more-information, problem in dependency, problem in custom component) on the pull request.

@home-assistant
Copy link

Hey there @home-assistant/cloud, mind taking a look at this pull request as it has been labeled with an integration (cloud) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of cloud can trigger bot actions by commenting:

  • @home-assistant close Closes the pull request.
  • @home-assistant rename Awesome new title Renames the pull request.
  • @home-assistant reopen Reopen the pull request.
  • @home-assistant unassign cloud Removes the current integration label and assignees on the pull request, add the integration domain after the command.
  • @home-assistant add-label needs-more-information Add a label (needs-more-information, problem in dependency, problem in custom component) to the pull request.
  • @home-assistant remove-label needs-more-information Remove a label (needs-more-information, problem in dependency, problem in custom component) on the pull request.

@@ -485,7 +493,7 @@ async def test_async_initiate_backup_with_agent_error(
assert result["event"] == {
"manager_state": BackupManagerState.CREATE_BACKUP,
"stage": None,
"state": CreateBackupState.COMPLETED,
"state": CreateBackupState.FAILED,
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that the backup will be marked as failed now if there are agent errors.

Copy link
Contributor

@emontnemery emontnemery Dec 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think that's fine. Maybe @piitaya can confirm it works for frontend?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I got confirmation from @bramkragten this doesn't break frontend 👍

@emontnemery emontnemery self-requested a review December 18, 2024 20:32
@MartinHjelmare
Copy link
Member Author

I recommend reviewing with hide whitespace.

@MartinHjelmare MartinHjelmare force-pushed the cloud-backup-add-manager-writer-errors branch 2 times, most recently from 8d71443 to 6153d6f Compare December 20, 2024 15:27
Copy link
Contributor

@emontnemery emontnemery left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good overall, but some comments and questions.

I've not reviewed the tests yet.

if invalid_agents := [
agent_id for agent_id in agent_ids if agent_id not in self.backup_agents
]:
raise BackupManagerError(f"Invalid agents selected: {invalid_agents}")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice improvement of the error logging 👍

self._async_finish_backup(agent_ids, with_automatic_settings),
name="backup_manager_finish_backup",
)
if not raise_task_error:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is confusingly named, by the name of the parameter it seems like it's about raising or not raising when something goes wrong, but it's about logging errors?
Also, why do we need to pass this down all the way here, why don't we just log the error in async_create_backup?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't want to log if we raise. This is about handling exceptions in the backup task when it is not awaited. If we don't await it we need to take care of the exception that may be raised and log it instead.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But we now pass this down through multiple layers, and all it does is log, there's no other difference in how the task is handled as far as I understand?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn't about logging. It's about handling exceptions. If we don't await a task and it raises an exception it will trigger the warning about unhandled exception. When we don't await the task we need to add the task done callback and call task.exception. This will avoid the unhandled exception warning.

Copy link
Contributor

@emontnemery emontnemery Jan 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't agree with this. It should be up to the caller to await tasks and decide if it is or is not interested in errors, there should not be a parameter passed to the called code.

We can merge the PR as is though and improve this later.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is up to the caller to pick one of the methods depending on what the caller wants.

Comment on lines +859 to 866
if with_automatic_settings:
self._update_issue_after_agent_upload(agent_errors)
# delete old backups more numerous than copies
# try this regardless of agent errors above
await delete_backups_exceeding_configured_count(self)
Copy link
Contributor

@emontnemery emontnemery Dec 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we introduce a race by issuing the COMPLETED event before we're really done with all operations?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're not idle, so I think it's ok.

Copy link
Contributor

@emontnemery emontnemery Dec 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, so only after the idle event has been fired, everything is done? I think that's fair.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes until idle the manager is busy. Completed means the backup is complete. Deleting old backups is a separate operation from the backup. So everything makes sense like it is now.

@995eko

This comment was marked as spam.

@MartinHjelmare MartinHjelmare added this to the 2025.1.0b0 milestone Dec 21, 2024
This reverts commit 28fd7a5.
@MartinHjelmare MartinHjelmare force-pushed the cloud-backup-add-manager-writer-errors branch from 6153d6f to 0f53259 Compare December 21, 2024 13:56
@frenck frenck removed this from the 2025.1.0b0 milestone Dec 23, 2024

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
@emontnemery emontnemery merged commit a329828 into dev Jan 2, 2025
64 checks passed
@emontnemery emontnemery deleted the cloud-backup-add-manager-writer-errors branch January 2, 2025 14:45
@emontnemery emontnemery added this to the 2025.1.0 milestone Jan 2, 2025
frenck pushed a commit that referenced this pull request Jan 2, 2025

Verified

This commit was signed with the committer’s verified signature. The key has expired.
frenck Franck Nijhof
* Add backup manager and read writer errors

* Clean up not needed default argument

* Clean up todo comment

* Trap agent bugs during upload

* Always release stream

* Clean up leftover

* Update test for backup with automatic settings

* Fix use of vol.Any

* Refactor test helper

* Only update successful timestamp if completed event is sent

* Always delete surplus copies

* Fix after rebase

* Fix after rebase

* Revert "Fix use of vol.Any"

This reverts commit 28fd7a5.

* Inherit BackupReaderWriterError in IncorrectPasswordError

---------

Co-authored-by: Erik Montnemery <erik@montnemery.com>
@github-actions github-actions bot locked and limited conversation to collaborators Jan 3, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants