Changed propagation of failure termination

[LanderOtto]

This commit changes how StreamFlow propagates the failure termination into the workflow steps.

Before this commit

The steps propagate SKIPPED status because the ports are empties.

New TerminationToken value

We use the 'TerminationToken' value, which was not used, to propagate the 'Step' exit status. This way, the following steps know if they can continue the execution or terminate immediately.

The possible step exit statuses are: CANCELLED, FAILED, COMPLETED and SKIPPED.

The step takes the exit status based on the exit status of all its jobs, combining them.
In general, the job statuses are unified. FAILED and CANCELLED absorbs all the other statuses, instead COMPLETED absorbs SKIPPED.
Some examples:

• [ COMPLETED, COMPLETED, COMPLETED, COMPLETED ] => COMPLETED
• [ COMPLETED, FAILED, COMPLETED, CANCELLED ] => FAILED
• [ CANCELLED, FAILED, COMPLETED, COMPLETED ] => CANCELLED
• [ COMPLETED, SKIPPED, COMPLETED, SKIPPED ] => COMPLETED
• [ SKIPPED, SKIPPED, SKIPPED, SKIPPED ] => SKIPPED

Forced Termination

First, a regression about how StreamFlow works. StreamFlow executes its entites as individual tasks, using the asyncio.Task corountine. So each manager and each Step is a task.
When a step fails, as said before, the TerminationToken is propagated to next steps and they terminate with FAILED status too. However, there are some StreamFlow entities which not receive the TerminationToken.

These entities are cancelled by the StreamFlow executor:

• some Step instances. In the case of the scatter feature, a job of the ExecuteStep can fail, so the other jobs are cancelled. The ExecuteStep propagates the TerminationToken with FAILED value. However, before the ExecuteStep can be other alive Step objects as TransferStep, ScheduleStep. They do not receive the TerminationToken with FAILED value, so they keep their activities until the StreamFlow executor will cancel all the asyncio.Task included the task in charge of these Step runs.
• the Scheduler. While the TerminationToken with FAILED value is propagated, the Scheduler continues its work. Meanwhile, the deployments are undeployed and can happens that the Scheduler was communicating with the location and its communication is interrupted.

[codecov]

Codecov Report

Attention: Patch coverage is 86.11111% with 10 lines in your changes are missing coverage. Please review.

Project coverage is 70.24%. Comparing base (6813bac) to head (ca181da).

Files	Patch %	Lines
streamflow/workflow/step.py	86.95%	4 Missing and 2 partials ⚠️
streamflow/workflow/executor.py	71.42%	1 Missing and 1 partial ⚠️
streamflow/workflow/token.py	80.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #373      +/-   ##
==========================================
+ Coverage   70.12%   70.24%   +0.12%     
==========================================
  Files          84       84              
  Lines       10611    10648      +37     
  Branches     2514     2532      +18     
==========================================
+ Hits         7441     7480      +39     
+ Misses       2714     2707       -7     
- Partials      456      461       +5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.