Exit code is 0 even on verification failure

[omerlh]

Please fill in the fields below to submit an issue or feature request. The
more information that is provided, the better.

Description of issue or feature request:

I'm testing in toto right now and noticed that the exit code is not set correctly on failure:

in-toto-verify --layout intoto-poc/scripts/root.layout --layout-key intoto-poc/alice.pub
Run command '['git', 'clone', '<>']' differs from expected command '['git', 'clone', **'<>']'**

echo $?
0

Current behavior:
Exit code is 0

Expected behavior:
Exit code should be non zero

[adityasaky]

Hi @omerlh, thanks for opening the issue. Currently, when the command that is run differs from the expected command in the layout, it merely shows a warning--this is not a failure of verification. You can check if verification passed successfully in this instance regardless of commands matching using the -v flag, for verbosity.

When verification does fail, the exit code should be non-zero.

[adityasaky]

saky ~/.../in-toto-demo/final_product > in-toto-verify -l root.layout -k alice.pub
(in-toto-verify) RuleVerificationError: 'DISALLOW *' matched the following artifacts: ['demo-project/foo.py']
Full trace for 'expected_materials' of item 'package':
Available materials (used for queue):
['demo-project/foo.py']
Available products:
['demo-project.tar.gz']
Queue after 'MATCH demo-project/* WITH PRODUCTS FROM update-version':
['demo-project/foo.py']

saky ~/.../in-toto-demo/final_product > echo $?
1

This is from in-toto/demo after the supply chain has been tampered with.

[trishankatdatadog]

I guess the documentation should clarify this 🙂

[adityasaky]

It's part of the spec (emphasis mine):

Finally, the "expected_command" field contains a string, COMMAND, describing the suggested command to run. It is important to mention that, in a case where a functionary’s key is compromised, this field can easily be forged (e.g., by changing the PATH environment variable in a host) and thus it should not be trusted for security checks. In addition, commands may be executed with different flags at the behest of the functionary's personal preference (e.g., a developer may run a command with --color=full or not). Because of this, during verification, clients should only show a warning if the expected command field does not match its counterpart in the link metadata. This warning may help auditors check whether something was out of the norm, but will not make verification fail, as these changes are not necessarily a problem.

I'm following up to see if it's also part of the documentation for the reference implementation. 😄

[adityasaky]

I see a reference to it here:https://in-toto.readthedocs.io/en/latest/layout-creation-example.html?highlight=expected%20command#layout-creation-example

But I agree, it needs to be clearer, and part of the docs for verification workflow. There, it says "soft-verify" which isn't very clear. I'll submit a fix.

[omerlh]

@adityasaky You comment makes a lot of sense to me, but I am still not sold on this. The expected_products only defines what files will be created, but there is no guarantee those files has not tampered with. I am thinking about the first step - which start the chain - usually clone command. If you don't verify the command, how do you trust it?

[adityasaky]

If I'm not mistaken, in that instance you'd ideally have in-toto metadata capturing the repository state that you're cloning and/or other solutions to secure the repository itself.

cc-ing @SantiagoTorres to chime in with his thoughts. I also wonder if it's worth moving this discussion to either a different thread or channel since it's not strictly a reference implementation discussion anymore.