invalid-publisher: valid token, but no corresponding publisher

[eifinger]

Hi,
I have a release workflow which suddenly stopped working:

• First run worked
• Second run failed because I tried to upload the same version number by accident
• Third run for a new version number shows invalid-publisher: valid token, but no corresponding publisher

Aside the version numbers I did not change anything.
I also tried to remove the trusted publisher and add it again and rerun the action but still no success.

What could be the reason that this worked the first time but not since then?

[webknjaz]

Sounds like a warehouse problem... @di @woodruffw could you look into this?

[webknjaz]

I tried to upload the same version number by accident

This wouldn't be related.

[webknjaz]

I also tried to remove the trusted publisher and add it again and rerun the action but still no success.

Was this right before the third run? Maybe, you made a typo somewhere and repo or env doesn't match?

[di]

Looks like your publisher is configured with an environment:

warehouse=> select * from github_oidc_publishers where repository_name = 'pywaze';
 repository_name | repository_owner | repository_owner_id | workflow_filename |                  id                  | environment
-----------------+------------------+---------------------+-------------------+--------------------------------------+-------------
 pywaze          | eifinger         | 1481961             | release.yml       | b5d2c695-2a19-417a-93ea-99ef38686566 | pypi
(1 row)

but I don't see that environment listed anywhere here: https://github.com/eifinger/pywaze/blob/main/.github/workflows/release.yml

I would expect to see something like environment: pypi listed in the workflow as documented here: https://docs.pypi.org/trusted-publishers/using-a-publisher/

Unfortunately I can't see how your original publisher was configured, did it have an environment set?

[eifinger]

I also tried to remove the trusted publisher and add it again and rerun the action but still no success.

Was this right before the third run? Maybe, you made a typo somewhere and repo or env doesn't match?

The third run has 3 attempts. I readded the publisher between the 2nd and 3rd attempt

[eifinger]

Looks like your publisher is configured with an environment:

warehouse=> select * from github_oidc_publishers where repository_name = 'pywaze';
 repository_name | repository_owner | repository_owner_id | workflow_filename |                  id                  | environment
-----------------+------------------+---------------------+-------------------+--------------------------------------+-------------
 pywaze          | eifinger         | 1481961             | release.yml       | b5d2c695-2a19-417a-93ea-99ef38686566 | pypi
(1 row)

but I don't see that environment listed anywhere here: https://github.com/eifinger/pywaze/blob/main/.github/workflows/release.yml

I would expect to see something like environment: pypi listed in the workflow as documented here: https://docs.pypi.org/trusted-publishers/using-a-publisher/

Unfortunately I can't see how your original publisher was configured, did it have an environment set?

I had some issues with GitHub in the last weeks showing different commits on some branches in different sessions. For me the environment is visible on line 13: https://github.com/eifinger/pywaze/blob/main/.github/workflows/release.yml#L13

[woodruffw]

Hmm, I can see the environment configured on the tag's changes as well:

https://github.com/eifinger/pywaze/blob/7a468b8d646d4ae0a07c13582e64d3bd5e0234e5/.github/workflows/release.yml

It's possible this was some kind of weird hiccup on GitHub's side, given the inconsistent results that people are seeing for the workflow's contents. Could you try running your release workflow again?

[di]

Ah, sorry, looks like I just can't read 😂

[eifinger]

Hmm, I can see the environment configured on the tag's changes as well:

https://github.com/eifinger/pywaze/blob/7a468b8d646d4ae0a07c13582e64d3bd5e0234e5/.github/workflows/release.yml

It's possible this was some kind of weird hiccup on GitHub's side, given the inconsistent results that people are seeing for the workflow's contents. Could you try running your release workflow again?

Attempt 4 failed as well https://github.com/eifinger/pywaze/actions/runs/5794194719/job/15724456812

[woodruffw]

Dang, thanks for trying.

Everything looks like to me here, and it's strange that it worked the first time but not the others. Could you share a screencap of the security event for the successful publish? It should be 1-2 events in the "Security History" page for the project that look something like this:

[eifinger]

Thank you for the fast and nice responses btw!

[woodruffw]

Thanks! Just for our reference, here's the workflow from that successful publish:

https://github.com/eifinger/pywaze/blob/bc8f7470f10691351cbe47a46ab71995b82e9cfb/.github/workflows/release.yml

This is pretty strange -- I don't see anything salient between that successful publish and the workflow that's failing 😕

[di]

The only thing that changed is that this was likely converted from a pending publisher to a regular publisher, so something doesn't agree in the publisher queries/verification between the two.

I'm working on adding some better error messages here (there are a lot of failure modes that would result in this one error message) which should at least help narrow this down.

[di]

@woodruffw The changes in question are here: pypi/warehouse#14308.

@eifinger Sorry you're having trouble here. Could I kindly ask that you don't change anything with your setup here for the time being? We should be able to ship this change shortly after which we can re-run your workflow to get some additional details on why this is failing.

[eifinger]

Doing nothing? Yeah I think I can handle that 😄

[di]

@eifinger Ok, these changes should be live now, can you attempt to re-run one of these failing workflow attempts and report back? It should still fail, but will have a different error message. Thanks!

[elupus]

Not alone: https://github.com/elupus/gardena-bluetooth/actions/runs/5811885732/job/15756416437#step:7:92

invalid-publisher: valid token, but no corresponding publisher (Check failed for required claim 'job_workflow_ref')

[woodruffw]

Hmm, I wonder if GitHub went and changed the job_workflow_ref format on us. That seems pretty unlikely though (and I'd expect way more publishers to break, if so).

[woodruffw]

For context, here's our job_workflow_ref check:

def _check_job_workflow_ref(ground_truth, signed_claim, all_signed_claims):
    # We expect a string formatted as follows:
    #   OWNER/REPO/.github/workflows/WORKFLOW.yml@REF
    # where REF is the value of the `ref` claim.

    # Defensive: GitHub should never give us an empty job_workflow_ref,
    # but we check for one anyways just in case.
    if not signed_claim:
        return False

    ref = all_signed_claims.get("ref")
    if not ref:
        return False

    return f"{ground_truth}@{ref}" == signed_claim

[di]

@woodruffw Just noticed that that check is actually looking at the ref claim, not the job_workflow_ref claim...

[woodruffw]

@woodruffw Just noticed that that check is actually looking at the ref claim, not the job_workflow_ref claim...

Just to copy state here: this should be fine, since signed_claim contains the contents of job_workflow_ref. The reason we check ref at all is so that we can do an exact match against job_workflow_ref, rather than a string prefix check.

[woodruffw]

NB: I just successfully published one of my own projects using a trusted publisher, so whatever's happening here isn't consistent.

As a further debugging step, I'll look into having this workflow decode the OIDC identity and dump some of its claims.

[eifinger]

Failed for me as well with: invalid-publisher: valid token, but no corresponding publisher (Check failed for required claim 'job_workflow_ref')

[woodruffw]

As an idea here: @elupus @eifinger are you seeing these same issues if you run a completely fresh workflow run, rather than re-running a workflow?

[elupus]

As an idea here: @elupus @eifinger are you seeing these same issues if you run a completely fresh workflow run, rather than re-running a workflow?

It was from a clean run initially. Had not run since a month or so back.

[woodruffw]

It was from a clean run initially. Had not run since a month or so back.

Got it, thanks for confirming. Could you humor me and try it again from a clean run? I don't expect it to change, but just for an additional datapoint 🙂

[eifinger]

It was from a clean run initially. Had not run since a month or so back.

Got it, thanks for confirming. Could you humor me and try it again from a clean run? I don't expect it to change, but just for an additional datapoint 🙂

Will try with a new release later today.

[woodruffw]

I've also opened #174 with some changes that should assist in debugging here: if you configure your publishing workflow to use that branch rather than the normal release, it should give us some additional context here 🙂

[di]

We've deployed some new changes with even more error messages, @eifinger can you re-run the workflow and let us know what you see?

[di]

@martibosch I don't see any Trusted Publishers configured for https://pypi.org/p/pylandstats, see https://docs.pypi.org/trusted-publishers/adding-a-publisher/ for details on how to do that.

[elupus]

Same:
Error: Trusted publishing exchange failure:
Token request failed: the server refused the request for the following reasons:

• invalid-publisher: valid token, but no corresponding publisher (The ref claim is empty)

[eifinger]

Same:

Error: Trusted publishing exchange failure: 
Token request failed: the server refused the request for the following reasons:

* `invalid-publisher`: valid token, but no corresponding publisher (The ref claim is empty)

[webknjaz]

I've also opened #174 with some changes that should assist in debugging here: if you configure your publishing workflow to use that branch rather than the normal release, it should give us some additional context here slightly_smiling_face

Released in v1.8.9

[di]

PyPI has also just shipped some other changes which should result in a new error message here with even more details. One last request to re-run the workflow @eifinger and @elupus.

[eifinger]

https://github.com/eifinger/pywaze/actions/runs/5794194719/job/15797156287

Traceback (most recent call last):
  File "/app/oidc-exchange.py", line 206, in <module>
    rendered_claims = render_claims(oidc_token)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/oidc-exchange.py", line 147, in render_claims
    claims = json.loads(base64.urlsafe_b64decode(payload))
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/base64.py", line [13](https://github.com/eifinger/pywaze/actions/runs/5794194719/job/15797156287#step:7:14)4, in urlsafe_b64decode
    return b64decode(s)
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/base64.py", line 88, in b64decode
    return binascii.a2b_base64(s, strict_mode=validate)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
binascii.Error: Incorrect padding

[woodruffw]

My bug, sorry. Fixing now.

[woodruffw]

@eifinger mind trying again with #177?

[webknjaz]

My bug, sorry. Fixing now.

Looks like we need to start having pytest-based tests too..

[woodruffw]

My bug, sorry. Fixing now.

Looks like we need to start having pytest-based tests too..

Yeah...I can look at adding some in the next few days, if you'd like.

[webknjaz]

If you have time, this would be welcome. Meanwhile, I released v1.8.10 with the hotfix.

[elupus]

Looking more lika a workable error:

* `invalid-publisher`: valid token, but no corresponding publisher (The claim does not match, expecting 'elupus/gardena-bluetooth/.github/workflows/python-publish.yml@', got 'elupus/gardena-bluetooth/.github/workflows/python-publish.yml@1abc19106b818d4e1c26b238c3654321bc94b8b1')

This generally indicates a trusted publisher configuration error, but could
also indicate an internal error on GitHub or PyPI's part.


The claims rendered below are **for debugging purposes only**. You should **not**
use them to configure a trusted publisher unless they already match your expectations.

If a claim is not present in the claim set, then it is rendered as `MISSING`.

* `sub`: `repo:elupus/gardena-bluetooth:environment:pypi`
* `repository`: `elupus/gardena-bluetooth`
* `repository_owner`: `elupus`
* `repository_owner_id`: `482976`
* `job_workflow_ref`: `elupus/gardena-bluetooth/.github/workflows/python-publish.yml@1abc19106b818d4e1c26b238c3654321bc94b8b1`
* `ref`: ``

[woodruffw]

Yeah, this is confirming the strange behavior we expected: GitHub is creating some OIDC JWTs with a SHA-1 ref in the job_workflow_ref, rather than using the value of ref (which for some reason is empty).

[di]

Oops, didn't expect that PR to close an issue over here, reopening...

[di]

This should now be resolved for those experiencing the original issue. Anyone in the future thinking they are experiencing this should ensure they have created and correctly configured a publisher by following these docs: https://docs.pypi.org/trusted-publishers/adding-a-publisher/

@webknjaz I'll let you determine if there's anything left to do here and if this can be closed.

[webknjaz]

Thanks, I think it's fine to close this. If anybody hits publishing issues in the future, they'll likely have different causes and would require a new issue.

But I invite the reporters to leave a comment confirming that the issue is fixed for them.

[eifinger]

I can confirm that the issue is resolved for me: https://github.com/eifinger/pywaze/actions/runs/5794194719/job/15810516147

[elupus]

Confirmed here too!

[martibosch]

it was indeed my mistake as I had not configured the trusted publishing in pypi. In any case, it may be helpful to add the link https://docs.pypi.org/trusted-publishers/adding-a-publisher/ in the "Trusted publishing" section of the README - especially since it is something that is quite new.

In any case, thank you @woodruffw and @di for pointing my issue out.

[di]

@martibosch Good point, I've added a link to it here: #179

[webknjaz]

Thanks everyone! The README update with the link merged.