Move from OAuth 1.0a to OAuth 2.0 for OSM logins

[pnorman]

OSM's OAuth 1.0a is in the process of being deprecated, as discussed at openstreetmap/operations#867. No timeline has been set, but we do not expect to shut off OAuth 1.0a this year. It would be good to move to OAuth 2.0 well before this time.

[davidbgk]

@pnorman thanks for the information, it's probably something to report upstream too, we use the backend from social-core and there might be other beneficiaries(?) if we mutualise our efforts: https://github.com/python-social-auth/social-core/blob/master/social_core/backends/openstreetmap.py

It looks like @Xmypblu made the initial/OAuth1 one: python-social-auth/social-core@5832454

[yohanboniface]

There is already an issue upstream :)

python-social-auth/social-core#758

[cayenne17]

OAuth 1.0a and HTTP Basic Auth shutdown

• June 1st, 2024: OAuth 1.0a and HTTP Basic Auth will be shut down.

[davidbgk]

There is now some code to test and a dedicated PR 🎉

[mmd-osm]

As a preview version, the new OAuth 2.0 provider is documented here: https://python-social-auth--202.org.readthedocs.build/en/202/backends/openstreetmap_oauth2.html

Most notable change is that the settings are now called SOCIAL_AUTH_OPENSTREETMAP_OAUTH2_KEY and SOCIAL_AUTH_OPENSTREETMAP_OAUTH2_SECRET, and the backend name has changed to social_core.backends.openstreetmap_oauth2.OpenStreetMapOAuth2

Relevant code to be adjusted: https://github.com/umap-project/umap/blob/master/umap/settings/base.py#L276-L281

I hope a new python-social-auth release including the new backend will be available soon.

[davidbgk]

Now on master! python-social-auth/social-core@f6d81fd

[mmd-osm]

Hi there. I'd really appreciate your help in doing some further testing of the new OAuth2 backend. In particular, can you please check, if the following scenario is working in uMap?

• Existing uMap user with associated OSM OAuth 1.0a login now tries to log on using OSM OAuth 2.0.
• Is the account association working properly in this case, and the user can continue using their existing account?

[yohanboniface]

I'll do some testing (but in the rush this week). AFAIK, changing the class name (OpenStreetMap => OpenStreetMap2) will result in changing the provider name in the database, so in some situation this can created unexpected behaviors: in the case of OSM OAuth, we do not have the email address (OSM does not provide it in the extra properties), so without further actions, in the Django app, I think an OpenStreetMap2 account will not be associate with an OpenStreetMap one, so Django will try to create a new account with the same username, which will fail because it's a unique value. So I'm expecting a migration to be needed (to rename providers in database from openstreetmap to openstreetmap2).

Anyway, thanks a lot for your work on this!

[mmd-osm]

Thank you for the quick feedback. Somehow I thought it might be possible to bring openstreetmap and openstreetmap-oauth2 accounts together based on the user id. I agree that linking based on email wouldn’t work, since this information is not exposed by the osm server.
I also checked some of the database tables and my first impression was that the parameters for both backends differ too much, making a direct migration to -oauth2 challenging. I will also try to dig a bit more into this topic. My hope is still that we can offer a smooth transition without users losing access to their previous accounts, even if oauth 1.0a is no longer available.

[mmd-osm]

I was playing a bit with the social-examples - example-django, and added the following function in local_settings.py:

The idea is to find out, if there's an openstreetmap registration for the same user id when logging in via openstreetmap-oauth2, and vice versa. I really have no clue, if this is the right approach. At least in my tests, results looked ok. No matter if I logged on using OAuth 1.0a or OAuth 2.0, in both cases, I ended up with the same backend user.

def lookup_osm_oauth(backend, response, details, user, *args, **kwargs):
    
    OSM_BACKENDS = ['openstreetmap', 'openstreetmap-oauth2']
    
    if backend.name in OSM_BACKENDS:
        user_id = backend.get_user_id(details,response)
        check_backend = OSM_BACKENDS[1 - OSM_BACKENDS.index(backend.name)]
        social_user = backend.strategy.storage.user.get_social_auth(check_backend, user_id)
        if social_user == None:
            return None
        else:
            return {"user": social_user.user, "is_new": False}

Then I defined the following pipeline:

SOCIAL_AUTH_PIPELINE = (
    'social_core.pipeline.social_auth.social_details',
    'social_core.pipeline.social_auth.social_uid',
    'social_core.pipeline.social_auth.auth_allowed',
    'social_core.pipeline.social_auth.social_user',
    'social_core.pipeline.user.get_username',
    'example.local_settings.lookup_osm_oauth',
    'social_core.pipeline.user.create_user',
    'social_core.pipeline.social_auth.associate_user',
    'social_core.pipeline.social_auth.load_extra_data',
    'social_core.pipeline.user.user_details',
    "social_core.pipeline.debug.debug",    
)

screen recording (click to open)

[yohanboniface]

@mmd-osm any clue on how to make the OAuth2 OSM app work with a non https local host ? I mean for testing.

[mmd-osm]

You can use http://127.0.0.1:8888 (or some other port number) as redirect URL. This will work even without https.

Please also check the example here: python-social-auth/social-core#758 (comment)

[yohanboniface]

Thanks!

You can use http://127.0.0.1:8888/ (or some other port number) as redirect URL. This will work even without https.

You mean specifically 127.0.0.1 instead of localhost ?

[mmd-osm]

Yes, exactly. You cannot use localhost without https, it is blocked by the underlying OAuth implementation on osm.org. See the discussion here: openstreetmap/openstreetmap-website#4287

[yohanboniface]

Yes, worked, thanks :)
Continuing to test the new client, thanks again for your work!

[yohanboniface]

Seems all good to me! Now we need a new python-social-auth release.

[yohanboniface]

I've tested your suggestion of using pipeline for fallback, but I think running a SQL once is simpler, otherwise we need to keep this fallback for ever (or until the SQL is run, but in this case is a bit more complex as some users will have both entries for oauth1 and oauth2).

Here is the SQL to be run for the migration, maybe this will become an automatic migration, as I'm thinking about making OSM OAuth2 mandatory for uMap 2.0.0:

UPDATE social_auth_usersocialauth SET provider = 'openstreetmap-oauth2' WHERE provider = 'openstreetmap';

---

For the record, here is a modification of your initial suggestion.

diff --git a/umap/settings/base.py b/umap/settings/base.py
index f4f96332..b6f445c5 100644
--- a/umap/settings/base.py
+++ b/umap/settings/base.py
@@ -283,6 +283,15 @@ if SOCIAL_AUTH_OPENSTREETMAP_KEY and SOCIAL_AUTH_OPENSTREETMAP_SECRET:
 
 AUTHENTICATION_BACKENDS += ("django.contrib.auth.backends.ModelBackend",)
 
+# Do not override
+from social_core.pipeline import DEFAULT_AUTH_PIPELINE
+
+SOCIAL_AUTH_PIPELINE = list(DEFAULT_AUTH_PIPELINE)
+SOCIAL_AUTH_PIPELINE.insert(
+    SOCIAL_AUTH_PIPELINE.index("social_core.pipeline.user.get_username") + 1,
+    "umap.utils.fallback_osm_oauth1",
+)
+
 LOGGING = {
     "version": 1,
     "disable_existing_loggers": False,
diff --git a/umap/utils.py b/umap/utils.py
index 003b9b01..c0df2c3b 100644
--- a/umap/utils.py
+++ b/umap/utils.py
@@ -162,3 +162,16 @@ def merge_features(reference: list, latest: list, incoming: list):
         merged.append(item)
 
     return merged
+
+
+def fallback_osm_oauth1(backend, response, details, user, *args, **kwargs):
+    # Try to make new OAuth2 work for users having a OAuth 1 local user
+    if backend.name == "openstreetmap-oauth2":
+        user_id = backend.get_user_id(details, response)
+        # "openstreetmap" is the OAuth1 backend name
+        social_user = backend.strategy.storage.user.get_social_auth(
+            "openstreetmap", user_id
+        )
+        if social_user:
+            return {"user": social_user.user, "is_new": False}
+        return None

[mmd-osm]

Maybe one thing to double check when migrating the social_auth_usersocialauth table to oauth2: When I was testing this scenario, I found some OAuth 1.0a specific info in there. I don't recall exactly, if these were stored in a JSON format string, or in dedicated database columns.

Now, when a user logs on the first time using OAuth 2.0, ideally all of the old OAuth 1.0a specific info should be deleted/removed.
In case of a single JSON string, this would probably happen automatically, and we will see a clean OAuth 2.0 entry.
In case of dedicated database columns, I'm not exactly sure about the outcome.

[yohanboniface]

It's a jsonb column, and AFAIK it is replaced at each login:

uMap itself does not use the extradata column, and it seems neither social-core itself, so I'd say it safe. FYI, we did exactly the migration that way when going from Twitter OAuth1 to Twitter OAuth2 in OSM France server, and we had no issue with this (but with Twitter in general, a lot, ha ha!).