Unsupported language is automatically added to preload option

[develohpanda]

Hi there,

On my express server, I am using the supportedLngs and preload options, and i18next-http-backend is configured to reload resources at a specified interval, via the reloadInterval option.

My array of supported languages is ['en-US', 'fr-FR'], for example.
My preload array is also ['en-US', 'fr-FR'].
I have fallbackLng configured such that when loading en-NZ, the fallback of en-US will be used, because en-NZ is not supported and my backend will return an error.

The issue I am facing is with the following line in particular, which extends the preload array with any new languages that are encountered, even if they are unsupported.

i18next/src/i18next.js

Line 500 in c9e2bf5

this.options.preload = preloaded.concat(newLngs);

Thus, what I observe is that during normal operation where i18n has only seen requests for en-US or fr-FR, those two languages are reloaded periodically.

But when a request comes through asking for en-NZ translations (perhaps via the Accept-Language header), although i18n will fallback to en-US correctly, the periodic reload continues to include en-NZ. Thus I see successful reloads of en-US and fr-FR as expected, but also failures of loading the translations for en-NZ, which will always fail as it is unsupported.

Is this a bug? Should the extension of the preload array be opt-in, or conditional on it being a part of supported languages? I'm not quite sure what the process should be here, but extending the user-configured preload array without any check feels a bit fishy. What would you suggest?

    "i18next": "^23.10.1",
    "i18next-http-backend": "^2.5.0",
    "i18next-http-middleware": "^3.5.0",

I can share a minimal reproduction if that will help. Thank you in advance!

[adrai]

Extending the preload option is expecetd... but I agree that only supportedLngs should be handled...
So v23.11.3 should address this.

[develohpanda]

Wonderful, thank you for such a quick response! I’ll test it out tomorrow 🎉

[adrai]

If you like this module don’t forget to star this repo. Make a tweet, share the word or have a look at our https://locize.com to support the devs of this project.

There are many ways to help this project 🙏

[develohpanda]

I've just pulled this down to test, and I've realized that I made a mistake in my initial report.

The supportedLngs option is set to ['en', 'fr'] (not ['en-US', 'fr-FR']), with nonExplicitSupportedLngs=true and we have fallbackLng set to { en: ['en-US'], fr: ['fr-FR'] }.

This means that if a request comes through for loading en-NZ translations, i18next will first match en in supportedLngs, and then fallbackLng kicks in to use the regional en-US language. There is always one failed request to load en-NZ translations from my backend, but it's just the reload interval + preload array (which I've recently introduced) where I'm now stuck reloading translations that I know don't exist.

FWIW I don't necessarily think this requires another change in i18next, although I agree with the extent of change you've made. This is probably specific to my project, however I'm not too sure how best I should be handling this. I've read through #1687 - it makes sense but doesn't quite address the case I've got.

What I've ended up doing is hijacking the loadPath function to avoid the request, which still creates an entry in the store for the unsupported language, just with an empty object. This looks to be working as I expect, but please let me know if a more elegant solution comes to mind. Perhaps I could change the response code from my backend for translations I know to not exist, though I'm not sure if i18next will take that into account.

[adrai]

The main question is: why are you trying to avoid those requests?
How is the error influencing your app?

[develohpanda]

If a users' browser has several regional language preferences, which are seen by the express server after which a matching supported language is chosen, all of the unsupported languages will still remain in i18next and it will continue to try and load them, failing always.

This doesn't really have any substantial impact on the application, and it leads to some "known failures" within telemetry, but the residue doesn't seem like a good thing.

There are fixed regional languages that we support. There are steps to normalize a user into one of those supported regional languages, and once normalized the user remains normalized, which is why it's a bit of residue to keep attempting to load the known unsupported regional languages.

[adrai]

if you have a known list of supported (regional) languages, just add them via supportedLngs...

[develohpanda]

It doesn't make a difference if I add them via supportedLngs.

Regardless of supportedLngs being ['en', 'fr'] or ['en', 'en-US', 'fr', 'fr-FR'], the nonExplicitSupportedLngs property still needs to be true to accept en-NZ, and fallbackLng still needs to be set to normalize from en-NZ -> en -> en-US, but extending the preload array doesn't consider the fallbackLng for normalization.

Here's a different question for you - is there a more appropriate way for me to normalize from en-NZ to en-US?

[adrai]

You could write your own language detector...

[adrai]

You could also just use the convertDetectedLanguage option...

https://github.com/i18next/i18next-browser-languageDetector

convertDetectedLanguage: (lng) => {
  if (lng === 'en-NZ') return 'en-US'
  return lng
}

[develohpanda]

I do want to keep the bells and whistles of the i18next language detector, however convertDetectedLanguage may just cater for everything… I will try it out and report back! 😊

[develohpanda]

It seems convertDetectedLanguage isn't being hit at all on the http middleware language detector (the https://github.com/i18next/i18next-http-middleware/blob/master/lib/LanguageDetector.js within i18next-http-middleware) each time a new request comes in 🤔 that checks out, as I don't see any code to handle that property

[adrai]

It was not yet implemented.... please try with v3.6.0

[develohpanda]

Tried with 3.6.0 - while that does work to normalize the detected language, the original unsupported regional language (en-NZ) still does ends up in the preload array. That extending of the preload array must be happening earlier in the pipeline 🤔

I may need to create a minimal reproduction to share. I won't be able to do that immediately but will report back here once I get a chance to