[4.x]: json_encode flags bug

[davidwebca]

What happened?

Description

I usually like to give as much detail as I can when reporting bug, but this one evades me a bit. I booted a new pro plan shared hosting on OVH and installed everything, coming from my staging server where everything ran correctly, and I thought everything was fine until I realized autosuggest fields didn't work. Everytime I made a change, I also realized that blocks made with Craft Neo were getting emptied.

I dug down and it was a json_encode bug. The default json_encode filter that passes the data in /src/templates/_includes/forms/autosuggest.twig:71 would return false without any errors reported from "json_last_error" or "json_last_error_msg". I was stumped. I played with the params passed to the filter and managed to make it work when the flags integer gets over 512. I don't know what specific flags that triggers because their integers are pretty high, but I thought maybe it was a utf-8 encoding issue so I overrode the json_encode filter from twig by creating a custom extension and now pass JSON_INVALID_UTF8_IGNORE. I took Indigo Viking's code and adapted it to add json_encode so that it overrides it, basically.

Now everything works fine, but I can't tell for the life of me if PHP installs usually have different default flags or not. I've never had that problem before, no one seems to have ever had it and I've already verified and it's not related to the php 8.1 bug that was introduced and fixed (when switching to php 8.2 and other versions on the host, it was still creating this issue weirdly).

Steps to reproduce

1. Get a pro hosting on OVH's server, this time it was in Germany
2. Install Craft on php 8.1
3. Autosuggest fields fail everywhere because no data get json_encoded (false returned without any error message).

Expected behavior

json_encode should work...

Actual behavior

The data seems to fail to get encoded, maybe because of some default flags on some hosting? Should Craft add a custom json_encode / decode Twig extension to be able to pass custom default flags?

Notes

Honestly, this is pretty niche and since I haven't found anyone else having this issue ever, even outside Craft's bubble, it may be safe to ignore this issue altogether, but I thought it would be a good idea to document it just in case someone else encounters this. Here's the code of the Json Twig extension I adapted: https://gist.github.com/davidwebca/ad2890ad8c0dc3156036051294b6e3ae

Craft CMS version

4.4.3

PHP version

8.1

Operating system and version

Debian / Linux 10 (buster)

Database type and version

Mysql 5.7

Image driver and version

GD 2.1 (bundled)

Installed plugins and versions

Name                Handle              Package Name                    Version  Installed  Enabled
    ------------------  ------------------  ------------------------------  -------  ---------  -------
    Field Manager       field-manager       verbb/field-manager             3.0.5    Yes        Yes
    Formie              formie              verbb/formie                    2.0.24   Yes        Yes
    Hyper               hyper               verbb/hyper                     1.1.2    Yes        Yes
    Icon Picker         icon-picker         verbb/icon-picker               2.0.12   Yes        Yes
    Inline SVG          inlinesvg           clubstudioltd/craft-inline-svg  2.0.0    Yes        Yes
    Maps                simplemap           ether/simplemap                 v4.0.3   Yes        Yes
    Navigation          navigation          verbb/navigation                2.0.16   Yes        Yes
    Neo                 neo                 spicyweb/craft-neo              3.7.1    Yes        Yes
    Redactor            redactor            craftcms/redactor               3.0.3    Yes        Yes
    SEOmatic            seomatic            nystudio107/craft-seomatic      4.0.22   Yes        Yes
    Translations admin  translations-admin  mutation/translate              3.0.1    Yes        Yes
    Vite                vite                nystudio107/craft-vite          4.0.5    Yes        Yes

[brandonkelly]

If JSON_INVALID_UTF8_IGNORE is fixing the issue, then you must have an invalid unicode character somewhere in that environment – either in an environment variable or perhaps in the DB content (depending on the autosuggest field).

I’ve just updated the |json_encode filter to start calling craft\helpers\Json::encode() internally, which will throw an exception if there’s an invalid value, rather than simply returning false, so the issue should be a little easier to track.

You can update to get that change by changing your craftcms/cms requirement in composer.json to dev-develop as 4.4.3, and running composer update.

[brandonkelly]

3.8.4 and 4.4.4 are out with that change.

[davidwebca]

Amazing! Thanks a lot.