Document server.servlet.encoding.* properties and server.servlet.encoding.mapping in particular #32472

Dmitrii-Iakovenko · 2022-09-22T00:03:37Z

Spring Boot 2.7.3

Spring MVC + Mustache = UTF-8 not working
Add CharacterEncodingFilter = UTF-8 works
Add Spring Security + config Mustache = UTF-8 display works, sending - no
Change CharacterEncodingFilter to OrderedCharacterEncodingFilter = UTF-8 works

Repository demonstrating the issue + commit history.

Old similar problem - #3912

wilkinsona · 2022-09-22T08:52:12Z

Thanks for the sample. Unfortunately, I've been unable to reproduce the problem using the first commit in the repository. If I disable the auto-configuration of the OrderedCharacterEncodingFilter by setting server.servlet.encoding.enabled=false, UTF-8 submitted by the form is corrupted while UTF-8 served directly from the mustache template remains correct.

I am on macOS which uses UTF-8 by default, but configuring the JVM to use US-ASCII (verified using Charset.getDefaultCharset()) makes no difference to the behaviour. What do I need to do so that the provided sample will reproduce the problem?

spring-projects-issues · 2022-09-29T08:55:11Z

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

wilkinsona · 2022-10-05T11:41:44Z

@lksh5737 As I described above, that doesn't match the behavior that I have observed. If you or @Dmitrii-Iakovenko can provide a sample that shows the problem, we can take another look.

wilkinsona · 2022-10-06T08:24:04Z

@lksh5737 Thanks, but that does not reproduce the problem for me either. It works fine for me on macOS, even when I force the JVM to use US-ASCII as its default encoding. As I asked above, what do I need to do so that the provided sample will reproduce the problem?

wilkinsona · 2022-10-06T08:46:34Z

That doesn't seem to make a difference:

wilkinsona · 2022-10-06T10:58:12Z

Perhaps you can debug your application with Spring Boot 2.6 where it works and Spring Boot 2.7 where it does not to find the difference? Unfortunately, we're unlikely to be able to help any further without knowing how to reproduce the problem.

wilkinsona · 2022-10-06T11:59:16Z

It's hard to offer specific guidance as we don't know where the problem is. I would step through the code using your IDE's debugger to try to identify the point at which the UTF-8 characters are corrupted. The rendering of the MustacheView that's done in renderMergedTemplateModel(Map<String, Object>, HttpServletRequest, HttpServletResponse) may be a good place to start.

spring-projects-issues · 2022-10-20T12:00:12Z

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.

swhyeon98 · 2023-07-15T08:13:17Z

https://github.com/swhyeon98/mustache_with_springboot_test.git

Can you find any errors on the above sample?

There are two solutions for this issue.

you can add the following tags to the application.properties file:
server.servlet.encoding.charset=UTF-8
server.servlet.encoding.force=true

Alternatively, you could downgrade your Spring Boot version to 2.6.

Both methods have solved the issue.

The followings are photos of the test run in different environments.

wilkinsona · 2023-07-18T08:56:48Z

@swhyeon98 Unfortunately, I cannot reproduce the problem with your sample either. I've tried on macOS using its default encoding (UTF-8) and also configured to use windows-1252 (-Dfile.encoding=cp1252). In both cases the UTF-8 text is returned:

$ curl -i localhost:8080
HTTP/1.1 200 
Content-Type: text/html;charset=UTF-8
Content-Language: en-GB
Content-Length: 239
Date: Tue, 18 Jul 2023 08:55:30 GMT

<!DOCTYPE HTML>
<html>
<head>
    <title>스프링 부트 웹서비스</title>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>
<body>
<h1>스프링 부트로 시작하는 웹 서비스</h1>
</body>
</html>

swhyeon98 · 2023-07-20T08:26:49Z

@wilkinsona
Thank you for your answer.
May I ask a few more questions?

did you use the IDE in your testing?
Could it be a problem with the IDE?
If you used an IDE, which IDE did you use? (I used IntelliJ.)
If you used an IDE, could a plug-in have affected the results?

wilkinsona · 2023-07-20T08:49:35Z

I ran it in Eclipse and on the command line. On the command line, I built the jar and then ran it using java -jar. The results were the same in both cases, both with and without file.encoding=cp1252. I've just tried it in IntelliJ, again with and without file.encoding=cp1252, and the problem did not occur there either.

swhyeon98 · 2023-07-23T06:42:09Z

@wilkinsona Thank you very much.

I will try to find the difference between you and me and the cause (java version, etc.).

If I find the cause, can I comment again then?

wilkinsona · 2023-07-24T07:30:18Z

If I find the cause, can I comment again then?

Yes, of course. Please do. We'd really like to understand what's going on as I'm sure it'll help others.

swhyeon98 · 2023-08-20T14:07:32Z

I could not find any difference between you and me, but I think I found the cause of the problem.

When I checked in developer mode - network, I saw that Content-Tpye shows ISO-8859-1.

So I checked the ViewResolver on Mustache's end,
The result of the check

if (mustache.getServlet().getContentType() ! = null) {
resolver.setContentType(mustache.getServlet().getContentType().toString());
{ }
It seems to occur because the above code only sends up to text/html on the ViewResolver side.

You can check this part in my previous sample.

I still haven't found the difference between you and me and why it's fixed when I downgrade SpringBoot.

How do you see the output when you check developer mode - network?

wilkinsona · 2023-08-31T10:58:29Z

How do you see the output when you check developer mode - network?

It's the same as it was with curl: text/html;charset=UTF-8

Looking at the headers in your request that's receiving an ISO-8859-1 response led me to try sending a request with an Accept-Language header. By default in Chrome, the header's value is en-GB,en-US;q=0.9,en;q=0.8 for me and curl doesn't send one at all by default. If I send the same header as your browser, I can reproduce the problem:

$ curl -i localhost:8080 -H "Accept-Language: ko-KR,ko;q=0.9"
HTTP/1.1 200 
Content-Type: text/html;charset=ISO-8859-1
Content-Language: ko-KR
Content-Length: 193
Date: Thu, 31 Aug 2023 10:54:47 GMT

<!DOCTYPE HTML>
<html>
<head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
    <title>??? ?? ????</title>
</head>
<body>
<h1>??? ??? ???? ? ???</h1>
</body>
</html>

I'm not yet sure why it has the effect that it does, but it gives me something to investigate.

wilkinsona · 2023-08-31T11:45:18Z

The problem is due to Tomcat's mapping of locales to charsets.

Without an Accept-Language header, the default locale is used. On my machine this is en_GB which is mapped to UTF-8. With an Accept-Language header of Accept-Language: ko-KR,ko;q=0.9 the locale is ko_KR. Tomcat has no charset mapping for this locale. This results in the default character encoding being used and the Servlet spec states that this is ISO-8859-1.

Tomcat provides a flag that can be used to disable its enforcement of the default character encoding but it doesn't really help. Not with curl at least. With the org.apache.catalina.connector.Response.ENFORCE_ENCODING_IN_GET_WRITER system property set to false, the Content-Type changes to omit the charset, but the UTF-8 characters are still corrupted:

curl -i localhost:8080 -H "Accept-Language: ko-KR,ko;q=0.9"
HTTP/1.1 200 
Content-Type: text/html
Content-Language: ko-KR
Content-Length: 193
Date: Thu, 31 Aug 2023 11:24:59 GMT

<!DOCTYPE HTML>
<html>
<head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
    <title>??? ?? ????</title>
</head>
<body>
<h1>??? ??? ???? ? ???</h1>
</body>
</html>

Instead, the problem can be addressed by mapping the ko locale to UTF-8:

server.servlet.encoding.mapping.ko=UTF-8

This property is equivalent to the <locale-encoding-mapping-list> entry in web.xml.

Without this custom mapping, the same problem occurs with Jetty as it defaults to ISO-8859-1 as required by the servlet spec. Interestingly, it does not occur with Undertow which uses UTF-8 regardless.

While server.servlet.encoding.mapping is listed in the configuration properties appendix, it isn't mentioned anywhere in the rest of the reference documentation as far as I can tell. I think we should use this issue to correct that.

spring-projects-issues added the status: waiting-for-triage label Sep 22, 2022

wilkinsona added the status: waiting-for-feedback label Sep 22, 2022

spring-projects-issues added status: feedback-reminder status: feedback-provided and removed status: waiting-for-feedback status: feedback-reminder labels Sep 29, 2022

wilkinsona added status: waiting-for-feedback and removed status: feedback-provided labels Oct 5, 2022

spring-projects-issues added status: feedback-provided and removed status: waiting-for-feedback labels Oct 6, 2022

wilkinsona added status: waiting-for-feedback and removed status: feedback-provided labels Oct 6, 2022

spring-projects-issues added status: feedback-provided and removed status: waiting-for-feedback labels Oct 6, 2022

wilkinsona added status: waiting-for-feedback and removed status: feedback-provided labels Oct 6, 2022

spring-projects-issues added status: feedback-provided and removed status: waiting-for-feedback labels Oct 6, 2022

bclozel added status: waiting-for-feedback and removed status: feedback-provided labels Oct 6, 2022

spring-projects-issues added status: feedback-provided and removed status: waiting-for-feedback labels Oct 6, 2022

wilkinsona removed the status: feedback-provided label Oct 6, 2022

spring-projects-issues added the status: feedback-reminder label Oct 13, 2022

spring-projects-issues closed this as completed Oct 20, 2022

spring-projects-issues removed status: waiting-for-feedback status: feedback-reminder status: waiting-for-triage labels Oct 20, 2022

wilkinsona reopened this Aug 31, 2023

wilkinsona added the status: waiting-for-triage label Aug 31, 2023

wilkinsona self-assigned this Aug 31, 2023

wilkinsona changed the title ~~Problems with UTF-8 when using Spring & Mustache~~ Document server.servlet.encoding.* properties and server.servlet.encoding.mapping in particular Aug 31, 2023

wilkinsona added type: documentation and removed status: waiting-for-triage labels Aug 31, 2023

wilkinsona added this to the 2.7.x milestone Aug 31, 2023

wilkinsona removed their assignment Aug 31, 2023

wilkinsona self-assigned this Oct 25, 2023

wilkinsona mentioned this issue Oct 25, 2023

Document server.servlet.encoding.* properties and server.servlet.encoding.mapping in particular #38033

Closed

wilkinsona modified the milestones: 2.7.x, 2.7.18 Oct 25, 2023

wilkinsona closed this as completed in ea39a33 Oct 25, 2023

wilkinsona mentioned this issue Feb 26, 2024

[3.2.3] broken umlauts in htmx req/resp #39745

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Document server.servlet.encoding.* properties and server.servlet.encoding.mapping in particular #32472

Document server.servlet.encoding.* properties and server.servlet.encoding.mapping in particular #32472

Dmitrii-Iakovenko commented Sep 22, 2022 •

edited by snicoll

Loading

wilkinsona commented Sep 22, 2022

spring-projects-issues commented Sep 29, 2022

wilkinsona commented Oct 5, 2022

wilkinsona commented Oct 6, 2022

wilkinsona commented Oct 6, 2022

wilkinsona commented Oct 6, 2022

wilkinsona commented Oct 6, 2022 •

edited

Loading

spring-projects-issues commented Oct 20, 2022

swhyeon98 commented Jul 15, 2023 •

edited

Loading

wilkinsona commented Jul 18, 2023

swhyeon98 commented Jul 20, 2023

wilkinsona commented Jul 20, 2023 •

edited

Loading

swhyeon98 commented Jul 23, 2023

wilkinsona commented Jul 24, 2023

swhyeon98 commented Aug 20, 2023 •

edited

Loading

wilkinsona commented Aug 31, 2023

wilkinsona commented Aug 31, 2023 •

edited

Loading

Document server.servlet.encoding.* properties and server.servlet.encoding.mapping in particular #32472

Document server.servlet.encoding.* properties and server.servlet.encoding.mapping in particular #32472

Comments

Dmitrii-Iakovenko commented Sep 22, 2022 • edited by snicoll Loading

wilkinsona commented Sep 22, 2022

spring-projects-issues commented Sep 29, 2022

wilkinsona commented Oct 5, 2022

wilkinsona commented Oct 6, 2022

wilkinsona commented Oct 6, 2022

wilkinsona commented Oct 6, 2022

wilkinsona commented Oct 6, 2022 • edited Loading

spring-projects-issues commented Oct 20, 2022

swhyeon98 commented Jul 15, 2023 • edited Loading

wilkinsona commented Jul 18, 2023

swhyeon98 commented Jul 20, 2023

wilkinsona commented Jul 20, 2023 • edited Loading

swhyeon98 commented Jul 23, 2023

wilkinsona commented Jul 24, 2023

swhyeon98 commented Aug 20, 2023 • edited Loading

wilkinsona commented Aug 31, 2023

wilkinsona commented Aug 31, 2023 • edited Loading

Dmitrii-Iakovenko commented Sep 22, 2022 •

edited by snicoll

Loading

wilkinsona commented Oct 6, 2022 •

edited

Loading

swhyeon98 commented Jul 15, 2023 •

edited

Loading

wilkinsona commented Jul 20, 2023 •

edited

Loading

swhyeon98 commented Aug 20, 2023 •

edited

Loading

wilkinsona commented Aug 31, 2023 •

edited

Loading