jetty client with proxy - ssl traffic between both proxy and servers

[Michal-Cho]

Jetty version
11.0.13
Java version
zulu 17.0.6
Question
Hello!

In our project we need to use jetty client with proxy that requires SSL, same as final destinations servers. We store needed certificates in keystore, both for proxy and dest servers

What I have figured out so far is that I need to create a HttpProxy object (using constructor with SslContextFactory.Client) and add it to client's ProxyConfiguration collections.
It is the same SslContextFactory.Client that is used in the client and it is working fine if we skip a proxy and run traffic client<--> server directly.

I can see in wireshark the following

• once a client needs to send a request it contacts a proxy

• TLS handshake is successful - seems to be correct

• Traffic between proxy and final server gets broken and my client receives java.nio.channels.ClosedChannelException

I would assume that the proxy server should try to establish connection to the final destination server using certificates that it could get from the client. But that does not work.

I've written simple program to reproduce the issue:

public class Main {
    public static void main(String[] args) throws Exception {

        System.out.println("Test proxy!");
        SslContextFactory.Client sslContextFactory = new SslContextFactory.Client();
        KeyStore keyStore = KeyStore.getInstance(new File("/Users/michal/Downloads/keystore_common.p12"), "123456".toCharArray());

        sslContextFactory.setKeyStorePassword("123456");
        sslContextFactory.setKeyStore(keyStore);
        sslContextFactory.setTrustStore(keyStore);
        sslContextFactory.setTrustStorePassword("123456");

        ClientConnector clientConnector = new ClientConnector();
        clientConnector.setSslContextFactory(sslContextFactory);
        HttpClient httpClient = new HttpClient(new HttpClientTransportDynamic(clientConnector));

        HttpProxy httpProxy = new HttpProxy(new Origin.Address("localhost", 8443),sslContextFactory);
        httpClient.getProxyConfiguration().addProxy(httpProxy);

        httpClient.start();

        Request req = httpClient.newRequest(new URI("https://our.server.net:30820/something"));

        req.method(req.getMethod()).send(new BufferingResponseListener() {
            @Override
            public void onComplete(Result result) {
                System.out.println(result.isFailed());
                System.out.println(result.getFailure());
                System.out.println(result.getResponse().getStatus());
                try {
                    httpClient.stop();
                } catch (Exception e) {
                    throw new RuntimeException(e);
                }
            }
        });
    }
}

If we run similar request from a simple python code:

#!/usr/bin/python3

import requests

url = 'https://our.server.net:30820/something' # SSL
PROXY = {'http': 'https://hej:hopp@localhost:8443', 'https': 'https://hej:hopp@localhost:8443'}
AUTH = ('user', 'myPassword')
KEY = '/Users/user/Downloads/both.pem'
verify = requests.get(url,
                      auth=AUTH,
                      proxies=PROXY,
                      verify=KEY)
print(verify.text, verify.status_code)

we get correct response.

Proxy does not require authentication for now.
My question here is - have I understood the jetty flow correctly? Or am I missing something? What confuses me is that it works with the pyton script and it works fine without proxy (so it means that my keystore is correct)

I will be really grateful for any hints/advice where I could find more information about setup described above

[Michal-Cho]

Traffic between client and proxy

Traffic between proxy and server

[sbordet]

I would assume that the proxy server should try to establish connection to the final destination server using certificates that it could get from the client.

No, the forward proxy only establishes a TCP connection to the server, not a TLS connection.
What proxy are you using? Is it Jetty too?

HttpProxy httpProxy = new HttpProxy(new Origin.Address("localhost", 8443),sslContextFactory);

Should be:

HttpProxy httpProxy = new HttpProxy(new Origin.Address("localhost", 8443), true);

I understand it is only a test, but:

        req.method(req.getMethod()).send(new BufferingResponseListener() {
            @Override
            public void onComplete(Result result) {
                System.out.println(result.isFailed());
                System.out.println(result.getFailure());
                System.out.println(result.getResponse().getStatus());
                try {
                    httpClient.stop();
                } catch (Exception e) {
                    throw new RuntimeException(e);
                }
            }
        });

You cannot stop the HttpClient from an HttpClient callback like onComplete(), see https://www.eclipse.org/jetty/documentation/jetty-11/programming-guide/index.html#pg-client-http-stop.

Your code should work, assuming the certificates are correct.
Please enable DEBUG logging for org.eclipse.jetty and attach the logs to this issue.

[Michal-Cho]

thank you for the input!
I've changed the proxy configuration to
HttpProxy httpProxy = new HttpProxy(new Origin.Address("localhost", 8443), true);

and re-run the test. I've also removed the stopping of the client, unfortunately no improvements.

The proxy we are using for testing is pproxy, that we start like this:
pproxy -l http+ssl://0.0.0.0:8443 -l http://0.0.0.0:8080 --ssl cert_proxy.pem,server.key -vv -d

You can find debug logs attached
jetty.log

looking.forward to hearing from you

[sbordet]

Same issue as #6483, the proxy sends a Connection: close in the 200 response 🤦🏼‍♂️

[Michal-Cho]

ok, let me look at the issue and try with a jetty ProxyServlet later.

What is strange that it works fine with python

[sbordet]

The python client is more forgiving when it receives a Connection: close, while the Jetty client is more strict.

You should use a ConnectHandler to handle the CONNECT method (not a ProxyServlet), and it will work because ConnectHandler will not send a Connection: close header.

I'm changing the Jetty client to ignore Connection: close for a 200 response to CONNECT to support these bad proxies 😩

[sbordet]

@Michal-Cho can you try the code in #9508 and report back if it works for you?

[Michal-Cho]

sure, let me check

[Michal-Cho]

@sbordet in a quick test I got a correct response 200 :)
so it looks like the fix has worked. Now I have to check if I can somehow apply it to the old version (9.x) that we are using as the upgrade from 9 to 10 is not that straight forward I guess - I've seen that there were quite a lot of changes done there

[sbordet]

as the upgrade from 9 to 10 is not that straight forward I guess

It should be, there have not been many changes in the HttpClient APIs, and the few ones are quite straightforward.

[Michal-Cho]

@sbordet fyi - I've backported the fix to jetty 9 and it seems to work in a quick test, but I guess we can not count on any new official release there ;)

[sbordet]

@Michal-Cho re: new Jetty 9.4.x releases you are correct.

Out of curiosity, what's stopping you from upgrading?

[Michal-Cho]

we will upgrade but having a fix in 9 would be much simpler as a short term solution :)

[joakime]

@sbordet fyi - I've backported the fix to jetty 9 and it seems to work in a quick test, but I guess we can not count on any new official release there ;)

The only way we cut new Jetty 9 releases anymore is through a support contract that sponsors the release.

[Michal-Cho]

@sbordet if we go with the upgrade - which jetty release could contain the fix and what would be a release date?

[sbordet]

The fix would be present in the next Jetty 10/11 release. As for the date we don't have a precise date for the next release.