Troubleshooting TLS Session Re-Use and Mutual Authentication in HAProxy

We take data protection seriously at Datto, which is why we’ve been increasingly using mutual TLS authentication to secure communications between components in our application stack. Our use of Hashicorp Vault has accelerated this security pattern, as Vault makes it easy to deploy and manage multiple CAs. Recently, we saw an increase in TLS-related errors for one of our mutually-authenticated application endpoints. In this article, I’ll walk you through how we debugged and resolved this problem. I’ll also take you on a deep dive into reproducing this issue, and I’ll hopefully teach you some fun OpenSSL commands along the way.

I’ll preface this article by saying that our issue was relatively simple, and this is a tale of troubleshooting and protocol deep-dives. Some of my favorite and most informative technical articles have described problem-resolution journeys. I always enjoy reading about how others troubleshoot interesting issues in their environment, and I hope you will too. Alright, let’s get started!

The Setup

Let’s start with an overview of the topology and basic app functionality. I’m greatly simplifying this for the sake of the article, but it should still give you an idea of what we’re working with.

This particular application contains endpoints that require clients to provide a certificate for authentication. We perform TLS offload at our load balancer cluster, and we pass along certificate disposition information to the backend app servers. Here’s what that looks like:

A client reaches out to our endpoint and provides its client certificate
The load balancer (HAProxy) inspects the certificate. It then forwards the request to the app servers. Crucially, it includes all of the information about the client certificate (and its verification status) as HTTP headers.
The app inspects these HTTP headers and makes an authorization decision based on what they contain. If the client included a valid certificate signed by an expected intermediate, then it may be authorized to perform certain restricted application functions.

Importantly, the client verification is optional. HAProxy will simply perform verification and pass the disposition, along with other certificate information, on to the application to make its own decisions. HAProxy makes this type of configuration easy, as shown by the partial configuration snippet below:

listen example-ssl
  log-tag haproxy-example-ssl
  bind 0.0.0.0:443 ssl crt /etc/haproxy/pki/haproxy.pem ca-file /etc/haproxy/pki/ca.pem verify optional
  mode http
  balance roundrobin
  http-request set-header SSL_CLIENT_A_KEY      %{+Q}[ssl_c_key_alg]
  http-request set-header SSL_CLIENT_A_SIG      %{+Q}[ssl_c_sig_alg]
  http-request set-header SSL_CLIENT_I_DN_CN    %{+Q}[ssl_c_i_dn(cn)]
  http-request set-header SSL_CLIENT_I_DN_C     %{+Q}[ssl_c_i_dn(c)]
  < additional headers omitted for brevity >

In practice, here’s what the HTTP request looks like from the backend application’s point of view. Notice that the headers contain all of the certificate information from the load balancer:

$  curl -k --cert client.pem --key key.pem --cacert ca.pem https://server.example.com
GET / HTTP/1.1
Host: server.example.com
Accept: */*
Ssl_client_a_key: rsaEncryption
Ssl_client_a_sig: RSA-SHA256
Ssl_client_cert: 1
Ssl_client_i_dn: /CN=App Intermediate
Ssl_client_i_dn_c: 
Ssl_client_i_dn_cn: App Intermediate
Ssl_client_i_dn_l: 
Ssl_client_i_dn_o: 
Ssl_client_i_dn_ou: 
Ssl_client_i_dn_st: 
Ssl_client_m_serial: 2B6E0CEECD2B26895A1A7923478B00477EC4ED21
Ssl_client_m_version: 3
Ssl_client_s_dn: /CN=client.example.com
Ssl_client_s_dn_c: 
Ssl_client_s_dn_cn: client.example.com
Ssl_client_s_dn_l: 
Ssl_client_s_dn_o: 
Ssl_client_s_dn_ou: 
Ssl_client_s_dn_st: 
Ssl_client_v_end: 220211112822Z
Ssl_client_v_start: 210213232753Z
Ssl_client_verify: 0
Ssl_tls_sni: server.example.com
User-Agent: curl/7.58.0
X-Forwarded-For: 10.72.4.147
X-Proxy-Ip: 10.40.155.211

(Credit to my colleague Jamie McKnight for his excellent header_parrot app that echoes HTTP headers).

The Symptom

The overall issue manifested as a trickle of occasional application errors that also appeared in a classic saw-tooth pattern during higher-load periods of time. This particular error had actually been appearing in our logs for quite some time. However, it wasn’t a particular area of concern: there were very few errors, and they had little impact on application performance because the API being protected by this setup is for a time-insensitive asynchronous operation that devices in our environment perform. Upon failure, the application will simply retry later, typically with success.

However, we recently began seeing an increase in this error rate among servers that are being upgraded as part of our routine OS upgrades. We wanted to rule out any possibility that these OS upgrades were the cause of the increased error rate, so we began to pay more attention to this problem. The tech-debt collector was knocking on our door at 3AM, and we were becoming fearful of our safety.

The application believed that certificate verification was failing. Specifically, the application expected to see the following headers set (this is a simplification, for the sake of this article):

If any of these checks failed, the application logged a generic error about certificate verification failing. This led to the initial assumption that the client certificate itself was not being verified properly on the load balancer.

These symptoms were compounded by two similar issues on different sides of the tech stack. The logging that we performed on our load balancer was limited: while we were collecting statistics and basic connection information, we weren’t logging specific details, such as client certificate information. The application-level logging was capturing data sent in the request body, but it wasn’t dumping any of the headers or information sent to it from the load balancer. We were initially flying blind.

Initial Troubleshooting

When we started troubleshooting, we could really only say a few things with certainty:

One of the headers that the application received was causing authentication to fail
The problem manifested more often with increased traffic loads

This wasn’t much to go on, and we could easily formulate a variety of highly unscientific theories:

The load balancer might be doing everything right, and the app servers could be mangling the headers
The client might occasionally be forgetting to send its certificate. Maybe it was hitting some code path that we didn’t think of.
The increased load might be triggering a (known or unknown) HAProxy or OpenSSL bug

It didn’t take long to recognize that we needed more logging on both sides of this equation. In this case, it was easier to bump up logging on HAProxy using a log formatter:

log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs {%[ssl_c_verify], %[ssl_fc_has_crt], %{+Q}[ssl_c_i_dn(cn)]} %{+Q}r"

HAProxy log formats are truly a Cthulhu-esque nightmare of unreadability, so if the above log line makes your brain hurt, then feel free to take a look at the docs. This is the standard HTTP log format, along with a few additional pieces of information (inside curly braces) that are made available via HAProxy variables:

ssl_c_verify: Error code from client certificate verification (or 0, if there is no error)
ssl_fc_has_crt: Indication of whether or not a client certificate was included with the connection.
ssl_c_i_dn(cn): The common name (CN) of the certificate issuer.

Sure enough, log messages started pouring in that correlated with the behavior that our application team was seeing. Specifically, we were seeing values indicating a lack of a client certificate (notice the 0,0 “App Intermediate”) within the curly braces:

2021-02-21T21:51:45.004358+00:00 local0.info acritelli-testbox-2 haproxy-example-ssl[3578986]: 10.72.4.147:57862 [21/Feb/2021:21:51:44.965] example-ssl~ example-ssl/<nosrv> -1/-1/-1/-1/39 400 211 - - CR-- 1/1/0/0/3 0/0 {0,0,"App Intermediate"} "GET / HTTP/1.1"</nosrv>

To compare the expected and logged values more clearly, here is what the above log line represents:

This seemed very strange: how could HAProxy be claiming that no certificate was being sent, yet the verification was passing and we could clearly see a value being populated for the issuer’s CN?

At this point, I personally started troubleshooting this issue (it had previously been handled by our on-call) because I had some passing familiarity with this environment. Like any good engineer who has spent time trying to solve a problem, I immediately started to Google for things that might be relevant. Since ssl_fc_has_cert was the relevant HAProxy variable, I started there and came across an issue logged against PFSense.

More Logging

I developed a hunch based on that PFSense issue, which directly quoted the HAProxy docs for the ssl_fc_has_cert variable:

Note: on SSL session resumption with Session ID or TLS ticket, client certificate is not present in the current connection but may be retrieved from the cache or the ticket. So prefer "ssl_c_used" if you want to check if current SSL session uses a client certificate.

There’s a feature in TLS called “session resumption” that behaves exactly like it’s described: with session resumption, a client is able to resume an existing connection with a server. Crucially, the client only sends certain authentication material (a ticket or session ID), and does not perform a complete handshake. Therefore, the client does not send its certificate. The session simply picks up where it left off.

When this happens, the ssl_fc_has_crt variable is correctly set to false, as the connection doesn’t include a certificate. However, ssl_c_verify remains true, because the client certificate was verified during previous connection attempts.

At this point, this was just a hunch, albeit one supported by documentation. How could we prove that TLS session resumption was really the cause of our issues? Thankfully, HAProxy also provides this as a variable that we can log: ssl_fc_is_resumed. As long as we were messing with the logging config, we figured we might as well log everything with a greatly expanded log formatter:

log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs {%{+Q}[ssl_c_key_alg],%{+Q}[ssl_c_sig_alg],%{+Q}[ssl_c_i_dn(cn)],%{+Q}[ssl_c_i_dn(c)],%{+Q}[ssl_c_i_dn(l)],%{+Q}[ssl_c_i_dn(o)],%{+Q}[ssl_c_i_dn(ou)],%{+Q}[ssl_c_i_dn(st)],%{+Q}[ssl_c_i_dn],%{+Q}[ssl_c_serial,hex],%{+Q}[ssl_c_version],%{+Q}[ssl_c_s_dn(cn)],%{+Q}[ssl_c_s_dn(c)],%{+Q}[ssl_c_s_dn(l)],%{+Q}[ssl_c_s_dn(o)],%{+Q}[ssl_c_s_dn(ou)],%{+Q}[ssl_c_s_dn(st)],%{+Q}[ssl_c_s_dn],%{+Q}[ssl_c_notafter],%{+Q}[ssl_c_notbefore],%{+Q}[ssl_fc_sni],%[ssl_c_verify],%[ssl_fc_has_crt],%[ssl_fc_is_resumed]} %{+Q}r"

Within a few minutes, the problem became apparent: sure enough, all of the application errors corresponded with cases where ssl_fc_is_resumed was 1, indicating that the TLS session was resumed and the client did not send a full handshake:

0 [21/Feb/2021:21:58:16.483] example-ssl~ example-ssl/<nosrv> -1/-1/-1/-1/39 400 211 - - CR-- 1/1/0/0/3 0/0 {"rsaEncryption","RSA-SHA256","App Intermediate","","","","","","/CN=App Intermediate","2B6E0CEECD2B26895A1A7923478B00477EC4ED21","3","client.example.com","","","","","","/CN=client.example.com","220211112822Z","210213232753Z","server.example.com",0,0,1} "GET / HTTP/1.1"</nosrv>

In the above log snippet, the final “1” right before the ending curly brace is the value of ssl_fc_is_resumed, which confirms that the session is being resumed. All of these cases correlated directly with application errors, so we knew that we found our culprit.

There are a few ways to solve this problem, but for reasons beyond the scope of this article, we decided to simply disable TLS session resumption by setting the no-tls-tickets flag and by disabling the SSL session cache (by setting its storage value to 0):

global
  tune.ssl.cachesize 0
listen example-ssl
  bind 0.0.0.0:443 ssl crt /etc/haproxy/pki/haproxy.pem ca-file /etc/haproxy/pki/ca.pem verify optional no-tls-tickets

Once the above configuration was in place, all of our errors immediately disappeared. On to the next on-call issue!

Digging Deeper

Before I started working in the Linux world, my professional background was network engineering (specifically, routing and switching). There’s three things that good network engineers love: protocol-level debugging, duct-taping TLVs onto the side of BGP instead of implementing better protocols, and talking about how we’re going to implement IPv6 someday. I happily left the world of networking before I got too deep into the latter two hobbies, but the former love of protocol and packet-level debugging has always stayed with me. Hardly a week goes by when I don’t fire up tcpdump to troubleshoot a strange issue. So while the previously described fix solved this issue in our production environment, I was curious to really see the protocol-level aspects of TLS session re-use in my lab.

I threw together a quick environment to reproduce this with, complete with a sanitized CA so that I could include logs and debug output in this article. My lab environment essentially matched the production environment from above, with a slightly simplified HAProxy configuration. I deployed a quick CA with Vault’s PKI and got to work digging into the protocol-level specifics of session resumption.

I’ve already mentioned that TLS supports either session identifiers or session tickets for session resumption. But what does each of this actually look like, and how can you test for them? Let’s take a look.

Without any configuration disabling it, HAProxy will support session tickets, as specified in RFC 5077. Session tickets allow the entire session state to be saved on the client, which provides an efficiency gain for the server.

We can test session resumption at the command line with openssl:

$ echo |  openssl s_client -CAfile ca.pem -cert client.pem -key key.pem -sess_out session.txt -state -no_tls1_3 -quiet -no_ign_eof -connect server.example.com:443

OpenSSL syntax can be a bit intimidating, so let me break this down:

openssl s_client: Use the generic TLS client included with OpenSSL to test a connection
-CAfile ca.pem: The CA used during server authentication and to construct the client certificate chain. In my lab, the same CA is used for both the server and client.
-cert_chain client.pem: The client’s certificate
-key: key.pem: The key for the client’s certificate
-sess_out: Writes the session out to a text file so that I can re-use it when re-establishing a session (see below)
-state: prints out log information about the session state
-no_tls1_3: There seems to be an issue with the way that OpenSSL behaves with TLS1.3 and session resumption, so this flag disables it so we can force it to work and observe its behavior.
-quiet: prevents the printing of verbose certificate and session information. If you’re doing this yourself, I’d recommend not including this flag. The entire output is worth seeing, but I’m omitting it in the interest of saving space.
-no_ign_eof: Tells OpenSSL to terminate the connection after the EOF sent by the echo command. This prevents needing to hit Ctrl+C to end the connection.
-connect server.example.com:443: The host and port to connect to

Running this command will produce a fairly typical mutual-authentication TLS handshake. Even without being familiar with the TLS handshake, it’s easy to follow based on the printed messages:

$ echo |  openssl s_client -CAfile ca.pem -cert client.pem -key key.pem -sess_out session.txt -state -no_tls1_3 -quiet -no_ign_eof -connect server.example.com:443
SSL_connect:before SSL initialization
SSL_connect:SSLv3/TLS write client hello
SSL_connect:SSLv3/TLS write client hello
SSL_connect:SSLv3/TLS read server hello
depth=1 CN = example.com
verify return:1
depth=0 CN = server.example.com
verify return:1
SSL_connect:SSLv3/TLS read server certificate
SSL_connect:SSLv3/TLS read server key exchange
SSL_connect:SSLv3/TLS read server certificate request
SSL_connect:SSLv3/TLS read server done
SSL_connect:SSLv3/TLS write client certificate
SSL_connect:SSLv3/TLS write client key exchange
SSL_connect:SSLv3/TLS write certificate verify
SSL_connect:SSLv3/TLS write change cipher spec
SSL_connect:SSLv3/TLS write finished
SSL_connect:SSLv3/TLS write finished
SSL_connect:SSLv3/TLS read server session ticket
SSL_connect:SSLv3/TLS read change cipher spec
SSL_connect:SSLv3/TLS read finished
DONE
SSL3 alert write:warning:close notify

In the HAProxy log, this connection attempt will produce a log message that looks like this:

2021-02-18T01:25:57.906880+00:00 local0.info acritelli-testbox-2 haproxy-example-ssl[1826329]: 10.72.4.124:45316 [18/Feb/2021:01:25:57.814] example-ssl~ example-ssl/<nosrv> -1/-1/-1/-1/90 400 211 - - CR-- 1/1/0/0/3 0/0 {"rsaEncryption","RSA-SHA256","App Intermediate","","","","","","/CN=App Intermediate","2B6E0CEECD2B26895A1A7923478B00477EC4ED21","3","client.example.com","","","","","","/CN=client.example.com","220211112822Z","210213232753Z","server.example.com",cert_verification:0,has_client_cert:1,session_is_resumed:0} "<badreq>"</badreq></nosrv>

This is obviously a verbose log message, so I’ve added some field labels to make the output more clear. Specifically, the cert verified successfully (cert_verification:0), a client cert was included in the request (has_client_cert:1), and the session has not been resumed (session_is_resumed:0).

OpenSSL even provides us with a way to inspect our session output. I’ve snipped the output below for brevity, but it should still give you a good idea of what this looks like:

$  openssl sess_id -noout -text -in session.txt | head -n 15
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: F77DDE8D5438BCB4140B91B5F87BD8AA1A54C49050146C687BE646225C3E0B96
    Session-ID-ctx:
    Master-Key: BD666A04C0B55C411E5C7269D649159FFD7AE26C851AC9997889C09706FDDA8C37B1AFC68E46F3E7B237431815FDA4B4
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:
    0000 - d1 b5 e6 a8 b5 d1 4d 1b-cf 4b 7b eb b0 c5 96 35   ......M..K{....5
    0010 - 30 e7 a0 1c 7e 03 df dc-db 34 15 53 be 21 9b ee   0...~....4.S.!..
    0020 - 84 05 68 d3 30 b6 3d 50-c4 0c e2 41 1c c8 56 b1   ..h.0.=P...A..V.
    0030 - 8f c3 a3 38 6a 29 cf 99-80 92 0e d5 7d 8f 3b 44   ...8j)......}.;D

I won’t cover the options passed to the sess_id command (you can read the manpage), but it should be fairly obvious what is going on here: OpenSSL is showing me the material that will be used to re-establish an existing session. Specifically, I can see the session ID and the TLS session ticket that will be sent to the server when I try to re-establish a session. Let’s try to establish a session using session reuse and see what that looks like:

$  echo |  openssl s_client -CAfile ca.pem -cert client.pem -key key.pem -sess_in session.txt -state -no_tls1_3 -quiet -no_ign_eof -connect server.example.com:443
SSL_connect:before SSL initialization
SSL_connect:SSLv3/TLS write client hello
SSL_connect:SSLv3/TLS write client hello
SSL_connect:SSLv3/TLS read server hello
SSL_connect:SSLv3/TLS read change cipher spec
SSL_connect:SSLv3/TLS read finished
SSL_connect:SSLv3/TLS write change cipher spec
SSL_connect:SSLv3/TLS write finished
DONE
SSL3 alert write:warning:close notify

Note that I’ve swapped the -sess_out flag for the -sess_in flag, since I want OpenSSL to re-use the session that I’ve written out to disk. As you can see above, this handshake process was much shorter than the first one. Fewer round trips can result in better performance, which is the entire benefit of session reuse.

To really dig into how this works, I can run the same command as above without the -quiet flag. This outputs quite a bit of verbose information, but if I scroll down in the output I can see the text below:

Reused, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: F77DDE8D5438BCB4140B91B5F87BD8AA1A54C49050146C687BE646225C3E0B96
    Session-ID-ctx:
    Master-Key: BD666A04C0B55C411E5C7269D649159FFD7AE26C851AC9997889C09706FDDA8C37B1AFC68E46F3E7B237431815FDA4B4
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:
    0000 - d1 b5 e6 a8 b5 d1 4d 1b-cf 4b 7b eb b0 c5 96 35   ......M..K{....5
    0010 - 30 e7 a0 1c 7e 03 df dc-db 34 15 53 be 21 9b ee   0...~....4.S.!..
    0020 - 84 05 68 d3 30 b6 3d 50-c4 0c e2 41 1c c8 56 b1   ..h.0.=P...A..V.
    0030 - 8f c3 a3 38 6a 29 cf 99-80 92 0e d5 7d 8f 3b 44   ...8j)......}.;D

That looks familiar, doesn’t it? It’s the session ticket that I previously displayed using openssl sess_id being sent to the server in lieu of a complete TLS handshake. On the HAProxy side of the connection, we can also see that the session is resumed:

[18/Feb/2021:01:31:38.208] example-ssl~ example-ssl/<nosrv> -1/-1/-1/-1/39 400 211 - - CR-- 1/1/0/0/3 0/0 {"rsaEncryption","RSA-SHA256","App Intermediate","","","","","","/CN=App Intermediate","2B6E0CEECD2B26895A1A7923478B00477EC4ED21","3","client.example.com","","","","","","/CN=client.example.com","220211112822Z","210213232753Z","server.example.com",cert_verification:0,has_client_cert:0,session_is_resumed:1} "<badreq>"</badreq></nosrv>

Notice that the certificate has been verified for the session (cert_verification:0), but now no client certificate has been sent (has_client_cert:0) because the session is resumed (session_is_resumed:1).

At this point, you probably get the idea of how TLS session resumption works. I won’t repeat this entire exercise with tickets disabled, as you probably get the gist of how this works. Without tickets, the client will simply pass a session ID to the server, and the server will be responsible for resuming the session based on this ID.

However, I will demonstrate what this looks like with session resumption completely disabled, which was our solution to the issue we were experiencing in production. As a reminder, TLS session resumption can be disabled completely by setting the cache size to 0 and disabling TLS tickets on the server bind:

global
  tune.ssl.cachesize 0
listen example-ssl
  bind 0.0.0.0:443 ssl crt /etc/haproxy/pki/haproxy.pem ca-file /etc/haproxy/pki/ca.pem verify optional no-tls-tickets

To confirm that session resumption is no longer possible, I can pass the -reconnect option to openssl s_client, which will attempt to reconnect to the server 5 times with the same session ID to test session resumption:

$  openssl s_client -CAfile ca.pem -cert client.pem -key key.pem -sess_out session.txt -state -no_tls1_3 -quiet -no_ign_eof -connect server.example.com:443 -reconnect
SSL_connect:before SSL initialization
SSL_connect:SSLv3/TLS write client hello
SSL_connect:SSLv3/TLS write client hello
SSL_connect:SSLv3/TLS read server hello
depth=1 CN = example.com
verify return:1
depth=0 CN = server.example.com
verify return:1
SSL_connect:SSLv3/TLS read server certificate
SSL_connect:SSLv3/TLS read server key exchange
SSL_connect:SSLv3/TLS read server certificate request
SSL_connect:SSLv3/TLS read server done
SSL_connect:SSLv3/TLS write client certificate
SSL_connect:SSLv3/TLS write client key exchange
SSL_connect:SSLv3/TLS write certificate verify
SSL_connect:SSLv3/TLS write change cipher spec
SSL_connect:SSLv3/TLS write finished
SSL_connect:SSLv3/TLS write finished
SSL_connect:SSLv3/TLS read change cipher spec
SSL_connect:SSLv3/TLS read finished
SSL3 alert write:warning:close notify
SSL_connect:before SSL initialization
SSL_connect:SSLv3/TLS write client hello
SSL_connect:SSLv3/TLS write client hello
SSL_connect:SSLv3/TLS read server hello
depth=1 CN = example.com
verify return:1
depth=0 CN = server.example.com
verify return:1
SSL_connect:SSLv3/TLS read server certificate
SSL_connect:SSLv3/TLS read server key exchange
SSL_connect:SSLv3/TLS read server certificate request
SSL_connect:SSLv3/TLS read server done
SSL_connect:SSLv3/TLS write client certificate
SSL_connect:SSLv3/TLS write client key exchange
SSL_connect:SSLv3/TLS write certificate verify
SSL_connect:SSLv3/TLS write change cipher spec
SSL_connect:SSLv3/TLS write finished
SSL_connect:SSLv3/TLS write finished
SSL_connect:SSLv3/TLS read change cipher spec
SSL_connect:SSLv3/TLS read finished
SSL3 alert write:warning:close notify

< handshake repeats 3 more times >

The above snippet shows a complete TLS handshake for each connection, indicating that OpenSSL is unable to use session resumption. Additionally, we can see that the -sess_out flag had no effect, and no session information was written to disk:

$  stat session.txt
stat: cannot stat 'session.txt': No such file or directory

While this is exactly what you would expect, it’s still important to appreciate what is going on here: with the session cache disabled by HAProxy, no session resumption can occur. Therefore, the full TLS handshake occurs on every connection.

Wrapping Up

I said in the introduction that this was a relatively simple problem if you know your way around TLS. In short: TLS session resumption exists, and you have to be mindful of it when you’re configuring your load balancers and the way they pass that information along to backend apps. But as they say: it’s about the journey, not the destination (I don’t say this, because I hate long car rides). I hope that you enjoyed learning about the process of troubleshooting this issue, and perhaps have a few new OpenSSL commands in your debugging Swiss Army Knife.

Interested in working on fun problems like this? We’re hiring.