Notes on Mac Remote Desktop Connection Encryption Errors

I pretty much live inside RDP, so getting almost constant “encryption errors” when trying to work on my xrdp machines from my Mac has been driving me crazy, because it instantly terminates the connection and completely breaks my flow.

Now, the Mac Remote Desktop client is… well, a work in progress. It regularly crashes when I reconnect to corporate AVD (especially if I have gone idle and was automatically disconnected, like at every lunch break), occasionally crashes while reconnecting to a session I started on another machine, very occasionally crashes when just… disconnecting, etc.

It’s great when it works, awesome when I have GPU acceleration configured on a server, but I am often reminded of how it could be better, and that certainly applies when trying to connect to xrdp.

So today I tried to be a little systematic about finding a solution for it, and since there doesn’t really seem to be a lot of knowledge/awareness on what this actually is, I decided to jot down my notes and the “solution” that seems to work for me (sample size of one for now).

I started out by looking at xrdp logs:

# typical connection start - these lines are not consecutive
[INFO ] Security protocol: configured [SSL|RDP], requested [SSL|HYBRID|RDP], selected [SSL]
[INFO ] TLS connection established from [client]:49866 TLSv1.3 with cipher TLS_AES_256_GCM_SHA384

# typical error sequence for a disconnect
[ERROR] SSL_read: Failure in SSL library (protocol error?)
[ERROR] SSL: error:0A000126:SSL routines::unexpected eof while reading

# another
[ERROR] SSL_read: Failure in SSL library (protocol error?)
[ERROR] SSL: error:0A0003FC:SSL routines::ssl/tls alert bad record mac

# and another
[ERROR] libxrdp_force_read: header read error
[ERROR] [ITU-T X.224] Connection Sequence: CR-TPDU (Connection Request) failed
[ERROR] xrdp_sec_incoming: xrdp_iso_incoming failed
[ERROR] xrdp_rdp_incoming: xrdp_sec_incoming failed

The iOS client works perfectly (as in, zero disconnects to the same machines), negotiates encryption in the same way (and often negotiates video acceleration better), so given the logs above I have to assume that there is a client-side bug with SSL handling that is somehow exclusive to the Mac builds.

The fact that the error happens at various different stages (during connection setup or after) and is always logged on the server side as an incoming thing (reading, checksumming a message, etc.) also points to some sort of client-side issue.

The Mac client (like the iOS one) exposes very little (if any) low-level parameters to control session encryption, but I had a look at the .rdp file it exported for one of my consistently failing connections and it had a strange setting that I couldn’t find documented anywhere:

allowed security protocols:s:*

This made me wonder what I could do server-side, so I spent 15 minutes looking at all of my xrdp servers (the only real difference is in one where I have an egfx acceleration patch, but that has zero impact on security).

These are the current defaults for xrdp.ini across Fedora, Ubuntu and Debian:

; security layer can be 'tls', 'rdp' or 'negotiate'
; for client compatible layer
security_layer=negotiate
; minimum security level allowed for client for classic RDP encryption
; use tls_ciphers to configure TLS encryption
; can be 'none', 'low', 'medium', 'high', 'fips'
crypt_level=high

Nobody has used native RDP encryption for ages, but I decided to see what happened if I configured my server like this:

security_layer=tls
crypt_level=medium

Sure enough, negotiation was a little different–but, surprisingly, the cipher didn’t change:

[INFO ] Security protocol: configured [SSL], requested [SSL|HYBRID|HYBRID_EX|RDP], selected [SSL]
[INFO ] TLS connection established from [client]:51012 TLSv1.3 with cipher TLS_AES_256_GCM_SHA384

But, even more surprisingly, that session never failed. It stayed on, perfectly stable, for almost eight hours (whereas before it failed minutes or seconds after reconnecting).

I don’t know exactly why this seemingly worked, but I decided to try to understand how cipher selection worked, and man xrdp.ini has this to say about it:

medium All data sent between the client and the server is protected by encryption based on the maximum key strength supported by the client (client compatible).

high All data sent between the client and the server is protected by encryption based on the server’s maximum key strength (sever compatible).

Since I am always getting TLS_AES_256_GCM_SHA384, it seems that both server and client are in agreement here, and that there is no security degradation whatsoever by using medium.

I have no real clue as to why this apparently worked (and to be fair, I probably need to use this for a few more days to confirm it is a real fix), but I think there might be some sort of session setup bug during security protocol negotiation that causes the client to configure SSL in such a way that it eventually fails later on. It might be either in the initial security mode selection or (more likely) during cipher selection.

It would be a lot more obvious if the negotiated cipher were completely different, though.

I suspect I may not have seen the end of this, but at least I was able to focus on doing actual work for a whole afternoon instead of getting interrupted by a modal dialog and having to babysit the connection.