OpenSSL is a widely used library for SSL and TLS protocol implementation that secures data using encryption and decryption based on cryptographic functions. However, a Security Bypass vulnerability – recently addressed in a patch by the OpenSSL Project –can be exploited to make vulnerable SSL clients or remote SSL servers send clean application data without encryption.
This Security Bypass vulnerability (CVE-2017-3737) is caused by an error when the SSL_read or SSL_write function handles an "error state" during an SSL handshake. In this paper the FortiGuard Labs team examines the root cause of this vulnerability.
The "error state" mechanism was introduced in OpenSSL beginning with version 1.0.2b, It is used to make OpenSSL move into an error state whenever a fatal error occurs during the SSL handshake that would fail if the SSL handshake continued. If SSL_read or SSL_write function is called directly, it checks the SSL handshake state and performs a new SSL handshake automatically if no handshake has been initiated. If a fatal error occurs during the SSL handshake, OpenSSL moves into the error state and returns an error message to the caller. However, the problem occurs if the caller doesn't check the error state and simply calls the SSL_read or SSL_write function again, because it then sends application data without encryption.
The following code snippet was taken from OpenSSL 1.0.2m. (Comments added by me have been highlighted.)
ssl3_write_bytes() is called by the SSL_write function to send the application data. It checks the SSL handshake state and performs the SSL handshake if needed.
We used a vulnerable SSL client as a target during the test. During the SSL handshake it received a malformed server hello message from the SSL server controlled by the attacker. ssl3_get_server_hello() is called to handle this server hello message and a fatal error occurs, causing OpenSSL to move into an error state by setting s->state from SSL_ST_INIT to SSL_ST_ERR.
If the vulnerable SSL client doesn't check the error state and call SSL_write function to send application data again, ssl3_write_bytes() is called and uses SSL_in_init() to check the handshake state again.
This time the check fails, the SSL handshake is bypassed and the application data will be sent without encryption.
The following traffic dump shows how the clean application data is sent:
During this attack, the attacker entices the vulnerable SSL client to connect to a malicious SSL server. The SSL client may bypass the handshake process and send the application data without encryption. The SSL server may also have the same vulnerability if SSL_read or SSL_write function is called directly.
NOTE: authentication is NOT required to exploit this vulnerability.