Invalid handling of X509_verify_cert() internal errors in libssl
(Moderate)
Internally libssl in OpenSSL calls X509_verify_cert() on the client
side to verify a certificate supplied by a server. That function may
return a negative return value to indicate an internal error (for
example out of memory). Such a negative return value is mishandled by
OpenSSL and will cause an IO function (such as SSL_connect() or
SSL_do_handshake()) to not indicate success and a subsequent call to
SSL_get_error() to return the value SSL_ERROR_WANT_RETRY_VERIFY.
This return value is only supposed to be returned by OpenSSL if the
application has previously called SSL_CTX_set_cert_verify_callback().
Since most applications do not do this the
SSL_ERROR_WANT_RETRY_VERIFY return value from SSL_get_error() will be
totally unexpected and applications may not behave correctly as a
result. The exact behaviour will depend on the application but it
could result in crashes, infinite loops or other similar incorrect
responses.