Infinite loop in BN_mod_sqrt() reachable when parsing certificates
(High)
The BN_mod_sqrt() function, which computes a modular square root,
contains a bug that can cause it to loop forever for non-prime
moduli.
Internally this function is used when parsing certificates that
contain elliptic curve public keys in compressed form or explicit
elliptic curve parameters with a base point encoded in compressed
form.
It is possible to trigger the infinite loop by crafting a
certificate that has invalid explicit curve parameters.
Since certificate parsing happens prior to verification of the
certificate signature, any process that parses an externally
supplied certificate may thus be subject to a denial of service
attack. The infinite loop can also be reached when parsing crafted
private keys as they can contain explicit elliptic curve
parameters.
Thus vulnerable situations include:
- TLS clients consuming server certificates
- TLS servers consuming client certificates
- Hosting providers taking certificates or private keys from
customers
- Certificate authorities parsing certification requests from
subscribers
- Anything else which parses ASN.1 elliptic curve parameters
Also any other applications that use the BN_mod_sqrt() where the
attacker can control the parameter values are vulnerable to this DoS
issue.