I’ve recently started using the Dispatch library for HTTP/HTTPS, which is quite a nice library, as long as you don’t need documentation. Dispatch uses the Ning/Sonatype AsyncHttpClient library, which is also quite nice, and although AsyncHttpClient is a library which I could recommend, it does have an insecure-by-default implementation of SSL. This post is a quick discussion of the AsyncHttpClient defaults and how to implement certificate verification to increase the security provided by SSL.
The information in this post is outdated. Thanks to the efforts of the Async
Http Client team, hostname validation was enabled by default in commit
3c9152e
from pull request
#510, which is
included in 2.0.0-alpha9 and later. The fix was also backported to
1.9.0-BETA1 in commit
a894583.
If you are using Async Http Client 1.9.0 or later, there is no need to use the
MyHostnameVerifier
class described in this post.
Background
This post will assume some familiarity with SSL and the need to verify certificates. If readers are unfamiliar with either of these topics, there are many online resources available and you are encouraged to explore the topic.
For background on how SSL is implemented on the Java platform, see the Java Secure Socket Extension (JSSE) Reference Guide. Readers who are not concerned with the implementation details may feel free to skip to the end of this article for “recipes” for SSL certificate verification.
Default SSLContext
The
SSLContext
class is central to the SSL implementation in Java in general and in
AsyncHttpClient in particular. The default SSLContext
for AsyncHttpClient
is dependent on whether the javax.net.ssl.keyStore
system property is set.
If this property is set, AsyncHttpClient will create a TLS SSLContext
with a
KeyManager
based on the specified key store (and configured based on the
values of many other javax.net.ssl
properties as described in the JSEE
Reference Guide linked above). Otherwise, it will create a TLS SSLContext
with no KeyManager
and a TrustManager
which accepts everything. In
effect, if javax.net.ssl.keyStore
is unspecified, any ol’ SSL certificate
will do.
If the trusted Certificate Authorities for the application should be the same
as the trusted CAs for the operating system, it is possible to avoid the
hassles of dealing with Java key stores by using the (JRE) default
SSLContext
. Simply instantiate a new SSLContext
and initialize it with
all null
values. This offloads the burden to the JRE provider and OS vendor
and works like a charm on my test system.
Unfortunately, there does not appear to be a way to set the default
SSLContext
used by AsyncHttpClient. Instead, applications must set their
preferred SSLContext
for each connection.
Default HostnameVerifier
Even if the SSLContext
can verify that a certificate is signed by a trusted
Certificate Authority, there is still room for problems. What happens if the
connection hostname doesn’t match the certificate hostname? Java provides the
HostnameVerifier
interface to give client code the option of providing a policy for handling
this situations. AsyncHttpClient adopts this interface for this purpose as
well. However, unlike the JDK, the default policy provided by AsyncHttpClient
is to allow all connections regardless of hostname.
Unlike SSLContext
, using the Java default
(HttpsURLConnection.getDefaultHostnameVerifier
)
is not a viable option because the default HostnameVerifier
expects to only
be called in the case that there is a mismatch (and therefore always returns
false
) while some of the AsyncHttpClient providers (e.g. Netty, the default)
call it on all
connections. To
make matters worse, the check is not trivial (consider SAN and wildcard matching) and is implemented in
sun.security.util.HostnameChecker
(a Sun internal proprietary API). This leaves the developer in the position
of either depending on an internal API or finding/copying/creating another
implementation of this functionality. For the examples in this article, I
have opted for the first option.
Unfortunately, as with SSLContext
, there does not appear to be a way to set
the default HostnameVerifier
used by AsyncHttpClient. Instead, applications
must set their preferred HostnameVerifier
for each connection.
Implementation
First, a quick note: The purpose of these example implementations is to demonstrate how to verify certificates. The programs should include better exception handling, logging, and a more modular functional decomposition, but this would lengthen the examples and obscure their core purpose. Please feel free to use your better judgement when copying and expanding on these examples.
Scala
First, an example program which downloads a given URL using Dispatch in Scala:
/* An example program using Dispatch with SSL certificate verification
*
* To the extent possible under law, Kevin Locke has waived all copyright and
* related or neighboring rights to this work.
*/
import com.ning.http.client.{AsyncHttpClient, AsyncHttpClientConfig}
import dispatch._
import java.security.cert.{CertificateException, X509Certificate}
import javax.net.ssl.{HostnameVerifier, SSLPeerUnverifiedException, SSLSession}
import javax.security.auth.kerberos.KerberosPrincipal
import sun.security.util.HostnameChecker
/** HostnameVerifier implementation which implements the same policy as the
* Java built-in pre-HostnameVerifier policy.
*/
object MyHostnameVerifier extends HostnameVerifier {
/** Checks if a given hostname matches the certificate or principal of a
* given session.
*/
private def hostnameMatches(hostname: String, session: SSLSession): Boolean = {
val checker = HostnameChecker.getInstance(HostnameChecker.TYPE_TLS);
try {
session.getPeerCertificates match {
case Array(cert: X509Certificate, _*) =>
try {
checker.`match`(hostname, cert)
// Certificate matches hostname
true
} catch {
case _: CertificateException =>
// Certificate does not match hostname
false
}
case _ =>
// Peer does not have any certificates or they aren't X.509
false
}
} catch {
case _: SSLPeerUnverifiedException =>
// Not using certificates for verification, try verifying the principal
try {
session.getPeerPrincipal match {
case principal: KerberosPrincipal =>
HostnameChecker.`match`(hostname, principal)
case _ =>
// Can't verify principal, not Kerberos
false
}
} catch {
case _: SSLPeerUnverifiedException =>
// Can't verify principal, no principal
false
}
}
}
def verify(hostname: String, session: SSLSession): Boolean = {
if (hostnameMatches(hostname, session)) {
true
} else {
// TODO: Add application-specific checks for hostname/certificate match
false
}
}
}
/** Extension of Http which uses an AsyncHttpClient configured with our
* customized SSLContext and HostnameVerifier
*/
object MyHttp extends Http {
override lazy val client = new AsyncHttpClient(
new AsyncHttpClientConfig.Builder()
.setSSLContext({
val ctx = javax.net.ssl.SSLContext.getInstance("TLS")
ctx.init(null, null, null)
ctx
})
.setHostnameVerifier(MyHostnameVerifier)
.build
)
}
/** Implements the "MyDownloader" application */
object MyDownloader {
def main(args: Array[String]) {
args match {
case Array(url) =>
val request = dispatch.url(url)
Console.err.println("Downloading " + url)
MyHttp(request OK as.String).either() match {
case Left(e) =>
// Something failed
Console.err.println("Failure downloading " + url + ": " + e)
case Right(content) =>
// Success
Console.err.println("Successfully downloaded " + url)
Console.out.println(content)
}
MyHttp.shutdown()
case _ =>
Console.err.println("Usage: myhttp <URL>")
System.exit(1)
}
}
}
The above code is also available as part of a GitHub Gist.
Java
Then, the same program using AsyncHttpClient directly from Java:
/* An example program using AsyncHttpClient with SSL certificate verification
*
* To the extent possible under law, Kevin Locke has waived all copyright and
* related or neighboring rights to this work.
* A legal description of this waiver is available in LICENSE.txt.
*/
import com.ning.http.client.AsyncHttpClient;
import com.ning.http.client.AsyncHttpClientConfig;
import com.ning.http.client.Response;
import java.io.IOException;
import java.security.KeyManagementException;
import java.security.NoSuchAlgorithmException;
import java.security.Principal;
import java.security.cert.Certificate;
import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;
import java.util.concurrent.ExecutionException;
import javax.net.ssl.HostnameVerifier;
import javax.net.ssl.SSLContext;
import javax.net.ssl.SSLPeerUnverifiedException;
import javax.net.ssl.SSLSession;
import javax.security.auth.kerberos.KerberosPrincipal;
import sun.security.util.HostnameChecker;
/** Implements the "MyDownloader" application */
public class MyDownloader {
/** HostnameVerifier implementation which implements the same policy as the
* Java built-in pre-HostnameVerifier policy.
*/
private static class MyHostnameVerifier implements HostnameVerifier {
/** Checks if a given hostname matches the certificate or principal of
* a given session.
*/
private boolean hostnameMatches(String hostname, SSLSession session) {
HostnameChecker checker =
HostnameChecker.getInstance(HostnameChecker.TYPE_TLS);
boolean validCertificate = false, validPrincipal = false;
try {
Certificate[] peerCertificates = session.getPeerCertificates();
if (peerCertificates.length > 0 &&
peerCertificates[0] instanceof X509Certificate) {
X509Certificate peerCertificate =
(X509Certificate)peerCertificates[0];
try {
checker.match(hostname, peerCertificate);
// Certificate matches hostname
validCertificate = true;
} catch (CertificateException ex) {
// Certificate does not match hostname
}
} else {
// Peer does not have any certificates or they aren't X.509
}
} catch (SSLPeerUnverifiedException ex) {
// Not using certificates for peers, try verifying the principal
try {
Principal peerPrincipal = session.getPeerPrincipal();
if (peerPrincipal instanceof KerberosPrincipal) {
validPrincipal = HostnameChecker.match(hostname,
(KerberosPrincipal)peerPrincipal);
} else {
// Can't verify principal, not Kerberos
}
} catch (SSLPeerUnverifiedException ex2) {
// Can't verify principal, no principal
}
}
return validCertificate || validPrincipal;
}
public boolean verify(String hostname, SSLSession session) {
if (hostnameMatches(hostname, session)) {
return true;
} else {
// TODO: Add application-specific checks for
// hostname/certificate match
return false;
}
}
}
public static void main(String[] args) {
if (args.length != 1) {
System.err.println("Usage: myhttp <URL>");
} else {
String url = args[0];
SSLContext context = null;
try {
context = SSLContext.getInstance("TLS");
} catch (NoSuchAlgorithmException e) {
e.printStackTrace();
return;
}
try {
context.init(null, null, null);
} catch (KeyManagementException e) {
e.printStackTrace();
return;
}
AsyncHttpClient client = new AsyncHttpClient(
new AsyncHttpClientConfig.Builder()
.setSSLContext(context)
.setHostnameVerifier(new MyHostnameVerifier())
.build()
);
Response response = null;
try {
response = client.prepareGet(url).execute().get();
} catch (InterruptedException e) {
e.printStackTrace();
return;
} catch (ExecutionException e) {
e.printStackTrace();
return;
} catch (IOException e) {
e.printStackTrace();
return;
}
if (response.getStatusCode() / 100 == 2) {
try {
String responseBody = response.getResponseBody();
System.err.println("Successfully downloaded " + url);
System.out.println(responseBody);
} catch (IOException e) {
e.printStackTrace();
return;
}
} else {
System.err.println("Failure downloading " + url +
": HTTP Status " + response.getStatusCode());
}
}
}
}
The above code is also available as part of a GitHub Gist.
Caveats
Although the default initialization of SSLContext
works quite well on my
test machine, I have not found a clear specification of its behavior and it
may not be guaranteed to use/trust the operating system certificate store on
all platforms.
In my testing I found that if the invalid certificate is returned by a process
on the local machine (e.g. using mitmproxy)
AsyncHttpClient will throw a java.io.IOException: Remotely Closed
rather
than java.net.ConnectException: General SSLEngine problem
. This is probably
a bug. In either
case, users should be wary of this behavior when troubleshooting a failing
SSL/TLS connection.