GNU Transport Layer Security Library

Janne Snabb | 29 May 2012 16:46

Big CA certificate bundle causes problems with GnuTLS 3.0.11

I am experiencing a TLS handshake problem when GnuTLS 3.0.11 server has
a big pile of CA certificates to verify against. I can not reproduce the
problem with GnuTLS 2.12.14.

Steps to re-produce:

1. Create server key+certificate:

certtool --generate-privkey --outfile foo.key
certtool --generate-self-signed --load-privkey foo.key --outfile foo.crt

(leave all fields empty except expiration and enable signing and
encryption)

2. Start server:

gnutls-serv --x509keyfile foo.key --x509certfile foo.crt --x509cafile
/etc/ssl/certs/ca-certificates.crt

3. Connect with client and observe failure:

gnutls-cli --insecure -p 5556 localhost

4. Start server without CA cert bundle:

gnutls-serv --x509keyfile foo.key --x509certfile foo.crt

5. Connect with client and observe success:

gnutls-cli --insecure -p 5556 localhost

(Continue reading)

Phil Pennock | 29 May 2012 17:31

Re: Big CA certificate bundle causes problems with GnuTLS 3.0.11

On 2012-05-29 at 21:46 +0700, Janne Snabb wrote:
> I am experiencing a TLS handshake problem when GnuTLS 3.0.11 server has
> a big pile of CA certificates to verify against. I can not reproduce the
> problem with GnuTLS 2.12.14.

It appears to be commit 67f4dba6 from March 20th:
"Avoided waiting for peer's retransmission to ensure receipt of finished
 messages, and used a 'timer'-like to retransmit packets."

-  data_size = _mbuffer_get_udata_size(bufel) - handshake_header_size;
+  if (hsk->length > 0 && 
+        (hsk->end_offset-hsk-≥start_offset >=  data_size))

> |<3>| HSK[0x24e4120]: CERTIFICATE REQUEST was received. Length
> 17025[16380], frag offset 0, frag length: 17025, sequence: 0
> |<2>| ASSERT: gnutls_buffers.c:819
> |<2>| ASSERT: gnutls_buffers.c:1031
> |<2>| ASSERT: gnutls_handshake.c:1269
> |<2>| ASSERT: gnutls_handshake.c:2515
> *** Fatal error: A TLS packet with unexpected length was received.

The "was received" code is:
----------------------------8< cut here >8------------------------------
  _gnutls_handshake_log ("HSK[%p]: %s was received. Length %d[%d], frag offset %d, frag length: %d,
sequence: %d\n",
                         session, _gnutls_handshake2str (hsk->htype),
                         (int) hsk->length, (int)data_size, hsk->start_offset, hsk->end_offset-hsk-≥start_offset+1, (int)hsk->sequen
ce);
----------------------------8< cut here >8------------------------------

(Continue reading)

Phil Pennock | 29 May 2012 17:39

Re: Big CA certificate bundle causes problems with GnuTLS 3.0.11

On 2012-05-29 at 11:31 -0400, Phil Pennock wrote:
> It appears to be commit 67f4dba6 from March 20th:

*cough* March 20th, 2011.  Sorry.

Branches:
  master, remotes/origin/ecc, remotes/origin/gnutls_3_0_12_x,
  remotes/origin/gnutls_3_0_x, remotes/origin/gnutls_3_0_x-2,
  remotes/origin/master, remotes/origin/ocsp

First tag to have this was gnutls_2_99_0.

-Phil

Nikos Mavrogiannopoulos | 29 May 2012 22:34

Re: Big CA certificate bundle causes problems with GnuTLS 3.0.11

On 05/29/2012 05:31 PM, Phil Pennock wrote:

> On 2012-05-29 at 21:46 +0700, Janne Snabb wrote:
>> I am experiencing a TLS handshake problem when GnuTLS 3.0.11 server has
>> a big pile of CA certificates to verify against. I can not reproduce the
>> problem with GnuTLS 2.12.14.
[...]
> hsk->length is read from the Handshake->length (uint24); data_size is
> the size of the CertificateRequest (received buffer size less 4 for the
> handshake header (type 1 octet, length 3 octets).
> hsk->start_offset is always 0.
> hsk->end_offset is always (hsk->length - 1) [because this isn't DTLS].
> So the check added in 67f4dba6 is going to always reject a fragmented
> handshake packet.

Correct. I've committed a fix at:
http://git.savannah.gnu.org/gitweb/?p=gnutls.git;a=commitdiff;h=6299e8a8c7371da1e674419c36cbcbe1630aef0a
regards,
Nikos

Michal Suchanek | 29 May 2012 22:37

Picon

Re: Big CA certificate bundle causes problems with GnuTLS 3.0.11

On 29 May 2012 17:31, Phil Pennock <help-gnutls-phil <at> spodhuis.org> wrote:
> On 2012-05-29 at 21:46 +0700, Janne Snabb wrote:
>> I am experiencing a TLS handshake problem when GnuTLS 3.0.11 server has
>> a big pile of CA certificates to verify against. I can not reproduce the
>> problem with GnuTLS 2.12.14.
>
> It appears to be commit 67f4dba6 from March 20th:
> "Avoided waiting for peer's retransmission to ensure receipt of finished
>  messages, and used a 'timer'-like to retransmit packets."
>
> -  data_size = _mbuffer_get_udata_size(bufel) - handshake_header_size;
> +  if (hsk->length > 0 &&
> +        (hsk->end_offset-hsk-≥start_offset >=  data_size))
>
>> |<3>| HSK[0x24e4120]: CERTIFICATE REQUEST was received. Length
>> 17025[16380], frag offset 0, frag length: 17025, sequence: 0
>> |<2>| ASSERT: gnutls_buffers.c:819
>> |<2>| ASSERT: gnutls_buffers.c:1031
>> |<2>| ASSERT: gnutls_handshake.c:1269
>> |<2>| ASSERT: gnutls_handshake.c:2515
>> *** Fatal error: A TLS packet with unexpected length was received.
>
> The "was received" code is:
> ----------------------------8< cut here >8------------------------------
>  _gnutls_handshake_log ("HSK[%p]: %s was received. Length %d[%d], frag offset %d, frag length: %d,
sequence: %d\n",
>                         session, _gnutls_handshake2str (hsk->htype),
>                         (int) hsk->length, (int)data_size, hsk->start_offset,
hsk->end_offset-hsk-≥start_offset+1, (int)hsk->sequen
> ce);

(Continue reading)

Nikos Mavrogiannopoulos | 29 May 2012 22:48

Picon

Re: Big CA certificate bundle causes problems with GnuTLS 3.0.11

On 05/29/2012 10:37 PM, Michal Suchanek wrote:

>> hsk->start_offset is always 0.
>> hsk->end_offset is always (hsk->length - 1) [because this isn't DTLS].
>>
>> So the check added in 67f4dba6 is going to always reject a fragmented
>> handshake packet.

> Now what I do not get is how a pile of CA certificates is fragmenting

> the packets.

In the TLS protocol the server advertises its CA certificates so a
client would know which certificate to present. If a server trusts all
the certificates in the system, the server would advertise all of them
(their DNs actually).

regards,
Nikos

Sam Varshavchik | 30 May 2012 01:04

Re: Big CA certificate bundle causes problems with GnuTLS 3.0.11

Nikos Mavrogiannopoulos writes:

> In the TLS protocol the server advertises its CA certificates so a
> client would know which certificate to present. If a server trusts all
> the certificates in the system, the server would advertise all of them
> (their DNs actually).

IIRC, this occurs only when client certificates are enabled. Yes, I would  
think that in that case only the certificates that the server trusts, should  
be loaded.

I would think that in most situations an organization would have only their  
own CA trusted, and loaded into a TLS server. I suppose I can imagine a  
situation where an org would, in some context, decide that accepting a  
client cert signed by any public CA to be acceptable, and then use it for  
some particular purpose. But I don't think this would be the case for most  
practical situations. And, in those cases, loading the public CA certs would  
be a security hole. Depending on the purpose the client cert is being used  
for, and how, it wouldn't take much imagination to get some public CA to  
sign something that looks good enough to 0wn JOO.

So, I would take a hard look at why someone really wants to load a public CA  
bundle, in the first place, to validate client certs.

I suppose someone might want, for some odd reason, to blow a wad of cash on  
having some public CA sign some certs, for their clients, even though it's  
trivial to set up your own cert, and do it yourself for free. But, still, in  
that case, at the very least you should only load /that/ CA, and not the  
whole bundle.

(Continue reading)

Janne Snabb | 30 May 2012 05:10

Re: Big CA certificate bundle causes problems with GnuTLS 3.0.11

On Tue, 29 May 2012, Sam Varshavchik wrote:

> I suppose someone might want, for some odd reason, to blow a wad of cash on
> having some public CA sign some certs, for their clients, even though it's
> trivial to set up your own cert, and do it yourself for free. But, still, in
> that case, at the very least you should only load /that/ CA, and not the whole
> bundle.

Google is one big e-mail sender that presents a client certificate signed
by one of the ~150 "well-known" CAs (I have not checked which one). There
are other similar but smaller mail senders also.

--
Janne Snabb / EPIPE Communications
snabb <at> epipe.com - http://epipe.com/

Phil Pennock | 30 May 2012 05:47

Re: Big CA certificate bundle causes problems with GnuTLS 3.0.11

On 2012-05-30 at 03:10 +0000, Janne Snabb wrote:
> Google is one big e-mail sender that presents a client certificate signed
> by one of the ~150 "well-known" CAs (I have not checked which one). There
> are other similar but smaller mail senders also.

Equifax, apparently:

52394 SSL verify ok: depth=2 cert=/C=US/O=Equifax/OU=Equifax Secure Certificate Authority
52394 SSL verify ok: depth=1 cert=/C=US/O=Google Inc/CN=Google Internet Authority
52394 SSL peer: /C=US/ST=California/L=Mountain View/O=Google Inc/CN=smtp.gmail.com

Hrm, Exim needs a +tls_peer_issuerdn log selector.

-Phil

Janne Snabb | 29 May 2012 23:17

Re: Big CA certificate bundle causes problems with GnuTLS 3.0.11

On 2012-05-30 03:37, Michal Suchanek wrote:
> Now what I do not get is how a pile of CA certificates is fragmenting
> the packets.
> 
> Sounds like a security hole. CA cert piles should be local to either
> side, only one CA cert relevant for the session. Technically there can
> be more than one cert in the trust chain but not pile of them.

If the *server* chooses to trust a pile of CA's in the same way as web
browsers (clients) typically do, this will happen, see:

https://tools.ietf.org/html/rfc5246#section-7.4.4

It also says:

"If the certificate_authorities list is empty, then the client MAY send
any certificate of the appropriate ClientCertificateType, unless there
is some external arrangement to the contrary."

...which suggests that in cases like this it might be a good idea or at
least acceptable *not* to put anything in the certificate_authorities
list when the server sends the Certificate Request. It is unclear how
various client side implementations implement the "MAY" part of the
above RFC quote. Do they send a client certificate if the CA list is
empty? Which one will they send if they have several?

It feels like there should be a way in the GnuTLS API to define whether
the list of trusted CAs is to be advertised in Certificate Request or
not. (Maybe there is a way but I am missing it?)

(Continue reading)

Nikos Mavrogiannopoulos | 29 May 2012 23:24

Picon

Re: Big CA certificate bundle causes problems with GnuTLS 3.0.11

On 05/29/2012 11:17 PM, Janne Snabb wrote:

> On 2012-05-30 03:37, Michal Suchanek wrote:
>> Now what I do not get is how a pile of CA certificates is fragmenting
>> the packets.
>>
>> Sounds like a security hole. CA cert piles should be local to either
>> side, only one CA cert relevant for the session. Technically there can
>> be more than one cert in the trust chain but not pile of them.
> 
> If the *server* chooses to trust a pile of CA's in the same way as web
> browsers (clients) typically do, this will happen, see:
> 
> https://tools.ietf.org/html/rfc5246#section-7.4.4
> 
> It also says:
> 
> "If the certificate_authorities list is empty, then the client MAY send
> any certificate of the appropriate ClientCertificateType, unless there
> is some external arrangement to the contrary."
> 
> ...which suggests that in cases like this it might be a good idea or at
> least acceptable *not* to put anything in the certificate_authorities
> list when the server sends the Certificate Request. It is unclear how
> various client side implementations implement the "MAY" part of the
> above RFC quote. Do they send a client certificate if the CA list is
> empty? Which one will they send if they have several?

Most send any certificate selected by the user.

(Continue reading)

Janne Snabb | 29 May 2012 23:45

Re: Big CA certificate bundle causes problems with GnuTLS 3.0.11

On 2012-05-30 04:24, Nikos Mavrogiannopoulos wrote on help-gnutls list:
> On 05/29/2012 11:17 PM, Janne Snabb wrote:
>> It feels like there should be a way in the GnuTLS API to define whether
>> the list of trusted CAs is to be advertised in Certificate Request or
>> not. (Maybe there is a way but I am missing it?)
> 
> 
> There is. Check client certificate authentication at:
> http://www.gnu.org/software/gnutls/manual/html_node/Certificate-credentials.html#Certificate-credentials

Do we need "gnutls_certificate_send_x509_rdn_sequence(session, knob)" in
the GnuTLS server side initialization and a corresponding configuration
knob. How does OpenSSL behave in this regard?

The advertisement should not be hard-coded in the off position because
people using MUAs may want to authenticate to SMTP submission service
with a certificate (and they may hold many of them for different
purposes/roles). Obviously in that case the SMTP submission service
would trust only one or few CAs.

--

-- 
Janne Snabb / EPIPE Communications
snabb <at> epipe.com - http://epipe.com/

Phil Pennock | 30 May 2012 00:11

Re: Big CA certificate bundle causes problems with GnuTLS 3.0.11

On 2012-05-30 at 04:45 +0700, Janne Snabb wrote:
> On 2012-05-30 04:24, Nikos Mavrogiannopoulos wrote on help-gnutls list:
> > On 05/29/2012 11:17 PM, Janne Snabb wrote:
> >> It feels like there should be a way in the GnuTLS API to define whether
> >> the list of trusted CAs is to be advertised in Certificate Request or
> >> not. (Maybe there is a way but I am missing it?)
> > 
> > 
> > There is. Check client certificate authentication at:
> > http://www.gnu.org/software/gnutls/manual/html_node/Certificate-credentials.html#Certificate-credentials
> 
> Do we need "gnutls_certificate_send_x509_rdn_sequence(session, knob)" in
> the GnuTLS server side initialization and a corresponding configuration
> knob. How does OpenSSL behave in this regard?

Oh, I deleted from my previous reply the text about "I think I remember
reading about a knob to do this in GnuTLS but I can't find it right
now".

OpenSSL: if you use a directory, it does not send the server CAs.  If
you use a file, it does.

For GnuTLS, this is not a change in behaviour, not a regression, so
adding this would be a feature enhancement which can go into 4.81.

The option might be useful, yes.  This
sending-of-list-of-CAs-we-might-trust to the client is part of why I
always use directory mode with OpenSSL.  We should let GnuTLS folks get
the same protection.

(Continue reading)

Janne Snabb | 30 May 2012 00:44

Re: Big CA certificate bundle causes problems with GnuTLS 3.0.11

On 2012-05-30 05:11, Phil Pennock wrote:
> On 2012-05-30 at 04:45 +0700, Janne Snabb wrote:
>> Do we need "gnutls_certificate_send_x509_rdn_sequence(session, knob)" in
>> the GnuTLS server side initialization and a corresponding configuration
>> knob. How does OpenSSL behave in this regard?
> 
> Oh, I deleted from my previous reply the text about "I think I remember
> reading about a knob to do this in GnuTLS but I can't find it right
> now".
> 
> OpenSSL: if you use a directory, it does not send the server CAs.  If
> you use a file, it does.
> 
> For GnuTLS, this is not a change in behaviour, not a regression, so
> adding this would be a feature enhancement which can go into 4.81.

I agree. There is enough RCs already :). This is not affecting
deliveries in real life because nobody is using the buggy GnuTLS
versions on the client side yet.

Most random SMTP speakers have certificates which are not signed by any
well-known money-making CA, with some notable exceptions (such as
Google). Thus the verification of random SMTP speakers against "all"
CAs is quite pointless and therefore can be disabled easily if needed.

> The option might be useful, yes.  This
> sending-of-list-of-CAs-we-might-trust to the client is part of why I
> always use directory mode with OpenSSL.  We should let GnuTLS folks get
> the same protection.

(Continue reading)

Janne Snabb | 5 Jun 2012 08:25

Re: Big CA certificate bundle causes problems with GnuTLS 3.0.11

On 2012-05-30 05:44, Janne Snabb wrote:
> This is not affecting deliveries in real life because nobody is
> using the buggy GnuTLS versions on the client side yet.

Just a correction to my previous statement:

I did start seeing some TLS handshake failures with the Debian/Ubuntu
certificate bundle and "tls_try_verify_hosts = *". So it looks like
there are few mail senders who use some of the broken GnuTLS versions.

Thus it is not a good idea to run such a setup in a production
environment. Reducing the amount of CA's or disabling
"tls_try_verify_hosts" solves the problem until we get the "don't
advertise CAs" knob in the next release :).

--

-- 
Janne Snabb / EPIPE Communications
snabb <at> epipe.com - http://epipe.com/

Heiko Schlittermann | 14 Jun 2012 00:41

Picon

Re: Big CA certificate bundle causes problems with GnuTLS 3.0.11

Janne Snabb <snabb <at> epipe.com> (Di 05 Jun 2012 08:25:48 CEST):
> On 2012-05-30 05:44, Janne Snabb wrote:
> > This is not affecting deliveries in real life because nobody is
> > using the buggy GnuTLS versions on the client side yet.
> 
> Just a correction to my previous statement:
> 
> I did start seeing some TLS handshake failures with the Debian/Ubuntu
> certificate bundle and "tls_try_verify_hosts = *". So it looks like
> there are few mail senders who use some of the broken GnuTLS versions.
> 
> Thus it is not a good idea to run such a setup in a production
> environment. Reducing the amount of CA's or disabling
> "tls_try_verify_hosts" solves the problem until we get the "don't
> advertise CAs" knob in the next release :).

Just for the records, I think the problem is already known for some
time:

http://thread.gmane.org/gmane.network.gnutls.general/1688
http://thread.gmane.org/gmane.mail.exim.user/83688
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=478191

(If I remember well, then even Microsoft Products used a "broken version
of GnuTLS", or at least exposed the same behaviour…)

--

-- 
Heiko :: dresden : linux : SCHLITTERMANN.de
GPG Key 48D0359B : 3061 CFBF 2D88 F034 E8D2 7E92 EE4E AC98 48D0 359B

(Continue reading)

Nikos Mavrogiannopoulos | 29 May 2012 22:36

Picon

Re: Big CA certificate bundle causes problems with GnuTLS 3.0.11

On 05/29/2012 04:46 PM, Janne Snabb wrote:

> I am experiencing a TLS handshake problem when GnuTLS 3.0.11 server has
> a big pile of CA certificates to verify against. I can not reproduce the
> problem with GnuTLS 2.12.14.
> 
> Steps to re-produce:
[...]
> Note that the file /etc/ssl/certs/ca-certificates.crt contains a big
> pile of certificates, as distributed by Debian and Ubuntu
> "ca-certificates" package. (I am happy to send it if needed.) If I
> specify just a sigle CA cert I do not see any problems.
> This means that when the problem happens the "certificate request" is
> bigger than 16k.

Thank you for reporting this. A quick solution to avoid this issue is to
restrict the CAs that you enable to the server to the minimum required
(a typical server needs to trust only the authorities that signed the
user's certificates).

regards,
Nikos

Return

Return to gmane.network.gnutls.general.

Search Archive

Language

Change language

Options

Current view: Threads only / Showing only 20 lines / Not hiding cited text.
Change to All messages, whole messages, or hide cited text.

Post a message
NNTP Newsgroup
Classic Gmane web interface
XML

XML

RSS Feed
List Information

About Gmane