IP Security Maintenance and Extensions (ipsecme)

Yoav Nir | 7 Jun 2012 17:20

Fragmentation causing IKE to fail

I work for a vendor selling and IKE/IPsec solution.

In the last few months, we've heard a troubling complaint from some of our customers. They say that some ISPs
have begun to drop IP fragments. While specific ISPs were not named, most of the issues seem to be in
south-east Asia. One customer has speculated that this has something to do with preparing to deploy CGNs.

According to draft-ietf-behave-lsn-requirements, CGNs MUST comply with the NAT requirements for UDP as
in RFC 4787, and that RFC says in section 11 that NAT devices MUST support fragments. So these routers are
not compliant, but that doesn't help our customers much.

Most traffic on the Internet is either TCP or small-packet UDP. The IKE protocol (both versions) has the
rather rare distinction of having large UDP packets. These are packets #5 and #6 in IKEv1 Main Mode, or the
IKE_AUTH packets in IKEv2, especially when using certificate authentication.

For now, the customers managed to "fix" the ISP with an angry phone call. That got them into an exception list
where fragments are not dropped. One had to change ISPs for that. While this can work for a while, it won't
work when the carrier doing the dropping is not the one that the customer has a business relationship with,
but another one downstream.

Trying to think up ways to deal with this, I can think of some:

* Get all ISPs to stop dropping fragments. This would be great, but as the saying goes, you are so not in charge.

* Find ways of making the packets smaller: move to PSK, fiddle with trust anchors so that only one cert is
needed, avoid sending CRLs, hash-and-URL, etc. These are not always successful, and often require more
configuration than we would like.

* Build a fragmentation layer within IKE, so IKE messages are broken into several packets that get

(Continue reading)

headers

Paul Hoffman | 7 Jun 2012 18:15

Re: Fragmentation causing IKE to fail

On Jun 7, 2012, at 8:20 AM, Yoav Nir wrote:

> Trying to think up ways to deal with this, I can think of some:
> 
> * Get all ISPs to stop dropping fragments. This would be great, but as the saying goes, you are so not in charge.
> 
> * Find ways of making the packets smaller: move to PSK, fiddle with trust anchors so that only one cert is
needed, avoid sending CRLs, hash-and-URL, etc. These are not always successful, and often require more
configuration than we would like.
> 
> * Build a fragmentation layer within IKE, so IKE messages are broken into several packets that get
reassembled at the destination. This is the path taken by one of our competitors [1]. This means that IKE
has segmentation in addition to other TCP-like features such as retransmission.
> 
> * Use IKE over TCP. Looking at the IANA registry ([2]) TCP port 500 is already allocated to "ISAKMP". We have
had IKE running over TCP for several years for remote access clients. This was done because remote access
clients connect from behind some very dodgy NAT devices, and some of those do drop fragments. Getting this
behavior at the ISP is novel.

* Use IKE over TCP only after IKE over UDP fails during transmission of a packet >512 bytes. That would be
interoperable with current deployments (although they would not see the second attempt, of course), it
costs little, and is trivial to implement.

--Paul Hoffman

headers

Paul Wouters | 7 Jun 2012 18:43

Re: Fragmentation causing IKE to fail

On Thu, 7 Jun 2012, Paul Hoffman wrote:

>> * Use IKE over TCP. Looking at the IANA registry ([2]) TCP port 500 is already allocated to "ISAKMP". We
have had IKE running over TCP for several years for remote access clients. This was done because remote
access clients connect from behind some very dodgy NAT devices, and some of those do drop fragments.
Getting this behavior at the ISP is novel.
>
> * Use IKE over TCP only after IKE over UDP fails during transmission of a packet >512 bytes. That would be
interoperable with current deployments (although they would not see the second attempt, of course), it
costs little, and is trivial to implement.

Is that compatible with some vendor's tcp port 10000 implementation?

Also, if we are doing this, I'd prefer to be able to signal which tcp
port to use, to make it more flexible to bypass port 500 blocks (which
is part of the tcp 10000 implementation I believe)

Paul

headers

Paul Hoffman | 7 Jun 2012 18:54

Re: Fragmentation causing IKE to fail


On Jun 7, 2012, at 9:43 AM, Paul Wouters wrote:

> On Thu, 7 Jun 2012, Paul Hoffman wrote:
> 
>>> * Use IKE over TCP. Looking at the IANA registry ([2]) TCP port 500 is already allocated to "ISAKMP". We
have had IKE running over TCP for several years for remote access clients. This was done because remote
access clients connect from behind some very dodgy NAT devices, and some of those do drop fragments.
Getting this behavior at the ISP is novel.
>> 
>> * Use IKE over TCP only after IKE over UDP fails during transmission of a packet >512 bytes. That would be
interoperable with current deployments (although they would not see the second attempt, of course), it
costs little, and is trivial to implement.
> 
> Is that compatible with some vendor's tcp port 10000 implementation?

Why should I care about that completely non-standard use? Seriously.

> Also, if we are doing this, I'd prefer to be able to signal which tcp
> port to use, to make it more flexible to bypass port 500 blocks (which
> is part of the tcp 10000 implementation I believe)

That seems fine to me. However, assuming that a firewall that blocks TCP/500 will not block
TCP/somerandomnewnumber is not wise.

--Paul Hoffman

headers

Nico Williams | 7 Jun 2012 19:26

Re: Fragmentation causing IKE to fail

On Thu, Jun 7, 2012 at 11:54 AM, Paul Hoffman <paul.hoffman <at> vpnc.org> wrote:
> On Jun 7, 2012, at 9:43 AM, Paul Wouters wrote:
>> Also, if we are doing this, I'd prefer to be able to signal which tcp
>> port to use, to make it more flexible to bypass port 500 blocks (which
>> is part of the tcp 10000 implementation I believe)
>
> That seems fine to me. However, assuming that a firewall that blocks TCP/500 will not block
TCP/somerandomnewnumber is not wise.

Use port 80.

(I'm being half facetious, half sarcastic, and half serious with this.)

Nico
--

headers

Paul Hoffman | 7 Jun 2012 19:40

Re: Fragmentation causing IKE to fail

On Jun 7, 2012, at 10:26 AM, Nico Williams wrote:

> On Thu, Jun 7, 2012 at 11:54 AM, Paul Hoffman <paul.hoffman <at> vpnc.org> wrote:
>> On Jun 7, 2012, at 9:43 AM, Paul Wouters wrote:
>>> Also, if we are doing this, I'd prefer to be able to signal which tcp
>>> port to use, to make it more flexible to bypass port 500 blocks (which
>>> is part of the tcp 10000 implementation I believe)
>> 
>> That seems fine to me. However, assuming that a firewall that blocks TCP/500 will not block
TCP/somerandomnewnumber is not wise.
> 
> Use port 80.
> 
> (I'm being half facetious, half sarcastic, and half serious with this.)

Being non-all-of-the-above: that won't work. Many firewalls that are blocking UDP fragments do deep
packet inspection on port 80 and will throw away traffic that doesn't look like HTTP. (Don't get me started
on "look like"...)

--Paul Hoffman

headers

Nico Williams | 7 Jun 2012 19:55

Re: Fragmentation causing IKE to fail

On Thu, Jun 7, 2012 at 12:40 PM, Paul Hoffman <paul.hoffman <at> vpnc.org> wrote:
> On Jun 7, 2012, at 10:26 AM, Nico Williams wrote:
>> Use port 80.
>>
>> (I'm being half facetious, half sarcastic, and half serious with this.)
>
> Being non-all-of-the-above: that won't work. Many firewalls that are blocking UDP fragments do deep
packet inspection on port 80 and will throw away traffic that doesn't look like HTTP. (Don't get me started
on "look like"...)

To be closer to 100% serious I'd have to advocate an HTTP mapping of
IKE and/or use of port 443.  I'm not sure that I want to go there, but
really, if you want to get past deep inspection nowadays then your
best bet seems to be port 443.  Which, of course, would not be enough.
 You'd find that ESP (encapsulated in UDP or not) also gets filtered,
so you'd have to start sending ESP over HTTPS.  And that's all kinds
of not fun.

At some point though one has to give up and declare the ISP useless.
If you're a dissident in Iran, well, you're not using IPsec anyways,
and Tor and all things port 443 are really your only friends, and if
in the end the great firewalls get good enough, well, what can we do
as far as *standards*?  Not much.  But I don't think Yoav was talking
about this case, just a lousy ISP case, and for that I suspect deep
packet inspection is not really the problem.  For Yoav I suspect that
IKE over TCP + UDP encapsulation of ESP is the way to go; worst case
scenario: ESP over TCP.

Nico
--

(Continue reading)

headers

Yoav Nir | 8 Jun 2012 00:05

Re: Fragmentation causing IKE to fail

On Jun 7, 2012, at 8:26 PM, Nico Williams wrote:

> On Thu, Jun 7, 2012 at 11:54 AM, Paul Hoffman <paul.hoffman <at> vpnc.org> wrote:
>> On Jun 7, 2012, at 9:43 AM, Paul Wouters wrote:
>>> Also, if we are doing this, I'd prefer to be able to signal which tcp
>>> port to use, to make it more flexible to bypass port 500 blocks (which
>>> is part of the tcp 10000 implementation I believe)
>> 
>> That seems fine to me. However, assuming that a firewall that blocks TCP/500 will not block
TCP/somerandomnewnumber is not wise.
> 
> Use port 80.

Well, we came up with IKE-over-TCP for clients in 2002. That worked OK with broken routers and NAT devices,
but not with firewalls. So in 2003 we came up with sending both IKE and IPsec over port 443 (because at the
time few firewalls other than ours validated that what goes on port 443 looks like SSL). Finally in 2005 we
came out with a client that actually uses SSL for traffic, so that firewalls have to either be SSL proxies or
do traffic flow analysis to recognize non-HTTPS.

But this arms race between tunneling clients and firewalls is not the issue here. Without getting into the
whole net neutrality controversy, ISPs are not supposed to, nor are they trying to block the creation of
tunnels. What they are doing is saving themselves the need to keep state on received fragments, which
allows better scale and better performance with the same hardware.

Yoav

headers

Yoav Nir | 7 Jun 2012 23:53

Re: Fragmentation causing IKE to fail

On Jun 7, 2012, at 7:15 PM, Paul Hoffman wrote:

>> * Use IKE over TCP. Looking at the IANA registry ([2]) TCP port 500 is already allocated to "ISAKMP". We
have had IKE running over TCP for several years for remote access clients. This was done because remote
access clients connect from behind some very dodgy NAT devices, and some of those do drop fragments.
Getting this behavior at the ISP is novel.
> 
> * Use IKE over TCP only after IKE over UDP fails during transmission of a packet >512 bytes. That would be
interoperable with current deployments (although they would not see the second attempt, of course), it
costs little, and is trivial to implement.

This is possible, but since UDP is not reliable, you'd have to retransmit several times before giving up on
UDP. Even if we shorten the "at least a dozen times over a period of at least several minutes", it's still
long enough for users to feel - get the "connection with Exchange lost" message in Outlook, for example. 

You could begin both UDP (first IKE message) and TCP's 3-way handshake at the same time. If the 3-way
handshake completed in time, the larger packets would go over that connection. If not, they would go over TCP.

But all this is implementation-specific details. I'm more interested in hearing whether others are
seeing this (I would guess yes, otherwise Cisco would not have developed the IKE fragments), and on
whether there is interest in the group in an IKE-over-TCP draft.

Yoav

headers

Paul Hoffman | 8 Jun 2012 00:01

Re: Fragmentation causing IKE to fail

On Jun 7, 2012, at 2:53 PM, Yoav Nir wrote:

> 
> On Jun 7, 2012, at 7:15 PM, Paul Hoffman wrote:
> 
>>> * Use IKE over TCP. Looking at the IANA registry ([2]) TCP port 500 is already allocated to "ISAKMP". We
have had IKE running over TCP for several years for remote access clients. This was done because remote
access clients connect from behind some very dodgy NAT devices, and some of those do drop fragments.
Getting this behavior at the ISP is novel.
>> 
>> * Use IKE over TCP only after IKE over UDP fails during transmission of a packet >512 bytes. That would be
interoperable with current deployments (although they would not see the second attempt, of course), it
costs little, and is trivial to implement.
> 
> This is possible, but since UDP is not reliable, you'd have to retransmit several times before giving up on
UDP. Even if we shorten the "at least a dozen times over a period of at least several minutes", it's still
long enough for users to feel - get the "connection with Exchange lost" message in Outlook, for example. 

Good point.

> You could begin both UDP (first IKE message) and TCP's 3-way handshake at the same time. If the 3-way
handshake completed in time, the larger packets would go over that connection. If not, they would go over TCP.

Yuck. But possibly the right answer.

> But all this is implementation-specific details. I'm more interested in hearing whether others are
seeing this (I would guess yes, otherwise Cisco would not have developed the IKE fragments), and on
whether there is interest in the group in an IKE-over-TCP draft.

Please consider "IKE-with-TCP-and-UDP" before "IKE-over-TCP".

(Continue reading)

headers

Yoav Nir | 8 Jun 2012 00:13

Re: Fragmentation causing IKE to fail


On Jun 8, 2012, at 1:01 AM, Paul Hoffman wrote:

> On Jun 7, 2012, at 2:53 PM, Yoav Nir wrote:
> 
>> 
>> On Jun 7, 2012, at 7:15 PM, Paul Hoffman wrote:
>> 
>>>> * Use IKE over TCP. Looking at the IANA registry ([2]) TCP port 500 is already allocated to "ISAKMP". We
have had IKE running over TCP for several years for remote access clients. This was done because remote
access clients connect from behind some very dodgy NAT devices, and some of those do drop fragments.
Getting this behavior at the ISP is novel.
>>> 
>>> * Use IKE over TCP only after IKE over UDP fails during transmission of a packet >512 bytes. That would be
interoperable with current deployments (although they would not see the second attempt, of course), it
costs little, and is trivial to implement.
>> 
>> This is possible, but since UDP is not reliable, you'd have to retransmit several times before giving up
on UDP. Even if we shorten the "at least a dozen times over a period of at least several minutes", it's still
long enough for users to feel - get the "connection with Exchange lost" message in Outlook, for example. 
> 
> Good point.
> 
>> You could begin both UDP (first IKE message) and TCP's 3-way handshake at the same time. If the 3-way
handshake completed in time, the larger packets would go over that connection. If not, they would go over TCP.
> 
> Yuck. But possibly the right answer.
> 
>> But all this is implementation-specific details. I'm more interested in hearing whether others are
seeing this (I would guess yes, otherwise Cisco would not have developed the IKE fragments), and on

(Continue reading)

headers

Paul Wouters | 8 Jun 2012 00:09

Re: Fragmentation causing IKE to fail

On Fri, 8 Jun 2012, Yoav Nir wrote:

> But all this is implementation-specific details. I'm more interested in hearing whether others are
seeing this (I would guess yes, otherwise Cisco would not have developed the IKE fragments), and on
whether there is interest in the group in an IKE-over-TCP draft.

Yes we have seen this in the past with openswan, though people affected
would usually use 2048 bit RSA keys instead of 1024 bit RSA keys.
And usually in combination with a CAcert without intermediary CAs.

We would advise them to use 1024 and the IKE fragemntation problem would
go away.

Paul

headers

Michael Richardson | 7 Jun 2012 21:18

Re: Fragmentation causing IKE to fail


>>>>> "Yoav" == Yoav Nir <ynir <at> checkpoint.com> writes:
    Yoav> Trying to think up ways to deal with this, I can think of some:

    Yoav> * Get all ISPs to stop dropping fragments. This would be
    Yoav> great, but as the saying goes, you are so not in charge. 

1) better diagnostics would help the end users point the finger
   properly.  I wish the POSIX/BSD APIs would give the application
   an error when a fragment assembly times out..

    Yoav> * Build a fragmentation layer within IKE, so IKE messages are
    Yoav> broken into several packets that get reassembled at the
    Yoav> destination. This is the path taken by one of our competitors
    Yoav> [1]. This means that IKE has segmentation in addition to other
    Yoav> TCP-like features such as retransmission. 

I proposed this for IKEv2 awhile ago.  I twould be worthwhile for people
who like certificates.

    Yoav> * Use IKE over TCP. Looking at the IANA registry ([2]) TCP
    Yoav> port 500 is already allocated to "ISAKMP". We have had IKE
    Yoav> running over TCP for several years for remote access
    Yoav> clients. This was done because remote access clients connect
    Yoav> from behind some very dodgy NAT devices, and some of those do
    Yoav> drop fragments. Getting this behavior at the ISP is novel. 

And ESP over TCP on port 4500?

--

--

(Continue reading)

headers

Valery Smyslov | 8 Jun 2012 06:05

Re: Fragmentation causing IKE to fail

Hi,

we've being running into this issue constantly. I think it is serious problem
for road warriers, who have to deal with all kinds of buggy or crippled SOHO
routers installed in hotels etc. Many our customers complain that they are
unable to connect to main office while being on business trip and most
offen the reason is inability (or unwilling) of some intermediate router(s) to pass
UDP fragments.

> Trying to think up ways to deal with this, I can think of some:
>
> * Get all ISPs to stop dropping fragments. This would be great, but as the saying goes,
you are so not in charge.

No an option.

> * Find ways of making the packets smaller: move to PSK, fiddle with trust anchors so
that only one cert is needed, avoid sending CRLs, hash-and-URL, etc. These are not always
successful, and often require more configuration than we would like.

Not an option either. Corporate PKI has plenty of requirements to deal with and
the requirement to make certificates smaller is the least. Hash-and-URL is a nice
feature, but it requires an additional infrastructure that not every customer is
willing to deploy.

> * Build a fragmentation layer within IKE, so IKE messages are broken into several
packets that get reassembled at the destination. This is the path taken by one of our
competitors [1]. This means that IKE has segmentation in addition to other TCP-like
features such as retransmission.

(Continue reading)

headers

Tero Kivinen | 15 Jun 2012 14:12

Re: Fragmentation causing IKE to fail

Valery Smyslov writes:
> > * Find ways of making the packets smaller: move to PSK, fiddle
> > with trust anchors so that only one cert is needed, avoid sending
> > CRLs, hash-and-URL, etc. These are not always successful, and
> > often require more configuration than we would like.
>
> Not an option either. Corporate PKI has plenty of requirements to
> deal with and the requirement to make certificates smaller is the
> least. Hash-and-URL is a nice feature, but it requires an additional
> infrastructure that not every customer is willing to deploy.

Hash-and-URL do require customer to deploy www-server. I admit that
for some enterprices that might be burden to deploy, but quite a lot
of other enterprices do already have working deployed www-server they
can use...

The url can be static, the certificate stored on that url can be
static, new certificates needs to be pushed to www-server only and
only when the new certiticate is enrolled. The url can be something
like http://certs.example.com/<ca-short-name>/<hash-of-cert>.

Hash-and-URL should make the packets small enough that fragmentation
is not needed (especially if network supports 1280 byte packets). 

> > * Build a fragmentation layer within IKE, so IKE messages are
> > broken into several packets that get reassembled at the
> > destination. This is the path taken by one of our competitors [1].
> > This means that IKE has segmentation in addition to other TCP-like
> > features such as retransmission.
>

(Continue reading)