headers
Bas van Schaik | 30 Jul 2012 15:25
Picon

Forcing track/routeback over OpenVPN connection

Hi all,

I've been struggling with a rather exotic routing challenge for a few 
days now, hope someone here can give me some hints. I'm running a 
virtualised server (Debian, Shorewall 4.5.5.3, IP 
123.123.123.123/255.255.255.0 - let's call this 'server') in on a 
location at which all ports except for 22 are firewalled. I need access 
to this server on ports 25, 143, 587, 80, and 443, so I decided to hire 
a VPS (Debian, Shorewall 4.4.11, IP 234.234.234.234/255.255.255.0) - 
let's call this 'vps-gateway') which runs OpenVPN. The server connects 
to the vps-gateway, and Shorewall on the vps-gateway is configured to 
DNAT the ports mentioned above to the server. A shorewall dump of the 
server is attached (vps-gateway is irrelevant, that one works fine), 
which will show you a set-up using two providers:
  - prov_main: main outgoing network connection (over LAN with gateway)
  - prov_vpn: provider created for traffic coming in through the 
vps-gateway (with 'track' option)

This works beautifully: traffic coming in on 234.234.234.234 gets 
DNATted to 192.168.103.6 (IP of server on VPN), is sent to the server, 
and the server routes replies back using tun0 via vps-gateway through 
the prov_vpn provider. Even though the default gateway for the server is 
123.123.123.254 on eth0 (123.123.123.123/255.255.255.0). So the routing 
responses over the same interface works.

One problematic exception: traffic from the server's LAN (e.g. a client 
at 123.123.123.100/255.255.255.0) with destination 234.234.234.234. The 
packets get DNATted at the vps-gateway and sent to the server through 
the VPN (as is to be expected), but the server acts in one of three ways 
(seemingly at random, not sure what's going on here):
(Continue reading)

Permalink | Reply |
headers
Tom Eastep | 30 Jul 2012 16:11
Favicon

Re: Forcing track/routeback over OpenVPN connection

On 7/30/12 6:25 AM, Bas van Schaik wrote:
> Hi all,
> 
> I've been struggling with a rather exotic routing challenge for a few
> days now, hope someone here can give me some hints. I'm running a
> virtualised server (Debian, Shorewall 4.5.5.3, IP
> 123.123.123.123/255.255.255.0 - let's call this 'server') in on a
> location at which all ports except for 22 are firewalled. I need access
> to this server on ports 25, 143, 587, 80, and 443, so I decided to hire
> a VPS (Debian, Shorewall 4.4.11, IP 234.234.234.234/255.255.255.0) -
> let's call this 'vps-gateway') which runs OpenVPN. The server connects
> to the vps-gateway, and Shorewall on the vps-gateway is configured to
> DNAT the ports mentioned above to the server. A shorewall dump of the
> server is attached (vps-gateway is irrelevant, that one works fine),
> which will show you a set-up using two providers:
>  - prov_main: main outgoing network connection (over LAN with gateway)
>  - prov_vpn: provider created for traffic coming in through the
> vps-gateway (with 'track' option)
> 
> This works beautifully: traffic coming in on 234.234.234.234 gets
> DNATted to 192.168.103.6 (IP of server on VPN), is sent to the server,
> and the server routes replies back using tun0 via vps-gateway through
> the prov_vpn provider. Even though the default gateway for the server is
> 123.123.123.254 on eth0 (123.123.123.123/255.255.255.0). So the routing
> responses over the same interface works.
> 
> One problematic exception: traffic from the server's LAN (e.g. a client
> at 123.123.123.100/255.255.255.0) with destination 234.234.234.234. The
> packets get DNATted at the vps-gateway and sent to the server through
> the VPN (as is to be expected), but the server acts in one of three ways
(Continue reading)

Permalink | Reply |
headers
Bas van Schaik | 30 Jul 2012 16:24
Picon

Re: Forcing track/routeback over OpenVPN connection

Hi Tom,

On 30/07/12 15:11, Tom Eastep wrote:
> Please describe your problem using the actual addresses and we'll try
> to help. Thanks, -Tom
OK, here it goes:

Hi all,

I've been struggling with a rather exotic routing challenge for a few
days now, hope someone here can give me some hints. I'm running a
virtualised server (Debian, Shorewall 4.5.5.3, IP
129.67.194.105/255.255.252.0 - let's call this 'server') in on a
location at which all ports except for 22 are firewalled. I need access
to this server on ports 25, 143, 587, 80, and 443, so I decided to hire
a VPS (Debian, Shorewall 4.4.11, IP 37.34.58.203 - let's call this
'vps-gateway') which runs OpenVPN. Note that I can't actually move the
server into a VPS provider for some practical reasons.

The server connects to the vps-gateway, and Shorewall on the vps-gateway
is configured to DNAT the ports mentioned above to the server. A
shorewall dump of the server is attached (vps-gateway is irrelevant,
that one works fine), which will show you a set-up using two providers:
 - prov_main: main outgoing network connection (over LAN with gateway)
 - prov_vpn: provider created for traffic coming in through the
vps-gateway (with 'track' option)

This works beautifully: traffic coming in on 37.34.58.203 gets DNATted
to 192.168.103.6 (IP of server on VPN), is sent to the server, and the
server routes replies back using tun0 via vps-gateway through the
(Continue reading)

Permalink | Reply |
headers
Tom Eastep | 31 Jul 2012 01:52
Favicon

Re: Forcing track/routeback over OpenVPN connection

On 7/30/12 7:24 AM, Bas van Schaik wrote:
> Hi Tom,
> 
> On 30/07/12 15:11, Tom Eastep wrote:
>> Please describe your problem using the actual addresses and we'll try
>> to help. Thanks, -Tom
> OK, here it goes:
> 
> 
> Hi all,
> 
> I've been struggling with a rather exotic routing challenge for a few
> days now, hope someone here can give me some hints. I'm running a
> virtualised server (Debian, Shorewall 4.5.5.3, IP
> 129.67.194.105/255.255.252.0 - let's call this 'server') in on a
> location at which all ports except for 22 are firewalled. I need access
> to this server on ports 25, 143, 587, 80, and 443, so I decided to hire
> a VPS (Debian, Shorewall 4.4.11, IP 37.34.58.203 - let's call this
> 'vps-gateway') which runs OpenVPN. Note that I can't actually move the
> server into a VPS provider for some practical reasons.
> 
> The server connects to the vps-gateway, and Shorewall on the vps-gateway
> is configured to DNAT the ports mentioned above to the server. A
> shorewall dump of the server is attached (vps-gateway is irrelevant,
> that one works fine), which will show you a set-up using two providers:
>  - prov_main: main outgoing network connection (over LAN with gateway)
>  - prov_vpn: provider created for traffic coming in through the
> vps-gateway (with 'track' option)
> 
> This works beautifully: traffic coming in on 37.34.58.203 gets DNATted
(Continue reading)

Permalink | Reply |
headers
Bas van Schaik | 31 Jul 2012 02:39
Picon

Re: Forcing track/routeback over OpenVPN connection

Hi Tom,

On 31/07/12 00:52, Tom Eastep wrote:
> Looks to me like your COPY column contents in /etc/shorewall/providers
> are wrong. The routes out of eth0 are copied into the VPN's routing
> table; see the entry marked <=== above.
Well spotted! Your observation was right and led to the solution (which
also caused a new problem - see below), but it was not the routes of
eth0 being copied into the 'prov_vpn' routing table, it was the 'main'
routing table being copied into the 'vpn_main' routing table. I removed
the values from the COPY column, and also removed the values from the
DUPLICATE column (which used to contain 'main', but is now '-').

If anyone else ever runs into the same trouble (track/routeback not
working), this setup of 'providers' might work for you:
> #NAME           NUMBER  MARK    DUPLICATE       INTERFACE      
> GATEWAY         OPTIONS                 COPY
> prov_main       1       1       -               eth0           
> detect          track,balance
> prov_vpn        2       2       -               tun0           
> detect          track,optional,loose

However, the new problem is that direct traffic from
129.67.194.0/255.255.252.0 to 129.67.194.105 is now routed through the
default gateway of 'provider_main', which is of course unnecessary and
causes new routing problems:
> Jul 31 01:21:08 guust kernel: [282101.945969] Redirect from
> 129.67.195.254 on eth0 about 129.67.194.110 ignored.

I tried adding a specific route using the 'routes' file (which was
(Continue reading)

Permalink | Reply |
headers
Tom Eastep | 31 Jul 2012 03:28
Favicon

Re: Forcing track/routeback over OpenVPN connection

On 7/30/12 5:39 PM, Bas van Schaik wrote:
> Hi Tom,
> 
> On 31/07/12 00:52, Tom Eastep wrote:
>> Looks to me like your COPY column contents in /etc/shorewall/providers
>> are wrong. The routes out of eth0 are copied into the VPN's routing
>> table; see the entry marked <=== above.
> Well spotted! Your observation was right and led to the solution (which
> also caused a new problem - see below), but it was not the routes of
> eth0 being copied into the 'prov_vpn' routing table, it was the 'main'
> routing table being copied into the 'vpn_main' routing table. I removed
> the values from the COPY column, and also removed the values from the
> DUPLICATE column (which used to contain 'main', but is now '-').
> 
> If anyone else ever runs into the same trouble (track/routeback not
> working), this setup of 'providers' might work for you:
>> #NAME           NUMBER  MARK    DUPLICATE       INTERFACE      
>> GATEWAY         OPTIONS                 COPY
>> prov_main       1       1       -               eth0           
>> detect          track,balance
>> prov_vpn        2       2       -               tun0           
>> detect          track,optional,loose
> 
> However, the new problem is that direct traffic from
> 129.67.194.0/255.255.252.0 to 129.67.194.105 is now routed through the
> default gateway of 'provider_main', which is of course unnecessary and
> causes new routing problems:
>> Jul 31 01:21:08 guust kernel: [282101.945969] Redirect from
>> 129.67.195.254 on eth0 about 129.67.194.110 ignored.
>
(Continue reading)

Permalink | Reply |
headers
Bas van Schaik | 31 Jul 2012 11:56
Picon

Re: Forcing track/routeback over OpenVPN connection

On 31/07/12 02:28, Tom Eastep wrote:
> Have you tried specifying 'none' in the COPY column as described in
> the shorewall-providers manpage?
Genius! I had to put 'main' back into the DUPLICATE column, and then
added 'none' to the COPY column. Now, everything works as expected. So,
that's:

> #NAME           NUMBER  MARK    DUPLICATE       INTERFACE      
> GATEWAY         OPTIONS                 COPY
> prov_main       1       1       main            eth0           
> detect          track,balance           none
> prov_vpn        2       2       main            tun0           
> detect          track,optional,loose    none
(in 'providers')

Thank you very much for your help.

  Bas

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
Permalink | Reply |
headers
Bas van Schaik | 1 Aug 2012 13:07
Picon

Re: Forcing track/routeback over OpenVPN connection

On 31/07/12 10:56, Bas van Schaik wrote:
> On 31/07/12 02:28, Tom Eastep wrote:
>> Have you tried specifying 'none' in the COPY column as described in
>> the shorewall-providers manpage?
> Genius! I had to put 'main' back into the DUPLICATE column, and then
> added 'none' to the COPY column. (..)

I noticed another minor issue: the server (eth0:129.67.194.105,
tun0:192.168.103.6) can't communicate with itself via the vps-gateway
(eth0:37.34.58.203). Assume www.mydomain.org resolves to the vps-gateway
at 37.34.58.203, which is actually served by Apache on the server. Now,
things go wrong when:

 1) 129.67.194.105 => www.mydomain.org (37.34.58.203)
 2) vps-gateway DNATs 37.34.58.203 to 192.168.103.6, resulting in:
 3) 129.67.194.105 => 192.168.103.6 (via tun0)
 4) server complains about martian on tun0:
     [72347.809855] martian source 192.168.103.6 from 129.67.194.105, on
dev tun0

The packet does qualify as a martian - after all, 129.67.194.105 would
usually be routed over eth0. However, in this setup, a "martian" are not
a bad thing: packets are allowed to travel via Mars (the vps-gateway),
and replies should be routed back via Mars. ROUTE_FILTER is still "No" -
see attached shorewall dump.

I'm guessing it's related to the routing rule with priority 0:
> Routing Rules
>
> 0:      from all lookup local
(Continue reading)

Permalink | Reply |
headers
Tom Eastep | 1 Aug 2012 15:29
Favicon

Re: Forcing track/routeback over OpenVPN connection

On 08/01/2012 04:07 AM, Bas van Schaik wrote:
> On 31/07/12 10:56, Bas van Schaik wrote:
>> On 31/07/12 02:28, Tom Eastep wrote:
>>> Have you tried specifying 'none' in the COPY column as described in
>>> the shorewall-providers manpage?
>> Genius! I had to put 'main' back into the DUPLICATE column, and then
>> added 'none' to the COPY column. (..)
>
> I noticed another minor issue: the server (eth0:129.67.194.105,
> tun0:192.168.103.6) can't communicate with itself via the vps-gateway
> (eth0:37.34.58.203). Assume www.mydomain.org resolves to the vps-gateway
> at 37.34.58.203, which is actually served by Apache on the server. Now,
> things go wrong when:
>
>   1) 129.67.194.105 => www.mydomain.org (37.34.58.203)
>   2) vps-gateway DNATs 37.34.58.203 to 192.168.103.6, resulting in:
>   3) 129.67.194.105 => 192.168.103.6 (via tun0)
>   4) server complains about martian on tun0:
>       [72347.809855] martian source 192.168.103.6 from 129.67.194.105, on
> dev tun0
>
>
> The packet does qualify as a martian - after all, 129.67.194.105 would
> usually be routed over eth0. However, in this setup, a "martian" are not
> a bad thing: packets are allowed to travel via Mars (the vps-gateway),
> and replies should be routed back via Mars. ROUTE_FILTER is still "No" -
> see attached shorewall dump.
>
> I'm guessing it's related to the routing rule with priority 0:
>> Routing Rules
(Continue reading)

Permalink | Reply |
headers
Bas van Schaik | 3 Aug 2012 16:09
Picon

Re: Forcing track/routeback over OpenVPN connection

Hi Tom,

On 01/08/12 14:29, Tom Eastep wrote:
> Why don't you simply turn of routefilter on tun0? -Tom
There is no 'routefilter' on any of the interfaces in the 'interfaces'
files (there is 'logmartians', though). Also, ROUTE_FILTER=No in
'shorewall.conf', and:

> $ cat /proc/sys/net/ipv4/conf/*/rp_filter
yields all zeroes.

  Bas

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
Permalink | Reply |
headers
Tom Eastep | 3 Aug 2012 17:44
Favicon

Re: Forcing track/routeback over OpenVPN connection

On 08/03/2012 07:09 AM, Bas van Schaik wrote:
> Hi Tom,
>
> On 01/08/12 14:29, Tom Eastep wrote:
>> Why don't you simply turn of routefilter on tun0? -Tom
> There is no 'routefilter' on any of the interfaces in the 'interfaces'
> files (there is 'logmartians', though). Also, ROUTE_FILTER=No in
> 'shorewall.conf', and:
>
>> $ cat /proc/sys/net/ipv4/conf/*/rp_filter
> yields all zeroes.

Then you can't possibly get getting martian messages.

-Tom
--

-- 
Tom Eastep        \ When I die, I want to go like my Grandfather who
Shoreline,         \ died peacefully in his sleep. Not screaming like
Washington, USA     \ all of the passengers in his car
http://shorewall.net \________________________________________________

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
Permalink | Reply |
headers
Bas van Schaik | 4 Aug 2012 14:58
Picon

Re: Forcing track/routeback over OpenVPN connection

On 03/08/12 16:44, Tom Eastep wrote:
> Then you can't possibly get getting martian messages.
That's what I thought, but they are most certainly there.

This is what I do to get them:
> $ for rtfile in `ls /proc/sys/net/ipv4/conf/*/rp_filter`; do echo -n
> "$rtfile:"; cat "$rtfile"; done; echo; wget --timeout=2 www.sjvs.nl
> --tries=1; tail -n1 /var/log/syslog
> /proc/sys/net/ipv4/conf/all/rp_filter:0
> /proc/sys/net/ipv4/conf/default/rp_filter:0
> /proc/sys/net/ipv4/conf/eth0/rp_filter:0
> /proc/sys/net/ipv4/conf/lo/rp_filter:0
> /proc/sys/net/ipv4/conf/tun0/rp_filter:0
> /proc/sys/net/ipv4/conf/tun1/rp_filter:0
>
> --2012-08-04 13:53:57--  http://www.sjvs.nl/
> Resolving www.sjvs.nl... 37.34.58.203
> Connecting to www.sjvs.nl|37.34.58.203|:80... failed: Connection timed
> out.
> Giving up.
>
> Aug  4 13:53:57 guust kernel: [77144.413105] martian source
> 192.168.103.6 from 129.67.194.105, on dev tun0

These commands a run on the server (eth0:129.67.194.105,
tun0:192.168.103.6, configuration unchanged from the shorewall dump
attached to my message of Aug 1, 12:07). Note that www.sjvs.nl resolves
to the vps-gateway at 37.34.58.203, which then routes the traffic back
to the server on tun0.
(Continue reading)

Permalink | Reply |
headers
Tom Eastep | 4 Aug 2012 19:49
Favicon

Re: Forcing track/routeback over OpenVPN connection

On 8/4/12 5:58 AM, "Bas van Schaik" <bas <at> tuxes.nl> wrote:

>On 03/08/12 16:44, Tom Eastep wrote:
>> Then you can't possibly get getting martian messages.
>That's what I thought, but they are most certainly there.
>
>This is what I do to get them:
>> $ for rtfile in `ls /proc/sys/net/ipv4/conf/*/rp_filter`; do echo -n
>> "$rtfile:"; cat "$rtfile"; done; echo; wget --timeout=2 www.sjvs.nl
>> --tries=1; tail -n1 /var/log/syslog
>> /proc/sys/net/ipv4/conf/all/rp_filter:0
>> /proc/sys/net/ipv4/conf/default/rp_filter:0
>> /proc/sys/net/ipv4/conf/eth0/rp_filter:0
>> /proc/sys/net/ipv4/conf/lo/rp_filter:0
>> /proc/sys/net/ipv4/conf/tun0/rp_filter:0
>> /proc/sys/net/ipv4/conf/tun1/rp_filter:0
>>
>> --2012-08-04 13:53:57--  http://www.sjvs.nl/
>> Resolving www.sjvs.nl... 37.34.58.203
>> Connecting to www.sjvs.nl|37.34.58.203|:80... failed: Connection timed
>> out.
>> Giving up.
>>
>> Aug  4 13:53:57 guust kernel: [77144.413105] martian source
>> 192.168.103.6 from 129.67.194.105, on dev tun0
>
>These commands a run on the server (eth0:129.67.194.105,
>tun0:192.168.103.6, configuration unchanged from the shorewall dump
>attached to my message of Aug 1, 12:07). Note that www.sjvs.nl resolves
>to the vps-gateway at 37.34.58.203, which then routes the traffic back
(Continue reading)

Permalink | Reply |

Gmane