Eugen Leitl | 2 Oct 12:19 2010

pfSense router/firewall in a Vmware ESXi guest for other guests


A customer needs to run VMWare instances on the cheap, so naturally I thought
about http://wiki.hetzner.de/index.php/VMware_ESXi_english

ESXi can't route by itself though, so I thought about putting
pfSense into one VMWare guest instance, and use that for a router/
firewall for the other guests.

Anyone here doing that? Works well? Care to share details of
your setup?

--

-- 
Eugen* Leitl <a href="http://leitl.org">leitl</a> http://leitl.org
______________________________________________________________
ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org
8B29F6BE: 099D 78BA 2FD3 B014 B08A  7779 75B0 2443 8B29 F6BE

---------------------------------------------------------------------
To unsubscribe, e-mail: discussion-unsubscribe <at> pfsense.com
For additional commands, e-mail: discussion-help <at> pfsense.com

Commercial support available - https://portal.pfsense.org

Adam Thompson | 2 Oct 20:27 2010
Picon

RE: pfSense router/firewall in a Vmware ESXi guest for other guests

It works, but performance is, in my experience, poor.  Don't use trunking 
(802.3ad / LACP) and VLANs together, or inter-vlan routing slows down 
drastically.  This appears to be a VMWare problem, not a pfSense problem. 
I recommend creating one virtual Ethernet device per network, and in fact 
mapping each virtual switch (or vlan) to a physical NIC on the host.
Basically, keep the networking as simple as possible, don't get fancy like 
I did.
-Adam Thompson
 athompso <at> athompso.net

> -----Original Message-----
> From: Eugen Leitl [mailto:eugen <at> leitl.org]
> Sent: Saturday, October 02, 2010 05:20
> To: discussion <at> pfsense.com
> Subject: [pfSense-discussion] pfSense router/firewall in a Vmware
> ESXi guest for other guests
>
>
> A customer needs to run VMWare instances on the cheap, so naturally
> I thought
> about http://wiki.hetzner.de/index.php/VMware_ESXi_english
>
> ESXi can't route by itself though, so I thought about putting
> pfSense into one VMWare guest instance, and use that for a router/
> firewall for the other guests.
>
> Anyone here doing that? Works well? Care to share details of
> your setup?
>
> --
(Continue reading)

Scott Ullrich | 2 Oct 20:37 2010
Picon

Re: pfSense router/firewall in a Vmware ESXi guest for other guests

On Sat, Oct 2, 2010 at 2:27 PM, Adam Thompson <athompso <at> c3a.ca> wrote:
> It works, but performance is, in my experience, poor.  Don't use trunking
> (802.3ad / LACP) and VLANs together, or inter-vlan routing slows down
> drastically.  This appears to be a VMWare problem, not a pfSense problem.
> I recommend creating one virtual Ethernet device per network, and in fact
> mapping each virtual switch (or vlan) to a physical NIC on the host.
> Basically, keep the networking as simple as possible, don't get fancy like
> I did.

Was this with 4.0 or 4.1?   4.1 seems to drastically improved across
the board in terms of I/O in general.

Scott

---------------------------------------------------------------------
To unsubscribe, e-mail: discussion-unsubscribe <at> pfsense.com
For additional commands, e-mail: discussion-help <at> pfsense.com

Commercial support available - https://portal.pfsense.org

Adam Thompson | 2 Oct 20:44 2010
Picon

RE: pfSense router/firewall in a Vmware ESXi guest for other guests

This started with 4.0, I have upgraded to 4.1 but haven't specifically 
tested performance since.  Routing from one VLAN to another entirely 
inside VMware is still slow, however.  AFAIK this is somehow related to 
interrupt handling and/or mitigation.  The bad news is that since 
upgrading to 4.1, the pfSense guest occasionally loses ALL network 
interrupts for about 15 minutes at a time - this happens at least once or 
twice a week.  It starts slowly, performance is merely degraded, then 
nothing, then slowly returns to normal - whole event takes ~15min.

Traffic arriving at or leaving the VMWare HOST shows normal performance 
levels, it's only traffic within the host that seems slow: SMB traffic 
across the pfSense router, no NAT involved, one pass-all pf rule, runs 
between 10Mbit/sec and 100Mbit/sec.  I also see lots of TCP badness if I 
run a sniffer on either end - dup acks, dup pkts, and missing packets.

I also have a lot (~7Mbyte/sec) of multicast traffic on one of the VLANs, 
which may contribute to the problem.

-Adam

> -----Original Message-----
> From: Scott Ullrich [mailto:sullrich <at> gmail.com]
> Sent: Saturday, October 02, 2010 13:37
> To: discussion <at> pfsense.com
> Subject: Re: [pfSense-discussion] pfSense router/firewall in a
> Vmware ESXi guest for other guests
>
> On Sat, Oct 2, 2010 at 2:27 PM, Adam Thompson <athompso <at> c3a.ca>
> wrote:
> > It works, but performance is, in my experience, poor.  Don't use
(Continue reading)

Chris Buechler | 2 Oct 21:53 2010
Picon

Re: pfSense router/firewall in a Vmware ESXi guest for other guests

On Sat, Oct 2, 2010 at 2:44 PM, Adam Thompson <athompso <at> c3a.ca> wrote:
> This started with 4.0, I have upgraded to 4.1 but haven't specifically
> tested performance since.  Routing from one VLAN to another entirely
> inside VMware is still slow, however.  AFAIK this is somehow related to
> interrupt handling and/or mitigation.  The bad news is that since
> upgrading to 4.1, the pfSense guest occasionally loses ALL network
> interrupts for about 15 minutes at a time - this happens at least once or
> twice a week.  It starts slowly, performance is merely degraded, then
> nothing, then slowly returns to normal - whole event takes ~15min.
>
> Traffic arriving at or leaving the VMWare HOST shows normal performance
> levels, it's only traffic within the host that seems slow: SMB traffic
> across the pfSense router, no NAT involved, one pass-all pf rule, runs
> between 10Mbit/sec and 100Mbit/sec.  I also see lots of TCP badness if I
> run a sniffer on either end - dup acks, dup pkts, and missing packets.
>

That's not the normal experience from what I've seen, sounds specific
to something in particular you're doing. I believe every environment
I've seen that routes between VLANs within ESX handles the VLANs
entirely at the ESX level, with one vswitch per VLAN and the firewall
connected to the individual vswitches, maybe that's the difference.

Running inside of VMware isn't nearly as fast as running on equivalent
bare metal, but most of the time you don't need that kind of
performance, 300 Mbps is easily achievable with e1000 NICs and
moderately new (anything with VT) server hardware. I've been on dozens
of such systems personally this year alone, across numerous different
customer environments. It's a common setup, and works well including
for routing between VLANs. I know at least a couple setups that route
(Continue reading)

Andrew C Burnette | 6 Oct 01:35 2010
Picon

Re: pfSense router/firewall in a Vmware ESXi guest for other guests

Make sure you've enabled all the various hardware assists in the 
server's BIOS  (Intel VT-x or AMD-V). Makes for a definite improvement 
in performance overall. No particular experience other than Intel 
chip'ed NIC cards, but they definitely seem to have excellent support 
for all the various configurations you'd need to have this perform well.

Good luck,
Andy

On 10/02/2010 03:53 PM, Chris Buechler wrote:
> On Sat, Oct 2, 2010 at 2:44 PM, Adam Thompson<athompso <at> c3a.ca>  wrote:
>> This started with 4.0, I have upgraded to 4.1 but haven't specifically
>> tested performance since.  Routing from one VLAN to another entirely
>> inside VMware is still slow, however.  AFAIK this is somehow related to
>> interrupt handling and/or mitigation.  The bad news is that since
>> upgrading to 4.1, the pfSense guest occasionally loses ALL network
>> interrupts for about 15 minutes at a time - this happens at least once or
>> twice a week.  It starts slowly, performance is merely degraded, then
>> nothing, then slowly returns to normal - whole event takes ~15min.
>>
>> Traffic arriving at or leaving the VMWare HOST shows normal performance
>> levels, it's only traffic within the host that seems slow: SMB traffic
>> across the pfSense router, no NAT involved, one pass-all pf rule, runs
>> between 10Mbit/sec and 100Mbit/sec.  I also see lots of TCP badness if I
>> run a sniffer on either end - dup acks, dup pkts, and missing packets.
>>
>
> That's not the normal experience from what I've seen, sounds specific
> to something in particular you're doing. I believe every environment
> I've seen that routes between VLANs within ESX handles the VLANs
(Continue reading)

Eugen Leitl | 7 Oct 15:43 2010

Re: pfSense router/firewall in a Vmware ESXi guest for other guests

On Sat, Oct 02, 2010 at 03:53:54PM -0400, Chris Buechler wrote:

> That's not the normal experience from what I've seen, sounds specific
> to something in particular you're doing. I believe every environment
> I've seen that routes between VLANs within ESX handles the VLANs
> entirely at the ESX level, with one vswitch per VLAN and the firewall
> connected to the individual vswitches, maybe that's the difference.
> 
> Running inside of VMware isn't nearly as fast as running on equivalent
> bare metal, but most of the time you don't need that kind of
> performance, 300 Mbps is easily achievable with e1000 NICs and
> moderately new (anything with VT) server hardware. I've been on dozens

Chris, how much memory do you recommend for a pfSense ESXi instance,
which handles 4 guests (one IP address each), 100 MBit/s switched 
setup? Do I need 1+ GByte, or can I risk allocating just 512 
MBytes to the guest? 

Can I allocate 1 virtual CPU to the pfSense instance, or should I
allocate 2? (This is a quadcore i7 box, with 8 GByte RAM).

Finally, will there be issues if I try for a pfSense carp+pfsync
failover, using two pfSense VMWare instances, each on abovementioned
i7 box? There's one Intel NIC present, each on a 100 MBit/s switched
port. Presumably, I can add another and connect both with a patch cable.
Nothing else heavy on the pfSense side, only haproxy.

Thanks!

> of such systems personally this year alone, across numerous different
(Continue reading)

Tim Dressel | 7 Oct 16:28 2010
Picon

Re: pfSense router/firewall in a Vmware ESXi guest for other guests

I gave mine a 10GB disk and 512MB ram, and two CPU's on a 4 core ESXi box (Xeon 3.2). I also ran squid on it. I find that VMware tends to give a perceived performance hit when you only assign a single core to a VM when the host is dual core and multi-processor. I'm not sure why this is.

Chris Buechler | 7 Oct 16:32 2010
Picon

Re: pfSense router/firewall in a Vmware ESXi guest for other guests

On Thu, Oct 7, 2010 at 3:43 PM, Eugen Leitl <eugen <at> leitl.org> wrote:
> On Sat, Oct 02, 2010 at 03:53:54PM -0400, Chris Buechler wrote:
>
>> That's not the normal experience from what I've seen, sounds specific
>> to something in particular you're doing. I believe every environment
>> I've seen that routes between VLANs within ESX handles the VLANs
>> entirely at the ESX level, with one vswitch per VLAN and the firewall
>> connected to the individual vswitches, maybe that's the difference.
>>
>> Running inside of VMware isn't nearly as fast as running on equivalent
>> bare metal, but most of the time you don't need that kind of
>> performance, 300 Mbps is easily achievable with e1000 NICs and
>> moderately new (anything with VT) server hardware. I've been on dozens
>
> Chris, how much memory do you recommend for a pfSense ESXi instance,
> which handles 4 guests (one IP address each), 100 MBit/s switched
> setup? Do I need 1+ GByte, or can I risk allocating just 512
> MBytes to the guest?
>

"It depends". Virtual sizing no diff from physical. Depends on
simultaneous connections, what packages and configurations they use,
etc. I use 128 MB RAM and 2 GB disks on most of my test and dev boxes,
they're mostly pretty basic though.

---------------------------------------------------------------------
To unsubscribe, e-mail: discussion-unsubscribe <at> pfsense.com
For additional commands, e-mail: discussion-help <at> pfsense.com

Commercial support available - https://portal.pfsense.org

Greg Hennessy | 7 Oct 16:45 2010
Picon

RE: pfSense router/firewall in a Vmware ESXi guest for other guests

If I may add one thought to this, 

Chokepoint have recently announced a virtual version of their 'blade' product which uses the VMSafe API to
enable more efficient inspection of traffic travelling between virtual machines and the outside world. 

http://www.networkworld.com/news/2010/090110-check-point-vmware-security.html?hpg1=bn

Dunno what the possibilty of such an approach is with pfSense. 

Given the innards of VMWare is linux based, the ABI is likely to be interesting for other operating systems
to interface against. 

Greg

________________________________________
From: Chris Buechler [cbuechler <at> gmail.com]
Sent: 07 October 2010 15:32
To: discussion <at> pfsense.com
Subject: Re: [pfSense-discussion] pfSense router/firewall in a Vmware ESXi guest for other guests

On Thu, Oct 7, 2010 at 3:43 PM, Eugen Leitl <eugen <at> leitl.org> wrote:
> On Sat, Oct 02, 2010 at 03:53:54PM -0400, Chris Buechler wrote:
>
>> That's not the normal experience from what I've seen, sounds specific
>> to something in particular you're doing. I believe every environment
>> I've seen that routes between VLANs within ESX handles the VLANs
>> entirely at the ESX level, with one vswitch per VLAN and the firewall
>> connected to the individual vswitches, maybe that's the difference.
>>
>> Running inside of VMware isn't nearly as fast as running on equivalent
>> bare metal, but most of the time you don't need that kind of
>> performance, 300 Mbps is easily achievable with e1000 NICs and
>> moderately new (anything with VT) server hardware. I've been on dozens
>
> Chris, how much memory do you recommend for a pfSense ESXi instance,
> which handles 4 guests (one IP address each), 100 MBit/s switched
> setup? Do I need 1+ GByte, or can I risk allocating just 512
> MBytes to the guest?
>

"It depends". Virtual sizing no diff from physical. Depends on
simultaneous connections, what packages and configurations they use,
etc. I use 128 MB RAM and 2 GB disks on most of my test and dev boxes,
they're mostly pretty basic though.

---------------------------------------------------------------------
To unsubscribe, e-mail: discussion-unsubscribe <at> pfsense.com
For additional commands, e-mail: discussion-help <at> pfsense.com

Commercial support available - https://portal.pfsense.org
---------------------------------------------------------------------
To unsubscribe, e-mail: discussion-unsubscribe <at> pfsense.com
For additional commands, e-mail: discussion-help <at> pfsense.com

Commercial support available - https://portal.pfsense.org

jason whitt | 7 Oct 21:20 2010
Picon

Re: pfSense router/firewall in a Vmware ESXi guest for other guests

As a note on cpu allocations for vnware vm's... this also applies to xen and hyper-v

Example:

Vmhost: 2 quad core cpu's

vm1 8 vcpu's
vm2 1 vcpu's
vm3 1 vcpu
vm4 1 vcpu
vm5 1vcpu

say vm1 is running something like active directory.. or any single threaded app. VMware will Que up all 8 vcpu's for vm1's cpu schedule. Once that's done it will release the cpu scheduler for the other vm's.

This is kinda hard to explain, however the bottom like is try out assigning no more that 1-2 cpu's per vm unless its needed. I use resource pools heavily to contain my cpu usage its often times ok to have a vm that's using 80% of its spu usage. this takes some time to test anc configure to achive optimal results. however you dont want to over provision your resources on the cpu side too much otherwise you'll run into contention, and if you are using vmware enterprise + with drs then you'll see vm's bounce from host to host in your cluster.... again not necessarily a bad thing just something to keep an eye on. I guess the answer is always it depends.



On Thu, Oct 7, 2010 at 8:45 AM, Greg Hennessy <Greg.Hennessy <at> nviz.net> wrote:
If I may add one thought to this,

Chokepoint have recently announced a virtual version of their 'blade' product which uses the VMSafe API to enable more efficient inspection of traffic travelling between virtual machines and the outside world.

http://www.networkworld.com/news/2010/090110-check-point-vmware-security.html?hpg1=bn

Dunno what the possibilty of such an approach is with pfSense.

Given the innards of VMWare is linux based, the ABI is likely to be interesting for other operating systems to interface against.



Greg


________________________________________
From: Chris Buechler [cbuechler <at> gmail.com]
Sent: 07 October 2010 15:32
To: discussion <at> pfsense.com
Subject: Re: [pfSense-discussion] pfSense router/firewall in a Vmware ESXi guest for other guests

On Thu, Oct 7, 2010 at 3:43 PM, Eugen Leitl <eugen <at> leitl.org> wrote:
> On Sat, Oct 02, 2010 at 03:53:54PM -0400, Chris Buechler wrote:
>
>> That's not the normal experience from what I've seen, sounds specific
>> to something in particular you're doing. I believe every environment
>> I've seen that routes between VLANs within ESX handles the VLANs
>> entirely at the ESX level, with one vswitch per VLAN and the firewall
>> connected to the individual vswitches, maybe that's the difference.
>>
>> Running inside of VMware isn't nearly as fast as running on equivalent
>> bare metal, but most of the time you don't need that kind of
>> performance, 300 Mbps is easily achievable with e1000 NICs and
>> moderately new (anything with VT) server hardware. I've been on dozens
>
> Chris, how much memory do you recommend for a pfSense ESXi instance,
> which handles 4 guests (one IP address each), 100 MBit/s switched
> setup? Do I need 1+ GByte, or can I risk allocating just 512
> MBytes to the guest?
>

"It depends". Virtual sizing no diff from physical. Depends on
simultaneous connections, what packages and configurations they use,
etc. I use 128 MB RAM and 2 GB disks on most of my test and dev boxes,
they're mostly pretty basic though.

---------------------------------------------------------------------
To unsubscribe, e-mail: discussion-unsubscribe <at> pfsense.com
For additional commands, e-mail: discussion-help <at> pfsense.com

Commercial support available - https://portal.pfsense.org
---------------------------------------------------------------------
To unsubscribe, e-mail: discussion-unsubscribe <at> pfsense.com
For additional commands, e-mail: discussion-help <at> pfsense.com

Commercial support available - https://portal.pfsense.org


Tim Dressel | 2 Oct 20:53 2010
Picon

Re: pfSense router/firewall in a Vmware ESXi guest for other guests

Hi folks,


I did this for about 6 months to do evaluations of Exchange 2010 and Zimbra.

My cluster had two VM hosts, each with 6 nics (2 onboard used for heartbeat, and an an in Intel PCIe quad port). I defined a LAN (vswitch) internal to the cluster only for traffic between all the VM's and the Lan side of the pfsense box. I also added one port from each of the VM hosts and connected to an external switch VLAN which was then directly connected to the internet. DRS and HA worked flawlessly.

This worked exceptionally well for the pfsense box. The VM hosts were dual processor dual core P4 Xeon's at 3.0Ghz. The internet connection was 100Mbit and I was easily able to get 80+Mbit across it. CPU use on the VM was never more than 20% of the single vCPU I assigned to it. In the 6 months we had it running it never burped once. It performed exactly like a hardware box. I did not install the VMware tools on pfsense.

I would not recommend this for a production scenario though, there are too many unknowns about the footprint that vmware might expose. Especially seeing any only computer will run pfsense very well if all you need is basic routing and NAT'ing.

This was on VMware ESXi 4.0 hosts, with a single vSphere manager.

We are currently playing with vyatta to do some really neat routing simulations for our larger network which is all cisco at the routing layer. We have several VRF's defined in our cisco's and have been playing with the open source patches to add this to the vyatta project that have not yet been integrated. For us, if we can prove this is stable in vmware, we will consider moving to hardware vyatta boxen.

Good luck!

Tim



Gmane