Martti Kuparinen | 1 Jul 2003 06:28
Picon
Picon
Favicon

Re: 1.6.1 eats memory

Andrey Petrov wrote:
> On Mon, Jun 30, 2003 at 03:59:55PM +0300, Martti Kuparinen wrote:
>
>>>>[ mbuf leak in 1.6.1 ]

> NFS? I saw similar things on -current a month back. Update helped
> so I havn't looked any more.

Yes, the host is using NFS to mount /home.

Martti

Martti Kuparinen | 1 Jul 2003 08:03
Picon
Picon
Favicon

Re: 1.6.1 eats memory

It's happening again :-( So this might be a NFS bug...

My www server running 1.6.1 without NFS (uptime 80+ days):

39 mbufs in use:
         20 mbufs allocated to data
         19 mbufs allocated to packet headers
53/62 mapped pages in use
196 Kbytes allocated to network (71% in use)
0 requests for memory denied
0 requests for memory delayed
1286 calls to protocol drain routines

The trouble maker running 1.6.1_STABLE (uptime 1+ days):

58469 mbufs in use:
         58461 mbufs allocated to data
         7 mbufs allocated to packet headers
         1 mbufs allocated to socket names and addresses
156/164 mapped pages in use
14964 Kbytes allocated to network (99% in use)
0 requests for memory denied
0 requests for memory delayed
54 calls to protocol drain routines

Martti

Martti Kuparinen | 2 Jul 2003 08:40
Picon
Picon
Favicon

Re: 1.6.1 eats memory

Martti Kuparinen wrote:

> The trouble maker running 1.6.1_STABLE (uptime 1+ days):
> 
> 58469 mbufs in use:
>         58461 mbufs allocated to data
>         7 mbufs allocated to packet headers
>         1 mbufs allocated to socket names and addresses
> 156/164 mapped pages in use
> 14964 Kbytes allocated to network (99% in use)
> 0 requests for memory denied
> 0 requests for memory delayed
> 54 calls to protocol drain routines

Just to clarify things: this is a NFS client (not a server!) mounting 
/home from another host.

Martti

Michael van Elst | 2 Jul 2003 08:48
Picon

Re: 1.6.1 eats memory

martti.kuparinen <at> iki.fi (Martti Kuparinen) writes:

>> 58469 mbufs in use:
>>         58461 mbufs allocated to data

>Just to clarify things: this is a NFS client (not a server!) mounting 
>/home from another host.

Maybe kern/13807 is the answer ?
-- 
--

-- 
                                Michael van Elst
Internet: mlelstv <at> serpens.de
                                "A potential Snark may lurk in every tree."

Andrey Petrov | 2 Jul 2003 08:44
Picon

Re: 1.6.1 eats memory

On Wed, Jul 02, 2003 at 09:40:16AM +0300, Martti Kuparinen wrote:
> Martti Kuparinen wrote:
> 
> >The trouble maker running 1.6.1_STABLE (uptime 1+ days):
> >
> >58469 mbufs in use:
> >        58461 mbufs allocated to data
> >        7 mbufs allocated to packet headers
> >        1 mbufs allocated to socket names and addresses
> >156/164 mapped pages in use
> >14964 Kbytes allocated to network (99% in use)
> >0 requests for memory denied
> >0 requests for memory delayed
> >54 calls to protocol drain routines
> 
> Just to clarify things: this is a NFS client (not a server!) mounting 
> /home from another host.
> 

If 1.6.1 doesn't leak memory then I'd look at diff between kernels
and try to find a commit which caused that. I'd pay extra attention
to nfs code. Test should be straightforward, revert changes, build and
try.

	Andrey

Martti Kuparinen | 2 Jul 2003 08:55
Picon
Picon
Favicon

Re: 1.6.1 eats memory

Andrey Petrov wrote:

>>Just to clarify things: this is a NFS client (not a server!) mounting 
>>/home from another host.
 >
> If 1.6.1 doesn't leak memory

But it does :-(

I wrote earlier that my 1.6.1 server *without* NFS is working great. 
This NFS client with IPsec leaked with 1.6.1 and still leaks with 
1.6.1_STABLE.

I'll take a look at kern/13807 and see if it fixes my problems.

Martti

Andrey Petrov | 2 Jul 2003 09:05
Picon

Re: 1.6.1 eats memory

On Wed, Jul 02, 2003 at 09:55:06AM +0300, Martti Kuparinen wrote:
> Andrey Petrov wrote:
> 
> >>Just to clarify things: this is a NFS client (not a server!) mounting 
> >>/home from another host.
> >
> >If 1.6.1 doesn't leak memory
> 
> But it does :-(
> 

Bummer.

> I wrote earlier that my 1.6.1 server *without* NFS is working great. 
> This NFS client with IPsec leaked with 1.6.1 and still leaks with 
> 1.6.1_STABLE.
> 
> I'll take a look at kern/13807 and see if it fixes my problems.
> 

You can also try NFS without IPsec, so problem will be more localized,
it would be interesting to know how -current does in your setup.

	Andrey

Michael van Elst | 2 Jul 2003 21:33
Picon

Re: 1.6.1 eats memory

petrov <at> netbsd.org (Andrey Petrov) writes:

>You can also try NFS without IPsec, so problem will be more localized,

NFS leaks memory when network traffic uses mbufs with 'aux' buffers
because the code manually pokes mbuf structures to align data, but
then ignores (and leaks) 'aux' buffers.

'aux' buffers are used by IPsec and maybe other subsystems. Not using
IPsec will hide the problem and (as it did to me) hints to IPsec
as the culprit, which is the wrong direction.

<rant>
After I wrote the patch and asked for review I got three different
answers from NetBSD developers:

- nobody wants to look at the NFS code, it's too ugly.
- nobody wants to look at mbuf code, it's too complex.
- 'aux' buffers have been a bad idea from the beginning
</rant>

-- 
--

-- 
                                Michael van Elst
Internet: mlelstv <at> serpens.de
                                "A potential Snark may lurk in every tree."

Martti Kuparinen | 2 Jul 2003 09:11
Picon
Picon
Favicon

Re: 1.6.1 eats memory

Andrey Petrov wrote:

>>I'll take a look at kern/13807 and see if it fixes my problems.
> 
> You can also try NFS without IPsec, so problem will be more localized,
> it would be interesting to know how -current does in your setup.

I applied the patch from kern/13807 and it makes things a lot better! 
This is my current statistics and it looks normal:

131 mbufs in use:
         128 mbufs allocated to data
         2 mbufs allocated to packet headers
         1 mbufs allocated to socket names and addresses
136/144 mapped pages in use
340 Kbytes allocated to network (93% in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines

I'll post more statistics later...

Martti

Martti Kuparinen | 3 Jul 2003 08:01
Picon
Picon
Favicon

Re: 1.6.1 eats memory

Martti Kuparinen wrote:

> I applied the patch from kern/13807 and it makes things a lot better! 
> This is my current statistics and it looks normal:
> 
> 131 mbufs in use:
>         128 mbufs allocated to data
>         2 mbufs allocated to packet headers
>         1 mbufs allocated to socket names and addresses
> 136/144 mapped pages in use
> 340 Kbytes allocated to network (93% in use)
> 0 requests for memory denied
> 0 requests for memory delayed
> 0 calls to protocol drain routines

uptime is now 1 day:

139 mbufs in use:
         133 mbufs allocated to data
         5 mbufs allocated to packet headers
         1 mbufs allocated to socket names and addresses
169/178 mapped pages in use
436 Kbytes allocated to network (94% in use)
0 requests for memory denied
0 requests for memory delayed
20 calls to protocol drain routines

So could our "NFS guru" take a look at this?

Martti
(Continue reading)


Gmane