headers
Dan Stromberg | 7 Sep 2005 00:39
Picon

Some code, and a question


OK, I know NFS isn't usually thought of as the fastest protocol under
the sun, but still, there are times when making NFS move along a little
faster can be worthwhile.

I've written a sort of NFS benchmark that I'm calling nfs-test.  It
tries a largish number of rsize's, wsize's, tcp vs udp, and version 2
or 3 (4 would be very easy to add), to see what gives the best
performance.  You can find it at
http://dcs.nac.uci.edu/~strombrg/nfs-test.html

My question is, before diving into trying to determine this empirically,
is there any theoretical reason why it would be better to have
rsize==wsize, or should it be better to just pick whatever rsize gives
the best read performance and pick whatever wsize gives the best write
performance, and not worry about if rsize!=wsize?

Thanks!

-------------------------------------------------------
SF.Net email is Sponsored by the Better Software Conference & EXPO
September 19-22, 2005 * San Francisco, CA * Development Lifecycle Practices
Agile & Plan-Driven Development * Managing Projects & Teams * Testing & QA
Security * Process Improvement & Measurement * http://www.sqe.com/bsce5sf
_______________________________________________
NFS maillist  -  NFS <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
Permalink | Reply |
headers
Greg Banks | 7 Sep 2005 03:02
Picon
Favicon

Re: Some code, and a question

On Tue, Sep 06, 2005 at 03:39:57PM -0700, Dan Stromberg wrote:
> 
> OK, I know NFS isn't usually thought of as the fastest protocol under
> the sun,

Why would you think that?  NFSv3 can be very efficient at moving
bits from point A to point B.

> My question is, before diving into trying to determine this empirically,
> is there any theoretical reason why it would be better to have
> rsize==wsize,

From a protocol point of view, no.

> or should it be better to just pick whatever rsize gives
> the best read performance and pick whatever wsize gives the best write
> performance, and not worry about if rsize!=wsize?

It will depend on the workload, but generally read and write throughput
will be better the larger the block size, up to a value beyond the
Linux kernel's ability to support.  I expect you will find your optimum
at rsize=wsize=32K.

Greg.
--

-- 
Greg Banks, R&D Software Engineer, SGI Australian Software Group.
I don't speak for SGI.

-------------------------------------------------------
SF.Net email is Sponsored by the Better Software Conference & EXPO
(Continue reading)

Permalink | Reply |
headers
Dan Stromberg | 7 Sep 2005 16:37
Picon

Re: Some code, and a question

On Wed, 2005-09-07 at 11:02 +1000, Greg Banks wrote: 
> On Tue, Sep 06, 2005 at 03:39:57PM -0700, Dan Stromberg wrote:
> > 
> > OK, I know NFS isn't usually thought of as the fastest protocol under
> > the sun,
> 
> Why would you think that?  NFSv3 can be very efficient at moving
> bits from point A to point B.

You mean aside from the troublesome back-and-forthing on a high latency
network?

Perhaps this'll be less of an issue when NFS becomes extent-based.

> > My question is, before diving into trying to determine this empirically,
> > is there any theoretical reason why it would be better to have
> > rsize==wsize,
> 
> From a protocol point of view, no.

That much is interesting.  Thank you.

I'm also thinking about resource contention on the wire...

> > or should it be better to just pick whatever rsize gives
> > the best read performance and pick whatever wsize gives the best write
> > performance, and not worry about if rsize!=wsize?
> 
> It will depend on the workload, but generally read and write throughput
> will be better the larger the block size, up to a value beyond the
(Continue reading)

Permalink | Reply |
headers
Peter Staubach | 7 Sep 2005 16:55
Picon
Favicon

Re: Some code, and a question

Dan Stromberg wrote:

>
>Here's the summary output from my script.  You may find it surprising.
>It may have bugs, but so far it seems to be coming up with results that
>one might not expect.  This was iterating rsize's and wsize's from 4K to
>64K in steps of 1K.  BTW, this is from an AIX 5.1 host to a Solaris 9
>host, but the script should run on nearly any unix or linux:
>

Presumably you have made the configuration changes at least on the Solaris
side, /etc/system or some such, to allow these systems to go all the way to
a 64K transfer size?  Vanilla Solaris 9 won't do that.

What have you done to factor out the file system on the server?

    Thanx...

       ps

-------------------------------------------------------
SF.Net email is Sponsored by the Better Software Conference & EXPO
September 19-22, 2005 * San Francisco, CA * Development Lifecycle Practices
Agile & Plan-Driven Development * Managing Projects & Teams * Testing & QA
Security * Process Improvement & Measurement * http://www.sqe.com/bsce5sf
_______________________________________________
NFS maillist  -  NFS <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
(Continue reading)

Permalink | Reply |
headers
Dan Stromberg | 7 Sep 2005 19:00
Picon

Re: Some code, and a question

On Wed, 2005-09-07 at 10:55 -0400, Peter Staubach wrote:
> Dan Stromberg wrote:
> 
> >
> >Here's the summary output from my script.  You may find it surprising.
> >It may have bugs, but so far it seems to be coming up with results that
> >one might not expect.  This was iterating rsize's and wsize's from 4K to
> >64K in steps of 1K.  BTW, this is from an AIX 5.1 host to a Solaris 9
> >host, but the script should run on nearly any unix or linux:
> >
> 
> Presumably you have made the configuration changes at least on the Solaris
> side, /etc/system or some such, to allow these systems to go all the way to
> a 64K transfer size?  Vanilla Solaris 9 won't do that.

No I haven't - great lead.  I'll see if I can google that up.  Or if you
have the incantation at your fingertips...

> What have you done to factor out the file system on the server?

Nothing.  Actually, I don't really want to in this case, because it's
the speed as seen by the enduser that I need to optimize, not the speed
of NFS alone.  That is, if there's something specific to the combination
of NFS and the underlying QFS filesystem, I don't want my benchmarking
to miss that.

Thanks!

-------------------------------------------------------
SF.Net email is Sponsored by the Better Software Conference & EXPO
(Continue reading)

Permalink | Reply |
headers
Peter Staubach | 7 Sep 2005 19:13
Picon
Favicon

Re: Some code, and a question

Dan Stromberg wrote:

>On Wed, 2005-09-07 at 10:55 -0400, Peter Staubach wrote:
>  
>
>>Dan Stromberg wrote:
>>
>>    
>>
>>>Here's the summary output from my script.  You may find it surprising.
>>>It may have bugs, but so far it seems to be coming up with results that
>>>one might not expect.  This was iterating rsize's and wsize's from 4K to
>>>64K in steps of 1K.  BTW, this is from an AIX 5.1 host to a Solaris 9
>>>host, but the script should run on nearly any unix or linux:
>>>
>>>      
>>>
>>Presumably you have made the configuration changes at least on the Solaris
>>side, /etc/system or some such, to allow these systems to go all the way to
>>a 64K transfer size?  Vanilla Solaris 9 won't do that.
>>    
>>
>
>No I haven't - great lead.  I'll see if I can google that up.  Or if you
>have the incantation at your fingertips...
>
>  
>

For Solaris, you might check out adding something like:
(Continue reading)

Permalink | Reply |
headers
Peter Staubach | 7 Sep 2005 19:25
Picon
Favicon

Re: Some code, and a question

Peter Staubach wrote:

>>
>> No I haven't - great lead.  I'll see if I can google that up.  Or if you
>> have the incantation at your fingertips...
>>
>
> For Solaris, you might check out adding something like:
>
>    set nfs:nfs3_max_transfer_size=1048576
>    set nfs:nfs4_max_transfer_size=1048576
>
> to /etc/system and then reboot the system.  This will increase the 
> maximum
> size of a transfer to 1M.  Alternately, you could use adb to patch a 
> running
> system.  The command, "nfsstat -m", should tell you what the limits 
> are for
> currently mounted file systems.  You will need to umount and mount any
> existing NFS mounted file systems in order for them to be able to use the
> new limits. 

You will probably need to increase the NFS client side blocksize
as well.  This is nfs[34]_bsize.  So, something like:

    set nfs:nfs3_bsize=1048576
    set nfs:nfs4_bsize=1048576

although, you will probably want it to match your rsize/wsize settings.
(Continue reading)

Permalink | Reply |
headers
Dan Stromberg | 7 Sep 2005 19:28
Picon

Re: Some code, and a question

On Wed, 2005-09-07 at 13:13 -0400, Peter Staubach wrote:
> Dan Stromberg wrote:
> 
> >On Wed, 2005-09-07 at 10:55 -0400, Peter Staubach wrote:
> >  
> >
> >>Dan Stromberg wrote:
> >>
> >>    
> >>
> >>>Here's the summary output from my script.  You may find it surprising.
> >>>It may have bugs, but so far it seems to be coming up with results that
> >>>one might not expect.  This was iterating rsize's and wsize's from 4K to
> >>>64K in steps of 1K.  BTW, this is from an AIX 5.1 host to a Solaris 9
> >>>host, but the script should run on nearly any unix or linux:
> >>>
> >>>      
> >>>
> >>Presumably you have made the configuration changes at least on the Solaris
> >>side, /etc/system or some such, to allow these systems to go all the way to
> >>a 64K transfer size?  Vanilla Solaris 9 won't do that.
> >>    
> >>
> >
> >No I haven't - great lead.  I'll see if I can google that up.  Or if you
> >have the incantation at your fingertips...
> >
> >  
> >
>
(Continue reading)

Permalink | Reply |
headers
Peter Staubach | 7 Sep 2005 19:27
Picon
Favicon

Re: Some code, and a question

Dan Stromberg wrote:

>
>I set up our QFS.  I made the QFS transfer size the same as the
>underlying stripe size in the underlying RAID 5's.  Is there more to it
>than that?  I studied some QFS notes that a QFS instructor shared with
>me, but I didn't see anything more about QFS tuning than that.
>

I don't know how to tune a QFS file system.  I do know that it was 
considered
to be a black art and LSC/Sun consulting would often get involved.

       ps

-------------------------------------------------------
SF.Net email is Sponsored by the Better Software Conference & EXPO
September 19-22, 2005 * San Francisco, CA * Development Lifecycle Practices
Agile & Plan-Driven Development * Managing Projects & Teams * Testing & QA
Security * Process Improvement & Measurement * http://www.sqe.com/bsce5sf
_______________________________________________
NFS maillist  -  NFS <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
Permalink | Reply |

Gmane