Discussion of technical issues relating to NetBSD's networking subsystem

Petar Bogdanovic | 27 Apr 2012 20:10

hpn_buffer_size

Hi,

I'm trying to figure out, why two nearly identical netbsd-6 hosts have
troubles using hpn.  They are connected through a 5000 Kb/s link and the
following (not very scientific) test:

	dd if=FILE bs=1m count=2 | ssh HOST dd bs=64k of=/dev/null

yields different results when defaults are in place (430KB/s) and when
HPNDisabled=yes and/or HPNBufferSize=2048 (590KB/s).

I did a diff on the -vvv output, but am not really sure how to interpret
the values:

	--- hpn.default   2012-04-27 19:42:26.000000000 +0200
	+++ hpn.disabled  2012-04-27 19:42:48.000000000 +0200
	 <at>  <at>  -126,329 +126,449  <at>  <at> 
	 debug2: channel 0: open confirm rwindow 0 rmax 32768
	 debug2: tcpwinsz: 33580 for connection: 5
	 debug2: tcpwinsz: 33580 for connection: 5
	-debug2: channel 0: rcvd adjust 32768
	+debug2: channel 0: rcvd adjust 131072
	 debug2: channel_input_status_confirm: type 99 id 0
	 debug2: exec request accepted on channel 0
	 debug2: tcpwinsz: 33580 for connection: 5
	 debug2: tcpwinsz: 33580 for connection: 5
	 debug2: tcpwinsz: 33580 for connection: 5
	 debug2: tcpwinsz: 33580 for connection: 5
	-debug2: channel 0: rcvd adjust 32768
	 debug2: tcpwinsz: 33580 for connection: 5

(Continue reading)

Thor Lancelot Simon | 27 Apr 2012 20:52

Picon

Re: hpn_buffer_size

On Fri, Apr 27, 2012 at 08:10:07PM +0200, Petar Bogdanovic wrote:
> Hi,
> 
> I'm trying to figure out, why two nearly identical netbsd-6 hosts have
> troubles using hpn.  They are connected through a 5000 Kb/s link and the
> following (not very scientific) test:
> 
> 	dd if=FILE bs=1m count=2 | ssh HOST dd bs=64k of=/dev/null
> 
> yields different results when defaults are in place (430KB/s) and when
> HPNDisabled=yes and/or HPNBufferSize=2048 (590KB/s).

590 KB/sec on a 50000 Kb/sec link is pretty good!

It does strike me that you aren't moving enough data for measurements to
be particularly useful.  The SSH negotiation will have a significant
impact.

Thor

Petar Bogdanovic | 27 Apr 2012 21:22

Re: hpn_buffer_size

On Fri, Apr 27, 2012 at 02:52:41PM -0400, Thor Lancelot Simon wrote:
> 
> It does strike me that you aren't moving enough data for measurements to
> be particularly useful.  The SSH negotiation will have a significant
> impact.

I tried more, FILE is a regular file (a cd-image) the results are pretty
much the same.  Also not every application gets the same results.  An
ssh-tunneled bacula restore for example drops down to 100-200 KB/s while
the same restore gets ~590 KB/s to a netbsd-4 / netbsd-6 host (w/o HPN)
or fantastic 60 MB/s over gigabit to a Laptop (Cipher: arcfour).

Chuck Swiger | 27 Apr 2012 20:17

Picon

Re: hpn_buffer_size

On Apr 27, 2012, at 11:10 AM, Petar Bogdanovic wrote:
> I'm trying to figure out, why two nearly identical netbsd-6 hosts have
> troubles using hpn.  They are connected through a 5000 Kb/s link and the
> following (not very scientific) test:
> 
> 	dd if=FILE bs=1m count=2 | ssh HOST dd bs=64k of=/dev/null
> 
> yields different results when defaults are in place (430KB/s) and when
> HPNDisabled=yes and/or HPNBufferSize=2048 (590KB/s).

You need to increase the TCP window size to take significant advantage
of the SSH HPN changes.  Take a look at the following sysctls:

kern.sbmax
net.inet.tcp.rfc1323
net.inet.tcp.recvbuf_max
net.inet.tcp.sendbuf_max
net.inet.tcp.recvspace
net.inet.tcp.sendspace

You'll also need to increase NMBCLUSTERS, unless that has become
a sysctl tunable as well...

Regards,
--

-- 
-Chuck

Petar Bogdanovic | 27 Apr 2012 20:43

Re: hpn_buffer_size

On Fri, Apr 27, 2012 at 11:17:40AM -0700, Chuck Swiger wrote:
> On Apr 27, 2012, at 11:10 AM, Petar Bogdanovic wrote:
> > I'm trying to figure out, why two nearly identical netbsd-6 hosts have
> > troubles using hpn.  They are connected through a 5000 Kb/s link and the
> > following (not very scientific) test:
> > 
> > 	dd if=FILE bs=1m count=2 | ssh HOST dd bs=64k of=/dev/null
> > 
> > yields different results when defaults are in place (430KB/s) and when
> > HPNDisabled=yes and/or HPNBufferSize=2048 (590KB/s).
> 
> You need to increase the TCP window size to take significant advantage
> of the SSH HPN changes.  Take a look at the following sysctls:
> 
> kern.sbmax
> net.inet.tcp.rfc1323
> net.inet.tcp.recvbuf_max
> net.inet.tcp.sendbuf_max
> net.inet.tcp.recvspace
> net.inet.tcp.sendspace
> 
> You'll also need to increase NMBCLUSTERS, unless that has become
> a sysctl tunable as well...

Thanks for the hints but I'm not sure I understand the logic behind
this.  HPNDisabled=no is the default setting, and I'm not running high
powered n Gigabit-Interfaces so why should a default setting in a rather
unspectacular environment require any additional tweaking?

Or in other words: Wouldn't a HPNDisabled=yes default make more sense?

(Continue reading)

Chuck Swiger | 27 Apr 2012 21:28

Picon

Re: hpn_buffer_size

On Apr 27, 2012, at 11:43 AM, Petar Bogdanovic wrote:
>> You need to increase the TCP window size to take significant advantage
>> of the SSH HPN changes.  Take a look at the following sysctls:
>> 
>> kern.sbmax
>> net.inet.tcp.rfc1323
>> net.inet.tcp.recvbuf_max
>> net.inet.tcp.sendbuf_max
>> net.inet.tcp.recvspace
>> net.inet.tcp.sendspace
>> 
>> You'll also need to increase NMBCLUSTERS, unless that has become
>> a sysctl tunable as well...
> 
> Thanks for the hints but I'm not sure I understand the logic behind
> this.  HPNDisabled=no is the default setting, and I'm not running high
> powered n Gigabit-Interfaces so why should a default setting in a rather
> unspectacular environment require any additional tweaking?

The defaults are chosen to be reasonable for a wide range of circumstances, without consuming a lot of
network buffer memory.  If you have a high-speed link and want to improve performance, then the default
values are not ideal.  (Note that these values date back to when 10Mbs ethernet was "fast", so even a gigabit
link could be improved by tuning....)

> Or in other words: Wouldn't a HPNDisabled=yes default make more sense?

Maybe.  HPN also permits some other things like a null crypto type, which is useful when you want to use SSH
auth, but you don't care about encrypting the data being sent....

Regards,

(Continue reading)

Mouse | 28 Apr 2012 02:38

Re: hpn_buffer_size

> Maybe.  HPN also permits some other things like a null crypto type, which is$

Any particular reason you're using paragraph-length lines?  I'd suggest
avoiding it for normal running text like this.

> HPN also permits some other things like a null crypto type, which is
> useful when you want to use SSH auth, but you don't care about
> encrypting the data being sent....

I'm of two minds about that.  For people who actually understand the
issues and find the tradeoffs acceptable, it's convenient.  But for
people who think they understand more than they actually do, it's a
nice stout piece of rope already tied into a noose....  I'm not sure
where I come down on that question.  My own implementation won't use
none by default, but if you tell it to it won't give you any backtalk.

/~\ The ASCII				  Mouse
\ / Ribbon Campaign
 X  Against HTML		mouse <at> rodents-montreal.org
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

Chuck Swiger | 28 Apr 2012 08:14

Picon

Re: hpn_buffer_size

On Apr 27, 2012, at 5:38 PM, Mouse wrote:
>> Maybe.  HPN also permits some other things like a null crypto type, which is$
> 
> Any particular reason you're using paragraph-length lines?  I'd suggest
> avoiding it for normal running text like this.

Yes, there are particular reasons:

  http://lists.ntp.org/pipermail/questions/2011-September/030457.html

>> HPN also permits some other things like a null crypto type, which is
>> useful when you want to use SSH auth, but you don't care about
>> encrypting the data being sent....
> 
> I'm of two minds about that.  For people who actually understand the
> issues and find the tradeoffs acceptable, it's convenient.  But for
> people who think they understand more than they actually do, it's a
> nice stout piece of rope already tied into a noose....  I'm not sure
> where I come down on that question.  My own implementation won't use
> none by default, but if you tell it to it won't give you any backtalk.

This sounds entirely reasonable.  SSH should use encryption for data by
default, but the user should be able to chose the null cipher if they like.

Regards,
--

-- 
-Chuck

Mouse | 28 Apr 2012 08:43

Re: hpn_buffer_size

>> Any particular reason you're using paragraph-length lines?
> Yes, there are particular reasons:

>   http://lists.ntp.org/pipermail/questions/2011-September/030457.html

Quoting from that (with long-line damage manually repaired)...

> $WORK obligates me to use a MUA which supports i18n well, by which I
> don't mean UTF-8 and ISO-Latin-1, but various UTF-16 encodings like
> Big5/GBK and ISO-2022-JP.  Unfortunately, Mail.app no longer
> implements the format=flowed MIME Content-type.

Then I suggest you switch to some other user agent for non-$WORK mail,
or, if your job involves writing to these lists, external-to-$WORK work
mail.  Or at least insert a line break every 70 characters or so.
Using paragraph-length lines for running text substantially impairs its
readability for many people (especially on the more technical lists,
where rewrapping text not marked as rewrappable severely mangles things
like quoted dmesg output).

> Feel free to file your own bug reports with Apple [...]

I am no customer of theirs and am not likely to be; they have no reason
to pay any attention whatsoever to me.  I have no idea how to go about
filing a bug report with Apple and see no reason to learn, especially
since I expect it would involve things I am not willing to do (such as
giving them a nontrivial amount of information about myself that is not
relevant to the bug).  In any case, I see no reason *I* should go to
any trouble to make up for *your* not only choosing to use an MTA which
doesn't know how to use perfectly good applicable standards, but then,

(Continue reading)

Chuck Swiger | 28 Apr 2012 19:44

Picon

Re: hpn_buffer_size

On Apr 27, 2012, at 11:43 PM, Mouse wrote:
[ ... ]
> Quoting from that (with long-line damage manually repaired)...
> 
>> $WORK obligates me to use a MUA which supports i18n well, by which I
>> don't mean UTF-8 and ISO-Latin-1, but various UTF-16 encodings like
>> Big5/GBK and ISO-2022-JP.  Unfortunately, Mail.app no longer
>> implements the format=flowed MIME Content-type.
> 
> Then I suggest you switch to some other user agent for non-$WORK mail,
> or, if your job involves writing to these lists, external-to-$WORK work
> mail.  Or at least insert a line break every 70 characters or so.

It's simply not practical for me to switch MUAs, most of the time.

Nothing in my job involves writing to NetBSD mailing lists:
I'm a network administrator, not in Apple Developer Relations.

And yes, if I make a point of it, I can manually wrap text.

> Using paragraph-length lines for running text substantially impairs its
> readability for many people (especially on the more technical lists,
> where rewrapping text not marked as rewrappable severely mangles things
> like quoted dmesg output).

You're not saying anything here which I don't already know.

Fifteen or so years ago, I used to do GNKSA reviews:

  http://www.newsreaders.com/gnksa/gnksa.txt

(Continue reading)

Thor Lancelot Simon | 30 Apr 2012 15:51

Picon

Re: hpn_buffer_size

On Sat, Apr 28, 2012 at 10:44:28AM -0700, Chuck Swiger wrote:
> On Apr 27, 2012, at 11:43 PM, Mouse wrote:
> [ ... ]
> > Quoting from that (with long-line damage manually repaired)...
> > 
> >> $WORK obligates me to use a MUA which supports i18n well, by which I
> >> don't mean UTF-8 and ISO-Latin-1, but various UTF-16 encodings like
> >> Big5/GBK and ISO-2022-JP.  Unfortunately, Mail.app no longer
> >> implements the format=flowed MIME Content-type.
> > 
> > Then I suggest you switch to some other user agent for non-$WORK mail,
> > or, if your job involves writing to these lists, external-to-$WORK work
> > mail.  Or at least insert a line break every 70 characters or so.
> 
> It's simply not practical for me to switch MUAs, most of the time.

What if your preferred user agent made it impossible to do inline
quotations (or, at least, as hard as it is to wrap text at 80 in
Mail.app), so if you used it it was much easier to top-post?  Would
you start top-posting on old Internet mailing lists, too?

At some point, you should show some respect for the cultural conventions
of the place you're visiting.  Not because they're necessarily more
rational nor in some other way better (though I think there are good
arguments to be made that both hard-wrapping text and inline quotation
are, when compared to endless lines and top-posting) but because when
you want to communicate with other people, you should respect their
cultural norms.  Why should everyone else have to do extra work to
interact with you?  That just screams that you don't really care what
anyone else gets out of your participation in the discussion; that you're

(Continue reading)

Gert Doering | 30 Apr 2012 15:58

Picon

On Proper E-Mail Usage (no longer: Re: hpn_buffer_size)

Hi,

On Mon, Apr 30, 2012 at 09:51:22AM -0400, Thor Lancelot Simon wrote:
[..]
> What if your preferred user agent made it impossible to do inline
> quotations (or, at least, as hard as it is to wrap text at 80 in
> Mail.app), so if you used it it was much easier to top-post?  Would
> you start top-posting on old Internet mailing lists, too?
[..]

while at the topic of "well-behaving on mailing lists"... reading the
From: and Subject:, I expected to be enlightened about buffer tuning
issues, high-performance SSH, and so on.

Instead I read another sermon about use of mail clients, proper
formatting, and so on - which I can agree to, but which I'm not 
interested in reading.

Would you, being so wisened in the use of e-mail and human communication,
please bother to change the Subject: line accordingly, then?

thanks,

Gert Doering

--

-- 
USENET is *not* the non-clickable part of WWW!
                                                           //www.muc.de/~gert/
Gert Doering - Munich, Germany                             gert <at> greenie.muc.de
fax: +49-89-35655025                        gert <at> net.informatik.tu-muenchen.de

(Continue reading)

Chuck Swiger | 30 Apr 2012 18:43

Picon

Re: On Proper E-Mail Usage (no longer: Re: hpn_buffer_size)

On 4/30/2012 9:58 AM, Gert Doering wrote:
> On Mon, Apr 30, 2012 at 09:51:22AM -0400, Thor Lancelot Simon wrote:
> [..]
>> What if your preferred user agent made it impossible to do inline
>> quotations (or, at least, as hard as it is to wrap text at 80 in
>> Mail.app), so if you used it it was much easier to top-post?  Would
>> you start top-posting on old Internet mailing lists, too?
> [..]
>
> while at the topic of "well-behaving on mailing lists"... reading the
> From: and Subject:, I expected to be enlightened about buffer tuning
> issues, high-performance SSH, and so on.

Absolutely-- talking about SSH HPN and network tuning sysctls had been my 
original intent.

> Instead I read another sermon about use of mail clients, proper
> formatting, and so on - which I can agree to, but which I'm not
> interested in reading.
>
> Would you, being so wisened in the use of e-mail and human communication,
> please bother to change the Subject: line accordingly, then?

As I've said before, and to an individual CC:ed here: "Many folks are more apt 
to fault others than they are to fault themselves for the same error."

Well, Thor and Mouse, feel free to keep throwing stones if you like:
"though boys throw stones at the frogs in sport, the frogs die in earnest."

It seems explicit that some folks in the NetBSD community want me to leave,

(Continue reading)

Thor Lancelot Simon | 30 Apr 2012 18:46

Picon

Re: On Proper E-Mail Usage (no longer: Re: hpn_buffer_size)

On Mon, Apr 30, 2012 at 12:43:31PM -0400, Chuck Swiger wrote:
> 
> It seems explicit that some folks in the NetBSD community want me to
> leave, and I'll respect such wishes by unsubscribing from these
> lists.

Explicit?  I'd appreciate a quotation, then -- who has said they want
you to leave?

Personally, I'd prefer that you not leave more than I'd prefer that you
stop posting infinite paragraphs.  But should that mean I am somehow not
permitted to tell you that I would like it if you'd stop posting infinite
paragraphs?

Thor

Mouse | 30 Apr 2012 21:06

Re: On Proper E-Mail Usage (no longer: Re: hpn_buffer_size)

> It seems explicit that some folks in the NetBSD community want me to
> leave, and I'll respect such wishes by unsubscribing from these
> lists.

It could be true.  But I don't think I'm part of that "some folks"; I
certainly did not intend to say any such thing.

If it comes down to a choice of having you here with paragraph-sized
lines or having you gone...I see arguments each way and don't know
which side I come down on; perhaps fortunately, I don't think I need to
figure that out, because it's not my choice to make.  I don't run the
list and most certainly don't run NetBSD.

/~\ The ASCII				  Mouse
\ / Ribbon Campaign
 X  Against HTML		mouse <at> rodents-montreal.org
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

David Young | 8 May 2012 01:34

Picon

the alternative to stupid email wars (was Re: hpn_buffer_size)

On Mon, Apr 30, 2012 at 09:51:22AM -0400, Thor Lancelot Simon wrote:
> For a reader (and perhaps more important, respondent) using the tools
> many of us traditionally use to interact here, on the NetBSD mailing lists,
> 500 word lines are a lot like that.  They display strangely in most user
> agents for character-cell terminals.  They're hard to quote properly when
> responding because one effectively has to reflow the text by hand.  Sure,
> one could quote them as 500 word lines; but that just kicks the can down
> the road to the next person who might want to respond to (and, as is
> polite, quote) only part of a paragraph.  They basically scream "I don't
> care how much work you have to do to have a conversation with me, it's
> all about what _I_ have to say."  Is it as bad as top-posting, where
> rearranging a message into a sane order for response could take 10 minutes?
> No.  But it's still rude, in the same way.  Or so it seems to me.

It seems to me that the time spent having these arguments would be
better spent designing or writing filters that do helpful things such as

* re-wrap lengthy emails at whichever width one prefers,

* convert a discussion thread consisting of top- and bottom-posted
  replies into one style or the other

* processes an mbox-format file containing a lengthy discussion thread
  containing quotes, and quotes of quotes, and re-prints the entire
  discussion as one document without any duplicate text

Dave

--

-- 
David Young

(Continue reading)

Matthew Mondor | 8 May 2012 02:16

Re: the alternative to stupid email wars (was Re: hpn_buffer_size)

On Mon, 7 May 2012 18:34:52 -0500
David Young <dyoung <at> pobox.com> wrote:

> It seems to me that the time spent having these arguments would be
> better spent designing or writing filters that do helpful things such as
> 
> * re-wrap lengthy emails at whichever width one prefers,
> 
> * convert a discussion thread consisting of top- and bottom-posted
>   replies into one style or the other
> 
> * processes an mbox-format file containing a lengthy discussion thread
>   containing quotes, and quotes of quotes, and re-prints the entire
>   discussion as one document without any duplicate text

Using client-side tools, we can indeed preprocess mail as we want/need,
but someone should also be able to simply use mutt or the client of
their choice to read the mail...

Unfortunately, tempering with mail content can also be problematic,
although I commonly see it on archive http frontends... Among the
problems (especially problematic if done server or transit-side):

- It should be able to distinguish between lines that should be wrapped
  and those that shouldn't, both of which commonly occur in the same
  message on tech lists (format-flowed apparently allows that but few
  clients support it correctly or allow to tag lines to be flowed and
  those which shouldn't; moreover some clients supporting it will
  output unreadable quoted-printable source).  Hence we'd need something
  like a text markup convention (I'll avoid mentioning HTML! :))

(Continue reading)

Thor Lancelot Simon | 30 Apr 2012 16:03

Picon

Re: hpn_buffer_size

On Fri, Apr 27, 2012 at 08:10:07PM +0200, Petar Bogdanovic wrote:
> Hi,
> 
> I'm trying to figure out, why two nearly identical netbsd-6 hosts have
> troubles using hpn.  They are connected through a 5000 Kb/s link and the
> following (not very scientific) test:
> 
> 	dd if=FILE bs=1m count=2 | ssh HOST dd bs=64k of=/dev/null
> 
> yields different results when defaults are in place (430KB/s) and when
> HPNDisabled=yes and/or HPNBufferSize=2048 (590KB/s).

I do think you need to test with larger files.  But the other thing that
occurs to me is that you're now using a kernel that automatically sizes
the TCP socket buffers; HPN has a tunable that needs to be set right if
this is the case (RcvBufferPolling?  Something like that.) which you
might want to investigate.

In fact, I think I'd first make sure it were set appropriately for a
kernel that _does_ auto-size the socket buffers, then try setting it
the other way and using systctl to set the socket buffer sizes by hand
(you will need to do this on *both* ends and restart the ssh daemon
process after each of these changes) to see whether the problem is either
A) HPN not playing nice with our socket buffer autosizing code due to wrong
default settings, or perhaps B) our socket buffer autosizing code being
broken.

Thor

(Continue reading)

Return

Return to gmane.os.netbsd.devel.network.

Search Archive

Language

Change language

Options

Current view: Threads only / Showing only 20 lines / Not hiding cited text.
Change to All messages, whole messages, or hide cited text.

Post a message
NNTP Newsgroup
Classic Gmane web interface
XML

XML

RSS Feed
List Information

About Gmane