headers
Eric Blake | 23 Nov 2008 15:56
Gravatar

Re: verbose testsuite output contains 1M newlines


[adding autoconf; this is an interesting question about autotest]

According to Ralf Wildenhues on 11/23/2008 6:23 AM:
> Hello Eric,

Hi Ralf,

> 
> running
>   make check-local TESTSUITEFLAGS=-v
> 
> on M4 (git master) produces lots of output on GNU/Linux/bash.  More
> precisely, the AT_CHECK in builtins.at:470 causes 1024*1024 newlines
> to be output, and I think there are more instances later.  Can this
> easily be avoided?

Hmm, this particular test _has_ to generate more than a megabyte of output
in order to trigger the code path in m4 being tested (namely, that m4
starts using temporary files after it collects enough in-memory diversion
output).  The problem stems from the fact that by default, when you
specify [stdout] as the third argument to AT_CHECK, autotest emits code
that both collects the file stdout (good) and copies the resulting file
into the log (annoying), and when -v is in effect, tees that copy to the
terminal (bad).  I think the idea behind this decision was that
testsuite.log should be self-contained, with everything needed to
reproduce why a test failed.

The point of specifying [stdout] is that we plan on processing the
resulting output file later on, so maybe we should quit trying to copy the
(Continue reading)

Permalink | Reply |
headers
Eric Blake | 11 May 2009 21:45
Gravatar

Re: verbose testsuite output contains 1M newlines

Eric Blake <ebb9 <at> byu.net> writes:

> According to Ralf Wildenhues on 11/23/2008 6:23 AM:

Hi Ralf, and sorry for the long delay,

> >   make check-local TESTSUITEFLAGS=-v
> > 
> > on M4 (git master) produces lots of output on GNU/Linux/bash.  More
> > precisely, the AT_CHECK in builtins.at:470 causes 1024*1024 newlines
> > to be output, and I think there are more instances later.  Can this
> > easily be avoided?
> 
> Hmm, this particular test _has_ to generate more than a megabyte of output
> in order to trigger the code path in m4 being tested (namely, that m4
> starts using temporary files after it collects enough in-memory diversion
> output).  The problem stems from the fact that by default, when you
> specify [stdout] as the third argument to AT_CHECK, autotest emits code
> that both collects the file stdout (good) and copies the resulting file
> into the log (annoying), and when -v is in effect, tees that copy to the
> terminal (bad).  I think the idea behind this decision was that
> testsuite.log should be self-contained, with everything needed to
> reproduce why a test failed.
> 
> The point of specifying [stdout] is that we plan on processing the
> resulting output file later on, so maybe we should quit trying to copy the
> captured file into the logs.  Maybe we should revisit this, and change it
> so that the file is just collected, with nothing output into the log.  Or
> even add a heuristic, where only the first and last 10 lines of the
> resulting file are output into the log.  If the test passed, then the log
(Continue reading)

Permalink | Reply |
headers
Ralf Wildenhues | 23 Nov 2008 17:37
Picon
Picon

Re: verbose testsuite output contains 1M newlines

* Eric Blake wrote on Sun, Nov 23, 2008 at 03:56:49PM CET:
> According to Ralf Wildenhues on 11/23/2008 6:23 AM:
> > running
> >   make check-local TESTSUITEFLAGS=-v
> > 
> > on M4 (git master) produces lots of output on GNU/Linux/bash.

> Hmm, this particular test _has_ to generate more than a megabyte of output
> in order to trigger the code path in m4 being tested (namely, that m4
> starts using temporary files after it collects enough in-memory diversion
> output).  The problem stems from the fact that by default, when you
> specify [stdout] as the third argument to AT_CHECK, autotest emits code
> that both collects the file stdout (good) and copies the resulting file
> into the log (annoying), and when -v is in effect, tees that copy to the
> terminal (bad).  I think the idea behind this decision was that
> testsuite.log should be self-contained, with everything needed to
> reproduce why a test failed.

Yes.  I think the Autotest behavior is more or less ok for most tests.
Maybe we can have AT_CHECK modifiers or sister macros that allow to
specify that some output may not be suitable for the screen or the log
file, due to size or content such as control characters.

> The point of specifying [stdout] is that we plan on processing the
> resulting output file later on, so maybe we should quit trying to copy the
> captured file into the logs.

Naah, I like the default.

> Maybe we should revisit this, and change it
(Continue reading)

Permalink | Reply |
headers
Eric Blake | 13 Apr 2009 15:14
Gravatar

Re: verbose testsuite output contains 1M newlines


According to Ralf Wildenhues on 11/23/2008 9:37 AM:
> * Eric Blake wrote on Sun, Nov 23, 2008 at 03:56:49PM CET:
>> According to Ralf Wildenhues on 11/23/2008 6:23 AM:
>>> running
>>>   make check-local TESTSUITEFLAGS=-v
>>>
>>> on M4 (git master) produces lots of output on GNU/Linux/bash.
> 
>> The problem stems from the fact that by default, when you
>> specify [stdout] as the third argument to AT_CHECK, autotest emits code
>> that both collects the file stdout (good) and copies the resulting file
>> into the log (annoying), and when -v is in effect, tees that copy to the
>> terminal (bad).  I think the idea behind this decision was that
>> testsuite.log should be self-contained, with everything needed to
>> reproduce why a test failed.
> 
> Yes.  I think the Autotest behavior is more or less ok for most tests.
> Maybe we can have AT_CHECK modifiers or sister macros that allow to
> specify that some output may not be suitable for the screen or the log
> file, due to size or content such as control characters.

Here's my proposal for the autotest side - ignore-nolog, stdout-nolog, and
stderr-nolog.  If you like this, then I can follow up with an m4 patch to
use them on the particular tests where we really don't need to see 1M of
newlines.

--
Don't work too hard, make some time for fun as well!
(Continue reading)

Permalink | Reply |
headers
Ralf Wildenhues | 14 Apr 2009 21:26
Picon
Picon

Re: verbose testsuite output contains 1M newlines

Hi Eric,

* Eric Blake wrote on Mon, Apr 13, 2009 at 03:14:21PM CEST:
> According to Ralf Wildenhues on 11/23/2008 9:37 AM:
> > Maybe we can have AT_CHECK modifiers or sister macros that allow to
> > specify that some output may not be suitable for the screen or the log
> > file, due to size or content such as control characters.
> 
> Here's my proposal for the autotest side - ignore-nolog, stdout-nolog, and
> stderr-nolog.  If you like this, then I can follow up with an m4 patch to
> use them on the particular tests where we really don't need to see 1M of
> newlines.

Almost.  It seems inconsistent that only ignore-nolog allows non-text
output.  What about ensuring that std{out,err}-nolog do likewise, by
sending some large lines to all three in a test?

Also, can we use 'diff' for comparing non-text files?  POSIX seems to
say yes, and autoconf.info doesn't disagree, but my memory isn't clear
on this.  A test could find this out for us.

Thanks,
Ralf
Permalink | Reply |
headers
Eric Blake | 15 Apr 2009 03:17
Gravatar

Re: verbose testsuite output contains 1M newlines


[dropping bug-m4 for this post]

According to Ralf Wildenhues on 4/14/2009 1:26 PM:
> Almost.  It seems inconsistent that only ignore-nolog allows non-text
> output.  What about ensuring that std{out,err}-nolog do likewise, by
> sending some large lines to all three in a test?

Or even files that lack a trailing newline, or contain non-printing
characters (or all three).  I'm working up a test, and will post it, along
with the other patches, as a series when I'm complete.

> Also, can we use 'diff' for comparing non-text files?  POSIX seems to
> say yes, and autoconf.info doesn't disagree, but my memory isn't clear
> on this.  A test could find this out for us.

POSIX requires it (think about the special rules for when a file does not
have a trailing newline, and about 'binary files differ'.  Depending on
the exact behavior is thus non-portable, but detecting same vs. different
should be portable.  But adding the above test should provide enough of a
stress test that we can find out about any broken diff apps out there.

--
Don't work too hard, make some time for fun as well!

Eric Blake             ebb9 <at> byu.net
Permalink | Reply |
headers
Eric Blake | 15 Apr 2009 14:47
Gravatar

Re: verbose testsuite output contains 1M newlines


According to Eric Blake on 4/14/2009 7:17 PM:
> According to Ralf Wildenhues on 4/14/2009 1:26 PM:
>> Almost.  It seems inconsistent that only ignore-nolog allows non-text
>> output.  What about ensuring that std{out,err}-nolog do likewise, by
>> sending some large lines to all three in a test?
> 
> Or even files that lack a trailing newline, or contain non-printing
> characters (or all three).  I'm working up a test, and will post it, along
> with the other patches, as a series when I'm complete.

Here's what I added to the series before pushing.  I don't really know how
to parse that diff gave reasonable output, as $at_diff may or may not be
using -u, and may print 'binary files differ' instead of comparing
contents (not to mention that grep and sed are not required to handle
binary files).  But by testing that valid binary tests compare equal, and
slightly modified binary tests compare different, I think I've covered the
intent that we want autotest to handle non-text files.

--
Don't work too hard, make some time for fun as well!

Eric Blake             ebb9 <at> byu.net

From 7ece247bdd194997f5a79aeaad3e7b02bc7130e2 Mon Sep 17 00:00:00 2001
From: Eric Blake <ebb9 <at> byu.net>
Date: Wed, 15 Apr 2009 06:38:58 -0600
Subject: [PATCH] Test that autotest handles binary output.
(Continue reading)

Permalink | Reply |
headers
Ralf Wildenhues | 15 Apr 2009 20:05
Picon
Picon

Re: verbose testsuite output contains 1M newlines

Hi Eric,

* Eric Blake wrote on Wed, Apr 15, 2009 at 02:47:26PM CEST:
> According to Eric Blake on 4/14/2009 7:17 PM:
> > According to Ralf Wildenhues on 4/14/2009 1:26 PM:
> >> Almost.  It seems inconsistent that only ignore-nolog allows non-text
> >> output.  What about ensuring that std{out,err}-nolog do likewise, by
> >> sending some large lines to all three in a test?
> > 
> > Or even files that lack a trailing newline, or contain non-printing
> > characters (or all three).  I'm working up a test, and will post it, along
> > with the other patches, as a series when I'm complete.
> 
> Here's what I added to the series before pushing.  I don't really know how
> to parse that diff gave reasonable output, as $at_diff may or may not be
> using -u, and may print 'binary files differ' instead of comparing
> contents (not to mention that grep and sed are not required to handle
> binary files).  But by testing that valid binary tests compare equal, and
> slightly modified binary tests compare different, I think I've covered the
> intent that we want autotest to handle non-text files.

Thanks for finishing this, it looks good.

I bet this will evoke some failures on legacy systems.  We'll see.

Cheers,
Ralf

> >From 7ece247bdd194997f5a79aeaad3e7b02bc7130e2 Mon Sep 17 00:00:00 2001
> From: Eric Blake <ebb9 <at> byu.net>
(Continue reading)

Permalink | Reply |

Gmane