gmane.comp.lang.tcl.core

Test case modularity

With the [chan pipe] patch and also with other, not-yet-TIPped stuff
of mine, I noticed that the checking of error messages in the test
suite is often made with strict string equality. This makes sense for
many messages which are likely frozen for eternity, but there is a
specific subpopulation which is more dynamic: messages enumerating
subcommands of an ensemble.

This means that every extension of an ensemble in the core needs
fixing test cases that are completely unrelated to the new subcommand:
tests expecting the message

    bad subcommand "foo": must be a, b, c, or d.

must be updated to check for

    bad subcommand "foo": must be a, b, c, d, or e.

Is there a good reason to do so ? Why not [regexp {^bad subcommand}] ?

-Alex

-------------------------------------------------------------------------
Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW!
Studies have shown that voting for your favorite open source project,
along with a healthy diet, reduces your potential for chronic lameness
and boredom. Vote Now at http://www.sourceforge.net/community/cca08

headers

Donal K. Fellows | 5 Jul 20:15

Re: Test case modularity

Alexandre Ferrieux wrote:
> This means that every extension of an ensemble in the core needs
> fixing test cases that are completely unrelated to the new subcommand:
> tests expecting the message
> 
>     bad subcommand "foo": must be a, b, c, or d.
> 
> must be updated to check for
> 
>     bad subcommand "foo": must be a, b, c, d, or e.
> 
> Is there a good reason to do so ? Why not [regexp {^bad subcommand}] ?

I've wondered about that in the past too. I'm not sure; perhaps it is
wiser to allow the checking that the list of subcommands is what it is
expected to be as it helps enforce coverage. But if you do want to do a
non-exact match, please do this instead:

   -match glob -result {bad subcommand "foo": must be *}

It's easier to read, and doesn't obscure what the string was when a
failure happens. A more complete example might be this:

   test chan-0.1 {bad chan subcommand} -returnCodes error -body {
       chan foobar
   } -match glob -result {bad subcommand "foobar": must be *}

Let's compare that with the traditional way of doing this (assuming
we're doing glob matching; regexps don't make this easier).

(Continue reading)

headers

Alexandre Ferrieux | 7 Jul 23:07

Re: Test case modularity

On 7/5/08, Donal K. Fellows <donal.k.fellows@...> wrote:
>
>  test chan-0.1 {bad chan subcommand} -returnCodes error -body {
>      chan foobar
>  } -match glob -result {bad subcommand "foobar": must be *}
>

FWIW, I have updated the ref impl of 304 to include these more generic
(and more modern) test cases in the three places where the addition of
[chan pipe] required updating anyway.

Also, thanks to TCT members who have already voted for 304.
I will be offline from now to July 21st, so when the deadline comes on
next Wednesday, if the outcome is positive, feel free to commit for
me.

-Alex

-------------------------------------------------------------------------
Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW!
Studies have shown that voting for your favorite open source project,
along with a healthy diet, reduces your potential for chronic lameness
and boredom. Vote Now at http://www.sourceforge.net/community/cca08

headers

Donald G Porter | 7 Jul 18:02

Re: Test case modularity

Alexandre Ferrieux wrote:
> This means that every extension of an ensemble in the core needs
> fixing test cases that are completely unrelated to the new subcommand:
> tests expecting the message
> 
>     bad subcommand "foo": must be a, b, c, or d.
> 
> must be updated to check for
> 
>     bad subcommand "foo": must be a, b, c, d, or e.
> 
> Is there a good reason to do so ? Why not [regexp {^bad subcommand}] ?

Let's take a step back away from implementation details to principles.

A test ought to exist for a purpose.  It ought to be testing something.

A test ought to be associated with some code getting tested.  The Tcl
test suite tends to follow the convention that tests in a file named
something like foo.test are for testing the code in tclFoo.c.

So if your test passes or fails based on the exact contents of an error
message, you are testing the code that constructs that error message,
whether that's your intent or not.

Thus, the text of an error message ought to be tested *at most* by
tests associated with the source code file where the content
of the error message is determined.  It makes no sense to test the
contents or formatting of an error message anywhere else.  All it does

(Continue reading)