David Abrahams | 5 Sep 03:48
Favicon
Gravatar

[testing] How not to mark up explicit failures


https://svn.boost.org/trac/boost/ticket/2258 describes a problem with
explicit failure markup that I suspect is not atypical.  I believe we
need to establish some ground rules for markup that ensure 

 * Boost problems don't pass unnoticed

 * there is a uniform standard for interpreting test results across
   libraries

 * "non-Boost problems" and non-problems don't add alarm bells for
   people reading test results

You can view the markup in question here:
https://svn.boost.org/trac/boost/browser/trunk/status/explicit-failures-markup.xml#L1581

Aside from the specifics in the ticket, the following problems jump out
at me just from looking at that XML:

 * even on compilers that do support nonstandard calling conventions,
   failures in these tests will never be flagged as problematic

 * even on compilers that don't support nonstandard calling conventions,
   these tests will fail and add useless noise to the chart.

I can think of a few general principles and practices that will prevent
such problems:

 * Boost regression tests should test the functionality of code that is
   expected to work, not to test the capabilities of the platform or C++
(Continue reading)


Gmane