5 Sep 03:48
[testing] How not to mark up explicit failures
From: David Abrahams <dave <at> boostpro.com>
Subject: [testing] How not to mark up explicit failures
Newsgroups: gmane.comp.lib.boost.devel
Date: 2008-09-05 01:52:25 GMT
Subject: [testing] How not to mark up explicit failures
Newsgroups: gmane.comp.lib.boost.devel
Date: 2008-09-05 01:52:25 GMT
https://svn.boost.org/trac/boost/ticket/2258 describes a problem with explicit failure markup that I suspect is not atypical. I believe we need to establish some ground rules for markup that ensure * Boost problems don't pass unnoticed * there is a uniform standard for interpreting test results across libraries * "non-Boost problems" and non-problems don't add alarm bells for people reading test results You can view the markup in question here: https://svn.boost.org/trac/boost/browser/trunk/status/explicit-failures-markup.xml#L1581 Aside from the specifics in the ticket, the following problems jump out at me just from looking at that XML: * even on compilers that do support nonstandard calling conventions, failures in these tests will never be flagged as problematic * even on compilers that don't support nonstandard calling conventions, these tests will fail and add useless noise to the chart. I can think of a few general principles and practices that will prevent such problems: * Boost regression tests should test the functionality of code that is expected to work, not to test the capabilities of the platform or C++(Continue reading)
RSS Feed