headers
Bertrik Sikken | 29 May 2012 18:58
Picon
Favicon

Pre-release testing framework

Hi all,

Last week at devcon euro 2012 we talked about pre-release testing and
I'm volunteering to help things forward w.r.t. testing.

One of the problems we've seen with the last release was that really
basic functionality like audio file playback and radio playback did
not work on some targets and we were not aware of it until many days
later. Back in 2010 Björn already proposed a framework to recruit
the help of users to test release candidates in a systematic way:
http://www.rockbox.org/mail/archive/rockbox-dev-archive-2010-10 /0114.shtml
This helps to at least be *aware* of any problems (what to do with
that information is another discussion).

I probably can't help with setting up a web-based framework, but
I can help with thinking up test cases.

The test case format I'm familiar with, defines the following
properties:
* some unique test case id, like
  "basics.playback.001"
* a descriptive title, like
  "Verify basic audio playback for all supported audio formats"
* (a reference to a requirement or spec, not really relevant for
  rockbox I think)
* a pre-condition, like:
  "The audio format test file set 1 has been copied to the player"
* action to perform the test, like:
  "Using the file browser, browse to the test file set and select the
  first audio file. Verify that all files play correctly."
(Continue reading)

Permalink | Reply |
headers
Lorenzo Miori | 29 May 2012 20:28
Picon

Re: Pre-release testing framework

I just read the email about the test cases: well it is a very good
idea. I completely agree this and actually doing a project with a
methodology can help quite a lot.
I don't know if you guys are aware of the "agile" methodologies, well
look for "user stories" and "acceptance tests" (XP/Scrum and Agile
methodologies)

Basically an user story is a table with a title, a description (with
point / risk and priority) and acceptance test name (every card is
enumerated with a name).
Acceptance test defines input(s), precondition(s), and expected result(s).

Example:

User story # 1: playAudio
Priority: 1
Points: 3
Description: player must be able to output some sound. [Avoid specific
terminology, implementations. Audio can be read also from a wifi
stream or a file, doesn't matter here]
Test: playAudioTest

playAudioTest
Input: user choses a file and playback starts.
Precondition: rockbox is running, no other audio are playing, volume
is set to a reasonable level...
Output: the audio output to the user [doesn't matter headphones or
other devices, we need to state what the user shall be able to do /
receive]
(Continue reading)

Permalink | Reply |
headers
Frank Gevaerts | 29 May 2012 20:42
Picon

Re: Pre-release testing framework

On Tue, May 29, 2012 at 06:58:53PM +0200, Bertrik Sikken wrote:
> With respect to test strategy: In my opinion, we can start with
> a set of rather basic tests to verify high-level behaviour, rather
> than to go for test completeness. A test suite taking about (say)
> one hour should keep the barrier low for people who want to
> participate in testing.

I agree in general, but I think one hour for the basic set of tests is
too much.

I stronly believe we need to have a *very* resstricted set of basic
tests that only tests basic playback, including things like playing a
file from the file browser or database, pause, skip, volume change, and
not much more. After that, we can (should) have more advanced tests,
again divided into small sets, that ideally cover all functionality.

The reasin I think this is better than larger chunks (at least for the
basics) is that I think we mainly need *wide* testing. We apparently
have targets (and I'd guess these days this goes for at least half our
stable targets) that see no testing at all during a release cycle, and
on some of those targets we then only get two or three comments after
the release about major issues (such as sound not working at all). We
*need* someone to test these targets, and if there are only (say) five
people who might be tempted to do that, fifteen minutes (for starters, I
hope they'll continue after the basic bit) is easier to sell than an
hour.

Frank

> Kind regards,
(Continue reading)

Permalink | Reply |
headers
Lorenzo Miori | 29 May 2012 20:48
Picon

Re: Pre-release testing framework

Yes. Atomic tests. Not just saying playback, you're right, it's too wide.
And yes, plugin or other high level functionalities have to be
debugged later: it's a music player after all :D

Lorenzo

2012/5/29 Frank Gevaerts <frank <at> gevaerts.be>:
> On Tue, May 29, 2012 at 06:58:53PM +0200, Bertrik Sikken wrote:
>> With respect to test strategy: In my opinion, we can start with
>> a set of rather basic tests to verify high-level behaviour, rather
>> than to go for test completeness. A test suite taking about (say)
>> one hour should keep the barrier low for people who want to
>> participate in testing.
>
> I agree in general, but I think one hour for the basic set of tests is
> too much.
>
> I stronly believe we need to have a *very* resstricted set of basic
> tests that only tests basic playback, including things like playing a
> file from the file browser or database, pause, skip, volume change, and
> not much more. After that, we can (should) have more advanced tests,
> again divided into small sets, that ideally cover all functionality.
>
> The reasin I think this is better than larger chunks (at least for the
> basics) is that I think we mainly need *wide* testing. We apparently
> have targets (and I'd guess these days this goes for at least half our
> stable targets) that see no testing at all during a release cycle, and
> on some of those targets we then only get two or three comments after
> the release about major issues (such as sound not working at all). We
> *need* someone to test these targets, and if there are only (say) five
(Continue reading)

Permalink | Reply |
headers
Frank Gevaerts | 29 May 2012 21:40
Picon

Re: Pre-release testing framework

On Tue, May 29, 2012 at 08:48:42PM +0200, Lorenzo Miori wrote:
> Yes. Atomic tests. Not just saying playback, you're right, it's too wide.

Yes, but that wasn't really my point :) What I mean is that we should
have a *small* basic test suite (which, yes, consists of clear
unambiguous and simple tests) that users can go through quickly in a few
minutes (ok, that's probably overoptimistic). As soon as they're through
that, we have usable results (not many, but today we have nothing) and
the user may well go on to the next (also reasonably short) set of less
basic tests (e.g. "DSP effects", or "bookmarking", or "the demo
plugins"). 

Keeping the sets short will avoid people getting discouraged before they
even get started, both on actual testing and on adding new tests, while
I don't see any disadvantages.

Of course some tests will take longer ("battery life not worse than in
previous release", "Zork is fully playable on the frotz plugin", ...)
think that's a reason not to try to keep the rest short.

Frank

--

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan
Permalink | Reply |
headers
Nils Wallménius | 29 May 2012 23:12
Picon

Re: Pre-release testing framework

On Tue, May 29, 2012 at 9:40 PM, Frank Gevaerts <frank <at> gevaerts.be> wrote:
> On Tue, May 29, 2012 at 08:48:42PM +0200, Lorenzo Miori wrote:
>> Yes. Atomic tests. Not just saying playback, you're right, it's too wide.
>
> Yes, but that wasn't really my point :) What I mean is that we should
> have a *small* basic test suite (which, yes, consists of clear
> unambiguous and simple tests) that users can go through quickly in a few
> minutes (ok, that's probably overoptimistic). As soon as they're through
> that, we have usable results (not many, but today we have nothing) and
> the user may well go on to the next (also reasonably short) set of less
> basic tests (e.g. "DSP effects", or "bookmarking", or "the demo
> plugins").
>
> Keeping the sets short will avoid people getting discouraged before they
> even get started, both on actual testing and on adding new tests, while
> I don't see any disadvantages.
>
> Of course some tests will take longer ("battery life not worse than in
> previous release", "Zork is fully playable on the frotz plugin", ...)
> think that's a reason not to try to keep the rest short.
>
> Frank
>
>
> --
> "Debugging is twice as hard as writing the code in the first place.
> Therefore, if you write the code as cleverly as possible, you are,
> by definition, not smart enough to debug it." - Brian W. Kernighan

I would suggest focusing on things that are more likely to be broken
(Continue reading)

Permalink | Reply |

Gmane