Roman Cheplyaka | 4 Oct 23:55 2012

ANNOUNCE: test-framework-golden-1.1

I am glad to announce the first public release of test-framework-golden — a
golden testing library.

Hackage: http://hackage.haskell.org/package/test-framework-golden
GitHub:  https://github.com/feuerbach/test-framework-golden

Golden tests are similar to unit tests (as implemented in HUnit), but the idea
is to store the expected result in files (called «golden» files).

I was introduced to the idea of golden testing by Bohdan Vlasyuk at ZuriHac in
2010. Since then I've discovered that this is a nice approach and it is already
used in variety of projects (e.g. ghc, haddock).

Surprisingly, not much is written about golden testing, and I've been unable to
find any golden testing libraries for Haskell or any other programming language.
Every project has its own ad-hoc implementation in Haskell/Python/Shell/etc.

The closest match on the market is Simon Michael's shelltestrunner. But to use
it you have to expose the tested functionality via command line, which may be
inconvenient.

So this is my attempt at a general golden testing library.

It consists of two modules.
Test.Golden has a simple interface that helps you get started very quickly. 
Test.Golden.Advanced provides a very generic testing function that you can use
to implement the testing system you dream about.

The library is integrated with test-framework, so you can use golden tests in
addition to SmallCheck/QuickCheck/HUnit tests.
(Continue reading)

Alfredo Di Napoli | 5 Oct 06:43 2012
Picon

Re: ANNOUNCE: test-framework-golden-1.1

That's cool, thank you.
A.

Sent from my iPad

On 04/ott/2012, at 23:55, Roman Cheplyaka <roma <at> ro-che.info> wrote:

> I am glad to announce the first public release of test-framework-golden — a
> golden testing library.
> 
> Hackage: http://hackage.haskell.org/package/test-framework-golden
> GitHub:  https://github.com/feuerbach/test-framework-golden
> 
> Golden tests are similar to unit tests (as implemented in HUnit), but the idea
> is to store the expected result in files (called «golden» files).
> 
> I was introduced to the idea of golden testing by Bohdan Vlasyuk at ZuriHac in
> 2010. Since then I've discovered that this is a nice approach and it is already
> used in variety of projects (e.g. ghc, haddock).
> 
> Surprisingly, not much is written about golden testing, and I've been unable to
> find any golden testing libraries for Haskell or any other programming language.
> Every project has its own ad-hoc implementation in Haskell/Python/Shell/etc.
> 
> The closest match on the market is Simon Michael's shelltestrunner. But to use
> it you have to expose the tested functionality via command line, which may be
> inconvenient.
> 
> So this is my attempt at a general golden testing library.
> 
(Continue reading)

Gianfranco Alongi | 5 Oct 06:57 2012
Picon

Re: ANNOUNCE: test-framework-golden-1.1

I'm a big fan of TDD and tend to approach all languages by learning
how to TDD in them.

As such, you mention that this is similar, could you please send some
links about this?

Cheers!

On Thu, Oct 4, 2012 at 11:55 PM, Roman Cheplyaka <roma <at> ro-che.info> wrote:
> I am glad to announce the first public release of test-framework-golden — a
> golden testing library.
>
> Hackage: http://hackage.haskell.org/package/test-framework-golden
> GitHub:  https://github.com/feuerbach/test-framework-golden
>
> Golden tests are similar to unit tests (as implemented in HUnit), but the idea
> is to store the expected result in files (called «golden» files).
>
> I was introduced to the idea of golden testing by Bohdan Vlasyuk at ZuriHac in
> 2010. Since then I've discovered that this is a nice approach and it is already
> used in variety of projects (e.g. ghc, haddock).
>
> Surprisingly, not much is written about golden testing, and I've been unable to
> find any golden testing libraries for Haskell or any other programming language.
> Every project has its own ad-hoc implementation in Haskell/Python/Shell/etc.
>
> The closest match on the market is Simon Michael's shelltestrunner. But to use
> it you have to expose the tested functionality via command line, which may be
> inconvenient.
>
(Continue reading)

Simon Hengel | 5 Oct 09:07 2012
Picon

Re: ANNOUNCE: test-framework-golden-1.1

Hi,

> I am glad to announce the first public release of
> test-framework-golden — a golden testing library.

Nice!

> The library is integrated with test-framework, so you can use golden
> tests in addition to SmallCheck/QuickCheck/HUnit tests.

I would suggest to rename the modules to

    Test.Framework.Golden

and

    Test.Framework.Golden.Advanced

to more clearly represent that fact.

Cheers,
Simon

_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe <at> haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Roman Cheplyaka | 5 Oct 09:29 2012

Re: ANNOUNCE: test-framework-golden-1.1

* Simon Hengel <sol <at> typeful.net> [2012-10-05 09:07:19+0200]
> Hi,
> 
> > I am glad to announce the first public release of
> > test-framework-golden — a golden testing library.
> 
> Nice!
> 
> > The library is integrated with test-framework, so you can use golden
> > tests in addition to SmallCheck/QuickCheck/HUnit tests.
> 
> I would suggest to rename the modules to
> 
>     Test.Framework.Golden
> 
> and
> 
>     Test.Framework.Golden.Advanced
> 
> to more clearly represent that fact.

According to the test-framework convention, they ought to be named

  Test.Framework.Providers.Golden

and

  Test.Framework.Providers.Golden.Advanced

respectively, and you can see that it was the case in the early
(Continue reading)

Simon Hengel | 5 Oct 10:05 2012
Picon

Re: ANNOUNCE: test-framework-golden-1.1

> My justification (which you may or may not buy) is that, unlike, say,
> Test.Framework.Providers.HUnit, this is not an adaptation of an existing
> testing library to test-framework, but is a new library that just
> happens to use test-framework. So it's more like Test.HUnit, although it
> already subsumes what would become Test.Framework.Providers.Golden.

Unless HUnit, it's unlikely that this will be integrated into other
testing frameworks.  So I think it is less general.  Anyway, it's up to
you.

Cheers,
Simon
Janek S. | 5 Oct 10:51 2012
Picon

Re: ANNOUNCE: test-framework-golden-1.1

Talking about good timing - I was just finishing my post on code testing in Haskell when your 
announcement came up, so your library made it as a last minute news :) I never used golden 
approach to testing but it is good to know that it exists. 

*I* think that it might be a good idea to separete your library into two parts: the golden 
functionality itself and test-framework provider. Right now it seems impossible to use your 
library outside of test-framework. For me that's not a big deal, cause I use test-framework 
anyway, but I suspect you could get more users that way. 

> According to the test-framework convention, they ought to be named
>
>   Test.Framework.Providers.Golden
>
> and
>
>   Test.Framework.Providers.Golden.Advanced
>
> respectively, and you can see that it was the case in the early
> versions:
> http://hackage.haskell.org/package/test-framework-golden-1.0
>
> But it's just too much to type.
Well, I'm already typing Test.Framework.Providers.QuickCheck2, so that's not a big deal really :)

Jan

_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe <at> haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
(Continue reading)

Roman Cheplyaka | 5 Oct 11:36 2012

Re: ANNOUNCE: test-framework-golden-1.1

* Janek S. <fremenzone <at> poczta.onet.pl> [2012-10-05 10:51:35+0200]
> Talking about good timing - I was just finishing my post on code testing in Haskell when your 
> announcement came up, so your library made it as a last minute news :) I never used golden 
> approach to testing but it is good to know that it exists. 

Cool, looking forward to reading it!

I hope you won't forget to cover SmallCheck in your article as well.
Being also the maintainer of SmallCheck, I want it to steal some fame
from QuickCheck :)

> *I* think that it might be a good idea to separete your library into two parts: the golden 
> functionality itself and test-framework provider. Right now it seems impossible to use your 
> library outside of test-framework. For me that's not a big deal, cause I use test-framework 
> anyway, but I suspect you could get more users that way. 

In theory I totally agree, but:

1. It's hard to guess at the moment how a good interface to the "pure
   golden" part should look like.
2. I'd rather avoid doing work when there's no demand.

So when I see someone interested in using it outside of test-framework,
I'm sure we'll figure something out.

Roman

> > According to the test-framework convention, they ought to be named
> >
> >   Test.Framework.Providers.Golden
(Continue reading)

Janek S. | 5 Oct 12:05 2012
Picon

Re: ANNOUNCE: test-framework-golden-1.1

> Cool, looking forward to reading it!
Well, the post is already finished: 
http://ics.p.lodz.pl/~stolarek/blog/2012/10/code-testing-in-haskell/
I was just going to publish it and then your email came up on the list.

> I hope you won't forget to cover SmallCheck in your article as well.
> Being also the maintainer of SmallCheck, I want it to steal some fame
> from QuickCheck :)
Sorry to disappoint you, but I did not mention SmallCheck. So far I'm relying on QuickCheck and 
didn't feel like I need to look for other testing library. I might be wrong of course.

> In theory I totally agree, but:
>
> 1. It's hard to guess at the moment how a good interface to the "pure
>    golden" part should look like.
> 2. I'd rather avoid doing work when there's no demand.
I guess these are valid points.

Jan
Simon Hengel | 5 Oct 13:43 2012
Picon

Re: ANNOUNCE: test-framework-golden-1.1

> 1. It's hard to guess at the moment how a good interface to the "pure
>    golden" part should look like.

Maybe just produce HUnit assertions, e.g.:

    goldenVsFile :: FilePath -> FilePath -> IO () -> Assertion

That way it works with plain HUnit

    main = runTestTT $ TestLabel "someAction produce desired output $
      goldenVsFile "ref.txt" "out.txt" someAction

test-framework

    main = defaultMain [
        testGroup "someAction" [
          testCase "produces some desired output" $
            goldenVsFile "ref.txt" "out.txt" someAction
        ]
      ]

Hspec

    main = hspec $ do
      describe "someAction" $ do
        it "produces some desired output" $ do
          goldenVsFile "ref.txt" "out.txt" someAction

and probably every other current or future Haskell test framework.

(Continue reading)

Roman Cheplyaka | 5 Oct 16:17 2012

Re: ANNOUNCE: test-framework-golden-1.1

I can do that indeed, and I guess I could reimplement everything I have
at the moment on top of HUnit.

However, an important part of functionality isn't there at the moment —
golden file management. You should be able to say, "for this test,
take its current output and write it to the corresponding golden file".

In order to do that, you need to have access to the list of golden tests
in the suite. This is where implementation details of different test
frameworks start to matter. Probably one can make an abstraction over
test frameworks that would give the list of all golden tests.
(Although when you start abstracting over test frameworks, which are
abstractions themselves, it becomes somewhat funny.)

Speaking of such functionality, correct me if I'm wrong, but neither
HUnit nor hspec won't be able to support it anyway, because they
represent tests as opaque IO actions.

Nor can HTF provide such support — although its TestSort type could be
extended to indicate that the test is a golden test, still there's no
way to get hold of the golden file name.

test-framework will support this once the Typeable constraint is added
for tests.

Roman

* Simon Hengel <sol <at> typeful.net> [2012-10-05 13:43:57+0200]
> > 1. It's hard to guess at the moment how a good interface to the "pure
> >    golden" part should look like.
(Continue reading)

Simon Hengel | 7 Oct 15:45 2012
Picon

Re: ANNOUNCE: test-framework-golden-1.1

On Fri, Oct 05, 2012 at 05:17:18PM +0300, Roman Cheplyaka wrote:
> I can do that indeed, and I guess I could reimplement everything I have
> at the moment on top of HUnit.
> 
> However, an important part of functionality isn't there at the moment —
> golden file management. You should be able to say, "for this test,
> take its current output and write it to the corresponding golden file".
> 
> In order to do that, you need to have access to the list of golden tests
> in the suite. This is where implementation details of different test
> frameworks start to matter. Probably one can make an abstraction over
> test frameworks that would give the list of all golden tests.
> (Although when you start abstracting over test frameworks, which are
> abstractions themselves, it becomes somewhat funny.)

Ok, makes sense.

I'm looking forward to give it a try, and see how it compares to using
operating system primitives (say `cp') for "golden file management".

> Speaking of such functionality, correct me if I'm wrong, but neither
> HUnit nor hspec won't be able to support it anyway, because they
> represent tests as opaque IO actions.

It would be easy to extend Hspec to support this in the same way you
extend test-framework to support this.  It requires existentials;  the
only substantial difference that I can see is that test-framework
already uses existentials, while Hspec does not.

Cheers,
(Continue reading)

Roman Cheplyaka | 7 Oct 20:09 2012

Re: ANNOUNCE: test-framework-golden-1.1

* Simon Hengel <sol <at> typeful.net> [2012-10-07 15:45:21+0200]
> On Fri, Oct 05, 2012 at 05:17:18PM +0300, Roman Cheplyaka wrote:
> > I can do that indeed, and I guess I could reimplement everything I have
> > at the moment on top of HUnit.
> > 
> > However, an important part of functionality isn't there at the moment —
> > golden file management. You should be able to say, "for this test,
> > take its current output and write it to the corresponding golden file".
> > 
> > In order to do that, you need to have access to the list of golden tests
> > in the suite. This is where implementation details of different test
> > frameworks start to matter. Probably one can make an abstraction over
> > test frameworks that would give the list of all golden tests.
> > (Although when you start abstracting over test frameworks, which are
> > abstractions themselves, it becomes somewhat funny.)
> 
> Ok, makes sense.
> 
> I'm looking forward to give it a try, and see how it compares to using
> operating system primitives (say `cp') for "golden file management".

1. You often want to update not just one test, but all, or some of the
   tests (when you've made a change and verified that the changes in
   output are expected). Doing it in command line is certainly possible,
   but not trivial nor convenient.
2. For some tests (like goldenVsString) the output is not captured in a
   file, so using cp directly is not possible.

> > Speaking of such functionality, correct me if I'm wrong, but neither
> > HUnit nor hspec won't be able to support it anyway, because they
(Continue reading)

Simon Hengel | 8 Oct 10:30 2012
Picon

Re: ANNOUNCE: test-framework-golden-1.1

On Sun, Oct 07, 2012 at 09:09:07PM +0300, Roman Cheplyaka wrote:
> * Simon Hengel <sol <at> typeful.net> [2012-10-07 15:45:21+0200]
> > On Fri, Oct 05, 2012 at 05:17:18PM +0300, Roman Cheplyaka wrote:
> > > I can do that indeed, and I guess I could reimplement everything I have
> > > at the moment on top of HUnit.
> > >
> > > However, an important part of functionality isn't there at the moment —
> > > golden file management. You should be able to say, "for this test,
> > > take its current output and write it to the corresponding golden file".
> > >
> > > In order to do that, you need to have access to the list of golden tests
> > > in the suite. This is where implementation details of different test
> > > frameworks start to matter. Probably one can make an abstraction over
> > > test frameworks that would give the list of all golden tests.
> > > (Although when you start abstracting over test frameworks, which are
> > > abstractions themselves, it becomes somewhat funny.)
> >
> > Ok, makes sense.
> >
> > I'm looking forward to give it a try, and see how it compares to using
> > operating system primitives (say `cp') for "golden file management".
>
> 1. You often want to update not just one test, but all, or some of the
>    tests (when you've made a change and verified that the changes in
>    output are expected). Doing it in command line is certainly possible,
>    but not trivial nor convenient.
> 2. For some tests (like goldenVsString) the output is not captured in a
>    file, so using cp directly is not possible.

Yes and yes.  I have no fixed expectations nor any idea how an "ideal"
(Continue reading)


Gmane