Tom Crayford | 16 Apr 11:17 2013
Picon

ANN: rematch, an library for composable assertions with human readable failure messages

I kept on running into this thing where I was calling error in quickcheck to get good error messages about the things I was comparing. In Java land, this stuff is handled by Hamcrest: a library for composable assertions with good error messages. This library is basically a port of hamcrest's core api, but I've been very pleased with how it turned out.

I've been using this in tests for production code for a month or so now, and I'm very pleased with it.

Running a matcher (in this example in an hunit test) looks like this:

expect [1] (is [1])

 The core API is very simple:

data Matcher a = Matcher {
    match :: a -> Bool
  -- ^ A function that returns True if the matcher should pass, False if it should fail
  , description :: String
  -- ^ A description of the matcher (usually of its success conditions)
  , describeMismatch :: a -> String
  -- ^ A description to be shown if the match fails.
  }

This means you can add/write your own matchers happily, which occasionally means you can write *very* nice test code (here's an example of using a custom matcher for checking the state of an "issue" in a hypothetical issue tracking app):

expect latestIssue (hasState Resolved)

-- I removed the supporting code to make this assertion actually run,
-- this email is already pretty long.

There are numerous matchers (and functions for creating matchers) in the rematch library, including some composition functions that provide good failure messages.

There are some shims to hook rematch into the common haskell test frameworks (specifically hunit and quickcheck).

The two libraries are up on hackage:

The code is all up on github:


I get rather frustrated when my tests give bad failure explanations, and using rematch goes a long way to fix that.

Lastly, rematch is pretty isolated from test frameworks/etc, with a very small and easy to understand surface api. Hopefully it'll help with the thing I've seen in other languages (cough ruby cough) with every test framework reinventing this idea, and not all frameworks having all the matchers I want to use.

Let me know if you have any feedback/thoughts

Tom
_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe <at> haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Tom Ellis | 16 Apr 12:24 2013
Picon

Re: ANN: rematch, an library for composable assertions with human readable failure messages

On Tue, Apr 16, 2013 at 10:17:48AM +0100, Tom Crayford wrote:
> I kept on running into this thing where I was calling error in quickcheck
> to get good error messages about the things I was comparing. In Java land,
> this stuff is handled by Hamcrest: a library for composable assertions with
> good error messages. This library is basically a port of hamcrest's core
> api, but I've been very pleased with how it turned out.
[...]
> Let me know if you have any feedback/thoughts

I've been feeling the need for something like this for some time.  I'll
check it out.  Thanks!

Tom
Oliver Charles | 16 Apr 12:31 2013
Picon

Re: ANN: rematch, an library for composable assertions with human readable failure messages

On 04/16/2013 11:24 AM, Tom Ellis wrote:
> On Tue, Apr 16, 2013 at 10:17:48AM +0100, Tom Crayford wrote:
>> I kept on running into this thing where I was calling error in quickcheck
>> to get good error messages about the things I was comparing. In Java land,
>> this stuff is handled by Hamcrest: a library for composable assertions with
>> good error messages. This library is basically a port of hamcrest's core
>> api, but I've been very pleased with how it turned out.
> [...]
>> Let me know if you have any feedback/thoughts
> I've been feeling the need for something like this for some time.  I'll
> check it out.  Thanks!
>
> Tom
This is beyond sorely needed. I can't wait to plug this into some of my 
own projects!

Thanks!
- Ollie
Roman Cheplyaka | 16 Apr 16:09 2013

Re: ANN: rematch, an library for composable assertions with human readable failure messages

Hi Tom,

This is a neat idea! I'd like to use something like this in smallcheck
and test-framework-golden.

The main obstacle to that is that your package depends on QuickCheck and
HUnit, and every package using rematch would transitively depend on
them, too. This has little sense, especially for smallcheck which is in
some sense a replacement for QuickCheck.

The alternative is either to put HUnit and QuickCheck interfaces in
the separate packages, or try to get them accepted into the HUnit and
QuickCheck directly.

Below are some more suggestions regarding the package:

1. You need to escape single and double quotes in the haddock
   documentation; otherwise they'll be turned into (bad) links.

2. Your 'join' function is a special case of 'intercalate' from
   Data.List.

3. The "Control" namespace doesn't quite match the purpose of your
   modules, since they are not about control flow. Perhaps "Test"?

I also wonder whether there is a more principled approach to such an API — 
say, based on applicative functors.

Roman

* Tom Crayford <tcrayford <at> gmail.com> [2013-04-16 10:17:48+0100]
> I kept on running into this thing where I was calling error in quickcheck
> to get good error messages about the things I was comparing. In Java land,
> this stuff is handled by Hamcrest: a library for composable assertions with
> good error messages. This library is basically a port of hamcrest's core
> api, but I've been very pleased with how it turned out.
> 
> I've been using this in tests for production code for a month or so now,
> and I'm very pleased with it.
> 
> Running a matcher (in this example in an hunit test) looks like this:
> 
> expect [1] (is [1])
> 
>  The core API is very simple:
> 
> data Matcher a = Matcher {
>     match :: a -> Bool
>   -- ^ A function that returns True if the matcher should pass, False if it
> should fail
>   , description :: String
>   -- ^ A description of the matcher (usually of its success conditions)
>   , describeMismatch :: a -> String
>   -- ^ A description to be shown if the match fails.
>   }
> 
> This means you can add/write your own matchers happily, which occasionally
> means you can write *very* nice test code (here's an example of using a
> custom matcher for checking the state of an "issue" in a hypothetical issue
> tracking app):
> 
> expect latestIssue (hasState Resolved)
> 
> -- I removed the supporting code to make this assertion actually run,
> -- this email is already pretty long.
> 
> There are numerous matchers (and functions for creating matchers) in the
> rematch library, including some composition functions that provide good
> failure messages.
> 
> There are some shims to hook rematch into the common haskell test
> frameworks (specifically hunit and quickcheck).
> 
> The two libraries are up on hackage:
> http://hackage.haskell.org/package/rematch
> http://hackage.haskell.org/package/rematch-text
> 
> The code is all up on github:
> 
> http://github.com/tcrayford/rematch
> 
> I get rather frustrated when my tests give bad failure explanations, and
> using rematch goes a long way to fix that.
> 
> Lastly, rematch is pretty isolated from test frameworks/etc, with a very
> small and easy to understand surface api. Hopefully it'll help with the
> thing I've seen in other languages (cough ruby cough) with every test
> framework reinventing this idea, and not all frameworks having all the
> matchers I want to use.
> 
> Let me know if you have any feedback/thoughts
> 
> Tom

> _______________________________________________
> Haskell-Cafe mailing list
> Haskell-Cafe <at> haskell.org
> http://www.haskell.org/mailman/listinfo/haskell-cafe

_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe <at> haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Tom Crayford | 16 Apr 16:24 2013
Picon

Re: ANN: rematch, an library for composable assertions with human readable failure messages

Roman,

Thanks for the feedback! I'd originally left the QuickCheck and HUnit implementations in this library for convenience, thinking that there aren't going to be many people who care about the transitive dep. But you care, so I'm happy moving them out of core. I'll release a 0.2 with both the HUnit and the QuickCheck runners in separate libraries soonish.

Thanks for the haddock tip and the implementation tips.

Re the Control namespace, these matchers aren't exclusively a testing tool. I've been using the core api for other purposes as well (primarily for validating forms in user interfaces in conjunction with digestive-functors). I couldn't figure anything better to put it in apart from Control (I definitely don't want it in Test, even though that's going to be what most people use it for). I guess it could be in `Data`, but that doesn't sound much better to me.

I'm not amazingly strong at building more principled interfaces right now, so I guess that's something I'll improve on. Are there any concrete suggestions you have there? I'd *like* these to have an `Alternative` instance, but making `Applicative`/`Functor` instances is beyond me right now (I guess I'd have to change the core API for that to work out).

Tom


On 16 April 2013 15:09, Roman Cheplyaka <roma <at> ro-che.info> wrote:
Hi Tom,

This is a neat idea! I'd like to use something like this in smallcheck
and test-framework-golden.

The main obstacle to that is that your package depends on QuickCheck and
HUnit, and every package using rematch would transitively depend on
them, too. This has little sense, especially for smallcheck which is in
some sense a replacement for QuickCheck.

The alternative is either to put HUnit and QuickCheck interfaces in
the separate packages, or try to get them accepted into the HUnit and
QuickCheck directly.

Below are some more suggestions regarding the package:

1. You need to escape single and double quotes in the haddock
   documentation; otherwise they'll be turned into (bad) links.

2. Your 'join' function is a special case of 'intercalate' from
   Data.List.

3. The "Control" namespace doesn't quite match the purpose of your
   modules, since they are not about control flow. Perhaps "Test"?

I also wonder whether there is a more principled approach to such an API —
say, based on applicative functors.

Roman

* Tom Crayford <tcrayford <at> gmail.com> [2013-04-16 10:17:48+0100]
> I kept on running into this thing where I was calling error in quickcheck
> to get good error messages about the things I was comparing. In Java land,
> this stuff is handled by Hamcrest: a library for composable assertions with
> good error messages. This library is basically a port of hamcrest's core
> api, but I've been very pleased with how it turned out.
>
> I've been using this in tests for production code for a month or so now,
> and I'm very pleased with it.
>
> Running a matcher (in this example in an hunit test) looks like this:
>
> expect [1] (is [1])
>
>  The core API is very simple:
>
> data Matcher a = Matcher {
>     match :: a -> Bool
>   -- ^ A function that returns True if the matcher should pass, False if it
> should fail
>   , description :: String
>   -- ^ A description of the matcher (usually of its success conditions)
>   , describeMismatch :: a -> String
>   -- ^ A description to be shown if the match fails.
>   }
>
> This means you can add/write your own matchers happily, which occasionally
> means you can write *very* nice test code (here's an example of using a
> custom matcher for checking the state of an "issue" in a hypothetical issue
> tracking app):
>
> expect latestIssue (hasState Resolved)
>
> -- I removed the supporting code to make this assertion actually run,
> -- this email is already pretty long.
>
> There are numerous matchers (and functions for creating matchers) in the
> rematch library, including some composition functions that provide good
> failure messages.
>
> There are some shims to hook rematch into the common haskell test
> frameworks (specifically hunit and quickcheck).
>
> The two libraries are up on hackage:
> http://hackage.haskell.org/package/rematch
> http://hackage.haskell.org/package/rematch-text
>
> The code is all up on github:
>
> http://github.com/tcrayford/rematch
>
> I get rather frustrated when my tests give bad failure explanations, and
> using rematch goes a long way to fix that.
>
> Lastly, rematch is pretty isolated from test frameworks/etc, with a very
> small and easy to understand surface api. Hopefully it'll help with the
> thing I've seen in other languages (cough ruby cough) with every test
> framework reinventing this idea, and not all frameworks having all the
> matchers I want to use.
>
> Let me know if you have any feedback/thoughts
>
> Tom

> _______________________________________________
> Haskell-Cafe mailing list
> Haskell-Cafe <at> haskell.org
> http://www.haskell.org/mailman/listinfo/haskell-cafe


_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe <at> haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Roman Cheplyaka | 16 Apr 16:51 2013

Re: ANN: rematch, an library for composable assertions with human readable failure messages

* Tom Crayford <tcrayford <at> gmail.com> [2013-04-16 15:24:07+0100]
> Re the Control namespace, these matchers aren't exclusively a testing tool.
> I've been using the core api for other purposes as well (primarily for
> validating forms in user interfaces in conjunction with
> digestive-functors). I couldn't figure anything better to put it in apart
> from Control (I definitely don't want it in Test, even though that's going
> to be what most people use it for). I guess it could be in `Data`, but that
> doesn't sound much better to me.

Yeah, the distinction between Control and Data is often borderline.
Also, I've just recalled that Control.Unification also uses the Control
namespace, although I don't see any rationale behind that.
It's not that important anyway.

> I'm not amazingly strong at building more principled interfaces right now,
> so I guess that's something I'll improve on. Are there any concrete
> suggestions you have there? I'd *like* these to have an `Alternative`
> instance, but making `Applicative`/`Functor` instances is beyond me right
> now (I guess I'd have to change the core API for that to work out).

I'll see if I can sketch something in the next couple of days.

Roman
Petr Pudlák | 16 Apr 18:37 2013
Picon

Re: ANN: rematch, an library for composable assertions with human readable failure messages

Hi tom,

I had problems installing version 0.1.2.1 on GHC  7.4.1:

Resolving dependencies...
Downloading rematch-0.1.2.1...
Configuring rematch-0.1.2.1...
Building rematch-0.1.2.1...
Preprocessing library rematch-0.1.2.1...
[1 of 4] Compiling Control.Rematch.Formatting ( Control/Rematch/Formatting.hs, dist/build/Control/Rematch/Formatting.o )
[2 of 4] Compiling Control.Rematch.Run ( Control/Rematch/Run.hs, dist/build/Control/Rematch/Run.o )
[3 of 4] Compiling Control.Rematch  ( Control/Rematch.hs, dist/build/Control/Rematch.o )
[4 of 4] Compiling Control.Rematch.QuickCheck ( Control/Rematch/QuickCheck.hs, dist/build/Control/Rematch/QuickCheck.o )

Control/Rematch/QuickCheck.hs:19:3:
    `exhaustive' is not a (visible) method of class `Testable'
Failed to install rematch-0.1.2.1
cabal: Error: some packages failed to install:
rematch-0.1.2.1 failed during the building phase. The exception was:
ExitFailure 1

I installed v 0.1.2.0 without problems.

  Best regards,
  Petr


2013/4/16 Tom Crayford <tcrayford <at> gmail.com>
I kept on running into this thing where I was calling error in quickcheck to get good error messages about the things I was comparing. In Java land, this stuff is handled by Hamcrest: a library for composable assertions with good error messages. This library is basically a port of hamcrest's core api, but I've been very pleased with how it turned out.

I've been using this in tests for production code for a month or so now, and I'm very pleased with it.

Running a matcher (in this example in an hunit test) looks like this:

expect [1] (is [1])

 The core API is very simple:

data Matcher a = Matcher {
    match :: a -> Bool
  -- ^ A function that returns True if the matcher should pass, False if it should fail
  , description :: String
  -- ^ A description of the matcher (usually of its success conditions)
  , describeMismatch :: a -> String
  -- ^ A description to be shown if the match fails.
  }

This means you can add/write your own matchers happily, which occasionally means you can write *very* nice test code (here's an example of using a custom matcher for checking the state of an "issue" in a hypothetical issue tracking app):

expect latestIssue (hasState Resolved)

-- I removed the supporting code to make this assertion actually run,
-- this email is already pretty long.

There are numerous matchers (and functions for creating matchers) in the rematch library, including some composition functions that provide good failure messages.

There are some shims to hook rematch into the common haskell test frameworks (specifically hunit and quickcheck).

The two libraries are up on hackage:

The code is all up on github:


I get rather frustrated when my tests give bad failure explanations, and using rematch goes a long way to fix that.

Lastly, rematch is pretty isolated from test frameworks/etc, with a very small and easy to understand surface api. Hopefully it'll help with the thing I've seen in other languages (cough ruby cough) with every test framework reinventing this idea, and not all frameworks having all the matchers I want to use.

Let me know if you have any feedback/thoughts

Tom

_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe <at> haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe


_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe <at> haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Ross Paterson | 16 Apr 19:23 2013
Picon

Re: ANN: rematch, an library for composable assertions with human readable failure messages

On Tue, Apr 16, 2013 at 10:17:48AM +0100, Tom Crayford wrote:
>  The core API is very simple:
> 
> data Matcher a = Matcher {
>     match :: a -> Bool
>   -- ^ A function that returns True if the matcher should pass, False if it
> should fail
>   , description :: String
>   -- ^ A description of the matcher (usually of its success conditions)
>   , describeMismatch :: a -> String
>   -- ^ A description to be shown if the match fails.
>   }

How about combining match and describeMismatch as a single function
of type a -> Match?  Then you wouldn't need the precondition on
describeMismatch.

Defining a Monoid instance for Match might also be useful.
Petr Pudlák | 16 Apr 19:46 2013
Picon

Re: ANN: rematch, an library for composable assertions with human readable failure messages

2013/4/16 Ross Paterson <R.Paterson <at> city.ac.uk>
On Tue, Apr 16, 2013 at 10:17:48AM +0100, Tom Crayford wrote:
>  The core API is very simple:
>
> data Matcher a = Matcher {
>     match :: a -> Bool
>   -- ^ A function that returns True if the matcher should pass, False if it
> should fail
>   , description :: String
>   -- ^ A description of the matcher (usually of its success conditions)
>   , describeMismatch :: a -> String
>   -- ^ A description to be shown if the match fails.
>   }

How about combining match and describeMismatch as a single function
of type a -> Match?  Then you wouldn't need the precondition on
describeMismatch.

And this way we'd get `runMatch` right away in the data type:

data Matcher a = Matcher {
    runMatch :: a -> Match
  , description :: String
  }
 
_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe <at> haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe

Gmane