headers
Till Berger | 7 Aug 2012 16:16
Picon
Picon

Possible bug in Criterion or Statistics package

Dear all,

I may have stumbled upon a bug in the Criterion package. When running  
the attached Haskell program (Benchmark.hs, a simple test case) on  
multiple cores (with +RTS -N, +RTS -N2, +RTS -N3 etc.) it sooner or  
later crashes with the following exception:

Benchmark: thread blocked indefinitely in an MVar operation

With profiling support enabled and run with the xc flag I get the  
following output before the crash:

*** Exception (reporting due to +RTS -xc): (THUNK_STATIC), stack trace:
   Statistics.Resampling.Bootstrap.bootstrapBCA,
   called from Main.main,
   called from Main.CAF
   --> evaluated by: Main.main,
   called from Main.CAF
*** Exception (reporting due to +RTS -xc): (THUNK_STATIC), stack trace:
   Statistics.Resampling.Bootstrap.bootstrapBCA,
   called from Main.main
*** Exception (reporting due to +RTS -xc): (THUNK_STATIC), stack trace:
   Statistics.Resampling.Bootstrap.bootstrapBCA,
   called from Main.main
*** Exception (reporting due to +RTS -xc): (THUNK_STATIC), stack trace:
   Statistics.Resampling.Bootstrap.bootstrapBCA,
   called from Main.main,
   called from Main.CAF

I have tested this with GHC versions 7.0.4 and 7.4.2 and Criterion 0.6.0.1.
(Continue reading)

Permalink | Reply |
headers
Johan Tibell | 7 Aug 2012 17:00
Picon
Gravatar

Re: Possible bug in Criterion or Statistics package

Hi Till,

This would make an excellent bug report at:

  https://github.com/bos/criterion/issues

Cheers,
Johan
Permalink | Reply |
headers
Aleksey Khudyakov | 7 Aug 2012 17:15
Picon
Gravatar

Re: Possible bug in Criterion or Statistics package

On 07.08.2012 18:16, Till Berger wrote:
> Dear all,
>
> I may have stumbled upon a bug in the Criterion package. When running
> the attached Haskell program (Benchmark.hs, a simple test case) on
> multiple cores (with +RTS -N, +RTS -N2, +RTS -N3 etc.) it sooner or
> later crashes with the following exception:
>
> Benchmark: thread blocked indefinitely in an MVar operation
>
> With profiling support enabled and run with the xc flag I get the
> following output before the crash:
>
> *** Exception (reporting due to +RTS -xc): (THUNK_STATIC), stack trace:
> Statistics.Resampling.Bootstrap.bootstrapBCA,
> called from Main.main,
> called from Main.CAF
> --> evaluated by: Main.main,
> called from Main.CAF
> *** Exception (reporting due to +RTS -xc): (THUNK_STATIC), stack trace:
> Statistics.Resampling.Bootstrap.bootstrapBCA,
> called from Main.main
> *** Exception (reporting due to +RTS -xc): (THUNK_STATIC), stack trace:
> Statistics.Resampling.Bootstrap.bootstrapBCA,
> called from Main.main
> *** Exception (reporting due to +RTS -xc): (THUNK_STATIC), stack trace:
> Statistics.Resampling.Bootstrap.bootstrapBCA,
> called from Main.main,
> called from Main.CAF
>
(Continue reading)

Permalink | Reply |
headers
Aleksey Khudyakov | 9 Aug 2012 17:37
Picon
Gravatar

Re: Possible bug in Criterion or Statistics package

On 07.08.2012 19:15, Aleksey Khudyakov wrote:
> On 07.08.2012 18:16, Till Berger wrote:
>> Dear all,
>>
>> So I am not sure if this is a bug in Criterion itself, the Statistics
>> package or any dependency or if I am doing something obviously wrong. I
>> would be grateful if someone could look into this as it is holding me
>> back from using Criterion for benchmarking my code.
>>
> I would suspect Statistics.Resampling.resample. From quick glance
> criterion doesn't use any concurrent stuff. I'll try create smaller test
> case

It looks like I'm wrong. I obtained event log from crashing program and 
resample completed its work without problems. Crash occured later. Next
suspect is bootstrapBCA itself. It uses monad-par to obtain parallelism[1].

I tried to create smaller test case without any success.

[1] 
https://github.com/bos/statistics/blob/master/Statistics/Resampling/Bootstrap.hs#L84
Permalink | Reply |
headers
Till Berger | 10 Aug 2012 20:20
Picon
Picon

Re: Possible bug in Criterion or Statistics package

>>> So I am not sure if this is a bug in Criterion itself, the Statistics
>>> package or any dependency or if I am doing something obviously wrong. I
>>> would be grateful if someone could look into this as it is holding me
>>> back from using Criterion for benchmarking my code.
>>>
>> I would suspect Statistics.Resampling.resample. From quick glance
>> criterion doesn't use any concurrent stuff. I'll try create smaller test
>> case
>
> It looks like I'm wrong. I obtained event log from crashing program  
> and resample completed its work without problems. Crash occured  
> later. Next
> suspect is bootstrapBCA itself. It uses monad-par to obtain parallelism[1].
>
> I tried to create smaller test case without any success.
>
>
>
> [1]  
> https://github.com/bos/statistics/blob/master/Statistics/Resampling/Bootstrap.hs#L84

Replacing "runPar $ parMap" with a simple "map" on that line seems to  
fix the bug. At least I could not reproduce it anymore on several runs  
with my original test case. So it seems to be a bug in the Par monad  
package as this change shouldn't alter the program's behaviour, should  
it?

Regards,
Till
(Continue reading)

Permalink | Reply |
headers
Aleksey Khudyakov | 12 Aug 2012 17:20
Picon
Gravatar

Re: Possible bug in Criterion or Statistics package

On 10.08.2012 22:20, Till Berger wrote:
>>>> So I am not sure if this is a bug in Criterion itself, the Statistics
>>>> package or any dependency or if I am doing something obviously wrong. I
>>>> would be grateful if someone could look into this as it is holding me
>>>> back from using Criterion for benchmarking my code.
>>>>
>>> I would suspect Statistics.Resampling.resample. From quick glance
>>> criterion doesn't use any concurrent stuff. I'll try create smaller test
>>> case
>>
>> It looks like I'm wrong. I obtained event log from crashing program
>> and resample completed its work without problems. Crash occured later.
>> Next
>> suspect is bootstrapBCA itself. It uses monad-par to obtain
>> parallelism[1].
>>
>> I tried to create smaller test case without any success.
>>
>>
>>
>> [1]
>> https://github.com/bos/statistics/blob/master/Statistics/Resampling/Bootstrap.hs#L84
>>
>
> Replacing "runPar $ parMap" with a simple "map" on that line seems to
> fix the bug. At least I could not reproduce it anymore on several runs
> with my original test case. So it seems to be a bug in the Par monad
> package as this change shouldn't alter the program's behaviour, should it?
>
Looks like this is the case. But reducing test case to reasonable size
(Continue reading)

Permalink | Reply |
headers
Ryan Newton | 13 Aug 2012 17:43
Picon
Gravatar

Re: Possible bug in Criterion or Statistics package

Terrible!  Quite sorry that this seems to be a bug in the monad-par library.

I'm copying some of the other monad-par authors and we hopefully can get to the bottom of this.  If it's not possible to create a smaller reproducer, is it possible to share the original test that triggers this problem?  In the meantime, it's good that you can at least run without parallelism.

Best,
  -Ryan

On Sun, Aug 12, 2012 at 11:20 AM, Aleksey Khudyakov <alexey.skladnoy <at> gmail.com> wrote:
On 10.08.2012 22:20, Till Berger wrote:
So I am not sure if this is a bug in Criterion itself, the Statistics
package or any dependency or if I am doing something obviously wrong. I
would be grateful if someone could look into this as it is holding me
back from using Criterion for benchmarking my code.

I would suspect Statistics.Resampling.resample. From quick glance
criterion doesn't use any concurrent stuff. I'll try create smaller test
case

It looks like I'm wrong. I obtained event log from crashing program
and resample completed its work without problems. Crash occured later.
Next
suspect is bootstrapBCA itself. It uses monad-par to obtain
parallelism[1].

I tried to create smaller test case without any success.



[1]
https://github.com/bos/statistics/blob/master/Statistics/Resampling/Bootstrap.hs#L84


Replacing "runPar $ parMap" with a simple "map" on that line seems to
fix the bug. At least I could not reproduce it anymore on several runs
with my original test case. So it seems to be a bug in the Par monad
package as this change shouldn't alter the program's behaviour, should it?

Looks like this is the case. But reducing test case to reasonable size (e.g. removing most of criterion and statistics could be quite difficult


_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe <at> haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe

_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe <at> haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Permalink | Reply |
headers
Till Berger | 13 Aug 2012 18:26
Picon
Picon

Re: Possible bug in Criterion or Statistics package

> Terrible!  Quite sorry that this seems to be a bug in the monad-par library.
>
> I'm copying some of the other monad-par authors and we hopefully can get to
> the bottom of this.  If it's not possible to create a smaller reproducer,
> is it possible to share the original test that triggers this problem?  In
> the meantime, it's good that you can at least run without parallelism.

I have attached an even simpler test that directly uses the monad-par  
library. The function "test" simply adds one to a list of numbers  
indefinitely using "parMap" and displays every intermediate result.  
When running the program on multiple cores the bug occurs every time  
for me.

Thanks for looking into this!

Regards,
Till
Attachment (MonadParBug.hs): text/x-haskell, 176 bytes
_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe <at> haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Permalink | Reply |
headers
Aleksey Khudyakov | 13 Aug 2012 18:32
Picon
Gravatar

Re: Possible bug in Criterion or Statistics package

On 13.08.2012 20:26, Till Berger wrote:
>> Terrible! Quite sorry that this seems to be a bug in the monad-par
>> library.
>>
>> I'm copying some of the other monad-par authors and we hopefully can
>> get to
>> the bottom of this. If it's not possible to create a smaller reproducer,
>> is it possible to share the original test that triggers this problem? In
>> the meantime, it's good that you can at least run without parallelism.
>
> I have attached an even simpler test that directly uses the monad-par
> library. The function "test" simply adds one to a list of numbers
> indefinitely using "parMap" and displays every intermediate result. When
> running the program on multiple cores the bug occurs every time for me.
>
> Thanks for looking into this!
>
I've tried your test case and it indeed fails every time. Usually it 
fails with "blocked on MVar indefinitely" but sometimes it fails with:

[2]
[3]
test: Impossible state in globalWorkComplete.
Permalink | Reply |
headers
Aleksey Khudyakov | 13 Aug 2012 18:27
Picon
Gravatar

Re: Possible bug in Criterion or Statistics package

On 13.08.2012 19:43, Ryan Newton wrote:
> Terrible!  Quite sorry that this seems to be a bug in the monad-par library.
>
> I'm copying some of the other monad-par authors and we hopefully can get
> to the bottom of this.  If it's not possible to create a smaller
> reproducer, is it possible to share the original test that triggers this
> problem?  In the meantime, it's good that you can at least run without
> parallelism.

Here is slightly simplified original test case. By itself program is 
very small but there is statistics and criterion on top of the monad-par
Failure occurs in the function 
Statistics.Resampling.Bootstrap.bootstrapBCA. However I couldn't trigger 
bug with mock data.

import Criterion.Main

test :: t -> ()
test _ = ()

main :: IO ()
main = defaultMain [ bench (show n) $ nf test () | n <- [0 .. 5000]]

P.S. I assume I've just got test failure report for the statistics from 
your buildbot. Failures reported are spurious. Also linux box cannot 
handle unicode in the output (wrong locale settings?)
Permalink | Reply |
headers
Ben Gamari | 1 Oct 2012 00:14
Picon
Gravatar

Re: Possible bug in Criterion or Statistics package

Aleksey Khudyakov <alexey.skladnoy <at> gmail.com> writes:

> On 13.08.2012 19:43, Ryan Newton wrote:
>> Terrible!  Quite sorry that this seems to be a bug in the monad-par library.
>>
>> I'm copying some of the other monad-par authors and we hopefully can get
>> to the bottom of this.  If it's not possible to create a smaller
>> reproducer, is it possible to share the original test that triggers this
>> problem?  In the meantime, it's good that you can at least run without
>> parallelism.
>
> Here is slightly simplified original test case. By itself program is 
> very small but there is statistics and criterion on top of the monad-par
> Failure occurs in the function 
> Statistics.Resampling.Bootstrap.bootstrapBCA. However I couldn't trigger 
> bug with mock data.
>
Has there been any progress or an official bug report on this?

Cheers,

- Ben
Permalink | Reply |
headers
Aleksey Khudyakov | 2 Oct 2012 11:56
Picon
Gravatar

Re: Possible bug in Criterion or Statistics package

On 01.10.2012 02:14, Ben Gamari wrote:
> Aleksey Khudyakov<alexey.skladnoy <at> gmail.com>  writes:
>
>> On 13.08.2012 19:43, Ryan Newton wrote:
>>> Terrible!  Quite sorry that this seems to be a bug in the monad-par library.
>>>
>>> I'm copying some of the other monad-par authors and we hopefully can get
>>> to the bottom of this.  If it's not possible to create a smaller
>>> reproducer, is it possible to share the original test that triggers this
>>> problem?  In the meantime, it's good that you can at least run without
>>> parallelism.
>>
>> Here is slightly simplified original test case. By itself program is
>> very small but there is statistics and criterion on top of the monad-par
>> Failure occurs in the function
>> Statistics.Resampling.Bootstrap.bootstrapBCA. However I couldn't trigger
>> bug with mock data.
>>
> Has there been any progress or an official bug report on this?
>
It appears it stalled. I filed bug report against monad-par

https://github.com/simonmar/monad-par/issues/23
Permalink | Reply |

Gmane