Hiromi ISHII | 1 Nov 05:26 2012
Picon

How to handle exceptions in conduit?

Hi, there

I'm writing a program communicating with external process, which can be sometimes fail, using conduit and
process-conduit package.

Consider the following example, which reads paths from the config file, and passes their contents to
external process, and output the results:

```exc.hs
module Main where
import qualified Data.ByteString.Char8 as BS
import           Data.Conduit
import qualified Data.Conduit.Binary   as BC
import qualified Data.Conduit.List     as LC
import           Data.Conduit.Process

main :: IO ()
main = runResourceT $
  BC.sourceFile "paths.dat" $$ BC.lines =$= myConduit =$= LC.mapM_ (unsafeLiftIO . BS.putStrLn)

myConduit :: MonadResource m => Conduit BS.ByteString m BS.ByteString
myConduit = awaitForever $ \path ->
  BC.sourceFile (BS.unpack path) =$= conduitCmd "./sometimes-fail"
```

```sometimes-fail.hs
module Main where
import System.Random

main :: IO ()
(Continue reading)

Michael Snoyman | 1 Nov 13:23 2012

Re: How to handle exceptions in conduit?

Due to various technical reasons regarding the nature of conduit, you can't currently catch exceptions within the Pipe monad. You have two options:

* Catch exceptions before `lift`ing.
* Catch exceptions thrown from the entire Pipe.

Since the exceptions are always originating in the underlying monad, the first choice is certainly possible in theory, though may require reworking the library you're using a bit.

One other possibility that I haven't actually tried would be to use transPipe[1] to catch all of the exceptions, though I'm not sure how well that would work in practice.

If people have ideas on how to improve the exception handling facilities of conduit, please let me know.

Michael



On Thu, Nov 1, 2012 at 6:26 AM, Hiromi ISHII <konn.jinro <at> gmail.com> wrote:
Hi, there

I'm writing a program communicating with external process, which can be sometimes fail, using conduit and process-conduit package.

Consider the following example, which reads paths from the config file, and passes their contents to external process, and output the results:

```exc.hs
module Main where
import qualified Data.ByteString.Char8 as BS
import           Data.Conduit
import qualified Data.Conduit.Binary   as BC
import qualified Data.Conduit.List     as LC
import           Data.Conduit.Process

main :: IO ()
main = runResourceT $
  BC.sourceFile "paths.dat" $$ BC.lines =$= myConduit =$= LC.mapM_ (unsafeLiftIO . BS.putStrLn)

myConduit :: MonadResource m => Conduit BS.ByteString m BS.ByteString
myConduit = awaitForever $ \path ->
  BC.sourceFile (BS.unpack path) =$= conduitCmd "./sometimes-fail"
```

```sometimes-fail.hs
module Main where
import System.Random

main :: IO ()
main = do
  b <- randomRIO (1,10 :: Int)
  if b < 9 then interact id else error "error!"
```

```paths.dat
txt/a.dat
txt/b.dat
txt/c.dat
...bra, bra, bra...
```

As you can see, `sometimes-fail` is a simple echoing program, but sometimes fail at random.

Successful result is below:

```
$ ./exc
this is a!

this is b!

this is c!

this was d!

this was e!

and this is f.
```

but if `sometimes-fail` fails in some place, `exc` exits with exception like below:

```
$ ./exc
this is a!

this is b!

this is c!
sometimes-fail: error!
```

But I want to write the program acts like below:

```
$ ./exc
this is a!

this is b!

this is c!
sometimes-fail: error!
this was e!

and this is f.
```

that is, ignore the exception and continue to process remaining streams.

So, the question is: how to handle the exception in `myConduit` and proceed to remaining works?

In `conduit` package, `Pipe` type is not an instance of `MonadBaseControl IO` so it cannot handle exceptions within it.
I think this is necessary to make `ResourceT` release resources correctly.

So, how to write the Conduit that ignores some kind of exceptions and proceed to remaining works?
One sometimes want to ignore the invalid input and/or output and just continue to process the remaining stream.

One solution is that libraries using conduit provide "failure-ignore" version for all the `Pipe`s included in the library, but I think it is too heavy solution. It is ideal that `conduit` can package provides combinator that makes exsiting `Pipe`s "failure-ignore".


-- Hiromi ISHII
konn.jinro <at> gmail.com




_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe <at> haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe

_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe <at> haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Hiromi ISHII | 5 Nov 13:41 2012
Picon

Re: How to handle exceptions in conduit?

Hi, there

On 2012/11/01, at 21:23, Michael Snoyman wrote:

> Due to various technical reasons regarding the nature of conduit, you can't currently catch exceptions
within the Pipe monad. You have two options:
> 
> * Catch exceptions before `lift`ing.
> * Catch exceptions thrown from the entire Pipe.
> 
> Since the exceptions are always originating in the underlying monad, the first choice is certainly
possible in theory, though may require reworking the library you're using a bit.

Thanks. In my case, used library is relatively small so I can rewrite it to ignore exception before lifting.
But I think it is more convenient doing the same thing without modifying existing code.

The second choice does not match my case because it cannot resume the process from the place just after an
exception occurred.

> One other possibility that I haven't actually tried would be to use transPipe[1] to catch all of the
exceptions, though I'm not sure how well that would work in practice.

The type of the first argument of `transPipe` should be general, so I think we can't compose it with `catch` function.

-- Hiromi ISHII
konn.jinro <at> gmail.com
Michael Snoyman | 5 Nov 20:51 2012

Re: How to handle exceptions in conduit?


On Nov 5, 2012 2:42 PM, "Hiromi ISHII" <konn.jinro <at> gmail.com> wrote:
>
> Hi, there
>
> On 2012/11/01, at 21:23, Michael Snoyman wrote:
>
> > Due to various technical reasons regarding the nature of conduit, you can't currently catch exceptions within the Pipe monad. You have two options:
> >
> > * Catch exceptions before `lift`ing.
> > * Catch exceptions thrown from the entire Pipe.
> >
> > Since the exceptions are always originating in the underlying monad, the first choice is certainly possible in theory, though may require reworking the library you're using a bit.
>
> Thanks. In my case, used library is relatively small so I can rewrite it to ignore exception before lifting.
> But I think it is more convenient doing the same thing without modifying existing code.
>
> The second choice does not match my case because it cannot resume the process from the place just after an exception occurred.

I agree that it would be great if conduit could meet your use case better. I haven't spent enough cycles looking at this yet to determine if the reason we don't have this support is a limitation in the conduit approach itself, or just a limitation in what I was able to implement so far. If you can think of a way to implement more fine-grained exception handling (or anyone else for that matter), I'd love to hear about it.

> > One other possibility that I haven't actually tried would be to use transPipe[1] to catch all of the exceptions, though I'm not sure how well that would work in practice.
>
> The type of the first argument of `transPipe` should be general, so I think we can't compose it with `catch` function.

That makes sense.

> -- Hiromi ISHII
> konn.jinro <at> gmail.com
>
>
>
>
> _______________________________________________
> Haskell-Cafe mailing list
> Haskell-Cafe <at> haskell.org
> http://www.haskell.org/mailman/listinfo/haskell-cafe

_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe <at> haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Michael Snoyman | 5 Nov 21:10 2012

Re: How to handle exceptions in conduit?




On Mon, Nov 5, 2012 at 9:51 PM, Michael Snoyman <michael <at> snoyman.com> wrote:


On Nov 5, 2012 2:42 PM, "Hiromi ISHII" <konn.jinro <at> gmail.com> wrote:
>
> Hi, there
>
> On 2012/11/01, at 21:23, Michael Snoyman wrote:
>
> > Due to various technical reasons regarding the nature of conduit, you can't currently catch exceptions within the Pipe monad. You have two options:
> >
> > * Catch exceptions before `lift`ing.
> > * Catch exceptions thrown from the entire Pipe.
> >
> > Since the exceptions are always originating in the underlying monad, the first choice is certainly possible in theory, though may require reworking the library you're using a bit.
>
> Thanks. In my case, used library is relatively small so I can rewrite it to ignore exception before lifting.
> But I think it is more convenient doing the same thing without modifying existing code.
>
> The second choice does not match my case because it cannot resume the process from the place just after an exception occurred.

I agree that it would be great if conduit could meet your use case better. I haven't spent enough cycles looking at this yet to determine if the reason we don't have this support is a limitation in the conduit approach itself, or just a limitation in what I was able to implement so far. If you can think of a way to implement more fine-grained exception handling (or anyone else for that matter), I'd love to hear about it.

> > One other possibility that I haven't actually tried would be to use transPipe[1] to catch all of the exceptions, though I'm not sure how well that would work in practice.
>
> The type of the first argument of `transPipe` should be general, so I think we can't compose it with `catch` function.

That makes sense.


> -- Hiromi ISHII
> konn.jinro <at> gmail.com
>
>
>
>
> _______________________________________________
> Haskell-Cafe mailing list
> Haskell-Cafe <at> haskell.org
> http://www.haskell.org/mailman/listinfo/haskell-cafe

Sorry, small follow-up. It's certainly possible to make some kind of catching function, e.g.:

catchPipe :: (MonadBaseControl IO m, Exception e) => Pipe l i o u m r -> (e -> Pipe l i o u m r) -> Pipe l i o u m r
catchPipe (HaveOutput p c o) f = HaveOutput (catchPipe p f) c o
catchPipe (NeedInput p c) f = NeedInput (flip catchPipe f . p) (flip catchPipe f . c)
catchPipe (Done r) _ = Done r
catchPipe (PipeM mp) f = PipeM $ Control.Exception.Lifted.catch (liftM (flip catchPipe f) mp) (return . f)
catchPipe (Leftover p l) f = Leftover (catchPipe p f) l

I'm just not certain how useful this is in practice, as it doesn't really give you any information on what else that Pipe was about to perform. So you can't really just pick up where you left off.

Michael
_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe <at> haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe

Gmane