Myles C. Maxfield | 27 Oct 21:20 2012
Picon

Auto-termination and leftovers in Conduits

Hey,
Say I have a stream of Data.Text.Text objects flowing through a
conduit, where the divisions between successive Data.Text.Text items
occur at arbitrary boundaries (maybe the source is sourceFile $=
decode utf8). I'd like to create a Sink that returns a tuple of (the
first line, the rest of the input).

My first attempt at this looks like this:

sink = do
  out1 <- CT.lines C.=$ CL.head
  out2 <- CL.consume
  return (out1, T.concat out2)

However, the following input provides:

runIdentity $ CL.sourceList ["abc\nde", "f\nghi"] C.$$ sink
(Just "abc","f\nghi")

But what I really want is
(Just "abc", "\ndef\nghi")

I think this is due to the auto-termination you mention in [1]. My
guess is that when CT.lines yields the first value, (CL.head then also
yields it,) and execution is auto-terminated before CT.lines gets a
chance to specify any leftovers.

How can I write this sink? (I know I can just use CL.consume and
T.break (== '\n'), but I'm not interested in that. I'm trying to
figure out how to get the behavior I'm looking for with conduits.)
(Continue reading)

Michael Snoyman | 28 Oct 04:35 2012

Re: Auto-termination and leftovers in Conduits

The important issue here is that, when using =$, $=, and =$=, leftovers will discarded. To see this more clearly, realize that the first line of sink is equivalent to:

  out1 <- C.injectLeftovers CT.lines C.>+> CL.head

So any leftovers from lines are lost once you move past that line. In order to get this to work, stick the consume inside the same composition:

sink = C.injectLeftovers CT.lines C.>+> do
    out1 <- CL.head
    out2 <- CL.consume
    return (out1, T.unlines out2)

Or:

sink = CT.lines C.=$ do
    out1 <- CL.head
    out2 <- CL.consume
    return (out1, T.unlines out2)

Michael

On Sat, Oct 27, 2012 at 9:20 PM, Myles C. Maxfield <myles.maxfield <at> gmail.com> wrote:
Hey,
Say I have a stream of Data.Text.Text objects flowing through a
conduit, where the divisions between successive Data.Text.Text items
occur at arbitrary boundaries (maybe the source is sourceFile $=
decode utf8). I'd like to create a Sink that returns a tuple of (the
first line, the rest of the input).

My first attempt at this looks like this:

sink = do
  out1 <- CT.lines C.=$ CL.head
  out2 <- CL.consume
  return (out1, T.concat out2)

However, the following input provides:

runIdentity $ CL.sourceList ["abc\nde", "f\nghi"] C.$$ sink
(Just "abc","f\nghi")

But what I really want is
(Just "abc", "\ndef\nghi")

I think this is due to the auto-termination you mention in [1]. My
guess is that when CT.lines yields the first value, (CL.head then also
yields it,) and execution is auto-terminated before CT.lines gets a
chance to specify any leftovers.

How can I write this sink? (I know I can just use CL.consume and
T.break (== '\n'), but I'm not interested in that. I'm trying to
figure out how to get the behavior I'm looking for with conduits.)

Thanks,
Myles

[1] http://hackage.haskell.org/packages/archive/conduit/0.5.2.7/doc/html/Data-Conduit.html

_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe <at> haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Myles C. Maxfield | 28 Oct 08:40 2012
Picon

Re: Auto-termination and leftovers in Conduits

Cool! Thanks so much!

--Myles

On Sat, Oct 27, 2012 at 8:35 PM, Michael Snoyman <michael <at> snoyman.com> wrote:
> The important issue here is that, when using =$, $=, and =$=, leftovers will
> discarded. To see this more clearly, realize that the first line of sink is
> equivalent to:
>
>   out1 <- C.injectLeftovers CT.lines C.>+> CL.head
>
> So any leftovers from lines are lost once you move past that line. In order
> to get this to work, stick the consume inside the same composition:
>
> sink = C.injectLeftovers CT.lines C.>+> do
>     out1 <- CL.head
>     out2 <- CL.consume
>     return (out1, T.unlines out2)
>
> Or:
>
> sink = CT.lines C.=$ do
>     out1 <- CL.head
>     out2 <- CL.consume
>     return (out1, T.unlines out2)
>
> Michael
>
> On Sat, Oct 27, 2012 at 9:20 PM, Myles C. Maxfield
> <myles.maxfield <at> gmail.com> wrote:
>>
>> Hey,
>> Say I have a stream of Data.Text.Text objects flowing through a
>> conduit, where the divisions between successive Data.Text.Text items
>> occur at arbitrary boundaries (maybe the source is sourceFile $=
>> decode utf8). I'd like to create a Sink that returns a tuple of (the
>> first line, the rest of the input).
>>
>> My first attempt at this looks like this:
>>
>> sink = do
>>   out1 <- CT.lines C.=$ CL.head
>>   out2 <- CL.consume
>>   return (out1, T.concat out2)
>>
>> However, the following input provides:
>>
>> runIdentity $ CL.sourceList ["abc\nde", "f\nghi"] C.$$ sink
>> (Just "abc","f\nghi")
>>
>> But what I really want is
>> (Just "abc", "\ndef\nghi")
>>
>> I think this is due to the auto-termination you mention in [1]. My
>> guess is that when CT.lines yields the first value, (CL.head then also
>> yields it,) and execution is auto-terminated before CT.lines gets a
>> chance to specify any leftovers.
>>
>> How can I write this sink? (I know I can just use CL.consume and
>> T.break (== '\n'), but I'm not interested in that. I'm trying to
>> figure out how to get the behavior I'm looking for with conduits.)
>>
>> Thanks,
>> Myles
>>
>> [1]
>> http://hackage.haskell.org/packages/archive/conduit/0.5.2.7/doc/html/Data-Conduit.html
>
>

Gmane