Ertugrul Söylemez | 10 May 16:56 2013
Picon

Stream processing

Hello everybody,

I'm trying to formulate the stream processing problem, which doesn't
seem to be solved fully by the currently existing patterns.  I'm
experimenting with a new idea, but I want to make sure that I don't miss
any defining features of the problem, so here is my list.  A stream
processing abstraction should:

  * have a categorically proven design (solved by iteratees, pipes),

  * be composable (solved by all of them),

  * be reasonably easy to understand and work with (solved by conduit,
    pipes),

  * support leftovers (solved by conduit and to some degree by
    iteratees),

  * be reliable in the presence of async exceptions (solved by conduit,
    pipes-safe),

  * hold on to resources only as long as necessary (solved by conduit
    and to some degree by pipes-safe),

  * ideally also allow upstream communication (solved by pipes and to
    some degree by conduit).

  * be fast (solved by all of them).

Anything else you would put in that list?
(Continue reading)

Gabriel Gonzalez | 11 May 05:37 2013
Picon

Re: Stream Processing

As far as I can tell, the only thing on that list not solved currently 
by `pipes` is leftovers, but I will be releasing that very soon.
> Hello everybody,
>
> I'm trying to formulate the stream processing problem, which doesn't
> seem to be solved fully by the currently existing patterns.  I'm
> experimenting with a new idea, but I want to make sure that I don't miss
> any defining features of the problem, so here is my list.  A stream
> processing abstraction should:
>
>    * have a categorically proven design (solved by iteratees, pipes),
>
>    * be composable (solved by all of them),
>
>    * be reasonably easy to understand and work with (solved by conduit,
>      pipes),
>
>    * support leftovers (solved by conduit and to some degree by
>      iteratees),
>
>    * be reliable in the presence of async exceptions (solved by conduit,
>      pipes-safe),
>
>    * hold on to resources only as long as necessary (solved by conduit
>      and to some degree by pipes-safe),
>
>    * ideally also allow upstream communication (solved by pipes and to
>      some degree by conduit).
>
>    * be fast (solved by all of them).
(Continue reading)

Michael Snoyman | 11 May 20:04 2013

Re: Stream processing

It's quite old at this point, but you may be interested in reading the initial motivations for creating conduit when the iteratee pattern (and enumerator library in particular) already existed:


I would say the only real component missing from your list is being able to structure significantly more complicated control flows, such as the use case of combining a web server and web client into a web proxy. That was probably the example which finally pushed me to start thinking seriously about an enumerator replacement. In conduit, this use case is addressed by connect-and-resume, which essentially allows you to escape the inversion of control normally introduced by the conduit pattern.


On Fri, May 10, 2013 at 5:56 PM, Ertugrul Söylemez <es <at> ertes.de> wrote:
Hello everybody,

I'm trying to formulate the stream processing problem, which doesn't
seem to be solved fully by the currently existing patterns.  I'm
experimenting with a new idea, but I want to make sure that I don't miss
any defining features of the problem, so here is my list.  A stream
processing abstraction should:

  * have a categorically proven design (solved by iteratees, pipes),

  * be composable (solved by all of them),

  * be reasonably easy to understand and work with (solved by conduit,
    pipes),

  * support leftovers (solved by conduit and to some degree by
    iteratees),

  * be reliable in the presence of async exceptions (solved by conduit,
    pipes-safe),

  * hold on to resources only as long as necessary (solved by conduit
    and to some degree by pipes-safe),

  * ideally also allow upstream communication (solved by pipes and to
    some degree by conduit).

  * be fast (solved by all of them).

Anything else you would put in that list?


Greets,
Ertugrul

--
Not to be or to be and (not to be or to be and (not to be or to be and
(not to be or to be and ... that is the list monad.

_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe <at> haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe


_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe <at> haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Gabriel Gonzalez | 11 May 21:15 2013
Picon

Re: Stream processing

On 05/11/2013 11:04 AM, Michael Snoyman wrote:
> It's quite old at this point, but you may be interested in reading the 
> initial motivations for creating conduit when the iteratee pattern 
> (and enumerator library in particular) already existed:
>
> https://github.com/snoyberg/conduit/blob/master/README.md#general-goal
>
> I would say the only real component missing from your list is being 
> able to structure significantly more complicated control flows, such 
> as the use case of combining a web server and web client into a web 
> proxy. That was probably the example which finally pushed me to start 
> thinking seriously about an enumerator replacement. In conduit, this 
> use case is addressed by connect-and-resume, which essentially allows 
> you to escape the inversion of control normally introduced by the 
> conduit pattern.
>
Speaking of escaping inversion of control, I'm going to take a moment to 
shamelessly plug this post of mine showing that you can escape the pipe 
monad using a trick uncannily similar to `io-streams`:

http://www.haskellforall.com/2013/04/pipes-and-io-streams.html

It's not equivalent to `conduit`'s connect and resume, though, which is 
very different.
Ertugrul Söylemez | 12 May 00:19 2013
Picon

Re: Stream processing

Michael Snoyman <michael <at> snoyman.com> wrote:

> In conduit, this use case is addressed by connect-and-resume, which
> essentially allows you to escape the inversion of control normally
> introduced by the conduit pattern.

Indeed, but this isn't even an extra feature in my case.  It follows
naturally from the automaton design.  Thanks for pointing it out,
though.  I need to think about how to expose this feature.

Greets,
Ertugrul

--

-- 
Key-ID: E5DD8D11 "Ertugrul Soeylemez <es <at> ertes.de>"
FPrint: BD28 3E3F BE63 BADD 4157  9134 D56A 37FA E5DD 8D11
Keysrv: hkp://subkeys.pgp.net/
_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe <at> haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe

Gmane