Nils | 24 Feb 19:23 2013

ANN: conduit-network-stream, A base layer for network protocols with Conduits

Hello everyone,

I just uploaded my "conduit-network-stream" package to hackage. It's a 
base layer for network protocols based on the "Conduit" package by 
Michael Snoyman which makes it possible to send multiple messages over a 
continuous network connection in a convenient way.

I wrote more about it at:

https://github.com/mcmaniac/conduit-network-stream/blob/master/README.md

It is available on hackage and github:

http://hackage.haskell.org/package/conduit-network-stream
https://github.com/mcmaniac/conduit-network-stream

Until the documentation on hackage is generated, I also host the haddock 
documentation at:

http://hs.nils.cc/conduit-network-stream/index.html

For any questions/problems/requests, please ask!

- Nils
Alexander V Vershilov | 25 Feb 06:52 2013
Picon

Re: ANN: conduit-network-stream, A base layer for network protocols with Conduits

Hello, Nils.

Can you describe if there any difference with latest conduit API  (yield, await) that can be
used to write functions in a very similar style, but without using exeternal packages:

>  runTCPServer settings app = appSource ap $$ go =$= CL.mapM_ encode =$ appSink app
>     where
>         go = forever $ do
>            bp <- decode <$> await {- decode is inlined here assuming it can return different types -}
>            {- logic here-}
>            yield $ result
>            mapM (yeild) [result1,result2...]

more over you don't need to write appSource $$ ... appSink on the top.
Here is an example of a client that authorizes and then reads commands
from TBMChan and requests server using json format.

> runTCPClient settings ad go
>   where
>       go = do
>          source $$ await
>          is <- authorize
>          when is loop'
>       authorize = do
>            yield u >> yield (S8.pack "\n") $$ sink -- send authorization token
>            mx <- source $$ sinkParser json -- get responce
>            ...
>        loop' = do
>            minfo <- atomically $ readTBMChan ch
>            case minfo of
>              Nothing -> return ()
>              Just (message, respBox) -> do
>                yield message $$ C.concatMap (SL.toChunks . encode) =$ sink
>                resp <- source $$ sinkParser json
>                atomically $ putTMVar respBox resp
>                loop'
>       source = ppSource ad
>       sink = appSink ad



On 24 February 2013 22:23, Nils <mail <at> nils.cc> wrote:
Hello everyone,

I just uploaded my "conduit-network-stream" package to hackage. It's a base layer for network protocols based on the "Conduit" package by Michael Snoyman which makes it possible to send multiple messages over a continuous network connection in a convenient way.

I wrote more about it at:

https://github.com/mcmaniac/conduit-network-stream/blob/master/README.md

It is available on hackage and github:

http://hackage.haskell.org/package/conduit-network-stream
https://github.com/mcmaniac/conduit-network-stream

Until the documentation on hackage is generated, I also host the haddock documentation at:

http://hs.nils.cc/conduit-network-stream/index.html

For any questions/problems/requests, please ask!


- Nils

_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe <at> haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe



--
Alexander
_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe <at> haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Nils | 25 Feb 12:53 2013

Re: ANN: conduit-network-stream, A base layer for network protocols with Conduits

Hi Alexander,

Am 25.02.2013 06:52, schrieb Alexander V Vershilov:
> Can you describe if there any difference with latest conduit API
> (yield, await) that can be
> used to write functions in a very similar style, but without using
> exeternal packages:

I have indeed written this library for the 0.5.6 API, so things might 
have changed a bit in the 1.0 API, but I'd be surprised if the 
fundamental flaws of this approach would have been fixed.

>  >  runTCPServer settings app = appSource ap $$ go =$= CL.mapM_ encode
> =$ appSink app
>  >     where
>  >         go = forever $ do
>  >            bp <- decode <$> await

`await` works on a conduit of strict `ByteString` chunks. The 
size/length of each `ByteString` is dependent on your network 
connection, so if you have a fast internet connection and your 
bytestrings are sufficiently separated because you're waiting for a 
response between each message your program might actually work as you 
expect it to do (with a little bit of luck). But consider a simple 
server/client application where the messages are not seperated by a 
small delay:

     server = runTCPServer (..) $ \ad ->
         appSource ad $$ Data.Conduit.List.mapM_ (liftIO . print)
     client = runTCPClient (..) $ \ad ->
         (yield "msg1" >> yield "msg2") $$ appSink ad

The server will simply print out "msg1msg2" as one message, not as two 
separate messages. Even worse, if your network connection is bad or your 
chunks are getting too big for buffering, you might end up with 
something like:

     "msg1ms"
     "g2"

`await` is not reliable in that regard because the network source is not 
consistent and non-deterministic. My libraray makes sure that every 
"yield" from the client corresponds to exactly one (not more or less) 
"await" at the server.

There are more benefits when using my library. For example consider a 
client which first sends an authorization message, then a couple of 
hashes from different files and then maybe some timestamps on some other 
files. Writing that server is straight forward:

     client = runTCPClient (..) $ \ad ->
         send1    ad $$ yield authenticationMsg
         sendList ad $$ mapM_ yield [file1hash, file2hash, file3hash]
         sendList ad $$ mapM_ yield [timestamp1, timestamp2]

     server = runTCPServer (..) $ \ad ->
         (next,[authmsg])  <- receive ad   $$ Data.Conduit.List.consume
         (next,hashes)     <- receive next $$ Data.Conduit.List.consume
         (next,timestamps) <- receive next $$ Data.Conduit.List.consume
         close next

Each `receive` corresponds to exactly one `send`. Without this library 
you have to manually check/verify/associate each message by hand in one 
big loop, whereas this library allows you to split your application into 
logical conduit "groups" which are straight forward to use.

- Nils
Alexander V Vershilov | 25 Feb 13:10 2013
Picon

Re: ANN: conduit-network-stream, A base layer for network protocols with Conduits

Thanks for explanation.

However I'm still confused as if I'll use one of protocols (protobuf, json, sereal message, binary message) all of them has
iteration interface i.e. function of type (ByteString -> Result a) that returns
data Result = Error Text ByteString | NeedMore (ByteString -> Result a) | HaveResult a
all of them can be used in conduits using sinkParser or same approach and so there will be
no problem in sending and receiving messages even if they are split, and that will be done in a logical block.

I see that there will be problems if I'll use raw data and I see that your library solves it, am I right?



On 25 February 2013 15:53, Nils <mail <at> nils.cc> wrote:
Hi Alexander,

Am 25.02.2013 06:52, schrieb Alexander V Vershilov:

Can you describe if there any difference with latest conduit API
(yield, await) that can be
used to write functions in a very similar style, but without using
exeternal packages:

I have indeed written this library for the 0.5.6 API, so things might have changed a bit in the 1.0 API, but I'd be surprised if the fundamental flaws of this approach would have been fixed.


 >  runTCPServer settings app = appSource ap $$ go =$= CL.mapM_ encode
=$ appSink app
 >     where
 >         go = forever $ do
 >            bp <- decode <$> await

`await` works on a conduit of strict `ByteString` chunks. The size/length of each `ByteString` is dependent on your network connection, so if you have a fast internet connection and your bytestrings are sufficiently separated because you're waiting for a response between each message your program might actually work as you expect it to do (with a little bit of luck). But consider a simple server/client application where the messages are not seperated by a small delay:

    server = runTCPServer (..) $ \ad ->
        appSource ad $$ Data.Conduit.List.mapM_ (liftIO . print)
    client = runTCPClient (..) $ \ad ->
        (yield "msg1" >> yield "msg2") $$ appSink ad

The server will simply print out "msg1msg2" as one message, not as two separate messages. Even worse, if your network connection is bad or your chunks are getting too big for buffering, you might end up with something like:

    "msg1ms"
    "g2"

`await` is not reliable in that regard because the network source is not consistent and non-deterministic. My libraray makes sure that every "yield" from the client corresponds to exactly one (not more or less) "await" at the server.

There are more benefits when using my library. For example consider a client which first sends an authorization message, then a couple of hashes from different files and then maybe some timestamps on some other files. Writing that server is straight forward:

    client = runTCPClient (..) $ \ad ->
        send1    ad $$ yield authenticationMsg
        sendList ad $$ mapM_ yield [file1hash, file2hash, file3hash]
        sendList ad $$ mapM_ yield [timestamp1, timestamp2]

    server = runTCPServer (..) $ \ad ->
        (next,[authmsg])  <- receive ad   $$ Data.Conduit.List.consume
        (next,hashes)     <- receive next $$ Data.Conduit.List.consume
        (next,timestamps) <- receive next $$ Data.Conduit.List.consume
        close next

Each `receive` corresponds to exactly one `send`. Without this library you have to manually check/verify/associate each message by hand in one big loop, whereas this library allows you to split your application into logical conduit "groups" which are straight forward to use.


- Nils



--
Alexander
_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe <at> haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Nils | 25 Feb 14:09 2013

Re: ANN: conduit-network-stream, A base layer for network protocols with Conduits

Of course you don't have to use my library if your library already 
solves the same problem.

Am 25.02.2013 13:10, schrieb Alexander V Vershilov:
> data Result = Error Text ByteString | NeedMore (ByteString -> Result a)
> | HaveResult a

I still see an issue with this if you're receiving multiple messages in 
one `ByteString` chunk. A type like `HaveResult a ByteString` which also 
returns the leftover `ByteString` after the first result has been found 
could solve this, but you'd still have to re-yield this bytestring by 
hand and you'll possible end up with more calls to "decode" than necessary.

Of course, with json you could avoid this problem by wrapping all 
messages into a json-list and sending this list as one big message. But 
that way you lose almost all advantages of a conduit - you cannot map 
over those list iems, you cannot process each item one by one, you 
cannot fold or filter them. When using this approach I don't really see 
the advantage over using the plain old "connectTo/hGetContents" method?

Anyway, my library is independent of the "bytestring encoding" you want 
to use with it. I think it's the most general solution and avoids having 
to re-implement the same solution for each and every "protocol design" 
over and over again.
Ozgun Ataman | 25 Feb 16:49 2013
Picon

Re: ANN: conduit-network-stream, A base layer for network protocols with Conduits

One question - and sorry that I didn't get a chance to try some examples myself - but can bi-directional conversations be carried between server/client in interleaving fashion using your library? Something like (simplified):

myClient = receive >>= send . compute

I have previously accomplished this using TBMChan and cereal-conduit to achieve the message delineation, but your library would certainly remove a layer of complexity if interleaved bi-directional talk is possible.

- Ozgun

On Monday, February 25, 2013 at 8:09 AM, Nils wrote:

Of course you don't have to use my library if your library already
solves the same problem.

Am 25.02.2013 13:10, schrieb Alexander V Vershilov:
data Result = Error Text ByteString | NeedMore (ByteString -> Result a)
| HaveResult a

I still see an issue with this if you're receiving multiple messages in
one `ByteString` chunk. A type like `HaveResult a ByteString` which also
returns the leftover `ByteString` after the first result has been found
could solve this, but you'd still have to re-yield this bytestring by
hand and you'll possible end up with more calls to "decode" than necessary.

Of course, with json you could avoid this problem by wrapping all
messages into a json-list and sending this list as one big message. But
that way you lose almost all advantages of a conduit - you cannot map
over those list iems, you cannot process each item one by one, you
cannot fold or filter them. When using this approach I don't really see
the advantage over using the plain old "connectTo/hGetContents" method?

Anyway, my library is independent of the "bytestring encoding" you want
to use with it. I think it's the most general solution and avoids having
to re-implement the same solution for each and every "protocol design"
over and over again.

_______________________________________________
Haskell-Cafe mailing list

_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe <at> haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe

Gmane