headers
andrea crotti | 2 Aug 2012 18:50
Picon

Designing a new architecture

I have to restructure a complex application and I would like to do this
with zeromq, I'm reading the doc and trying out things but I'm a bit
lost now, so some hints might be very useful..

The application I'm writing is very parallel, there are many
long-running processes that do things, managed by a central daemon.

I would like to be able to communicate with these processes to check the
current status, and interact with it, and these processes should write
the results on a common sink.

So it looks like the multiple worker with a sink might be the right
choice, but I also want to be able to communicate with every single
process separately.

Another thing which I'd like to do is to use Python Cmd class, and I
thought I could create a zmq class that behaves like a file, and thus be
passed in the Cmd object.

So I did the following, but apparently I just can't get it working,
because every time I receive I should acknoledge with a write and
vice-versa..
Do you think it's possible/makes sense to do something like this?

__metaclass__ = type

import zmq

ADDR = "tcp://127.0.0.1:5556"
context = zmq.Context()
(Continue reading)

Permalink | Reply |
headers
Michel Pelletier | 2 Aug 2012 20:07
Picon

Re: Designing a new architecture

On Thu, Aug 2, 2012 at 9:50 AM, andrea crotti <andrea.crotti.0 <at> gmail.com> wrote:
> I have to restructure a complex application and I would like to do this
> with zeromq, I'm reading the doc and trying out things but I'm a bit
> lost now, so some hints might be very useful..
>
> The application I'm writing is very parallel, there are many
> long-running processes that do things, managed by a central daemon.
>
> I would like to be able to communicate with these processes to check the
> current status, and interact with it, and these processes should write
> the results on a common sink.
>
> So it looks like the multiple worker with a sink might be the right
> choice, but I also want to be able to communicate with every single
> process separately.

I would suggest looking at the ROUTER socket in detail and
specifically the LRU pattern in the guide.  You may only need the
"back-half" of the LRU pattern to do what you need.  This allows you
to have bidirectional communication with a bunch of workers.

> Another thing which I'd like to do is to use Python Cmd class, and I
> thought I could create a zmq class that behaves like a file, and thus be
> passed in the Cmd object.

I would avoid stretching the file abstraction over a zmq socket like
that.   Maybe another library that is more flexible for command line
and simple interpreters like cliff
(http://pypi.python.org/pypi/cliff/) would be easier for you to
integrate with zmq.  Cmd after all, is pretty archaic and file
(Continue reading)

Permalink | Reply |
headers
Michel Pelletier | 2 Aug 2012 20:09
Picon

Re: Designing a new architecture

Specifically, check out cliff's cool interactive mode:

http://cliff.readthedocs.org/en/latest/interactive_mode.html

On Thu, Aug 2, 2012 at 11:07 AM, Michel Pelletier
<pelletier.michel <at> gmail.com> wrote:
> On Thu, Aug 2, 2012 at 9:50 AM, andrea crotti <andrea.crotti.0 <at> gmail.com> wrote:
>> I have to restructure a complex application and I would like to do this
>> with zeromq, I'm reading the doc and trying out things but I'm a bit
>> lost now, so some hints might be very useful..
>>
>> The application I'm writing is very parallel, there are many
>> long-running processes that do things, managed by a central daemon.
>>
>> I would like to be able to communicate with these processes to check the
>> current status, and interact with it, and these processes should write
>> the results on a common sink.
>>
>> So it looks like the multiple worker with a sink might be the right
>> choice, but I also want to be able to communicate with every single
>> process separately.
>
> I would suggest looking at the ROUTER socket in detail and
> specifically the LRU pattern in the guide.  You may only need the
> "back-half" of the LRU pattern to do what you need.  This allows you
> to have bidirectional communication with a bunch of workers.
>
>> Another thing which I'd like to do is to use Python Cmd class, and I
>> thought I could create a zmq class that behaves like a file, and thus be
>> passed in the Cmd object.
(Continue reading)

Permalink | Reply |
headers
andrea crotti | 3 Aug 2012 14:13
Picon

Re: Designing a new architecture

2012/8/2 Michel Pelletier <pelletier.michel <at> gmail.com>:
> Specifically, check out cliff's cool interactive mode:
>
> http://cliff.readthedocs.org/en/latest/interactive_mode.html
>
>>
>> I would suggest looking at the ROUTER socket in detail and
>> specifically the LRU pattern in the guide.  You may only need the
>> "back-half" of the LRU pattern to do what you need.  This allows you
>> to have bidirectional communication with a bunch of workers.
>>
>>> Another thing which I'd like to do is to use Python Cmd class, and I
>>> thought I could create a zmq class that behaves like a file, and thus be
>>> passed in the Cmd object.
>>
>> I would avoid stretching the file abstraction over a zmq socket like
>> that.   Maybe another library that is more flexible for command line
>> and simple interpreters like cliff
>> (http://pypi.python.org/pypi/cliff/) would be easier for you to
>> integrate with zmq.  Cmd after all, is pretty archaic and file
>> oriented.
>>
>> -Michel
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev <at> lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev

So far I haven't found much more in the InteractiveClass to help me
putting zeromq and cliff together actually:
(Continue reading)

Permalink | Reply |
headers
andrea crotti | 7 Aug 2012 15:19
Picon

Re: Designing a new architecture

2012/8/3 andrea crotti <andrea.crotti.0 <at> gmail.com>:
>
>
>
> So far I haven't found much more in the InteractiveClass to help me
> putting zeromq and cliff together actually:
>
> class cliff.interactive.InteractiveApp(parent_app, command_manager,
> stdin, stdout)
>
> it still takes a stdin and stdout objects, so it expects files..
> Instead of making a zeromq object behave like a file we could make it
> write and read from a file which is written and read by different
> processes, could that be a solution?

Since I am stubborn I was still trying to get the stdin/stdout
emulation with something like:

class ZmqStdin:
    """Fake a stdin file,
    """
    def __init__(self):
        context = zmq.Context()
        self.socket = context.socket(zmq.PULL)
        self.socket.connect(STDIN)

    def read(self):
        msg = self.socket.recv()
        # should I replace also the others?
        return msg
(Continue reading)

Permalink | Reply |
headers
andrea crotti | 7 Aug 2012 15:47
Picon

Re: Designing a new architecture

So now I tried the following, the smart process runs a subprocess in
background, the samplecmd should send the LIST query, but it just
hangs there, and I don't get any answer..
Is there anything missing (can't find anything there)?

import zmq

import cmd2 as cmd
from multiprocessing import Process
from subprocess import Popen, PIPE

SOCK = "tcp://127.0.0.1:4444"

class SampleCmd(cmd.Cmd):
    def __init__(self, query):
        cmd.Cmd.__init__(self)
        self.query = query

    def do_list(self, testid):
        """List all the commands
        """
        self.query.send('LIST')
        print(self.query.recv().split(' '))

class SmartProcess(Process):
    def __init__(self):
        Process.__init__(self)
        context = zmq.Context()
        self.socket = context.socket(zmq.REP)
        self.socket.bind(SOCK)
(Continue reading)

Permalink | Reply |
headers
Paul Colomiets | 8 Aug 2012 22:44
Gravatar

Re: Designing a new architecture

On Tue, Aug 7, 2012 at 4:47 PM, andrea crotti <andrea.crotti.0 <at> gmail.com> wrote:
> So now I tried the following, the smart process runs a subprocess in
> background, the samplecmd should send the LIST query, but it just
> hangs there, and I don't get any answer..
> Is there anything missing (can't find anything there)?
>

Are you building something like Salt? (http://saltstack.org) I think
you should try salt, or look at it's codebase.

--

-- 
Paul
Permalink | Reply |
headers
andrea crotti | 3 Aug 2012 14:06
Picon

Re: Designing a new architecture

2012/8/2 Michel Pelletier <pelletier.michel <at> gmail.com>:
> I would avoid stretching the file abstraction over a zmq socket like
> that.   Maybe another library that is more flexible for command line
> and simple interpreters like cliff
> (http://pypi.python.org/pypi/cliff/) would be easier for you to
> integrate with zmq.  Cmd after all, is pretty archaic and file
> oriented.
>

Thanks for the answer, I'm playing with cliff now and I will try to
integrate it with zeromq, it might be very nice if I get that working..
If anyone already did I'm all ears ;)
All I want is that every process listens on a port and I can connect
to its argument parser.

As a comment from a fresh lerner, the documentation is great, but I'm
missing a few things:
- some good references on theoretical simple explanation about the
various models.
- there are many examples, but at least the Python ones are not very
commented, and
  also most of them need to run two different files to actually get
something done,
  and it's not very clear to me yet how to do.

I'll go back to chapter 3 to see if I can understand something more..
Permalink | Reply |

Gmane