Re: patterns with back-pressure

Andrew,

You can get information about whether a worker is blocked or slow by
having it send back data along another channel.  Collect this and you
know how fast it's working, compare number of responses to number of
requests and you have the real total queue length.

You can then route dynamically by constructing envelopes and passing
them to an XREP socket to which your herds or chokepoints are connected.

I hope this makes sense...

-Pieter

On Tue, Sep 7, 2010 at 5:06 PM, Andrew Hume <andrew <at> research.att.com> wrote:
> martin,
> thanks.
> to the extent you can set an HWM, then there ought to be a way
> of measuring how much of that is being used. sure, its an estimate
> and not exact. but if i have the PUSHer being blocked because of one client,
> it would surely be good to know who that was. or if a client was
> unexpectedly
> slow at servicing requests.
> in my case, because i have an anonymous herd of PUSHers feeding an
> anonymous herd of PULLers (for which there is no pattern), i have well-known
> chokepoints (one per node in a cluster), and i can measure the traffic
> flowing
> through each chokepoint. thus, i can tell the first herd of PUSHers how
> to statistically shape their traffic to keep per-node queues balanced.
> arguably, this might be done inside 0mq, but as per the discussion
> we had a while back on distribution policy, for now it seems wise
> to let all that stuff sit above 0mq. (even for that, though, it would be
> nice to be able to get an estimate of queue length.)
> andrew
> On Sep 7, 2010, at 7:35 AM, Martin Sustrik wrote:
>
> Andrew,
> No. There's no such API. It's not even obvious what the semantics
> should be. Some messages are in 0MQ tx buffer, some are in TCP tx buffer
> or PGM tx buffer, some may be on the wire but not yet at the receiver.
> Moreover, the whole system is in state of flux so any figure you would
> get would be outdated already etc.
> Martin
> On 7/9/2010, "Andrew Hume" <andrew <at> research.att.com> wrote:
>
> thanks!
> is there an API to get the queue length?
> On Sep 7, 2010, at 6:43 AM, Martin Sustrik wrote:
>
> Andrew,
>
> just checking:
> is there any pattern that incorporates back-pressure?
> that is, fair share but taking queue length into account?
>
> Both req/rep and push/pull patterns do apply backpressure _if_ there's
> high watermark set (ZMQ_HWM socket option) for the length of the
> queue.
> Martin
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev <at> lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
> ------------------
> Andrew Hume  (best -> Telework) +1 732-886-1886
> andrew <at> research.att.com  (Work) +1 973-360-8651
> AT&T Labs - Research; member of USENIX and LOPSA
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev <at> lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
> ------------------
> Andrew Hume  (best -> Telework) +1 732-886-1886
> andrew <at> research.att.com  (Work) +1 973-360-8651
> AT&T Labs - Research; member of USENIX and LOPSA
>
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev <at> lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>

--

-- 
-
Pieter Hintjens
iMatix - www.imatix.com