This is beanstalk-talk, the mailing list for developer and user discussion for beanstalkd and its client libraries. ()

Keith Rarick | 9 Aug 2009 11:13

Re: Architecturally lost!?

On Sat, Aug 8, 2009 at 12:21 PM, Tom<qklxmf@...> wrote:
> On Aug 8, 9:14 pm, Tom <qkl...@...> wrote:
>> I am a queue newbie and are completely lost in the plethora of
>> possibilities. My question is, what I will draft here, the best and
>> MOST simple way to reach my goal - and especially is beanstalk the way
>> to go? Any pointers really appreciated!

Hi Tom,

Your situation sounds like a perfect fit for beanstalkd.

Simplicity is one thing beanstalkd has over every AMQP program.

Several million messages per day throughput is easy.

If you want to get started with basics, there is a tutorial in ruby at
http://devver.net/blog/2008/10/ruby-beanstalkd-distributed-worker-basics/
. It has lots of example code.

> Concrete questions:
> 1.) Should I have two queues per protocol (incoming and outgoing). So
> I end up having 6 queues if I use twitter, sms and xmpp?

If you have a process that only knows how to handle, say, outgoing
sms, it should get its own tube (aka queue).

How are the incoming jobs created? Can you generate them all in the
same format? If so, you could make one incoming tube, and have one
type of worker that handles all incoming jobs, and puts appropriate

(Continue reading)

headers

Tom | 9 Aug 2009 22:16

Re: Architecturally lost!?


Hey Keith,

thank you for your prompt reply! I am currently experimenting with the
devver examples, looks promising so far.

> If you have a process that only knows how to handle, say, outgoing
> sms, it should get its own tube (aka queue).
>
> How are the incoming jobs created? Can you generate them all in the
> same format? If so, you could make one incoming tube, and have one
> type of worker that handles all incoming jobs, and puts appropriate
> outgoing jobs into all three outgoing tubes.

I will most probably have one worker per protocol, so that i can
handle e.g. incoming rest-calls from sms or twitter via the same
worker and xmpp with another.

> I have never heard of anyone outgrowing the performance of a single
> beanstalkd process on a single box. So, probably, the only reason you
> would want more than one is for high availability of the service -- if
> a box goes down, you can still submit jobs to the other one.

Thats sounds inviting, even if performance is not my main problem.

shares an/a few internal ruby queue(s)?
>
> I'd recommend beanstalkd, but maybe I'm biased. :) Seriously, though,
> this is just what beanstalkd was designed for. A textbook example.

(Continue reading)

headers

Keith Rarick | 9 Aug 2009 22:33

Re: Architecturally lost!?


On Sun, Aug 9, 2009 at 1:16 PM, Tom<qklxmf@...> wrote:
> Do you have any tutorial or examples on async_observer and its uses?
> It seems like a natural fit because I am on RoR or would you start
> with the plain beanstalk-client gem and later (when?) 'upgrade' to the
> async_observer? Or use it right away?

If you plan on using it eventually, it's easier if you start with
async_observer.

I don't know of any tutorials other than the very basic guide on
http://async-observer.rubyforge.org/ . If you get that much working,
the rest is pretty easy.

> Did I get it right, that it can make most method calls asynchronous,
> without any additional work on the backend (just a little config)?

That's the idea. :)

kr

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "beanstalk-talk" group.
To post to this group, send email to beanstalk-talk@...
To unsubscribe from this group, send email to beanstalk-talk+unsubscribe@...
For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en
-~----------~----~----~----~------~----~------~--~---

headers

Tom | 14 Aug 2009 16:37

Re: Architecturally lost!?


Just to let you know: I decided to got with beanstalk and am happy so
far. In case someone is curious my setup, it looks like this:

A handful of separate processes encapsulated by the very capable
daemon-kit (see github) connected to the rails mothership via
beanstalk. Tests work nice so far!

Thanks.

On Aug 9, 10:33 pm, Keith Rarick <k...@...> wrote:
> On Sun, Aug 9, 2009 at 1:16 PM, Tom<qkl...@...> wrote:
> > Do you have any tutorial or examples on async_observer and its uses?
> > It seems like a natural fit because I am on RoR or would you start
> > with the plain beanstalk-client gem and later (when?) 'upgrade' to the
> > async_observer? Or use it right away?
>
> If you plan on using it eventually, it's easier if you start with
> async_observer.
>
> I don't know of any tutorials other than the very basic guide onhttp://async-observer.rubyforge.org/.
If you get that much working,
> the rest is pretty easy.
>
> > Did I get it right, that it can make most method calls asynchronous,
> > without any additional work on the backend (just a little config)?
>
> That's the idea. :)
>
> kr

(Continue reading)