headers
Jon Brisbin | 1 Jun 2011 15:38
Favicon
Gravatar

Re: How to handle full exchange history delivery

I've been seriously considering adding this functionality to the Riak-backed custom exchange. It was something I thought I would add first time through but didn't.

You'd have to use a custom exchange. Not sure if that's an issue in your scenario or not.


Thanks!

Jon Brisbin
http//jbrisbin.com


From: "Simone Busoli" <Simone.Busoli-sKiPNUh8iG9BDgjK7y7TUQ@public.gmane.org>
To: rabbitmq-discuss <at> lists.rabbitmq.com
Sent: Wednesday, June 1, 2011 5:52:24 AM
Subject: [rabbitmq-discuss] How to handle full exchange history delivery

Hi,

 

I am trying to achieve a scenario in which messages are published from some data source into a RabbitMQ exchange and clients should be able to receive the full history of messages flown through that exchange, regardless of the exact moment in time when they connect.

 

I’ve been thinking to solve it by setting up a proxy which subscribes to the exchange where messages are published using its own queue, keeps an internal storage of the messages and accepts requests from clients.

When a request from a client arrives it just replays the full message history to the client by publishing the messages to the client queue (supplied in the ReplyTo field), then binds the client queue to the messages exchange, so that the client from this moment onwards starts receiving the “live” messages, and is pretty much unaware of what’s happened; it simply just receives the whole history.

 

Now I am under the impression that there’s a fundamental flaw in this reasoning, since the proxy cannot know when the message history has completed and he can switch the client to the live stream, because while it is replaying historical messages to the client other live messages might be arriving on the proxy’s queue, thus getting into an endless loop.

 

I’m probably missing something obvious, any advice about how others would approach this scenario is appreciated.

 

Simone

_________________________________________________________________________________________________________________
Questo messaggio e da intendersi esclusivamente ad uso del destinatario e puo contenere informazioni che sono di natura privilegiata, confidenziale
o non divulgabile secondo le leggi vigenti. Se il lettore del presente messaggio non e il destinatario designato, o il dipendente/agente responsabile
per la consegna del messaggio al destinatario designato, si informa che ogni disseminazione, distribuzione o copiatura di questa comunicazione e
strettamente proibita anche ai sensi del decreto legislativo 196/03 . Se avete ricevuto questo messaggio per errore, vi preghiamo di notificarcelo
immediatamente a mezzo e-mail di risposta e successivamente di procedere alla cancellazione di questa e-mail e relativi allegati dal vostro sistema.
_________________________________________________________________________________________________________________
This message is intended only for the use of the addressee and may contain information that is privileged, confidential and exempt from
disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the
message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly
prohibited. If you have received this e-mail in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from
your system.
_________________________________________________________________________________________________________________


_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

<div><div>I've been seriously considering adding this functionality to the Riak-backed custom exchange. It was something I thought I would add first time through but didn't.<br><br>You'd have to use a custom exchange. Not sure if that's an issue in your scenario or not.<br><span><br><span name="x"></span><br>Thanks!<br><br>Jon Brisbin<br>http//jbrisbin.com<br><span name="x"></span><br></span><br><blockquote>From: "Simone Busoli" &lt;Simone.Busoli@...&gt;<br>To: rabbitmq-discuss <at> lists.rabbitmq.com<br>Sent: Wednesday, June 1, 2011 5:52:24 AM<br>Subject: [rabbitmq-discuss] How to handle full exchange history delivery<br><br><div class="WordSection1">
<p class="MsoNormal">Hi,</p>
<p class="MsoNormal">&nbsp;</p>
<p class="MsoNormal"><span lang="EN-US">I am trying to achieve a scenario in which messages are published from some data source into a RabbitMQ exchange and clients should be able to receive the full history of messages flown through that exchange, regardless
 of the exact moment in time when they connect.</span></p>
<p class="MsoNormal"><span lang="EN-US">&nbsp;</span></p>
<p class="MsoNormal"><span lang="EN-US">I&rsquo;ve been thinking to solve it by setting up a proxy which subscribes to the exchange where messages are published using its own queue, keeps an internal storage of the messages and accepts requests from clients.
</span></p>
<p class="MsoNormal"><span lang="EN-US">When a request from a client arrives it just replays the full message history to the client by publishing the messages to the client queue (supplied in the ReplyTo field),
then binds the client queue to the messages exchange, so that the client from this moment onwards starts receiving the &ldquo;live&rdquo; messages, and is pretty much unaware of what&rsquo;s happened; it simply just receives the whole history.</span></p>
<p class="MsoNormal"><span lang="EN-US">&nbsp;</span></p>
<p class="MsoNormal"><span lang="EN-US">Now I am under the impression that there&rsquo;s a fundamental flaw in this reasoning, since the proxy cannot know when the message history has completed and he can switch the client to the live stream, because while it is
 replaying historical messages to the client other live messages might be arriving on the proxy&rsquo;s queue, thus getting into an endless loop.</span></p>
<p class="MsoNormal"><span lang="EN-US">&nbsp;</span></p>
<p class="MsoNormal"><span lang="EN-US">I&rsquo;m probably missing something obvious, any advice about how others would approach this scenario is appreciated.</span></p>
<p class="MsoNormal"><span lang="EN-US">&nbsp;</span></p>
<p class="MsoNormal"><span lang="EN-US">Simone</span></p>
</div>
<p>_________________________________________________________________________________________________________________<br>Questo messaggio e da intendersi esclusivamente ad uso del destinatario e puo contenere informazioni che sono di natura privilegiata, confidenziale<br>o non divulgabile secondo le leggi vigenti. Se il lettore del presente messaggio non e il destinatario designato, o il dipendente/agente responsabile<br>per la consegna del messaggio al destinatario designato, si informa che ogni disseminazione, distribuzione o copiatura di questa comunicazione e <br>strettamente proibita anche ai sensi del decreto legislativo 196/03 . Se avete ricevuto questo messaggio per errore, vi preghiamo di notificarcelo<br>immediatamente a mezzo e-mail di risposta e successivamente di procedere alla cancellazione di questa e-mail e relativi allegati dal vostro sistema.<br>_________________________________________________________________________________________________________________<br>This message is intended only for the use of the addressee and may contain information that is privileged, confidential and exempt from <br>disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the <br>message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly <br>prohibited. If you have received this e-mail in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from <br>your system.<br>_________________________________________________________________________________________________________________<br></p>
<br>_______________________________________________<br>rabbitmq-discuss mailing list<br>rabbitmq-discuss@...<br>https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss<br>
</blockquote>
<br>
</div></div>
Permalink | Reply |
headers
Busoli, Simone | 1 Jun 2011 15:43
Favicon

Re: How to handle full exchange history delivery

That’s what I thought as well, although I’m thinking about a couple of issues about it:

 

·         Lack of knowledge about Erlang, which is the language I’d have to use to build a custom exchange, I guess

·         Amount of memory it would require to store the entire history in memory. It needn’t be in memory actually, but I’m not sure what would be the recommended way if the storage has to be carried out by exchange itself. Mnesia perhaps, would it make sense?

_______________________________________________________________

Simone Busoli
Direzione risorse umane e segreteria generale – Sistemi informativi gestione sportiva
Collaboratore

Ferrari S.p.A.     Via Ascari 55/57, 41053, Maranello (MO); Italy
T (+39) 0536-949.850
E simone.busoli-sKiPNUh8iG9BDgjK7y7TUQ@public.gmane.org  www.ferrari.com

 

From: Jon Brisbin [mailto:jon-foCUeXAHXPNWk0Htik3J/w@public.gmane.org]
Sent: Wednesday, June 01, 2011 3:38 PM
To: Busoli, Simone
Cc: rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org
Subject: Re: [rabbitmq-discuss] How to handle full exchange history delivery

 

I've been seriously considering adding this functionality to the Riak-backed custom exchange. It was something I thought I would add first time through but didn't.

You'd have to use a custom exchange. Not sure if that's an issue in your scenario or not.


Thanks!

Jon Brisbin
http//jbrisbin.com

From: "Simone Busoli" <Simone.Busoli-sKiPNUh8iG9BDgjK7y7TUQ@public.gmane.org>
To: rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org
Sent: Wednesday, June 1, 2011 5:52:24 AM
Subject: [rabbitmq-discuss] How to handle full exchange history delivery


Hi,

 

I am trying to achieve a scenario in which messages are published from some data source into a RabbitMQ exchange and clients should be able to receive the full history of messages flown through that exchange, regardless of the exact moment in time when they connect.

 

I’ve been thinking to solve it by setting up a proxy which subscribes to the exchange where messages are published using its own queue, keeps an internal storage of the messages and accepts requests from clients.

When a request from a client arrives it just replays the full message history to the client by publishing the messages to the client queue (supplied in the ReplyTo field), then binds the client queue to the messages exchange, so that the client from this moment onwards starts receiving the “live” messages, and is pretty much unaware of what’s happened; it simply just receives the whole history.

 

Now I am under the impression that there’s a fundamental flaw in this reasoning, since the proxy cannot know when the message history has completed and he can switch the client to the live stream, because while it is replaying historical messages to the client other live messages might be arriving on the proxy’s queue, thus getting into an endless loop.

 

I’m probably missing something obvious, any advice about how others would approach this scenario is appreciated.

 

Simone

_________________________________________________________________________________________________________________
Questo messaggio e da intendersi esclusivamente ad uso del destinatario e puo contenere informazioni che sono di natura privilegiata, confidenziale
o non divulgabile secondo le leggi vigenti. Se il lettore del presente messaggio non e il destinatario designato, o il dipendente/agente responsabile
per la consegna del messaggio al destinatario designato, si informa che ogni disseminazione, distribuzione o copiatura di questa comunicazione e
strettamente proibita anche ai sensi del decreto legislativo 196/03 . Se avete ricevuto questo messaggio per errore, vi preghiamo di notificarcelo
immediatamente a mezzo e-mail di risposta e successivamente di procedere alla cancellazione di questa e-mail e relativi allegati dal vostro sistema.
_________________________________________________________________________________________________________________
This message is intended only for the use of the addressee and may contain information that is privileged, confidential and exempt from
disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the
message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly
prohibited. If you have received this e-mail in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from
your system.
_________________________________________________________________________________________________________________


_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

 

_________________________________________________________________________________________________________________
Questo messaggio è da intendersi esclusivamente ad uso del destinatario e può contenere informazioni che sono di natura privilegiata, confidenziale
o non divulgabile secondo le leggi vigenti. Se il lettore del presente messaggio non è il destinatario designato, o il dipendente/agente responsabile
per la consegna del messaggio al destinatario designato, si informa che ogni disseminazione, distribuzione o copiatura di questa comunicazione è
strettamente proibita anche ai sensi del decreto legislativo 196/03 . Se avete ricevuto questo messaggio per errore, vi preghiamo di notificarcelo
immediatamente a mezzo e-mail di risposta e successivamente di procedere alla cancellazione di questa e-mail e relativi allegati dal vostro sistema.
_________________________________________________________________________________________________________________
This message is intended only for the use of the addressee and may contain information that is privileged, confidential and exempt from
disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the
message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly
prohibited. If you have received this e-mail in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from
your system.
_________________________________________________________________________________________________________________

<div>
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US">That&rsquo;s what I thought as well, although I&rsquo;m thinking about a couple of issues about it:<p></p></span></p>
<p class="MsoNormal"><span lang="EN-US"><p>&nbsp;</p></span></p>
<p class="MsoListParagraph"><span lang="EN-US"><span>&middot;<span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
</span></span></span><span lang="EN-US">Lack of knowledge about Erlang, which is the language I&rsquo;d have to use to build a custom exchange, I guess<p></p></span></p>
<p class="MsoListParagraph"><span lang="EN-US"><span>&middot;<span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
</span></span></span><span lang="EN-US">Amount of memory it would require to store the entire history in memory. It needn&rsquo;t be in memory actually, but I&rsquo;m not sure what would be the recommended way if the storage has to be carried
 out by exchange itself. Mnesia perhaps, would it make sense?<p></p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">_______________________________________________________</span><span>________</span><span>
<p></p></span></p>
<p class="MsoNormal"><span>Simone Busoli</span><span><br></span><span>Direzione risorse umane e segreteria generale</span><span>
</span><span>&ndash; Sistemi informativi gestione sportiva
<br>Collaboratore</span><span><p></p></span></p>
<p class="MsoNormal"><span>Ferrari S.p.A.&nbsp;&nbsp;&nbsp;&nbsp;</span><span>
</span><span>Via Ascari 55/57, 41053, Maranello (MO); Italy</span><span><br></span><span lang="DE">T</span><span>&nbsp;(+39) 0536-949.850</span><span><br></span><span>E</span><span>
<a href="mailto:simone.busoli@..."><span>simone.busoli@...</span></a>&nbsp;
<a href="http://www.ferrari.com/"><span>www.ferrari.com</span></a></span><span>
<p></p></span></p>
</div>
<p class="MsoNormal"><span><p>&nbsp;</p></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">From:</span><span lang="EN-US"> Jon Brisbin
 [mailto:jon@...] <br>Sent: Wednesday, June 01, 2011 3:38 PM<br>To: Busoli, Simone<br>Cc: rabbitmq-discuss@...<br>Subject: Re: [rabbitmq-discuss] How to handle full exchange history delivery<p></p></span></p>
</div>
</div>
<p class="MsoNormal"><p>&nbsp;</p></p>
<div>
<p class="MsoNormal"><span>I've been seriously considering adding this functionality to the Riak-backed custom exchange. It was something I thought I would add first time through but didn't.<br><br>
You'd have to use a custom exchange. Not sure if that's an issue in your scenario or not.<br><br><br>
Thanks!<br><br>
Jon Brisbin<br>
http//jbrisbin.com<br><br><p></p></span></p>
<div class="MsoNormal" align="center"><span>
</span></div>
<blockquote>
<p class="MsoNormal"><span>From:
</span><span>"Simone Busoli" &lt;Simone.Busoli@...&gt;<br>To: rabbitmq-discuss@...<br>Sent: Wednesday, June 1, 2011 5:52:24 AM<br>Subject: [rabbitmq-discuss] How to handle full exchange history delivery<br><br><br></span><span><p></p></span></p>
<p class="MsoNormal"><span>Hi,<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;<p></p></span></p>
<p class="MsoNormal"><span lang="EN-US">I am trying to achieve a scenario in which messages are published from some data source into a RabbitMQ exchange and clients should be able to receive the full history of messages flown through that
 exchange, regardless of the exact moment in time when they connect.</span><span><p></p></span></p>
<p class="MsoNormal"><span lang="EN-US">&nbsp;</span><span><p></p></span></p>
<p class="MsoNormal"><span lang="EN-US">I&rsquo;ve been thinking to solve it by setting up a proxy which subscribes to the exchange where messages are published using its own queue, keeps an internal storage of the messages and accepts requests
 from clients. </span><span><p></p></span></p>
<p class="MsoNormal"><span lang="EN-US">When a request from a client arrives it just replays the full message history to the client by publishing the messages to the client queue (supplied in the ReplyTo field),
then binds the client queue to the messages exchange, so that the client from this moment onwards starts receiving the &ldquo;live&rdquo; messages, and is pretty much unaware of what&rsquo;s happened; it simply just receives the whole history.</span><span><p></p></span></p>
<p class="MsoNormal"><span lang="EN-US">&nbsp;</span><span><p></p></span></p>
<p class="MsoNormal"><span lang="EN-US">Now I am under the impression that there&rsquo;s a fundamental flaw in this reasoning, since the proxy cannot know when the message history has completed and he can switch the client to the live stream,
 because while it is replaying historical messages to the client other live messages might be arriving on the proxy&rsquo;s queue, thus getting into an endless loop.</span><span><p></p></span></p>
<p class="MsoNormal"><span lang="EN-US">&nbsp;</span><span><p></p></span></p>
<p class="MsoNormal"><span lang="EN-US">I&rsquo;m probably missing something obvious, any advice about how others would approach this scenario is appreciated.</span><span><p></p></span></p>
<p class="MsoNormal"><span lang="EN-US">&nbsp;</span><span><p></p></span></p>
<p class="MsoNormal"><span lang="EN-US">Simone</span><span><p></p></span></p>
<p><span>_________________________________________________________________________________________________________________<br>
Questo messaggio e da intendersi esclusivamente ad uso del destinatario e puo contenere informazioni che sono di natura privilegiata, confidenziale<br>
o non divulgabile secondo le leggi vigenti. Se il lettore del presente messaggio non e il destinatario designato, o il dipendente/agente responsabile<br>
per la consegna del messaggio al destinatario designato, si informa che ogni disseminazione, distribuzione o copiatura di questa comunicazione e
<br>
strettamente proibita anche ai sensi del decreto legislativo 196/03 . Se avete ricevuto questo messaggio per errore, vi preghiamo di notificarcelo<br>
immediatamente a mezzo e-mail di risposta e successivamente di procedere alla cancellazione di questa e-mail e relativi allegati dal vostro sistema.<br>
_________________________________________________________________________________________________________________<br>
This message is intended only for the use of the addressee and may contain information that is privileged, confidential and exempt from
<br>
disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the
<br>
message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly
<br>
prohibited. If you have received this e-mail in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from
<br>
your system.<br>
_________________________________________________________________________________________________________________</span><span><p></p></span></p>
<p class="MsoNormal"><span><br></span><span lang="EN-US">_______________________________________________<br>
rabbitmq-discuss mailing list<br></span><span><a href="mailto:rabbitmq-discuss@..."><span lang="EN-US">rabbitmq-discuss@...</span></a></span><span lang="EN-US"><br></span><span><a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss"><span lang="EN-US">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</span></a></span><span lang="EN-US"><p></p></span></p>
</blockquote>
<p class="MsoNormal"><span lang="EN-US"><p>&nbsp;</p></span></p>
</div>
</div>
<p align="left">_________________________________________________________________________________________________________________<br>Questo messaggio &egrave; da intendersi esclusivamente ad uso del destinatario e pu&ograve; contenere informazioni che sono di natura privilegiata, confidenziale<br>o non divulgabile secondo le leggi vigenti. Se il lettore del presente messaggio non &egrave; il destinatario designato, o il dipendente/agente responsabile<br>per la consegna del messaggio al destinatario designato, si informa che ogni disseminazione, distribuzione o copiatura di questa comunicazione &egrave; <br>strettamente proibita anche ai sensi del decreto legislativo 196/03 . Se avete ricevuto questo messaggio per errore, vi preghiamo di notificarcelo<br>immediatamente a mezzo e-mail di risposta e successivamente di procedere alla cancellazione di questa e-mail e relativi allegati dal vostro sistema.<br>_________________________________________________________________________________________________________________<br>This message is intended only for the use of the addressee and may contain information that is privileged, confidential and exempt from <br>disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the <br>message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly <br>prohibited. If you have received this e-mail in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from <br>your system.<br>_________________________________________________________________________________________________________________<br></p>
</div>
Permalink | Reply |
headers
Jon Brisbin | 1 Jun 2011 16:58
Favicon
Gravatar

Re: How to handle full exchange history delivery

Maybe I wasn't being clear:

I've already written a custom exchange that stores every message it receives in Riak (http://wiki.basho.com/). I'm going to be talking about it next week at Erlang Factory in London, as a matter of fact. :)

https://github.com/jbrisbin/riak-exchange

When I was writing it, I thought it would be really easy to, when a new queue is bound to this exchange, go fetch all the messages in that bucket and send them to the queue.

Since I'm using Riak, the messages themselves are not all stored in memory.

Thanks!

Jon Brisbin
http//jbrisbin.com


From: "Simone Busoli" <Simone.Busoli-sKiPNUh8iG9BDgjK7y7TUQ@public.gmane.org>
To: rabbitmq-discuss-ETbvJ2rUIr7IiUczSzxo9w@public.gmane.orgmq.com
Sent: Wednesday, June 1, 2011 8:43:58 AM
Subject: Re: [rabbitmq-discuss] How to handle full exchange history delivery

That’s what I thought as well, although I’m thinking about a couple of issues about it:

 

·         Lack of knowledge about Erlang, which is the language I’d have to use to build a custom exchange, I guess

·         Amount of memory it would require to store the entire history in memory. It needn’t be in memory actually, but I’m not sure what would be the recommended way if the storage has to be carried out by exchange itself. Mnesia perhaps, would it make sense?

_______________________________________________________________

Simone Busoli
Direzione risorse umane e segreteria generale – Sistemi informativi gestione sportiva
Collaboratore

Ferrari S.p.A.     Via Ascari 55/57, 41053, Maranello (MO); Italy
T (+39) 0536-949.850
E simone.busoli-sKiPNUh8iG9BDgjK7y7TUQ@public.gmane.org  www.ferrari.com

 

From: Jon Brisbin [mailto:jon-foCUeXAHXPNWk0Htik3J/w@public.gmane.org]
Sent: Wednesday, June 01, 2011 3:38 PM
To: Busoli, Simone
Cc: rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org
Subject: Re: [rabbitmq-discuss] How to handle full exchange history delivery

 

I've been seriously considering adding this functionality to the Riak-backed custom exchange. It was something I thought I would add first time through but didn't.

You'd have to use a custom exchange. Not sure if that's an issue in your scenario or not.


Thanks!

Jon Brisbin
http//jbrisbin.com

From: "Simone Busoli" <Simone.Busoli-sKiPNUh8iG9BDgjK7y7TUQ@public.gmane.org>
To: rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org
Sent: Wednesday, June 1, 2011 5:52:24 AM
Subject: [rabbitmq-discuss] How to handle full exchange history delivery


Hi,

 

I am trying to achieve a scenario in which messages are published from some data source into a RabbitMQ exchange and clients should be able to receive the full history of messages flown through that exchange, regardless of the exact moment in time when they connect.

 

I’ve been thinking to solve it by setting up a proxy which subscribes to the exchange where messages are published using its own queue, keeps an internal storage of the messages and accepts requests from clients.

When a request from a client arrives it just replays the full message history to the client by publishing the messages to the client queue (supplied in the ReplyTo field), then binds the client queue to the messages exchange, so that the client from this moment onwards starts receiving the “live” messages, and is pretty much unaware of what’s happened; it simply just receives the whole history.

 

Now I am under the impression that there’s a fundamental flaw in this reasoning, since the proxy cannot know when the message history has completed and he can switch the client to the live stream, because while it is replaying historical messages to the client other live messages might be arriving on the proxy’s queue, thus getting into an endless loop.

 

I’m probably missing something obvious, any advice about how others would approach this scenario is appreciated.

 

Simone

_________________________________________________________________________________________________________________
Questo messaggio e da intendersi esclusivamente ad uso del destinatario e puo contenere informazioni che sono di natura privilegiata, confidenziale
o non divulgabile secondo le leggi vigenti. Se il lettore del presente messaggio non e il destinatario designato, o il dipendente/agente responsabile
per la consegna del messaggio al destinatario designato, si informa che ogni disseminazione, distribuzione o copiatura di questa comunicazione e
strettamente proibita anche ai sensi del decreto legislativo 196/03 . Se avete ricevuto questo messaggio per errore, vi preghiamo di notificarcelo
immediatamente a mezzo e-mail di risposta e successivamente di procedere alla cancellazione di questa e-mail e relativi allegati dal vostro sistema.
_________________________________________________________________________________________________________________
This message is intended only for the use of the addressee and may contain information that is privileged, confidential and exempt from
disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the
message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly
prohibited. If you have received this e-mail in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from
your system.
_________________________________________________________________________________________________________________


_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss <at> lists.rabbitmq.com
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

 

_________________________________________________________________________________________________________________
Questo messaggio è da intendersi esclusivamente ad uso del destinatario e può contenere informazioni che sono di natura privilegiata, confidenziale
o non divulgabile secondo le leggi vigenti. Se il lettore del presente messaggio non è il destinatario designato, o il dipendente/agente responsabile
per la consegna del messaggio al destinatario designato, si informa che ogni disseminazione, distribuzione o copiatura di questa comunicazione è
strettamente proibita anche ai sensi del decreto legislativo 196/03 . Se avete ricevuto questo messaggio per errore, vi preghiamo di notificarcelo
immediatamente a mezzo e-mail di risposta e successivamente di procedere alla cancellazione di questa e-mail e relativi allegati dal vostro sistema.
_________________________________________________________________________________________________________________
This message is intended only for the use of the addressee and may contain information that is privileged, confidential and exempt from
disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the
message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly
prohibited. If you have received this e-mail in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from
your system.
_________________________________________________________________________________________________________________


_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

<div><div>Maybe I wasn't being clear:<br><br>I've already written a custom exchange that stores every message it receives in Riak (http://wiki.basho.com/). I'm going to be talking about it next week at Erlang Factory in London, as a matter of fact. :)<br><br>https://github.com/jbrisbin/riak-exchange<br><span><br>When I was writing it, I thought it would be really easy to, when a new queue is bound to this exchange, go fetch all the messages in that bucket and send them to the queue.<br><br>Since I'm using Riak, the messages themselves are not all stored in memory.<br><span name="x"></span><br>Thanks!<br><br>Jon Brisbin<br>http//jbrisbin.com<br><span name="x"></span><br></span><br><blockquote>From: "Simone Busoli" &lt;Simone.Busoli@...&gt;<br>To: rabbitmq-discuss@...mq.com<br>Sent: Wednesday, June 1, 2011 8:43:58 AM<br>Subject: Re: [rabbitmq-discuss] How to handle full exchange history delivery<br><br><div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US">That&rsquo;s what I thought as well, although I&rsquo;m thinking about a couple of issues about it:</span></p>
<p class="MsoNormal"><span lang="EN-US">&nbsp;</span></p>
<p class="MsoListParagraph"><span lang="EN-US"><span>&middot;<span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
</span></span></span><span lang="EN-US">Lack of knowledge about Erlang, which is the language I&rsquo;d have to use to build a custom exchange, I guess</span></p>
<p class="MsoListParagraph"><span lang="EN-US"><span>&middot;<span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
</span></span></span><span lang="EN-US">Amount of memory it would require to store the entire history in memory. It needn&rsquo;t be in memory actually, but I&rsquo;m not sure what would be the recommended way if the storage has to be carried
 out by exchange itself. Mnesia perhaps, would it make sense?</span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">_______________________________________________________</span><span>________</span><span>
</span></p>
<p class="MsoNormal"><span>Simone Busoli</span><span><br></span><span>Direzione risorse umane e segreteria generale</span><span>
</span><span>&ndash; Sistemi informativi gestione sportiva
<br>Collaboratore</span><span></span></p>
<p class="MsoNormal"><span>Ferrari S.p.A.&nbsp;&nbsp;&nbsp;&nbsp;</span><span>
</span><span>Via Ascari 55/57, 41053, Maranello (MO); Italy</span><span><br></span><span lang="DE">T</span><span>&nbsp;(+39) 0536-949.850</span><span><br></span><span>E</span><span>
<a href="mailto:simone.busoli@..." target="_blank"><span>simone.busoli@...</span></a>&nbsp;
<a href="http://www.ferrari.com/" target="_blank"><span>www.ferrari.com</span></a></span><span>
</span></p>
</div>
<p class="MsoNormal"><span>&nbsp;</span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">From:</span><span lang="EN-US"> Jon Brisbin
 [mailto:jon@...] <br>Sent: Wednesday, June 01, 2011 3:38 PM<br>To: Busoli, Simone<br>Cc: rabbitmq-discuss@...<br>Subject: Re: [rabbitmq-discuss] How to handle full exchange history delivery</span></p>
</div>
</div>
<p class="MsoNormal">&nbsp;</p>
<div>
<p class="MsoNormal"><span>I've been seriously considering adding this functionality to the Riak-backed custom exchange. It was something I thought I would add first time through but didn't.<br><br>
You'd have to use a custom exchange. Not sure if that's an issue in your scenario or not.<br><br><br>
Thanks!<br><br>
Jon Brisbin<br>
http//jbrisbin.com<br><br></span></p>
<div class="MsoNormal" align="center"><span>
</span></div>
<blockquote>
<p class="MsoNormal"><span>From:
</span><span>"Simone Busoli" &lt;Simone.Busoli@...&gt;<br>To: rabbitmq-discuss@...<br>Sent: Wednesday, June 1, 2011 5:52:24 AM<br>Subject: [rabbitmq-discuss] How to handle full exchange history delivery<br><br><br></span><span></span></p>
<p class="MsoNormal"><span>Hi,</span></p>
<p class="MsoNormal"><span>&nbsp;</span></p>
<p class="MsoNormal"><span lang="EN-US">I am trying to achieve a scenario in which messages are published from some data source into a RabbitMQ exchange and clients should be able to receive the full history of messages flown through that
 exchange, regardless of the exact moment in time when they connect.</span><span></span></p>
<p class="MsoNormal"><span lang="EN-US">&nbsp;</span><span></span></p>
<p class="MsoNormal"><span lang="EN-US">I&rsquo;ve been thinking to solve it by setting up a proxy which subscribes to the exchange where messages are published using its own queue, keeps an internal storage of the messages and accepts requests
 from clients. </span><span></span></p>
<p class="MsoNormal"><span lang="EN-US">When a request from a client arrives it just replays the full message history to the client by publishing the messages to the client queue (supplied in the ReplyTo field),
then binds the client queue to the messages exchange, so that the client from this moment onwards starts receiving the &ldquo;live&rdquo; messages, and is pretty much unaware of what&rsquo;s happened; it simply just receives the whole history.</span><span></span></p>
<p class="MsoNormal"><span lang="EN-US">&nbsp;</span><span></span></p>
<p class="MsoNormal"><span lang="EN-US">Now I am under the impression that there&rsquo;s a fundamental flaw in this reasoning, since the proxy cannot know when the message history has completed and he can switch the client to the live stream,
 because while it is replaying historical messages to the client other live messages might be arriving on the proxy&rsquo;s queue, thus getting into an endless loop.</span><span></span></p>
<p class="MsoNormal"><span lang="EN-US">&nbsp;</span><span></span></p>
<p class="MsoNormal"><span lang="EN-US">I&rsquo;m probably missing something obvious, any advice about how others would approach this scenario is appreciated.</span><span></span></p>
<p class="MsoNormal"><span lang="EN-US">&nbsp;</span><span></span></p>
<p class="MsoNormal"><span lang="EN-US">Simone</span><span></span></p>
<p><span>_________________________________________________________________________________________________________________<br>
Questo messaggio e da intendersi esclusivamente ad uso del destinatario e puo contenere informazioni che sono di natura privilegiata, confidenziale<br>
o non divulgabile secondo le leggi vigenti. Se il lettore del presente messaggio non e il destinatario designato, o il dipendente/agente responsabile<br>
per la consegna del messaggio al destinatario designato, si informa che ogni disseminazione, distribuzione o copiatura di questa comunicazione e
<br>
strettamente proibita anche ai sensi del decreto legislativo 196/03 . Se avete ricevuto questo messaggio per errore, vi preghiamo di notificarcelo<br>
immediatamente a mezzo e-mail di risposta e successivamente di procedere alla cancellazione di questa e-mail e relativi allegati dal vostro sistema.<br>
_________________________________________________________________________________________________________________<br>
This message is intended only for the use of the addressee and may contain information that is privileged, confidential and exempt from
<br>
disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the
<br>
message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly
<br>
prohibited. If you have received this e-mail in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from
<br>
your system.<br>
_________________________________________________________________________________________________________________</span><span></span></p>
<p class="MsoNormal"><span><br></span><span lang="EN-US">_______________________________________________<br>
rabbitmq-discuss mailing list<br></span><span><a href="mailto:rabbitmq-discuss <at> lists.rabbitmq.com" target="_blank"><span lang="EN-US">rabbitmq-discuss <at> lists.rabbitmq.com</span></a></span><span lang="EN-US"><br></span><span><a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank"><span lang="EN-US">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</span></a></span><span lang="EN-US"></span></p>
</blockquote>
<p class="MsoNormal"><span lang="EN-US">&nbsp;</span></p>
</div>
</div>
<p align="left">_________________________________________________________________________________________________________________<br>Questo messaggio &egrave; da intendersi esclusivamente ad uso del destinatario e pu&ograve; contenere informazioni che sono di natura privilegiata, confidenziale<br>o non divulgabile secondo le leggi vigenti. Se il lettore del presente messaggio non &egrave; il destinatario designato, o il dipendente/agente responsabile<br>per la consegna del messaggio al destinatario designato, si informa che ogni disseminazione, distribuzione o copiatura di questa comunicazione &egrave; <br>strettamente proibita anche ai sensi del decreto legislativo 196/03 . Se avete ricevuto questo messaggio per errore, vi preghiamo di notificarcelo<br>immediatamente a mezzo e-mail di risposta e successivamente di procedere alla cancellazione di questa e-mail e relativi allegati dal vostro sistema.<br>_________________________________________________________________________________________________________________<br>This message is intended only for the use of the addressee and may contain information that is privileged, confidential and exempt from <br>disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the <br>message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly <br>prohibited. If you have received this e-mail in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from <br>your system.<br>_________________________________________________________________________________________________________________<br></p>
<br>_______________________________________________<br>rabbitmq-discuss mailing list<br>rabbitmq-discuss@...<br>https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss<br>
</blockquote>
<br>
</div></div>
Permalink | Reply |
headers
Busoli, Simone | 1 Jun 2011 17:15
Favicon

Re: How to handle full exchange history delivery

Thanks Jon, I misunderstood you probably.

 

WRT this:

 

When I was writing it, I thought it would be really easy to, when a new queue is bound to this exchange, go fetch all the messages in that bucket and send them to the queue.

 

I’m not sure it is, that was my main question actually. How do you figure out how to join the messages stored in the bucket and those arriving live so that the last message you send to the queue from those picked up from the bucket is followed exactly by its logical successor from those coming live?

 

Say I have messages 1..100 being published to the exchange, each time one arrives it’s stored. The exchange has received the first 50 messages and a client performs a binding of a queue to the exchange. So you start pulling the messages from the storage and sending them to the queue before forwarding any live messages. While you are pulling them out of the storage messages 51..60 are published to the exchange, how do you deal with them? Do you always pull from the storage? When do you stop pulling from the storage and relay to the queue directly what is being published to the exchange if you don’t know whether any new messages have arrived while you were pulling from the storage?

_______________________________________________________________

Simone Busoli
Direzione risorse umane e segreteria generale – Sistemi informativi gestione sportiva
Collaboratore

Ferrari S.p.A.     Via Ascari 55/57, 41053, Maranello (MO); Italy
T (+39) 0536-949.850
E simone.busoli-sKiPNUh8iG9BDgjK7y7TUQ@public.gmane.org  www.ferrari.com

 

From: Jon Brisbin [mailto:jon-foCUeXAHXPNWk0Htik3J/w@public.gmane.org]
Sent: Wednesday, June 01, 2011 4:59 PM
To: Busoli, Simone
Cc: rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org
Subject: Re: [rabbitmq-discuss] How to handle full exchange history delivery

 

Maybe I wasn't being clear:

I've already written a custom exchange that stores every message it receives in Riak (http://wiki.basho.com/). I'm going to be talking about it next week at Erlang Factory in London, as a matter of fact. :)

https://github.com/jbrisbin/riak-exchange

When I was writing it, I thought it would be really easy to, when a new queue is bound to this exchange, go fetch all the messages in that bucket and send them to the queue.

Since I'm using Riak, the messages themselves are not all stored in memory.

Thanks!

Jon Brisbin
http//jbrisbin.com

From: "Simone Busoli" <Simone.Busoli-sKiPNUh8iG9BDgjK7y7TUQ@public.gmane.org>
To: rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org
Sent: Wednesday, June 1, 2011 8:43:58 AM
Subject: Re: [rabbitmq-discuss] How to handle full exchange history delivery


That’s what I thought as well, although I’m thinking about a couple of issues about it:

 

·         Lack of knowledge about Erlang, which is the language I’d have to use to build a custom exchange, I guess

·         Amount of memory it would require to store the entire history in memory. It needn’t be in memory actually, but I’m not sure what would be the recommended way if the storage has to be carried out by exchange itself. Mnesia perhaps, would it make sense?

_______________________________________________________________

Simone Busoli
Direzione risorse umane e segreteria generale – Sistemi informativi gestione sportiva
Collaboratore

Ferrari S.p.A.     Via Ascari 55/57, 41053, Maranello (MO); Italy
T (+39) 0536-949.850
E simone.busoli-sKiPNUh8iG9BDgjK7y7TUQ@public.gmane.org  www.ferrari.com

 

From: Jon Brisbin [mailto:jon-foCUeXAHXPNWk0Htik3J/w@public.gmane.org]
Sent: Wednesday, June 01, 2011 3:38 PM
To: Busoli, Simone
Cc: rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org
Subject: Re: [rabbitmq-discuss] How to handle full exchange history delivery

 

I've been seriously considering adding this functionality to the Riak-backed custom exchange. It was something I thought I would add first time through but didn't.

You'd have to use a custom exchange. Not sure if that's an issue in your scenario or not.


Thanks!

Jon Brisbin
http//jbrisbin.com

From: "Simone Busoli" <Simone.Busoli-sKiPNUh8iG9BDgjK7y7TUQ@public.gmane.org>
To: rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org
Sent: Wednesday, June 1, 2011 5:52:24 AM
Subject: [rabbitmq-discuss] How to handle full exchange history delivery

Hi,

 

I am trying to achieve a scenario in which messages are published from some data source into a RabbitMQ exchange and clients should be able to receive the full history of messages flown through that exchange, regardless of the exact moment in time when they connect.

 

I’ve been thinking to solve it by setting up a proxy which subscribes to the exchange where messages are published using its own queue, keeps an internal storage of the messages and accepts requests from clients.

When a request from a client arrives it just replays the full message history to the client by publishing the messages to the client queue (supplied in the ReplyTo field), then binds the client queue to the messages exchange, so that the client from this moment onwards starts receiving the “live” messages, and is pretty much unaware of what’s happened; it simply just receives the whole history.

 

Now I am under the impression that there’s a fundamental flaw in this reasoning, since the proxy cannot know when the message history has completed and he can switch the client to the live stream, because while it is replaying historical messages to the client other live messages might be arriving on the proxy’s queue, thus getting into an endless loop.

 

I’m probably missing something obvious, any advice about how others would approach this scenario is appreciated.

 

Simone

_________________________________________________________________________________________________________________
Questo messaggio e da intendersi esclusivamente ad uso del destinatario e puo contenere informazioni che sono di natura privilegiata, confidenziale
o non divulgabile secondo le leggi vigenti. Se il lettore del presente messaggio non e il destinatario designato, o il dipendente/agente responsabile
per la consegna del messaggio al destinatario designato, si informa che ogni disseminazione, distribuzione o copiatura di questa comunicazione e
strettamente proibita anche ai sensi del decreto legislativo 196/03 . Se avete ricevuto questo messaggio per errore, vi preghiamo di notificarcelo
immediatamente a mezzo e-mail di risposta e successivamente di procedere alla cancellazione di questa e-mail e relativi allegati dal vostro sistema.
_________________________________________________________________________________________________________________
This message is intended only for the use of the addressee and may contain information that is privileged, confidential and exempt from
disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the
message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly
prohibited. If you have received this e-mail in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from
your system.
_________________________________________________________________________________________________________________


_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

 

_________________________________________________________________________________________________________________
Questo messaggio è da intendersi esclusivamente ad uso del destinatario e può contenere informazioni che sono di natura privilegiata, confidenziale
o non divulgabile secondo le leggi vigenti. Se il lettore del presente messaggio non è il destinatario designato, o il dipendente/agente responsabile
per la consegna del messaggio al destinatario designato, si informa che ogni disseminazione, distribuzione o copiatura di questa comunicazione è
strettamente proibita anche ai sensi del decreto legislativo 196/03 . Se avete ricevuto questo messaggio per errore, vi preghiamo di notificarcelo
immediatamente a mezzo e-mail di risposta e successivamente di procedere alla cancellazione di questa e-mail e relativi allegati dal vostro sistema.
_________________________________________________________________________________________________________________
This message is intended only for the use of the addressee and may contain information that is privileged, confidential and exempt from
disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the
message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly
prohibited. If you have received this e-mail in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from
your system.
_________________________________________________________________________________________________________________


_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

 

_________________________________________________________________________________________________________________
Questo messaggio è da intendersi esclusivamente ad uso del destinatario e può contenere informazioni che sono di natura privilegiata, confidenziale
o non divulgabile secondo le leggi vigenti. Se il lettore del presente messaggio non è il destinatario designato, o il dipendente/agente responsabile
per la consegna del messaggio al destinatario designato, si informa che ogni disseminazione, distribuzione o copiatura di questa comunicazione è
strettamente proibita anche ai sensi del decreto legislativo 196/03 . Se avete ricevuto questo messaggio per errore, vi preghiamo di notificarcelo
immediatamente a mezzo e-mail di risposta e successivamente di procedere alla cancellazione di questa e-mail e relativi allegati dal vostro sistema.
_________________________________________________________________________________________________________________
This message is intended only for the use of the addressee and may contain information that is privileged, confidential and exempt from
disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the
message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly
prohibited. If you have received this e-mail in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from
your system.
_________________________________________________________________________________________________________________

<div>
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US">Thanks Jon, I misunderstood you probably.<p></p></span></p>
<p class="MsoNormal"><span lang="EN-US"><p>&nbsp;</p></span></p>
<p class="MsoNormal"><span lang="EN-US">WRT this:<p></p></span></p>
<p class="MsoNormal"><span lang="EN-US"><p>&nbsp;</p></span></p>
<p class="MsoNormal"><span lang="EN-US">When I was writing it, I thought it would be really easy to, when a new queue is bound to this exchange, go fetch all the messages in that bucket and send them to the queue.</span><span lang="EN-US"><p></p></span></p>
<p class="MsoNormal"><span lang="EN-US"><p>&nbsp;</p></span></p>
<p class="MsoNormal"><span lang="EN-US">I&rsquo;m not sure it is, that was my main question actually. How do you figure out how to join the messages stored in the bucket and those arriving live so that the last message you send to the queue
 from those picked up from the bucket is followed exactly by its logical successor from those coming live?<p></p></span></p>
<p class="MsoNormal"><span lang="EN-US"><p>&nbsp;</p></span></p>
<p class="MsoNormal"><span lang="EN-US">Say I have messages 1..100 being published to the exchange, each time one arrives it&rsquo;s stored. The exchange has received the first 50 messages and a client performs a binding of a queue to the exchange.
 So you start pulling the messages from the storage and sending them to the queue before forwarding any live messages. While you are pulling them out of the storage messages 51..60 are published to the exchange, how do you deal with them? Do you always pull
 from the storage? When do you stop pulling from the storage and relay to the queue directly what is being published to the exchange if you don&rsquo;t know whether any new messages have arrived while you were pulling from the storage?<p></p></span></p>
<div>
<p class="MsoNormal"><span>_______________________________________________________________</span><span>
<p></p></span></p>
<p class="MsoNormal"><span>Simone Busoli</span><span><br></span><span>Direzione risorse umane e segreteria generale</span><span>
</span><span>&ndash; Sistemi informativi gestione sportiva
<br>Collaboratore</span><span><p></p></span></p>
<p class="MsoNormal"><span>Ferrari S.p.A.&nbsp;&nbsp;&nbsp;&nbsp;</span><span>
</span><span>Via Ascari 55/57, 41053, Maranello (MO); Italy</span><span><br></span><span lang="DE">T</span><span>&nbsp;(+39) 0536-949.850</span><span><br></span><span>E</span><span>
<a href="mailto:simone.busoli@..."><span>simone.busoli@...</span></a>&nbsp;
<a href="http://www.ferrari.com/"><span>www.ferrari.com</span></a></span><span>
<p></p></span></p>
</div>
<p class="MsoNormal"><span><p>&nbsp;</p></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">From:</span><span lang="EN-US"> Jon Brisbin
 [mailto:jon@...] <br>Sent: Wednesday, June 01, 2011 4:59 PM<br>To: Busoli, Simone<br>Cc: rabbitmq-discuss@...<br>Subject: Re: [rabbitmq-discuss] How to handle full exchange history delivery<p></p></span></p>
</div>
</div>
<p class="MsoNormal"><p>&nbsp;</p></p>
<div>
<p class="MsoNormal"><span lang="EN-US">Maybe I wasn't being clear:<br><br>
I've already written a custom exchange that stores every message it receives in Riak (</span><span><a href="http://wiki.basho.com/"><span lang="EN-US">http://wiki.basho.com/</span></a></span><span lang="EN-US">). I'm
 going to be talking about it next week at Erlang Factory in London, as a matter of fact. :)<br><br></span><span><a href="https://github.com/jbrisbin/riak-exchange"><span lang="EN-US">https://github.com/jbrisbin/riak-exchange</span></a></span><span lang="EN-US"><br><br>
When I was writing it, I thought it would be really easy to, when a new queue is bound to this exchange, go fetch all the messages in that bucket and send them to the queue.<br><br></span><span>Since I'm using Riak, the messages themselves are not all stored in memory.<br><br>
Thanks!<br><br>
Jon Brisbin<br>
http//jbrisbin.com<br><br><p></p></span></p>
<div class="MsoNormal" align="center"><span>
</span></div>
<blockquote>
<p class="MsoNormal"><span>From:
</span><span>"Simone Busoli" &lt;Simone.Busoli@...&gt;<br>To: rabbitmq-discuss@...<br>Sent: Wednesday, June 1, 2011 8:43:58 AM<br>Subject: Re: [rabbitmq-discuss] How to handle full exchange history delivery<br><br><br></span><span><p></p></span></p>
<p class="MsoNormal"><span lang="EN-US">That&rsquo;s what I thought as well, although I&rsquo;m thinking about a couple of issues about it:</span><span><p></p></span></p>
<p class="MsoNormal"><span lang="EN-US">&nbsp;</span><span><p></p></span></p>
<p class="MsoListParagraph"><span lang="EN-US">&middot;</span><span lang="EN-US">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
</span><span lang="EN-US">Lack of knowledge about Erlang, which is the language I&rsquo;d have to use to build a custom exchange, I guess</span><span><p></p></span></p>
<p class="MsoListParagraph"><span lang="EN-US">&middot;</span><span lang="EN-US">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
</span><span lang="EN-US">Amount of memory it would require to store the entire history in memory. It needn&rsquo;t be in memory actually, but I&rsquo;m not sure what would be the recommended way if the storage has to be carried out by exchange itself.
 Mnesia perhaps, would it make sense?</span><span><p></p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">_______________________________________________________</span><span>________</span><span>
</span><span><p></p></span></p>
<p class="MsoNormal"><span>Simone Busoli</span><span><br></span><span>Direzione risorse umane e segreteria generale</span><span>
</span><span>&ndash; Sistemi informativi gestione sportiva
<br>Collaboratore</span><span><p></p></span></p>
<p class="MsoNormal"><span>Ferrari S.p.A.&nbsp;&nbsp;&nbsp;&nbsp;</span><span>
</span><span>Via Ascari 55/57, 41053, Maranello (MO); Italy</span><span><br></span><span lang="DE">T</span><span>&nbsp;(+39) 0536-949.850</span><span><br></span><span>E</span><span>
<a href="mailto:simone.busoli@..." target="_blank">simone.busoli@...</a>&nbsp;
<a href="http://www.ferrari.com/" target="_blank">www.ferrari.com</a></span><span>
</span><span><p></p></span></p>
</div>
<p class="MsoNormal"><span>&nbsp;</span><span><p></p></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">From:</span><span lang="EN-US"> Jon Brisbin [mailto:jon@...]
<br>Sent: Wednesday, June 01, 2011 3:38 PM<br>To: Busoli, Simone<br>Cc: rabbitmq-discuss@...<br>Subject: Re: [rabbitmq-discuss] How to handle full exchange history delivery</span><span><p></p></span></p>
</div>
</div>
<p class="MsoNormal"><span>&nbsp;<p></p></span></p>
<div>
<p class="MsoNormal"><span>I've been seriously considering adding this functionality to the Riak-backed custom exchange. It was something I thought I would add first time through but didn't.<br><br>
You'd have to use a custom exchange. Not sure if that's an issue in your scenario or not.<br><br><br>
Thanks!<br><br>
Jon Brisbin<br>
http//jbrisbin.com<p></p></span></p>
<div class="MsoNormal" align="center"><span>
</span></div>
<blockquote>
<p class="MsoNormal"><span>From:
</span><span>"Simone Busoli" &lt;Simone.Busoli@...&gt;<br>To: rabbitmq-discuss@...<br>Sent: Wednesday, June 1, 2011 5:52:24 AM<br>Subject: [rabbitmq-discuss] How to handle full exchange history delivery<br><br></span><span><p></p></span></p>
<p class="MsoNormal"><span>Hi,<p></p></span></p>
<p class="MsoNormal"><span>&nbsp;<p></p></span></p>
<p class="MsoNormal"><span lang="EN-US">I am trying to achieve a scenario in which messages are published from some data source into a RabbitMQ exchange and clients should be able to receive the full history of messages flown through that
 exchange, regardless of the exact moment in time when they connect.</span><span><p></p></span></p>
<p class="MsoNormal"><span lang="EN-US">&nbsp;</span><span><p></p></span></p>
<p class="MsoNormal"><span lang="EN-US">I&rsquo;ve been thinking to solve it by setting up a proxy which subscribes to the exchange where messages are published using its own queue, keeps an internal storage of the messages and accepts requests
 from clients. </span><span><p></p></span></p>
<p class="MsoNormal"><span lang="EN-US">When a request from a client arrives it just replays the full message history to the client by publishing the messages to the client queue (supplied in the ReplyTo field),
then binds the client queue to the messages exchange, so that the client from this moment onwards starts receiving the &ldquo;live&rdquo; messages, and is pretty much unaware of what&rsquo;s happened; it simply just receives the whole history.</span><span><p></p></span></p>
<p class="MsoNormal"><span lang="EN-US">&nbsp;</span><span><p></p></span></p>
<p class="MsoNormal"><span lang="EN-US">Now I am under the impression that there&rsquo;s a fundamental flaw in this reasoning, since the proxy cannot know when the message history has completed and he can switch the client to the live stream,
 because while it is replaying historical messages to the client other live messages might be arriving on the proxy&rsquo;s queue, thus getting into an endless loop.</span><span><p></p></span></p>
<p class="MsoNormal"><span lang="EN-US">&nbsp;</span><span><p></p></span></p>
<p class="MsoNormal"><span lang="EN-US">I&rsquo;m probably missing something obvious, any advice about how others would approach this scenario is appreciated.</span><span><p></p></span></p>
<p class="MsoNormal"><span lang="EN-US">&nbsp;</span><span><p></p></span></p>
<p class="MsoNormal"><span lang="EN-US">Simone</span><span><p></p></span></p>
<p><span>_________________________________________________________________________________________________________________<br>
Questo messaggio e da intendersi esclusivamente ad uso del destinatario e puo contenere informazioni che sono di natura privilegiata, confidenziale<br>
o non divulgabile secondo le leggi vigenti. Se il lettore del presente messaggio non e il destinatario designato, o il dipendente/agente responsabile<br>
per la consegna del messaggio al destinatario designato, si informa che ogni disseminazione, distribuzione o copiatura di questa comunicazione e
<br>
strettamente proibita anche ai sensi del decreto legislativo 196/03 . Se avete ricevuto questo messaggio per errore, vi preghiamo di notificarcelo<br>
immediatamente a mezzo e-mail di risposta e successivamente di procedere alla cancellazione di questa e-mail e relativi allegati dal vostro sistema.<br>
_________________________________________________________________________________________________________________<br>
This message is intended only for the use of the addressee and may contain information that is privileged, confidential and exempt from
<br>
disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the
<br>
message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly
<br>
prohibited. If you have received this e-mail in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from
<br>
your system.<br>
_________________________________________________________________________________________________________________</span><span><p></p></span></p>
<p class="MsoNormal"><span><br></span><span lang="EN-US">_______________________________________________<br>
rabbitmq-discuss mailing list<br></span><span><a href="mailto:rabbitmq-discuss@..." target="_blank"><span lang="EN-US">rabbitmq-discuss@...</span></a></span><span lang="EN-US"><br></span><span><a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank"><span lang="EN-US">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</span></a></span><span><p></p></span></p>
</blockquote>
<p class="MsoNormal"><span lang="EN-US">&nbsp;</span><span><p></p></span></p>
</div>
<p><span>_________________________________________________________________________________________________________________<br>
Questo messaggio &egrave; da intendersi esclusivamente ad uso del destinatario e pu&ograve; contenere informazioni che sono di natura privilegiata, confidenziale<br>
o non divulgabile secondo le leggi vigenti. Se il lettore del presente messaggio non &egrave; il destinatario designato, o il dipendente/agente responsabile<br>
per la consegna del messaggio al destinatario designato, si informa che ogni disseminazione, distribuzione o copiatura di questa comunicazione &egrave;
<br>
strettamente proibita anche ai sensi del decreto legislativo 196/03 . Se avete ricevuto questo messaggio per errore, vi preghiamo di notificarcelo<br>
immediatamente a mezzo e-mail di risposta e successivamente di procedere alla cancellazione di questa e-mail e relativi allegati dal vostro sistema.<br>
_________________________________________________________________________________________________________________<br>
This message is intended only for the use of the addressee and may contain information that is privileged, confidential and exempt from
<br>
disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the
<br>
message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly
<br>
prohibited. If you have received this e-mail in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from
<br>
your system.<br>
_________________________________________________________________________________________________________________</span><span><p></p></span></p>
<p class="MsoNormal"><span><br></span><span lang="EN-US">_______________________________________________<br>
rabbitmq-discuss mailing list<br></span><span><a href="mailto:rabbitmq-discuss@..."><span lang="EN-US">rabbitmq-discuss@...</span></a></span><span lang="EN-US"><br></span><span><a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss"><span lang="EN-US">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</span></a></span><span lang="EN-US"><p></p></span></p>
</blockquote>
<p class="MsoNormal"><span lang="EN-US"><p>&nbsp;</p></span></p>
</div>
</div>
<p align="left">_________________________________________________________________________________________________________________<br>Questo messaggio &egrave; da intendersi esclusivamente ad uso del destinatario e pu&ograve; contenere informazioni che sono di natura privilegiata, confidenziale<br>o non divulgabile secondo le leggi vigenti. Se il lettore del presente messaggio non &egrave; il destinatario designato, o il dipendente/agente responsabile<br>per la consegna del messaggio al destinatario designato, si informa che ogni disseminazione, distribuzione o copiatura di questa comunicazione &egrave; <br>strettamente proibita anche ai sensi del decreto legislativo 196/03 . Se avete ricevuto questo messaggio per errore, vi preghiamo di notificarcelo<br>immediatamente a mezzo e-mail di risposta e successivamente di procedere alla cancellazione di questa e-mail e relativi allegati dal vostro sistema.<br>_________________________________________________________________________________________________________________<br>This message is intended only for the use of the addressee and may contain information that is privileged, confidential and exempt from <br>disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the <br>message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly <br>prohibited. If you have received this e-mail in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from <br>your system.<br>_________________________________________________________________________________________________________________<br></p>
</div>
Permalink | Reply |
headers
Jon Brisbin | 1 Jun 2011 17:50
Favicon
Gravatar

Re: How to handle full exchange history delivery

I'm not a big fan of forcing the message broker to keep fascist control over message ordering. Spring Integration has great tools for handling splitting and aggregation that can do this message ordering for you, which means your message broker situation can be much more flexible.

In this particular case, the only way to guarantee you won't receive new messages before you've finished sending them the old ones is to suspend message delivery of new messages to the newly-bound queue until it is finished sending the old ones. One would have to literally queue requests to route messages internally (to the exchange). The problem is that exchanges are stateless. One would have to actually store new message routing requests in mnesia or something based on whether or not the currently-being-filled queue has a new routing request pending.

All that said, it's technically possible to implement this kind of behaviour in the broker. I'm just not sure it would be a good idea. :) Messaging applications are, after all, supposed to be "A"synchronous, which means, necessarily, undetermined ordering. If ordering is really that important, then you don't want a message broker, you want a datastore and a cursor of some sort to sequentially process ordered items. You need, in fact, a datastore just like the one I'm currently proof-of-concepting (is that a word? ;) in my spare time, which is an asynchronous datastore that mixes messaging and data storage to achieve very closely to what you're wanting here. ;)

Thanks!

Jon Brisbin
http//jbrisbin.com


From: "Simone Busoli" <Simone.Busoli-sKiPNUh8iG9BDgjK7y7TUQ@public.gmane.org>
To: rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org
Sent: Wednesday, June 1, 2011 10:15:26 AM
Subject: Re: [rabbitmq-discuss] How to handle full exchange history delivery

Thanks Jon, I misunderstood you probably.

 

WRT this:

 

When I was writing it, I thought it would be really easy to, when a new queue is bound to this exchange, go fetch all the messages in that bucket and send them to the queue.

 

I’m not sure it is, that was my main question actually. How do you figure out how to join the messages stored in the bucket and those arriving live so that the last message you send to the queue from those picked up from the bucket is followed exactly by its logical successor from those coming live?

 

Say I have messages 1..100 being published to the exchange, each time one arrives it’s stored. The exchange has received the first 50 messages and a client performs a binding of a queue to the exchange. So you start pulling the messages from the storage and sending them to the queue before forwarding any live messages. While you are pulling them out of the storage messages 51..60 are published to the exchange, how do you deal with them? Do you always pull from the storage? When do you stop pulling from the storage and relay to the queue directly what is being published to the exchange if you don’t know whether any new messages have arrived while you were pulling from the storage?

_______________________________________________________________

Simone Busoli
Direzione risorse umane e segreteria generale – Sistemi informativi gestione sportiva
Collaboratore

Ferrari S.p.A.     Via Ascari 55/57, 41053, Maranello (MO); Italy
T (+39) 0536-949.850
E simone.busoli-sKiPNUh8iG9BDgjK7y7TUQ@public.gmane.org  www.ferrari.com

 

From: Jon Brisbin [mailto:jon-foCUeXAHXPNWk0Htik3J/w@public.gmane.org]
Sent: Wednesday, June 01, 2011 4:59 PM
To: Busoli, Simone
Cc: rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org
Subject: Re: [rabbitmq-discuss] How to handle full exchange history delivery

 

Maybe I wasn't being clear:

I've already written a custom exchange that stores every message it receives in Riak (http://wiki.basho.com/). I'm going to be talking about it next week at Erlang Factory in London, as a matter of fact. :)

https://github.com/jbrisbin/riak-exchange

When I was writing it, I thought it would be really easy to, when a new queue is bound to this exchange, go fetch all the messages in that bucket and send them to the queue.

Since I'm using Riak, the messages themselves are not all stored in memory.

Thanks!

Jon Brisbin
http//jbrisbin.com

From: "Simone Busoli" <Simone.Busoli-sKiPNUh8iG9BDgjK7y7TUQ@public.gmane.org>
To: rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org
Sent: Wednesday, June 1, 2011 8:43:58 AM
Subject: Re: [rabbitmq-discuss] How to handle full exchange history delivery


That’s what I thought as well, although I’m thinking about a couple of issues about it:

 

·         Lack of knowledge about Erlang, which is the language I’d have to use to build a custom exchange, I guess

·         Amount of memory it would require to store the entire history in memory. It needn’t be in memory actually, but I’m not sure what would be the recommended way if the storage has to be carried out by exchange itself. Mnesia perhaps, would it make sense?

_______________________________________________________________

Simone Busoli
Direzione risorse umane e segreteria generale – Sistemi informativi gestione sportiva
Collaboratore

Ferrari S.p.A.     Via Ascari 55/57, 41053, Maranello (MO); Italy
T (+39) 0536-949.850
E simone.busoli-sKiPNUh8iG9BDgjK7y7TUQ@public.gmane.org  www.ferrari.com

 

From: Jon Brisbin [mailto:jon-foCUeXAHXPNWk0Htik3J/w@public.gmane.org]
Sent: Wednesday, June 01, 2011 3:38 PM
To: Busoli, Simone
Cc: rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org
Subject: Re: [rabbitmq-discuss] How to handle full exchange history delivery

 

I've been seriously considering adding this functionality to the Riak-backed custom exchange. It was something I thought I would add first time through but didn't.

You'd have to use a custom exchange. Not sure if that's an issue in your scenario or not.


Thanks!

Jon Brisbin
http//jbrisbin.com

From: "Simone Busoli" <Simone.Busoli-sKiPNUh8iG9BDgjK7y7TUQ@public.gmane.org>
To: rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org
Sent: Wednesday, June 1, 2011 5:52:24 AM
Subject: [rabbitmq-discuss] How to handle full exchange history delivery

Hi,

 

I am trying to achieve a scenario in which messages are published from some data source into a RabbitMQ exchange and clients should be able to receive the full history of messages flown through that exchange, regardless of the exact moment in time when they connect.

 

I’ve been thinking to solve it by setting up a proxy which subscribes to the exchange where messages are published using its own queue, keeps an internal storage of the messages and accepts requests from clients.

When a request from a client arrives it just replays the full message history to the client by publishing the messages to the client queue (supplied in the ReplyTo field), then binds the client queue to the messages exchange, so that the client from this moment onwards starts receiving the “live” messages, and is pretty much unaware of what’s happened; it simply just receives the whole history.

 

Now I am under the impression that there’s a fundamental flaw in this reasoning, since the proxy cannot know when the message history has completed and he can switch the client to the live stream, because while it is replaying historical messages to the client other live messages might be arriving on the proxy’s queue, thus getting into an endless loop.

 

I’m probably missing something obvious, any advice about how others would approach this scenario is appreciated.

 

Simone

_________________________________________________________________________________________________________________
Questo messaggio e da intendersi esclusivamente ad uso del destinatario e puo contenere informazioni che sono di natura privilegiata, confidenziale
o non divulgabile secondo le leggi vigenti. Se il lettore del presente messaggio non e il destinatario designato, o il dipendente/agente responsabile
per la consegna del messaggio al destinatario designato, si informa che ogni disseminazione, distribuzione o copiatura di questa comunicazione e
strettamente proibita anche ai sensi del decreto legislativo 196/03 . Se avete ricevuto questo messaggio per errore, vi preghiamo di notificarcelo
immediatamente a mezzo e-mail di risposta e successivamente di procedere alla cancellazione di questa e-mail e relativi allegati dal vostro sistema.
_________________________________________________________________________________________________________________
This message is intended only for the use of the addressee and may contain information that is privileged, confidential and exempt from
disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the
message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly
prohibited. If you have received this e-mail in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from
your system.
_________________________________________________________________________________________________________________


_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss <at> lists.rabbitmq.com
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

 

_________________________________________________________________________________________________________________
Questo messaggio è da intendersi esclusivamente ad uso del destinatario e può contenere informazioni che sono di natura privilegiata, confidenziale
o non divulgabile secondo le leggi vigenti. Se il lettore del presente messaggio non è il destinatario designato, o il dipendente/agente responsabile
per la consegna del messaggio al destinatario designato, si informa che ogni disseminazione, distribuzione o copiatura di questa comunicazione è
strettamente proibita anche ai sensi del decreto legislativo 196/03 . Se avete ricevuto questo messaggio per errore, vi preghiamo di notificarcelo
immediatamente a mezzo e-mail di risposta e successivamente di procedere alla cancellazione di questa e-mail e relativi allegati dal vostro sistema.
_________________________________________________________________________________________________________________
This message is intended only for the use of the addressee and may contain information that is privileged, confidential and exempt from
disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the
message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly
prohibited. If you have received this e-mail in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from
your system.
_________________________________________________________________________________________________________________


_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss <at> lists.rabbitmq.com
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

 

_________________________________________________________________________________________________________________
Questo messaggio è da intendersi esclusivamente ad uso del destinatario e può contenere informazioni che sono di natura privilegiata, confidenziale
o non divulgabile secondo le leggi vigenti. Se il lettore del presente messaggio non è il destinatario designato, o il dipendente/agente responsabile
per la consegna del messaggio al destinatario designato, si informa che ogni disseminazione, distribuzione o copiatura di questa comunicazione è
strettamente proibita anche ai sensi del decreto legislativo 196/03 . Se avete ricevuto questo messaggio per errore, vi preghiamo di notificarcelo
immediatamente a mezzo e-mail di risposta e successivamente di procedere alla cancellazione di questa e-mail e relativi allegati dal vostro sistema.
_________________________________________________________________________________________________________________
This message is intended only for the use of the addressee and may contain information that is privileged, confidential and exempt from
disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the
message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly
prohibited. If you have received this e-mail in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from
your system.
_________________________________________________________________________________________________________________


_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

<div><div>I'm not a big fan of forcing the message broker to keep fascist control over<span> message ordering. Spring Integration has great tools for handling splitting and aggregation that can do this message ordering for you, which means your message broker situation can be much more flexible.<br><br>In this particular case, the only way to guarantee you won't receive new messages before you've finished sending them the old ones is to suspend message delivery of new messages to the newly-bound queue until it is finished sending the old ones. One would have to literally queue requests to route messages internally (to the exchange). The problem is that exchanges are stateless. One would have to actually store new message routing requests in mnesia or something based on whether or not the currently-being-filled queue has a new routing request pending.<br><br>All that said, it's technically possible to implement this kind of behaviour in the broker. I'm just not sure it would be a good idea. :) Messaging applications are, after all, supposed to be "A"synchronous, which means, necessarily, undetermined ordering. If ordering is really that important, then you don't want a message broker, you want a datastore and a cursor of some sort to sequentially process ordered items. You need, in fact, a datastore just like the one I'm currently proof-of-concepting (is that a word? ;) in my spare time, which is an asynchronous datastore that mixes messaging and data storage to achieve very closely to what you're wanting here. ;)<br><span name="x"></span><br>Thanks!<br><br>Jon Brisbin<br>http//jbrisbin.com<br><span name="x"></span><br></span><br><blockquote>From: "Simone Busoli" &lt;Simone.Busoli@...&gt;<br>To: rabbitmq-discuss@...<br>Sent: Wednesday, June 1, 2011 10:15:26 AM<br>Subject: Re: [rabbitmq-discuss] How to handle full exchange history delivery<br><br><div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US">Thanks Jon, I misunderstood you probably.</span></p>
<p class="MsoNormal"><span lang="EN-US">&nbsp;</span></p>
<p class="MsoNormal"><span lang="EN-US">WRT this:</span></p>
<p class="MsoNormal"><span lang="EN-US">&nbsp;</span></p>
<p class="MsoNormal"><span lang="EN-US">When I was writing it, I thought it would be really easy to, when a new queue is bound to this exchange, go fetch all the messages in that bucket and send them to the queue.</span><span lang="EN-US"></span></p>
<p class="MsoNormal"><span lang="EN-US">&nbsp;</span></p>
<p class="MsoNormal"><span lang="EN-US">I&rsquo;m not sure it is, that was my main question actually. How do you figure out how to join the messages stored in the bucket and those arriving live so that the last message you send to the queue
 from those picked up from the bucket is followed exactly by its logical successor from those coming live?</span></p>
<p class="MsoNormal"><span lang="EN-US">&nbsp;</span></p>
<p class="MsoNormal"><span lang="EN-US">Say I have messages 1..100 being published to the exchange, each time one arrives it&rsquo;s stored. The exchange has received the first 50 messages and a client performs a binding of a queue to the exchange.
 So you start pulling the messages from the storage and sending them to the queue before forwarding any live messages. While you are pulling them out of the storage messages 51..60 are published to the exchange, how do you deal with them? Do you always pull
 from the storage? When do you stop pulling from the storage and relay to the queue directly what is being published to the exchange if you don&rsquo;t know whether any new messages have arrived while you were pulling from the storage?</span></p>
<div>
<p class="MsoNormal"><span>_______________________________________________________________</span><span>
</span></p>
<p class="MsoNormal"><span>Simone Busoli</span><span><br></span><span>Direzione risorse umane e segreteria generale</span><span>
</span><span>&ndash; Sistemi informativi gestione sportiva
<br>Collaboratore</span><span></span></p>
<p class="MsoNormal"><span>Ferrari S.p.A.&nbsp;&nbsp;&nbsp;&nbsp;</span><span>
</span><span>Via Ascari 55/57, 41053, Maranello (MO); Italy</span><span><br></span><span lang="DE">T</span><span>&nbsp;(+39) 0536-949.850</span><span><br></span><span>E</span><span>
<a href="mailto:simone.busoli@..." target="_blank"><span>simone.busoli@...</span></a>&nbsp;
<a href="http://www.ferrari.com/" target="_blank"><span>www.ferrari.com</span></a></span><span>
</span></p>
</div>
<p class="MsoNormal"><span>&nbsp;</span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">From:</span><span lang="EN-US"> Jon Brisbin
 [mailto:jon@...] <br>Sent: Wednesday, June 01, 2011 4:59 PM<br>To: Busoli, Simone<br>Cc: rabbitmq-discuss@...<br>Subject: Re: [rabbitmq-discuss] How to handle full exchange history delivery</span></p>
</div>
</div>
<p class="MsoNormal">&nbsp;</p>
<div>
<p class="MsoNormal"><span lang="EN-US">Maybe I wasn't being clear:<br><br>
I've already written a custom exchange that stores every message it receives in Riak (</span><span><a href="http://wiki.basho.com/" target="_blank"><span lang="EN-US">http://wiki.basho.com/</span></a></span><span lang="EN-US">). I'm
 going to be talking about it next week at Erlang Factory in London, as a matter of fact. :)<br><br></span><span><a href="https://github.com/jbrisbin/riak-exchange" target="_blank"><span lang="EN-US">https://github.com/jbrisbin/riak-exchange</span></a></span><span lang="EN-US"><br><br>
When I was writing it, I thought it would be really easy to, when a new queue is bound to this exchange, go fetch all the messages in that bucket and send them to the queue.<br><br></span><span>Since I'm using Riak, the messages themselves are not all stored in memory.<br><br>
Thanks!<br><br>
Jon Brisbin<br>
http//jbrisbin.com<br><br></span></p>
<div class="MsoNormal" align="center"><span>
</span></div>
<blockquote>
<p class="MsoNormal"><span>From:
</span><span>"Simone Busoli" &lt;Simone.Busoli@...&gt;<br>To: rabbitmq-discuss@...<br>Sent: Wednesday, June 1, 2011 8:43:58 AM<br>Subject: Re: [rabbitmq-discuss] How to handle full exchange history delivery<br><br><br></span><span></span></p>
<p class="MsoNormal"><span lang="EN-US">That&rsquo;s what I thought as well, although I&rsquo;m thinking about a couple of issues about it:</span><span></span></p>
<p class="MsoNormal"><span lang="EN-US">&nbsp;</span><span></span></p>
<p class="MsoListParagraph"><span lang="EN-US">&middot;</span><span lang="EN-US">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
</span><span lang="EN-US">Lack of knowledge about Erlang, which is the language I&rsquo;d have to use to build a custom exchange, I guess</span><span></span></p>
<p class="MsoListParagraph"><span lang="EN-US">&middot;</span><span lang="EN-US">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
</span><span lang="EN-US">Amount of memory it would require to store the entire history in memory. It needn&rsquo;t be in memory actually, but I&rsquo;m not sure what would be the recommended way if the storage has to be carried out by exchange itself.
 Mnesia perhaps, would it make sense?</span><span></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">_______________________________________________________</span><span>________</span><span>
</span><span></span></p>
<p class="MsoNormal"><span>Simone Busoli</span><span><br></span><span>Direzione risorse umane e segreteria generale</span><span>
</span><span>&ndash; Sistemi informativi gestione sportiva
<br>Collaboratore</span><span></span></p>
<p class="MsoNormal"><span>Ferrari S.p.A.&nbsp;&nbsp;&nbsp;&nbsp;</span><span>
</span><span>Via Ascari 55/57, 41053, Maranello (MO); Italy</span><span><br></span><span lang="DE">T</span><span>&nbsp;(+39) 0536-949.850</span><span><br></span><span>E</span><span>
<a href="mailto:simone.busoli@..." target="_blank">simone.busoli@...</a>&nbsp;
<a href="http://www.ferrari.com/" target="_blank">www.ferrari.com</a></span><span>
</span><span></span></p>
</div>
<p class="MsoNormal"><span>&nbsp;</span><span></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">From:</span><span lang="EN-US"> Jon Brisbin [mailto:jon@...]
<br>Sent: Wednesday, June 01, 2011 3:38 PM<br>To: Busoli, Simone<br>Cc: rabbitmq-discuss@...<br>Subject: Re: [rabbitmq-discuss] How to handle full exchange history delivery</span><span></span></p>
</div>
</div>
<p class="MsoNormal"><span>&nbsp;</span></p>
<div>
<p class="MsoNormal"><span>I've been seriously considering adding this functionality to the Riak-backed custom exchange. It was something I thought I would add first time through but didn't.<br><br>
You'd have to use a custom exchange. Not sure if that's an issue in your scenario or not.<br><br><br>
Thanks!<br><br>
Jon Brisbin<br>
http//jbrisbin.com</span></p>
<div class="MsoNormal" align="center"><span>
</span></div>
<blockquote>
<p class="MsoNormal"><span>From:
</span><span>"Simone Busoli" &lt;Simone.Busoli@...&gt;<br>To: rabbitmq-discuss@...<br>Sent: Wednesday, June 1, 2011 5:52:24 AM<br>Subject: [rabbitmq-discuss] How to handle full exchange history delivery<br><br></span><span></span></p>
<p class="MsoNormal"><span>Hi,</span></p>
<p class="MsoNormal"><span>&nbsp;</span></p>
<p class="MsoNormal"><span lang="EN-US">I am trying to achieve a scenario in which messages are published from some data source into a RabbitMQ exchange and clients should be able to receive the full history of messages flown through that
 exchange, regardless of the exact moment in time when they connect.</span><span></span></p>
<p class="MsoNormal"><span lang="EN-US">&nbsp;</span><span></span></p>
<p class="MsoNormal"><span lang="EN-US">I&rsquo;ve been thinking to solve it by setting up a proxy which subscribes to the exchange where messages are published using its own queue, keeps an internal storage of the messages and accepts requests
 from clients. </span><span></span></p>
<p class="MsoNormal"><span lang="EN-US">When a request from a client arrives it just replays the full message history to the client by publishing the messages to the client queue (supplied in the ReplyTo field),
then binds the client queue to the messages exchange, so that the client from this moment onwards starts receiving the &ldquo;live&rdquo; messages, and is pretty much unaware of what&rsquo;s happened; it simply just receives the whole history.</span><span></span></p>
<p class="MsoNormal"><span lang="EN-US">&nbsp;</span><span></span></p>
<p class="MsoNormal"><span lang="EN-US">Now I am under the impression that there&rsquo;s a fundamental flaw in this reasoning, since the proxy cannot know when the message history has completed and he can switch the client to the live stream,
 because while it is replaying historical messages to the client other live messages might be arriving on the proxy&rsquo;s queue, thus getting into an endless loop.</span><span></span></p>
<p class="MsoNormal"><span lang="EN-US">&nbsp;</span><span></span></p>
<p class="MsoNormal"><span lang="EN-US">I&rsquo;m probably missing something obvious, any advice about how others would approach this scenario is appreciated.</span><span></span></p>
<p class="MsoNormal"><span lang="EN-US">&nbsp;</span><span></span></p>
<p class="MsoNormal"><span lang="EN-US">Simone</span><span></span></p>
<p><span>_________________________________________________________________________________________________________________<br>
Questo messaggio e da intendersi esclusivamente ad uso del destinatario e puo contenere informazioni che sono di natura privilegiata, confidenziale<br>
o non divulgabile secondo le leggi vigenti. Se il lettore del presente messaggio non e il destinatario designato, o il dipendente/agente responsabile<br>
per la consegna del messaggio al destinatario designato, si informa che ogni disseminazione, distribuzione o copiatura di questa comunicazione e
<br>
strettamente proibita anche ai sensi del decreto legislativo 196/03 . Se avete ricevuto questo messaggio per errore, vi preghiamo di notificarcelo<br>
immediatamente a mezzo e-mail di risposta e successivamente di procedere alla cancellazione di questa e-mail e relativi allegati dal vostro sistema.<br>
_________________________________________________________________________________________________________________<br>
This message is intended only for the use of the addressee and may contain information that is privileged, confidential and exempt from
<br>
disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the
<br>
message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly
<br>
prohibited. If you have received this e-mail in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from
<br>
your system.<br>
_________________________________________________________________________________________________________________</span><span></span></p>
<p class="MsoNormal"><span><br></span><span lang="EN-US">_______________________________________________<br>
rabbitmq-discuss mailing list<br></span><span><a href="mailto:rabbitmq-discuss <at> lists.rabbitmq.com" target="_blank"><span lang="EN-US">rabbitmq-discuss <at> lists.rabbitmq.com</span></a></span><span lang="EN-US"><br></span><span><a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank"><span lang="EN-US">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</span></a></span><span></span></p>
</blockquote>
<p class="MsoNormal"><span lang="EN-US">&nbsp;</span><span></span></p>
</div>
<p><span>_________________________________________________________________________________________________________________<br>
Questo messaggio &egrave; da intendersi esclusivamente ad uso del destinatario e pu&ograve; contenere informazioni che sono di natura privilegiata, confidenziale<br>
o non divulgabile secondo le leggi vigenti. Se il lettore del presente messaggio non &egrave; il destinatario designato, o il dipendente/agente responsabile<br>
per la consegna del messaggio al destinatario designato, si informa che ogni disseminazione, distribuzione o copiatura di questa comunicazione &egrave;
<br>
strettamente proibita anche ai sensi del decreto legislativo 196/03 . Se avete ricevuto questo messaggio per errore, vi preghiamo di notificarcelo<br>
immediatamente a mezzo e-mail di risposta e successivamente di procedere alla cancellazione di questa e-mail e relativi allegati dal vostro sistema.<br>
_________________________________________________________________________________________________________________<br>
This message is intended only for the use of the addressee and may contain information that is privileged, confidential and exempt from
<br>
disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the
<br>
message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly
<br>
prohibited. If you have received this e-mail in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from
<br>
your system.<br>
_________________________________________________________________________________________________________________</span><span></span></p>
<p class="MsoNormal"><span><br></span><span lang="EN-US">_______________________________________________<br>
rabbitmq-discuss mailing list<br></span><span><a href="mailto:rabbitmq-discuss <at> lists.rabbitmq.com" target="_blank"><span lang="EN-US">rabbitmq-discuss <at> lists.rabbitmq.com</span></a></span><span lang="EN-US"><br></span><span><a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank"><span lang="EN-US">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</span></a></span><span lang="EN-US"></span></p>
</blockquote>
<p class="MsoNormal"><span lang="EN-US">&nbsp;</span></p>
</div>
</div>
<p align="left">_________________________________________________________________________________________________________________<br>Questo messaggio &egrave; da intendersi esclusivamente ad uso del destinatario e pu&ograve; contenere informazioni che sono di natura privilegiata, confidenziale<br>o non divulgabile secondo le leggi vigenti. Se il lettore del presente messaggio non &egrave; il destinatario designato, o il dipendente/agente responsabile<br>per la consegna del messaggio al destinatario designato, si informa che ogni disseminazione, distribuzione o copiatura di questa comunicazione &egrave; <br>strettamente proibita anche ai sensi del decreto legislativo 196/03 . Se avete ricevuto questo messaggio per errore, vi preghiamo di notificarcelo<br>immediatamente a mezzo e-mail di risposta e successivamente di procedere alla cancellazione di questa e-mail e relativi allegati dal vostro sistema.<br>_________________________________________________________________________________________________________________<br>This message is intended only for the use of the addressee and may contain information that is privileged, confidential and exempt from <br>disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the <br>message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly <br>prohibited. If you have received this e-mail in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from <br>your system.<br>_________________________________________________________________________________________________________________<br></p>
<br>_______________________________________________<br>rabbitmq-discuss mailing list<br>rabbitmq-discuss@...<br>https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss<br>
</blockquote>
<br>
</div></div>
Permalink | Reply |
headers
Simone Busoli | 2 Jun 2011 15:36
Picon
Gravatar

Re: How to handle full exchange history delivery

Thanks Jon, I see your point. I'm really just evaluating ways to accomplish this scenario and went down to trying to carry it out by relying on RabbitMQ alone, which I get doesn't fit very well here.
I will probably fallback to creating a layer on top of it which takes care of message ordering and merging of storage and live streams. Thanks for sharing your ideas!

On Jun 1, 2011 5:51 PM, "Jon Brisbin" <jon-foCUeXAHXPNWk0Htik3J/w@public.gmane.org> wrote:
> I'm not a big fan of forcing the message broker to keep fascist control over message ordering. Spring Integration has great tools for handling splitting and aggregation that can do this message ordering for you, which means your message broker situation can be much more flexible.
>
> In this particular case, the only way to guarantee you won't receive new messages before you've finished sending them the old ones is to suspend message delivery of new messages to the newly-bound queue until it is finished sending the old ones. One would have to literally queue requests to route messages internally (to the exchange). The problem is that exchanges are stateless. One would have to actually store new message routing requests in mnesia or something based on whether or not the currently-being-filled queue has a new routing request pending.
>
> All that said, it's technically possible to implement this kind of behaviour in the broker. I'm just not sure it would be a good idea. :) Messaging applications are, after all, supposed to be "A"synchronous, which means, necessarily, undetermined ordering. If ordering is really that important, then you don't want a message broker, you want a datastore and a cursor of some sort to sequentially process ordered items. You need, in fact, a datastore just like the one I'm currently proof-of-concepting (is that a word? ;) in my spare time, which is an asynchronous datastore that mixes messaging and data storage to achieve very closely to what you're wanting here. ;)
>
> Thanks!
>
> Jon Brisbin
> http//jbrisbin.com
>
> ----- Original Message -----
>
>> From: "Simone Busoli" <Simone.Busoli-sKiPNUh8iG9BDgjK7y7TUQ@public.gmane.org>
>> To: rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org
>> Sent: Wednesday, June 1, 2011 10:15:26 AM
>> Subject: Re: [rabbitmq-discuss] How to handle full exchange history
>> delivery
>
>> Thanks Jon, I misunderstood you probably.
>
>> WRT this:
>
>> When I was writing it, I thought it would be really easy to, when a
>> new queue is bound to this exchange, go fetch all the messages in
>> that bucket and send them to the queue.
>
>> I’m not sure it is, that was my main question actually. How do you
>> figure out how to join the messages stored in the bucket and those
>> arriving live so that the last message you send to the queue from
>> those picked up from the bucket is followed exactly by its logical
>> successor from those coming live?
>
>> Say I have messages 1..100 being published to the exchange, each time
>> one arrives it’s stored. The exchange has received the first 50
>> messages and a client performs a binding of a queue to the exchange.
>> So you start pulling the messages from the storage and sending them
>> to the queue before forwarding any live messages. While you are
>> pulling them out of the storage messages 51..60 are published to the
>> exchange, how do you deal with them? Do you always pull from the
>> storage? When do you stop pulling from the storage and relay to the
>> queue directly what is being published to the exchange if you don’t
>> know whether any new messages have arrived while you were pulling
>> from the storage?
>
>> _______________________________________________________________
>> Simone Busoli
>> Direzione risorse umane e segreteria generale – Sistemi informativi
>> gestione sportiva
>> Collaboratore
>> Ferrari S.p.A. Via Ascari 55/57, 41053, Maranello (MO); Italy
>> T (+39) 0536-949.850
>> E simone.busoli-sKiPNUh8iG9BDgjK7y7TUQ@public.gmane.org www.ferrari.com
>
>> From: Jon Brisbin [mailto:jon-foCUeXAHXPNWk0Htik3J/w@public.gmane.org]
>> Sent: Wednesday, June 01, 2011 4:59 PM
>> To: Busoli, Simone
>> Cc: rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org
>> Subject: Re: [rabbitmq-discuss] How to handle full exchange history
>> delivery
>
>> Maybe I wasn't being clear:
>
>> I've already written a custom exchange that stores every message it
>> receives in Riak ( http://wiki.basho.com/ ). I'm going to be talking
>> about it next week at Erlang Factory in London, as a matter of fact.
>> :)
>
>> https://github.com/jbrisbin/riak-exchange
>
>> When I was writing it, I thought it would be really easy to, when a
>> new queue is bound to this exchange, go fetch all the messages in
>> that bucket and send them to the queue.
>
>> Since I'm using Riak, the messages themselves are not all stored in
>> memory.
>
>> Thanks!
>
>> Jon Brisbin
>> http//jbrisbin.com
>
>> ----- Original Message -----
>
>> > From: "Simone Busoli" <Simone.Busoli-sKiPNUh8iG9BDgjK7y7TUQ@public.gmane.org>
>>
>> > To: rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org
>>
>> > Sent: Wednesday, June 1, 2011 8:43:58 AM
>>
>> > Subject: Re: [rabbitmq-discuss] How to handle full exchange history
>> > delivery
>>
>
>> > That’s what I thought as well, although I’m thinking about a couple
>> > of issues about it:
>>
>
>> > · Lack of knowledge about Erlang, which is the language I’d have to
>> > use to build a custom exchange, I guess
>>
>> > · Amount of memory it would require to store the entire history in
>> > memory. It needn’t be in memory actually, but I’m not sure what
>> > would be the recommended way if the storage has to be carried out
>> > by
>> > exchange itself. Mnesia perhaps, would it make sense?
>>
>
>> > _______________________________________________________ ________
>>
>> > Simone Busoli
>>
>> > Direzione risorse umane e segreteria generale – Sistemi informativi
>> > gestione sportiva
>>
>> > Collaboratore
>>
>> > Ferrari S.p.A. Via Ascari 55/57, 41053, Maranello (MO); Italy
>>
>> > T (+39) 0536-949.850
>>
>> > E simone.busoli-sKiPNUh8iG9BDgjK7y7TUQ@public.gmane.org www.ferrari.com
>>
>
>> > From: Jon Brisbin [mailto:jon-foCUeXAHXPNWk0Htik3J/w@public.gmane.org]
>>
>> > Sent: Wednesday, June 01, 2011 3:38 PM
>>
>> > To: Busoli, Simone
>>
>> > Cc: rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org
>>
>> > Subject: Re: [rabbitmq-discuss] How to handle full exchange history
>> > delivery
>>
>
>> > I've been seriously considering adding this functionality to the
>> > Riak-backed custom exchange. It was something I thought I would add
>> > first time through but didn't.
>>
>
>> > You'd have to use a custom exchange. Not sure if that's an issue in
>> > your scenario or not.
>>
>
>> > Thanks!
>>
>
>> > Jon Brisbin
>>
>> > http//jbrisbin.com
>>
>
>> > > From: "Simone Busoli" <Simone.Busoli-sKiPNUh8iG9BDgjK7y7TUQ@public.gmane.org>
>> >
>>
>> > > To: rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org
>> >
>>
>> > > Sent: Wednesday, June 1, 2011 5:52:24 AM
>> >
>>
>> > > Subject: [rabbitmq-discuss] How to handle full exchange history
>> > > delivery
>> >
>>
>
>> > > Hi,
>> >
>>
>
>> > > I am trying to achieve a scenario in which messages are published
>> > > from some data source into a RabbitMQ exchange and clients should
>> > > be
>> > > able to receive the full history of messages flown through that
>> > > exchange, regardless of the exact moment in time when they
>> > > connect.
>> >
>>
>
>> > > I’ve been thinking to solve it by setting up a proxy which
>> > > subscribes
>> > > to the exchange where messages are published using its own queue,
>> > > keeps an internal storage of the messages and accepts requests
>> > > from
>> > > clients.
>> >
>>
>> > > When a request from a client arrives it just replays the full
>> > > message
>> > > history to the client by publishing the messages to the client
>> > > queue
>> > > (supplied in the ReplyTo field), then binds the client queue to
>> > > the
>> > > messages exchange, so that the client from this moment onwards
>> > > starts receiving the “live” messages, and is pretty much unaware
>> > > of
>> > > what’s happened; it simply just receives the whole history.
>> >
>>
>
>> > > Now I am under the impression that there’s a fundamental flaw in
>> > > this
>> > > reasoning, since the proxy cannot know when the message history
>> > > has
>> > > completed and he can switch the client to the live stream,
>> > > because
>> > > while it is replaying historical messages to the client other
>> > > live
>> > > messages might be arriving on the proxy’s queue, thus getting
>> > > into
>> > > an endless loop.
>> >
>>
>
>> > > I’m probably missing something obvious, any advice about how
>> > > others
>> > > would approach this scenario is appreciated.
>> >
>>
>
>> > > Simone
>> >
>>
>> > > _________________________________________________________________________________________________________________
>> >
>>
>> > > Questo messaggio e da intendersi esclusivamente ad uso del
>> > > destinatario e puo contenere informazioni che sono di natura
>> > > privilegiata, confidenziale
>> >
>>
>> > > o non divulgabile secondo le leggi vigenti. Se il lettore del
>> > > presente messaggio non e il destinatario designato, o il
>> > > dipendente/agente responsabile
>> >
>>
>> > > per la consegna del messaggio al destinatario designato, si
>> > > informa
>> > > che ogni disseminazione, distribuzione o copiatura di questa
>> > > comunicazione e
>> >
>>
>> > > strettamente proibita anche ai sensi del decreto legislativo
>> > > 196/03
>> > > .
>> > > Se avete ricevuto questo messaggio per errore, vi preghiamo di
>> > > notificarcelo
>> >
>>
>> > > immediatamente a mezzo e-mail di risposta e successivamente di
>> > > procedere alla cancellazione di questa e-mail e relativi allegati
>> > > dal vostro sistema.
>> >
>>
>> > > _________________________________________________________________________________________________________________
>> >
>>
>> > > This message is intended only for the use of the addressee and
>> > > may
>> > > contain information that is privileged, confidential and exempt
>> > > from
>> >
>>
>> > > disclosure under applicable law. If the reader of this message is
>> > > not
>> > > the intended recipient, or the employee or agent responsible for
>> > > delivering the
>> >
>>
>> > > message to the intended recipient, you are hereby notified that
>> > > any
>> > > dissemination, distribution or copying of this communication is
>> > > strictly
>> >
>>
>> > > prohibited. If you have received this e-mail in error, please
>> > > notify
>> > > us immediately by return e-mail and delete this e-mail and all
>> > > attachments from
>> >
>>
>> > > your system.
>> >
>>
>> > > _________________________________________________________________________________________________________________
>> >
>>
>
>> > > _______________________________________________
>> >
>>
>> > > rabbitmq-discuss mailing list
>> >
>>
>> > > rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org
>> >
>>
>> > > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>> >
>>
>> > _________________________________________________________________________________________________________________
>>
>> > Questo messaggio è da intendersi esclusivamente ad uso del
>> > destinatario e può contenere informazioni che sono di natura
>> > privilegiata, confidenziale
>>
>> > o non divulgabile secondo le leggi vigenti. Se il lettore del
>> > presente messaggio non è il destinatario designato, o il
>> > dipendente/agente responsabile
>>
>> > per la consegna del messaggio al destinatario designato, si informa
>> > che ogni disseminazione, distribuzione o copiatura di questa
>> > comunicazione è
>>
>> > strettamente proibita anche ai sensi del decreto legislativo 196/03
>> > .
>> > Se avete ricevuto questo messaggio per errore, vi preghiamo di
>> > notificarcelo
>>
>> > immediatamente a mezzo e-mail di risposta e successivamente di
>> > procedere alla cancellazione di questa e-mail e relativi allegati
>> > dal vostro sistema.
>>
>> > _________________________________________________________________________________________________________________
>>
>> > This message is intended only for the use of the addressee and may
>> > contain information that is privileged, confidential and exempt
>> > from
>>
>> > disclosure under applicable law. If the reader of this message is
>> > not
>> > the intended recipient, or the employee or agent responsible for
>> > delivering the
>>
>> > message to the intended recipient, you are hereby notified that any
>> > dissemination, distribution or copying of this communication is
>> > strictly
>>
>> > prohibited. If you have received this e-mail in error, please
>> > notify
>> > us immediately by return e-mail and delete this e-mail and all
>> > attachments from
>>
>> > your system.
>>
>> > _________________________________________________________________________________________________________________
>>
>
>> > _______________________________________________
>>
>> > rabbitmq-discuss mailing list
>>
>> > rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org
>>
>> > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>> _________________________________________________________________________________________________________________
>> Questo messaggio è da intendersi esclusivamente ad uso del
>> destinatario e può contenere informazioni che sono di natura
>> privilegiata, confidenziale
>> o non divulgabile secondo le leggi vigenti. Se il lettore del
>> presente messaggio non è il destinatario designato, o il
>> dipendente/agente responsabile
>> per la consegna del messaggio al destinatario designato, si informa
>> che ogni disseminazione, distribuzione o copiatura di questa
>> comunicazione è
>> strettamente proibita anche ai sensi del decreto legislativo 196/03 .
>> Se avete ricevuto questo messaggio per errore, vi preghiamo di
>> notificarcelo
>> immediatamente a mezzo e-mail di risposta e successivamente di
>> procedere alla cancellazione di questa e-mail e relativi allegati
>> dal vostro sistema.
>> _________________________________________________________________________________________________________________
>> This message is intended only for the use of the addressee and may
>> contain information that is privileged, confidential and exempt from
>> disclosure under applicable law. If the reader of this message is not
>> the intended recipient, or the employee or agent responsible for
>> delivering the
>> message to the intended recipient, you are hereby notified that any
>> dissemination, distribution or copying of this communication is
>> strictly
>> prohibited. If you have received this e-mail in error, please notify
>> us immediately by return e-mail and delete this e-mail and all
>> attachments from
>> your system.
>> _________________________________________________________________________________________________________________
>
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
<div>
<p>Thanks Jon, I see your point. I'm really just evaluating ways to accomplish this scenario and went down to trying to carry it out by relying on RabbitMQ alone, which I get doesn't fit very well here.<br>
I will probably fallback to creating a layer on top of it which takes care of message ordering and merging of storage and live streams. Thanks for sharing your ideas!</p>
<div class="gmail_quote">On Jun 1, 2011 5:51 PM, "Jon Brisbin" &lt;<a href="mailto:jon@...">jon@...</a>&gt; wrote:<br type="attribution">&gt; I'm not a big fan of forcing the message broker to keep fascist control over message ordering. Spring Integration has great tools for handling splitting and aggregation that can do this message ordering for you, which means your message broker situation can be much more flexible. <br>
&gt; <br>&gt; In this particular case, the only way to guarantee you won't receive new messages before you've finished sending them the old ones is to suspend message delivery of new messages to the newly-bound queue until it is finished sending the old ones. One would have to literally queue requests to route messages internally (to the exchange). The problem is that exchanges are stateless. One would have to actually store new message routing requests in mnesia or something based on whether or not the currently-being-filled queue has a new routing request pending. <br>
&gt; <br>&gt; All that said, it's technically possible to implement this kind of behaviour in the broker. I'm just not sure it would be a good idea. :) Messaging applications are, after all, supposed to be "A"synchronous, which means, necessarily, undetermined ordering. If ordering is really that important, then you don't want a message broker, you want a datastore and a cursor of some sort to sequentially process ordered items. You need, in fact, a datastore just like the one I'm currently proof-of-concepting (is that a word? ;) in my spare time, which is an asynchronous datastore that mixes messaging and data storage to achieve very closely to what you're wanting here. ;) <br>
&gt; <br>&gt; Thanks! <br>&gt; <br>&gt; Jon Brisbin <br>&gt; http//<a href="http://jbrisbin.com">jbrisbin.com</a> <br>&gt; <br>&gt; ----- Original Message -----<br>&gt; <br>&gt;&gt; From: "Simone Busoli" &lt;<a href="mailto:Simone.Busoli@...">Simone.Busoli@...</a>&gt;<br>
&gt;&gt; To: <a href="mailto:rabbitmq-discuss@...">rabbitmq-discuss@...</a><br>&gt;&gt; Sent: Wednesday, June 1, 2011 10:15:26 AM<br>&gt;&gt; Subject: Re: [rabbitmq-discuss] How to handle full exchange history<br>
&gt;&gt; delivery<br>&gt; <br>&gt;&gt; Thanks Jon, I misunderstood you probably.<br>&gt; <br>&gt;&gt; WRT this:<br>&gt; <br>&gt;&gt; When I was writing it, I thought it would be really easy to, when a<br>&gt;&gt; new queue is bound to this exchange, go fetch all the messages in<br>
&gt;&gt; that bucket and send them to the queue.<br>&gt; <br>&gt;&gt; I&rsquo;m not sure it is, that was my main question actually. How do you<br>&gt;&gt; figure out how to join the messages stored in the bucket and those<br>&gt;&gt; arriving live so that the last message you send to the queue from<br>
&gt;&gt; those picked up from the bucket is followed exactly by its logical<br>&gt;&gt; successor from those coming live?<br>&gt; <br>&gt;&gt; Say I have messages 1..100 being published to the exchange, each time<br>&gt;&gt; one arrives it&rsquo;s stored. The exchange has received the first 50<br>
&gt;&gt; messages and a client performs a binding of a queue to the exchange.<br>&gt;&gt; So you start pulling the messages from the storage and sending them<br>&gt;&gt; to the queue before forwarding any live messages. While you are<br>
&gt;&gt; pulling them out of the storage messages 51..60 are published to the<br>&gt;&gt; exchange, how do you deal with them? Do you always pull from the<br>&gt;&gt; storage? When do you stop pulling from the storage and relay to the<br>
&gt;&gt; queue directly what is being published to the exchange if you don&rsquo;t<br>&gt;&gt; know whether any new messages have arrived while you were pulling<br>&gt;&gt; from the storage?<br>&gt; <br>&gt;&gt; _______________________________________________________________<br>
&gt;&gt; Simone Busoli<br>&gt;&gt; Direzione risorse umane e segreteria generale &ndash; Sistemi informativi<br>&gt;&gt; gestione sportiva<br>&gt;&gt; Collaboratore<br>&gt;&gt; Ferrari S.p.A. Via Ascari 55/57, 41053, Maranello (MO); Italy<br>
&gt;&gt; T (+39) 0536-949.850<br>&gt;&gt; E <a href="mailto:simone.busoli <at> ferrari.com">simone.busoli@...</a> <a href="http://www.ferrari.com">www.ferrari.com</a><br>&gt; <br>&gt;&gt; From: Jon Brisbin [mailto:<a href="mailto:jon@...">jon@...</a>]<br>
&gt;&gt; Sent: Wednesday, June 01, 2011 4:59 PM<br>&gt;&gt; To: Busoli, Simone<br>&gt;&gt; Cc: <a href="mailto:rabbitmq-discuss@...">rabbitmq-discuss@...</a><br>&gt;&gt; Subject: Re: [rabbitmq-discuss] How to handle full exchange history<br>
&gt;&gt; delivery<br>&gt; <br>&gt;&gt; Maybe I wasn't being clear:<br>&gt; <br>&gt;&gt; I've already written a custom exchange that stores every message it<br>&gt;&gt; receives in Riak ( <a href="http://wiki.basho.com/">http://wiki.basho.com/</a> ). I'm going to be talking<br>
&gt;&gt; about it next week at Erlang Factory in London, as a matter of fact.<br>&gt;&gt; :)<br>&gt; <br>&gt;&gt; <a href="https://github.com/jbrisbin/riak-exchange">https://github.com/jbrisbin/riak-exchange</a><br>&gt; <br>
&gt;&gt; When I was writing it, I thought it would be really easy to, when a<br>&gt;&gt; new queue is bound to this exchange, go fetch all the messages in<br>&gt;&gt; that bucket and send them to the queue.<br>&gt; <br>&gt;&gt; Since I'm using Riak, the messages themselves are not all stored in<br>
&gt;&gt; memory.<br>&gt; <br>&gt;&gt; Thanks!<br>&gt; <br>&gt;&gt; Jon Brisbin<br>&gt;&gt; http//<a href="http://jbrisbin.com">jbrisbin.com</a><br>&gt; <br>&gt;&gt; ----- Original Message -----<br>&gt; <br>&gt;&gt; &gt; From: "Simone Busoli" &lt;<a href="mailto:Simone.Busoli@...m">Simone.Busoli@...</a>&gt;<br>
&gt;&gt; <br>&gt;&gt; &gt; To: <a href="mailto:rabbitmq-discuss@...bitmq.com">rabbitmq-discuss@...</a><br>&gt;&gt; <br>&gt;&gt; &gt; Sent: Wednesday, June 1, 2011 8:43:58 AM<br>&gt;&gt; <br>&gt;&gt; &gt; Subject: Re: [rabbitmq-discuss] How to handle full exchange history<br>
&gt;&gt; &gt; delivery<br>&gt;&gt; <br>&gt; <br>&gt;&gt; &gt; That&rsquo;s what I thought as well, although I&rsquo;m thinking about a couple<br>&gt;&gt; &gt; of issues about it:<br>&gt;&gt; <br>&gt; <br>&gt;&gt; &gt; &middot; Lack of knowledge about Erlang, which is the language I&rsquo;d have to<br>
&gt;&gt; &gt; use to build a custom exchange, I guess<br>&gt;&gt; <br>&gt;&gt; &gt; &middot; Amount of memory it would require to store the entire history in<br>&gt;&gt; &gt; memory. It needn&rsquo;t be in memory actually, but I&rsquo;m not sure what<br>
&gt;&gt; &gt; would be the recommended way if the storage has to be carried out<br>&gt;&gt; &gt; by<br>&gt;&gt; &gt; exchange itself. Mnesia perhaps, would it make sense?<br>&gt;&gt; <br>&gt; <br>&gt;&gt; &gt; _______________________________________________________ ________<br>
&gt;&gt; <br>&gt;&gt; &gt; Simone Busoli<br>&gt;&gt; <br>&gt;&gt; &gt; Direzione risorse umane e segreteria generale &ndash; Sistemi informativi<br>&gt;&gt; &gt; gestione sportiva<br>&gt;&gt; <br>&gt;&gt; &gt; Collaboratore<br>
&gt;&gt; <br>&gt;&gt; &gt; Ferrari S.p.A. Via Ascari 55/57, 41053, Maranello (MO); Italy<br>&gt;&gt; <br>&gt;&gt; &gt; T (+39) 0536-949.850<br>&gt;&gt; <br>&gt;&gt; &gt; E <a href="mailto:simone.busoli@...">simone.busoli@...</a> <a href="http://www.ferrari.com">www.ferrari.com</a><br>
&gt;&gt; <br>&gt; <br>&gt;&gt; &gt; From: Jon Brisbin [mailto:<a href="mailto:jon@...">jon@...</a>]<br>&gt;&gt; <br>&gt;&gt; &gt; Sent: Wednesday, June 01, 2011 3:38 PM<br>&gt;&gt; <br>&gt;&gt; &gt; To: Busoli, Simone<br>
&gt;&gt; <br>&gt;&gt; &gt; Cc: <a href="mailto:rabbitmq-discuss@...bitmq.com">rabbitmq-discuss@...</a><br>&gt;&gt; <br>&gt;&gt; &gt; Subject: Re: [rabbitmq-discuss] How to handle full exchange history<br>
&gt;&gt; &gt; delivery<br>&gt;&gt; <br>&gt; <br>&gt;&gt; &gt; I've been seriously considering adding this functionality to the<br>&gt;&gt; &gt; Riak-backed custom exchange. It was something I thought I would add<br>&gt;&gt; &gt; first time through but didn't.<br>
&gt;&gt; <br>&gt; <br>&gt;&gt; &gt; You'd have to use a custom exchange. Not sure if that's an issue in<br>&gt;&gt; &gt; your scenario or not.<br>&gt;&gt; <br>&gt; <br>&gt;&gt; &gt; Thanks!<br>&gt;&gt; <br>&gt; <br>
&gt;&gt; &gt; Jon Brisbin<br>&gt;&gt; <br>&gt;&gt; &gt; http//<a href="http://jbrisbin.com">jbrisbin.com</a><br>&gt;&gt; <br>&gt; <br>&gt;&gt; &gt; &gt; From: "Simone Busoli" &lt;<a href="mailto:Simone.Busoli <at> ferrari.com">Simone.Busoli@...</a>&gt;<br>
&gt;&gt; &gt; <br>&gt;&gt; <br>&gt;&gt; &gt; &gt; To: <a href="mailto:rabbitmq-discuss@...">rabbitmq-discuss@...</a><br>&gt;&gt; &gt; <br>&gt;&gt; <br>&gt;&gt; &gt; &gt; Sent: Wednesday, June 1, 2011 5:52:24 AM<br>
&gt;&gt; &gt; <br>&gt;&gt; <br>&gt;&gt; &gt; &gt; Subject: [rabbitmq-discuss] How to handle full exchange history<br>&gt;&gt; &gt; &gt; delivery<br>&gt;&gt; &gt; <br>&gt;&gt; <br>&gt; <br>&gt;&gt; &gt; &gt; Hi,<br>&gt;&gt; &gt; <br>
&gt;&gt; <br>&gt; <br>&gt;&gt; &gt; &gt; I am trying to achieve a scenario in which messages are published<br>&gt;&gt; &gt; &gt; from some data source into a RabbitMQ exchange and clients should<br>&gt;&gt; &gt; &gt; be<br>
&gt;&gt; &gt; &gt; able to receive the full history of messages flown through that<br>&gt;&gt; &gt; &gt; exchange, regardless of the exact moment in time when they<br>&gt;&gt; &gt; &gt; connect.<br>&gt;&gt; &gt; <br>&gt;&gt; <br>
&gt; <br>&gt;&gt; &gt; &gt; I&rsquo;ve been thinking to solve it by setting up a proxy which<br>&gt;&gt; &gt; &gt; subscribes<br>&gt;&gt; &gt; &gt; to the exchange where messages are published using its own queue,<br>&gt;&gt; &gt; &gt; keeps an internal storage of the messages and accepts requests<br>
&gt;&gt; &gt; &gt; from<br>&gt;&gt; &gt; &gt; clients.<br>&gt;&gt; &gt; <br>&gt;&gt; <br>&gt;&gt; &gt; &gt; When a request from a client arrives it just replays the full<br>&gt;&gt; &gt; &gt; message<br>&gt;&gt; &gt; &gt; history to the client by publishing the messages to the client<br>
&gt;&gt; &gt; &gt; queue<br>&gt;&gt; &gt; &gt; (supplied in the ReplyTo field), then binds the client queue to<br>&gt;&gt; &gt; &gt; the<br>&gt;&gt; &gt; &gt; messages exchange, so that the client from this moment onwards<br>
&gt;&gt; &gt; &gt; starts receiving the &ldquo;live&rdquo; messages, and is pretty much unaware<br>&gt;&gt; &gt; &gt; of<br>&gt;&gt; &gt; &gt; what&rsquo;s happened; it simply just receives the whole history.<br>&gt;&gt; &gt; <br>&gt;&gt; <br>
&gt; <br>&gt;&gt; &gt; &gt; Now I am under the impression that there&rsquo;s a fundamental flaw in<br>&gt;&gt; &gt; &gt; this<br>&gt;&gt; &gt; &gt; reasoning, since the proxy cannot know when the message history<br>&gt;&gt; &gt; &gt; has<br>
&gt;&gt; &gt; &gt; completed and he can switch the client to the live stream,<br>&gt;&gt; &gt; &gt; because<br>&gt;&gt; &gt; &gt; while it is replaying historical messages to the client other<br>&gt;&gt; &gt; &gt; live<br>
&gt;&gt; &gt; &gt; messages might be arriving on the proxy&rsquo;s queue, thus getting<br>&gt;&gt; &gt; &gt; into<br>&gt;&gt; &gt; &gt; an endless loop.<br>&gt;&gt; &gt; <br>&gt;&gt; <br>&gt; <br>&gt;&gt; &gt; &gt; I&rsquo;m probably missing something obvious, any advice about how<br>
&gt;&gt; &gt; &gt; others<br>&gt;&gt; &gt; &gt; would approach this scenario is appreciated.<br>&gt;&gt; &gt; <br>&gt;&gt; <br>&gt; <br>&gt;&gt; &gt; &gt; Simone<br>&gt;&gt; &gt; <br>&gt;&gt; <br>&gt;&gt; &gt; &gt; _________________________________________________________________________________________________________________<br>
&gt;&gt; &gt; <br>&gt;&gt; <br>&gt;&gt; &gt; &gt; Questo messaggio e da intendersi esclusivamente ad uso del<br>&gt;&gt; &gt; &gt; destinatario e puo contenere informazioni che sono di natura<br>&gt;&gt; &gt; &gt; privilegiata, confidenziale<br>
&gt;&gt; &gt; <br>&gt;&gt; <br>&gt;&gt; &gt; &gt; o non divulgabile secondo le leggi vigenti. Se il lettore del<br>&gt;&gt; &gt; &gt; presente messaggio non e il destinatario designato, o il<br>&gt;&gt; &gt; &gt; dipendente/agente responsabile<br>
&gt;&gt; &gt; <br>&gt;&gt; <br>&gt;&gt; &gt; &gt; per la consegna del messaggio al destinatario designato, si<br>&gt;&gt; &gt; &gt; informa<br>&gt;&gt; &gt; &gt; che ogni disseminazione, distribuzione o copiatura di questa<br>
&gt;&gt; &gt; &gt; comunicazione e<br>&gt;&gt; &gt; <br>&gt;&gt; <br>&gt;&gt; &gt; &gt; strettamente proibita anche ai sensi del decreto legislativo<br>&gt;&gt; &gt; &gt; 196/03<br>&gt;&gt; &gt; &gt; .<br>&gt;&gt; &gt; &gt; Se avete ricevuto questo messaggio per errore, vi preghiamo di<br>
&gt;&gt; &gt; &gt; notificarcelo<br>&gt;&gt; &gt; <br>&gt;&gt; <br>&gt;&gt; &gt; &gt; immediatamente a mezzo e-mail di risposta e successivamente di<br>&gt;&gt; &gt; &gt; procedere alla cancellazione di questa e-mail e relativi allegati<br>
&gt;&gt; &gt; &gt; dal vostro sistema.<br>&gt;&gt; &gt; <br>&gt;&gt; <br>&gt;&gt; &gt; &gt; _________________________________________________________________________________________________________________<br>&gt;&gt; &gt; <br>
&gt;&gt; <br>&gt;&gt; &gt; &gt; This message is intended only for the use of the addressee and<br>&gt;&gt; &gt; &gt; may<br>&gt;&gt; &gt; &gt; contain information that is privileged, confidential and exempt<br>&gt;&gt; &gt; &gt; from<br>
&gt;&gt; &gt; <br>&gt;&gt; <br>&gt;&gt; &gt; &gt; disclosure under applicable law. If the reader of this message is<br>&gt;&gt; &gt; &gt; not<br>&gt;&gt; &gt; &gt; the intended recipient, or the employee or agent responsible for<br>
&gt;&gt; &gt; &gt; delivering the<br>&gt;&gt; &gt; <br>&gt;&gt; <br>&gt;&gt; &gt; &gt; message to the intended recipient, you are hereby notified that<br>&gt;&gt; &gt; &gt; any<br>&gt;&gt; &gt; &gt; dissemination, distribution or copying of this communication is<br>
&gt;&gt; &gt; &gt; strictly<br>&gt;&gt; &gt; <br>&gt;&gt; <br>&gt;&gt; &gt; &gt; prohibited. If you have received this e-mail in error, please<br>&gt;&gt; &gt; &gt; notify<br>&gt;&gt; &gt; &gt; us immediately by return e-mail and delete this e-mail and all<br>
&gt;&gt; &gt; &gt; attachments from<br>&gt;&gt; &gt; <br>&gt;&gt; <br>&gt;&gt; &gt; &gt; your system.<br>&gt;&gt; &gt; <br>&gt;&gt; <br>&gt;&gt; &gt; &gt; _________________________________________________________________________________________________________________<br>
&gt;&gt; &gt; <br>&gt;&gt; <br>&gt; <br>&gt;&gt; &gt; &gt; _______________________________________________<br>&gt;&gt; &gt; <br>&gt;&gt; <br>&gt;&gt; &gt; &gt; rabbitmq-discuss mailing list<br>&gt;&gt; &gt; <br>&gt;&gt; <br>
&gt;&gt; &gt; &gt; <a href="mailto:rabbitmq-discuss@...">rabbitmq-discuss@...</a><br>&gt;&gt; &gt; <br>&gt;&gt; <br>&gt;&gt; &gt; &gt; <a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
&gt;&gt; &gt; <br>&gt;&gt; <br>&gt;&gt; &gt; _________________________________________________________________________________________________________________<br>&gt;&gt; <br>&gt;&gt; &gt; Questo messaggio &egrave; da intendersi esclusivamente ad uso del<br>
&gt;&gt; &gt; destinatario e pu&ograve; contenere informazioni che sono di natura<br>&gt;&gt; &gt; privilegiata, confidenziale<br>&gt;&gt; <br>&gt;&gt; &gt; o non divulgabile secondo le leggi vigenti. Se il lettore del<br>&gt;&gt; &gt; presente messaggio non &egrave; il destinatario designato, o il<br>
&gt;&gt; &gt; dipendente/agente responsabile<br>&gt;&gt; <br>&gt;&gt; &gt; per la consegna del messaggio al destinatario designato, si informa<br>&gt;&gt; &gt; che ogni disseminazione, distribuzione o copiatura di questa<br>
&gt;&gt; &gt; comunicazione &egrave;<br>&gt;&gt; <br>&gt;&gt; &gt; strettamente proibita anche ai sensi del decreto legislativo 196/03<br>&gt;&gt; &gt; .<br>&gt;&gt; &gt; Se avete ricevuto questo messaggio per errore, vi preghiamo di<br>
&gt;&gt; &gt; notificarcelo<br>&gt;&gt; <br>&gt;&gt; &gt; immediatamente a mezzo e-mail di risposta e successivamente di<br>&gt;&gt; &gt; procedere alla cancellazione di questa e-mail e relativi allegati<br>&gt;&gt; &gt; dal vostro sistema.<br>
&gt;&gt; <br>&gt;&gt; &gt; _________________________________________________________________________________________________________________<br>&gt;&gt; <br>&gt;&gt; &gt; This message is intended only for the use of the addressee and may<br>
&gt;&gt; &gt; contain information that is privileged, confidential and exempt<br>&gt;&gt; &gt; from<br>&gt;&gt; <br>&gt;&gt; &gt; disclosure under applicable law. If the reader of this message is<br>&gt;&gt; &gt; not<br>&gt;&gt; &gt; the intended recipient, or the employee or agent responsible for<br>
&gt;&gt; &gt; delivering the<br>&gt;&gt; <br>&gt;&gt; &gt; message to the intended recipient, you are hereby notified that any<br>&gt;&gt; &gt; dissemination, distribution or copying of this communication is<br>&gt;&gt; &gt; strictly<br>
&gt;&gt; <br>&gt;&gt; &gt; prohibited. If you have received this e-mail in error, please<br>&gt;&gt; &gt; notify<br>&gt;&gt; &gt; us immediately by return e-mail and delete this e-mail and all<br>&gt;&gt; &gt; attachments from<br>
&gt;&gt; <br>&gt;&gt; &gt; your system.<br>&gt;&gt; <br>&gt;&gt; &gt; _________________________________________________________________________________________________________________<br>&gt;&gt; <br>&gt; <br>&gt;&gt; &gt; _______________________________________________<br>
&gt;&gt; <br>&gt;&gt; &gt; rabbitmq-discuss mailing list<br>&gt;&gt; <br>&gt;&gt; &gt; <a href="mailto:rabbitmq-discuss@...">rabbitmq-discuss@...</a><br>&gt;&gt; <br>&gt;&gt; &gt; <a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
&gt;&gt; <br>&gt;&gt; _________________________________________________________________________________________________________________<br>&gt;&gt; Questo messaggio &egrave; da intendersi esclusivamente ad uso del<br>&gt;&gt; destinatario e pu&ograve; contenere informazioni che sono di natura<br>
&gt;&gt; privilegiata, confidenziale<br>&gt;&gt; o non divulgabile secondo le leggi vigenti. Se il lettore del<br>&gt;&gt; presente messaggio non &egrave; il destinatario designato, o il<br>&gt;&gt; dipendente/agente responsabile<br>
&gt;&gt; per la consegna del messaggio al destinatario designato, si informa<br>&gt;&gt; che ogni disseminazione, distribuzione o copiatura di questa<br>&gt;&gt; comunicazione &egrave;<br>&gt;&gt; strettamente proibita anche ai sensi del decreto legislativo 196/03 .<br>
&gt;&gt; Se avete ricevuto questo messaggio per errore, vi preghiamo di<br>&gt;&gt; notificarcelo<br>&gt;&gt; immediatamente a mezzo e-mail di risposta e successivamente di<br>&gt;&gt; procedere alla cancellazione di questa e-mail e relativi allegati<br>
&gt;&gt; dal vostro sistema.<br>&gt;&gt; _________________________________________________________________________________________________________________<br>&gt;&gt; This message is intended only for the use of the addressee and may<br>
&gt;&gt; contain information that is privileged, confidential and exempt from<br>&gt;&gt; disclosure under applicable law. If the reader of this message is not<br>&gt;&gt; the intended recipient, or the employee or agent responsible for<br>
&gt;&gt; delivering the<br>&gt;&gt; message to the intended recipient, you are hereby notified that any<br>&gt;&gt; dissemination, distribution or copying of this communication is<br>&gt;&gt; strictly<br>&gt;&gt; prohibited. If you have received this e-mail in error, please notify<br>
&gt;&gt; us immediately by return e-mail and delete this e-mail and all<br>&gt;&gt; attachments from<br>&gt;&gt; your system.<br>&gt;&gt; _________________________________________________________________________________________________________________<br>
&gt; <br>&gt;&gt; _______________________________________________<br>&gt;&gt; rabbitmq-discuss mailing list<br>&gt;&gt; <a href="mailto:rabbitmq-discuss@...">rabbitmq-discuss@...</a><br>&gt;&gt; <a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
</div>
</div>
Permalink | Reply |
headers
Michael Bridgen | 2 Jun 2011 12:40
Favicon
Gravatar

Re: How to handle full exchange history delivery

> WRT this:
>
> /When I was writing it, I thought it would be really easy to, when a new
> queue is bound to this exchange, go fetch all the messages in that
> bucket and send them to the queue.///
>
> I’m not sure it is, that was my main question actually. How do you
> figure out how to join the messages stored in the bucket and those
> arriving live so that the last message you send to the queue from those
> picked up from the bucket is followed exactly by its logical successor
> from those coming live?
>
> Say I have messages 1..100 being published to the exchange, each time
> one arrives it’s stored. The exchange has received the first 50 messages
> and a client performs a binding of a queue to the exchange. So you start
> pulling the messages from the storage and sending them to the queue
> before forwarding any live messages. While you are pulling them out of
> the storage messages 51..60 are published to the exchange, how do you
> deal with them? Do you always pull from the storage? When do you stop
> pulling from the storage and relay to the queue directly what is being
> published to the exchange if you don’t know whether any new messages
> have arrived while you were pulling from the storage?

My inclination with this and similar use cases* is that they deserve 
their own content class (alongside 'exchange', 'queue', etc.).

By way of illustration: the last-value-caching exchange, though it was 
developed as a verification of the exchange type mechanism, is rather 
awkward; in the main because it is encoded in the existing AMQP 
operations (bind, consume), and these aren't the right vocabulary for 
the intended use.  See the "Limitations" section of
https://github.com/squaremo/rabbitmq-lvc-plugin.

There's two natural generalisations of LVC: one is to keep an arbitrary 
number of values around, and the other is to make the hold higher-order 
(i.e., instead of replaying values, replay some aggregate of values). 
Neither of these really fit in AMQP as it stands, not in the available 
methods or their semantics.

Anyway.  So that this is helpful and not just me kicking AMQP, here is a 
suggestion: if the consumers are made to co-operate just slightly you 
can get your proxy idea to work. Just use different queues for the 
replay and the live.  Subscribe the live first and note the most recent 
message you know about at that time; then publish your saved messages 
through to that message into the replay queue.  The consumer, when it 
hears back from the proxy, drains the replay queue then starts on the 
live queue.

Michael

*I call the category "value+updates", and the synchronisation problem 
you describe above is the essence of it.

> _______________________________________________________________
>
> *Simone Busoli*
> Direzione risorse umane e segreteria generale – Sistemi informativi
> gestione sportiva
> /Collaboratore/
>
> *Ferrari S.p.A. * Via Ascari 55/57, 41053, Maranello (MO); Italy
> *T* (+39) 0536-949.850
> *E* _simone.busoli <at> ferrari.com <mailto:simone.busoli <at> ferrari.com>_
> _www.ferrari.com <http://www.ferrari.com/>_
>
> *From:* Jon Brisbin [mailto:jon <at> jbrisbin.com]
> *Sent:* Wednesday, June 01, 2011 4:59 PM
> *To:* Busoli, Simone
> *Cc:* rabbitmq-discuss <at> lists.rabbitmq.com
> *Subject:* Re: [rabbitmq-discuss] How to handle full exchange history
> delivery
>
> Maybe I wasn't being clear:
>
> I've already written a custom exchange that stores every message it
> receives in Riak (http://wiki.basho.com/). I'm going to be talking about
> it next week at Erlang Factory in London, as a matter of fact. :)
>
> https://github.com/jbrisbin/riak-exchange
>
> When I was writing it, I thought it would be really easy to, when a new
> queue is bound to this exchange, go fetch all the messages in that
> bucket and send them to the queue.
>
> Since I'm using Riak, the messages themselves are not all stored in memory.
>
> Thanks!
>
> Jon Brisbin
> http//jbrisbin.com
>
> ------------------------------------------------------------------------
>
>     *From: *"Simone Busoli" <Simone.Busoli <at> ferrari.com>
>     *To: *rabbitmq-discuss <at> lists.rabbitmq.com
>     *Sent: *Wednesday, June 1, 2011 8:43:58 AM
>     *Subject: *Re: [rabbitmq-discuss] How to handle full exchange
>     history delivery
>
>
>     That’s what I thought as well, although I’m thinking about a couple
>     of issues about it:
>
>     · Lack of knowledge about Erlang, which is the language I’d have to
>     use to build a custom exchange, I guess
>
>     · Amount of memory it would require to store the entire history in
>     memory. It needn’t be in memory actually, but I’m not sure what
>     would be the recommended way if the storage has to be carried out by
>     exchange itself. Mnesia perhaps, would it make sense?
>
>     _______________________________________________________________
>
>     *Simone Busoli*
>     Direzione risorse umane e segreteria generale – Sistemi informativi
>     gestione sportiva
>     /Collaboratore/
>
>     *Ferrari S.p.A. * Via Ascari 55/57, 41053, Maranello (MO); Italy
>     *T* (+39) 0536-949.850
>     *E* _simone.busoli <at> ferrari.com <mailto:simone.busoli <at> ferrari.com>_
>     _www.ferrari.com <http://www.ferrari.com/>_
>
>     *From:* Jon Brisbin [mailto:jon <at> jbrisbin.com]
>     *Sent:* Wednesday, June 01, 2011 3:38 PM
>     *To:* Busoli, Simone
>     *Cc:* rabbitmq-discuss <at> lists.rabbitmq.com
>     *Subject:* Re: [rabbitmq-discuss] How to handle full exchange
>     history delivery
>
>     I've been seriously considering adding this functionality to the
>     Riak-backed custom exchange. It was something I thought I would add
>     first time through but didn't.
>
>     You'd have to use a custom exchange. Not sure if that's an issue in
>     your scenario or not.
>
>
>     Thanks!
>
>     Jon Brisbin
>     http//jbrisbin.com
>
>     ------------------------------------------------------------------------
>
>         *From: *"Simone Busoli" <Simone.Busoli <at> ferrari.com>
>         *To: *rabbitmq-discuss <at> lists.rabbitmq.com
>         *Sent: *Wednesday, June 1, 2011 5:52:24 AM
>         *Subject: *[rabbitmq-discuss] How to handle full exchange
>         history delivery
>
>         Hi,
>
>         I am trying to achieve a scenario in which messages are
>         published from some data source into a RabbitMQ exchange and
>         clients should be able to receive the full history of messages
>         flown through that exchange, regardless of the exact moment in
>         time when they connect.
>
>         I’ve been thinking to solve it by setting up a proxy which
>         subscribes to the exchange where messages are published using
>         its own queue, keeps an internal storage of the messages and
>         accepts requests from clients.
>
>         When a request from a client arrives it just replays the full
>         message history to the client by publishing the messages to the
>         client queue (supplied in the ReplyTo field), *then* binds the
>         client queue to the messages exchange, so that the client from
>         this moment onwards starts receiving the “live” messages, and is
>         pretty much unaware of what’s happened; it simply just receives
>         the whole history.
>
>         Now I am under the impression that there’s a fundamental flaw in
>         this reasoning, since the proxy cannot know when the message
>         history has completed and he can switch the client to the live
>         stream, because while it is replaying historical messages to the
>         client other live messages might be arriving on the proxy’s
>         queue, thus getting into an endless loop.
>
>         I’m probably missing something obvious, any advice about how
>         others would approach this scenario is appreciated.
>
>         Simone
>
>         _________________________________________________________________________________________________________________
>         Questo messaggio e da intendersi esclusivamente ad uso del
>         destinatario e puo contenere informazioni che sono di natura
>         privilegiata, confidenziale
>         o non divulgabile secondo le leggi vigenti. Se il lettore del
>         presente messaggio non e il destinatario designato, o il
>         dipendente/agente responsabile
>         per la consegna del messaggio al destinatario designato, si
>         informa che ogni disseminazione, distribuzione o copiatura di
>         questa comunicazione e
>         strettamente proibita anche ai sensi del decreto legislativo
>         196/03 . Se avete ricevuto questo messaggio per errore, vi
>         preghiamo di notificarcelo
>         immediatamente a mezzo e-mail di risposta e successivamente di
>         procedere alla cancellazione di questa e-mail e relativi
>         allegati dal vostro sistema.
>         _________________________________________________________________________________________________________________
>         This message is intended only for the use of the addressee and
>         may contain information that is privileged, confidential and
>         exempt from
>         disclosure under applicable law. If the reader of this message
>         is not the intended recipient, or the employee or agent
>         responsible for delivering the
>         message to the intended recipient, you are hereby notified that
>         any dissemination, distribution or copying of this communication
>         is strictly
>         prohibited. If you have received this e-mail in error, please
>         notify us immediately by return e-mail and delete this e-mail
>         and all attachments from
>         your system.
>         _________________________________________________________________________________________________________________
>
>
>         _______________________________________________
>         rabbitmq-discuss mailing list
>         rabbitmq-discuss <at> lists.rabbitmq.com
>         <mailto:rabbitmq-discuss <at> lists.rabbitmq.com>
>         https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>     _________________________________________________________________________________________________________________
>     Questo messaggio è da intendersi esclusivamente ad uso del
>     destinatario e può contenere informazioni che sono di natura
>     privilegiata, confidenziale
>     o non divulgabile secondo le leggi vigenti. Se il lettore del
>     presente messaggio non è il destinatario designato, o il
>     dipendente/agente responsabile
>     per la consegna del messaggio al destinatario designato, si informa
>     che ogni disseminazione, distribuzione o copiatura di questa
>     comunicazione è
>     strettamente proibita anche ai sensi del decreto legislativo 196/03
>     . Se avete ricevuto questo messaggio per errore, vi preghiamo di
>     notificarcelo
>     immediatamente a mezzo e-mail di risposta e successivamente di
>     procedere alla cancellazione di questa e-mail e relativi allegati
>     dal vostro sistema.
>     _________________________________________________________________________________________________________________
>     This message is intended only for the use of the addressee and may
>     contain information that is privileged, confidential and exempt from
>     disclosure under applicable law. If the reader of this message is
>     not the intended recipient, or the employee or agent responsible for
>     delivering the
>     message to the intended recipient, you are hereby notified that any
>     dissemination, distribution or copying of this communication is
>     strictly
>     prohibited. If you have received this e-mail in error, please notify
>     us immediately by return e-mail and delete this e-mail and all
>     attachments from
>     your system.
>     _________________________________________________________________________________________________________________
>
>
>     _______________________________________________
>     rabbitmq-discuss mailing list
>     rabbitmq-discuss <at> lists.rabbitmq.com
>     <mailto:rabbitmq-discuss <at> lists.rabbitmq.com>
>     https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
> _________________________________________________________________________________________________________________
> Questo messaggio è da intendersi esclusivamente ad uso del destinatario
> e può contenere informazioni che sono di natura privilegiata, confidenziale
> o non divulgabile secondo le leggi vigenti. Se il lettore del presente
> messaggio non è il destinatario designato, o il dipendente/agente
> responsabile
> per la consegna del messaggio al destinatario designato, si informa che
> ogni disseminazione, distribuzione o copiatura di questa comunicazione è
> strettamente proibita anche ai sensi del decreto legislativo 196/03 . Se
> avete ricevuto questo messaggio per errore, vi preghiamo di notificarcelo
> immediatamente a mezzo e-mail di risposta e successivamente di procedere
> alla cancellazione di questa e-mail e relativi allegati dal vostro sistema.
> _________________________________________________________________________________________________________________
> This message is intended only for the use of the addressee and may
> contain information that is privileged, confidential and exempt from
> disclosure under applicable law. If the reader of this message is not
> the intended recipient, or the employee or agent responsible for
> delivering the
> message to the intended recipient, you are hereby notified that any
> dissemination, distribution or copying of this communication is strictly
> prohibited. If you have received this e-mail in error, please notify us
> immediately by return e-mail and delete this e-mail and all attachments
> from
> your system.
> _________________________________________________________________________________________________________________
>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss <at> lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss <at> lists.rabbitmq.com
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
Permalink | Reply |
headers
Simone Busoli | 2 Jun 2011 15:43
Picon
Gravatar

Re: How to handle full exchange history delivery

Hi Michael, now I realize this scenario is not one RabbitMQ is supposed to handle out of the box, thanks for clearing it out. Nonetheless I'm afraid building support for it inside the broker is not something I can afford, therefore I think I will build something at a higher level externally, as you suggested with some collaboration from the client side.

On Jun 2, 2011 12:30 PM, "Michael Bridgen" <mikeb-mQ7lE4MOPXtWk0Htik3J/w@public.gmane.org> wrote:
>> WRT this:
>>
>> /When I was writing it, I thought it would be really easy to, when a new
>> queue is bound to this exchange, go fetch all the messages in that
>> bucket and send them to the queue.///
>>
>> I’m not sure it is, that was my main question actually. How do you
>> figure out how to join the messages stored in the bucket and those
>> arriving live so that the last message you send to the queue from those
>> picked up from the bucket is followed exactly by its logical successor
>> from those coming live?
>>
>> Say I have messages 1..100 being published to the exchange, each time
>> one arrives it’s stored. The exchange has received the first 50 messages
>> and a client performs a binding of a queue to the exchange. So you start
>> pulling the messages from the storage and sending them to the queue
>> before forwarding any live messages. While you are pulling them out of
>> the storage messages 51..60 are published to the exchange, how do you
>> deal with them? Do you always pull from the storage? When do you stop
>> pulling from the storage and relay to the queue directly what is being
>> published to the exchange if you don’t know whether any new messages
>> have arrived while you were pulling from the storage?
>
> My inclination with this and similar use cases* is that they deserve
> their own content class (alongside 'exchange', 'queue', etc.).
>
> By way of illustration: the last-value-caching exchange, though it was
> developed as a verification of the exchange type mechanism, is rather
> awkward; in the main because it is encoded in the existing AMQP
> operations (bind, consume), and these aren't the right vocabulary for
> the intended use. See the "Limitations" section of
> https://github.com/squaremo/rabbitmq-lvc-plugin.
>
> There's two natural generalisations of LVC: one is to keep an arbitrary
> number of values around, and the other is to make the hold higher-order
> (i.e., instead of replaying values, replay some aggregate of values).
> Neither of these really fit in AMQP as it stands, not in the available
> methods or their semantics.
>
> Anyway. So that this is helpful and not just me kicking AMQP, here is a
> suggestion: if the consumers are made to co-operate just slightly you
> can get your proxy idea to work. Just use different queues for the
> replay and the live. Subscribe the live first and note the most recent
> message you know about at that time; then publish your saved messages
> through to that message into the replay queue. The consumer, when it
> hears back from the proxy, drains the replay queue then starts on the
> live queue.
>
>
> Michael
>
> *I call the category "value+updates", and the synchronisation problem
> you describe above is the essence of it.
>
>> _______________________________________________________________
>>
>> *Simone Busoli*
>> Direzione risorse umane e segreteria generale – Sistemi informativi
>> gestione sportiva
>> /Collaboratore/
>>
>> *Ferrari S.p.A. * Via Ascari 55/57, 41053, Maranello (MO); Italy
>> *T* (+39) 0536-949.850
>> *E* _simone.busoli-sKiPNUh8iG9BDgjK7y7TUQ@public.gmane.org <mailto:simone.busoli-sKiPNUh8iG9BDgjK7y7TUQ@public.gmane.org>_
>> _www.ferrari.com <http://www.ferrari.com/>_
>>
>> *From:* Jon Brisbin [mailto:jon-foCUeXAHXPNWk0Htik3J/w@public.gmane.org]
>> *Sent:* Wednesday, June 01, 2011 4:59 PM
>> *To:* Busoli, Simone
>> *Cc:* rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org
>> *Subject:* Re: [rabbitmq-discuss] How to handle full exchange history
>> delivery
>>
>> Maybe I wasn't being clear:
>>
>> I've already written a custom exchange that stores every message it
>> receives in Riak (http://wiki.basho.com/). I'm going to be talking about
>> it next week at Erlang Factory in London, as a matter of fact. :)
>>
>> https://github.com/jbrisbin/riak-exchange
>>
>> When I was writing it, I thought it would be really easy to, when a new
>> queue is bound to this exchange, go fetch all the messages in that
>> bucket and send them to the queue.
>>
>> Since I'm using Riak, the messages themselves are not all stored in memory.
>>
>> Thanks!
>>
>> Jon Brisbin
>> http//jbrisbin.com
>>
>> ------------------------------------------------------------------------
>>
>> *From: *"Simone Busoli" <Simone.Busoli-sKiPNUh8iG9BDgjK7y7TUQ@public.gmane.org>
>> *To: *rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org
>> *Sent: *Wednesday, June 1, 2011 8:43:58 AM
>> *Subject: *Re: [rabbitmq-discuss] How to handle full exchange
>> history delivery
>>
>>
>> That’s what I thought as well, although I’m thinking about a couple
>> of issues about it:
>>
>> · Lack of knowledge about Erlang, which is the language I’d have to
>> use to build a custom exchange, I guess
>>
>> · Amount of memory it would require to store the entire history in
>> memory. It needn’t be in memory actually, but I’m not sure what
>> would be the recommended way if the storage has to be carried out by
>> exchange itself. Mnesia perhaps, would it make sense?
>>
>> _______________________________________________________________
>>
>> *Simone Busoli*
>> Direzione risorse umane e segreteria generale – Sistemi informativi
>> gestione sportiva
>> /Collaboratore/
>>
>> *Ferrari S.p.A. * Via Ascari 55/57, 41053, Maranello (MO); Italy
>> *T* (+39) 0536-949.850
>> *E* _simone.busoli-sKiPNUh8iG9BDgjK7y7TUQ@public.gmane.org <mailto:simone.busoli-sKiPNUh8iG9BDgjK7y7TUQ@public.gmane.org>_
>> _www.ferrari.com <http://www.ferrari.com/>_
>>
>> *From:* Jon Brisbin [mailto:jon-foCUeXAHXPNWk0Htik3J/w@public.gmane.org]
>> *Sent:* Wednesday, June 01, 2011 3:38 PM
>> *To:* Busoli, Simone
>> *Cc:* rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org
>> *Subject:* Re: [rabbitmq-discuss] How to handle full exchange
>> history delivery
>>
>> I've been seriously considering adding this functionality to the
>> Riak-backed custom exchange. It was something I thought I would add
>> first time through but didn't.
>>
>> You'd have to use a custom exchange. Not sure if that's an issue in
>> your scenario or not.
>>
>>
>> Thanks!
>>
>> Jon Brisbin
>> http//jbrisbin.com
>>
>> ------------------------------------------------------------------------
>>
>> *From: *"Simone Busoli" <Simone.Busoli <at> ferrari.com>
>> *To: *rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org
>> *Sent: *Wednesday, June 1, 2011 5:52:24 AM
>> *Subject: *[rabbitmq-discuss] How to handle full exchange
>> history delivery
>>
>> Hi,
>>
>> I am trying to achieve a scenario in which messages are
>> published from some data source into a RabbitMQ exchange and
>> clients should be able to receive the full history of messages
>> flown through that exchange, regardless of the exact moment in
>> time when they connect.
>>
>> I’ve been thinking to solve it by setting up a proxy which
>> subscribes to the exchange where messages are published using
>> its own queue, keeps an internal storage of the messages and
>> accepts requests from clients.
>>
>> When a request from a client arrives it just replays the full
>> message history to the client by publishing the messages to the
>> client queue (supplied in the ReplyTo field), *then* binds the
>> client queue to the messages exchange, so that the client from
>> this moment onwards starts receiving the “live” messages, and is
>> pretty much unaware of what’s happened; it simply just receives
>> the whole history.
>>
>> Now I am under the impression that there’s a fundamental flaw in
>> this reasoning, since the proxy cannot know when the message
>> history has completed and he can switch the client to the live
>> stream, because while it is replaying historical messages to the
>> client other live messages might be arriving on the proxy’s
>> queue, thus getting into an endless loop.
>>
>> I’m probably missing something obvious, any advice about how
>> others would approach this scenario is appreciated.
>>
>> Simone
>>
>> _________________________________________________________________________________________________________________
>> Questo messaggio e da intendersi esclusivamente ad uso del
>> destinatario e puo contenere informazioni che sono di natura
>> privilegiata, confidenziale
>> o non divulgabile secondo le leggi vigenti. Se il lettore del
>> presente messaggio non e il destinatario designato, o il
>> dipendente/agente responsabile
>> per la consegna del messaggio al destinatario designato, si
>> informa che ogni disseminazione, distribuzione o copiatura di
>> questa comunicazione e
>> strettamente proibita anche ai sensi del decreto legislativo
>> 196/03 . Se avete ricevuto questo messaggio per errore, vi
>> preghiamo di notificarcelo
>> immediatamente a mezzo e-mail di risposta e successivamente di
>> procedere alla cancellazione di questa e-mail e relativi
>> allegati dal vostro sistema.
>> _________________________________________________________________________________________________________________
>> This message is intended only for the use of the addressee and
>> may contain information that is privileged, confidential and
>> exempt from
>> disclosure under applicable law. If the reader of this message
>> is not the intended recipient, or the employee or agent
>> responsible for delivering the
>> message to the intended recipient, you are hereby notified that
>> any dissemination, distribution or copying of this communication
>> is strictly
>> prohibited. If you have received this e-mail in error, please
>> notify us immediately by return e-mail and delete this e-mail
>> and all attachments from
>> your system.
>> _________________________________________________________________________________________________________________
>>
>>
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss-ETbvJ2rUIr6rLlSb+tPU7A@public.gmane.orgitmq.com
>> <mailto:rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org>
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>> _________________________________________________________________________________________________________________
>> Questo messaggio è da intendersi esclusivamente ad uso del
>> destinatario e può contenere informazioni che sono di natura
>> privilegiata, confidenziale
>> o non divulgabile secondo le leggi vigenti. Se il lettore del
>> presente messaggio non è il destinatario designato, o il
>> dipendente/agente responsabile
>> per la consegna del messaggio al destinatario designato, si informa
>> che ogni disseminazione, distribuzione o copiatura di questa
>> comunicazione è
>> strettamente proibita anche ai sensi del decreto legislativo 196/03
>> . Se avete ricevuto questo messaggio per errore, vi preghiamo di
>> notificarcelo
>> immediatamente a mezzo e-mail di risposta e successivamente di
>> procedere alla cancellazione di questa e-mail e relativi allegati
>> dal vostro sistema.
>> _________________________________________________________________________________________________________________
>> This message is intended only for the use of the addressee and may
>> contain information that is privileged, confidential and exempt from
>> disclosure under applicable law. If the reader of this message is
>> not the intended recipient, or the employee or agent responsible for
>> delivering the
>> message to the intended recipient, you are hereby notified that any
>> dissemination, distribution or copying of this communication is
>> strictly
>> prohibited. If you have received this e-mail in error, please notify
>> us immediately by return e-mail and delete this e-mail and all
>> attachments from
>> your system.
>> _________________________________________________________________________________________________________________
>>
>>
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org
>> <mailto:rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org>
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>> _________________________________________________________________________________________________________________
>> Questo messaggio è da intendersi esclusivamente ad uso del destinatario
>> e può contenere informazioni che sono di natura privilegiata, confidenziale
>> o non divulgabile secondo le leggi vigenti. Se il lettore del presente
>> messaggio non è il destinatario designato, o il dipendente/agente
>> responsabile
>> per la consegna del messaggio al destinatario designato, si informa che
>> ogni disseminazione, distribuzione o copiatura di questa comunicazione è
>> strettamente proibita anche ai sensi del decreto legislativo 196/03 . Se
>> avete ricevuto questo messaggio per errore, vi preghiamo di notificarcelo
>> immediatamente a mezzo e-mail di risposta e successivamente di procedere
>> alla cancellazione di questa e-mail e relativi allegati dal vostro sistema.
>> _________________________________________________________________________________________________________________
>> This message is intended only for the use of the addressee and may
>> contain information that is privileged, confidential and exempt from
>> disclosure under applicable law. If the reader of this message is not
>> the intended recipient, or the employee or agent responsible for
>> delivering the
>> message to the intended recipient, you are hereby notified that any
>> dissemination, distribution or copying of this communication is strictly
>> prohibited. If you have received this e-mail in error, please notify us
>> immediately by return e-mail and delete this e-mail and all attachments
>> from
>> your system.
>> _________________________________________________________________________________________________________________
>>
>>
>>
>> _______________________________________________
>> rabbitmq-discuss mailing list
>> rabbitmq-discuss <at> lists.rabbitmq.com
>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
<div>
<p>Hi Michael, now I realize this scenario is not one RabbitMQ is supposed to handle out of the box, thanks for clearing it out. Nonetheless I'm afraid building support for it inside the broker is not something I can afford, therefore I think I will build something at a higher level externally, as you suggested with some collaboration from the client side.</p>

<div class="gmail_quote">On Jun 2, 2011 12:30 PM, "Michael Bridgen" &lt;<a href="mailto:mikeb@...">mikeb@...</a>&gt; wrote:<br type="attribution">&gt;&gt; WRT this:<br>&gt;&gt;<br>&gt;&gt; /When I was writing it, I thought it would be really easy to, when a new<br>
&gt;&gt; queue is bound to this exchange, go fetch all the messages in that<br>&gt;&gt; bucket and send them to the queue.///<br>&gt;&gt;<br>&gt;&gt; I&rsquo;m not sure it is, that was my main question actually. How do you<br>&gt;&gt; figure out how to join the messages stored in the bucket and those<br>
&gt;&gt; arriving live so that the last message you send to the queue from those<br>&gt;&gt; picked up from the bucket is followed exactly by its logical successor<br>&gt;&gt; from those coming live?<br>&gt;&gt;<br>&gt;&gt; Say I have messages 1..100 being published to the exchange, each time<br>
&gt;&gt; one arrives it&rsquo;s stored. The exchange has received the first 50 messages<br>&gt;&gt; and a client performs a binding of a queue to the exchange. So you start<br>&gt;&gt; pulling the messages from the storage and sending them to the queue<br>
&gt;&gt; before forwarding any live messages. While you are pulling them out of<br>&gt;&gt; the storage messages 51..60 are published to the exchange, how do you<br>&gt;&gt; deal with them? Do you always pull from the storage? When do you stop<br>
&gt;&gt; pulling from the storage and relay to the queue directly what is being<br>&gt;&gt; published to the exchange if you don&rsquo;t know whether any new messages<br>&gt;&gt; have arrived while you were pulling from the storage?<br>
&gt; <br>&gt; My inclination with this and similar use cases* is that they deserve <br>&gt; their own content class (alongside 'exchange', 'queue', etc.).<br>&gt; <br>&gt; By way of illustration: the last-value-caching exchange, though it was <br>
&gt; developed as a verification of the exchange type mechanism, is rather <br>&gt; awkward; in the main because it is encoded in the existing AMQP <br>&gt; operations (bind, consume), and these aren't the right vocabulary for <br>
&gt; the intended use.  See the "Limitations" section of<br>&gt; <a href="https://github.com/squaremo/rabbitmq-lvc-plugin">https://github.com/squaremo/rabbitmq-lvc-plugin</a>.<br>&gt; <br>&gt; There's two natural generalisations of LVC: one is to keep an arbitrary <br>
&gt; number of values around, and the other is to make the hold higher-order <br>&gt; (i.e., instead of replaying values, replay some aggregate of values). <br>&gt; Neither of these really fit in AMQP as it stands, not in the available <br>
&gt; methods or their semantics.<br>&gt; <br>&gt; Anyway.  So that this is helpful and not just me kicking AMQP, here is a <br>&gt; suggestion: if the consumers are made to co-operate just slightly you <br>&gt; can get your proxy idea to work. Just use different queues for the <br>
&gt; replay and the live.  Subscribe the live first and note the most recent <br>&gt; message you know about at that time; then publish your saved messages <br>&gt; through to that message into the replay queue.  The consumer, when it <br>
&gt; hears back from the proxy, drains the replay queue then starts on the <br>&gt; live queue.<br>&gt; <br>&gt; <br>&gt; Michael<br>&gt; <br>&gt; *I call the category "value+updates", and the synchronisation problem <br>
&gt; you describe above is the essence of it.<br>&gt; <br>&gt;&gt; _______________________________________________________________<br>&gt;&gt;<br>&gt;&gt; *Simone Busoli*<br>&gt;&gt; Direzione risorse umane e segreteria generale &ndash; Sistemi informativi<br>
&gt;&gt; gestione sportiva<br>&gt;&gt; /Collaboratore/≤br>&gt;&gt;<br>&gt;&gt; *Ferrari S.p.A. * Via Ascari 55/57, 41053, Maranello (MO); Italy<br>&gt;&gt; *T* (+39) 0536-949.850<br>&gt;&gt; *E* _<a href="mailto:simone.busoli@...">simone.busoli@...</a> &lt;mailto:<a href="mailto:simone.busoli@...">simone.busoli@...</a>&gt;_<br>
&gt;&gt; _<a href="http://www.ferrari.com">www.ferrari.com</a> &lt;<a href="http://www.ferrari.com/">http://www.ferrari.com/</a>&gt;_<br>&gt;&gt;<br>&gt;&gt; *From:* Jon Brisbin [mailto:<a href="mailto:jon@...">jon@...</a>]<br>
&gt;&gt; *Sent:* Wednesday, June 01, 2011 4:59 PM<br>&gt;&gt; *To:* Busoli, Simone<br>&gt;&gt; *Cc:* <a href="mailto:rabbitmq-discuss@....com">rabbitmq-discuss@...</a><br>&gt;&gt; *Subject:* Re: [rabbitmq-discuss] How to handle full exchange history<br>
&gt;&gt; delivery<br>&gt;&gt;<br>&gt;&gt; Maybe I wasn't being clear:<br>&gt;&gt;<br>&gt;&gt; I've already written a custom exchange that stores every message it<br>&gt;&gt; receives in Riak (<a href="http://wiki.basho.com/">http://wiki.basho.com/</a>). I'm going to be talking about<br>
&gt;&gt; it next week at Erlang Factory in London, as a matter of fact. :)<br>&gt;&gt;<br>&gt;&gt; <a href="https://github.com/jbrisbin/riak-exchange">https://github.com/jbrisbin/riak-exchange</a><br>&gt;&gt;<br>&gt;&gt; When I was writing it, I thought it would be really easy to, when a new<br>
&gt;&gt; queue is bound to this exchange, go fetch all the messages in that<br>&gt;&gt; bucket and send them to the queue.<br>&gt;&gt;<br>&gt;&gt; Since I'm using Riak, the messages themselves are not all stored in memory.<br>
&gt;&gt;<br>&gt;&gt; Thanks!<br>&gt;&gt;<br>&gt;&gt; Jon Brisbin<br>&gt;&gt; http//<a href="http://jbrisbin.com">jbrisbin.com</a><br>&gt;&gt;<br>&gt;&gt; ------------------------------------------------------------------------<br>
&gt;&gt;<br>&gt;&gt;     *From: *"Simone Busoli" &lt;<a href="mailto:Simone.Busoli@...">Simone.Busoli@...</a>&gt;<br>&gt;&gt;     *To: *<a href="mailto:rabbitmq-discuss@...">rabbitmq-discuss@...</a><br>
&gt;&gt;     *Sent: *Wednesday, June 1, 2011 8:43:58 AM<br>&gt;&gt;     *Subject: *Re: [rabbitmq-discuss] How to handle full exchange<br>&gt;&gt;     history delivery<br>&gt;&gt;<br>&gt;&gt;<br>&gt;&gt;     That&rsquo;s what I thought as well, although I&rsquo;m thinking about a couple<br>
&gt;&gt;     of issues about it:<br>&gt;&gt;<br>&gt;&gt;     &middot; Lack of knowledge about Erlang, which is the language I&rsquo;d have to<br>&gt;&gt;     use to build a custom exchange, I guess<br>&gt;&gt;<br>&gt;&gt;     &middot; Amount of memory it would require to store the entire history in<br>
&gt;&gt;     memory. It needn&rsquo;t be in memory actually, but I&rsquo;m not sure what<br>&gt;&gt;     would be the recommended way if the storage has to be carried out by<br>&gt;&gt;     exchange itself. Mnesia perhaps, would it make sense?<br>
&gt;&gt;<br>&gt;&gt;     _______________________________________________________________<br>&gt;&gt;<br>&gt;&gt;     *Simone Busoli*<br>&gt;&gt;     Direzione risorse umane e segreteria generale &ndash; Sistemi informativi<br>&gt;&gt;     gestione sportiva<br>
&gt;&gt;     /Collaboratore/≤br>&gt;&gt;<br>&gt;&gt;     *Ferrari S.p.A. * Via Ascari 55/57, 41053, Maranello (MO); Italy<br>&gt;&gt;     *T* (+39) 0536-949.850<br>&gt;&gt;     *E* _<a href="mailto:simone.busoli@...">simone.busoli@...</a> &lt;mailto:<a href="mailto:simone.busoli <at> ferrari.com">simone.busoli@...</a>&gt;_<br>
&gt;&gt;     _<a href="http://www.ferrari.com">www.ferrari.com</a> &lt;<a href="http://www.ferrari.com/">http://www.ferrari.com/</a>&gt;_<br>&gt;&gt;<br>&gt;&gt;     *From:* Jon Brisbin [mailto:<a href="mailto:jon <at> jbrisbin.com">jon@...</a>]<br>
&gt;&gt;     *Sent:* Wednesday, June 01, 2011 3:38 PM<br>&gt;&gt;     *To:* Busoli, Simone<br>&gt;&gt;     *Cc:* <a href="mailto:rabbitmq-discuss <at> lists.rabbitmq.com">rabbitmq-discuss@...</a><br>&gt;&gt;     *Subject:* Re: [rabbitmq-discuss] How to handle full exchange<br>
&gt;&gt;     history delivery<br>&gt;&gt;<br>&gt;&gt;     I've been seriously considering adding this functionality to the<br>&gt;&gt;     Riak-backed custom exchange. It was something I thought I would add<br>&gt;&gt;     first time through but didn't.<br>
&gt;&gt;<br>&gt;&gt;     You'd have to use a custom exchange. Not sure if that's an issue in<br>&gt;&gt;     your scenario or not.<br>&gt;&gt;<br>&gt;&gt;<br>&gt;&gt;     Thanks!<br>&gt;&gt;<br>&gt;&gt;     Jon Brisbin<br>
&gt;&gt;     http//<a href="http://jbrisbin.com">jbrisbin.com</a><br>&gt;&gt;<br>&gt;&gt;     ------------------------------------------------------------------------<br>&gt;&gt;<br>&gt;&gt;         *From: *"Simone Busoli" &lt;<a href="mailto:Simone.Busoli@...">Simone.Busoli <at> ferrari.com</a>&gt;<br>
&gt;&gt;         *To: *<a href="mailto:rabbitmq-discuss@...m">rabbitmq-discuss@...</a><br>&gt;&gt;         *Sent: *Wednesday, June 1, 2011 5:52:24 AM<br>&gt;&gt;         *Subject: *[rabbitmq-discuss] How to handle full exchange<br>
&gt;&gt;         history delivery<br>&gt;&gt;<br>&gt;&gt;         Hi,<br>&gt;&gt;<br>&gt;&gt;         I am trying to achieve a scenario in which messages are<br>&gt;&gt;         published from some data source into a RabbitMQ exchange and<br>
&gt;&gt;         clients should be able to receive the full history of messages<br>&gt;&gt;         flown through that exchange, regardless of the exact moment in<br>&gt;&gt;         time when they connect.<br>&gt;&gt;<br>
&gt;&gt;         I&rsquo;ve been thinking to solve it by setting up a proxy which<br>&gt;&gt;         subscribes to the exchange where messages are published using<br>&gt;&gt;         its own queue, keeps an internal storage of the messages and<br>
&gt;&gt;         accepts requests from clients.<br>&gt;&gt;<br>&gt;&gt;         When a request from a client arrives it just replays the full<br>&gt;&gt;         message history to the client by publishing the messages to the<br>
&gt;&gt;         client queue (supplied in the ReplyTo field), *then* binds the<br>&gt;&gt;         client queue to the messages exchange, so that the client from<br>&gt;&gt;         this moment onwards starts receiving the &ldquo;live&rdquo; messages, and is<br>
&gt;&gt;         pretty much unaware of what&rsquo;s happened; it simply just receives<br>&gt;&gt;         the whole history.<br>&gt;&gt;<br>&gt;&gt;         Now I am under the impression that there&rsquo;s a fundamental flaw in<br>&gt;&gt;         this reasoning, since the proxy cannot know when the message<br>
&gt;&gt;         history has completed and he can switch the client to the live<br>&gt;&gt;         stream, because while it is replaying historical messages to the<br>&gt;&gt;         client other live messages might be arriving on the proxy&rsquo;s<br>
&gt;&gt;         queue, thus getting into an endless loop.<br>&gt;&gt;<br>&gt;&gt;         I&rsquo;m probably missing something obvious, any advice about how<br>&gt;&gt;         others would approach this scenario is appreciated.<br>
&gt;&gt;<br>&gt;&gt;         Simone<br>&gt;&gt;<br>&gt;&gt;         _________________________________________________________________________________________________________________<br>&gt;&gt;         Questo messaggio e da intendersi esclusivamente ad uso del<br>
&gt;&gt;         destinatario e puo contenere informazioni che sono di natura<br>&gt;&gt;         privilegiata, confidenziale<br>&gt;&gt;         o non divulgabile secondo le leggi vigenti. Se il lettore del<br>&gt;&gt;         presente messaggio non e il destinatario designato, o il<br>
&gt;&gt;         dipendente/agente responsabile<br>&gt;&gt;         per la consegna del messaggio al destinatario designato, si<br>&gt;&gt;         informa che ogni disseminazione, distribuzione o copiatura di<br>&gt;&gt;         questa comunicazione e<br>
&gt;&gt;         strettamente proibita anche ai sensi del decreto legislativo<br>&gt;&gt;         196/03 . Se avete ricevuto questo messaggio per errore, vi<br>&gt;&gt;         preghiamo di notificarcelo<br>&gt;&gt;         immediatamente a mezzo e-mail di risposta e successivamente di<br>
&gt;&gt;         procedere alla cancellazione di questa e-mail e relativi<br>&gt;&gt;         allegati dal vostro sistema.<br>&gt;&gt;         _________________________________________________________________________________________________________________<br>
&gt;&gt;         This message is intended only for the use of the addressee and<br>&gt;&gt;         may contain information that is privileged, confidential and<br>&gt;&gt;         exempt from<br>&gt;&gt;         disclosure under applicable law. If the reader of this message<br>
&gt;&gt;         is not the intended recipient, or the employee or agent<br>&gt;&gt;         responsible for delivering the<br>&gt;&gt;         message to the intended recipient, you are hereby notified that<br>&gt;&gt;         any dissemination, distribution or copying of this communication<br>
&gt;&gt;         is strictly<br>&gt;&gt;         prohibited. If you have received this e-mail in error, please<br>&gt;&gt;         notify us immediately by return e-mail and delete this e-mail<br>&gt;&gt;         and all attachments from<br>
&gt;&gt;         your system.<br>&gt;&gt;         _________________________________________________________________________________________________________________<br>&gt;&gt;<br>&gt;&gt;<br>&gt;&gt;         _______________________________________________<br>
&gt;&gt;         rabbitmq-discuss mailing list<br>&gt;&gt;         <a href="mailto:rabbitmq-discuss@...">rabbitmq-discuss@...itmq.com</a><br>&gt;&gt;         &lt;mailto:<a href="mailto:rabbitmq-discuss@...">rabbitmq-discuss@...</a>&gt;<br>
&gt;&gt;         <a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>&gt;&gt;<br>&gt;&gt;     _________________________________________________________________________________________________________________<br>
&gt;&gt;     Questo messaggio &egrave; da intendersi esclusivamente ad uso del<br>&gt;&gt;     destinatario e pu&ograve; contenere informazioni che sono di natura<br>&gt;&gt;     privilegiata, confidenziale<br>&gt;&gt;     o non divulgabile secondo le leggi vigenti. Se il lettore del<br>
&gt;&gt;     presente messaggio non &egrave; il destinatario designato, o il<br>&gt;&gt;     dipendente/agente responsabile<br>&gt;&gt;     per la consegna del messaggio al destinatario designato, si informa<br>&gt;&gt;     che ogni disseminazione, distribuzione o copiatura di questa<br>
&gt;&gt;     comunicazione &egrave;<br>&gt;&gt;     strettamente proibita anche ai sensi del decreto legislativo 196/03<br>&gt;&gt;     . Se avete ricevuto questo messaggio per errore, vi preghiamo di<br>&gt;&gt;     notificarcelo<br>
&gt;&gt;     immediatamente a mezzo e-mail di risposta e successivamente di<br>&gt;&gt;     procedere alla cancellazione di questa e-mail e relativi allegati<br>&gt;&gt;     dal vostro sistema.<br>&gt;&gt;     _________________________________________________________________________________________________________________<br>
&gt;&gt;     This message is intended only for the use of the addressee and may<br>&gt;&gt;     contain information that is privileged, confidential and exempt from<br>&gt;&gt;     disclosure under applicable law. If the reader of this message is<br>
&gt;&gt;     not the intended recipient, or the employee or agent responsible for<br>&gt;&gt;     delivering the<br>&gt;&gt;     message to the intended recipient, you are hereby notified that any<br>&gt;&gt;     dissemination, distribution or copying of this communication is<br>
&gt;&gt;     strictly<br>&gt;&gt;     prohibited. If you have received this e-mail in error, please notify<br>&gt;&gt;     us immediately by return e-mail and delete this e-mail and all<br>&gt;&gt;     attachments from<br>
&gt;&gt;     your system.<br>&gt;&gt;     _________________________________________________________________________________________________________________<br>&gt;&gt;<br>&gt;&gt;<br>&gt;&gt;     _______________________________________________<br>
&gt;&gt;     rabbitmq-discuss mailing list<br>&gt;&gt;     <a href="mailto:rabbitmq-discuss@...">rabbitmq-discuss@...</a><br>&gt;&gt;     &lt;mailto:<a href="mailto:rabbitmq-discuss@...bitmq.com">rabbitmq-discuss@...</a>&gt;<br>
&gt;&gt;     <a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>&gt;&gt;<br>&gt;&gt; _________________________________________________________________________________________________________________<br>
&gt;&gt; Questo messaggio &egrave; da intendersi esclusivamente ad uso del destinatario<br>&gt;&gt; e pu&ograve; contenere informazioni che sono di natura privilegiata, confidenziale<br>&gt;&gt; o non divulgabile secondo le leggi vigenti. Se il lettore del presente<br>
&gt;&gt; messaggio non &egrave; il destinatario designato, o il dipendente/agente<br>&gt;&gt; responsabile<br>&gt;&gt; per la consegna del messaggio al destinatario designato, si informa che<br>&gt;&gt; ogni disseminazione, distribuzione o copiatura di questa comunicazione &egrave;<br>
&gt;&gt; strettamente proibita anche ai sensi del decreto legislativo 196/03 . Se<br>&gt;&gt; avete ricevuto questo messaggio per errore, vi preghiamo di notificarcelo<br>&gt;&gt; immediatamente a mezzo e-mail di risposta e successivamente di procedere<br>
&gt;&gt; alla cancellazione di questa e-mail e relativi allegati dal vostro sistema.<br>&gt;&gt; _________________________________________________________________________________________________________________<br>&gt;&gt; This message is intended only for the use of the addressee and may<br>
&gt;&gt; contain information that is privileged, confidential and exempt from<br>&gt;&gt; disclosure under applicable law. If the reader of this message is not<br>&gt;&gt; the intended recipient, or the employee or agent responsible for<br>
&gt;&gt; delivering the<br>&gt;&gt; message to the intended recipient, you are hereby notified that any<br>&gt;&gt; dissemination, distribution or copying of this communication is strictly<br>&gt;&gt; prohibited. If you have received this e-mail in error, please notify us<br>
&gt;&gt; immediately by return e-mail and delete this e-mail and all attachments<br>&gt;&gt; from<br>&gt;&gt; your system.<br>&gt;&gt; _________________________________________________________________________________________________________________<br>
&gt;&gt;<br>&gt;&gt;<br>&gt;&gt;<br>&gt;&gt; _______________________________________________<br>&gt;&gt; rabbitmq-discuss mailing list<br>&gt;&gt; <a href="mailto:rabbitmq-discuss@...">rabbitmq-discuss <at> lists.rabbitmq.com</a><br>
&gt;&gt; <a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>&gt; <br>&gt; _______________________________________________<br>
&gt; rabbitmq-discuss mailing list<br>&gt; <a href="mailto:rabbitmq-discuss@...">rabbitmq-discuss@...</a><br>&gt; <a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
</div>
</div>
Permalink | Reply |
headers
Tony Garnock-Jones | 2 Jun 2011 18:43
Picon

Re: How to handle full exchange history delivery

(Dammit, forgot to reply-to-all. Sorry about the dup, mikeb!)

On 2 June 2011 06:40, Michael Bridgen <mikeb-mQ7lE4MOPXtWk0Htik3J/w@public.gmane.org> wrote:

By way of illustration: the last-value-caching exchange, though it was developed as a verification of the exchange type mechanism, is rather awkward; in the main because it is encoded in the existing AMQP operations (bind, consume), and these aren't the right vocabulary for the intended use.  See the "Limitations" section of
https://github.com/squaremo/rabbitmq-lvc-plugin.

Aren't these RabbitMQ implementation restrictions rather than things that, in principle, require new or different AMQP vocabulary? If new vocabulary really is required, what kind of thing do you have in mind?

Tony
<div>
<p>(Dammit, forgot to reply-to-all. Sorry about the dup, mikeb!)<br><br>On 2 June 2011 06:40, Michael Bridgen <span dir="ltr">&lt;<a href="mailto:mikeb@...">mikeb@...</a>&gt;</span> wrote:<br></p>
<div class="gmail_quote">
<blockquote class="gmail_quote">By way of illustration: the last-value-caching exchange, though it was developed as a verification of the exchange type mechanism, is rather awkward; in the main because it is encoded in the existing AMQP operations (bind, consume), and these aren't the right vocabulary for the intended use. &nbsp;See the "Limitations" section of<br><a href="https://github.com/squaremo/rabbitmq-lvc-plugin" target="_blank">https://github.com/squaremo/rabbitmq-lvc-plugin</a>.<br>
</blockquote>
<div>
<br>Aren't these RabbitMQ implementation restrictions rather than things 
that, in principle, require new or different AMQP vocabulary? If new 
vocabulary really is required, what kind of thing do you have in mind?<br><br>Tony<br>
</div>
</div>
</div>
Permalink | Reply |
headers
Simone Busoli | 3 Jun 2011 01:14
Picon
Gravatar

Re: How to handle full exchange history delivery

I have started implementing it with a layer on top of rabbit, and it gets tricky soon. There are two main challenges:

- figure out how to merge the stored and live message streams
- make sure that the replayer component replays messages which partially overlap with those the client subscribed to before asking for stored messages (this is a case of slow message store)

To solve the first I have to decorated the messages with a unique identifier so that the client knows when to merge live and stored messages based on id match.
The second one is a bit trickier and I didn't solve it yet. This happens if the component which stores messages is slow and upon replay request is only capable of replaying messages up to a point in time earlier than the first message the client gets from the live stream receives, thus leaving a client with a hole in the message stream.

On Thu, Jun 2, 2011 at 12:40, Michael Bridgen <mikeb-mQ7lE4MOPXtWk0Htik3J/w@public.gmane.org> wrote:
WRT this:

/When I was writing it, I thought it would be really easy to, when a new
queue is bound to this exchange, go fetch all the messages in that
bucket and send them to the queue.///

I’m not sure it is, that was my main question actually. How do you
figure out how to join the messages stored in the bucket and those
arriving live so that the last message you send to the queue from those
picked up from the bucket is followed exactly by its logical successor
from those coming live?

Say I have messages 1..100 being published to the exchange, each time
one arrives it’s stored. The exchange has received the first 50 messages
and a client performs a binding of a queue to the exchange. So you start
pulling the messages from the storage and sending them to the queue
before forwarding any live messages. While you are pulling them out of
the storage messages 51..60 are published to the exchange, how do you
deal with them? Do you always pull from the storage? When do you stop
pulling from the storage and relay to the queue directly what is being
published to the exchange if you don’t know whether any new messages
have arrived while you were pulling from the storage?

My inclination with this and similar use cases* is that they deserve their own content class (alongside 'exchange', 'queue', etc.).

By way of illustration: the last-value-caching exchange, though it was developed as a verification of the exchange type mechanism, is rather awkward; in the main because it is encoded in the existing AMQP operations (bind, consume), and these aren't the right vocabulary for the intended use.  See the "Limitations" section of
https://github.com/squaremo/rabbitmq-lvc-plugin.

There's two natural generalisations of LVC: one is to keep an arbitrary number of values around, and the other is to make the hold higher-order (i.e., instead of replaying values, replay some aggregate of values). Neither of these really fit in AMQP as it stands, not in the available methods or their semantics.

Anyway.  So that this is helpful and not just me kicking AMQP, here is a suggestion: if the consumers are made to co-operate just slightly you can get your proxy idea to work. Just use different queues for the replay and the live.  Subscribe the live first and note the most recent message you know about at that time; then publish your saved messages through to that message into the replay queue.  The consumer, when it hears back from the proxy, drains the replay queue then starts on the live queue.


Michael

*I call the category "value+updates", and the synchronisation problem you describe above is the essence of it.

_______________________________________________________________

*Simone Busoli*

Direzione risorse umane e segreteria generale – Sistemi informativi
gestione sportiva
/Collaboratore/

*Ferrari S.p.A. * Via Ascari 55/57, 41053, Maranello (MO); Italy
*T* (+39) 0536-949.850
*E* _simone.busoli-sKiPNUh8iG9BDgjK7y7TUQ@public.gmane.org <mailto:simone.busoli-sKiPNUh8iG9BDgjK7y7TUQ@public.gmane.org>_
_www.ferrari.com <http://www.ferrari.com/>_


*From:* Jon Brisbin [mailto:jon-foCUeXAHXPNWk0Htik3J/w@public.gmane.org]
*Sent:* Wednesday, June 01, 2011 4:59 PM
*To:* Busoli, Simone
*Cc:* rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org
*Subject:* Re: [rabbitmq-discuss] How to handle full exchange history
delivery

Maybe I wasn't being clear:

I've already written a custom exchange that stores every message it
receives in Riak (http://wiki.basho.com/). I'm going to be talking about
it next week at Erlang Factory in London, as a matter of fact. :)

https://github.com/jbrisbin/riak-exchange

When I was writing it, I thought it would be really easy to, when a new
queue is bound to this exchange, go fetch all the messages in that
bucket and send them to the queue.

Since I'm using Riak, the messages themselves are not all stored in memory.

Thanks!

Jon Brisbin
http//jbrisbin.com

------------------------------------------------------------------------

   *From: *"Simone Busoli" <Simone.Busoli-sKiPNUh8iG9BDgjK7y7TUQ@public.gmane.org>
   *To: *rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org
   *Sent: *Wednesday, June 1, 2011 8:43:58 AM
   *Subject: *Re: [rabbitmq-discuss] How to handle full exchange
   history delivery


   That’s what I thought as well, although I’m thinking about a couple
   of issues about it:

   · Lack of knowledge about Erlang, which is the language I’d have to
   use to build a custom exchange, I guess

   · Amount of memory it would require to store the entire history in
   memory. It needn’t be in memory actually, but I’m not sure what
   would be the recommended way if the storage has to be carried out by
   exchange itself. Mnesia perhaps, would it make sense?

   _______________________________________________________________

   *Simone Busoli*
   Direzione risorse umane e segreteria generale – Sistemi informativi
   gestione sportiva
   /Collaboratore/

   *Ferrari S.p.A. * Via Ascari 55/57, 41053, Maranello (MO); Italy
   *T* (+39) 0536-949.850
   *E* _simone.busoli-sKiPNUh8iG9BDgjK7y7TUQ@public.gmane.org <mailto:simone.busoli-sKiPNUh8iG9BDgjK7y7TUQ@public.gmane.org>_
   _www.ferrari.com <http://www.ferrari.com/>_


   *From:* Jon Brisbin [mailto:jon-foCUeXAHXPNWk0Htik3J/w@public.gmane.org]
   *Sent:* Wednesday, June 01, 2011 3:38 PM
   *To:* Busoli, Simone
   *Cc:* rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org
   *Subject:* Re: [rabbitmq-discuss] How to handle full exchange
   history delivery

   I've been seriously considering adding this functionality to the
   Riak-backed custom exchange. It was something I thought I would add
   first time through but didn't.

   You'd have to use a custom exchange. Not sure if that's an issue in
   your scenario or not.


   Thanks!

   Jon Brisbin
   http//jbrisbin.com

   ------------------------------------------------------------------------

       *From: *"Simone Busoli" <Simone.Busoli-sKiPNUh8iG9BDgjK7y7TUQ@public.gmane.org>
       *To: *rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org
       *Sent: *Wednesday, June 1, 2011 5:52:24 AM
       *Subject: *[rabbitmq-discuss] How to handle full exchange
       history delivery

       Hi,

       I am trying to achieve a scenario in which messages are
       published from some data source into a RabbitMQ exchange and
       clients should be able to receive the full history of messages
       flown through that exchange, regardless of the exact moment in
       time when they connect.

       I’ve been thinking to solve it by setting up a proxy which
       subscribes to the exchange where messages are published using
       its own queue, keeps an internal storage of the messages and
       accepts requests from clients.

       When a request from a client arrives it just replays the full
       message history to the client by publishing the messages to the
       client queue (supplied in the ReplyTo field), *then* binds the
       client queue to the messages exchange, so that the client from
       this moment onwards starts receiving the “live” messages, and is
       pretty much unaware of what’s happened; it simply just receives
       the whole history.

       Now I am under the impression that there’s a fundamental flaw in
       this reasoning, since the proxy cannot know when the message
       history has completed and he can switch the client to the live
       stream, because while it is replaying historical messages to the
       client other live messages might be arriving on the proxy’s
       queue, thus getting into an endless loop.

       I’m probably missing something obvious, any advice about how
       others would approach this scenario is appreciated.

       Simone

       _________________________________________________________________________________________________________________
       Questo messaggio e da intendersi esclusivamente ad uso del
       destinatario e puo contenere informazioni che sono di natura
       privilegiata, confidenziale
       o non divulgabile secondo le leggi vigenti. Se il lettore del
       presente messaggio non e il destinatario designato, o il
       dipendente/agente responsabile
       per la consegna del messaggio al destinatario designato, si
       informa che ogni disseminazione, distribuzione o copiatura di
       questa comunicazione e
       strettamente proibita anche ai sensi del decreto legislativo
       196/03 . Se avete ricevuto questo messaggio per errore, vi
       preghiamo di notificarcelo
       immediatamente a mezzo e-mail di risposta e successivamente di
       procedere alla cancellazione di questa e-mail e relativi
       allegati dal vostro sistema.
       _________________________________________________________________________________________________________________
       This message is intended only for the use of the addressee and
       may contain information that is privileged, confidential and
       exempt from
       disclosure under applicable law. If the reader of this message
       is not the intended recipient, or the employee or agent
       responsible for delivering the
       message to the intended recipient, you are hereby notified that
       any dissemination, distribution or copying of this communication
       is strictly
       prohibited. If you have received this e-mail in error, please
       notify us immediately by return e-mail and delete this e-mail
       and all attachments from
       your system.
       _________________________________________________________________________________________________________________


       _______________________________________________
       rabbitmq-discuss mailing list
       rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org
       <mailto:rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org>

       https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

   _________________________________________________________________________________________________________________
   Questo messaggio è da intendersi esclusivamente ad uso del
   destinatario e può contenere informazioni che sono di natura
   privilegiata, confidenziale
   o non divulgabile secondo le leggi vigenti. Se il lettore del
   presente messaggio non è il destinatario designato, o il
   dipendente/agente responsabile
   per la consegna del messaggio al destinatario designato, si informa
   che ogni disseminazione, distribuzione o copiatura di questa
   comunicazione è
   strettamente proibita anche ai sensi del decreto legislativo 196/03
   . Se avete ricevuto questo messaggio per errore, vi preghiamo di
   notificarcelo
   immediatamente a mezzo e-mail di risposta e successivamente di
   procedere alla cancellazione di questa e-mail e relativi allegati
   dal vostro sistema.
   _________________________________________________________________________________________________________________
   This message is intended only for the use of the addressee and may
   contain information that is privileged, confidential and exempt from
   disclosure under applicable law. If the reader of this message is
   not the intended recipient, or the employee or agent responsible for
   delivering the
   message to the intended recipient, you are hereby notified that any
   dissemination, distribution or copying of this communication is
   strictly
   prohibited. If you have received this e-mail in error, please notify
   us immediately by return e-mail and delete this e-mail and all
   attachments from
   your system.
   _________________________________________________________________________________________________________________


   _______________________________________________
   rabbitmq-discuss mailing list
   rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org
   <mailto:rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org>

   https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

_________________________________________________________________________________________________________________
Questo messaggio è da intendersi esclusivamente ad uso del destinatario
e può contenere informazioni che sono di natura privilegiata, confidenziale
o non divulgabile secondo le leggi vigenti. Se il lettore del presente
messaggio non è il destinatario designato, o il dipendente/agente
responsabile
per la consegna del messaggio al destinatario designato, si informa che
ogni disseminazione, distribuzione o copiatura di questa comunicazione è
strettamente proibita anche ai sensi del decreto legislativo 196/03 . Se
avete ricevuto questo messaggio per errore, vi preghiamo di notificarcelo
immediatamente a mezzo e-mail di risposta e successivamente di procedere
alla cancellazione di questa e-mail e relativi allegati dal vostro sistema.
_________________________________________________________________________________________________________________
This message is intended only for the use of the addressee and may
contain information that is privileged, confidential and exempt from
disclosure under applicable law. If the reader of this message is not
the intended recipient, or the employee or agent responsible for
delivering the
message to the intended recipient, you are hereby notified that any
dissemination, distribution or copying of this communication is strictly
prohibited. If you have received this e-mail in error, please notify us
immediately by return e-mail and delete this e-mail and all attachments
from
your system.
_________________________________________________________________________________________________________________



_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

<div>I have started implementing it with a layer on top of rabbit, and it gets tricky soon. There are two main challenges:<div><br></div>
<div>- figure out how to merge the stored and live message streams</div>
<div>- make sure that the replayer component replays messages which partially overlap with those the client subscribed to before asking for stored messages (this is a case of slow message store)</div>

<div><br></div>
<div>To solve the first I have to decorated the messages with a unique identifier so that the client knows when to merge live and stored messages based on id match.</div>

<div>The second one is a bit trickier and I didn't solve it yet. This happens if the component which stores messages is slow and upon replay request is only capable of replaying messages up to a point in time earlier than the first message the client gets from the live stream receives, thus leaving a client with a hole in the message stream.</div>

<div><div>
<br><div class="gmail_quote">On Thu, Jun 2, 2011 at 12:40, Michael Bridgen <span dir="ltr">&lt;<a href="mailto:mikeb@...">mikeb@...</a>&gt;</span> wrote:<br><blockquote class="gmail_quote">

<div class="im">
<blockquote class="gmail_quote">
WRT this:<br><br>
/When I was writing it, I thought it would be really easy to, when a new<br>
queue is bound to this exchange, go fetch all the messages in that<br>
bucket and send them to the queue.///<br><br>
I&rsquo;m not sure it is, that was my main question actually. How do you<br>
figure out how to join the messages stored in the bucket and those<br>
arriving live so that the last message you send to the queue from those<br>
picked up from the bucket is followed exactly by its logical successor<br>
from those coming live?<br><br>
Say I have messages 1..100 being published to the exchange, each time<br>
one arrives it&rsquo;s stored. The exchange has received the first 50 messages<br>
and a client performs a binding of a queue to the exchange. So you start<br>
pulling the messages from the storage and sending them to the queue<br>
before forwarding any live messages. While you are pulling them out of<br>
the storage messages 51..60 are published to the exchange, how do you<br>
deal with them? Do you always pull from the storage? When do you stop<br>
pulling from the storage and relay to the queue directly what is being<br>
published to the exchange if you don&rsquo;t know whether any new messages<br>
have arrived while you were pulling from the storage?<br>
</blockquote>
<br>
</div>
My inclination with this and similar use cases* is that they deserve their own content class (alongside 'exchange', 'queue', etc.).<br><br>
By way of illustration: the last-value-caching exchange, though it was developed as a verification of the exchange type mechanism, is rather awkward; in the main because it is encoded in the existing AMQP operations (bind, consume), and these aren't the right vocabulary for the intended use. &nbsp;See the "Limitations" section of<br><a href="https://github.com/squaremo/rabbitmq-lvc-plugin" target="_blank">https://github.com/squaremo/rabbitmq-lvc-plugin</a>.<br><br>
There's two natural generalisations of LVC: one is to keep an arbitrary number of values around, and the other is to make the hold higher-order (i.e., instead of replaying values, replay some aggregate of values). Neither of these really fit in AMQP as it stands, not in the available methods or their semantics.<br><br>
Anyway. &nbsp;So that this is helpful and not just me kicking AMQP, here is a suggestion: if the consumers are made to co-operate just slightly you can get your proxy idea to work. Just use different queues for the replay and the live. &nbsp;Subscribe the live first and note the most recent message you know about at that time; then publish your saved messages through to that message into the replay queue. &nbsp;The consumer, when it hears back from the proxy, drains the replay queue then starts on the live queue.<br><br><br>
Michael<br><br>
*I call the category "value+updates", and the synchronisation problem you describe above is the essence of it.<br><br><blockquote class="gmail_quote">
_______________________________________________________________<br><br>
*Simone Busoli*<div class="im">
<br>
Direzione risorse umane e segreteria generale &ndash; Sistemi informativi<br>
gestione sportiva<br>
/Collaboratore/≤br><br>
*Ferrari S.p.A. * Via Ascari 55/57, 41053, Maranello (MO); Italy<br>
*T* <a href="tel:%28%2B39%29%200536-949.850" value="+390536949850" target="_blank">(+39) 0536-949.850</a><br>
</div>
*E* _<a href="mailto:simone.busoli@..." target="_blank">simone.busoli@...</a> &lt;mailto:<a href="mailto:simone.busoli <at> ferrari.com" target="_blank">simone.busoli@...</a>&gt;_<br>
_<a href="http://www.ferrari.com" target="_blank">www.ferrari.com</a> &lt;<a href="http://www.ferrari.com/" target="_blank">http://www.ferrari.com/</a>&gt;_<div>
<div></div>
<div class="h5">
<br><br>
*From:* Jon Brisbin [mailto:<a href="mailto:jon@..." target="_blank">jon@...</a>]<br>
*Sent:* Wednesday, June 01, 2011 4:59 PM<br>
*To:* Busoli, Simone<br>
*Cc:* <a href="mailto:rabbitmq-discuss@..." target="_blank">rabbitmq-discuss@...</a><br>
*Subject:* Re: [rabbitmq-discuss] How to handle full exchange history<br>
delivery<br><br>
Maybe I wasn't being clear:<br><br>
I've already written a custom exchange that stores every message it<br>
receives in Riak (<a href="http://wiki.basho.com/" target="_blank">http://wiki.basho.com/</a>). I'm going to be talking about<br>
it next week at Erlang Factory in London, as a matter of fact. :)<br><br><a href="https://github.com/jbrisbin/riak-exchange" target="_blank">https://github.com/jbrisbin/riak-exchange</a><br><br>
When I was writing it, I thought it would be really easy to, when a new<br>
queue is bound to this exchange, go fetch all the messages in that<br>
bucket and send them to the queue.<br><br>
Since I'm using Riak, the messages themselves are not all stored in memory.<br><br>
Thanks!<br><br>
Jon Brisbin<br>
http//<a href="http://jbrisbin.com" target="_blank">jbrisbin.com</a><br><br>
------------------------------------------------------------------------<br><br>
 &nbsp; &nbsp;*From: *"Simone Busoli" &lt;<a href="mailto:Simone.Busoli@..." target="_blank">Simone.Busoli@...</a>&gt;<br>
 &nbsp; &nbsp;*To: *<a href="mailto:rabbitmq-discuss@..." target="_blank">rabbitmq-discuss@...</a><br>
 &nbsp; &nbsp;*Sent: *Wednesday, June 1, 2011 8:43:58 AM<br>
 &nbsp; &nbsp;*Subject: *Re: [rabbitmq-discuss] How to handle full exchange<br>
 &nbsp; &nbsp;history delivery<br><br><br>
 &nbsp; &nbsp;That&rsquo;s what I thought as well, although I&rsquo;m thinking about a couple<br>
 &nbsp; &nbsp;of issues about it:<br><br>
 &nbsp; &nbsp;&middot; Lack of knowledge about Erlang, which is the language I&rsquo;d have to<br>
 &nbsp; &nbsp;use to build a custom exchange, I guess<br><br>
 &nbsp; &nbsp;&middot; Amount of memory it would require to store the entire history in<br>
 &nbsp; &nbsp;memory. It needn&rsquo;t be in memory actually, but I&rsquo;m not sure what<br>
 &nbsp; &nbsp;would be the recommended way if the storage has to be carried out by<br>
 &nbsp; &nbsp;exchange itself. Mnesia perhaps, would it make sense?<br><br>
 &nbsp; &nbsp;_______________________________________________________________<br><br>
 &nbsp; &nbsp;*Simone Busoli*<br>
 &nbsp; &nbsp;Direzione risorse umane e segreteria generale &ndash; Sistemi informativi<br>
 &nbsp; &nbsp;gestione sportiva<br>
 &nbsp; &nbsp;/Collaboratore/≤br><br>
 &nbsp; &nbsp;*Ferrari S.p.A. * Via Ascari 55/57, 41053, Maranello (MO); Italy<br>
 &nbsp; &nbsp;*T* <a href="tel:%28%2B39%29%200536-949.850" value="+390536949850" target="_blank">(+39) 0536-949.850</a><br>
</div>
</div>
 &nbsp; &nbsp;*E* _<a href="mailto:simone.busoli@..." target="_blank">simone.busoli@...</a> &lt;mailto:<a href="mailto:simone.busoli <at> ferrari.com" target="_blank">simone.busoli@...</a>&gt;_<br>
 &nbsp; &nbsp;_<a href="http://www.ferrari.com" target="_blank">www.ferrari.com</a> &lt;<a href="http://www.ferrari.com/" target="_blank">http://www.ferrari.com/</a>&gt;_<div>
<div></div>
<div class="h5">
<br><br>
 &nbsp; &nbsp;*From:* Jon Brisbin [mailto:<a href="mailto:jon@..." target="_blank">jon@...</a>]<br>
 &nbsp; &nbsp;*Sent:* Wednesday, June 01, 2011 3:38 PM<br>
 &nbsp; &nbsp;*To:* Busoli, Simone<br>
 &nbsp; &nbsp;*Cc:* <a href="mailto:rabbitmq-discuss@..." target="_blank">rabbitmq-discuss@...</a><br>
 &nbsp; &nbsp;*Subject:* Re: [rabbitmq-discuss] How to handle full exchange<br>
 &nbsp; &nbsp;history delivery<br><br>
 &nbsp; &nbsp;I've been seriously considering adding this functionality to the<br>
 &nbsp; &nbsp;Riak-backed custom exchange. It was something I thought I would add<br>
 &nbsp; &nbsp;first time through but didn't.<br><br>
 &nbsp; &nbsp;You'd have to use a custom exchange. Not sure if that's an issue in<br>
 &nbsp; &nbsp;your scenario or not.<br><br><br>
 &nbsp; &nbsp;Thanks!<br><br>
 &nbsp; &nbsp;Jon Brisbin<br>
 &nbsp; &nbsp;http//<a href="http://jbrisbin.com" target="_blank">jbrisbin.com</a><br><br>
 &nbsp; &nbsp;------------------------------------------------------------------------<br><br>
 &nbsp; &nbsp; &nbsp; &nbsp;*From: *"Simone Busoli" &lt;<a href="mailto:Simone.Busoli@..." target="_blank">Simone.Busoli@...</a>&gt;<br>
 &nbsp; &nbsp; &nbsp; &nbsp;*To: *<a href="mailto:rabbitmq-discuss@..." target="_blank">rabbitmq-discuss@...</a><br>
 &nbsp; &nbsp; &nbsp; &nbsp;*Sent: *Wednesday, June 1, 2011 5:52:24 AM<br>
 &nbsp; &nbsp; &nbsp; &nbsp;*Subject: *[rabbitmq-discuss] How to handle full exchange<br>
 &nbsp; &nbsp; &nbsp; &nbsp;history delivery<br><br>
 &nbsp; &nbsp; &nbsp; &nbsp;Hi,<br><br>
 &nbsp; &nbsp; &nbsp; &nbsp;I am trying to achieve a scenario in which messages are<br>
 &nbsp; &nbsp; &nbsp; &nbsp;published from some data source into a RabbitMQ exchange and<br>
 &nbsp; &nbsp; &nbsp; &nbsp;clients should be able to receive the full history of messages<br>
 &nbsp; &nbsp; &nbsp; &nbsp;flown through that exchange, regardless of the exact moment in<br>
 &nbsp; &nbsp; &nbsp; &nbsp;time when they connect.<br><br>
 &nbsp; &nbsp; &nbsp; &nbsp;I&rsquo;ve been thinking to solve it by setting up a proxy which<br>
 &nbsp; &nbsp; &nbsp; &nbsp;subscribes to the exchange where messages are published using<br>
 &nbsp; &nbsp; &nbsp; &nbsp;its own queue, keeps an internal storage of the messages and<br>
 &nbsp; &nbsp; &nbsp; &nbsp;accepts requests from clients.<br><br>
 &nbsp; &nbsp; &nbsp; &nbsp;When a request from a client arrives it just replays the full<br>
 &nbsp; &nbsp; &nbsp; &nbsp;message history to the client by publishing the messages to the<br>
 &nbsp; &nbsp; &nbsp; &nbsp;client queue (supplied in the ReplyTo field), *then* binds the<br>
 &nbsp; &nbsp; &nbsp; &nbsp;client queue to the messages exchange, so that the client from<br>
 &nbsp; &nbsp; &nbsp; &nbsp;this moment onwards starts receiving the &ldquo;live&rdquo; messages, and is<br>
 &nbsp; &nbsp; &nbsp; &nbsp;pretty much unaware of what&rsquo;s happened; it simply just receives<br>
 &nbsp; &nbsp; &nbsp; &nbsp;the whole history.<br><br>
 &nbsp; &nbsp; &nbsp; &nbsp;Now I am under the impression that there&rsquo;s a fundamental flaw in<br>
 &nbsp; &nbsp; &nbsp; &nbsp;this reasoning, since the proxy cannot know when the message<br>
 &nbsp; &nbsp; &nbsp; &nbsp;history has completed and he can switch the client to the live<br>
 &nbsp; &nbsp; &nbsp; &nbsp;stream, because while it is replaying historical messages to the<br>
 &nbsp; &nbsp; &nbsp; &nbsp;client other live messages might be arriving on the proxy&rsquo;s<br>
 &nbsp; &nbsp; &nbsp; &nbsp;queue, thus getting into an endless loop.<br><br>
 &nbsp; &nbsp; &nbsp; &nbsp;I&rsquo;m probably missing something obvious, any advice about how<br>
 &nbsp; &nbsp; &nbsp; &nbsp;others would approach this scenario is appreciated.<br><br>
 &nbsp; &nbsp; &nbsp; &nbsp;Simone<br><br>
 &nbsp; &nbsp; &nbsp; &nbsp;_________________________________________________________________________________________________________________<br>
 &nbsp; &nbsp; &nbsp; &nbsp;Questo messaggio e da intendersi esclusivamente ad uso del<br>
 &nbsp; &nbsp; &nbsp; &nbsp;destinatario e puo contenere informazioni che sono di natura<br>
 &nbsp; &nbsp; &nbsp; &nbsp;privilegiata, confidenziale<br>
 &nbsp; &nbsp; &nbsp; &nbsp;o non divulgabile secondo le leggi vigenti. Se il lettore del<br>
 &nbsp; &nbsp; &nbsp; &nbsp;presente messaggio non e il destinatario designato, o il<br>
 &nbsp; &nbsp; &nbsp; &nbsp;dipendente/agente responsabile<br>
 &nbsp; &nbsp; &nbsp; &nbsp;per la consegna del messaggio al destinatario designato, si<br>
 &nbsp; &nbsp; &nbsp; &nbsp;informa che ogni disseminazione, distribuzione o copiatura di<br>
 &nbsp; &nbsp; &nbsp; &nbsp;questa comunicazione e<br>
 &nbsp; &nbsp; &nbsp; &nbsp;strettamente proibita anche ai sensi del decreto legislativo<br>
 &nbsp; &nbsp; &nbsp; &nbsp;196/03 . Se avete ricevuto questo messaggio per errore, vi<br>
 &nbsp; &nbsp; &nbsp; &nbsp;preghiamo di notificarcelo<br>
 &nbsp; &nbsp; &nbsp; &nbsp;immediatamente a mezzo e-mail di risposta e successivamente di<br>
 &nbsp; &nbsp; &nbsp; &nbsp;procedere alla cancellazione di questa e-mail e relativi<br>
 &nbsp; &nbsp; &nbsp; &nbsp;allegati dal vostro sistema.<br>
 &nbsp; &nbsp; &nbsp; &nbsp;_________________________________________________________________________________________________________________<br>
 &nbsp; &nbsp; &nbsp; &nbsp;This message is intended only for the use of the addressee and<br>
 &nbsp; &nbsp; &nbsp; &nbsp;may contain information that is privileged, confidential and<br>
 &nbsp; &nbsp; &nbsp; &nbsp;exempt from<br>
 &nbsp; &nbsp; &nbsp; &nbsp;disclosure under applicable law. If the reader of this message<br>
 &nbsp; &nbsp; &nbsp; &nbsp;is not the intended recipient, or the employee or agent<br>
 &nbsp; &nbsp; &nbsp; &nbsp;responsible for delivering the<br>
 &nbsp; &nbsp; &nbsp; &nbsp;message to the intended recipient, you are hereby notified that<br>
 &nbsp; &nbsp; &nbsp; &nbsp;any dissemination, distribution or copying of this communication<br>
 &nbsp; &nbsp; &nbsp; &nbsp;is strictly<br>
 &nbsp; &nbsp; &nbsp; &nbsp;prohibited. If you have received this e-mail in error, please<br>
 &nbsp; &nbsp; &nbsp; &nbsp;notify us immediately by return e-mail and delete this e-mail<br>
 &nbsp; &nbsp; &nbsp; &nbsp;and all attachments from<br>
 &nbsp; &nbsp; &nbsp; &nbsp;your system.<br>
 &nbsp; &nbsp; &nbsp; &nbsp;_________________________________________________________________________________________________________________<br><br><br>
 &nbsp; &nbsp; &nbsp; &nbsp;_______________________________________________<br>
 &nbsp; &nbsp; &nbsp; &nbsp;rabbitmq-discuss mailing list<br>
 &nbsp; &nbsp; &nbsp; &nbsp;<a href="mailto:rabbitmq-discuss@..." target="_blank">rabbitmq-discuss@...</a><br>
</div>
</div>
 &nbsp; &nbsp; &nbsp; &nbsp;&lt;mailto:<a href="mailto:rabbitmq-discuss@...q.com" target="_blank">rabbitmq-discuss@...</a>&gt;<div>
<div></div>
<div class="h5">
<br>
 &nbsp; &nbsp; &nbsp; &nbsp;<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br><br>
 &nbsp; &nbsp;_________________________________________________________________________________________________________________<br>
 &nbsp; &nbsp;Questo messaggio &egrave; da intendersi esclusivamente ad uso del<br>
 &nbsp; &nbsp;destinatario e pu&ograve; contenere informazioni che sono di natura<br>
 &nbsp; &nbsp;privilegiata, confidenziale<br>
 &nbsp; &nbsp;o non divulgabile secondo le leggi vigenti. Se il lettore del<br>
 &nbsp; &nbsp;presente messaggio non &egrave; il destinatario designato, o il<br>
 &nbsp; &nbsp;dipendente/agente responsabile<br>
 &nbsp; &nbsp;per la consegna del messaggio al destinatario designato, si informa<br>
 &nbsp; &nbsp;che ogni disseminazione, distribuzione o copiatura di questa<br>
 &nbsp; &nbsp;comunicazione &egrave;<br>
 &nbsp; &nbsp;strettamente proibita anche ai sensi del decreto legislativo 196/03<br>
 &nbsp; &nbsp;. Se avete ricevuto questo messaggio per errore, vi preghiamo di<br>
 &nbsp; &nbsp;notificarcelo<br>
 &nbsp; &nbsp;immediatamente a mezzo e-mail di risposta e successivamente di<br>
 &nbsp; &nbsp;procedere alla cancellazione di questa e-mail e relativi allegati<br>
 &nbsp; &nbsp;dal vostro sistema.<br>
 &nbsp; &nbsp;_________________________________________________________________________________________________________________<br>
 &nbsp; &nbsp;This message is intended only for the use of the addressee and may<br>
 &nbsp; &nbsp;contain information that is privileged, confidential and exempt from<br>
 &nbsp; &nbsp;disclosure under applicable law. If the reader of this message is<br>
 &nbsp; &nbsp;not the intended recipient, or the employee or agent responsible for<br>
 &nbsp; &nbsp;delivering the<br>
 &nbsp; &nbsp;message to the intended recipient, you are hereby notified that any<br>
 &nbsp; &nbsp;dissemination, distribution or copying of this communication is<br>
 &nbsp; &nbsp;strictly<br>
 &nbsp; &nbsp;prohibited. If you have received this e-mail in error, please notify<br>
 &nbsp; &nbsp;us immediately by return e-mail and delete this e-mail and all<br>
 &nbsp; &nbsp;attachments from<br>
 &nbsp; &nbsp;your system.<br>
 &nbsp; &nbsp;_________________________________________________________________________________________________________________<br><br><br>
 &nbsp; &nbsp;_______________________________________________<br>
 &nbsp; &nbsp;rabbitmq-discuss mailing list<br>
 &nbsp; &nbsp;<a href="mailto:rabbitmq-discuss@..." target="_blank">rabbitmq-discuss@...</a><br>
</div>
</div>
 &nbsp; &nbsp;&lt;mailto:<a href="mailto:rabbitmq-discuss@..." target="_blank">rabbitmq-discuss@...</a>&gt;<div>
<div></div>
<div class="h5">
<br>
 &nbsp; &nbsp;<a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br><br>
_________________________________________________________________________________________________________________<br>
Questo messaggio &egrave; da intendersi esclusivamente ad uso del destinatario<br>
e pu&ograve; contenere informazioni che sono di natura privilegiata, confidenziale<br>
o non divulgabile secondo le leggi vigenti. Se il lettore del presente<br>
messaggio non &egrave; il destinatario designato, o il dipendente/agente<br>
responsabile<br>
per la consegna del messaggio al destinatario designato, si informa che<br>
ogni disseminazione, distribuzione o copiatura di questa comunicazione &egrave;<br>
strettamente proibita anche ai sensi del decreto legislativo 196/03 . Se<br>
avete ricevuto questo messaggio per errore, vi preghiamo di notificarcelo<br>
immediatamente a mezzo e-mail di risposta e successivamente di procedere<br>
alla cancellazione di questa e-mail e relativi allegati dal vostro sistema.<br>
_________________________________________________________________________________________________________________<br>
This message is intended only for the use of the addressee and may<br>
contain information that is privileged, confidential and exempt from<br>
disclosure under applicable law. If the reader of this message is not<br>
the intended recipient, or the employee or agent responsible for<br>
delivering the<br>
message to the intended recipient, you are hereby notified that any<br>
dissemination, distribution or copying of this communication is strictly<br>
prohibited. If you have received this e-mail in error, please notify us<br>
immediately by return e-mail and delete this e-mail and all attachments<br>
from<br>
your system.<br>
_________________________________________________________________________________________________________________<br><br><br><br>
_______________________________________________<br>
rabbitmq-discuss mailing list<br><a href="mailto:rabbitmq-discuss@..." target="_blank">rabbitmq-discuss@...</a><br><a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
</div>
</div>
</blockquote>
<div>
<div></div>
<div class="h5">
<br>
_______________________________________________<br>
rabbitmq-discuss mailing list<br><a href="mailto:rabbitmq-discuss@..." target="_blank">rabbitmq-discuss@...</a><br><a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
</div>
</div>
</blockquote>
</div>
<br>
</div></div>
</div>
Permalink | Reply |
headers
Jim Irrer | 1 Jun 2011 15:46
Picon

Re: How to handle full exchange history delivery

This sounds similar to this conversation:



Maybe the plugin is ready?

Thanks,

- Jim

Jim Irrer     irrer-63aXycvo3TyHXe+LvDLADg@public.gmane.org       (734) 647-4409
University of Michigan Hospital Radiation Oncology
519 W. William St.             Ann Arbor, MI 48103


On Wed, Jun 1, 2011 at 9:38 AM, Jon Brisbin <jon <at> jbrisbin.com> wrote:
I've been seriously considering adding this functionality to the Riak-backed custom exchange. It was something I thought I would add first time through but didn't.

You'd have to use a custom exchange. Not sure if that's an issue in your scenario or not.


Thanks!

Jon Brisbin
http//jbrisbin.com


From: "Simone Busoli" <Simone.Busoli-sKiPNUh8iG9BDgjK7y7TUQ@public.gmane.org>
To: rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org
Sent: Wednesday, June 1, 2011 5:52:24 AM
Subject: [rabbitmq-discuss] How to handle full exchange history delivery


Hi,

 

I am trying to achieve a scenario in which messages are published from some data source into a RabbitMQ exchange and clients should be able to receive the full history of messages flown through that exchange, regardless of the exact moment in time when they connect.

 

I’ve been thinking to solve it by setting up a proxy which subscribes to the exchange where messages are published using its own queue, keeps an internal storage of the messages and accepts requests from clients.

When a request from a client arrives it just replays the full message history to the client by publishing the messages to the client queue (supplied in the ReplyTo field), then binds the client queue to the messages exchange, so that the client from this moment onwards starts receiving the “live” messages, and is pretty much unaware of what’s happened; it simply just receives the whole history.

 

Now I am under the impression that there’s a fundamental flaw in this reasoning, since the proxy cannot know when the message history has completed and he can switch the client to the live stream, because while it is replaying historical messages to the client other live messages might be arriving on the proxy’s queue, thus getting into an endless loop.

 

I’m probably missing something obvious, any advice about how others would approach this scenario is appreciated.

 

Simone

_________________________________________________________________________________________________________________
Questo messaggio e da intendersi esclusivamente ad uso del destinatario e puo contenere informazioni che sono di natura privilegiata, confidenziale
o non divulgabile secondo le leggi vigenti. Se il lettore del presente messaggio non e il destinatario designato, o il dipendente/agente responsabile
per la consegna del messaggio al destinatario designato, si informa che ogni disseminazione, distribuzione o copiatura di questa comunicazione e
strettamente proibita anche ai sensi del decreto legislativo 196/03 . Se avete ricevuto questo messaggio per errore, vi preghiamo di notificarcelo
immediatamente a mezzo e-mail di risposta e successivamente di procedere alla cancellazione di questa e-mail e relativi allegati dal vostro sistema.
_________________________________________________________________________________________________________________
This message is intended only for the use of the addressee and may contain information that is privileged, confidential and exempt from
disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the
message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly
prohibited. If you have received this e-mail in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from
your system.
_________________________________________________________________________________________________________________


_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss-ETbvJ2rUIr4qBm01orBoR9BPR1lH4CV8@public.gmane.org
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss


_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss <at> lists.rabbitmq.com
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss


<div>
<p>This sounds similar to this conversation:</p>
<div><br></div>
<blockquote class="webkit-indent-blockquote">
<div><a href="http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2011-March/011933.html">http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2011-March/011933.html</a></div>

</blockquote>
<div><br></div>
<div>Maybe the plugin is ready?</div>
<div>
<br>Thanks,<br><br>- Jim<br><br>Jim Irrer&nbsp; &nbsp;&nbsp; <a href="mailto:irrer <at> umich.edu">irrer@...</a>&nbsp; &nbsp; &nbsp;&nbsp; (734) 647-4409<br>University of Michigan Hospital Radiation Oncology<br>

519 W. William St.&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp; Ann Arbor, MI 48103<br><br><br><div class="gmail_quote">On Wed, Jun 1, 2011 at 9:38 AM, Jon Brisbin <span dir="ltr">&lt;<a href="mailto:jon@...">jon <at> jbrisbin.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote">

<div><div>I've been seriously considering adding this functionality to the Riak-backed custom exchange. It was something I thought I would add first time through but didn't.<br><br>You'd have to use a custom exchange. Not sure if that's an issue in your scenario or not.<br><span><br><span name="x"></span><br>Thanks!<br><br>Jon Brisbin<br>http//<a href="http://jbrisbin.com" target="_blank">jbrisbin.com</a><br><span name="x"></span><br></span><br><blockquote>

From: "Simone Busoli" &lt;<a href="mailto:Simone.Busoli <at> ferrari.com" target="_blank">Simone.Busoli@...</a>&gt;<br>To: <a href="mailto:rabbitmq-discuss@..." target="_blank">rabbitmq-discuss@...</a><br>Sent: Wednesday, June 1, 2011 5:52:24 AM<br>Subject: [rabbitmq-discuss] How to handle full exchange history delivery<div>
<div></div>
<div class="h5">
<br><br><div>
<p class="MsoNormal">Hi,</p>
<p class="MsoNormal">&nbsp;</p>
<p class="MsoNormal"><span lang="EN-US">I am trying to achieve a scenario in which messages are published from some data source into a RabbitMQ exchange and clients should be able to receive the full history of messages flown through that exchange, regardless
 of the exact moment in time when they connect.</span></p>
<p class="MsoNormal"><span lang="EN-US">&nbsp;</span></p>
<p class="MsoNormal"><span lang="EN-US">I&rsquo;ve been thinking to solve it by setting up a proxy which subscribes to the exchange where messages are published using its own queue, keeps an internal storage of the messages and accepts requests from clients.
</span></p>
<p class="MsoNormal"><span lang="EN-US">When a request from a client arrives it just replays the full message history to the client by publishing the messages to the client queue (supplied in the ReplyTo field),
then binds the client queue to the messages exchange, so that the client from this moment onwards starts receiving the &ldquo;live&rdquo; messages, and is pretty much unaware of what&rsquo;s happened; it simply just receives the whole history.</span></p>

<p class="MsoNormal"><span lang="EN-US">&nbsp;</span></p>
<p class="MsoNormal"><span lang="EN-US">Now I am under the impression that there&rsquo;s a fundamental flaw in this reasoning, since the proxy cannot know when the message history has completed and he can switch the client to the live stream, because while it is
 replaying historical messages to the client other live messages might be arriving on the proxy&rsquo;s queue, thus getting into an endless loop.</span></p>
<p class="MsoNormal"><span lang="EN-US">&nbsp;</span></p>
<p class="MsoNormal"><span lang="EN-US">I&rsquo;m probably missing something obvious, any advice about how others would approach this scenario is appreciated.</span></p>
<p class="MsoNormal"><span lang="EN-US">&nbsp;</span></p>
<p class="MsoNormal"><span lang="EN-US">Simone</span></p>
</div>
<p>_________________________________________________________________________________________________________________<br>Questo messaggio e da intendersi esclusivamente ad uso del destinatario e puo contenere informazioni che sono di natura privilegiata, confidenziale<br>

o non divulgabile secondo le leggi vigenti. Se il lettore del presente messaggio non e il destinatario designato, o il dipendente/agente responsabile<br>per la consegna del messaggio al destinatario designato, si informa che ogni disseminazione, distribuzione o copiatura di questa comunicazione e <br>

strettamente proibita anche ai sensi del decreto legislativo 196/03 . Se avete ricevuto questo messaggio per errore, vi preghiamo di notificarcelo<br>immediatamente a mezzo e-mail di risposta e successivamente di procedere alla cancellazione di questa e-mail e relativi allegati dal vostro sistema.<br>

_________________________________________________________________________________________________________________<br>This message is intended only for the use of the addressee and may contain information that is privileged, confidential and exempt from <br>

disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the <br>message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly <br>

prohibited. If you have received this e-mail in error, please notify us immediately by return e-mail and delete this e-mail and all attachments from <br>your system.<br>_________________________________________________________________________________________________________________<br></p>
<br>
</div>
</div>_______________________________________________<br>rabbitmq-discuss mailing list<br><a href="mailto:rabbitmq-discuss <at> lists.rabbitmq.com" target="_blank">rabbitmq-discuss@...</a><br><a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br>
</blockquote>
<br>
</div></div>
<br>_______________________________________________<br>

rabbitmq-discuss mailing list<br><a href="mailto:rabbitmq-discuss@...">rabbitmq-discuss <at> lists.rabbitmq.com</a><br><a href="https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss" target="_blank">https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss</a><br><br>
</blockquote>
</div>
<br>
</div>
</div>
Permalink | Reply |

Gmane