headers
Martin Sustrik | 10 Jul 2011 12:57
Gravatar

ZMQ_IDENTITY revisited

Hi all,

Some of you may recall I've asked about whether ZMQ_IDENTITY option can 
be retired in 0MQ/3.0 some time ago. The majority asked not to remove 
the option at the time as it's being used in products.

I've promised to retain the option and that's the status quo right now. 
ZMQ_IDENITY is still available in 0MQ/3.0 (master).

However, I see a problem with this arrangement.

Given, that ZMQ_IDENTITY is a biggest source of complexity in the 
codebase (while at the same time being semantically unsound) I'll have 
to remove it from the codebase sooner or later be able to develop the 
product further.

So, what will happen is that we'll have 0MQ/2.x (old API, ZMQ_IDENTITY), 
0MQ/3.x (new API, ZMQ_IDENTITY) and 0MQ/4.x (new API, no ZMQ_IDENTITY).

0MQ/2.x is maintained by Pieter, 0MQ/4.x (the master) will be maintained 
by myself, but it's not clear who's going to do maintenance (backporting 
bugfixes etc.) of 0MQ/3.x.

If there's no maintainer, the version is going to bitrot pretty quickly, 
adding additional trouble for binding maintainers while providing no 
added value.

So, my question is: Is there anybody out there who needs ZMQ_IDENTITY so 
badly as to volunteer for maintaining 0MQ/3.0?
(Continue reading)

Permalink | Reply |
headers
Pieter Hintjens | 10 Jul 2011 15:23
Gravatar

Re: ZMQ_IDENTITY revisited

On Sun, Jul 10, 2011 at 12:57 PM, Martin Sustrik <sustrik <at> 250bpm.com> wrote:

> So, what will happen is that we'll have 0MQ/2.x (old API, ZMQ_IDENTITY),
> 0MQ/3.x (new API, ZMQ_IDENTITY) and 0MQ/4.x (new API, no ZMQ_IDENTITY).

So you're saying you want to skip from stable 2.x to 4.x without a
stable 3.x in between?

> 0MQ/2.x is maintained by Pieter, 0MQ/4.x (the master) will be maintained
> by myself, but it's not clear who's going to do maintenance (backporting
> bugfixes etc.) of 0MQ/3.x.

Backporting & maintenance is 100% a function of demand. No users = no
demand. Since there have been no packages for 3.x, it's fair to assume
there are _very_ few users.

> If not so, there's no much point in releasing 0MQ/3.0 and we should move
> directly to 0MQ/4.0, ie. new API without ZMQ_IDENTITY.

There's a few points here.

Most dangerously, Version 2 Syndrome. You are planning, afaics, to
change three major aspects of the stack at once. The external API, the
wire level protocol, and the internal architecture. This is risky. I'd
generally advise against it, except that (a) it worked for v2, and (b)
we have v2, so if this fails, it's not a major problem. But do watch
out for "change everything at once" syndrome. It's safer and often
easier to get stability in each area of change before starting on the
next one.
(Continue reading)

Permalink | Reply |
headers
Pieter Hintjens | 10 Jul 2011 15:56
Gravatar

Re: ZMQ_IDENTITY revisited

OK, after some IRC discussion, our plan is to:

- package up 3.0 soon, as a new 'unstable' release
- maintain the 3.0 branch as we've done for 2.0 and 2.1
- principal new feature being subscription forwarding
- continue to plan for removal of durable sockets
- aim to remove those in 3.1, once 3.0 is stable
- as 3.0 becomes stable, move 2.1.x to legacy status

So IOW I'm volunteering to package / maintain 3.0, taking it to stable
status. This should take us about 6 months from now.

-Pieter
Permalink | Reply |
headers
Martin Sustrik | 10 Jul 2011 17:45
Gravatar

Re: ZMQ_IDENTITY revisited

On 07/10/2011 03:56 PM, Pieter Hintjens wrote:

> So IOW I'm volunteering to package / maintain 3.0, taking it to stable
> status. This should take us about 6 months from now.

Great! Thanks!

For reference, here's the list of changes introduced by 0MQ/3.0:

* Obsolete constants ZMQ_UPSTREAM and ZMQ_DOWNSTREAM removed. Use 
ZMQ_PUSH and ZMQ_PULL instead.

* Timeout in zmq_poll is in milliseconds instead of microseconds. This 
makes zmq_poll() compliant with POSIX poll()

* ZMQ_MCAST_LOOP removed. There's no support for multicast over loopback 
any more. Use IPC or TCP isntead.

* Pre-built devices and zmq_device() removed. Should be made available 
as a separate project(s).

* ZMQ_SWAP removed. Writing data to disk should be done on top of 0MQ, 
on inside it.

* C++ binding removed from the core. Now it's a separate project, same 
way as any other binding.

* zmq_send/zmq_recv was renamed zmq_sendmsg/zmq_recvmsg.

* POSIX-compliant zmq_send and zmq_recv introduced (uses raw buffer
(Continue reading)

Permalink | Reply |
headers
Dirkjan Ochtman | 11 Jul 2011 09:19
Picon
Gravatar

Re: ZMQ_IDENTITY revisited

On Sun, Jul 10, 2011 at 17:45, Martin Sustrik <sustrik <at> 250bpm.com> wrote:
> * ZMQ_SWAP removed. Writing data to disk should be done on top of 0MQ,
> on inside it.

What's the rationale for this? Was there a lot of complexity?

Cheers,

Dirkjan
Permalink | Reply |
headers
Martin Sustrik | 11 Jul 2011 09:35
Gravatar

Re: ZMQ_IDENTITY revisited

On 07/11/2011 09:19 AM, Dirkjan Ochtman wrote:

>> * ZMQ_SWAP removed. Writing data to disk should be done on top of 0MQ,
>> on inside it.
>
> What's the rationale for this? Was there a lot of complexity?

It's wrong layering. It's something like if TCP buffers were kept on disk.

Writing data to disk normally happens in two layers:

- in the OS ("swap" per se)
- in the business logic (databases)

So, ZMQ_SWAP can be either considered to be a swapping feature that 
kicks in in case of memory overload, in which case it should be (and it 
is) implemented by the OS.

Or, it could be considered to a database, in which case it should be 
moved to the application on top of 0MQ.

Martin
Permalink | Reply |
headers
Pieter Hintjens | 11 Jul 2011 14:43
Gravatar

Re: ZMQ_IDENTITY revisited

On Mon, Jul 11, 2011 at 9:19 AM, Dirkjan Ochtman <dirkjan <at> ochtman.nl> wrote:
> On Sun, Jul 10, 2011 at 17:45, Martin Sustrik <sustrik <at> 250bpm.com> wrote:
>> * ZMQ_SWAP removed. Writing data to disk should be done on top of 0MQ,
>> on inside it.
>
> What's the rationale for this? Was there a lot of complexity?

Mainly, having it as a device, rather than a socket option, would make
it more useful. One could configure it, allow different qualities of
service, etc.

The big advantage of having this in libzmq is that it's available to
all bindings. But we can get much the same result by providing a
standard utility library, libzutil, that does this and other
functionality layered on top of the core.

-Pieter
Permalink | Reply |
headers
Dirkjan Ochtman | 11 Jul 2011 15:02
Picon
Gravatar

Re: ZMQ_IDENTITY revisited

On Mon, Jul 11, 2011 at 14:43, Pieter Hintjens <ph <at> imatix.com> wrote:
> Mainly, having it as a device, rather than a socket option, would make
> it more useful. One could configure it, allow different qualities of
> service, etc.

Ah, if there are helpers to get this right, that sounds fine.

Cheers,

Dirkjan
Permalink | Reply |
headers
Brian Granger | 13 Jul 2011 19:59
Picon
Gravatar

Re: ZMQ_IDENTITY revisited

Martin,

> Given, that ZMQ_IDENTITY is a biggest source of complexity in the
> codebase (while at the same time being semantically unsound) I'll have
> to remove it from the codebase sooner or later be able to develop the
> product further.
>
> So, what will happen is that we'll have 0MQ/2.x (old API, ZMQ_IDENTITY),
> 0MQ/3.x (new API, ZMQ_IDENTITY) and 0MQ/4.x (new API, no ZMQ_IDENTITY).

I think this plan sounds good as long as the new API has the same
capabilities as ZMQ_IDENTITY.

Cheers,

Brian

> 0MQ/2.x is maintained by Pieter, 0MQ/4.x (the master) will be maintained
> by myself, but it's not clear who's going to do maintenance (backporting
> bugfixes etc.) of 0MQ/3.x.
>
> If there's no maintainer, the version is going to bitrot pretty quickly,
> adding additional trouble for binding maintainers while providing no
> added value.
>
> So, my question is: Is there anybody out there who needs ZMQ_IDENTITY so
> badly as to volunteer for maintaining 0MQ/3.0?
>
> If not so, there's no much point in releasing 0MQ/3.0 and we should move
> directly to 0MQ/4.0, ie. new API without ZMQ_IDENTITY.
(Continue reading)

Permalink | Reply |

Gmane