User interface development mailing list for OLPC. ()

Re: Data Transport between nodes

Erik Blankinship wrote:
> How would you change the HelloMesh example to send binary data or an
> object?  Could you post a diff?  That would be useful for a reference. 
> Thanks!

This is a dangerous suggestion, but you may want to look at the source code for
my Acoustic Measure activity:
http://dev.laptop.org/git?p=projects/acoustic-measure-activity;a=summary

My code is directly based on HelloMesh, but I needed to send binary data over
the Tube.  The answer to your question specifically is basically one word:
base64.  Python has a base64 module providing b64encode and b64decode functions.
 I used these to send arbitrary bytes over the Tube, which only supports utf8.

--Ben

headers

Simon McVittie | 3 Oct 12:24

Re: Data Transport between nodes

On Tue, 02 Oct 2007 at 17:43:41 -0400, Benjamin M. Schwartz wrote:
>  I used these to send arbitrary bytes over the Tube, which only supports utf8.

This isn't actually true. If a D-Bus method is declared with signature 's'
(string) it only supports unicode or UTF-8 str objects. If no signature is
declared but you use a str argument, dbus-python will guess that you
wanted signature 's' and the string has to be UTF-8.

However, if your method is declared with signature 'ay' (byte array) you
can pass arbitrary binary data. You probably want to put
byte_arrays=True in the @method decorator, so the method will get its
input as a dbus.ByteArray (a subclass of str) rather than as a
dbus.Array of dbus.Bytes (subclasses of: a list of ints).

If your data has some other internal structure, other D-Bus types may be
more appropriate (e.g. you might want a list of floats, which in D-Bus
terminology would be an array of double, signature 'ad').

Similar notes apply to signals. See
http://dbus.freedesktop.org/doc/dbus-python/doc/tutorial.html for more
information.

headers

Erik Blankinship | 3 Oct 15:15

Re: Data Transport between nodes

On 10/3/07, Simon McVittie <simon.mcvittie <at> collabora.co.uk> wrote:

On Tue, 02 Oct 2007 at 17:43:41 -0400, Benjamin M. Schwartz wrote:
> I used these to send arbitrary bytes over the Tube, which only supports utf8.

This isn't actually true. If a D-Bus method is declared with signature 's'
(string) it only supports unicode or UTF-8 str objects. If no signature is
declared but you use a str argument, dbus-python will guess that you
wanted signature 's' and the string has to be UTF-8.

However, if your method is declared with signature 'ay' (byte array) you
can pass arbitrary binary data. You probably want to put
byte_arrays=True in the <at> method decorator, so the method will get its
input as a dbus.ByteArray (a subclass of str) rather than as a
dbus.Array of dbus.Bytes (subclasses of: a list of ints).

I hope this is a simple question: what is the preferred way to convert a file into a ByteArray for use with dBus?

_______________________________________________
Sugar mailing list
Sugar <at> lists.laptop.org
http://lists.laptop.org/listinfo/sugar

headers

Dafydd Harries | 14 Dec 13:03

Re: Data Transport between nodes

Ar 03/10/2007 am 09:15, ysgrifennodd Erik Blankinship:
> On 10/3/07, Simon McVittie <simon.mcvittie <at> collabora.co.uk> wrote:
> >
> > On Tue, 02 Oct 2007 at 17:43:41 -0400, Benjamin M. Schwartz wrote:
> > >  I used these to send arbitrary bytes over the Tube, which only supports
> > utf8.
> >
> > This isn't actually true. If a D-Bus method is declared with signature 's'
> > (string) it only supports unicode or UTF-8 str objects. If no signature is
> > declared but you use a str argument, dbus-python will guess that you
> > wanted signature 's' and the string has to be UTF-8.
> >
> > However, if your method is declared with signature 'ay' (byte array) you
> > can pass arbitrary binary data. You probably want to put
> > byte_arrays=True in the @method decorator, so the method will get its
> > input as a dbus.ByteArray (a subclass of str) rather than as a
> > dbus.Array of dbus.Bytes (subclasses of: a list of ints).
> 
> 
> I hope this is a simple question: what is the preferred way to convert a
> file into a ByteArray for use with dBus?

contents = file(path).read()
dbus.ByteArray(contents)

--

-- 
Dafydd

headers

Ivan Krstić | 15 Dec 01:58

Re: Data Transport between nodes

On Dec 14, 2007, at 7:03 AM, Dafydd Harries wrote:
> contents = file(path).read()
> dbus.ByteArray(contents)

How is that possibly supposed to scale?

--
Ivan Krstić <krstic <at> solarsail.hcs.harvard.edu> | http://radian.org

_______________________________________________
Sugar mailing list
Sugar <at> lists.laptop.org
http://lists.laptop.org/listinfo/sugar

headers

Dafydd Harries | 15 Dec 02:16

Re: Data Transport between nodes

Ar 14/12/2007 am 19:58, ysgrifennodd Ivan Krstić:
> On Dec 14, 2007, at 7:03 AM, Dafydd Harries wrote:
> > contents = file(path).read()
> > dbus.ByteArray(contents)
> 
> 
> How is that possibly supposed to scale?

My intention was to illustrate how to use byte arrays. I did not mean to imply
any guarantee of scalability.

D-Bus, and D-Bus tubes in particular, are not suited to very large messages.
If you need to transport large files, I suggest using a stream tube instead.
Failing that, splitting the file into pieces before transporting it over D-Bus
would be prudent.

At any rate, I've already discussed this with Erik, and I'm sure knows what
he's doing.

--

-- 
Dafydd
_______________________________________________
Sugar mailing list
Sugar <at> lists.laptop.org
http://lists.laptop.org/listinfo/sugar

headers

Ivan Krstić | 15 Dec 02:21

Re: Data Transport between nodes

On Dec 14, 2007, at 8:16 PM, Dafydd Harries wrote:
> If you need to transport large files, I suggest using a stream tube  
> instead.

Do we have stream tube documentation?

> At any rate, I've already discussed this with Erik, and I'm sure  
> knows what
> he's doing.

Is this conversation or its outcome documented?

Cheers,

--
Ivan Krstić <krstic <at> solarsail.hcs.harvard.edu> | http://radian.org

_______________________________________________
Sugar mailing list
Sugar <at> lists.laptop.org
http://lists.laptop.org/listinfo/sugar

headers

Dafydd Harries | 15 Dec 13:48

Re: Data Transport between nodes

Ar 14/12/2007 am 20:21, ysgrifennodd Ivan Krstić:
> On Dec 14, 2007, at 8:16 PM, Dafydd Harries wrote:
> > If you need to transport large files, I suggest using a stream tube  
> > instead.
> 
> Do we have stream tube documentation?

Like all Telepathy interfaces, it's documented in the specification which Bert
linked to.

> > At any rate, I've already discussed this with Erik, and I'm sure  
> > knows what
> > he's doing.
> 
> 
> Is this conversation or its outcome documented?

No, this was an informal conversation we had in person a few months ago.

--

-- 
Dafydd
_______________________________________________
Sugar mailing list
Sugar <at> lists.laptop.org
http://lists.laptop.org/listinfo/sugar

headers

Bert Freudenberg | 15 Dec 08:02

Re: Data Transport between nodes

On Dec 15, 2007, at 7:06 , Ivan Krstić wrote:

> On Dec 14, 2007, at 8:16 PM, Dafydd Harries wrote:
>> If you need to transport large files, I suggest using a stream tube
>> instead.
>
> Do we have stream tube documentation?

Well, beginnings of "understandable" documentation are here:

http://wiki.laptop.org/go/Low-level_Activity_API#Tubes

And the real hard-core documentation is here:

http://telepathy.freedesktop.org/ 
spec-0.16.html#org.freedesktop.Telepathy.Channel.Type.Tubes

- Bert -

_______________________________________________
Sugar mailing list
Sugar <at> lists.laptop.org
http://lists.laptop.org/listinfo/sugar

headers

Ivan Krstić | 15 Dec 08:08

Re: Data Transport between nodes

On Dec 15, 2007, at 2:02 AM, Bert Freudenberg wrote:
> Well, beginnings of "understandable" documentation are here:
> http://wiki.laptop.org/go/Low-level_Activity_API#Tubes

Hmm. Does that mean sending a file over a stream tube requires the  
sender to bind a socket which, when connected to, starts spewing back  
the file contents? That seems awfully awkward.

--
Ivan Krstić <krstic <at> solarsail.hcs.harvard.edu> | http://radian.org

_______________________________________________
Sugar mailing list
Sugar <at> lists.laptop.org
http://lists.laptop.org/listinfo/sugar

headers

Bert Freudenberg | 15 Dec 09:03

Re: Data Transport between nodes

On Dec 15, 2007, at 12:53 , Ivan Krstić wrote:

> On Dec 15, 2007, at 2:02 AM, Bert Freudenberg wrote:
>> Well, beginnings of "understandable" documentation are here:
>> http://wiki.laptop.org/go/Low-level_Activity_API#Tubes
>
> Hmm. Does that mean sending a file over a stream tube requires the  
> sender to bind a socket which, when connected to, starts spewing  
> back the file contents? That seems awfully awkward.

Stream tubes are nothing more than socket forwarding/tunneling a TCP  
connection, so yes, you're on your own.

What the Read activity does is actually run a Web server, and making  
that available as a stream tube. At least this does not invent a new  
protocol ...

- Bert -

_______________________________________________
Sugar mailing list
Sugar <at> lists.laptop.org
http://lists.laptop.org/listinfo/sugar

headers

Bert Freudenberg | 15 Dec 09:03

[sugar] Data Transport between nodes

On Dec 15, 2007, at 12:53 , Ivan Krsti? wrote:

> On Dec 15, 2007, at 2:02 AM, Bert Freudenberg wrote:
>> Well, beginnings of "understandable" documentation are here:
>> http://wiki.laptop.org/go/Low-level_Activity_API#Tubes
>
> Hmm. Does that mean sending a file over a stream tube requires the  
> sender to bind a socket which, when connected to, starts spewing  
> back the file contents? That seems awfully awkward.

Stream tubes are nothing more than socket forwarding/tunneling a TCP  
connection, so yes, you're on your own.

What the Read activity does is actually run a Web server, and making  
that available as a stream tube. At least this does not invent a new  
protocol ...

- Bert -

headers

Ivan Krstić | 15 Dec 08:08

[sugar] Data Transport between nodes

On Dec 15, 2007, at 2:02 AM, Bert Freudenberg wrote:
> Well, beginnings of "understandable" documentation are here:
> http://wiki.laptop.org/go/Low-level_Activity_API#Tubes

Hmm. Does that mean sending a file over a stream tube requires the  
sender to bind a socket which, when connected to, starts spewing back  
the file contents? That seems awfully awkward.

--
Ivan Krsti? <krstic at solarsail.hcs.harvard.edu> | http://radian.org

headers

Dafydd Harries | 15 Dec 13:48

[sugar] Data Transport between nodes

Ar 14/12/2007 am 20:21, ysgrifennodd Ivan Krsti?:
> On Dec 14, 2007, at 8:16 PM, Dafydd Harries wrote:
> > If you need to transport large files, I suggest using a stream tube  
> > instead.
> 
> Do we have stream tube documentation?

Like all Telepathy interfaces, it's documented in the specification which Bert
linked to.

> > At any rate, I've already discussed this with Erik, and I'm sure  
> > knows what
> > he's doing.
> 
> 
> Is this conversation or its outcome documented?

No, this was an informal conversation we had in person a few months ago.

--

-- 
Dafydd

headers

Bert Freudenberg | 15 Dec 08:02

[sugar] Data Transport between nodes

On Dec 15, 2007, at 7:06 , Ivan Krsti? wrote:

> On Dec 14, 2007, at 8:16 PM, Dafydd Harries wrote:
>> If you need to transport large files, I suggest using a stream tube
>> instead.
>
> Do we have stream tube documentation?

Well, beginnings of "understandable" documentation are here:

http://wiki.laptop.org/go/Low-level_Activity_API#Tubes

And the real hard-core documentation is here:

http://telepathy.freedesktop.org/ 
spec-0.16.html#org.freedesktop.Telepathy.Channel.Type.Tubes

- Bert -

headers

Ivan Krstić | 15 Dec 02:21

[sugar] Data Transport between nodes

On Dec 14, 2007, at 8:16 PM, Dafydd Harries wrote:
> If you need to transport large files, I suggest using a stream tube  
> instead.

Do we have stream tube documentation?

> At any rate, I've already discussed this with Erik, and I'm sure  
> knows what
> he's doing.

Is this conversation or its outcome documented?

Cheers,

--
Ivan Krsti? <krstic at solarsail.hcs.harvard.edu> | http://radian.org

headers

Dafydd Harries | 15 Dec 02:16

[sugar] Data Transport between nodes

Ar 14/12/2007 am 19:58, ysgrifennodd Ivan Krsti?:
> On Dec 14, 2007, at 7:03 AM, Dafydd Harries wrote:
> > contents = file(path).read()
> > dbus.ByteArray(contents)
> 
> 
> How is that possibly supposed to scale?

My intention was to illustrate how to use byte arrays. I did not mean to imply
any guarantee of scalability.

D-Bus, and D-Bus tubes in particular, are not suited to very large messages.
If you need to transport large files, I suggest using a stream tube instead.
Failing that, splitting the file into pieces before transporting it over D-Bus
would be prudent.

At any rate, I've already discussed this with Erik, and I'm sure knows what
he's doing.

--

-- 
Dafydd

headers

Ivan Krstić | 15 Dec 01:58

[sugar] Data Transport between nodes

On Dec 14, 2007, at 7:03 AM, Dafydd Harries wrote:
> contents = file(path).read()
> dbus.ByteArray(contents)

How is that possibly supposed to scale?

--
Ivan Krsti? <krstic at solarsail.hcs.harvard.edu> | http://radian.org

headers

Dafydd Harries | 14 Dec 13:03

[sugar] Data Transport between nodes

Ar 03/10/2007 am 09:15, ysgrifennodd Erik Blankinship:
> On 10/3/07, Simon McVittie <simon.mcvittie at collabora.co.uk> wrote:
> >
> > On Tue, 02 Oct 2007 at 17:43:41 -0400, Benjamin M. Schwartz wrote:
> > >  I used these to send arbitrary bytes over the Tube, which only supports
> > utf8.
> >
> > This isn't actually true. If a D-Bus method is declared with signature 's'
> > (string) it only supports unicode or UTF-8 str objects. If no signature is
> > declared but you use a str argument, dbus-python will guess that you
> > wanted signature 's' and the string has to be UTF-8.
> >
> > However, if your method is declared with signature 'ay' (byte array) you
> > can pass arbitrary binary data. You probably want to put
> > byte_arrays=True in the @method decorator, so the method will get its
> > input as a dbus.ByteArray (a subclass of str) rather than as a
> > dbus.Array of dbus.Bytes (subclasses of: a list of ints).
> 
> 
> I hope this is a simple question: what is the preferred way to convert a
> file into a ByteArray for use with dBus?

contents = file(path).read()
dbus.ByteArray(contents)

--

-- 
Dafydd

headers

Erik Blankinship | 3 Oct 15:15

[sugar] Data Transport between nodes

On 10/3/07, Simon McVittie <simon.mcvittie at collabora.co.uk> wrote:
>
> On Tue, 02 Oct 2007 at 17:43:41 -0400, Benjamin M. Schwartz wrote:
> >  I used these to send arbitrary bytes over the Tube, which only supports
> utf8.
>
> This isn't actually true. If a D-Bus method is declared with signature 's'
> (string) it only supports unicode or UTF-8 str objects. If no signature is
> declared but you use a str argument, dbus-python will guess that you
> wanted signature 's' and the string has to be UTF-8.
>
> However, if your method is declared with signature 'ay' (byte array) you
> can pass arbitrary binary data. You probably want to put
> byte_arrays=True in the @method decorator, so the method will get its
> input as a dbus.ByteArray (a subclass of str) rather than as a
> dbus.Array of dbus.Bytes (subclasses of: a list of ints).

I hope this is a simple question: what is the preferred way to convert a
file into a ByteArray for use with dBus?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.laptop.org/pipermail/sugar/attachments/20071003/4d9bd66b/attachment.htm

headers

Simon McVittie | 3 Oct 12:24

[sugar] Data Transport between nodes

On Tue, 02 Oct 2007 at 17:43:41 -0400, Benjamin M. Schwartz wrote:
>  I used these to send arbitrary bytes over the Tube, which only supports utf8.

This isn't actually true. If a D-Bus method is declared with signature 's'
(string) it only supports unicode or UTF-8 str objects. If no signature is
declared but you use a str argument, dbus-python will guess that you
wanted signature 's' and the string has to be UTF-8.

However, if your method is declared with signature 'ay' (byte array) you
can pass arbitrary binary data. You probably want to put
byte_arrays=True in the @method decorator, so the method will get its
input as a dbus.ByteArray (a subclass of str) rather than as a
dbus.Array of dbus.Bytes (subclasses of: a list of ints).

If your data has some other internal structure, other D-Bus types may be
more appropriate (e.g. you might want a list of floats, which in D-Bus
terminology would be an array of double, signature 'ad').

Similar notes apply to signals. See
http://dbus.freedesktop.org/doc/dbus-python/doc/tutorial.html for more
information.