Re: Unpickling from Java

That looks tantalizingly close to what I need, but I'm running Jython
embedded in Java, so I figure I'd need to hack the Jython jar.  I'm not sure
that would go over well in my project, unfortunately.  For what it's worth,
Jython 2.5 unpickles my data just fine.  :-/

Any word on the stability of 2.5alpha?

Thanks,

Ulises Melendez

-----Original Message-----
From: Stefan Eischet [mailto:stefan <at> eischet.com] 
Sent: Friday, August 08, 2008 6:26 PM
To: Melendez,Ulises
Cc: Jeff Emanuel; jython-users <at> lists.sourceforge.net
Subject: Re: [Jython-users] Unpickling from Java

Okay, maybe this helps:

[stefan <at> flagship:~/pickletest]$ python
Python 2.5.1 (r251:54863, Apr 15 2008, 22:57:26)
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
 >>> obj = {'patches': ['73', '74', '75', '76', '77', '78', '79',  
'79x02'], 'source':
... 'host1.local.net', 'system_revision': '76', 'system_type': 'SA',
... 'creation_time': 1215221963, 'user': 'def', 'reason': 'USER',  
'comment': '',
... 'pol_version': (8, 1.21501435305E9), 'bund_type': 'FULL', 'domains':
... ['host1.local.net']}
 >>> import pickle
 >>> pickle.dumps(obj, protocol=2)
'\x80\x02}q\x00(U\x07commentq\x01U\x00q\x02U\x07patchesq\x03]q\x04(U 
\x0273q\x05U\x0274q\x06U\x0275q\x07U\x0276q\x08U\x0277q\tU\x0278q\nU 
\x0279q\x0bU\x0579x02q\x0ceU\rcreation_timeq\rJ\xcb\xd0nHU 
\x0fsystem_revisionq\x0eh\x08U\x06sourceq\x0fU\x0fhost1.local.netq\x10U 
\x06reasonq\x11U\x04USERq\x12U\x0bpol_versionq\x13K\x08GA\xd2\x1a 
\xe9tC33\x86q\x14U\x04userq\x15U\x03defq\x16U\x07domainsq\x17]q\x18h 
\x10aU\tbund_typeq\x19U\x04FULLq\x1aU\x0bsystem_typeq\x1bU\x02SAq\x1cu.'
 >>> ^D
[stefan <at> flagship:~/pickletest]$ ../opt/jython221/jython
Jython 2.2.1 on java1.5.0_13
Type "copyright", "credits" or "license" for more information.
 >>> import pickle
 >>> pickle.loads('\x80\x02}q\x00(U\x07commentq\x01U\x00q\x02U 
\x07patchesq\x03]q\x04(U\x0273q\x05U\x0274q\x06U\x0275q\x07U\x0276q 
\x08U\x0277q\tU\x0278q\nU\x0279q\x0bU\x0579x02q\x0ceU\rcreation_timeq 
\rJ\xcb\xd0nHU\x0fsystem_revisionq\x0eh\x08U\x06sourceq\x0fU 
\x0fhost1.local.netq\x10U\x06reasonq\x11U\x04USERq\x12U\x0bpol_versionq 
\x13K\x08GA\xd2\x1a\xe9tC33\x86q\x14U\x04userq\x15U\x03defq\x16U 
\x07domainsq\x17]q\x18h\x10aU\tbund_typeq\x19U\x04FULLq\x1aU 
\x0bsystem_typeq\x1bU\x02SAq\x1cu.')
{'source': 'host1.local.net', 'patches': ['73', '74', '75', '76',  
'77', '78', '79', '79x02'], 'system_revision': '76', 'bund_type':  
'FULL', 'system_type': 'SA', 'creation_time': 1215221963, 'user':  
'def', 'reason': 'USER', 'pol_version': (8, 1.21501435305E9),  
'comment': '', 'domains': ['host1.local.net']}

But Jython 2.2.1 does not support protocol=2, right? Well...   I  
just copied the Jython 2.5 version of pickle.py and copy_reg.py to the  
current directory. That means that "import pickle" does not use the  
Jython 2.2.1 version but instead loads the one I copied from 2.5. Does  
that work for you, too?

Regards,
Stefan

On 2008-08-08, at 23:29 , Melendez,Ulises wrote:

> Stefan,
>
> Thanks again for your explanation.  I ran similar tests myself, but I
> wondered if there might not be some setting that needed switching in  
> order
> to support protocol v2 (particularly because I noticed the  
> HIGHEST_PROTOCOL
> field was present).  I understand that's not the case.
>
> My data structure is only marginally more complex than the ones you
> illustrated.  Here's an example:
>
> {'patches': ['73', '74', '75', '76', '77', '78', '79', '79x02'],  
> 'source':
> 'host1.local.net', 'system_revision': '76', 'system_type': 'SA',
> 'creation_time': 1215221963, 'user': 'def', 'reason': 'USER',  
> 'comment': '',
> 'pol_version': (8, 1.21501435305E9), 'bund_type': 'FULL', 'domains':
> ['host1.local.net']}
>
> I see you're able to unpickle protocol v2 using Jython 2.5.  Any  
> idea when
> that's going to become stable?  I would be using Jython exclusively  
> for
> unpickling.  Would it still be inadvisable to use 2.5 in this  
> limited scope?
>
> Oh, and the pickled data that I receive is out of my control, so I'm  
> not
> able to revert to an earlier protocol.
>
>
> Ulises Melendez
>
> -----Original Message-----
> From: Stefan Eischet [mailto:stefan <at> eischet.com]
> Sent: Friday, August 08, 2008 5:11 PM
> To: Melendez,Ulises
> Cc: Jeff Emanuel; jython-users <at> lists.sourceforge.net
> Subject: Re: [Jython-users] Unpickling from Java
>
> Okay, let's do some more analysis. The idea here is to pickle a simple
> object (a tuple of strings) in Python and to unpickle it in Jython
> 2.2.1 and 2.5a1.
> Let's see what happens:
>
> [stefan <at> flagship:~]$ python
> Python 2.5.1 (r251:54863, Apr 15 2008, 22:57:26)
> [GCC 4.0.1 (Apple Inc. build 5465)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
>>>> import pickle
>>>> obj = ('I', 'am', 'a', 'simple', 'tuple', )
>>>> pickle.dumps(obj, protocol=0) # use protocol=0 - backwards
> compatible ASCII encoding
> "(S'I'\np0\nS'am'\np1\nS'a'\np2\nS'simple'\np3\nS'tuple'\np4\ntp5\n."
>>>> pickle.dumps(obj, protocol=1) # another protocol
> '(U\x01Iq\x00U\x02amq\x01U\x01aq\x02U\x06simpleq\x03U\x05tupleq\x04tq
> \x05.'
>>>> pickle.dumps(obj, protocol=2) # yet another protocol
> '\x80\x02(U\x01Iq\x00U\x02amq\x01U\x01aq\x02U\x06simpleq\x03U 
> \x05tupleq
> \x04tq\x05.'
>>>> ^D
>
> You'll notice that, depending on the protocol parameter, you get
> different string representations.
> When unpickling, the pickle module should, theoretically,
> automatically recognize the encoding and turn this string back into a
> proper object.
>
> With Jython 2.2.1:
>
> [stefan <at> flagship:~]$ opt/jython221/jython
> Jython 2.2.1 on java1.5.0_13
> Type "copyright", "credits" or "license" for more information.
>>>> import pickle
>>>>
> pickle
> .loads
> ("(S'I
> '\np0\nS'am'\np1\nS'a'\np2\nS'simple'\np3\nS'tuple'\np4\ntp5\n.") #
> protocol=0
> ('I', 'am', 'a', 'simple', 'tuple')
>>>> pickle.loads('(U\x01Iq\x00U\x02amq\x01U\x01aq\x02U\x06simpleq\x03U
> \x05tupleq\x04tq\x05.') # protocol=1
> ('I', 'am', 'a', 'simple', 'tuple')
>>>> pickle.loads('\x80\x02(U\x01Iq\x00U\x02amq\x01U\x01aq\x02U
> \x06simpleq\x03U\x05tupleq\x04tq\x05.') # protocol=2
> Traceback (innermost last):
>   File "<console>", line 1, in ?
>   File "/Users/stefan/opt/jython221/Lib/pickle.py", line 990, in loads
>   File "/Users/stefan/opt/jython221/Lib/pickle.py", line 599, in load
> KeyError: ?
>>>> ^D
>
> Aha, the last one doesn't work. Obviously, Jython 2.2.1 does not
> implement that protocol... Which makes perfect sense when you look at
> http://docs.python.org/lib/node315.html
> , which states that protocol=2 has been added in Python 2.3. (I have
> to admit that the docs are slightly confusing here - the protocol
> parameter itself was also added in Python 2.3, but it IS available in
> Jython 2.2.1. Oh well, that why we have the interactive interpreter -
> to try things out ourselves.  )
>
> Using Jython 2.5a1, I get better results:
>
> [stefan <at> flagship:~]$ opt/jython25a1/jython
> Jython 2.5a1+ (asm:4943:4945, Jul 15 2008, 15:30:04)
> [Java HotSpot(TM) Client VM (Apple Inc.)] on java1.5.0_13
> Type "help", "copyright", "credits" or "license" for more information.
>>>> import pickle
>>>>
> pickle
> .loads
> ("(S'I
> '\np0\nS'am'\np1\nS'a'\np2\nS'simple'\np3\nS'tuple'\np4\ntp5\n.") #
> protocol=0
> ('I', 'am', 'a', 'simple', 'tuple')
>>>> pickle.loads('\x80\x02(U\x01Iq\x00U\x02amq\x01U\x01aq\x02U
> \x06simpleq\x03U\x05tupleq\x04tq\x05.') # protocol=1
> ('I', 'am', 'a', 'simple', 'tuple')
>>>> pickle.loads('(U\x01Iq\x00U\x02amq\x01U\x01aq\x02U\x06simpleq\x03U
> \x05tupleq\x04tq\x05.') # protocol=2
> ('I', 'am', 'a', 'simple', 'tuple')
>
> Here, all protocol versions work. Which makes sense, because the
> Jython developers are now using up-to-date Python library modules,
> eliminating the "lag" Jython has had in the past when compared to the
> Python library. I can't praise them enough for this. 
>
> I think you should make sure that you're using protocol=0 or just omit
> the protocol parameter when pickling your objects.
> Also, in your code, look for data types that have been introduced
> after Python 2.3... maybe that's your problem:
>
> Python 2.5.1 (r251:54863, Apr 15 2008, 22:57:26)
> [GCC 4.0.1 (Apple Inc. build 5465)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
>>>> import pickle
>>>> pickle.dumps(set())
> 'c__builtin__\nset\np0\n((lp1\ntp2\nRp3\n.'
>>>> pickle.dumps(set((1,2,3,)))
> 'c__builtin__\nset\np0\n((lp1\nI1\naI2\naI3\natp2\nRp3\n.'
>
> [stefan <at> flagship:~]$ opt/jython221/jython
> Jython 2.2.1 on java1.5.0_13
> Type "copyright", "credits" or "license" for more information.
>>>> import pickle
>>>> pickle.loads('c__builtin__\nset\np0\n((lp1\ntp2\nRp3\n.')
> Traceback (innermost last):
>   File "<console>", line 1, in ?
>   File "/Users/stefan/opt/jython221/Lib/pickle.py", line 990, in loads
>   File "/Users/stefan/opt/jython221/Lib/pickle.py", line 599, in load
>   File "/Users/stefan/opt/jython221/Lib/pickle.py", line 820, in
> load_global
>   File "/Users/stefan/opt/jython221/Lib/pickle.py", line 827, in
> find_class
> AttributeError: 'module' object has no attribute 'set'
>>>>
>
> Here, Python nicely pickles sets in protocol=0, but Jython doesn't
> have that builtin type and so it cannot unpickle them, even though it
> understands the format.
> But we can cheat a bit, because (in this case) there is a similar
> class available that might just do the trick:
>
> In Jython 2.2.1 again:
>>>> import sets
>>>> import __builtin__
>>>> __builtin__.set = sets.Set
>>>> pickle.loads('c__builtin__\nset\np0\n((lp1\ntp2\nRp3\n.')
> Set([])
>>>> pickle.loads('c__builtin__\nset
> \np0\n((lp1\nI1\naI2\naI3\natp2\nRp3\n.') # not shown in Python:
> pickle.dumps(set((1,2,3,)))
> Set([2, 1, 3])
>
> Maybe you can do something similar to unpickle your objects. Without
> seeing the objects you're actually trying to unpickle, I'm afraid
> there's not much else we can do to help you. After searching the
> Python library docs for possible incompatibilities and doing some
> experiments yourself, maybe you can post some sample code we could
> look at.
>
> Hope this helps,
> Stefan
>
>
> On 2008-08-08, at 19:09 , Melendez,Ulises wrote:
>
>> Jeff,
>>
>> Thanks for the reply.  I realize now I was encountering several
>> problems in
>> trying to unpickle.  I think I've narrowed down my hurdles to just
>> one.  I
>> seem to be hitting a roadblock due to the pickling protocol.  I'm
>> not sure
>> how to determine what protocol version I'm unpickling, but I think an
>> incompatible version is what's preventing jython from unpickling.
>>
>> Following is the error I get:
>> Exception in thread "main" Traceback (innermost last):
>>   (no code object) at line 0
>> TypeError: unhashable type
>>
>> I'm not sure whether that's useful, but I'm fairly confident the
>> problem is
>> unpickling protocol 2.  Is there a way to get this to work using
>> Jython?
>> I'm using version 2.2.1 of the jython jar.
>>
>> Thanks,
>>
>>
>> Ulises Melendez
>> -----Original Message-----
>> From: Jeff Emanuel [mailto:jemanuel <at> frii.com]
>> Sent: Thursday, August 07, 2008 11:15 AM
>> To: Melendez,Ulises
>> Cc: Stefan Eischet; jython-users <at> lists.sourceforge.net
>> Subject: Re: [Jython-users] Unpickling from Java
>>
>> Just execute the same code with your embedded
>> Jython interpreter.  Then use get() if you want
>> the data in Java.
>>
>> http://www.jython.org/Project/userguide.html#embedding-jython
>>
>> Melendez,Ulises wrote:
>>> Stefan,
>>>
>>>
>>>
>>> Thanks very much for the example and suggestions.  Unfortunately, I
>>> have
>> no
>>> control over how the data are serialized.
>>>
>>>
>>>
>>> Would you mind also providing an example of depickling from within
>>> Java
>>> (i.e. using the jython jar)?  This is really the usage I'm trying
>>> to get
>>> running, and I've had a very tough time.
>>>
>>>
>>>
>>> Thanks again,
>>>
>>>
>>>
>>>
>>>
>>> Ulises Melendez
>>>
>>> ________________________________
>>>
>>> From: Stefan Eischet [mailto:stefan <at> eischet.com]
>>> Sent: Thursday, August 07, 2008 3:05 AM
>>> To: Melendez,Ulises
>>> Cc: jython-users <at> lists.sourceforge.net
>>> Subject: Re: [Jython-users] Unpickling from Java
>>>
>>>
>>>
>>> Hello Ulises,
>>>
>>>
>>>
>>> to confirm that the basics work, I've just tried unpickling a
>>> simple tuple
>>> successfully:
>>>
>>>
>>>
>>> $ python
>>>
>>> Python 2.5.1 (r251:54863, Apr 15 2008, 22:57:26)
>>>
>>> [GCC 4.0.1 (Apple Inc. build 5465)] on darwin
>>>
>>> Type "help", "copyright", "credits" or "license" for more
>>> information.
>>>
>>>
>>>>>> import pickle
>>>>>>
>>>
>>>
>>>>>> data = (1,2,3,4,5,)
>>>>>>
>>>
>>>
>>>>>> pickle.dumps(data)
>>>>>>
>>>
>>> '(I1\nI2\nI3\nI4\nI5\ntp0\n.'
>>>
>>>
>>>>>> ^D
>>>>>>
>>>
>>>
>>>
>>> $ jython
>>>
>>> Jython 2.5a1+ (asm:4943:4945, Jul 15 2008, 15:30:04)
>>>
>>> [Java HotSpot(TM) Client VM (Apple Inc.)] on java1.5.0_13
>>>
>>> Type "help", "copyright", "credits" or "license" for more
>>> information.
>>>
>>>
>>>>>> import pickle
>>>>>>
>>>
>>>
>>>>>> pickle.loads('(I1\nI2\nI3\nI4\nI5\ntp0\n.')
>>>>>>
>>>
>>> (1, 2, 3, 4, 5)
>>>
>>>
>>>>>> ^D
>>>>>>
>>>
>>>
>>>
>>> YMMV with other Jython versions, of course. Two things you might
>>> want to
>>> look at are the protocol parameter of the pickle module (which
>>> defaults to
>>> 0, and which should be fine... see
>> http://docs.python.org/lib/node315.html).
>>> There's also a discussion about certain problems and workarounds for
>>> floating point values:
>>>
>>
>
http://coding.derkeiler.com/Archive/Python/comp.lang.python/2005-09/msg01079
>>> .html
>>>
>>>
>>>
>>> What I usually do instead of pickling is to serialize/deserialize
>>> simpler
>>> data structures (mainly dicts) into JSON: http://www.JSON.org/ and
>>> read
>> them
>>> from Java code. So, instead of pickling an object I just dump its
>>> interesting properties into a dict and read them back from the dict
>>> representation when needed. This also works fine when using plain
>>> Java
>>> instead of Java+Jython. Maybe you'll have more success with that
>>> approach.
>>>
>>>
>>>
>>> Regards
>>>
>>> Stefan
>>>
>>>
>>>
>>> On 2008-08-07, at 02:10 , Melendez,Ulises wrote:
>>>
>>>
>>>
>>>
>>>
>>> I have a python object that's been pickled to a file, and I need to
>> unpickle
>>> it from within a Java VM.  I've searched around and fooled around
>>> with
>>> cPickle (.java), but I'm having no such luck.  Can anyone point me
>>> to an
>>> example of unpickling a pickled object within Java?  I can also
>>> provide
>> some
>>> of my failed attempts for public ridicule, if that would help.
>>>
>>>
>>>
>>> Thanks in advance,
>>>
>>>
>>>
>>>
>>>
>>> Ulises Melendez
>>>
>>>
>>>
>>>
-------------------------------------------------------------------------
>>> This SF.Net email is sponsored by the Moblin Your Move Developer's
>> challenge
>>> Build the coolest Linux based applications with Moblin SDK & win
>>> great
>>> prizes
>>> Grand prize is a trip for two to an Open Source event anywhere in  
>>> the
>> world
>>>
>>
>
http://moblin-contest.org/redirect.php?banner_id=100&url=/__________________
>>> _____________________________
>>> Jython-users mailing list
>>> Jython-users <at> lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/jython-users
>>>
>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------
>>>
>>>
-------------------------------------------------------------------------
>>> This SF.Net email is sponsored by the Moblin Your Move Developer's
>> challenge
>>> Build the coolest Linux based applications with Moblin SDK & win
>>> great
>> prizes
>>> Grand prize is a trip for two to an Open Source event anywhere in  
>>> the
>> world
>>> http://moblin-contest.org/redirect.php?banner_id=100&url=/
>>> ------------------------------------------------------------------------
>>>
>>> _______________________________________________
>>> Jython-users mailing list
>>> Jython-users <at> lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/jython-users
>>>
>>
>>
>> -------------------------------------------------------------------------
>> This SF.Net email is sponsored by the Moblin Your Move Developer's
>> challenge
>> Build the coolest Linux based applications with Moblin SDK & win
>> great prizes
>> Grand prize is a trip for two to an Open Source event anywhere in
>> the world
>> http://moblin-contest.org/redirect.php?banner_id=100&url=/
>> _______________________________________________
>> Jython-users mailing list
>> Jython-users <at> lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/jython-users
>
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's  
> challenge
> Build the coolest Linux based applications with Moblin SDK & win  
> great prizes
> Grand prize is a trip for two to an Open Source event anywhere in  
> the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> Jython-users mailing list
> Jython-users <at> lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/jython-users

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/