Matthew Brett | 20 Jun 2012 22:48
Picon
Gravatar

Bizarre errors with byteswapping, complex256, PPC

Hi,

Our Debian friends were hammering our code tests before the upcoming
freeze, and found the following very odd thing on 32-bit PPC running
Debian squeeze and numpy 1.6.2 or current trunk.

Consider the following script:

<script>
import numpy as np

arr = np.arange(10, dtype=np.complex256)
bs_arr = arr.byteswap().newbyteorder('S')
print arr
print bs_arr
print arr == bs_arr
print arr == bs_arr
print arr == bs_arr
</script>

Here is the output from some example runs of this script:

np-devel)[mb312 <at> joshlegacy ~/tmp]$ python funny_bs.py
[ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
  7.0+0.0j  8.0+0.0j  9.0+0.0j]
[ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
  7.0+0.0j  8.0+0.0j  9.0+0.0j]
[ True False False False False False False False False False]
[ True False False False False False False False False False]
[ True False False False False False False False False False]
(Continue reading)

Travis Oliphant | 20 Jun 2012 22:56
Favicon
Gravatar

Re: Bizarre errors with byteswapping, complex256, PPC

This looks like a problem with comparisons of floating point numbers rather than a byteswapping problem
per-say.   Try to use an almost equal comparison instead.  

-Travis

On Jun 20, 2012, at 3:48 PM, Matthew Brett wrote:

> Hi,
> 
> Our Debian friends were hammering our code tests before the upcoming
> freeze, and found the following very odd thing on 32-bit PPC running
> Debian squeeze and numpy 1.6.2 or current trunk.
> 
> Consider the following script:
> 
> <script>
> import numpy as np
> 
> arr = np.arange(10, dtype=np.complex256)
> bs_arr = arr.byteswap().newbyteorder('S')
> print arr
> print bs_arr
> print arr == bs_arr
> print arr == bs_arr
> print arr == bs_arr
> </script>
> 
> Here is the output from some example runs of this script:
> 
> np-devel)[mb312 <at> joshlegacy ~/tmp]$ python funny_bs.py
(Continue reading)

Matthew Brett | 21 Jun 2012 00:00
Picon
Gravatar

Re: Bizarre errors with byteswapping, complex256, PPC

Hi,

On Wed, Jun 20, 2012 at 1:56 PM, Travis Oliphant <travis <at> continuum.io> wrote:
> This looks like a problem with comparisons of floating point numbers rather than a byteswapping problem
per-say.   Try to use an almost equal comparison instead.

Is that right - that the byteswapped versions might not be strictly
equal to identical numbers but not byteswapped?

But I should maybe have been clearer - they also subtract wrongly:

<script>
import numpy as np

arr = np.arange(10, dtype=np.complex256)
bs_arr = arr.byteswap().newbyteorder('S')
print arr
print bs_arr
print arr - bs_arr
print arr - bs_arr
print arr - bs_arr
</script>

(np-devel)[mb312 <at> joshlegacy ~/tmp]$ python funny_bs.py
[ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
  7.0+0.0j  8.0+0.0j  9.0+0.0j]
[ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
  7.0+0.0j  8.0+0.0j  9.0+0.0j]
[ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
  7.0+0.0j  8.0+0.0j  9.0+0.0j]
(Continue reading)

Travis Oliphant | 21 Jun 2012 00:04
Favicon
Gravatar

Re: Bizarre errors with byteswapping, complex256, PPC

That is clearly wrong and worth a bug report.

Travis 

--
Travis Oliphant
(on a mobile)
512-826-7480

On Jun 20, 2012, at 5:00 PM, Matthew Brett <matthew.brett <at> gmail.com> wrote:

> Hi,
> 
> On Wed, Jun 20, 2012 at 1:56 PM, Travis Oliphant <travis <at> continuum.io> wrote:
>> This looks like a problem with comparisons of floating point numbers rather than a byteswapping problem
per-say.   Try to use an almost equal comparison instead.
> 
> Is that right - that the byteswapped versions might not be strictly
> equal to identical numbers but not byteswapped?
> 
> But I should maybe have been clearer - they also subtract wrongly:
> 
> <script>
> import numpy as np
> 
> arr = np.arange(10, dtype=np.complex256)
> bs_arr = arr.byteswap().newbyteorder('S')
> print arr
> print bs_arr
> print arr - bs_arr
(Continue reading)

Charles R Harris | 21 Jun 2012 00:05
Picon

Re: Bizarre errors with byteswapping, complex256, PPC



On Wed, Jun 20, 2012 at 4:00 PM, Matthew Brett <matthew.brett <at> gmail.com> wrote:
Hi,

On Wed, Jun 20, 2012 at 1:56 PM, Travis Oliphant <travis <at> continuum.io> wrote:
> This looks like a problem with comparisons of floating point numbers rather than a byteswapping problem per-say.   Try to use an almost equal comparison instead.

Is that right - that the byteswapped versions might not be strictly
equal to identical numbers but not byteswapped?

But I should maybe have been clearer - they also subtract wrongly:

<script>
import numpy as np

arr = np.arange(10, dtype=np.complex256)
bs_arr = arr.byteswap().newbyteorder('S')
print arr
print bs_arr
print arr - bs_arr
print arr - bs_arr
print arr - bs_arr
</script>

(np-devel)[mb312 <at> joshlegacy ~/tmp]$ python funny_bs.py
[ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
 7.0+0.0j  8.0+0.0j  9.0+0.0j]
[ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
 7.0+0.0j  8.0+0.0j  9.0+0.0j]
[ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
 7.0+0.0j  8.0+0.0j  9.0+0.0j]
[ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
 7.0+0.0j  8.0+0.0j  9.0+0.0j]
[ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
 7.0+0.0j  8.0+0.0j  9.0+0.0j]

(wrong)

(np-devel)[mb312 <at> joshlegacy ~/tmp]$ python funny_bs.py
[ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
 7.0+0.0j  8.0+0.0j  9.0+0.0j]
[ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
 7.0+0.0j  8.0+0.0j  9.0+0.0j]
[ 0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j
 0.0+0.0j  0.0+0.0j  0.0+0.0j]
[ 0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j
 0.0+0.0j  0.0+0.0j  0.0+0.0j]
[ 0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j
 0.0+0.0j  0.0+0.0j  0.0+0.0j]

(right)

See you,


Long doubles on PPC consist of two doubles, so I expect you need to swap both doubles instead of 16 bytes. Strictly speaking, numpy doesn't support non ieee floats.

Chuck

_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion <at> scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Matthew Brett | 21 Jun 2012 00:11
Picon
Gravatar

Re: Bizarre errors with byteswapping, complex256, PPC

Hi,

On Wed, Jun 20, 2012 at 3:05 PM, Charles R Harris
<charlesr.harris <at> gmail.com> wrote:
>
>
> On Wed, Jun 20, 2012 at 4:00 PM, Matthew Brett <matthew.brett <at> gmail.com>
> wrote:
>>
>> Hi,
>>
>> On Wed, Jun 20, 2012 at 1:56 PM, Travis Oliphant <travis <at> continuum.io>
>> wrote:
>> > This looks like a problem with comparisons of floating point numbers
>> > rather than a byteswapping problem per-say.   Try to use an almost equal
>> > comparison instead.
>>
>> Is that right - that the byteswapped versions might not be strictly
>> equal to identical numbers but not byteswapped?
>>
>> But I should maybe have been clearer - they also subtract wrongly:
>>
>> <script>
>> import numpy as np
>>
>> arr = np.arange(10, dtype=np.complex256)
>> bs_arr = arr.byteswap().newbyteorder('S')
>> print arr
>> print bs_arr
>> print arr - bs_arr
>> print arr - bs_arr
>> print arr - bs_arr
>> </script>
>>
>> (np-devel)[mb312 <at> joshlegacy ~/tmp]$ python funny_bs.py
>> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
>>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
>>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
>>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
>>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
>>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>>
>> (wrong)
>>
>> (np-devel)[mb312 <at> joshlegacy ~/tmp]$ python funny_bs.py
>> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
>>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
>>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>> [ 0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j
>>  0.0+0.0j  0.0+0.0j  0.0+0.0j]
>> [ 0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j
>>  0.0+0.0j  0.0+0.0j  0.0+0.0j]
>> [ 0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j
>>  0.0+0.0j  0.0+0.0j  0.0+0.0j]
>>
>> (right)
>>
>> See you,
>>
>
> Long doubles on PPC consist of two doubles, so I expect you need to swap
> both doubles instead of 16 bytes. Strictly speaking, numpy doesn't support
> non ieee floats.

Well - the byteswapping appears to be correct in that the array is
displayed with the correct values, but then, when doing a subtraction
on the array, most of the time it is incorrect, but whether it is
correct or incorrect, appears to be random even with the same
variables and memory.

Float128 and other numpy dtypes appear to be correct using the same tests.

Best,

Matthew
Matthew Brett | 21 Jun 2012 00:46
Picon
Gravatar

Re: Bizarre errors with byteswapping, complex256, PPC

Hi,

On Wed, Jun 20, 2012 at 3:11 PM, Matthew Brett <matthew.brett <at> gmail.com> wrote:
> Hi,
>
> On Wed, Jun 20, 2012 at 3:05 PM, Charles R Harris
> <charlesr.harris <at> gmail.com> wrote:
>>
>>
>> On Wed, Jun 20, 2012 at 4:00 PM, Matthew Brett <matthew.brett <at> gmail.com>
>> wrote:
>>>
>>> Hi,
>>>
>>> On Wed, Jun 20, 2012 at 1:56 PM, Travis Oliphant <travis <at> continuum.io>
>>> wrote:
>>> > This looks like a problem with comparisons of floating point numbers
>>> > rather than a byteswapping problem per-say.   Try to use an almost equal
>>> > comparison instead.
>>>
>>> Is that right - that the byteswapped versions might not be strictly
>>> equal to identical numbers but not byteswapped?
>>>
>>> But I should maybe have been clearer - they also subtract wrongly:
>>>
>>> <script>
>>> import numpy as np
>>>
>>> arr = np.arange(10, dtype=np.complex256)
>>> bs_arr = arr.byteswap().newbyteorder('S')
>>> print arr
>>> print bs_arr
>>> print arr - bs_arr
>>> print arr - bs_arr
>>> print arr - bs_arr
>>> </script>
>>>
>>> (np-devel)[mb312 <at> joshlegacy ~/tmp]$ python funny_bs.py
>>> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
>>>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>>> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
>>>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>>> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
>>>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>>> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
>>>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>>> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
>>>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>>>
>>> (wrong)
>>>
>>> (np-devel)[mb312 <at> joshlegacy ~/tmp]$ python funny_bs.py
>>> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
>>>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>>> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
>>>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>>> [ 0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j
>>>  0.0+0.0j  0.0+0.0j  0.0+0.0j]
>>> [ 0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j
>>>  0.0+0.0j  0.0+0.0j  0.0+0.0j]
>>> [ 0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j
>>>  0.0+0.0j  0.0+0.0j  0.0+0.0j]
>>>
>>> (right)
>>>
>>> See you,
>>>
>>
>> Long doubles on PPC consist of two doubles, so I expect you need to swap
>> both doubles instead of 16 bytes. Strictly speaking, numpy doesn't support
>> non ieee floats.
>
> Well - the byteswapping appears to be correct in that the array is
> displayed with the correct values, but then, when doing a subtraction
> on the array, most of the time it is incorrect, but whether it is
> correct or incorrect, appears to be random even with the same
> variables and memory.
>
> Float128 and other numpy dtypes appear to be correct using the same tests.

http://projects.scipy.org/numpy/ticket/2174

Best,

Matthew
Charles R Harris | 21 Jun 2012 07:43
Picon

Re: Bizarre errors with byteswapping, complex256, PPC



On Wed, Jun 20, 2012 at 4:11 PM, Matthew Brett <matthew.brett <at> gmail.com> wrote:
Hi,

On Wed, Jun 20, 2012 at 3:05 PM, Charles R Harris
<charlesr.harris <at> gmail.com> wrote:
>
>
> On Wed, Jun 20, 2012 at 4:00 PM, Matthew Brett <matthew.brett <at> gmail.com>
> wrote:
>>
>> Hi,
>>
>> On Wed, Jun 20, 2012 at 1:56 PM, Travis Oliphant <travis <at> continuum.io>
>> wrote:
>> > This looks like a problem with comparisons of floating point numbers
>> > rather than a byteswapping problem per-say.   Try to use an almost equal
>> > comparison instead.
>>
>> Is that right - that the byteswapped versions might not be strictly
>> equal to identical numbers but not byteswapped?
>>
>> But I should maybe have been clearer - they also subtract wrongly:
>>
>> <script>
>> import numpy as np
>>
>> arr = np.arange(10, dtype=np.complex256)
>> bs_arr = arr.byteswap().newbyteorder('S')
>> print arr
>> print bs_arr
>> print arr - bs_arr
>> print arr - bs_arr
>> print arr - bs_arr
>> </script>
>>
>> (np-devel)[mb312 <at> joshlegacy ~/tmp]$ python funny_bs.py
>> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
>>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
>>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
>>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
>>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
>>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>>
>> (wrong)
>>
>> (np-devel)[mb312 <at> joshlegacy ~/tmp]$ python funny_bs.py
>> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
>>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
>>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>> [ 0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j
>>  0.0+0.0j  0.0+0.0j  0.0+0.0j]
>> [ 0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j
>>  0.0+0.0j  0.0+0.0j  0.0+0.0j]
>> [ 0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j
>>  0.0+0.0j  0.0+0.0j  0.0+0.0j]
>>
>> (right)
>>
>> See you,
>>
>
> Long doubles on PPC consist of two doubles, so I expect you need to swap
> both doubles instead of 16 bytes. Strictly speaking, numpy doesn't support
> non ieee floats.

Well - the byteswapping appears to be correct in that the array is
displayed with the correct values, but then, when doing a subtraction
on the array, most of the time it is incorrect, but whether it is
correct or incorrect, appears to be random even with the same
variables and memory.

Float128 and other numpy dtypes appear to be correct using the same tests.

Thinking about it, that makes sense because the swapped version is probably incorrect ;) That is, the PPC was (is?) selectable to run either little endian or big endian, so the real test would be if long doubles were portable between machines set up different ways. The only machines I know of are little endian, but IIRC, there was at least one brand that was big endian. However, I suspect we are just reversing the whole 16 bytes, so even though that is pretty much a meaningless thing to do, it should work...

Is this something that only happens on 32 bit machines?

Chuck

_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion <at> scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Matthew Brett | 21 Jun 2012 08:11
Picon
Gravatar

Re: Bizarre errors with byteswapping, complex256, PPC

Hi,

On Wed, Jun 20, 2012 at 10:43 PM, Charles R Harris
<charlesr.harris <at> gmail.com> wrote:
>
>
> On Wed, Jun 20, 2012 at 4:11 PM, Matthew Brett <matthew.brett <at> gmail.com>
> wrote:
>>
>> Hi,
>>
>> On Wed, Jun 20, 2012 at 3:05 PM, Charles R Harris
>> <charlesr.harris <at> gmail.com> wrote:
>> >
>> >
>> > On Wed, Jun 20, 2012 at 4:00 PM, Matthew Brett <matthew.brett <at> gmail.com>
>> > wrote:
>> >>
>> >> Hi,
>> >>
>> >> On Wed, Jun 20, 2012 at 1:56 PM, Travis Oliphant <travis <at> continuum.io>
>> >> wrote:
>> >> > This looks like a problem with comparisons of floating point numbers
>> >> > rather than a byteswapping problem per-say.   Try to use an almost
>> >> > equal
>> >> > comparison instead.
>> >>
>> >> Is that right - that the byteswapped versions might not be strictly
>> >> equal to identical numbers but not byteswapped?
>> >>
>> >> But I should maybe have been clearer - they also subtract wrongly:
>> >>
>> >> <script>
>> >> import numpy as np
>> >>
>> >> arr = np.arange(10, dtype=np.complex256)
>> >> bs_arr = arr.byteswap().newbyteorder('S')
>> >> print arr
>> >> print bs_arr
>> >> print arr - bs_arr
>> >> print arr - bs_arr
>> >> print arr - bs_arr
>> >> </script>
>> >>
>> >> (np-devel)[mb312 <at> joshlegacy ~/tmp]$ python funny_bs.py
>> >> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
>> >>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>> >> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
>> >>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>> >> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
>> >>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>> >> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
>> >>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>> >> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
>> >>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>> >>
>> >> (wrong)
>> >>
>> >> (np-devel)[mb312 <at> joshlegacy ~/tmp]$ python funny_bs.py
>> >> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
>> >>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>> >> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
>> >>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>> >> [ 0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j
>> >>  0.0+0.0j  0.0+0.0j  0.0+0.0j]
>> >> [ 0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j
>> >>  0.0+0.0j  0.0+0.0j  0.0+0.0j]
>> >> [ 0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j
>> >>  0.0+0.0j  0.0+0.0j  0.0+0.0j]
>> >>
>> >> (right)
>> >>
>> >> See you,
>> >>
>> >
>> > Long doubles on PPC consist of two doubles, so I expect you need to swap
>> > both doubles instead of 16 bytes. Strictly speaking, numpy doesn't
>> > support
>> > non ieee floats.
>>
>> Well - the byteswapping appears to be correct in that the array is
>> displayed with the correct values, but then, when doing a subtraction
>> on the array, most of the time it is incorrect, but whether it is
>> correct or incorrect, appears to be random even with the same
>> variables and memory.
>>
>> Float128 and other numpy dtypes appear to be correct using the same tests.
>
>
> Thinking about it, that makes sense because the swapped version is probably
> incorrect ;) That is, the PPC was (is?) selectable to run either little
> endian or big endian, so the real test would be if long doubles were
> portable between machines set up different ways. The only machines I know of
> are little endian, but IIRC, there was at least one brand that was big
> endian. However, I suspect we are just reversing the whole 16 bytes, so even
> though that is pretty much a meaningless thing to do, it should work...
>
> Is this something that only happens on 32 bit machines?

The PPC machines I have record themselves as big endian (2 running OSX
and 1 running Debian wheezy).

I can only get the Debian wheezy machine to misbehave in this way.

The original report of the problem was on a POWER7 machine running
Debian - wheezy I think.  I guess this is 64 bit - Yarik - do you
know?

See you,

Matthew
Charles R Harris | 21 Jun 2012 08:57
Picon

Re: Bizarre errors with byteswapping, complex256, PPC



On Thu, Jun 21, 2012 at 12:11 AM, Matthew Brett <matthew.brett <at> gmail.com> wrote:
Hi,

On Wed, Jun 20, 2012 at 10:43 PM, Charles R Harris
<charlesr.harris <at> gmail.com> wrote:
>
>
> On Wed, Jun 20, 2012 at 4:11 PM, Matthew Brett <matthew.brett <at> gmail.com>
> wrote:
>>
>> Hi,
>>
>> On Wed, Jun 20, 2012 at 3:05 PM, Charles R Harris
>> <charlesr.harris <at> gmail.com> wrote:
>> >
>> >
>> > On Wed, Jun 20, 2012 at 4:00 PM, Matthew Brett <matthew.brett <at> gmail.com>
>> > wrote:
>> >>
>> >> Hi,
>> >>
>> >> On Wed, Jun 20, 2012 at 1:56 PM, Travis Oliphant <travis <at> continuum.io>
>> >> wrote:
>> >> > This looks like a problem with comparisons of floating point numbers
>> >> > rather than a byteswapping problem per-say.   Try to use an almost
>> >> > equal
>> >> > comparison instead.
>> >>
>> >> Is that right - that the byteswapped versions might not be strictly
>> >> equal to identical numbers but not byteswapped?
>> >>
>> >> But I should maybe have been clearer - they also subtract wrongly:
>> >>
>> >> <script>
>> >> import numpy as np
>> >>
>> >> arr = np.arange(10, dtype=np.complex256)
>> >> bs_arr = arr.byteswap().newbyteorder('S')
>> >> print arr
>> >> print bs_arr
>> >> print arr - bs_arr
>> >> print arr - bs_arr
>> >> print arr - bs_arr
>> >> </script>
>> >>
>> >> (np-devel)[mb312 <at> joshlegacy ~/tmp]$ python funny_bs.py
>> >> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
>> >>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>> >> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
>> >>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>> >> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
>> >>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>> >> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
>> >>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>> >> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
>> >>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>> >>
>> >> (wrong)
>> >>
>> >> (np-devel)[mb312 <at> joshlegacy ~/tmp]$ python funny_bs.py
>> >> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
>> >>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>> >> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
>> >>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>> >> [ 0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j
>> >>  0.0+0.0j  0.0+0.0j  0.0+0.0j]
>> >> [ 0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j
>> >>  0.0+0.0j  0.0+0.0j  0.0+0.0j]
>> >> [ 0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j
>> >>  0.0+0.0j  0.0+0.0j  0.0+0.0j]
>> >>
>> >> (right)
>> >>
>> >> See you,
>> >>
>> >
>> > Long doubles on PPC consist of two doubles, so I expect you need to swap
>> > both doubles instead of 16 bytes. Strictly speaking, numpy doesn't
>> > support
>> > non ieee floats.
>>
>> Well - the byteswapping appears to be correct in that the array is
>> displayed with the correct values, but then, when doing a subtraction
>> on the array, most of the time it is incorrect, but whether it is
>> correct or incorrect, appears to be random even with the same
>> variables and memory.
>>
>> Float128 and other numpy dtypes appear to be correct using the same tests.
>
>
> Thinking about it, that makes sense because the swapped version is probably
> incorrect ;) That is, the PPC was (is?) selectable to run either little
> endian or big endian, so the real test would be if long doubles were
> portable between machines set up different ways. The only machines I know of
> are little endian, but IIRC, there was at least one brand that was big
> endian. However, I suspect we are just reversing the whole 16 bytes, so even
> though that is pretty much a meaningless thing to do, it should work...
>
> Is this something that only happens on 32 bit machines?

The PPC machines I have record themselves as big endian (2 running OSX
and 1 running Debian wheezy).

I can only get the Debian wheezy machine to misbehave in this way.

The original report of the problem was on a POWER7 machine running
Debian - wheezy I think.  I guess this is 64 bit - Yarik - do you
know?


Looks like I got little/big endian reversed. Anyway, this is very strange. Same compilers on both machines? I don't understand why this wouldn't show up on other machines with float128, and in particular why is should only happen for complex256 and Debian. What is the extended precision type in OSX on PPC?

Chuck

_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion <at> scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Matthew Brett | 21 Jun 2012 11:01
Picon
Gravatar

Re: Bizarre errors with byteswapping, complex256, PPC

Hi,

On Wed, Jun 20, 2012 at 11:57 PM, Charles R Harris
<charlesr.harris <at> gmail.com> wrote:
>
>
> On Thu, Jun 21, 2012 at 12:11 AM, Matthew Brett <matthew.brett <at> gmail.com>
> wrote:
>>
>> Hi,
>>
>> On Wed, Jun 20, 2012 at 10:43 PM, Charles R Harris
>> <charlesr.harris <at> gmail.com> wrote:
>> >
>> >
>> > On Wed, Jun 20, 2012 at 4:11 PM, Matthew Brett <matthew.brett <at> gmail.com>
>> > wrote:
>> >>
>> >> Hi,
>> >>
>> >> On Wed, Jun 20, 2012 at 3:05 PM, Charles R Harris
>> >> <charlesr.harris <at> gmail.com> wrote:
>> >> >
>> >> >
>> >> > On Wed, Jun 20, 2012 at 4:00 PM, Matthew Brett
>> >> > <matthew.brett <at> gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> Hi,
>> >> >>
>> >> >> On Wed, Jun 20, 2012 at 1:56 PM, Travis Oliphant
>> >> >> <travis <at> continuum.io>
>> >> >> wrote:
>> >> >> > This looks like a problem with comparisons of floating point
>> >> >> > numbers
>> >> >> > rather than a byteswapping problem per-say.   Try to use an almost
>> >> >> > equal
>> >> >> > comparison instead.
>> >> >>
>> >> >> Is that right - that the byteswapped versions might not be strictly
>> >> >> equal to identical numbers but not byteswapped?
>> >> >>
>> >> >> But I should maybe have been clearer - they also subtract wrongly:
>> >> >>
>> >> >> <script>
>> >> >> import numpy as np
>> >> >>
>> >> >> arr = np.arange(10, dtype=np.complex256)
>> >> >> bs_arr = arr.byteswap().newbyteorder('S')
>> >> >> print arr
>> >> >> print bs_arr
>> >> >> print arr - bs_arr
>> >> >> print arr - bs_arr
>> >> >> print arr - bs_arr
>> >> >> </script>
>> >> >>
>> >> >> (np-devel)[mb312 <at> joshlegacy ~/tmp]$ python funny_bs.py
>> >> >> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j
>> >> >>  6.0+0.0j
>> >> >>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>> >> >> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j
>> >> >>  6.0+0.0j
>> >> >>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>> >> >> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j
>> >> >>  6.0+0.0j
>> >> >>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>> >> >> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j
>> >> >>  6.0+0.0j
>> >> >>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>> >> >> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j
>> >> >>  6.0+0.0j
>> >> >>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>> >> >>
>> >> >> (wrong)
>> >> >>
>> >> >> (np-devel)[mb312 <at> joshlegacy ~/tmp]$ python funny_bs.py
>> >> >> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j
>> >> >>  6.0+0.0j
>> >> >>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>> >> >> [ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j
>> >> >>  6.0+0.0j
>> >> >>  7.0+0.0j  8.0+0.0j  9.0+0.0j]
>> >> >> [ 0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j
>> >> >>  0.0+0.0j
>> >> >>  0.0+0.0j  0.0+0.0j  0.0+0.0j]
>> >> >> [ 0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j
>> >> >>  0.0+0.0j
>> >> >>  0.0+0.0j  0.0+0.0j  0.0+0.0j]
>> >> >> [ 0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j  0.0+0.0j
>> >> >>  0.0+0.0j
>> >> >>  0.0+0.0j  0.0+0.0j  0.0+0.0j]
>> >> >>
>> >> >> (right)
>> >> >>
>> >> >> See you,
>> >> >>
>> >> >
>> >> > Long doubles on PPC consist of two doubles, so I expect you need to
>> >> > swap
>> >> > both doubles instead of 16 bytes. Strictly speaking, numpy doesn't
>> >> > support
>> >> > non ieee floats.
>> >>
>> >> Well - the byteswapping appears to be correct in that the array is
>> >> displayed with the correct values, but then, when doing a subtraction
>> >> on the array, most of the time it is incorrect, but whether it is
>> >> correct or incorrect, appears to be random even with the same
>> >> variables and memory.
>> >>
>> >> Float128 and other numpy dtypes appear to be correct using the same
>> >> tests.
>> >
>> >
>> > Thinking about it, that makes sense because the swapped version is
>> > probably
>> > incorrect ;) That is, the PPC was (is?) selectable to run either little
>> > endian or big endian, so the real test would be if long doubles were
>> > portable between machines set up different ways. The only machines I
>> > know of
>> > are little endian, but IIRC, there was at least one brand that was big
>> > endian. However, I suspect we are just reversing the whole 16 bytes, so
>> > even
>> > though that is pretty much a meaningless thing to do, it should work...
>> >
>> > Is this something that only happens on 32 bit machines?
>>
>> The PPC machines I have record themselves as big endian (2 running OSX
>> and 1 running Debian wheezy).
>>
>> I can only get the Debian wheezy machine to misbehave in this way.
>>
>> The original report of the problem was on a POWER7 machine running
>> Debian - wheezy I think.  I guess this is 64 bit - Yarik - do you
>> know?
>>
>
> Looks like I got little/big endian reversed. Anyway, this is very strange.
> Same compilers on both machines?

gcc-4.6.3-1 on the machine I have access to.

The POWER7 machine appears to be running an older kernel, so my guess
is it has an older gcc too, but I don't have access I'm afraid.

>  I don't understand why this wouldn't show
> up on other machines with float128, and in particular why is should only
> happen for complex256 and Debian. What is the extended precision type in OSX
> on PPC?

OSX uses double pairs too - and I've been testing on them for a while
- so I was also surprised to see this when we got to Debian...

See you,

Matthew
Charles R Harris | 21 Jun 2012 00:08
Picon

Re: Bizarre errors with byteswapping, complex256, PPC



On Wed, Jun 20, 2012 at 2:48 PM, Matthew Brett <matthew.brett <at> gmail.com> wrote:
Hi,

Our Debian friends were hammering our code tests before the upcoming
freeze, and found the following very odd thing on 32-bit PPC running
Debian squeeze and numpy 1.6.2 or current trunk.

Consider the following script:

<script>
import numpy as np

arr = np.arange(10, dtype=np.complex256)
bs_arr = arr.byteswap().newbyteorder('S')
print arr
print bs_arr
print arr == bs_arr
print arr == bs_arr
print arr == bs_arr
</script>

Here is the output from some example runs of this script:

np-devel)[mb312 <at> joshlegacy ~/tmp]$ python funny_bs.py
[ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
 7.0+0.0j  8.0+0.0j  9.0+0.0j]
[ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
 7.0+0.0j  8.0+0.0j  9.0+0.0j]
[ True False False False False False False False False False]
[ True False False False False False False False False False]
[ True False False False False False False False False False]

(that's the most common result)

(np-devel)[mb312 <at> joshlegacy ~/tmp]$ python funny_bs.py
[ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
 7.0+0.0j  8.0+0.0j  9.0+0.0j]
[ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
 7.0+0.0j  8.0+0.0j  9.0+0.0j]
[ True  True  True  True  True  True  True  True  True  True]
[ True  True  True  True  True  True  True  True  True  True]
[ True  True  True  True  True  True  True  True  True  True]

(this happens maybe 10% of the time)

(np-devel)[mb312 <at> joshlegacy ~/tmp]$ python funny_bs.py
[ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
 7.0+0.0j  8.0+0.0j  9.0+0.0j]
[ 0.0+0.0j  1.0+0.0j  2.0+0.0j  3.0+0.0j  4.0+0.0j  5.0+0.0j  6.0+0.0j
 7.0+0.0j  8.0+0.0j  9.0+0.0j]
[ True  True  True  True  True  True  True  True  True  True]
[ True False False False False False False False False False]
[ True  True  True  True  True  True  True  True  True  True]

(less than 10% of the time - order of True, False prints _of the same
comparison_ is random.  This only seems to happen with complex256.

Is there anything I can do to debug this further?  Does anyone want a
login to this machine to have a look?

Try swapped float128 and see what that looks like.

Chuck

_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion <at> scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Gmane