headers
Neil Mayhew | 22 May 16:49

[integer] Promotion of endian library code from vault

On 2006-07-22 12:21:01, Beman Dawes wrote:
> [boost] [integer] endian-0.6 posted w/ names changed
> endian-0.6.zip has been posted in the integer directory of the file 
> vault at http://boost-consulting.com/vault/

This code is still in the vault, but I'd like to see it in the main part 
of boost. What needs to be done to make this happen?

The endian library is exactly what I've been looking for, and I would 
really like to use it in my work, but I can't justify that to my 
colleagues if it's not an official part of boost. I would hate to become 
yet another person that reinvents this particular wheel. However, I 
notice there was a lot of discussion 
<http://lists.boost.org/Archives/boost/2006/07/108125.php> about PODs vs 
constructors, because of unions, so maybe not enough consensus was 
reached? There were some interesting proposals but no agreed-upon 
solution. In which case, I suspect that people will need to copy the 
code and each hack it to suit their own purposes, which is not what 
boost is all about.

I have two use cases from my own work: TrueType fonts, and Mac HFS+ file 
systems. Both of these are big-endian file formats that need to be read 
into memory and processed, so (FWIW) I would support the POD approach. 
Currently, I can't use the code as-is because of the need to use the 
types in unions.

References:

http://lists.boost.org/Archives/boost/2000/03/2495.php
http://lists.boost.org/Archives/boost/2006/05/105621.php
(Continue reading)

Permalink | Reply |
headers
Roland Schwarz | 23 May 13:05
Favicon

Re: [integer] Promotion of endian library code from vault


Neil Mayhew wrote:
> This code is still in the vault, but I'd like to see it in the main part 
> of boost. What needs to be done to make this happen?
> 
> The endian library is exactly what I've been looking for, and I would 
> really like to use it in my work, but I can't justify that to my 
> colleagues if it's not an official part of boost.

This is an interesting library indeed! Thank you for having this
brought up on the list again.

However I have some concerns about usefulness when it comes to
compiler independence and platform independence.

One of the goals of such a library to be useful (for me at least) would
be to be able to create compiler/platform independent binary files.

I can see two problems here:

1) struct layout.
   The standard gives no provisions for struct layout. So e.g. for

   struct foo {
      big8_t a;
      big32_t b;
   };

   one cannot predict the alignment of the members. (Or am I wrong
   in this respect?)
(Continue reading)

Permalink | Reply |
headers
dizzy | 23 May 13:46

Re: [integer] Promotion of endian library code from vault

On Friday 23 May 2008 14:05:56 Roland Schwarz wrote:
> Neil Mayhew wrote:
> > This code is still in the vault, but I'd like to see it in the main part
> > of boost. What needs to be done to make this happen?
> >
> > The endian library is exactly what I've been looking for, and I would
> > really like to use it in my work, but I can't justify that to my
> > colleagues if it's not an official part of boost.
>
> This is an interesting library indeed! Thank you for having this
> brought up on the list again.
>
> However I have some concerns about usefulness when it comes to
> compiler independence and platform independence.
>
> One of the goals of such a library to be useful (for me at least) would
> be to be able to create compiler/platform independent binary files.
>
> I can see two problems here:
>
> 1) struct layout.
>    The standard gives no provisions for struct layout. So e.g. for
>
>    struct foo {
>       big8_t a;
>       big32_t b;
>    };
>
>    one cannot predict the alignment of the members. (Or am I wrong
>    in this respect?)
(Continue reading)

Permalink | Reply |
headers
Roland Schwarz | 23 May 14:19
Favicon

Re: [integer] Promotion of endian library code from vault


dizzy wrote:
> On Friday 23 May 2008 14:05:56 Roland Schwarz wrote:
>>    one cannot predict the alignment of the members. (Or am I wrong
>>    in this respect?)
> 
> Correct, which is why protocol binary structures are never mapped directly in 
> memory (you can with some compiler extensions but you won't gain anything 
> since I/O is the bottleneck in such cases and not memory copy). Instead a 
> serialization aproach should solve such issues.

It seems I was wrong with my assumptions about endian.hpp:

The proposed lib actually does have an unaligned type, i.e. it
actually maps the types to char bytes[..] arrays. So the question
remains what the standard has to say about alignment of

struct foo {
   char bytes_1 [3];
   char bytes_2 [2];
   char bytes_3 [1];
   char bytes_4 [4];
};

Will such a struct be equivalent to
char bytes[3+2+1+4] ?

Hmm, and foo isn't necessarily a POD. Writing this out with
binary write... what is guaranteed? I fear not much.
(Continue reading)

Permalink | Reply |
headers
Neil Mayhew | 23 May 15:28

Re: [integer] Promotion of endian library code from vault

On 5/23/08 6:19 AM, Roland Schwarz wrote:

> The proposed lib actually does have an unaligned type, i.e. it
> actually maps the types to char bytes[..] arrays. So the question
> remains what the standard has to say about alignment of
>
> struct foo {
>    char bytes_1 [3];
>    char bytes_2 [2];
>    char bytes_3 [1];
>    char bytes_4 [4];
> };
>
> Will such a struct be equivalent to
> char bytes[3+2+1+4] ?
>   

I am not a language lawyer, but my understanding is that struct members 
will be aligned according to their needs, and a type that contains only 
chars will need no special alignment. The endian library therefore takes 
care of alignment down to the byte level.

> 1) I wanted a way to control layout on a per struct basis. I wanted
>    to be able to go as low as bit position.
>   

I think it is very unlikely that a portable library would be able to get 
control over bit positions. At least, not in a purely data-declarative way.

However, you could use a boost endian type as the underlying storage and
(Continue reading)

Permalink | Reply |
headers
dizzy | 23 May 15:36

Re: [integer] Promotion of endian library code from vault

On Friday 23 May 2008 15:19:32 Roland Schwarz wrote:
> > That kinda looks like reinventing boost.serialization although with a
> > different API (I did something similar in my code).
>
> Indeed there are similarities. But there are differences as well.
>
> 1) I wanted a way to control layout on a per struct basis. I wanted
>    to be able to go as low as bit position.
>
> 2) boost.serialization is fine when I want to make my in memory
>    classes persistent, but it is of little help for the decoding
>    of binary protocol packages. ( I do not claim it is not possible).
>
> 3) boost.serialization solves two orthogonal problems with a single
>    implementation. One problem is mapping native types to portable
>    types (partial overlap with standard lib << operators), the other
>    is writing out a tree of objects (serialization). I was aiming
>    only at the first problem, and so does endian.hpp I guess.
>    Perhaps serialization could be refactored to allow more
>    control over the layout within the serialize functions?

Correct, I agree, that's why I guess I have written my own portable 
serialization library (although I never thought too much about reasons but 
those listed by you do apply in my case).

I have something similar as you said, basically something like this (I'm 
showing only the integral case because is the basis of the framework, I also 
have utility serialization code for strings and such but those are based on 
the integral too):
(Continue reading)

Permalink | Reply |
headers
Giovanni Piero Deretta | 23 May 15:45

Re: [integer] Promotion of endian library code from vault

On Fri, May 23, 2008 at 1:46 PM, dizzy <dizzy <at> roedu.net> wrote:
> On Friday 23 May 2008 14:05:56 Roland Schwarz wrote:
>> Neil Mayhew wrote:
>> > This code is still in the vault, but I'd like to see it in the main part
>> > of boost. What needs to be done to make this happen?
>> >
>> > The endian library is exactly what I've been looking for, and I would
>> > really like to use it in my work, but I can't justify that to my
>> > colleagues if it's not an official part of boost.
>>
>> This is an interesting library indeed! Thank you for having this
>> brought up on the list again.
>>
>> However I have some concerns about usefulness when it comes to
>> compiler independence and platform independence.
>>
>> One of the goals of such a library to be useful (for me at least) would
>> be to be able to create compiler/platform independent binary files.
>>
>> I can see two problems here:
>>
>> 1) struct layout.
>>    The standard gives no provisions for struct layout. So e.g. for
>>
>>    struct foo {
>>       big8_t a;
>>       big32_t b;
>>    };
>>
>>    one cannot predict the alignment of the members. (Or am I wrong
(Continue reading)

Permalink | Reply |
headers
Neil Mayhew | 23 May 17:27

Re: [integer] Promotion of endian library code from vault

On 23/05/08 07:45 AM Giovanni Piero Deretta wrote:
>  On Fri, May 23, 2008 at 1:46 PM, dizzy <dizzy <at> roedu.net> wrote:
> > ... protocol binary structures are never mapped directly in memory
> > (you can with some compiler extensions but you won't gain anything
> > since I/O is the bottleneck in such cases and not memory copy).
> > Instead a serialization aproach should solve such issues.
>
>  With disk I/O this is certainly true, on the other hand, high speed
>  LAN networks might actually be faster than (uncached) memory accesses
>  (think for example 10G ethernet). Zero copy I/O is certainly an
>  useful property.

Also, protocols aren't the only use case. For example, dealing with
large binary files may involve memory-mapped disk i/o, in which case
mapping directly structures accurately in memory is essential. Endian
works very well for this, and I don't see a need for special serialization.

--Neil

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Permalink | Reply |
headers
Phil Endecott | 23 May 18:32

Re: [integer] Promotion of endian library code from vault

dizzy wrote:
> On Friday 23 May 2008 14:05:56 Roland Schwarz wrote:
>>    The standard gives no provisions for struct layout. So e.g. for
>>
>>    struct foo {
>>       big8_t a;
>>       big32_t b;
>>    };
>>
>>    one cannot predict the alignment of the members. (Or am I wrong
>>    in this respect?)
>
> Correct, which is why protocol binary structures are never mapped directly in 
> memory (you can with some compiler extensions but you won't gain anything 
> since I/O is the bottleneck in such cases and not memory copy). Instead a 
> serialization aproach should solve such issues.

Never say never :-)

I find that mmap()ing binary files into memory has the following 
significant advantage compared to read()ing them in: if the file is 
large and memory is tight, then read-in data must be swapped out i.e. 
written to disk.  In contrast, read-only mmap()ed pages can simply be 
discarded from RAM.  Even if the data is never actually swapped out, 
unless the OS over-commits swap space, disk will be reserved for this 
data.  So mmap() has a performance benefit on all memory-constrained 
systems and also a disk (i.e. flash) space benefit on embedded systems.

So while I'm generally quite pedantic about standards-compliance 
issues, struct layout is an area where I'm prepared to assume that the
(Continue reading)

Permalink | Reply |
headers
Beman Dawes | 24 May 13:34
Favicon

Re: [integer] Promotion of endian library code from vault

Roland Schwarz wrote:
> 
 >...
> The bottom line: For the endian lib to go into boost I really would
> want to require it being able to produce platform independent
> binary data.

Yes, of course. That's the most important requirement.

While the interface in endian.hpp is modern C++, the C implementation of 
underlying struct has been in use since about 1984 on a very wide 
variety of platforms. Data files have been exchanged without problems 
between those platforms since then. Others have reported successful use 
of the same technique. The standards committees understand the 
importance of preserving layout compatibility for such structs.

--Beman

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Permalink | Reply |
headers
Beman Dawes | 23 May 15:11
Favicon

Re: [integer] Promotion of endian library code from vault

Neil Mayhew wrote:
> On 2006-07-22 12:21:01, Beman Dawes wrote:
>> [boost] [integer] endian-0.6 posted w/ names changed
>> endian-0.6.zip has been posted in the integer directory of the file 
>> vault at http://boost-consulting.com/vault/
> 
> This code is still in the vault, but I'd like to see it in the main part 
> of boost. What needs to be done to make this happen?

Here's what has happened:

In working on endian, I became convince that the C++ standard's POD 
specification was a serious impediment and needed major revision. The 
standards committee agreed, and so C++0x will include a major relaxation 
on the requirements for POD's. Among other changes, base classes and 
constructors are permitted (under certain conditions).

See http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2007/n2342.htm

That work led Lawrence Crowl to come up with the idea of deleted and 
defaulted function for C++0x, which works synergistically with the POD's 
changes.

See http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2007/n2346.htm

A third C++0x feature, constexpr, may also be useful for endian.

See http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2007/n2235.pdf

So the next step is to revisit the current endian design, and apply
(Continue reading)

Permalink | Reply |
headers
Neil Mayhew | 23 May 15:37

Re: [integer] Promotion of endian library code from vault

On 5/23/08 7:11 AM, Beman Dawes wrote:
> In working on endian, I became convince that the C++ standard's POD 
> specification was a serious impediment and needed major revision. The 
> standards committee agreed, and so C++0x will include a major 
> relaxation on the requirements for POD's... So the next step is to 
> revisit the current endian design, and apply C++0x features where useful.

Thanks for this clarification. The standard changes sound great, and it 
makes sense to take advantage of these in endian, where possible.

> Some of these features are starting to become available in compilers, 
> so they can be tested.

However, I think it will be a while before I myself am able to use a 
compiler with these features, whereas I'd like to use endian asap. How 
about we get the current implementation into boost more or less as-is, 
and then work on improving it with new compiler features as these become 
available?

> I'm buried with other commitments, so if someone else wants to help 
> with endian it might speed things up quite a bit.

What kind of help did you have in mind? Is there anything that needs to 
be done that doesn't involve using new compiler features?

The areas that I am interested in seeing more work done on are (1) 
making constructors "conditional" and (2) adding a bit-range access method.

--Neil
_______________________________________________
(Continue reading)

Permalink | Reply |
headers
Beman Dawes | 24 May 13:45
Favicon

Re: [integer] Promotion of endian library code from vault

Neil Mayhew wrote:
> On 5/23/08 7:11 AM, Beman Dawes wrote:
>> In working on endian, I became convince that the C++ standard's POD 
>> specification was a serious impediment and needed major revision. The 
>> standards committee agreed, and so C++0x will include a major 
>> relaxation on the requirements for POD's... So the next step is to 
>> revisit the current endian design, and apply C++0x features where useful.
> 
> Thanks for this clarification. The standard changes sound great, and it 
> makes sense to take advantage of these in endian, where possible.
> 
>> Some of these features are starting to become available in compilers, 
>> so they can be tested.
> 
> However, I think it will be a while before I myself am able to use a 
> compiler with these features, whereas I'd like to use endian asap. How 
> about we get the current implementation into boost more or less as-is, 
> and then work on improving it with new compiler features as these become 
> available?
> 
>> I'm buried with other commitments, so if someone else wants to help 
>> with endian it might speed things up quite a bit.
> 
> What kind of help did you have in mind?

I just took a look at the current state of the library, and it seems to 
be in pretty good shape. So the main help would be careful review and 
commenting on a revised version. I'll try to get that out in a week or so.

> Is there anything that needs to
(Continue reading)

Permalink | Reply |
headers
Scott McMurray | 24 May 17:23

Re: [integer] Promotion of endian library code from vault

On Sat, May 24, 2008 at 7:45 AM, Beman Dawes <bdawes <at> acm.org> wrote:
> Neil Mayhew wrote:
>> The areas that I am interested in seeing more work done on are (1)
>> making constructors "conditional" and (2) adding a bit-range access method.
>
> Do you have a proposed design for bit-range access?
>
> What is the use case or motivation for bit-range access?
>

Perhaps something based off of erlang's Bit Syntax?
http://www.erlang.org/doc/reference_manual/expressions.html#6.16

Armstrong's Erlang book gives MPEG headers as an example of bit range
access, among other things.

So something like

typedef bit_sequence<
    uint_t<11>,
    uint_t<2>,
    uint_t<2>,
    uint_t<1>,
    uint_t<4>,
    uint_t<2>,
    uint_t<1>,
    uint_t<9>
> MPEG_header;

Or if you're writing a linker to generate EXEs for windows (another
(Continue reading)

Permalink | Reply |
headers
vicente.botet | 24 May 18:05
Gravatar

Re: [integer] Promotion of endian library code from vault

----- Original Message ----- 
From: "Beman Dawes" <bdawes <at> acm.org>
To: <boost <at> lists.boost.org>
Sent: Saturday, May 24, 2008 1:45 PM
Subject: Re: [boost] [integer] Promotion of endian library code from vault

> Neil Mayhew wrote:
>> The areas that I am interested in seeing more work done on are (1)
>> making constructors "conditional" and (2) adding a bit-range access 
>> method.
>
> Do you have a proposed design for bit-range access?

What about the bitfield library? The one on the Vault could be a good base.

Vicente 

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Permalink | Reply |
headers
Neil Mayhew | 26 May 22:31

Re: [integer] Promotion of endian library code from vault

On 24/05/08 05:45 AM Beman Dawes wrote:
> Neil Mayhew wrote:
>> The areas that I am interested in seeing more work done on are (1)
>> making constructors "conditional" and (2) adding a bit-range access 
>> method.
>>
> * Should there be a way (#ifdef? separate class?) to provide an
> interoperable version that meets C++03 POD requirements, so endian
> objects can be put in unions? Is that what you mean by (1)?

Yes, this is what I meant. I was thinking that the most elegant solution 
would be to have a base class that contains everything except the 
constructors, and a derived class that adds the constructors (I think 
this was in a post in 2006). The typedefs to big32_t etc. would then be 
#if'd to correspond to the POD or non-POD classes as desired. I think 
this is better than putting the #if's around the constructors themselves.

> Do you have a proposed design for bit-range access?

I was thinking of adding methods to read and write the given bits within 
the value:

value_type endian::bits(std::size_t offset, std::size_t length) const;
value_type endian::bits(std::size_t offset, std::size_t length, 
value_type value);

Making these template methods with offset and length being template 
arguments might allow for more compile-time unrolling, but since these 
methods just do shifting and masking, the compiler will probably do all 
the needed optimization anyway.
(Continue reading)

Permalink | Reply |
headers
Beman Dawes | 29 May 00:25
Favicon

Re: [integer] Promotion of endian library code from vault

Neil Mayhew wrote:
> On 24/05/08 05:45 AM Beman Dawes wrote:
>> Neil Mayhew wrote:
>>> The areas that I am interested in seeing more work done on are (1)
>>> making constructors "conditional" and (2) adding a bit-range access 
>>> method.
>>>
>> * Should there be a way (#ifdef? separate class?) to provide an
>> interoperable version that meets C++03 POD requirements, so endian
>> objects can be put in unions? Is that what you mean by (1)?
> 
> Yes, this is what I meant. I was thinking that the most elegant solution 
> would be to have a base class that contains everything except the 
> constructors, and a derived class that adds the constructors (I think 
> this was in a post in 2006).

There are some issues with the base class approach. For it to be a POD, 
it can't have constructors, base classes, or private members. No base 
classes means no arithmetic operations unless they are 100% supplied by 
the class itself. No private members means the non-POD version would 
have to use private inheritance and then forward all operations to the 
private operations.

Messy, but I can't think of any other approach. It might be better to 
just have two separate types, endian and old_endian. (old_endian isn't 
very clear as a name, but calling it pod_endian becomes invalid the 
moment C++0x ships.)

 > The typedefs to big32_t etc. would then be
> #if'd to correspond to the POD or non-POD classes as desired. I think
(Continue reading)

Permalink | Reply |
headers
Neil Mayhew | 29 May 01:50

Re: [integer] Promotion of endian library code from vault

On 28/05/08 04:25 PM Beman Dawes wrote:
> Neil Mayhew wrote:
>> ... have a base class that contains everything except the 
>> constructors, and a derived class that adds the constructors
>
> There are some issues with the base class approach. For it to be a 
> POD, it can't have constructors, base classes, or private members.

Strictly speaking, yes, although we are only concerned about memory 
layout here, so the class doesn't have to be a true POD. The other 
concern is for the class to be able to live inside a union, and I think 
the constructors are the the only thing stopping that. (I #if'd the 
constructors and I was then able to use endian in a union.)

> It might be better to just have two separate types, endian and 
> old_endian. (old_endian isn't very clear as a name, but calling it 
> pod_endian becomes invalid the moment C++0x ships.)
>
>> The typedefs to big32_t etc. would then be #if'd to correspond to the 
>> POD or non-POD classes as desired. I think this is better than 
>> putting the #if's around the constructors themselves.
>
> Hummm... Seems too obscure. Better to have two sets of typedefs, with 
> the C++03 set prefixed by old_ (or whatever better name anyone can 
> come up with.)

I don't like "old_", at least not if it has to appear in my code! :-)

I would like my client code to remain essentially the same when I 
upgrade to a C++0x compiler. This means having the same type names,
(Continue reading)

Permalink | Reply |
headers
Neil Mayhew | 29 May 21:54

Re: [integer] Promotion of endian library code from vault

On 28/05/08 05:50 PM Neil Mayhew wrote:
> Perhaps the simplest and best solution is therefore:
>
>     class endian< ... >
>       : cover_operators< ... >
>     {
>       public:
> #if defined(CXX_0X) || !defined(ENDIANS_IN_UNIONS)
>         endian() {}
>         endian(T val) { ... }
> #endif

I just discovered that an operator=(T) is needed as well:

endian& operator=(T i) { detail::store_big_endian<T, n_bits/8>(bytes, i); }

Without this, and without the endian(T val) constructor, a lot of things 
just don't work - for example, the binary arithmetic operators.

This makes me think that there should have been an operator= all along. 
For a start, this is just good practice: anywhere there's a constructor 
initializing from a particular type, there should usually also be an 
assignment operator taking the same type.

Second, I think the generated code for binary operators must have been 
suboptimal, since it seems that the computed result (a native integer) 
of adding two endians was being used to construct a temporary endian 
which was then copy-assigned into the actual result. At least, that's my 
interpretation of the compilation errors I was getting before I put 
operator= in. To test this, take out the constructors and do:
(Continue reading)

Permalink | Reply |
headers
Beman Dawes | 30 May 02:49
Favicon

Re: [integer] Promotion of endian library code from vault

Neil Mayhew wrote:
> On 28/05/08 05:50 PM Neil Mayhew wrote:
>> Perhaps the simplest and best solution is therefore:
>>
>>     class endian< ... >
>>       : cover_operators< ... >
>>     {
>>       public:
>> #if defined(CXX_0X) || !defined(ENDIANS_IN_UNIONS)
>>         endian() {}
>>         endian(T val) { ... }
>> #endif
> 
> I just discovered that an operator=(T) is needed as well:
> 
> endian& operator=(T i) { detail::store_big_endian<T, n_bits/8>(bytes, i); }
> 
> Without this, and without the endian(T val) constructor, a lot of things 
> just don't work - for example, the binary arithmetic operators.
> 
> This makes me think that there should have been an operator= all along. 
> For a start, this is just good practice: anywhere there's a constructor 
> initializing from a particular type, there should usually also be an 
> assignment operator taking the same type.

Did you get a chance to look at the new version in the vault? See the 
message I posted two or three days ago: http://tinyurl.com/6xejo5

It provides operator=(T), for the reasons you identified.
(Continue reading)

Permalink | Reply |
headers
Neil Mayhew | 30 May 05:24

Re: [integer] Promotion of endian library code from vault

On 2008-05-29 18:49, Beman Dawes wrote:
> Neil Mayhew wrote:
>> I just discovered that an operator=(T) is needed as well:
>
> Did you get a chance to look at the new version in the vault?...
> It provides operator=(T), for the reasons you identified.

My apologies. I did look at it, but didn't notice the addition of 
operator=, and also didn't notice that I still had the older (0.6) 
version in my working directory.

It might be a good idea, when you post new versions of things to the 
vault, to keep one or two older versions around, to make it easier for 
people to see what's changed. I did still have 0.6 around, but it was on 
another machine and I didn't bother to copy it over to diff against it.

--Neil
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Permalink | Reply |
headers
Beman Dawes | 30 May 14:04
Favicon

Re: [integer] Promotion of endian library code from vault

Neil Mayhew wrote:
> On 2008-05-29 18:49, Beman Dawes wrote:
>> Neil Mayhew wrote:
>>> I just discovered that an operator=(T) is needed as well:
>> Did you get a chance to look at the new version in the vault?...
>> It provides operator=(T), for the reasons you identified.
> 
> My apologies. I did look at it, but didn't notice the addition of 
> operator=, and also didn't notice that I still had the older (0.6) 
> version in my working directory.
> 
> It might be a good idea, when you post new versions of things to the 
> vault, to keep one or two older versions around, to make it easier for 
> people to see what's changed. I did still have 0.6 around, but it was on 
> another machine and I didn't bother to copy it over to diff against it.

I really should put it into the subversion sandbox. I'll try to do that 
later today.

--Beman

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Permalink | Reply |
headers
vicente.botet | 27 Sep 09:51
Gravatar

Re: [integer] Promotion of endian library code from vault

----- Original Message ----- 
From: "Beman Dawes" <bdawes <at> acm.org>
To: <boost <at> lists.boost.org>
Sent: Friday, May 30, 2008 2:04 PM
Subject: Re: [boost] [integer] Promotion of endian library code from vault

> I really should put it into the subversion sandbox. I'll try to do that 
> later today.

Hi Beman, 
there is a copy/paste bug on the sandbox.

#     ifdef BOOST_BIG_ENDIAN            // BUG !!!!!!!!!!

must be 

#     ifdef BOOST_LITTLE_ENDIAN

isn't it?

Vicente

    template <typename T, std::size_t n_bits>
    class endian< endianness::little, T, n_bits, alignment::aligned  >
      : cover_operators< endian< endianness::little, T, n_bits, 
alignment::aligned >, T >
    {
        BOOST_STATIC_ASSERT( (n_bits/8)*8 == n_bits );
        BOOST_STATIC_ASSERT( sizeof(T) == n_bits/8 );
      public:
(Continue reading)

Permalink | Reply |
headers
Beman Dawes | 1 Oct 17:33
Favicon

Re: [integer] Promotion of endian library code from vault

vicente.botet wrote:
> ----- Original Message ----- From: "Beman Dawes" <bdawes <at> acm.org>
> To: <boost <at> lists.boost.org>
> Sent: Friday, May 30, 2008 2:04 PM
> Subject: Re: [boost] [integer] Promotion of endian library code from vault
> 
> 
>> I really should put it into the subversion sandbox. I'll try to do 
>> that later today.
> 
> Hi Beman, there is a copy/paste bug on the sandbox.
> 
> #     ifdef BOOST_BIG_ENDIAN            // BUG !!!!!!!!!!
> 
> must be
> #     ifdef BOOST_LITTLE_ENDIAN
> 
> isn't it?

Nice catch! Fixed.

This would have been detected by the regression tests if they were run 
on a big endian machine.

--Beman

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
(Continue reading)

Permalink | Reply |
headers
Beman Dawes | 30 May 03:51
Favicon

Re: [integer] Promotion of endian library code from vault

Neil Mayhew wrote:
> On 28/05/08 04:25 PM Beman Dawes wrote:
>> Neil Mayhew wrote:
>>> ... have a base class that contains everything except the 
>>> constructors, and a derived class that adds the constructors
>> There are some issues with the base class approach. For it to be a 
>> POD, it can't have constructors, base classes, or private members.
> 
> Strictly speaking, yes, although we are only concerned about memory 
> layout here, so the class doesn't have to be a true POD. The other 
> concern is for the class to be able to live inside a union, and I think 
> the constructors are the the only thing stopping that. (I #if'd the 
> constructors and I was then able to use endian in a union.)

I was wrong above - you are correct that just the constructs have to be 
removed. Private members and base classes are OK.

>> It might be better to just have two separate types, endian and 
>> old_endian. (old_endian isn't very clear as a name, but calling it 
>> pod_endian becomes invalid the moment C++0x ships.)
>>
>>> The typedefs to big32_t etc. would then be #if'd to correspond to the 
>>> POD or non-POD classes as desired. I think this is better than 
>>> putting the #if's around the constructors themselves.
>> Hummm... Seems too obscure. Better to have two sets of typedefs, with 
>> the C++03 set prefixed by old_ (or whatever better name anyone can 
>> come up with.)
> 
> I don't like "old_", at least not if it has to appear in my code! :-)
>
(Continue reading)

Permalink | Reply |
headers
Neil Mayhew | 30 May 05:53

Re: [integer] Promotion of endian library code from vault

On 2008-05-29 19:51, Beman Dawes wrote:
>>> There are some issues with the base class approach. For it to be a 
>>> POD, it can't have constructors, base classes, or private members.
>
> I was wrong above ... just the constructs have to be removed. Private 
> members and base classes are OK.

Does that mean the FAQ entry "Are endian types POD's?" in the 
documentation needs to be changed? Or do you just mean what's said in 
the next FAQ entry, that "this problem has never been observed in a real 
compiler"?

If we do implement the suggestion of a switch to turn off the definition 
of constructors, there will of course need to be some adjustments to the 
documentation. I'm still thinking about what those changes might 
usefully be.

BTW, if you are making changes to the documentation, I have a very minor 
suggestion. In "What are the implications of C++03 endian types not 
being POD's?", it would make for easier reading if you made it clearer 
that there are two points being made; eg "Also, compilers aren't 
required to align or lay out storage in portable ways for non-POD types, 
although in practice this problem has never been observed in a real 
compiler."

--Neil
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
(Continue reading)

Permalink | Reply |
headers
Neil Mayhew | 29 May 01:59

Re: [integer] Promotion of endian library code from vault

On 26/05/08 02:31 PM Neil Mayhew wrote:
> On 24/05/08 05:45 AM Beman Dawes wrote:
>> Do you have a proposed design for bit-range access?
>
> I was thinking of adding methods to read and write the given bits within
> the value:
>
> value_type endian::bits(std::size_t offset, std::size_t length) const;
> value_type endian::bits(std::size_t offset, std::size_t length,
>                         value_type value);
>
> struct packet
> {
>     ubig24_t abc;
>
>     uint_least32_t field_a() const
>     {
>         return abc.bits(0, 6);
>     }
>     uint_least32_t field_b() const
>     {
>         return abc.bits(6, 4);
>     }
>     uint_least32_t field_c() const
>     {
>         return abc.bits(10, 14);
>     }
> ...
> };
(Continue reading)

Permalink | Reply |
headers
vicente.botet | 26 Sep 01:23
Gravatar

Re: [integer] Promotion of endian library code from vault

----- Original Message ----- 
From: "Neil Mayhew" <neil_mayhew <at> users.sourceforge.net>
To: <boost <at> lists.boost.org>
Sent: Thursday, May 29, 2008 1:59 AM
Subject: Re: [boost] [integer] Promotion of endian library code from vault

> On 26/05/08 02:31 PM Neil Mayhew wrote:
>> On 24/05/08 05:45 AM Beman Dawes wrote:
>>> Do you have a proposed design for bit-range access?
>>
>> I was thinking of adding methods to read and write the given bits within
>> the value:
>>
>> value_type endian::bits(std::size_t offset, std::size_t length) const;
>> value_type endian::bits(std::size_t offset, std::size_t length,
>>                         value_type value);
>>
>
> On second thoughts, I've realised this would be easier to read, and
> safer, if the bits() methods used start and end rather than offset and
> length. Then the values in the methods would match up, and it would be
> more obvious if there were gaps or overlaps:
>
> return abc.bits(0, 6);
> ...
> return abc.bits(6, 10);
> ...
> return abc.bits(10, 14);

Yes I think this is eassier to read and write.
(Continue reading)

Permalink | Reply |
headers
Roland Schwarz | 27 May 09:19
Favicon

Re: [integer] Promotion of endian library code from vault

Beman Dawes schrieb:
> So the main help would be careful review and 
> commenting on a revised version. I'll try to get that out in a week or so.
>
>   
I use version 0.8:

Questions:

1) Would it be possible to add a conversion operator to char* ? At least 
for unaligned types
     this looks useful to me since it would avoid ugly casts in cases 
where conversion is safe.

2) I'd like to be able to initialize a struct containing endian<> types 
by making it a reference
    to some buffer. E.g.

    char buf[1234];
    my_struct& ms(buf);

    What would be needed to make this possible? (I'd expect this only to 
work if the
     entire struct contains of unaligned char[].) Is this feasible at all?

3) The naming endian for the lib does not caption its "real" purpose. At 
least I was not
    aware of it altough I was searching for it. I'd suggest something as 
ptype.hpp for
    _P_ortable _TYPES_.
(Continue reading)

Permalink | Reply |
headers
Neil Mayhew | 27 May 15:44

Re: [integer] Promotion of endian library code from vault

On 5/27/08 1:19 AM, Roland Schwarz wrote:
> 1) Would it be possible to add a conversion operator to char* ? At least
> for unaligned types.
>
> 2) I'd like to be able to initialize a struct containing endian<> types
> by making it a reference to some buffer.

I think perhaps you've misunderstood the purpose of endian. The idea is
to be able to define a structure like this:

struct example_t
{
    big8_t a;
    big24_t b;
    big32_t c;
};

example_t data;

You read it from a file or a socket with read(fd, &data, sizeof(data))
and then use data.a, data.b and data.c as if they were built-in integer
types.

Reading into a character buffer and then "mapping" a structure on top of
it could be done with reinterpret_cast<example_t*>(buffer), although
this is not the preferred approach. The idea is not to take individual
endian integers and force them to a particular position within a buffer,
but to let the whole struct do the work of computing the offsets. Of
course, that can be still done with reinterpret_cast, but it's not how
the library is designed to work.
(Continue reading)

Permalink | Reply |
headers
Hansi | 27 May 16:41

Re: [integer] Promotion of endian library code from vault

It would be also great if it is possible to directly use lexical_cast 
with the endian library!

Neil Mayhew schrieb:
> On 5/27/08 1:19 AM, Roland Schwarz wrote:
>> 1) Would it be possible to add a conversion operator to char* ? At least
>> for unaligned types.
>>
>> 2) I'd like to be able to initialize a struct containing endian<> types
>> by making it a reference to some buffer.
> 
> I think perhaps you've misunderstood the purpose of endian. The idea is
> to be able to define a structure like this:
> 
> struct example_t
> {
>     big8_t a;
>     big24_t b;
>     big32_t c;
> };
> 
> example_t data;
> 
> You read it from a file or a socket with read(fd, &data, sizeof(data))
> and then use data.a, data.b and data.c as if they were built-in integer
> types.
> 
> Reading into a character buffer and then "mapping" a structure on top of
> it could be done with reinterpret_cast<example_t*>(buffer), although
> this is not the preferred approach. The idea is not to take individual
(Continue reading)

Permalink | Reply |
headers
Scott McMurray | 28 May 06:04

Re: [integer] Promotion of endian library code from vault

On Tue, May 27, 2008 at 10:41 AM, Hansi <hansipet <at> web.de> wrote:
> It would be also great if it is possible to directly use lexical_cast
> with the endian library!
>

I don't follow.

What would you want to want to use lexical_cast for?
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Permalink | Reply |
headers
Hansi | 28 May 07:33

Re: [integer] Promotion of endian library code from vault

sorry for the bad explanation...
I would use it only to convert to strings and backwards...

Scott McMurray schrieb:
> On Tue, May 27, 2008 at 10:41 AM, Hansi <hansipet <at> web.de> wrote:
>> It would be also great if it is possible to directly use lexical_cast
>> with the endian library!
>>
> 
> I don't follow.
> 
> What would you want to want to use lexical_cast for?
> _______________________________________________
> Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
> 

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Permalink | Reply |
headers
Roland Schwarz | 28 May 07:58
Favicon

Re: [integer] Promotion of endian library code from vault


Neil Mayhew wrote:
> I think perhaps you've misunderstood the purpose of endian. The idea is
> to be able to define a structure like this:
> 
> struct example_t
> {
>     big8_t a;
>     big24_t b;
>     big32_t c;
> };
> 
> example_t data;
> 
> You read it from a file or a socket with read(fd, &data, sizeof(data))
> and then use data.a, data.b and data.c as if they were built-in integer
> types.
> 
> Reading into a character buffer and then "mapping" a structure on top of
> it could be done with reinterpret_cast<example_t*>(buffer), although
> this is not the preferred approach.

I don't think I misunderstood the purpose. It is just that in my code I
ended up exactly as you said:

I have buffer in terms of char, i.e. char buffer[]. This buffer is not
from stream, file or socket. It comes from a fifo in my application.
The I want to make use of the buffer as:

if (type == id_foo) {
(Continue reading)

Permalink | Reply |
headers
Roland Schwarz | 28 May 11:33
Favicon

Re: [integer] Promotion of endian library code from vault

Just checked:
iostream::write
can't write to a void*

This means:

struct A_t {
  ulittle32_t a;
  ulittle32_t b;
} A;

std::fstream f("test.bin", ios::out|ios::binary);

f.write(&A, sizeof(A_t);

won't compile. And I think one should be able to expect this
to work for an endian lib.

Or am am I missing something here?

--

-- 
_________________________________________
  _  _  | Roland Schwarz
 |_)(_  | aka. speedsnail
 | \__) | mailto:roland.schwarz <at> chello.at
________| http://www.blackspace.at
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
(Continue reading)

Permalink | Reply |
headers
Beman Dawes | 29 May 00:09
Favicon

Re: [integer] Promotion of endian library code from vault

Roland Schwarz wrote:
> Just checked:
> iostream::write
> can't write to a void*

Actually, its expecting a const char*.

> 
> This means:
> 
> struct A_t {
>   ulittle32_t a;
>   ulittle32_t b;
> } A;
> 
> std::fstream f("test.bin", ios::out|ios::binary);
> 
> f.write(&A, sizeof(A_t);
> 
> won't compile. And I think one should be able to expect this
> to work for an endian lib.
> 
> Or am am I missing something here?

A reinterpret_cast is needed for a struct. It doesn't matter what is 
inside the struct.

--beman

_______________________________________________
(Continue reading)

Permalink | Reply |
headers
Scott McMurray | 29 May 03:33

Re: [integer] Promotion of endian library code from vault

On Wed, May 28, 2008 at 1:58 AM, Roland Schwarz
<roland.schwarz <at> chello.at> wrote:
> I have buffer in terms of char, i.e. char buffer[]. This buffer is not
> from stream, file or socket. It comes from a fifo in my application.
> The I want to make use of the buffer as:
>
> if (type == id_foo) {
>        foo& f(*reinterpret_cast<foo*>(&buffer[0]));
>        ... do something with foo ...
>
> }
>
> And I think this is not only ugly but also unnecessary, since foo
> already might be expressed in terms of char[]. If not, e.g. because
> foo is not made of unaligned data, I'd prefer a compiler error
> instead.
>

I think that there's a strong possibility that violates strong
aliasing requirements.

> Btw. read(&foo, sizeof(foo)) also will need to do an equivalent
> of reinterpret cast internally. Doesn't it?
>
> I just think that endian is a fairly general idea that should not
> be limited to stream, file and socket.
>

Is there a usage for it outside of I/O though?  I can't think of one.
(Continue reading)

Permalink | Reply |
headers
Neil Mayhew | 29 May 07:54

Re: [integer] Promotion of endian library code from vault

On 2008-05-27 23:58, Roland Schwarz wrote:
> I have buffer in terms of char, i.e. char buffer[]. This buffer is not 
> from stream, file or socket. It comes from a fifo in my application.

If you are using a FIFO, then I assume the data is originating on the 
same machine. In which case, there should be no need to use endian, as 
you can arrange that the endianness, and even the alignment, is native 
to the machine.

> The I want to make use of the buffer as:
>
> if (type == id_foo) {
>     foo& f(*reinterpret_cast<foo*>(&buffer[0]));
>     ... do something with foo ...
> }
>
> And I think this is not only ugly but also unnecessary, since foo 
> already might be expressed in terms of char[]. If not, e.g. because 
> foo is not made of unaligned data, I'd prefer a compiler error instead.
>
> Yes of course I always can do the reinterpret_cast, but if we can do 
> better, why shouldn't we?

I think reinterpret_cast is the honest thing to do, because that is 
actually what you are doing, reinterpreting a memory location. Just 
because an endian is defined in terms of chars internally, that's not 
actually what it is. Also, people reading your code will find it easier 
to understand what is going on if they see the reinterpret_cast, since 
this sort of poking around in memory, based on a type id, conventionally 
uses reinterpret_cast. It may be a little long-winded, but it's not
(Continue reading)

Permalink | Reply |

Gmane