Nick Stokes | 19 Jan 01:39 2011
Picon

g++ cross distro compilation problem

Dear all:

I have very specific trouble with building GCC for our compute clusters.

Our setup is a fairly common compute cluster environment, where users
login to a front-end and compile their codes. Then they submit to a
batch queue to run them on  the compute nodes. This is not exactly a
cross compilation problem as the hardware on the compute nodes and the
login node are fairly similar (both x86_64). The particular GNU/Linux
distro on them, however, don't match (a fact we unfortunately can do
nothing about). The login node runs OpenSUSE 11.3, and the compute
nodes are CentOS 5.5. (and these are upgraded time and again)

The current distro versions of GCC (4.5.0 for login, and 4.1.2 for
computes) are incompatible and insufficient anyway since we need to
maintain multiple versions of GCC to address user needs. The idea was
to install all of 4.3, 4.4 and 4.5 versions on different prefixes
(/opt/gcc/4.3, /opt/gcc/4.4 and /opt/gcc/4.5, respectively) and NFS
mount these on the login node for users.

We stage the compilers, install, and test them on the compute nodes.
Everything seems ok. But on the login node the C++ compiler spits out
the following errors (for a simple hello world program):

$ g++ hello.cpp
In file included from
/opt/gcc/4.4.3/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.4.3/../../../../include/c++/4.4.3/bits/localefwd.h:42,
                 from
/opt/gcc/4.4.3/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.4.3/../../../../include/c++/4.4.3/ios:42,
                 from
(Continue reading)

Ian Lance Taylor | 19 Jan 01:57 2011
Picon

Re: g++ cross distro compilation problem

Nick Stokes <randomaccessiterator <at> gmail.com> writes:

> The current distro versions of GCC (4.5.0 for login, and 4.1.2 for
> computes) are incompatible and insufficient anyway since we need to
> maintain multiple versions of GCC to address user needs. The idea was
> to install all of 4.3, 4.4 and 4.5 versions on different prefixes
> (/opt/gcc/4.3, /opt/gcc/4.4 and /opt/gcc/4.5, respectively) and NFS
> mount these on the login node for users.
>
> We stage the compilers, install, and test them on the compute nodes.
> Everything seems ok. But on the login node the C++ compiler spits out
> the following errors (for a simple hello world program):

This must have something to do with the different versions of glibc
being used on the different machines.  I'm not sure what the exact
problem is, though.

Ian

Jonathan Wakely | 19 Jan 02:14 2011
Picon

Re: g++ cross distro compilation problem

On 19 January 2011 00:39, Nick Stokes  wrote:
>
> We stage the compilers, install, and test them on the compute nodes.
> Everything seems ok. But on the login node the C++ compiler spits out
> the following errors (for a simple hello world program):
>
> $ g++ hello.cpp
> In file included from
> /opt/gcc/4.4.3/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.4.3/../../../../include/c++/4.4.3/bits/localefwd.h:42,
>                 from
> /opt/gcc/4.4.3/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.4.3/../../../../include/c++/4.4.3/ios:42,
>                 from
> /opt/gcc/4.4.3/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.4.3/../../../../include/c++/4.4.3/ostream:40,
>                 from
> /opt/gcc/4.4.3/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.4.3/../../../../include/c++/4.4.3/iostream:40,
>                 from main.cpp:1:
> /opt/gcc/4.4.3/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.4.3/../../../../include/c++/4.4.3/x86_64-unknown-linux-gnu/bits/c++locale.h:52:
> error: 'uselocale' was not declared in this scope

This indicates that the compiler was built on a system which had the
necessary pieces for the C++ runtime library to be configured with
--enable-clocale=gnu (it will be used automatically if configure
detects it is supported)

Apparently on the system where it's installed something is missing.
Probably something in glibc, as Ian suggests.

Jonathan Wakely | 19 Jan 02:17 2011
Picon

Re: g++ cross distro compilation problem

On 19 January 2011 01:14, Jonathan Wakely <jwakely.gcc <at> gmail.com> wrote:
> On 19 January 2011 00:39, Nick Stokes  wrote:
>>
>> We stage the compilers, install, and test them on the compute nodes.
>> Everything seems ok. But on the login node the C++ compiler spits out
>> the following errors (for a simple hello world program):
>>
>> $ g++ hello.cpp
>> In file included from
>> /opt/gcc/4.4.3/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.4.3/../../../../include/c++/4.4.3/bits/localefwd.h:42,
>>                 from
>> /opt/gcc/4.4.3/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.4.3/../../../../include/c++/4.4.3/ios:42,
>>                 from
>> /opt/gcc/4.4.3/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.4.3/../../../../include/c++/4.4.3/ostream:40,
>>                 from
>> /opt/gcc/4.4.3/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.4.3/../../../../include/c++/4.4.3/iostream:40,
>>                 from main.cpp:1:
>> /opt/gcc/4.4.3/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.4.3/../../../../include/c++/4.4.3/x86_64-unknown-linux-gnu/bits/c++locale.h:52:
>> error: 'uselocale' was not declared in this scope
>
> This indicates that the compiler was built on a system which had the
> necessary pieces for the C++ runtime library to be configured with
> --enable-clocale=gnu (it will be used automatically if configure
> detects it is supported)
>
> Apparently on the system where it's installed something is missing.
> Probably something in glibc, as Ian suggests.

I would try to build gcc on both systems, with the same options, and
compare the $TARGET/libstdc++-v3/config.log files to see what choice
(Continue reading)

Nick Stokes | 20 Jan 00:27 2011
Picon

Re: g++ cross distro compilation problem

Hi Ian and Jonathan,

On Tue, Jan 18, 2011 at 8:17 PM, Jonathan Wakely <jwakely.gcc <at> gmail.com> wrote:
> On 19 January 2011 01:14, Jonathan Wakely <jwakely.gcc <at> gmail.com> wrote:
>> On 19 January 2011 00:39, Nick Stokes  wrote:
>>>
>>> [..]
>>
>> This indicates that the compiler was built on a system which had the
>> necessary pieces for the C++ runtime library to be configured with
>> --enable-clocale=gnu (it will be used automatically if configure
>> detects it is supported)
>>
>> Apparently on the system where it's installed something is missing.
>> Probably something in glibc, as Ian suggests.
>
> I would try to build gcc on both systems, with the same options, and
> compare the $TARGET/libstdc++-v3/config.log files to see what choice
> of locale model is used.
>
> I expect you'll find a difference.  You could force the basic model to
> be used with --enable-clocale=generic, which should work the same
> everywhere.  Ideally though you'd want to find out why the gnu model
> doesn't work, and fix that.
>

Ian,  you are right on. The versions are different:
compute node (where gcc is built):  /lib64/libc-2.5.so
login node (where gcc is used):  /lib64/libc-2.11.2.so

(Continue reading)

Jonathan Wakely | 20 Jan 01:47 2011
Picon

Re: g++ cross distro compilation problem

On 19 January 2011 23:27, Nick Stokes wrote:
>
> Ian,  you are right on. The versions are different:
> compute node (where gcc is built):  /lib64/libc-2.5.so
> login node (where gcc is used):  /lib64/libc-2.11.2.so
>
> Jonathan, I looked at the config.logs (attached). Both seem to use gnu.

Yes, both your distros have a new enough glibc, so the problem might be simpler:
does it make any difference if you define _GNU_SOURCE when compiling?

That should ensure uselocale is defined by <locale.h>

> When I configure with --enable-clocale=generic it indeed remedies the
> issue and g++ works without errors. Are the any serious implications
> of not using gnu model? (any performance issues, or anything like that
> sort?)

The GNU model, which requires glibc 2.3 or later, supports a
per-thread locale (via the uselocale function) so that the locale can
be changed temporarily by the C++ runtime library without affecting
the global process-wide locale.  I believe this avoids possible race
conditions in multithreaded programs which make use of locales.

Nick Stokes | 20 Jan 16:53 2011
Picon

Re: g++ cross distro compilation problem

On Wed, Jan 19, 2011 at 7:47 PM, Jonathan Wakely <jwakely.gcc <at> gmail.com> wrote:
> On 19 January 2011 23:27, Nick Stokes wrote:
>>
>> Ian,  you are right on. The versions are different:
>> compute node (where gcc is built):  /lib64/libc-2.5.so
>> login node (where gcc is used):  /lib64/libc-2.11.2.so
>>
>> Jonathan, I looked at the config.logs (attached). Both seem to use gnu.
>
> Yes, both your distros have a new enough glibc, so the problem might be simpler:
> does it make any difference if you define _GNU_SOURCE when compiling?
>

Assuming you are referring to compiling a code with g++  (and not
compiling gcc itself), then no it doesn't make a difference. (in fact,
g++ -v shows that _GNU_SOURCE is already defined).

Could it be some other silly mistake on my part, e.g forgetting to set
an environment variable or something? Indeed there is the distro's g++
compiler installed on the login node, and most of these headers --
that are potentially incompatible with the gcc versions I am trying to
install under /opt -- are under common places  /usr/include etc.

>
> The GNU model, which requires glibc 2.3 or later, supports a
> per-thread locale (via the uselocale function) so that the locale can
> be changed temporarily by the C++ runtime library without affecting
> the global process-wide locale.  I believe this avoids possible race
> conditions in multithreaded programs which make use of locales.
>
(Continue reading)

Jonathan Wakely | 20 Jan 17:40 2011
Picon

Re: g++ cross distro compilation problem

On 20 January 2011 15:53, Nick Stokes wrote:
>
> Assuming you are referring to compiling a code with g++  (and not

Yep, that's what I meant.

> compiling gcc itself), then no it doesn't make a difference. (in fact,
> g++ -v shows that _GNU_SOURCE is already defined).
>
> Could it be some other silly mistake on my part, e.g forgetting to set
> an environment variable or something? Indeed there is the distro's g++
> compiler installed on the login node, and most of these headers --
> that are potentially incompatible with the gcc versions I am trying to
> install under /opt -- are under common places  /usr/include etc.

You don't need to set any environment variables to use GCC, the 4.4.3
compiler under /opt will find its own c++ headers not the ones in
/usr/include/c++/x.y.z (you can check the paths printed with -v to see
that for yourself)

You need to find why uselocale is not defined here:

/opt/gcc/4.4.3/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.4.3/../../../../include/c++/4.4.3/x86_64-unknown-linux-gnu/bits/c++locale.h:52:
error: 'uselocale' was not declared in this scope

Try using the -E flag and examining the preprocessed output. Do that
on both machines and look for differences in the path or contents of
the locale.h header.  To minimise the output it should be sufficient
to do:

(Continue reading)

Nick Stokes | 20 Jan 20:28 2011
Picon

Re: g++ cross distro compilation problem

On Thu, Jan 20, 2011 at 11:40 AM, Jonathan Wakely <jwakely.gcc <at> gmail.com> wrote:
> On 20 January 2011 15:53, Nick Stokes wrote:
>>
>> Assuming you are referring to compiling a code with g++  (and not
>
> Yep, that's what I meant.
>
>> compiling gcc itself), then no it doesn't make a difference. (in fact,
>> g++ -v shows that _GNU_SOURCE is already defined).
>>
>> Could it be some other silly mistake on my part, e.g forgetting to set
>> an environment variable or something? Indeed there is the distro's g++
>> compiler installed on the login node, and most of these headers --
>> that are potentially incompatible with the gcc versions I am trying to
>> install under /opt -- are under common places  /usr/include etc.
>
> You don't need to set any environment variables to use GCC, the 4.4.3
> compiler under /opt will find its own c++ headers not the ones in
> /usr/include/c++/x.y.z (you can check the paths printed with -v to see
> that for yourself)
>
> You need to find why uselocale is not defined here:
>
> /opt/gcc/4.4.3/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.4.3/../../../../include/c++/4.4.3/x86_64-unknown-linux-gnu/bits/c++locale.h:52:
> error: 'uselocale' was not declared in this scope
>
> Try using the -E flag and examining the preprocessed output. Do that
> on both machines and look for differences in the path or contents of
> the locale.h header.  To minimise the output it should be sufficient
> to do:
(Continue reading)

Jonathan Wakely | 20 Jan 20:57 2011
Picon

Re: g++ cross distro compilation problem

On 20 January 2011 19:28, Nick Stokes wrote:
>
> Great! This indeed revealed it.  In /usr/include/locale.h (same
> location, line  133, in both distros actually)  there is #ifdef
> __USE_GNU  on CentOS version, which is  #ifdef __USE_XOPEN2K8 in
> SUSE's version.   So, in fact if I define `__USE_XOPEN2K8'  while
> compiling on SUSE, it works. Hmm, go figure.. This can not be the
> right way to do this. What am I missing?

I don't know why they're different (on my glibc 2.12 system the
uselocale definition is guarded by __USE_GNU, just like your CentOS
system) but it looks like you've found the solution.

Users are not supposed to use the __USE_XXX macros, instead you should
define _GNU_SOURCE to enable __USE_GNU and _POSIX_C_SOURCE=200809L (or
greater) to enable __USE_XOPEN2K8.

Jonathan Wakely | 20 Jan 21:01 2011
Picon

Re: g++ cross distro compilation problem

On 20 January 2011 19:57, Jonathan Wakely <jwakely.gcc <at> gmail.com> wrote:
> On 20 January 2011 19:28, Nick Stokes wrote:
>>
>> Great! This indeed revealed it.  In /usr/include/locale.h (same
>> location, line  133, in both distros actually)  there is #ifdef
>> __USE_GNU  on CentOS version, which is  #ifdef __USE_XOPEN2K8 in
>> SUSE's version.   So, in fact if I define `__USE_XOPEN2K8'  while
>> compiling on SUSE, it works. Hmm, go figure.. This can not be the
>> right way to do this. What am I missing?
>
> I don't know why they're different (on my glibc 2.12 system the
> uselocale definition is guarded by __USE_GNU, just like your CentOS
> system) but it looks like you've found the solution.
>
> Users are not supposed to use the __USE_XXX macros, instead you should
> define _GNU_SOURCE to enable __USE_GNU and _POSIX_C_SOURCE=200809L (or
> greater) to enable __USE_XOPEN2K8.

It looks as though you can also define _XOPEN_SOURCE=700 (or greater)
to set __USE_XOPEN2K8

Either way, you should use one of those standard feature test macros,
not the __USE_XOPEN2K8 one which is an internal implementation
details, see man feature_test_macros for more details.

Nick Stokes | 21 Jan 01:57 2011
Picon

Re: g++ cross distro compilation problem

On Thu, Jan 20, 2011 at 3:01 PM, Jonathan Wakely <jwakely.gcc <at> gmail.com> wrote:
> On 20 January 2011 19:57, Jonathan Wakely <jwakely.gcc <at> gmail.com> wrote:
>> On 20 January 2011 19:28, Nick Stokes wrote:
>>>
>>> Great! This indeed revealed it.  In /usr/include/locale.h (same
>>> location, line  133, in both distros actually)  there is #ifdef
>>> __USE_GNU  on CentOS version, which is  #ifdef __USE_XOPEN2K8 in
>>> SUSE's version.   So, in fact if I define `__USE_XOPEN2K8'  while
>>> compiling on SUSE, it works. Hmm, go figure.. This can not be the
>>> right way to do this. What am I missing?
>>
>> I don't know why they're different (on my glibc 2.12 system the
>> uselocale definition is guarded by __USE_GNU, just like your CentOS
>> system) but it looks like you've found the solution.
>>
>> Users are not supposed to use the __USE_XXX macros, instead you should
>> define _GNU_SOURCE to enable __USE_GNU and _POSIX_C_SOURCE=200809L (or
>> greater) to enable __USE_XOPEN2K8.
>
> It looks as though you can also define _XOPEN_SOURCE=700 (or greater)
> to set __USE_XOPEN2K8
>
> Either way, you should use one of those standard feature test macros,
> not the __USE_XOPEN2K8 one which is an internal implementation
> details, see man feature_test_macros for more details.
>

Thanks, these are great leads!

But unfortunately this didn't work either. The reason is subtle (and
(Continue reading)

Nick Stokes | 21 Jan 03:03 2011
Picon

Re: g++ cross distro compilation problem

On Thu, Jan 20, 2011 at 7:57 PM, Nick Stokes
<randomaccessiterator <at> gmail.com> wrote:
> On Thu, Jan 20, 2011 at 3:01 PM, Jonathan Wakely <jwakely.gcc <at> gmail.com> wrote:
>> On 20 January 2011 19:57, Jonathan Wakely <jwakely.gcc <at> gmail.com> wrote:
>>> On 20 January 2011 19:28, Nick Stokes wrote:
>>>>
>>>> Great! This indeed revealed it.  In /usr/include/locale.h (same
>>>> location, line  133, in both distros actually)  there is #ifdef
>>>> __USE_GNU  on CentOS version, which is  #ifdef __USE_XOPEN2K8 in
>>>> SUSE's version.   So, in fact if I define `__USE_XOPEN2K8'  while
>>>> compiling on SUSE, it works. Hmm, go figure.. This can not be the
>>>> right way to do this. What am I missing?
>>>
>>> I don't know why they're different (on my glibc 2.12 system the
>>> uselocale definition is guarded by __USE_GNU, just like your CentOS
>>> system) but it looks like you've found the solution.
>>>
>>> Users are not supposed to use the __USE_XXX macros, instead you should
>>> define _GNU_SOURCE to enable __USE_GNU and _POSIX_C_SOURCE=200809L (or
>>> greater) to enable __USE_XOPEN2K8.
>>
>> It looks as though you can also define _XOPEN_SOURCE=700 (or greater)
>> to set __USE_XOPEN2K8
>>
>> Either way, you should use one of those standard feature test macros,
>> not the __USE_XOPEN2K8 one which is an internal implementation
>> details, see man feature_test_macros for more details.
>>
>
> Thanks, these are great leads!
(Continue reading)

Jonathan Wakely | 21 Jan 10:16 2011
Picon

Re: g++ cross distro compilation problem

On 21 January 2011 02:03, Nick Stokes wrote:
>>
>> But unfortunately this didn't work either. The reason is subtle (and
>> elusive!): On CentOS (where gcc is built) the GCC features.h header is
>> defining __USE_XOPEN2K, and not __USE_XOPEN2K8  conditioned on
>> _XOPEN_SOURCE (or _POSIX_C_SOURCE) being defined.   But on the
>> front-end SUSE, the /usr/include/locale.h is expecting __USE_XOPEN2K8,
>> hence fails.

By "GCC features.h header" do you mean one under the GCC installation
tree in /opt, or /usr/include/features.h?  If the former, that
explains the problem - you have a fixinclude'd features.h created on
the old system, which has a glibc that predates POSIX 2008, so
_GNU_SOURCE doesn't define __USE_XOPEN2K8 (as I think it would on a
newer glibc)

> I guess I can sweep under the rug for now by defining __USE_XOPEN2K8.
> Incidentally, I need to make this transparent for users. Is it
> possible to add such "custom" definitions to gcc at configure time (or
> is there some other way to accomplish this)?

http://gcc.gnu.org/onlinedocs/gcc/Spec-Files.html

Or you can provide a wrapper script which invokes the actual gcc
driver with the additional flags you need.

Rather than defining __USE_XOPEN2K8 (which is definitely not meant to
be defined/undefined by users) I would consider modifying the
features.h file so that _GNU_SOURCE sets __XOPEN28K.  You could try
building the same GCC on a login machine, and compare the features.h
(Continue reading)


Gmane