gmane.linux.gentoo.devel

ICC Profile

The intel C Compiler (icc) has an ebuild for gentoo and the wiki has a script to integrate it with portage. 
This script works will in terms of building binaries, however when mixed with gcc environments there are
massive linking issues.  I propose that an ICC profile is made which contains specific versions and
default flags for people who want to build a mixed icc-gcc environment.  ICC is much faster than GCC and
although not free, offers a free non-commercial license.  I would be very interested in this project and
more than willing to help to the best of my abilities.  I've already been trying to maintain a mixed
environment with some luck, while there have been a lot of problems using dynamically linked libraries
(ld from intel and ld from gcc don't always get along), my system is sub
 statially faster.  The kernel obviously will still be built under gcc as well as bash (unless intel helps
submit patches to make the code work with their compiler).  There are many tools icc
  !
has to offer for vectorization.  If these were streamlined into Gentoo with a fetch restriction for ICC, a
bootsrapping boot disk could be made and result in a very fast
distribution. 
--

-- 
gentoo-dev <at> lists.gentoo.org mailing list

headers

Luca Barbato | 18 Jul 04:34

Re: ICC Profile

Adam Stylinski wrote:
> The intel C Compiler (icc)

icc, xlc, llvm, sunstudio could be interesting fields of discovery.

Which are the pitfalls of using icc?

lu

-- 

Luca Barbato
Gentoo Council Member
Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero

--

-- 
gentoo-dev <at> lists.gentoo.org mailing list

headers

Robert Bridge | 18 Jul 05:15

Re: ICC Profile

On Fri, 18 Jul 2008 11:34:11 +0900
Luca Barbato <lu_zero <at> gentoo.org> wrote:

> Adam Stylinski wrote:
> > The intel C Compiler (icc)
> 
> icc, xlc, llvm, sunstudio could be interesting fields of discovery.
> 
> Which are the pitfalls of using icc?
> 
> lu
> 

If I recall correctly, needing some packages compiled with ICC flags
set one way, and needing others compiled with the same flag set the
oter way.

Can it compile a kernel yet?

Rob.
--

-- 
gentoo-dev <at> lists.gentoo.org mailing list

headers

Doug Goldstein | 18 Jul 15:55

Re: ICC Profile

Robert Bridge wrote:
> On Fri, 18 Jul 2008 11:34:11 +0900
> Luca Barbato <lu_zero <at> gentoo.org> wrote:
>
>   
>> Adam Stylinski wrote:
>>     
>>> The intel C Compiler (icc)
>>>       
>> icc, xlc, llvm, sunstudio could be interesting fields of discovery.
>>
>> Which are the pitfalls of using icc?
>>
>> lu
>>
>>     
>
> If I recall correctly, needing some packages compiled with ICC flags
> set one way, and needing others compiled with the same flag set the
> oter way.
>
> Can it compile a kernel yet?
>
> Rob.
>   
No. It doesn't implement all of GCCs extensions and all GNUisms that the 
kernel relies on.
--

-- 
gentoo-dev <at> lists.gentoo.org mailing list

(Continue reading)

headers

Adam Stylinski | 18 Jul 05:20

Re: ICC Profile

There are very few pitfalls, none of which I see as real killers. These include:

1.) Closed source compiler: Yes this stands against what we believe, and yes by closing their source
they're protecting the trade secrets of their architecture. It also could be more difficult to debug,
although that's highly unlikely, they have the idb (intel debugger) which works very much like gdb.

2.) Linking issues: So far it's pretty versatile, but it doesn't always cooperate with gcc compiled apps.
It may be a good strategy to make the troublesome apps which won't compile with ICC compile with ICC.

Pro's:

1.) Bloody fast machine code. Intel obfuscates their architecture but they give back to the community as
much as possible to make their hardware marketable toward the open source sysadmin, developer, etc etc.
Their drivers are open and they develop for the kernel constantly. This cooperation leads me to believe
that they would assist a team of developers in making 100% icc compatible code.

2.) Bloody fast compilation time. In my experience the compiler works much faster even with heavy optimization.

3.) Takes full advantage of SSE enabled hardware. SIMD instructions are quite useful, code is extremely vectorized.

4.) will project gentoo toward the power user more, helps the gentoo image, and overall will make linux a
more professional operating system (and a quite competitive alternative to something like a
SPARC+Solaris configuration). This would also make cluster farms and science application more
respectful toward the gentoo community. The academic and research world already uses ICC to compile
their apps for the sake of speed. The interprocedural optimizations for both the fortran and c/c++
compilers make it a must.

5.) It's free, albeit a commercial product. As gentoo is entirely non-profit, there is no restriction when
it comes to licensing. The binaries won't be sold for the intel-compiled livecd, and the compiler itself
with a fetch restriction allows the user to legally register for their free non-commercial license.

(Continue reading)

headers

Donnie Berkholz | 18 Jul 07:02

Re: ICC Profile

On 23:24 Thu 17 Jul     , Adam Stylinski wrote:
> There are very few pitfalls, none of which I see as real killers.  
> These include:
> 
> 1.) Closed source compiler: Yes this stands against what we believe, 
> and yes by closing their source they're protecting the trade secrets 
> of their architecture.  It also could be more difficult to debug, 
> although that's highly unlikely, they have the idb (intel debugger) 
> which works very much like gdb.

Doesn't really matter, although we could never move to it as the default 
without violating our social contract.

> 2.) Linking issues: So far it's pretty versatile, but it doesn't 
> always cooperate with gcc compiled apps.  It may be a good strategy to 
> make the troublesome apps which won't compile with ICC compile with 
> ICC.

Pretty sure you didn't mean ICC twice here, but sure.

> Pro's:
> 
> 1.) Bloody fast machine code.  Intel obfuscates their architecture but 
> they give back to the community as much as possible to make their 
> hardware marketable toward the open source sysadmin, developer, etc 
> etc.  Their drivers are open and they develop for the kernel 
> constantly.  This cooperation leads me to believe that they would 
> assist a team of developers in making 100% icc compatible code.

OK. This involves upstream projects more than us, though.

(Continue reading)

headers

Sébastien Fabbro | 18 Jul 16:16

Re: ICC Profile

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Thursday July 17, Adam Stylinski wrote:

> Pro's:
> 
> 1.) Bloody fast machine code.  Intel obfuscates their architecture
> but they give back to the community as much as possible to make their
> hardware marketable toward the open source sysadmin, developer, etc
> etc.  Their drivers are open and they develop for the kernel
> constantly.  This cooperation leads me to believe that they would
> assist a team of developers in making 100% icc compatible code.  

Gentoo is not supported from Intel, and they had not plans doing so. As
of October 2007, I asked their Software channel whether Gentoo
users have similar support as RedHat or SUSE users and the answer was:

"No, we have no current plans to support Gentoo. Also, Gentoo is NOT a
derivative of a Linux we do support.  My understanding is that it is
independently derived from Kernel.org.  Thus it is less likely to work
than a distro which is a derivative of a supported distribution.

Meanwhile, Debian/Ubuntu got support, so things might change if
Gentoo re-becomes/remains popular. Any Intel dev reading this list,
please contact us.

And as Luca mentioned, having sunstudio, xlc (is this one free?) or llvm
would not make Intel a privileged case for Gentoo.

(Continue reading)

headers

Adam Stylinski | 18 Jul 05:24

Re: ICC Profile

There is some record of a version of the kernel being compiled with some patches involved.  It's probably
possible, I'd imagine.  Though, this is not necessary for the first builds.
--

-- 
gentoo-dev <at> lists.gentoo.org mailing list

headers

Adam Stylinski | 18 Jul 05:33

Re: ICC Profile

Oh yes, and we can also take advantage of the free support that intel offers for all users:

http://www.intel.com/support/performancetools/sb/CS-017156.htm

Not to mention they have forums filled with intel developers/experts which we can get involved.  I'm sure
intel would benefit from this and be more than excited to collaborate:
http://softwarecommunity.intel.com/isn/Community/en-US/forums/default.aspx

--

-- 
gentoo-dev <at> lists.gentoo.org mailing list

headers

Donnie Berkholz | 18 Jul 06:56

Re: ICC Profile

On 14:23 Thu 17 Jul     , Adam Stylinski wrote:
> The intel C Compiler (icc) has an ebuild for gentoo and the wiki has a 
> script to integrate it with portage.  This script works will in terms 
> of building binaries, however when mixed with gcc environments there 
> are massive linking issues.  I propose that an ICC profile is made 
> which contains specific versions and default flags for people who want 
> to build a mixed icc-gcc environment.  ICC is much faster than GCC and 
> although not free, offers a free non-commercial license.  I would be 
> very interested in this project and more than willing to help to the 
> best of my abilities.  I've already been trying to maintain a mixed 
> environment with some luck, while there have been a lot of problems 
> using dynamically linked libraries (ld from intel and ld from gcc 
> don't always get along), my system is substatially faster.  The kernel 
> obviously will still be built under gcc as well as bash (unless intel 
> helps submit patches to make the code work with their compiler).  
> There are many tools icc has to offer for vectorization.  If these 
> were streamlined into Gentoo with a fetch restriction for ICC, a 
> bootsrapping boot disk could be made and result in a very fast 
> distribution. -- gentoo-dev <at> lists.gentoo.org mailing list

Adam,

I've already got a mixed system that works reasonably well. I use 
/etc/portage/env/ to control which packages build with icc and 
icc-specific CFLAGS by having /etc/portage/env/$CATEGORY/$PN be symlinks 
to /etc/portage/env/env.icc with these contents:

  echo " * Exporting: Intel compilers in $EBUILD_PHASE"
  CC="icc"
  CXX="icc"

(Continue reading)

headers

Nikos Chantziaras | 18 Jul 11:55

Re: ICC Profile

Adam Stylinski wrote:
> The intel C Compiler (icc) has an ebuild for gentoo and the wiki has
> a script to integrate it with portage.

I'm using ICC for programs where I have numbers about performance; for 
example, bzip2 is 30% faster here when compiled with ICC.

However, one thing I don't like about ICC is that when you post in the 
Intel forums about a problem you have on Gentoo, the reply is "Gentoo is 
not supported, please use RedHat."  If Gentoo is going to "support" ICC, 
ICC should be supporting Gentoo as well.

--

-- 
gentoo-dev <at> lists.gentoo.org mailing list

headers

Adam Stylinski | 18 Jul 15:22

Re: ICC Profile

I'm not suggesting that it be sold. Gentoo is
non-profit anyway, the livecd could be available for
download only. The binaries don't have to be
licensed if you're not selling them, however the
compiler does. This is where the non-commercial free
license comes in (with a fetch restriction requiring
the user to register for it). This is completely
free and I don't see it much more of a pain than it
is to go sign up for the IBM developers work to
extract their PPC version of java.
--

-- 
gentoo-dev <at> lists.gentoo.org mailing list

headers

Adam Stylinski | 18 Jul 15:22

Re: Re: ICC Profile

I actually know somebody working at intel, maybe he can get them more involved.
--

-- 
gentoo-dev <at> lists.gentoo.org mailing list

headers

Branko Badrljica | 18 Jul 15:43

Re: Re: ICC Profile

BTW: Is ICC really worth the fuss ?
I have checked around and reported that newest gcc-4.3 is able to to 
catch and sometimes even outperform icc ( not always, naturally).

Big news seemed to be thatnew gcc si close and sometimes better than icc.

Is it any truth to that and if it is, what is the motive of having 
non-open icc option ?

Adam Stylinski wrote:
> I actually know somebody working at intel, maybe he can get them more involved.
>   

--

-- 
gentoo-dev <at> lists.gentoo.org mailing list

headers

Nikos Chantziaras | 18 Jul 16:32

Re: ICC Profile

Branko Badrljica wrote:
> BTW: Is ICC really worth the fuss ?
> I have checked around and reported that newest gcc-4.3 is able to to 
> catch and sometimes even outperform icc ( not always, naturally).
> 
> Big news seemed to be thatnew gcc si close and sometimes better than icc.
> 
> Is it any truth to that and if it is, what is the motive of having 
> non-open icc option ?

The programs where I care about speed the most are gzip, bzip2, oggenc, 
lame, x264...  I guess you get the "pattern", they're 
encoders/compressors.  ICC wins in every one of them (speed increase is 
quite dramatic in the case of gzip and bzip2; 20% to 30%).  GCC won with 
diffutils though; 2% faster than ICC.

Testing other tools is difficult; how do you measure if X is faster with 
ICC?  Or KDE?  (If it even compiles with ICC; didnt' test.)

There's the issue of bugs though; programs break even with different 
versions of GCC (see Thunderbird; it breaks when compiled with *any* 
form of optimization turned on with GCC 4.3. See bugs 223375 and 
217805).  Who knows what breakage can occur with ICC.  The vast majority 
of projects out there are developed and tested only on GCC systems.

Then there's the show stopper #1 bug: every C++ source file that has an 
#include <limits.h> refuses to compile with ICC (at least on my system; 
AMD64).  Intel responds with "use RedHat where it works, we don't 
support Gentoo."  Supporting ICC in Gentoo is probably going to be too 
difficult if Intel refuses to even try to fix bugs that don't show up in

(Continue reading)

headers

Sébastien Fabbro | 18 Jul 16:16

Re: ICC Profile

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Apologies if you received this mail already, I'm having problems with my
smtp.

On Thursday July 17 Adam Stylinski wrote:

> The intel C Compiler (icc) has an ebuild for gentoo and the wiki has
> a script to integrate it with portage.  This script works will in
> terms of building binaries, however when mixed with gcc environments
> there are massive linking issues.  I propose that an ICC profile is
> made which contains specific versions and default flags for people
> who want to build a mixed icc-gcc environment.  ICC is much faster
> than GCC and although not free, offers a free non-commercial
> license.  I would be very interested in this project and more than
> willing to help to the best of my abilities.  I've already been
> trying to maintain a mixed environment with some luck, while there
> have been a lot of problems using dynamically linked libraries (ld
> from intel and ld from gcc don't always get along), my system is
> substatially faster.  The kernel obviously will still be built under
> gcc as well as bash (unless intel helps submit patches to make the
> code work with their compiler).  There are many tools icc ! has to
> offer for vectorization.  If these were streamlined into Gentoo with
> a fetch restriction for ICC, a bootsrapping boot disk could be made
> and result in a very fast distribution. 


An icc profile would be welcome. I've been the maintainer of icc (and
other Intel tools) for the last year or so more by default than

(Continue reading)

headers

Robert Bridge | 18 Jul 17:29

Re: ICC Profile

On Fri, 18 Jul 2008 15:16:18 +0100
Sébastien Fabbro <bicatali <at> gentoo.org> wrote:

> There was some attempts a few years ago for rolling up a full Gentoo
> with icc, but it hit several problems if I recall. Now both icc and
> gcc have improved since then.

Including needing package specific CFLAGS because some packages in
@system needed mutually exclusive flag settings. 

I'll see if I can dig up the link for an old blog on the topic later.
--

-- 
gentoo-dev <at> lists.gentoo.org mailing list

headers

Adam Stylinski | 18 Jul 16:21

Re: Re: ICC Profile

GCC 4.3 is catching up, but they are no where near utilizing SSE4 or SSE5 instructions.  

http://blog.alphagemini.org/2008/03/icc-vs-gcc-43.html

He concludes that it's not worth pursuing, but I beg to differ.  Those are signifcant differences for a processor.
--

-- 
gentoo-dev <at> lists.gentoo.org mailing list

headers

Ciaran McCreesh | 18 Jul 16:34

Re: Re: ICC Profile

On Fri, 18 Jul 2008 10:24:58 -0400 (EDT)
Adam Stylinski <stylinae <at> email.uc.edu> wrote:
> GCC 4.3 is catching up, but they are no where near utilizing SSE4 or
> SSE5 instructions.  
> 
> http://blog.alphagemini.org/2008/03/icc-vs-gcc-43.html
> 
> He concludes that it's not worth pursuing, but I beg to differ.
> Those are signifcant differences for a processor.

He doesn't establish whether the code in question is highly cpu-bound
or not when run on his system. For a lot of memory- and i/o-bound code,
there's little practical difference between gcc with optimisations
turned off and gcc with -frice-my-shorts except that the former
compiles an order of magnitude faster.

The more interesting question, then, is whether users run any
non-trivial cpu-bound programs. We know the applied science types do,
but they tend to be the ones who're doing clever things with icc
anyway. What about normal users?

--

-- 
Ciaran McCreesh

headers

Richard Freeman | 18 Jul 21:34

Re: Re: ICC Profile

Ciaran McCreesh wrote:
> 
> The more interesting question, then, is whether users run any
> non-trivial cpu-bound programs. We know the applied science types do,
> but they tend to be the ones who're doing clever things with icc
> anyway. What about normal users?
> 

I'm sure they do on some occasion if they encode compressed audio/video, 
or when compressing data with zip/etc.  That is probably the biggest 
application of cpu-bound software.

Personally, I use -Os across the board when it doesn't break.  As you 
said I tend to be memory/IO bound, and optimizing for size helps with 
both (swapping causes IO).  I'd probably benefit from using -O3 on the 
aforementioned CPU-intensive apps.
--

-- 
gentoo-dev <at> lists.gentoo.org mailing list

headers

Ciaran McCreesh | 18 Jul 21:44

Re: Re: ICC Profile

On Fri, 18 Jul 2008 15:34:53 -0400
Richard Freeman <rich0 <at> gentoo.org> wrote:
> Ciaran McCreesh wrote:
> > The more interesting question, then, is whether users run any
> > non-trivial cpu-bound programs. We know the applied science types
> > do, but they tend to be the ones who're doing clever things with icc
> > anyway. What about normal users?
> 
> I'm sure they do on some occasion if they encode compressed
> audio/video, or when compressing data with zip/etc.  That is probably
> the biggest application of cpu-bound software.

How much of that is memory bound? Of the things that aren't, how many
aren't written in assembly anyway? Of those things, what proportion of
the runtime is spent in those areas?

If you double the speed of something that takes up 2% of the overall
execution time, you can't measure the improvement.

Or looking at it the other way -- is there any reason to believe that
using icc (which can end up being a substantial pain in the arse, given
the way it tries to use gcc's c++ headers but doesn't support some of
the extensions or quirks that g++ does) will provide a genuine gain
for people who aren't already doing clever profile-directed trickery
anyway?

> I'd probably benefit from using -O3 on the aforementioned
> CPU-intensive apps.

The problem with -O3 is that function inlining can lead to a

(Continue reading)

headers

Richard Freeman | 18 Jul 21:53

Re: Re: ICC Profile

Ciaran McCreesh wrote:
> 
> How much of that is memory bound? Of the things that aren't, how many
> aren't written in assembly anyway? Of those things, what proportion of
> the runtime is spent in those areas?
> 
> If you double the speed of something that takes up 2% of the overall
> execution time, you can't measure the improvement.
> 
> Or looking at it the other way -- is there any reason to believe that
> using icc (which can end up being a substantial pain in the arse, given
> the way it tries to use gcc's c++ headers but doesn't support some of
> the extensions or quirks that g++ does) will provide a genuine gain
> for people who aren't already doing clever profile-directed trickery
> anyway?
> 
> 
> The problem with -O3 is that function inlining can lead to a
> substantial cache hit. Unless you're using profile-directed
> optimisations, which Gentoo doesn't support, it's extremely hit and
> miss as to whether O3 helps or hurts.
> 

I agree with all of the above.   Gentoo is about choice, so if people 
want to make ICC work well more power to them.  I agree that it would be 
hard to make it THE ONLY system compiler.  For those who do try it I'd 
be really interested in their findings.
--

-- 
gentoo-dev <at> lists.gentoo.org mailing list

(Continue reading)

headers

Duncan | 19 Jul 09:50

Re: ICC Profile

Richard Freeman <rich0 <at> gentoo.org> posted 4880F05D.4010905 <at> gentoo.org,
excerpted below, on  Fri, 18 Jul 2008 15:34:53 -0400:

> Personally, I use -Os across the board when it doesn't break.  As you
> said I tend to be memory/IO bound, and optimizing for size helps with
> both (swapping causes IO).  I'd probably benefit from using -O3 on the
> aforementioned CPU-intensive apps.

I did before gcc-4.3, but with 4.3, several of the other flags (-freorder-
blocks-and-partition, for instance, which increases cache hit rates but 
also increases size) I was using folded into -O2 but not -Os, and I 
decided -O2 was a better choice as a result.

Has anyone done a study of -Os vs -O2 with gcc-4.3.x, similar to the ICC/
gcc-4.x study linked upthread?  On x86_64 would be an extra bonus, 
particularly if it were on AMD, or if both Intel and AMD were studied, as 
would the effect of such flags as -freorder-blocks-and-partition, as 
mentioned above.  I'd love to see links, if so!

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

--

-- 
gentoo-dev <at> lists.gentoo.org mailing list

headers

Rémi Cardona | 19 Jul 12:51

Re: Re: ICC Profile

Duncan wrote:
> Has anyone done a study of -Os vs -O2 with gcc-4.3.x,

Just a quick note while on the subject : -Os is known to break some 
packages.

Although it has been a while since I've last had a full -Os system, 
there was a time when -Os was a _very_bad_idea_. That's why the Gnome 
Herd (and upstream Gnome) won't support anything more than -O2.

Of course, if anyone wants to use -Os to actually fix bugs... :)

Cheers

Rémi
--

-- 
gentoo-dev <at> lists.gentoo.org mailing list

headers

Adam Stylinski | 18 Jul 16:51

Re: Re: ICC Profile

He's also doing it on a core 2 duo.  It would be interesting to compare this with some mildly legacy hardware
(netburst pipelines) in order to see whether GCC does a comparable job.  My guess would be no, seeing as
netburst was extremely ugly and complicated, only intel would be able to write a compiler that took
advantage of it.  
--

-- 
gentoo-dev <at> lists.gentoo.org mailing list

headers

Adam Stylinski | 18 Jul 19:25

Re: ICC Profile

Also, in the academic environment the grad student/university can pay for the license that the student
slipstreams into their gentoo installation, making it 100% legal depending on how many seats he or she buys.
--

-- 
gentoo-dev <at> lists.gentoo.org mailing list

headers

Adam Stylinski | 18 Jul 22:07

Re: Re: ICC Profile

Perhaps we could write a script that compiles packages in portage with both ICC and GCC and runs them with
different flags.  I think there was an effort on the GCC side already to test flags with specific packages. 
We can then have the script run time on the applications doing work (again, that's a lot of packages for a lot
of different tasks).  If we can store these in the portage tree (yes, it's more metadata), the user can
actually benefit well from the optimizations.
--

-- 
gentoo-dev <at> lists.gentoo.org mailing list

headers

Michael Hammer | 19 Jul 13:01

Re: ICC Profile

Cold you please fix your reply?

--

-- 
------------------------------------------------------------
Michael Hammer    |    <mueli <at> gentoo.org>     |     Graz, AT
Gentoo Developer (Kerberos)  |  http://www.michael-hammer.at

headers

Adam Stylinski | 19 Jul 17:15

Re: ICC Profile

Well, it looks like we're not alone on this project.  This email was just sent to me:

Adam,
I managed to track down someone who can probably help you with your ICC work, if you're nice to him. :)
See the forwarded message:

Have your friend contact bill dot hilliard at intel dot com.  He's on the ICC team and can either answer any
questions or works next to someone who can.

Scott
--

-- 
gentoo-dev <at> lists.gentoo.org mailing list