Technical discussion about the development of the HotSpot virtual machine

Kurt Miller | 15 Mar 16:44

precompiled headers and includeDB*

Hi,

The use of precompiled headers obscures problems with
the includeDB* files and they have become out of date.
Building on linux without defining USE_PRECOMPILED_HEADER
in build/linux/makefiles/gcc.make reveals the problems
with the includeDB* files.

I've taken a stab at updating the includeDB files but
I'm not fully sure I've got it all right so far. compiler2
completes ok, but compiler1 bails like this:

make[4]: Entering directory `/home/truk/openjdk/hotspot/build/linux/linux_i486_compiler1/product'
make[4]: *** No rule to make target `node.hpp', needed by `compactingPermGenGen.o'. Stop.

I could use some guidance at this point. Do the patches
below look correct so far?

Thanks,
-Kurt

--- src/share/vm/includeDB_core.orig	Fri Mar  9 09:52:24 2007
+++ src/share/vm/includeDB_core	Sat Mar 10 19:43:08 2007
@@ -429,7 +429,7 @@ cardTableModRefBS.cpp			cardTableRS.hpp
 cardTableModRefBS.cpp			java.hpp
 cardTableModRefBS.cpp			mutexLocker.hpp
 cardTableModRefBS.cpp			sharedHeap.hpp
-cardTableModRefBS.cpp			space.hpp
+cardTableModRefBS.cpp			space.inline.hpp
 cardTableModRefBS.cpp			virtualspace.hpp
 cardTableModRefBS.cpp			universe.hpp

@@ -706,7 +706,6 @@ compactPermGen.hpp			permGen.hpp

 compactingPermGenGen.cpp                compactingPermGenGen.hpp
 compactingPermGenGen.cpp                filemap.hpp
-compactingPermGenGen.cpp                generation.inline.hpp
 compactingPermGenGen.cpp                generationSpec.hpp
 compactingPermGenGen.cpp                genOopClosures.inline.hpp
 compactingPermGenGen.cpp                concurrentMarkSweepGeneration.inline.hpp
@@ -715,7 +714,8 @@ compactingPermGenGen.cpp                
 compactingPermGenGen.cpp                symbolTable.hpp
 compactingPermGenGen.cpp                systemDictionary.hpp

-compactingPermGenGen.hpp                space.hpp
+compactingPermGenGen.hpp                space.inline.hpp
+compactingPermGenGen.hpp                generation.inline.hpp

 compile.hpp				jvmpi.h

@@ -911,6 +911,7 @@ debug.cpp                               
 debug.cpp                               vframe.hpp
 debug.cpp                               vmError.hpp
 debug.cpp                               vtableStubs.hpp
+debug.cpp                               node.hpp

 debug.hpp                               globalDefinitions.hpp

@@ -1067,7 +1068,7 @@ filemap.cpp                             
 filemap.cpp                             symbolTable.hpp

 filemap.hpp                             compactingPermGenGen.hpp
-filemap.hpp                             space.hpp
+filemap.hpp                             node.hpp

 forte.cpp                               collectedHeap.inline.hpp
 forte.cpp                               forte.hpp
@@ -1093,6 +1094,7 @@ fprofiler.cpp                           
 fprofiler.cpp                           task.hpp
 fprofiler.cpp                           universe.inline.hpp
 fprofiler.cpp                           vframe.hpp
+fprofiler.cpp                           node.hpp

 fprofiler.hpp                           thread_<os_family>.inline.hpp
 fprofiler.hpp                           timer.hpp
@@ -1318,7 +1320,7 @@ genOopClosures.inline.hpp               
 genOopClosures.inline.hpp               genRemSet.hpp
 genOopClosures.inline.hpp               parNewGeneration.hpp
 genOopClosures.inline.hpp               sharedHeap.hpp
-genOopClosures.inline.hpp               space.hpp
+genOopClosures.inline.hpp               node.hpp

 generationSpec.cpp                      asParNewGeneration.hpp
 generationSpec.cpp                      compactPermGen.hpp
@@ -2343,7 +2345,7 @@ klass.hpp                               
 klass.hpp                               specialized_oop_closures.hpp

 klass.inline.hpp                        klass.hpp
-klass.inline.hpp                        markOop.hpp
+klass.inline.hpp                        markOop.inline.hpp

 klassKlass.cpp                          collectedHeap.hpp
 klassKlass.cpp                          collectedHeap.inline.hpp
@@ -2885,6 +2887,7 @@ oopMap.cpp                              
 oopMap.cpp                              oopMap.hpp
 oopMap.cpp                              resourceArea.hpp
 oopMap.cpp                              signature.hpp
+oopMap.cpp                              node.hpp

 oopMap.hpp                              allocation.hpp
 oopMap.hpp                              compressedStream.hpp
@@ -3506,7 +3509,7 @@ space.hpp                               
 space.hpp                               blockOffsetTable.hpp
 space.hpp                               cardTableModRefBS.hpp
 space.hpp                               iterator.hpp
-space.hpp                               markOop.hpp
+space.hpp                               markOop.inline.hpp
 space.hpp                               memRegion.hpp
 space.hpp                               mutexLocker.hpp
 space.hpp                               os_<os_family>.inline.hpp
@@ -3678,7 +3681,7 @@ symbolTable.hpp	                        
 symbolTable.hpp                         symbolOop.hpp

 synchronizer.hpp                        handles.hpp
-synchronizer.hpp                        markOop.hpp
+synchronizer.hpp                        markOop.inline.hpp
 synchronizer.hpp                        top.hpp
 synchronizer.hpp                        perfData.hpp

@@ -4150,6 +4153,7 @@ virtualspace.cpp                        
 virtualspace.hpp                        allocation.hpp

 vmError.hpp                             globalDefinitions.hpp
+vmError.hpp                             thread_<os_family>.inline.hpp

 vmError.cpp                             arguments.hpp
 vmError.cpp                             debug.hpp
--- src/share/vm/includeDB_compiler2.orig	Fri Mar  9 10:06:49 2007
+++ src/share/vm/includeDB_compiler2	Sat Mar 10 13:55:25 2007
@@ -140,7 +140,7 @@ c2compiler.cpp                          

 c2compiler.hpp                          abstractCompiler.hpp

-c2_init_≤arch>.cpp                      compile.hpp
+c2_init_≤arch>.cpp                      node.hpp

 callGenerator.hpp                       callnode.hpp
 callGenerator.hpp                       compile.hpp
@@ -1052,6 +1052,7 @@ phase.cpp                               
 phase.cpp                               compileBroker.hpp
 phase.cpp                               nmethod.hpp
 phase.cpp                               phase.hpp
+phase.cpp                               node.hpp

 phase.hpp                               port.hpp
 phase.hpp                               timer.hpp
--- src/share/vm/gc_implementation/includeDB_gc_shared.orig	Fri Mar  9 09:45:44 2007
+++ src/share/vm/gc_implementation/includeDB_gc_shared	Fri Mar  9 09:46:07 2007
@@ -44,7 +44,7 @@ ageTable.cpp                            
 ageTable.cpp                            resourceArea.hpp
 ageTable.cpp                            sharedHeap.hpp

-ageTable.hpp                            markOop.hpp
+ageTable.hpp                            markOop.inline.hpp
 ageTable.hpp                            oop.hpp
 ageTable.hpp                            perfData.hpp

@@ -114,7 +114,7 @@ markSweep.cpp                           

 markSweep.hpp                           growableArray.hpp
 markSweep.hpp                           jvmpi.hpp
-markSweep.hpp                           markOop.hpp
+markSweep.hpp                           markOop.inline.hpp
 markSweep.hpp                           oop.hpp
 markSweep.hpp                           timer.hpp
 markSweep.hpp                           universe.hpp
--- src/share/vm/gc_implementation/includeDB_gc_parallelScavenge.orig	Fri Mar  9 11:43:24 2007
+++ src/share/vm/gc_implementation/includeDB_gc_parallelScavenge	Fri Mar  9 11:44:05 2007
@@ -92,6 +92,7 @@ gcTaskManager.cpp                       
 gcTaskManager.cpp                       gcTaskThread.hpp
 gcTaskManager.cpp                       mutex.hpp
 gcTaskManager.cpp                       mutexLocker.hpp
+gcTaskManager.cpp                       thread_<os_family>.inline.hpp

 gcTaskThread.hpp                        thread.hpp

Kurt Miller | 15 Mar 23:04

Re: precompiled headers and includeDB*

*

On Thursday 15 March 2007 11:44:45 am Kurt Miller wrote:
> Hi,
> 
> The use of precompiled headers obscures problems with
> the includeDB* files and they have become out of date.
> Building on linux without defining USE_PRECOMPILED_HEADER
> in build/linux/makefiles/gcc.make reveals the problems
> with the includeDB* files.

Ok I see where I went wrong now. Here is the full
update for consideration.

--- src/share/vm/includeDB_core.orig	Thu Mar  1 08:58:41 2007
+++ src/share/vm/includeDB_core	Thu Mar 15 15:02:21 2007
@@ -429,7 +429,7 @@ cardTableModRefBS.cpp			cardTableRS.hpp
 cardTableModRefBS.cpp			java.hpp
 cardTableModRefBS.cpp			mutexLocker.hpp
 cardTableModRefBS.cpp			sharedHeap.hpp
-cardTableModRefBS.cpp			space.hpp
+cardTableModRefBS.cpp			space.inline.hpp
 cardTableModRefBS.cpp			virtualspace.hpp
 cardTableModRefBS.cpp			universe.hpp

@@ -706,7 +706,6 @@ compactPermGen.hpp			permGen.hpp

 compactingPermGenGen.cpp                compactingPermGenGen.hpp
 compactingPermGenGen.cpp                filemap.hpp
-compactingPermGenGen.cpp                generation.inline.hpp
 compactingPermGenGen.cpp                generationSpec.hpp
 compactingPermGenGen.cpp                genOopClosures.inline.hpp
 compactingPermGenGen.cpp                concurrentMarkSweepGeneration.inline.hpp
@@ -715,7 +714,8 @@ compactingPermGenGen.cpp                
 compactingPermGenGen.cpp                symbolTable.hpp
 compactingPermGenGen.cpp                systemDictionary.hpp

-compactingPermGenGen.hpp                space.hpp
+compactingPermGenGen.hpp                space.inline.hpp
+compactingPermGenGen.hpp                generation.inline.hpp

 compile.hpp				jvmpi.h

@@ -2343,7 +2343,7 @@ klass.hpp                               
 klass.hpp                               specialized_oop_closures.hpp

 klass.inline.hpp                        klass.hpp
-klass.inline.hpp                        markOop.hpp
+klass.inline.hpp                        markOop.inline.hpp

 klassKlass.cpp                          collectedHeap.hpp
 klassKlass.cpp                          collectedHeap.inline.hpp
@@ -3506,7 +3506,7 @@ space.hpp                               
 space.hpp                               blockOffsetTable.hpp
 space.hpp                               cardTableModRefBS.hpp
 space.hpp                               iterator.hpp
-space.hpp                               markOop.hpp
+space.hpp                               markOop.inline.hpp
 space.hpp                               memRegion.hpp
 space.hpp                               mutexLocker.hpp
 space.hpp                               os_<os_family>.inline.hpp
@@ -3678,7 +3678,7 @@ symbolTable.hpp	                        
 symbolTable.hpp                         symbolOop.hpp

 synchronizer.hpp                        handles.hpp
-synchronizer.hpp                        markOop.hpp
+synchronizer.hpp                        markOop.inline.hpp
 synchronizer.hpp                        top.hpp
 synchronizer.hpp                        perfData.hpp

@@ -4150,6 +4150,7 @@ virtualspace.cpp                        
 virtualspace.hpp                        allocation.hpp

 vmError.hpp                             globalDefinitions.hpp
+vmError.hpp                             thread_<os_family>.inline.hpp

 vmError.cpp                             arguments.hpp
 vmError.cpp                             debug.hpp
--- src/share/vm/includeDB_compiler2.orig	Thu Mar  1 08:58:40 2007
+++ src/share/vm/includeDB_compiler2	Thu Mar 15 15:01:43 2007
@@ -140,7 +140,7 @@ c2compiler.cpp                          

 c2compiler.hpp                          abstractCompiler.hpp

-c2_init_≤arch>.cpp                      compile.hpp
+c2_init_≤arch>.cpp                      node.hpp

 callGenerator.hpp                       callnode.hpp
 callGenerator.hpp                       compile.hpp
@@ -392,6 +392,7 @@ debug.cpp                               
 debug.cpp                               icBuffer.hpp
 debug.cpp                               nmethod.hpp
 debug.cpp                               vtableStubs.hpp
+debug.cpp                               node.hpp

 debug_<arch>.cpp                        nmethod.hpp

@@ -499,11 +500,14 @@ exceptionHandlerTable.hpp               

 exceptions.cpp                          compileBroker.hpp

+filemap.hpp                             node.hpp
+
 fprofiler.cpp                           compile.hpp
 fprofiler.cpp                           deoptimization.hpp
 fprofiler.cpp                           icBuffer.hpp
 fprofiler.cpp                           nmethod.hpp
 fprofiler.cpp                           vtableStubs.hpp
+fprofiler.cpp                           node.hpp

 frame.cpp                               codeCache.hpp
 frame.cpp                               nmethod.hpp
@@ -551,6 +555,8 @@ generateOptoStub.cpp                    
 generateOptoStub.cpp                    runtime.hpp
 generateOptoStub.cpp                    type.hpp

+genOopClosures.inline.hpp               node.hpp
+
 globals.hpp                             c2_globals_≤arch>.hpp
 globals.hpp                             c2_globals_≤os_family>.hpp

@@ -945,6 +951,7 @@ oopMap.cpp                              
 oopMap.cpp                              oopMap.hpp
 oopMap.cpp                              scopeDesc.hpp
 oopMap.cpp                              signature.hpp
+oopMap.cpp                              node.hpp

 oopMap.hpp                              allocation.hpp
 oopMap.hpp                              compressedStream.hpp
@@ -1052,6 +1059,7 @@ phase.cpp                               
 phase.cpp                               compileBroker.hpp
 phase.cpp                               nmethod.hpp
 phase.cpp                               phase.hpp
+phase.cpp                               node.hpp

 phase.hpp                               port.hpp
 phase.hpp                               timer.hpp
--- src/share/vm/gc_implementation/includeDB_gc_shared.orig	Fri Mar  9 09:45:44 2007
+++ src/share/vm/gc_implementation/includeDB_gc_shared	Fri Mar  9 09:46:07 2007
@@ -44,7 +44,7 @@ ageTable.cpp                            
 ageTable.cpp                            resourceArea.hpp
 ageTable.cpp                            sharedHeap.hpp

-ageTable.hpp                            markOop.hpp
+ageTable.hpp                            markOop.inline.hpp
 ageTable.hpp                            oop.hpp
 ageTable.hpp                            perfData.hpp

@@ -114,7 +114,7 @@ markSweep.cpp                           

 markSweep.hpp                           growableArray.hpp
 markSweep.hpp                           jvmpi.hpp
-markSweep.hpp                           markOop.hpp
+markSweep.hpp                           markOop.inline.hpp
 markSweep.hpp                           oop.hpp
 markSweep.hpp                           timer.hpp
 markSweep.hpp                           universe.hpp
--- src/share/vm/gc_implementation/includeDB_gc_parallelScavenge.orig	Fri Mar  9 11:43:24 2007
+++ src/share/vm/gc_implementation/includeDB_gc_parallelScavenge	Fri Mar  9 11:44:05 2007
@@ -92,6 +92,7 @@ gcTaskManager.cpp                       
 gcTaskManager.cpp                       gcTaskThread.hpp
 gcTaskManager.cpp                       mutex.hpp
 gcTaskManager.cpp                       mutexLocker.hpp
+gcTaskManager.cpp                       thread_<os_family>.inline.hpp

 gcTaskThread.hpp                        thread.hpp

tom fogal | 15 Mar 17:07

Re: precompiled headers and includeDB*

 <200703151144.45721.lists@...>Kurt Miller writes:
> The use of precompiled headers obscures problems with
> the includeDB* files and they have become out of date.
[snip]
> 
> I could use some guidance at this point. Do the patches
> below look correct so far?
[snip]

While we're on the subject, is there a compelling reason for the
includeDB system in general?  I guess I don't really understand that
aspect of the build system very well.  It exists because of a potential
performance gain in build times through the use of precompiled headers,
correct?

As far as I know, the system uses these includeDB files and MakeDep to
somehow build a single giant include file, which gets precompiled.
Then all files end up including this precompiled header.

This seems like a lot of trouble to go through however, and I'm not
convinced of the performance gain.  In this scheme, compilation of
file.C won't have to parse its include of file.h, but it still has to
read 1 giant precompiled header file from disk.  It would seem that
reading + parsing a small number of includes wouldn't be that much
slower (or maybe even faster) then just a read of one large precompiled
header.  Unless most headers really need to get included in a
significant portion of the JDK, which seems unlikely but possible.

So, making the unreasonable assumption that I understand things
correctly, then dependencies && includes could be done in a more
traditional way.  Performance would hopefully be similar, and that
could deprecate `MakeDeps', making OpenJDK one step closer to not
requiring a bootstrap JDK (I'm not sure if thats a reasonable eventual
goal, but from posts it seems like that would be nice).

Thoughts / corrections to my logic?

-tom

steve goldman | 15 Mar 19:12

Picon

Re: precompiled headers and includeDB*

tom fogal wrote:
> 
> While we're on the subject, is there a compelling reason for the
> includeDB system in general?  I guess I don't really understand that
> aspect of the build system very well.  It exists because of a potential
> performance gain in build times through the use of precompiled headers,
> correct?

No that is not the reason it exists at all. The original reason it 
existed was because the original developers of hotspot intended to 
license different parts of the source base as different products. 
includeDB was the database used to manage which pieces were sent with 
which products.

Nowadays it generates the project files for Visual Studio and will 
likely generate the project files for NetBeans in addition to generating 
the dependencies for make.

> 
> As far as I know, the system uses these includeDB files and MakeDep to
> somehow build a single giant include file, which gets precompiled.
> Then all files end up including this precompiled header.
> 
> This seems like a lot of trouble to go through however, and I'm not
> convinced of the performance gain.  In this scheme, compilation of
> file.C won't have to parse its include of file.h, but it still has to
> read 1 giant precompiled header file from disk.  It would seem that
> reading + parsing a small number of includes wouldn't be that much
> slower (or maybe even faster) then just a read of one large precompiled
> header.  Unless most headers really need to get included in a
> significant portion of the JDK, which seems unlikely but possible.

since precompiled headers weren't the reason behind includeDB whether 
they are faster or not wasn't exactly the issue. On the platforms where 
precompiled headers exist the compilation is much faster though.

> 
> So, making the unreasonable assumption that I understand things
> correctly, then dependencies && includes could be done in a more
> traditional way.  Performance would hopefully be similar, and that
> could deprecate `MakeDeps', making OpenJDK one step closer to not
> requiring a bootstrap JDK (I'm not sure if thats a reasonable eventual
> goal, but from posts it seems like that would be nice).

What would you do about jvmti? It needs something to transform all that 
xml into code.

We have very recently had this issue come up internally at Sun. I think 
the consensus view is that if we didn't have MakeDeps we wouldn't go out 
and invent it. My view was that if MakeDeps is removed we make a large 
change to the system that only makes the system different not better. 
I'd rather see changes that made the system better. For example the 
earlier question about why is there a linux and solaris directory and 
not a shared unix directory. Refactoring that code would be a lot higher 
on my list.

--

-- 
Steve

John Rose | 15 Mar 19:35

Picon

Re: precompiled headers and includeDB*

On Mar 15, 2007, at 11:12 AM, steve goldman wrote:

> No that is not the reason it exists at all. The original reason it  
> existed was because the original developers of hotspot intended to  
> license different parts of the source base as different products.  
> includeDB was the database used to manage which pieces were sent  
> with which products.
...
> I think the consensus view is that if we didn't have MakeDeps we  
> wouldn't go out and invent it.

True.  There are other ways to manage this information, some of which  
would be more pleasant or familiar to some people.
But  in the big picture there are many more profitable changes needed.

It's probably a matter of personal preference, but I and others enjoy  
managing the inter-file dependencies in a centralized way.
And (and Ken Russell and others said last time the topic came up)  
checking the dependencies for circularity up front is handy.
This is the more valuable because there are "build flavors" of VM  
(C1, C2).

A final bit of consensus:  This is only a minor part of the learning  
curve for working on Hotspot.

-- John

Florian Weimer | 15 Mar 17:50

Picon

Re: precompiled headers and includeDB*

* tom fogal:

*

> This seems like a lot of trouble to go through however, and I'm not
> convinced of the performance gain.  In this scheme, compilation of
> file.C won't have to parse its include of file.h, but it still has to
> read 1 giant precompiled header file from disk.

The file is not read, but mapped into the process address space AFAIK.

Keith McGuigan | 15 Mar 17:53

Picon

Re: precompiled headers and includeDB*

Hi Tom,

tom fogal wrote:
*

> parsing a small number of includes wouldn't be that much
> slower (or maybe even faster) then just a read of one large precompiled
> header.  Unless most headers really need to get included in a
> significant portion of the JDK, which seems unlikely but possible.

It actually is the case that many, many headers get included in each source 
file.  Some are worse than others, of course, and this is aggravated by the fact 
that many headers files require other header files.

I believe that we could still get the precompiled header support without 
MakeDeps/includeDB (i.e, tradition includes), but there's a surprising amount of 
fondness for this system at this point.

--
- Keith

Ivan Krylov | 15 Mar 17:23

Picon

Re: precompiled headers and includeDB*

Hi Tom,

Using precompiled headers does help a lot to reduce hotspot compilation 
time.
The can be clearly observed if you compare with compilation time using 
Sun Studio that we use for Solaris builds.
In the mean time we continue to convince sun studio folks to improve 
precompiled header support. The way it is done now it will not help much 
hotspot compilation.

Now to you points;
Looking at http://gcc.gnu.org/onlinedocs/gcc/Precompiled-Headers.html
there is a restriction "Only one precompiled header can be used in a 
particular compilation"
So we can't have many small precompiled files

Regards,

Ivan.

tom fogal wrote:
*

>  <200703151144.45721.lists@...>Kurt Miller writes:
>> The use of precompiled headers obscures problems with
>> the includeDB* files and they have become out of date.
> [snip]
>> I could use some guidance at this point. Do the patches
>> below look correct so far?
> [snip]
> 
> While we're on the subject, is there a compelling reason for the
> includeDB system in general?  I guess I don't really understand that
> aspect of the build system very well.  It exists because of a potential
> performance gain in build times through the use of precompiled headers,
> correct?
> 
> As far as I know, the system uses these includeDB files and MakeDep to
> somehow build a single giant include file, which gets precompiled.
> Then all files end up including this precompiled header.
> 
> This seems like a lot of trouble to go through however, and I'm not
> convinced of the performance gain.  In this scheme, compilation of
> file.C won't have to parse its include of file.h, but it still has to
> read 1 giant precompiled header file from disk.  It would seem that
> reading + parsing a small number of includes wouldn't be that much
> slower (or maybe even faster) then just a read of one large precompiled
> header.  Unless most headers really need to get included in a
> significant portion of the JDK, which seems unlikely but possible.
> 
> So, making the unreasonable assumption that I understand things
> correctly, then dependencies && includes could be done in a more
> traditional way.  Performance would hopefully be similar, and that
> could deprecate `MakeDeps', making OpenJDK one step closer to not
> requiring a bootstrap JDK (I'm not sure if thats a reasonable eventual
> goal, but from posts it seems like that would be nice).
> 
> Thoughts / corrections to my logic?
> 
> -tom

--

-- 

Ivan Krylov
J2SE Hotspot Runtime Team
x(70)33368 (+7 812 3346368)

Return

Return to gmane.comp.java.openjdk.hotspot.devel.

Advertisement

Project Web Page

Technical discussion about the development of the HotSpot virtual machine

Search Archive

Language

Change language

Options

Current view: All messages / Showing whole messages / Hiding cited text.
Change to Threads only, shortened messages, or show cited text.

Post a message
NNTP Newsgroup
Classic Gmane web interface
XML

XML

RSS Feed
List Information

About Gmane