The engineering (developer) team uses this to discuss technical issues and for announcements.

Brett Wilson | 4 Jan 2009 21:12

Re: V8Bindings_prebuild slowness

On Sun, Jan 4, 2009 at 11:28 AM, Mike Belshe <mbelshe <at> google.com> wrote:
> It's been this way for a while; the good news is that it should only
> happen on initial build; subsequent builds don't regenerate the
> bindings.

The problem is that I (and it seems like many other people) have
learned that you have to do a full build basically every time you
sync, because the dependencies aren't computed properly. Ironically
the biggest problem of incorrect dependencies is the generated files!
That's why I was suggesting the solution that will help both of these
problems.

 If 3 minutes is really too much for a full build, we could
> do your #2; hopefully the dependency checker would be smart enough not
> to rebuild them if they were checked in.   We'd have to find where to
> check them in as well, because currently they are built into the
> output directory (for both Release and Debug; perhaps we could just
> generate one set, as I don't think they differ between release/debug
> builds)

I think we would need a new DEPS entry for them. Ojan's WebKit sync
script could probably do the management of this automatically.

Brett

--~--~---------~--~----~------------~-------~--~----~
Chromium Developers mailing list: chromium-dev <at> googlegroups.com 
View archives, change email options, or unsubscribe: 
    http://groups.google.com/group/chromium-dev

(Continue reading)

headers

Darin Fisher | 4 Jan 2009 22:23

Re: V8Bindings_prebuild slowness

This problem could also be solved by ignoring DerivedSources.make, and instead just add the "source" files to the vcproj. Then write a custom .rules file for each file type that runs the appropriate batch command to create the generated file. Then, dependency tracking would work just as it does for .cpp files.

Our DerivedSources.make is already so tremendously out of sync with the one upstream that there doesn't seem to be much point in using it.

-Darin

On Sun, Jan 4, 2009 at 3:12 PM, Brett Wilson <brettw <at> chromium.org> wrote:

On Sun, Jan 4, 2009 at 11:28 AM, Mike Belshe <mbelshe <at> google.com> wrote:
> It's been this way for a while; the good news is that it should only
> happen on initial build; subsequent builds don't regenerate the
> bindings.

The problem is that I (and it seems like many other people) have
learned that you have to do a full build basically every time you
sync, because the dependencies aren't computed properly. Ironically
the biggest problem of incorrect dependencies is the generated files!
That's why I was suggesting the solution that will help both of these
problems.

If 3 minutes is really too much for a full build, we could
> do your #2; hopefully the dependency checker would be smart enough not
> to rebuild them if they were checked in. We'd have to find where to
> check them in as well, because currently they are built into the
> output directory (for both Release and Debug; perhaps we could just
> generate one set, as I don't think they differ between release/debug
> builds)

I think we would need a new DEPS entry for them. Ojan's WebKit sync
script could probably do the management of this automatically.

Brett

--~--~---------~--~----~------------~-------~--~----~
Chromium Developers mailing list: chromium-dev <at> googlegroups.com
View archives, change email options, or unsubscribe:
http://groups.google.com/group/chromium-dev

-~----------~----~----~----~------~----~------~--~---

headers

Brett Wilson | 4 Jan 2009 22:45

Re: V8Bindings_prebuild slowness


On Sun, Jan 4, 2009 at 1:23 PM, Darin Fisher <darin <at> chromium.org> wrote:
> This problem could also be solved by ignoring DerivedSources.make, and
> instead just add the "source" files to the vcproj.  Then write a custom
> .rules file for each file type that runs the appropriate batch command to
> create the generated file.  Then, dependency tracking would work just as it
> does for .cpp files.

Well, but you'd still have the slowness of spawing perl hundreds of
times. I'm not sure that would speed up the build at all (though it
would improve the dependency management).

> Our DerivedSources.make is already so tremendously out of sync with the one
> upstream that there doesn't seem to be much point in using it.

I agree.

Brett

--~--~---------~--~----~------------~-------~--~----~
Chromium Developers mailing list: chromium-dev <at> googlegroups.com 
View archives, change email options, or unsubscribe: 
    http://groups.google.com/group/chromium-dev
-~----------~----~----~----~------~----~------~--~---

headers

tony | 5 Jan 2009 02:44

Re: V8Bindings_prebuild slowness

On Sun, 4 Jan 2009, Brett Wilson wrote:

> 
> On Sun, Jan 4, 2009 at 1:23 PM, Darin Fisher <darin <at> chromium.org> wrote:
> > This problem could also be solved by ignoring DerivedSources.make, and
> > instead just add the "source" files to the vcproj.  Then write a custom
> > .rules file for each file type that runs the appropriate batch command to
> > create the generated file.  Then, dependency tracking would work just as it
> > does for .cpp files.
> 
> Well, but you'd still have the slowness of spawing perl hundreds of
> times. I'm not sure that would speed up the build at all (though it
> would improve the dependency management).

But you would only pay it once.  Using native vcproj files may get the
dependencies right and maybe you wouldn't need to do a full rebuild after
each sync then.  Or maybe I put too much faith in MSVC's dependency
management.

> > Our DerivedSources.make is already so tremendously out of sync with the one
> > upstream that there doesn't seem to be much point in using it.
> 
> I agree.

It's not used in the scons build which properly tracks dependencies so
the files are only generated once.  Would Incredibuild be able to
parallelize the perl scripts or does it only know how to parallelize c++
compiles?

--~--~---------~--~----~------------~-------~--~----~
Chromium Developers mailing list: chromium-dev <at> googlegroups.com 
View archives, change email options, or unsubscribe: 
    http://groups.google.com/group/chromium-dev
-~----------~----~----~----~------~----~------~--~---

headers

Darin Fisher | 10 Jan 2009 00:37

Re: V8Bindings_prebuild slowness

On Sun, Jan 4, 2009 at 5:44 PM, <tony <at> chromium.org> wrote:

On Sun, 4 Jan 2009, Brett Wilson wrote:

>
> On Sun, Jan 4, 2009 at 1:23 PM, Darin Fisher <darin <at> chromium.org> wrote:
> > This problem could also be solved by ignoring DerivedSources.make, and
> > instead just add the "source" files to the vcproj. Then write a custom
> > .rules file for each file type that runs the appropriate batch command to
> > create the generated file. Then, dependency tracking would work just as it
> > does for .cpp files.
>
> Well, but you'd still have the slowness of spawing perl hundreds of
> times. I'm not sure that would speed up the build at all (though it
> would improve the dependency management).

But you would only pay it once. Using native vcproj files may get the
dependencies right and maybe you wouldn't need to do a full rebuild after
each sync then. Or maybe I put too much faith in MSVC's dependency
management.

Right. This is what I was thinking too. I do clobber builds primarily (only?) because of issues related to DerivedSources.

> > Our DerivedSources.make is already so tremendously out of sync with the one
> > upstream that there doesn't seem to be much point in using it.
>
> I agree.

It's not used in the scons build which properly tracks dependencies so
the files are only generated once. Would Incredibuild be able to
parallelize the perl scripts or does it only know how to parallelize c++
compiles?

That's a great question :)

I suspect it only runs those locally.

-Darin

headers

Pam Greene | 10 Jan 2009 01:06

Re: V8Bindings_prebuild slowness

On Fri, Jan 9, 2009 at 3:37 PM, Darin Fisher <darin <at> chromium.org> wrote:

On Sun, Jan 4, 2009 at 5:44 PM, <tony <at> chromium.org> wrote:

On Sun, 4 Jan 2009, Brett Wilson wrote:

>
> On Sun, Jan 4, 2009 at 1:23 PM, Darin Fisher <darin <at> chromium.org> wrote:
> > This problem could also be solved by ignoring DerivedSources.make, and
> > instead just add the "source" files to the vcproj. Then write a custom
> > .rules file for each file type that runs the appropriate batch command to
> > create the generated file. Then, dependency tracking would work just as it
> > does for .cpp files.
>
> Well, but you'd still have the slowness of spawing perl hundreds of
> times. I'm not sure that would speed up the build at all (though it
> would improve the dependency management).

But you would only pay it once. Using native vcproj files may get the
dependencies right and maybe you wouldn't need to do a full rebuild after
each sync then. Or maybe I put too much faith in MSVC's dependency
management.

Right. This is what I was thinking too. I do clobber builds primarily (only?) because of issues related to DerivedSources.

> > Our DerivedSources.make is already so tremendously out of sync with the one
> > upstream that there doesn't seem to be much point in using it.
>
> I agree.

It's not used in the scons build which properly tracks dependencies so
the files are only generated once. Would Incredibuild be able to
parallelize the perl scripts or does it only know how to parallelize c++
compiles?

That's a great question :)

I suspect it only runs those locally.

Without an additional extension license, only locally. But if it only had to do it once, that'd still be a big help. Alternatively, if we find we still need to speed clobber builds and are willing to move away from DerivedSources.make, I'd expect that only launching perl once rather than 200+ times -- just move the "do this for every file in this long list" from make into perl -- would provide a dramatic speedup.

- Pam

-Darin

headers

Eric Seidel | 10 Jan 2009 00:51

Re: V8Bindings_prebuild slowness


There have long been talks of re-writing the idl bindings auto-gen
system, it's just not been done yet.  I'm sure WebKit folks would be
happy for any work in the area.

-eric

On Fri, Jan 9, 2009 at 3:37 PM, Darin Fisher <darin <at> chromium.org> wrote:
>
>
> On Sun, Jan 4, 2009 at 5:44 PM, <tony <at> chromium.org> wrote:
>>
>> On Sun, 4 Jan 2009, Brett Wilson wrote:
>>
>> >
>> > On Sun, Jan 4, 2009 at 1:23 PM, Darin Fisher <darin <at> chromium.org> wrote:
>> > > This problem could also be solved by ignoring DerivedSources.make, and
>> > > instead just add the "source" files to the vcproj.  Then write a
>> > > custom
>> > > .rules file for each file type that runs the appropriate batch command
>> > > to
>> > > create the generated file.  Then, dependency tracking would work just
>> > > as it
>> > > does for .cpp files.
>> >
>> > Well, but you'd still have the slowness of spawing perl hundreds of
>> > times. I'm not sure that would speed up the build at all (though it
>> > would improve the dependency management).
>>
>> But you would only pay it once.  Using native vcproj files may get the
>> dependencies right and maybe you wouldn't need to do a full rebuild after
>> each sync then.  Or maybe I put too much faith in MSVC's dependency
>> management.
>
> Right.  This is what I was thinking too.  I do clobber builds primarily
> (only?) because of issues related to DerivedSources.
>
>>
>> > > Our DerivedSources.make is already so tremendously out of sync with
>> > > the one
>> > > upstream that there doesn't seem to be much point in using it.
>> >
>> > I agree.
>>
>> It's not used in the scons build which properly tracks dependencies so
>> the files are only generated once.  Would Incredibuild be able to
>> parallelize the perl scripts or does it only know how to parallelize c++
>> compiles?
>
> That's a great question :)
> I suspect it only runs those locally.
> -Darin
>
>
> >
>

--~--~---------~--~----~------------~-------~--~----~
Chromium Developers mailing list: chromium-dev <at> googlegroups.com 
View archives, change email options, or unsubscribe: 
    http://groups.google.com/group/chromium-dev
-~----------~----~----~----~------~----~------~--~---

headers

Amanda Walker | 4 Jan 2009 23:38

Re: V8Bindings_prebuild slowness

On Sun, Jan 4, 2009 at 4:45 PM, Brett Wilson <brettw <at> chromium.org> wrote:
> Well, but you'd still have the slowness of spawing perl hundreds of
> times. I'm not sure that would speed up the build at all (though it
> would improve the dependency management).

Perhaps we've outgrown perl .  I haven't looked at what the perl
code does (I've been willfully ignoring v8, since it has been "just
working" ), but is it something that a chunk of C++ linked against
PCRE and knows how to iterate directories and stat file times could
do?

--Amanda

--~--~---------~--~----~------------~-------~--~----~
Chromium Developers mailing list: chromium-dev <at> googlegroups.com 
View archives, change email options, or unsubscribe: 
    http://groups.google.com/group/chromium-dev
-~----------~----~----~----~------~----~------~--~---

headers

Dean McNamee | 4 Jan 2009 23:40

Re: V8Bindings_prebuild slowness


The perl code is the upstream webkit's IDL compiler (but heavily
modified for v8).  It reads in the IDL files and generates the .h /
.cpp files that are the actual binding code.  It would be nice to
rewrite this, but it would be a lot of work and also would need
coordination upstream.

On Sun, Jan 4, 2009 at 11:38 PM, Amanda Walker <amanda <at> chromium.org> wrote:
>
> On Sun, Jan 4, 2009 at 4:45 PM, Brett Wilson <brettw <at> chromium.org> wrote:
>> Well, but you'd still have the slowness of spawing perl hundreds of
>> times. I'm not sure that would speed up the build at all (though it
>> would improve the dependency management).
>
> Perhaps we've outgrown perl .  I haven't looked at what the perl
> code does (I've been willfully ignoring v8, since it has been "just
> working" ), but is it something that a chunk of C++ linked against
> PCRE and knows how to iterate directories and stat file times could
> do?
>
> --Amanda
>
> >
>

--~--~---------~--~----~------------~-------~--~----~
Chromium Developers mailing list: chromium-dev <at> googlegroups.com 
View archives, change email options, or unsubscribe: 
    http://groups.google.com/group/chromium-dev
-~----------~----~----~----~------~----~------~--~---

headers

Amanda Walker | 5 Jan 2009 01:36

Re: V8Bindings_prebuild slowness

On Sun, Jan 4, 2009 at 5:40 PM, Dean McNamee <deanm <at> chromium.org> wrote:
> The perl code is the upstream webkit's IDL compiler (but heavily
> modified for v8).  It reads in the IDL files and generates the .h /
> .cpp files that are the actual binding code.  It would be nice to
> rewrite this, but it would be a lot of work and also would need
> coordination upstream.

OK, makes sense.  There would certainly be a certain amount of
up-front work (and as I said, I haven't looked at it, so I don't know
how much work that might be).  Though if we're maintaining our own
heavily modified version, it sounds like we're already paying a
"having to coordinate our version with upstream webkit" price.

I've ended up in the habit of doing clobber builds fairly often in
Windows, thanks to the broken dependency tracking (and the fact that I
only update my windows tree once or twice a week), so I share some of
Brett's frustration (though my time ratios aren't as big, since we
don't have the benefit of incredibuild out here).

--Amanda

--~--~---------~--~----~------------~-------~--~----~
Chromium Developers mailing list: chromium-dev <at> googlegroups.com 
View archives, change email options, or unsubscribe: 
    http://groups.google.com/group/chromium-dev
-~----------~----~----~----~------~----~------~--~---

headers

Dean McNamee | 4 Jan 2009 23:39

Re: V8Bindings_prebuild slowness


I have wanted to check in the generated binding source for a long time
(there is a bug on it).  The benefit of checking it in is then it's
visible from the source server, now you won't get the bindings source
when source debugging w/ the symbol / source server.

b/1346719

-- dean

On Sun, Jan 4, 2009 at 11:38 PM, Amanda Walker <amanda <at> chromium.org> wrote:
>
> On Sun, Jan 4, 2009 at 4:45 PM, Brett Wilson <brettw <at> chromium.org> wrote:
>> Well, but you'd still have the slowness of spawing perl hundreds of
>> times. I'm not sure that would speed up the build at all (though it
>> would improve the dependency management).
>
> Perhaps we've outgrown perl .  I haven't looked at what the perl
> code does (I've been willfully ignoring v8, since it has been "just
> working" ), but is it something that a chunk of C++ linked against
> PCRE and knows how to iterate directories and stat file times could
> do?
>
> --Amanda
>
> >
>

--~--~---------~--~----~------------~-------~--~----~
Chromium Developers mailing list: chromium-dev <at> googlegroups.com 
View archives, change email options, or unsubscribe: 
    http://groups.google.com/group/chromium-dev
-~----------~----~----~----~------~----~------~--~---