Rob Landley | 2 Jan 09:07 2009
Picon

PATCH [0/3]: Simplify the kernel build by removing perl.

Before 2.6.25 (specifically git bdc807871d58285737d50dc6163d0feb72cb0dc2 ) 
building a Linux kernel never required perl to be installed on the build 
system.  (Various development and debugging scripts were written in perl and 
python and such, but they weren't involved in actually building a kernel.)  
Building a kernel before 2.6.25 could be done with a minimal system built from 
gcc, binutils, bash, make, busybox, uClibc, and the Linux kernel, and nothing 
else.  (Embedded developers creating clean cross compile environments that 
won't leak bits of the host system into the target, or bootstrapping native 
development environments to run on target hardware or under emulators, tend to 
care about this sort of thing.)

The perl checkin for 2.6.25 was the camel's nose under the tent flap, and 
since then two more instances of perl have shown up in the core kernel build.  
This patch series removes the three required to build a basic kernel for qemu 
for the targets I've tested (arm, mips, powerpc, sparc, m68k, sh4, and of 
course x86 and x86-64), replacing them with shell scripts.

Historically the kernel has gone out of its way to minimize environmental 
dependencies for the build.  For example, the plethora of *_shipped files mean 
we don't need lex and yacc to build the kernel.  The kconfig infrastructure 
once required curses to run "make oldconfig", but that requirement was 
restricted to just menuconfig so the kernel could build on systems without 
ncurses.

That was a very nice policy.  Kernel development already requires an in-depth 
knowledge of C, make, and shell, plus things like the kconfig language.  A 
similarly in-depth knowledge of perl is bigger than all that combined (even 
Larry Wall probably doesn't know all of Perl anymore), and then what's the 
excuse _not_ to add Python to the core build?  And after that why not java, or 
lua?  Where does it end?  What's the criteria to say "no" here?
(Continue reading)

Rob Landley | 2 Jan 09:13 2009
Picon

[PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh

From: Rob Landley <rob <at> landley.net>

Replace kernel/timeconst.pl with kernel/timeconst.sh.  The new shell script
is much simpler, about 1/4 the size, and runs on Red Hat 9 from 2003.

Peter Anvin added this perl to 2.6.25.  Before that, the kernel had never
required perl to build.

Signed-off-by: Rob Landley <rob <at> landley.net>
---

 kernel/Makefile     |    4 
 kernel/timeconst.pl |  378 ------------------------------------------
 kernel/timeconst.sh |   91 ++++++++++
 3 files changed, 93 insertions(+), 380 deletions(-)

diff -r d0c7611dcfd6 kernel/Makefile
--- a/kernel/Makefile	Tue Dec 30 17:48:25 2008 -0800
+++ b/kernel/Makefile	Fri Jan 02 00:10:44 2009 -0600
 <at>  <at>  -115,7 +115,7  <at>  <at> 
 $(obj)/time.o: $(obj)/timeconst.h

 quiet_cmd_timeconst  = TIMEC   $ <at> 
-      cmd_timeconst  = $(PERL) $< $(CONFIG_HZ) > $ <at> 
+      cmd_timeconst  = $(CONFIG_SHELL) $< $(CONFIG_HZ) > $ <at> 
 targets += timeconst.h
-$(obj)/timeconst.h: $(src)/timeconst.pl FORCE
+$(obj)/timeconst.h: $(src)/timeconst.sh FORCE
 	$(call if_changed,timeconst)
--- /dev/null	1969-12-31 00:00:00.000000000 -0600
(Continue reading)

Sam Ravnborg | 2 Jan 10:04 2009

Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh

Hi Rob.

On Fri, Jan 02, 2009 at 02:13:30AM -0600, Rob Landley wrote:
> From: Rob Landley <rob <at> landley.net>
> 
> Replace kernel/timeconst.pl with kernel/timeconst.sh.  The new shell script
> is much simpler, about 1/4 the size, and runs on Red Hat 9 from 2003.

This part of the changelog is OK except that is fails to
address why we want to get away from perl.

Please drop the remining part of the changelog (but not the s-o-b).

> Peter Anvin added this perl to 2.6.25.  Before that, the kernel had never
> required perl to build.
> 
> Signed-off-by: Rob Landley <rob <at> landley.net>
> ---
> 
>  kernel/Makefile     |    4 
>  kernel/timeconst.pl |  378 ------------------------------------------
>  kernel/timeconst.sh |   91 ++++++++++
>  3 files changed, 93 insertions(+), 380 deletions(-)
> 
> diff -r d0c7611dcfd6 kernel/Makefile
> --- a/kernel/Makefile	Tue Dec 30 17:48:25 2008 -0800
> +++ b/kernel/Makefile	Fri Jan 02 00:10:44 2009 -0600
>  <at>  <at>  -115,7 +115,7  <at>  <at> 
>  $(obj)/time.o: $(obj)/timeconst.h
>  
(Continue reading)

Rob Landley | 2 Jan 13:00 2009
Picon

Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh

On Friday 02 January 2009 03:04:39 Sam Ravnborg wrote:
> Hi Rob.
>
> On Fri, Jan 02, 2009 at 02:13:30AM -0600, Rob Landley wrote:
> > From: Rob Landley <rob <at> landley.net>
> >
> > Replace kernel/timeconst.pl with kernel/timeconst.sh.  The new shell
> > script is much simpler, about 1/4 the size, and runs on Red Hat 9 from
> > 2003.
>
> This part of the changelog is OK except that is fails to
> address why we want to get away from perl.

You mean "The new shell script is much simpler, about 1/4 the size, runs on 
Red Hat 9 from 2003, and isn't perl?" :)

> Please drop the remining part of the changelog (but not the s-o-b).

ok.

> > Peter Anvin added this perl to 2.6.25.  Before that, the kernel had never
> > required perl to build.
> >
> > Signed-off-by: Rob Landley <rob <at> landley.net>
> > ---
> >
> >  kernel/Makefile     |    4
> >  kernel/timeconst.pl |  378 ------------------------------------------
> >  kernel/timeconst.sh |   91 ++++++++++
> >  3 files changed, 93 insertions(+), 380 deletions(-)
(Continue reading)

H. Peter Anvin | 2 Jan 20:33 2009

Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh

Rob Landley wrote:
> 
> You mean "The new shell script is much simpler, about 1/4 the size, runs on 
> Red Hat 9 from 2003, and isn't perl?" :)
> 

And introduces unclear environment dependencies depending on how
external utilities are implemented.

The whole point of why that script was written in Perl was to have
access to arbitrary-precision arithmetic -- after it was shown that bc
would simply lock up on some systems.

	-hpa

--

-- 
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.

Rob Landley | 4 Jan 02:32 2009
Picon

Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh

On Friday 02 January 2009 13:33:02 H. Peter Anvin wrote:
> Rob Landley wrote:
> > You mean "The new shell script is much simpler, about 1/4 the size, runs
> > on Red Hat 9 from 2003, and isn't perl?" :)
>
> And introduces unclear environment dependencies depending on how
> external utilities are implemented.

I note that sed and printf and such are all susv3.  I have an explicit test 
for 32 bit math in the script that cares, and this worked in Red Hat 9 circa 
2003.

I consider this a step up from code with an implicit dependency on a CPAN 
library.

> The whole point of why that script was written in Perl was to have
> access to arbitrary-precision arithmetic -- after it was shown that bc
> would simply lock up on some systems.

A) I'm not using bc.

B) You don't need arbitrary precision arithmetic, you need around 72 bits 
worth of arithmetic for the pathological case.

C) Your definition of "access to arbitrary-precision arithmetic" includes the 
following, cut and paste verbatim from your old script:

# Precomputed values for systems without Math::BigInt
# Generated by:
# timeconst.pl --can 24 32 48 64 100 122 128 200 250 256 300 512 1000 1024 
(Continue reading)

H. Peter Anvin | 4 Jan 02:35 2009

Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh

Rob Landley wrote:
> 
> I consider this a step up from code with an implicit dependency on a CPAN 
> library.
> 

There is no CPAN library in use.  Math::BigInt is a standard part of
Perl, and the canned values is there only to support extremely old
versions of Perl, or weird system configurations, as a
belt-and-suspenders measure.

	-hpa
Alan Cox | 4 Jan 13:07 2009
Picon

Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh

> I note that sed and printf and such are all susv3.  I have an explicit test 
> for 32 bit math in the script that cares, and this worked in Red Hat 9 circa 
> 2003.

If you are trying to do arbitary precision maths using standard posix
tools just use dc. That way the standard is explicit about what you will
get.
H. Peter Anvin | 4 Jan 19:36 2009

Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh

Alan Cox wrote:
>> I note that sed and printf and such are all susv3.  I have an explicit test 
>> for 32 bit math in the script that cares, and this worked in Red Hat 9 circa 
>> 2003.
> 
> If you are trying to do arbitary precision maths using standard posix
> tools just use dc. That way the standard is explicit about what you will
> get.

The original patch used bc.  Unfortunately, it ran into bc bugs -- on
some platforms like some of akpm's older test machines it would just hang.

	-hpa

--

-- 
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.

Rob Landley | 4 Jan 20:03 2009
Picon

Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh

On Sunday 04 January 2009 06:07:35 Alan Cox wrote:
> > I note that sed and printf and such are all susv3.  I have an explicit
> > test for 32 bit math in the script that cares, and this worked in Red Hat
> > 9 circa 2003.
>
> If you are trying to do arbitary precision maths using standard posix
> tools just use dc. That way the standard is explicit about what you will
> get.

I looked at that, but:

A) the Open Group Base Specifications (which I normally go by, since they're 
roughly synonymous with SUSv3 and Posix and available free on the web; they 
just released version 7 a week or so back) don't list dc as one of their 
utilities.  (They mention bc, but not dc.)

B) The busybox implementation of dc is crap.  I got 'em to fix the bug where 
the output defaulted to binary instead of decimal, but the math is all done as 
floating point rather than arbitrary precision, and they don't implement the 
<< operator.

C) The only calculation which can overflow 64 bits (the ADJ32 one) turns out 
not to need arbitrary precision math, just 72 bits, and if it ever uses more 
than 32 then bottom 32 are all zero before the divide so you can do it in 
three lines.

Essentially, the ADJ32 calculation is "(($NUMBER-1)<<$SHIFT)/$NUMBER".

$SHIFT maxes out at 51 and the largest interesting $NUMBER is 1000000.  
(That's the pathological case of HZ=1, calculating the USEC_TO_HZ direction.  
(Continue reading)

H. Peter Anvin | 4 Jan 21:39 2009

Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh

Rob Landley wrote:
> 
> C) The only calculation which can overflow 64 bits (the ADJ32 one) turns out 
> not to need arbitrary precision math, just 72 bits, and if it ever uses more 
> than 32 then bottom 32 are all zero before the divide so you can do it in 
> three lines.
> 

... for the current code (32 bits).  When we get an overflow-less 64-bit
implementation, this code will have to be redone, which is not true for
a properly done implementation.

	-hpa

--

-- 
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.

Rob Landley | 5 Jan 01:59 2009
Picon

Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh

On Sunday 04 January 2009 14:39:36 H. Peter Anvin wrote:
> Rob Landley wrote:
> > C) The only calculation which can overflow 64 bits (the ADJ32 one) turns
> > out not to need arbitrary precision math, just 72 bits, and if it ever
> > uses more than 32 then bottom 32 are all zero before the divide so you
> > can do it in three lines.
>
> ... for the current code (32 bits).  When we get an overflow-less 64-bit
> implementation, this code will have to be redone, which is not true for
> a properly done implementation.

One extra mask and add is a strange definition of "redone", but I can add it 
now if you like.  (I'd personally prefer to wait for something to actually 
need it, but...)

> 	-hpa

Rob
Harvey Harrison | 3 Jan 07:28 2009
Picon

Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh

On Fri, 2009-01-02 at 06:00 -0600, Rob Landley wrote:
> On Friday 02 January 2009 03:04:39 Sam Ravnborg wrote:
> > > +# Output start of header file
> > > +
> > > +cat << EOF
> > > +/* Automatically generated by kernel/timeconst.sh */
> > > +/* Conversion constants for HZ == $HZ */
> > > +
> > > +#ifndef KERNEL_TIMECONST_H
> > > +#define KERNEL_TIMECONST_H
> >
> > Please use __KERNEL_TIMECONST_H. The two underscores are standard.
> 
> Sure thing.  (I was just copying the perl there, I'll post an updated patch 
> after I get some sleep.)

In the x86 discussions recently regarding header guards, I though a single
underscore was eventually decided on as 'standard'.

Harvey

Ingo Oeser | 3 Jan 13:28 2009
Picon

Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh

Hi Robert,

I fully support your argument here: Requiring lots of hacks in various languages
to build a system core sucks. But now sth. more productive:

On Friday 02 January 2009, Rob Landley wrote:
> diff -r d0c7611dcfd6 kernel/Makefile
> --- a/kernel/Makefile	Tue Dec 30 17:48:25 2008 -0800
> +++ b/kernel/Makefile	Fri Jan 02 00:10:44 2009 -0600
>  <at>  <at>  -115,7 +115,7  <at>  <at> 
>  $(obj)/time.o: $(obj)/timeconst.h
>  
>  quiet_cmd_timeconst  = TIMEC   $ <at> 
> -      cmd_timeconst  = $(PERL) $< $(CONFIG_HZ) > $ <at> 
> +      cmd_timeconst  = $(CONFIG_SHELL) $< $(CONFIG_HZ) > $ <at> 
>  targets += timeconst.h
> -$(obj)/timeconst.h: $(src)/timeconst.pl FORCE
> +$(obj)/timeconst.h: $(src)/timeconst.sh FORCE
>  	$(call if_changed,timeconst)
> --- /dev/null	1969-12-31 00:00:00.000000000 -0600
> +++ hg/kernel/timeconst.sh	2009-01-01 23:53:17.000000000 -0600
>  <at>  <at>  -0,0 +1,91  <at>  <at> 
> +#!/bin/bash
> +
> +if [ $# -ne 1 ]
> +then
> +	echo "Usage: timeconst.sh HZ"
> +	exit 1
> +fi
> +
(Continue reading)

Rob Landley | 4 Jan 02:36 2009
Picon

Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh

On Saturday 03 January 2009 06:28:22 Ingo Oeser wrote:
> > +for i in "MSEC 1000" "USEC 1000000"
> > +do
> > +	NAME=$(echo $i | awk '{print $1}')
>
> cut -d' ' -f1  does the same
>
> > +	PERIOD=$(echo $i | awk '{print $2}')
>
> cut -d' ' -f2  does the same

From a standards perspective 
http://www.opengroup.org/onlinepubs/9699919799/utilities/cut.html vs 
http://www.opengroup.org/onlinepubs/9699919799/utilities/awk.html is probably 
a wash, but from a simplicity perspective using the tool that _isn't_ its own 
programming language is probably a win. :)

I made the change in the second round of patches I just posted.

Thanks,

Rob
Valdis.Kletnieks | 4 Jan 06:07 2009
Picon

Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh

On Sat, 03 Jan 2009 19:36:04 CST, Rob Landley said:
> On Saturday 03 January 2009 06:28:22 Ingo Oeser wrote:
> > > +for i in "MSEC 1000" "USEC 1000000"
> > > +do
> > > +	NAME=$(echo $i | awk '{print $1}')
> >
> > cut -d' ' -f1  does the same
> >
> > > +	PERIOD=$(echo $i | awk '{print $2}')
> >
> > cut -d' ' -f2  does the same

Close, but no cee-gar.  cut does something counter-intuitive with multiple
blanks:

% echo 'a    b' | awk '{print $2}'
b
% echo 'a    b' | cut -d' ' -f2

% echo 'a    b' | sed -r 's/[ ]+/ /g' | cut -d' ' -f2
b

Unfortunately, 'sed -r' isn't in the opengroup.org list of required options,
and sed 's/  / /g' doesn't DTRT for 3 or more blanks (as it won't recursively
apply the change to a *new* double blank formed by the previous change).

Rob Landley | 4 Jan 07:43 2009
Picon

Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh

On Saturday 03 January 2009 23:07:55 Valdis.Kletnieks <at> vt.edu wrote:
> On Sat, 03 Jan 2009 19:36:04 CST, Rob Landley said:
> > On Saturday 03 January 2009 06:28:22 Ingo Oeser wrote:
> > > > +for i in "MSEC 1000" "USEC 1000000"
> > > > +do
> > > > +	NAME=$(echo $i | awk '{print $1}')
> > >
> > > cut -d' ' -f1  does the same
> > >
> > > > +	PERIOD=$(echo $i | awk '{print $2}')
> > >
> > > cut -d' ' -f2  does the same
>
> Close, but no cee-gar.  cut does something counter-intuitive with multiple
> blanks:

Yes, but in this case I'm the one passing in the data so I know there aren't 
multiple blanks.  (Or tabs instead of spaces.)

In a private email, Bernd Petrovitsch suggested "set -- $i" and then using 
NAME=$1; PERIOD=$2.  (I keep getting private email responses to these sort of 
threads, and then getting dismissed as the only one who cares about the issue.  
Less so this time around, but still...)  This apparently works all the way 
back to the bourne shell.  Not worth rolling another patch for, but if I do 
wind up rolling another patch for other reasons I might switch over to that.  
Both "cut" and "awk" are susv3, though.

Rob
Jamie Lokier | 4 Jan 23:13 2009

Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh

Rob Landley wrote:
> In a private email, Bernd Petrovitsch suggested "set -- $i" and then
> using NAME=$1; PERIOD=$2.  (I keep getting private email responses
> to these sort of threads, and then getting dismissed as the only one
> who cares about the issue.  Less so this time around, but still...)
> This apparently works all the way back to the bourne shell.

If you're going "all the way back to the bourne shell", don't use "set
-- $i"; use "set x $i" instead, and don't expect to do any arithmetic
in the shell; use "expr" or "awk" for arithmetic.

(Not relevant to kernel scripts, imho, since you can always assume
something a bit more modern and not too stripped down).

(I have 850 Linux boxes on my network with a bourne shell which
doesn't do $((...)).  I won't be building kernels on them though :-)

-- Jamie
Bernd Petrovitsch | 5 Jan 01:15 2009
Picon

Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh

On Son, 2009-01-04 at 22:13 +0000, Jamie Lokier wrote:
> Rob Landley wrote:
> > In a private email, Bernd Petrovitsch suggested "set -- $i" and then
> > using NAME=$1; PERIOD=$2.  (I keep getting private email responses
> > to these sort of threads, and then getting dismissed as the only one
> > who cares about the issue.  Less so this time around, but still...)
> > This apparently works all the way back to the bourne shell.
> 
> If you're going "all the way back to the bourne shell", don't use "set

"Going back to a Bourne shell" was neither the intention nor makes it
sense IMHO.
I mentioned it to point out that the `set -- ' (or `set x `) is nothing
new or even a bash-ism.

> -- $i"; use "set x $i" instead, and don't expect to do any arithmetic
> in the shell; use "expr" or "awk" for arithmetic.
> 
> (Not relevant to kernel scripts, imho, since you can always assume
> something a bit more modern and not too stripped down).

ACK. A bash can IMHO be expected. Even going for `dash` is IMHO somewhat
too extreme.

> (I have 850 Linux boxes on my network with a bourne shell which
> doesn't do $((...)).  I won't be building kernels on them though :-)

Believe it or not, but there are folks out there who build the firmware
on ARM 200 MHz NFS-mounted systems natively  (and not simply
cross-compile it on a 2GHz PC .....).
(Continue reading)

Jamie Lokier | 5 Jan 03:23 2009

Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh

Bernd Petrovitsch wrote:
> > (I have 850 Linux boxes on my network with a bourne shell which
> > doesn't do $((...)).  I won't be building kernels on them though :-)
> 
> Believe it or not, but there are folks out there who build the firmware
> on ARM 200 MHz NFS-mounted systems natively  (and not simply
> cross-compile it on a 2GHz PC .....).

Really?

My 850 Linux boxes are 166MHz ARMs and occasionally NFS-mounted.
Their /bin/sh does not do $((...)), and Bash is not there at all.

If I were installing GCC natively on them, I'd install GNU Make and a
proper shell while I were at it.  But I don't know if Bash works
properly without fork()* - or even if GCC does :-)

Perl might be hard, as shared libraries aren't supported by the
toolchain which targets my ARMs* and Perl likes its loadable modules.

I'm not sure why I would want to build a kernel on these devices.

But I see why people with mobile ARM devices like gphones might
want to, when they're out travelling.

-- Jamie

(* - No MMU on some ARMs, but I'm working on ARM FDPIC-ELF to add
     proper shared libs.  Feel free to fund this :-)
(Continue reading)

Bernd Petrovitsch | 5 Jan 11:46 2009
Picon

Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh

On Mon, 2009-01-05 at 02:23 +0000, Jamie Lokier wrote:
> Bernd Petrovitsch wrote:
> > > (I have 850 Linux boxes on my network with a bourne shell which
> > > doesn't do $((...)).  I won't be building kernels on them though :-)
> > 
> > Believe it or not, but there are folks out there who build the firmware
> > on ARM 200 MHz NFS-mounted systems natively  (and not simply
> > cross-compile it on a 2GHz PC .....).
> 
> Really?
> 
> My 850 Linux boxes are 166MHz ARMs and occasionally NFS-mounted.
> Their /bin/sh does not do $((...)), and Bash is not there at all.

I assume that the NFS-mounted root filesystem is a real distribution.
And on the local flash is a usual busybox based firmware.

> If I were installing GCC natively on them, I'd install GNU Make and a
> proper shell while I were at it.  But I don't know if Bash works

ACK.

> properly without fork()* - or even if GCC does :-)
> 
> Perl might be hard, as shared libraries aren't supported by the
> toolchain which targets my ARMs* and Perl likes its loadable modules.

The simplest way to go is probably to use CentOS or Debian or another
ready binary distribution on ARM (or MIPS or PPC or whatever core the
embedded system has) possibly on a custom build kernel (if necessary).
(Continue reading)

Jamie Lokier | 5 Jan 16:01 2009

Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh

Bernd Petrovitsch wrote:
> I assume that the NFS-mounted root filesystem is a real distribution.

Not unless you call uClinux (MMU-less) a real distribution, no.

> > (* - No MMU on some ARMs, but I'm working on ARM FDPIC-ELF to add
> >      proper shared libs.  Feel free to fund this :-)
> 
> The above mentioned ARMs have a MMU. Without MMU, it would be truly
> insane IMHO.

We have similar cross-build issues without MMUs... I.e. that a lot of
useful packages don't cross-build properly (including many which use
Autoconf), and it might be easier to make a native build environment
than to debug and patch all the broken-for-cross-build packages.
Especially as sometimes they build, but fail at run-time in some
conditions.

But you're right it's probably insane to try.  I haven't dared as I
suspect GCC and/or Binutils would break too :-)

I'm sticking instead with "oh well cross-build a few packages by hand
and just don't even _try_ to use most of the handy software out there".

You mentioned ARM Debian.  According to
http://wiki.debian.org/ArmEabiPort one recommended method of
bootstrapping it is building natively on an emulated ARM, because
cross-building is fragile.

-- Jamie
(Continue reading)

Bernd Petrovitsch | 5 Jan 17:18 2009
Picon

Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh

On Mon, 2009-01-05 at 15:01 +0000, Jamie Lokier wrote:
> Bernd Petrovitsch wrote:
> > I assume that the NFS-mounted root filesystem is a real distribution.
> 
> Not unless you call uClinux (MMU-less) a real distribution, no.

Not really.

> > > (* - No MMU on some ARMs, but I'm working on ARM FDPIC-ELF to add
> > >      proper shared libs.  Feel free to fund this :-)
> > 
> > The above mentioned ARMs have a MMU. Without MMU, it would be truly
> > insane IMHO.
> 
> We have similar cross-build issues without MMUs... I.e. that a lot of

Of course.

> useful packages don't cross-build properly (including many which use
> Autoconf), and it might be easier to make a native build environment

Tell me about it - AC_TRY_RUN() is the culprit.
And `pkg-config` supports cross-compilation only since 18 months or so.
Before one had to rewrite the generated .pc files.

[...]
> You mentioned ARM Debian.  According to
> http://wiki.debian.org/ArmEabiPort one recommended method of
> bootstrapping it is building natively on an emulated ARM, because
> cross-building is fragile.
(Continue reading)

Rob Landley | 6 Jan 01:06 2009
Picon

Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh

On Monday 05 January 2009 09:01:56 Jamie Lokier wrote:
> Bernd Petrovitsch wrote:
> > I assume that the NFS-mounted root filesystem is a real distribution.
>
> Not unless you call uClinux (MMU-less) a real distribution, no.

I want things to be orthogonal.  The following should be completely separate 
steps:

1) Creating a cross compiler
2) building a native development environment
3) booting a native development environment (on real hardware or under and 
emulator)
4) natively building your target system.

You should be able to mix and match.  Crosstool for #1, go download "fedora 
for arm" instead of #2, qemu or real hardware is your choice for #3, and then 
you should be able to natively build gentoo under an ubuntu host or vice 
versa.  (How is not currently properly documented, but I'm working on that.)

My objection to build systems like buildroot or uClinux is that they bundle 
all this together into a big hairball.  They create their own cross compiler, 
build their own root file system, use their own packaging system, and you have 
to take it all or nothing.

My build system is ruthlessly orthogonal.  I try not to make it depend on 
other bits of _itself_ more than necessary.

> > > (* - No MMU on some ARMs, but I'm working on ARM FDPIC-ELF to add
> > >      proper shared libs.  Feel free to fund this :-)
(Continue reading)

Rob Landley | 5 Jan 22:07 2009
Picon

Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh

On Monday 05 January 2009 04:46:18 Bernd Petrovitsch wrote:
> > My 850 Linux boxes are 166MHz ARMs and occasionally NFS-mounted.
> > Their /bin/sh does not do $((...)), and Bash is not there at all.
>
> I assume that the NFS-mounted root filesystem is a real distribution.
> And on the local flash is a usual busybox based firmware.

Building on an nfs mount is evil.  Make cares greatly about timestamp 
accuracy, and NFS's dentry cacheing doesn't really, especially when it 
discards cached copies and re-fetches them, and the server and client's clocks 
are a half-second off from each other.

Sometimes you haven't got a choice, but I hate having to debug the build 
problems this intermittently causes.  If you never do anything except "make 
all" it should suck less.

> > If I were installing GCC natively on them, I'd install GNU Make and a
> > proper shell while I were at it.  But I don't know if Bash works
>
> ACK.
>
> > properly without fork()* - or even if GCC does :-)
> >
> > Perl might be hard, as shared libraries aren't supported by the
> > toolchain which targets my ARMs* and Perl likes its loadable modules.
>
> The simplest way to go is probably to use CentOS or Debian or another
> ready binary distribution on ARM (or MIPS or PPC or whatever core the
> embedded system has) possibly on a custom build kernel (if necessary).

(Continue reading)

Rob Landley | 5 Jan 05:50 2009
Picon

Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh

On Sunday 04 January 2009 18:15:30 Bernd Petrovitsch wrote:
> On Son, 2009-01-04 at 22:13 +0000, Jamie Lokier wrote:
> > Rob Landley wrote:
> > > In a private email, Bernd Petrovitsch suggested "set -- $i" and then
> > > using NAME=$1; PERIOD=$2.  (I keep getting private email responses
> > > to these sort of threads, and then getting dismissed as the only one
> > > who cares about the issue.  Less so this time around, but still...)
> > > This apparently works all the way back to the bourne shell.
> >
> > If you're going "all the way back to the bourne shell", don't use "set
>
> "Going back to a Bourne shell" was neither the intention nor makes it
> sense IMHO.
> I mentioned it to point out that the `set -- ' (or `set x `) is nothing
> new or even a bash-ism.
>
> > -- $i"; use "set x $i" instead, and don't expect to do any arithmetic
> > in the shell; use "expr" or "awk" for arithmetic.
> >
> > (Not relevant to kernel scripts, imho, since you can always assume
> > something a bit more modern and not too stripped down).
>
> ACK. A bash can IMHO be expected. Even going for `dash` is IMHO somewhat
> too extreme.

I have yet to encounter a system that uses dash _without_ bash.  (All ubuntu 
variants, even jeos, install bash by default.  They moved the /bin/sh symlink 
but they didn't stop installing bash, and the kernel will preferentially use 
bash if it finds it.)  People keep telling me they exist.  I suppose you could 
uninstall bash.  You could also uninstall gcc.  Not sure what that proves. 
(Continue reading)

Bernd Petrovitsch | 5 Jan 13:29 2009
Picon

Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh

On Son, 2009-01-04 at 22:50 -0600, Rob Landley wrote:
> On Sunday 04 January 2009 18:15:30 Bernd Petrovitsch wrote:
[...]
> > ACK. A bash can IMHO be expected. Even going for `dash` is IMHO somewhat
> > too extreme.
> 
> I have yet to encounter a system that uses dash _without_ bash.  (All ubuntu 

Hmm, should be doable with a chroot environment quite cheap and simple.

> variants, even jeos, install bash by default.  They moved the /bin/sh symlink 

Yes, I know (small) embedded systems that have a bash (and not "only"
one of busybox shells). It eases writing somewhat fast shell scripts
without the need for lots of fork()s+exec()s too .....

	Bernd
--

-- 
Firmix Software GmbH                   http://www.firmix.at/
mobil: +43 664 4416156                 fax: +43 1 7890849-55
          Embedded Linux Development and Services

Alejandro Mery | 4 Jan 22:51 2009

Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh

Valdis.Kletnieks <at> vt.edu wrote:
> Close, but no cee-gar. cut does something counter-intuitive with multiple
> blanks:
>
> % echo 'a    b' | awk '{print $2}'
> b
> % echo 'a    b' | cut -d' ' -f2
>
> % echo 'a    b' | sed -r 's/[ ]+/ /g' | cut -d' ' -f2
> b
>
> Unfortunately, 'sed -r' isn't in the opengroup.org list of required options,
> and sed 's/  / /g' doesn't DTRT for 3 or more blanks (as it won't recursively
> apply the change to a *new* double blank formed by the previous change).
echo 'a    b' | tr -s ' ' | cut -d' ' -f2
b

that is the light way ;-)

Alejandro Mery
Attachment (smime.p7s): application/x-pkcs7-signature, 6776 bytes
Michal Jaegermann | 4 Jan 08:15 2009
Picon
Picon

Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh

On Sat, Jan 03, 2009 at 07:36:04PM -0600, Rob Landley wrote:
> On Saturday 03 January 2009 06:28:22 Ingo Oeser wrote:
> > > +for i in "MSEC 1000" "USEC 1000000"
> > > +do
> > > +	NAME=$(echo $i | awk '{print $1}')
> >
> > cut -d' ' -f1  does the same
> >
> > > +	PERIOD=$(echo $i | awk '{print $2}')
> >
> > cut -d' ' -f2  does the same
> 
> From a standards perspective 
> http://www.opengroup.org/onlinepubs/9699919799/utilities/cut.html vs 
> http://www.opengroup.org/onlinepubs/9699919799/utilities/awk.html is probably 
> a wash, but from a simplicity perspective using the tool that _isn't_ its own 
> programming language is probably a win. :)

Vagaries of 'cut' aside you can limit yourself here to just shell:

set_name_period () {
   NAME=$1 ; PERIOD=$2
}
for i in "MSEC 1000" "USEC 1000000"
do
   set_name_period $i
....
done

or you may skip a shell function and do 'set $i' within a loop plus
(Continue reading)

Ray Lee | 5 Jan 01:41 2009

Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh

On Fri, Jan 2, 2009 at 12:13 AM, Rob Landley <rob <at> landley.net> wrote:
> Replace kernel/timeconst.pl with kernel/timeconst.sh.  The new shell script
> is much simpler, about 1/4 the size, and runs on Red Hat 9 from 2003.
>
> Peter Anvin added this perl to 2.6.25.  Before that, the kernel had never
> required perl to build.

Nice work. As the computations can all be done in 64-bit precision
now, and there have been concerns expressed about some shells not
supporting 64 bit integers, is there any reason this can't be done
using long longs in C?

Other than ruining a good bike shed argument, anyway.
Rob Landley | 5 Jan 06:08 2009
Picon

Re: [PATCH 1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh

On Sunday 04 January 2009 18:41:15 Ray Lee wrote:
> On Fri, Jan 2, 2009 at 12:13 AM, Rob Landley <rob <at> landley.net> wrote:
> > Replace kernel/timeconst.pl with kernel/timeconst.sh.  The new shell
> > script is much simpler, about 1/4 the size, and runs on Red Hat 9 from
> > 2003.
> >
> > Peter Anvin added this perl to 2.6.25.  Before that, the kernel had never
> > required perl to build.
>
> Nice work.

Thanks.  You'll definitely want to look at the _second_ version of that patch 
rather than the first, though. :)

> As the computations can all be done in 64-bit precision
> now, and there have been concerns expressed about some shells not
> supporting 64 bit integers, is there any reason this can't be done
> using long longs in C?

Nope.  Any of this could be done in C.  (And that's the approach Sam Ravnborg 
prefers to take for the second patch in the series, upgrading unifdef.c to do 
everything itself.)

I tend to lean towards scripts that create header files rather than programs 
that create header files, but as long as you remember to use HOSTCC it's 
fairly straightforward. :)

> Other than ruining a good bike shed argument, anyway.

Oh pile on.  It beats being dismissed as the only one on the planet who cares 
(Continue reading)

Rob Landley | 2 Jan 09:14 2009
Picon

[PATCH 2/3]: Remove perl from make headers_install

From: Rob Landley <rob <at> landley.net>

Remove perl from make headers_install by replacing a perl script (doing
a simple regex search and replace) with a smaller and faster shell script
implementation.  The new shell script is a single for loop calling sed and
piping its output through unifdef to produce the target file.

Sam Ravnborg added this perl to 2.6.27.

Signed-off-by: Rob Landley <rob <at> landley.net>
---

 scripts/Makefile.headersinst |    6 ++--
 scripts/headers_install.pl   |   46 ---------------------------------
 scripts/headers_install.sh   |   23 ++++++++++++++++
 3 files changed, 26 insertions(+), 49 deletions(-)

--- /dev/null	2008-11-21 04:46:41.000000000 -0600
+++ b/scripts/headers_install.sh	2008-12-15 22:09:45.000000000 -0600
 <at>  <at>  -0,0 +1,23  <at>  <at> 
+#!/bin/bash
+
+# Grab arguments
+
+INDIR="$1"
+shift
+OUTDIR="$1"
+shift
+ARCH="$1"
+shift
(Continue reading)

Sam Ravnborg | 2 Jan 10:09 2009

Re: [PATCH 2/3]: Remove perl from make headers_install

On Fri, Jan 02, 2009 at 02:14:32AM -0600, Rob Landley wrote:
> From: Rob Landley <rob <at> landley.net>
> 
> Remove perl from make headers_install by replacing a perl script (doing
> a simple regex search and replace) with a smaller and faster shell script
> implementation.  The new shell script is a single for loop calling sed and
> piping its output through unifdef to produce the target file.

OK
> 
> Sam Ravnborg added this perl to 2.6.27.

Drop this part - this is just to make you happy and for no use for others.

> 
> Signed-off-by: Rob Landley <rob <at> landley.net>
> ---
> 
>  scripts/Makefile.headersinst |    6 ++--
>  scripts/headers_install.pl   |   46 ---------------------------------
>  scripts/headers_install.sh   |   23 ++++++++++++++++
>  3 files changed, 26 insertions(+), 49 deletions(-)
> 
> --- /dev/null	2008-11-21 04:46:41.000000000 -0600
> +++ b/scripts/headers_install.sh	2008-12-15 22:09:45.000000000 -0600
>  <at>  <at>  -0,0 +1,23  <at>  <at> 
> +#!/bin/bash
> +
> +# Grab arguments
> +
(Continue reading)

Rob Landley | 2 Jan 09:15 2009
Picon

[PATCH 3/3]: Convert mkcapflags.pl to mkcapflags.sh

From: Rob Landley <rob <at> landley.net>

Convert kernel/cpu/mkcapflags.pl to kernel/cpu/mkcapflags.sh.

This script generates kernel/cpu/capflags.c from include/asm/cpufeature.h.

Peter Anvin added this perl to 2.6.28.

Signed-off-by: Rob Landley <rob <at> landley.net>
---

 arch/x86/kernel/cpu/Makefile      |    4 +--
 arch/x86/kernel/cpu/mkcapflags.pl |   32 ----------------------------
 arch/x86/kernel/cpu/mkcapflags.sh |   28 ++++++++++++++++++++++++
 3 files changed, 30 insertions(+), 34 deletions(-)

diff -ruN alt-linux/arch/x86/kernel/cpu/Makefile alt-linux2/arch/x86/kernel/cpu/Makefile
--- alt-linux/arch/x86/kernel/cpu/Makefile	2008-12-24 17:26:37.000000000 -0600
+++ alt-linux2/arch/x86/kernel/cpu/Makefile	2008-12-28 18:10:51.000000000 -0600
 <at>  <at>  -23,10 +23,10  <at>  <at> 
 obj-$(CONFIG_X86_LOCAL_APIC) += perfctr-watchdog.o

 quiet_cmd_mkcapflags = MKCAP   $ <at> 
-      cmd_mkcapflags = $(PERL) $(srctree)/$(src)/mkcapflags.pl $< $ <at> 
+      cmd_mkcapflags = $(CONFIG_SHELL) $(srctree)/$(src)/mkcapflags.sh $< $ <at> 

 cpufeature = $(src)/../../include/asm/cpufeature.h

 targets += capflags.c
-$(obj)/capflags.c: $(cpufeature) $(src)/mkcapflags.pl FORCE
(Continue reading)

Sam Ravnborg | 2 Jan 10:12 2009

Re: [PATCH 3/3]: Convert mkcapflags.pl to mkcapflags.sh

On Fri, Jan 02, 2009 at 02:15:36AM -0600, Rob Landley wrote:
> From: Rob Landley <rob <at> landley.net>
> 
> Convert kernel/cpu/mkcapflags.pl to kernel/cpu/mkcapflags.sh.
> 
> This script generates kernel/cpu/capflags.c from include/asm/cpufeature.h.
OK

> Peter Anvin added this perl to 2.6.28.
Drop it.

Implementation looks OK.

	Sam
Arkadiusz Miskiewicz | 2 Jan 10:26 2009
Picon

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

On Friday 02 of January 2009, Rob Landley wrote:
> Before 2.6.25 (specifically git bdc807871d58285737d50dc6163d0feb72cb0dc2 )
> building a Linux kernel never required perl to be installed on the build
> system.  (Various development and debugging scripts were written in perl
> and python and such, but they weren't involved in actually building a
> kernel.) Building a kernel before 2.6.25 could be done with a minimal
> system built from gcc, binutils, bash, make, busybox, uClibc, and the Linux
> kernel, and nothing else.

And now bash is going to be required... while some distros don't need/have 
bash. /bin/sh should be enough.

Heh,
--

-- 
Arkadiusz Miśkiewicz        PLD/Linux Team
arekm / maven.pl            http://ftp.pld-linux.org/
Christoph Hellwig | 2 Jan 10:49 2009

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

On Fri, Jan 02, 2009 at 10:26:37AM +0100, Arkadiusz Miskiewicz wrote:
> On Friday 02 of January 2009, Rob Landley wrote:
> > Before 2.6.25 (specifically git bdc807871d58285737d50dc6163d0feb72cb0dc2 )
> > building a Linux kernel never required perl to be installed on the build
> > system.  (Various development and debugging scripts were written in perl
> > and python and such, but they weren't involved in actually building a
> > kernel.) Building a kernel before 2.6.25 could be done with a minimal
> > system built from gcc, binutils, bash, make, busybox, uClibc, and the Linux
> > kernel, and nothing else.
> 
> And now bash is going to be required... while some distros don't need/have 
> bash. /bin/sh should be enough.

*nod*  bash is in many ways a worse requirement than perl.  strict posix
/bin/sh + awk + sed would be nicest, but if that's too much work perl
seems reasonable.

Alejandro Mery | 2 Jan 11:16 2009

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

Christoph Hellwig escribió:
> On Fri, Jan 02, 2009 at 10:26:37AM +0100, Arkadiusz Miskiewicz wrote:
>   
>> On Friday 02 of January 2009, Rob Landley wrote:
>>     
>>> Before 2.6.25 (specifically git bdc807871d58285737d50dc6163d0feb72cb0dc2 )
>>> building a Linux kernel never required perl to be installed on the build
>>> system.  (Various development and debugging scripts were written in perl
>>> and python and such, but they weren't involved in actually building a
>>> kernel.) Building a kernel before 2.6.25 could be done with a minimal
>>> system built from gcc, binutils, bash, make, busybox, uClibc, and the Linux
>>> kernel, and nothing else.
>>>       
>> And now bash is going to be required... while some distros don't need/have 
>> bash. /bin/sh should be enough.
>>     
>
> *nod*  bash is in many ways a worse requirement than perl.  strict posix
> /bin/sh + awk + sed would be nicest, but if that's too much work perl
> seems reasonable.
well, bash is not worse as bash is trivial to cross-compile to run on a
constrained sandbox and perl is a nightmare, but I agree bash should be
avoided too.

I think the $(( ... )) bash-ism can be replaced with a simple .c helper toy.

Thank Rob for reopening the topic.

Alejandro Mery

(Continue reading)

Mark Miller | 2 Jan 11:30 2009

Re: PATCH [0/3]: Simplify the kernel build by removing perl.


On Jan 2, 2009, at 4:16 AM, Alejandro Mery wrote:

> Christoph Hellwig escribió:
>> On Fri, Jan 02, 2009 at 10:26:37AM +0100, Arkadiusz Miskiewicz wrote:
>>
>>> On Friday 02 of January 2009, Rob Landley wrote:
>>>
>>>> Before 2.6.25 (specifically git  
>>>> bdc807871d58285737d50dc6163d0feb72cb0dc2 )
>>>> building a Linux kernel never required perl to be installed on  
>>>> the build
>>>> system.  (Various development and debugging scripts were written  
>>>> in perl
>>>> and python and such, but they weren't involved in actually  
>>>> building a
>>>> kernel.) Building a kernel before 2.6.25 could be done with a  
>>>> minimal
>>>> system built from gcc, binutils, bash, make, busybox, uClibc, and  
>>>> the Linux
>>>> kernel, and nothing else.
>>>>
>>> And now bash is going to be required... while some distros don't  
>>> need/have
>>> bash. /bin/sh should be enough.
>>>
>>
>> *nod*  bash is in many ways a worse requirement than perl.  strict  
>> posix
>> /bin/sh + awk + sed would be nicest, but if that's too much work perl
(Continue reading)

Matt Keenan | 2 Jan 12:18 2009
Picon

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

On Fri, 2009-01-02 at 04:30 -0600, Mark Miller wrote:
> On Jan 2, 2009, at 4:16 AM, Alejandro Mery wrote:
> 
> > Christoph Hellwig escribió:
> >> On Fri, Jan 02, 2009 at 10:26:37AM +0100, Arkadiusz Miskiewicz wrote:
> >>
> >>> On Friday 02 of January 2009, Rob Landley wrote:
> >>>
> >>>> Before 2.6.25 (specifically git  
> >>>> bdc807871d58285737d50dc6163d0feb72cb0dc2 )
> >>>> building a Linux kernel never required perl to be installed on  
> >>>> the build
> >>>> system.  (Various development and debugging scripts were written  
> >>>> in perl
> >>>> and python and such, but they weren't involved in actually  
> >>>> building a
> >>>> kernel.) Building a kernel before 2.6.25 could be done with a  
> >>>> minimal
> >>>> system built from gcc, binutils, bash, make, busybox, uClibc, and  
> >>>> the Linux
> >>>> kernel, and nothing else.
> >>>>
> >>> And now bash is going to be required... while some distros don't  
> >>> need/have
> >>> bash. /bin/sh should be enough.
> >>>
> >>
> >> *nod*  bash is in many ways a worse requirement than perl.  strict  
> >> posix
> >> /bin/sh + awk + sed would be nicest, but if that's too much work perl
(Continue reading)

Måns Rullgård | 2 Jan 11:41 2009

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

Alejandro Mery <amery <at> opensde.org> writes:

> Christoph Hellwig escribió:
>> On Fri, Jan 02, 2009 at 10:26:37AM +0100, Arkadiusz Miskiewicz wrote:
>>   
>>> On Friday 02 of January 2009, Rob Landley wrote:
>>>     
>>>> Before 2.6.25 (specifically git bdc807871d58285737d50dc6163d0feb72cb0dc2 )
>>>> building a Linux kernel never required perl to be installed on the build
>>>> system.  (Various development and debugging scripts were written in perl
>>>> and python and such, but they weren't involved in actually building a
>>>> kernel.) Building a kernel before 2.6.25 could be done with a minimal
>>>> system built from gcc, binutils, bash, make, busybox, uClibc, and the Linux
>>>> kernel, and nothing else.
>>>>       
>>> And now bash is going to be required... while some distros don't need/have 
>>> bash. /bin/sh should be enough.
>>>     
>>
>> *nod*  bash is in many ways a worse requirement than perl.  strict posix
>> /bin/sh + awk + sed would be nicest, but if that's too much work perl
>> seems reasonable.
> well, bash is not worse as bash is trivial to cross-compile to run on a
> constrained sandbox and perl is a nightmare, but I agree bash should be
> avoided too.
>
> I think the $(( ... )) bash-ism can be replaced with a simple .c helper toy.

The $(( ... )) construct is standard POSIX shell syntax, see
http://www.opengroup.org/onlinepubs/000095399/utilities/xcu_chap02.html#tag_02_06_04
(Continue reading)

Pádraig Brady | 15 Jan 13:59 2009

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

Måns Rullgård wrote:
> Alejandro Mery <amery <at> opensde.org> writes:
> 
>> I think the $(( ... )) bash-ism can be replaced with a simple .c helper toy.
> 
> The $(( ... )) construct is standard POSIX shell syntax, see
> http://www.opengroup.org/onlinepubs/000095399/utilities/xcu_chap02.html#tag_02_06_04
> 
> Bash supports $[ ... ] as an alternate syntax for the same thing.
> Perhaps you were thinking of that.

I think the misconception that $(( ... )) is a bashism is caused by
the wrong highlighting defaults chosen by vim. To fix this add this to ~/.vimrc

let g:is_posix = 1

That will also allow you to use the $(command) POSIX construct.
BTW, the vim syntax maintainers don't agree with changing this default:
http://groups.google.com/group/vim_dev/browse_thread/thread/41139a32772b2f5f

cheers,
Pádraig.
Jamie Lokier | 15 Jan 19:52 2009

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

Pádraig Brady wrote:
> > The $(( ... )) construct is standard POSIX shell syntax, see
> > http://www.opengroup.org/onlinepubs/000095399/utilities/xcu_chap02.html#tag_02_06_04
> > 
> > Bash supports $[ ... ] as an alternate syntax for the same thing.
> > Perhaps you were thinking of that.
> 
> I think the misconception that $(( ... )) is a bashism is caused by
> the wrong highlighting defaults chosen by vim.

I think the misconception is because traditional unix bourne shells
don't implement that construct.  I just tried it on a few machines,
and it failed on 4 of them.  Admittedly, the only up to date one is
running Solaris 10; the others are older unixes that you're unlikely
to build Linux on.

-- Jamie
Måns Rullgård | 15 Jan 20:45 2009

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

Jamie Lokier <jamie <at> shareable.org> writes:

> Pádraig Brady wrote:
>> > The $(( ... )) construct is standard POSIX shell syntax, see
>> > http://www.opengroup.org/onlinepubs/000095399/utilities/xcu_chap02.html#tag_02_06_04
>> > 
>> > Bash supports $[ ... ] as an alternate syntax for the same thing.
>> > Perhaps you were thinking of that.
>> 
>> I think the misconception that $(( ... )) is a bashism is caused by
>> the wrong highlighting defaults chosen by vim.
>
> I think the misconception is because traditional unix bourne shells
> don't implement that construct.  I just tried it on a few machines,
> and it failed on 4 of them.  Admittedly, the only up to date one is
> running Solaris 10; the others are older unixes that you're unlikely
> to build Linux on.

On Solaris, /usr/xpg4/bin/sh is usually a POSIX-compliant shell.  I
don't know how long it has been there.

--

-- 
Måns Rullgård
mans <at> mansr.com
Rob Landley | 2 Jan 12:15 2009
Picon

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

On Friday 02 January 2009 04:16:53 Alejandro Mery wrote:
> Christoph Hellwig escribió:
> > On Fri, Jan 02, 2009 at 10:26:37AM +0100, Arkadiusz Miskiewicz wrote:
> >> On Friday 02 of January 2009, Rob Landley wrote:
> >>> Before 2.6.25 (specifically git
> >>> bdc807871d58285737d50dc6163d0feb72cb0dc2 ) building a Linux kernel
> >>> never required perl to be installed on the build system.  (Various
> >>> development and debugging scripts were written in perl and python and
> >>> such, but they weren't involved in actually building a kernel.)
> >>> Building a kernel before 2.6.25 could be done with a minimal system
> >>> built from gcc, binutils, bash, make, busybox, uClibc, and the Linux
> >>> kernel, and nothing else.
> >>
> >> And now bash is going to be required... while some distros don't
> >> need/have bash. /bin/sh should be enough.
> >
> > *nod*  bash is in many ways a worse requirement than perl.  strict posix
> > /bin/sh + awk + sed would be nicest, but if that's too much work perl
> > seems reasonable.
>
> well, bash is not worse as bash is trivial to cross-compile to run on a
> constrained sandbox and perl is a nightmare, but I agree bash should be
> avoided too.
>
> I think the $(( ... )) bash-ism can be replaced with a simple .c helper
> toy.

No, $[ ] is the bashism, $(( )) is susv3:
http://www.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06_04

(Continue reading)

Sam Ravnborg | 2 Jan 12:44 2009

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

On Fri, Jan 02, 2009 at 05:15:32AM -0600, Rob Landley wrote:
> On Friday 02 January 2009 04:16:53 Alejandro Mery wrote:
> > Christoph Hellwig escribió:
> > > On Fri, Jan 02, 2009 at 10:26:37AM +0100, Arkadiusz Miskiewicz wrote:
> > >> On Friday 02 of January 2009, Rob Landley wrote:
> > >>> Before 2.6.25 (specifically git
> > >>> bdc807871d58285737d50dc6163d0feb72cb0dc2 ) building a Linux kernel
> > >>> never required perl to be installed on the build system.  (Various
> > >>> development and debugging scripts were written in perl and python and
> > >>> such, but they weren't involved in actually building a kernel.)
> > >>> Building a kernel before 2.6.25 could be done with a minimal system
> > >>> built from gcc, binutils, bash, make, busybox, uClibc, and the Linux
> > >>> kernel, and nothing else.
> > >>
> > >> And now bash is going to be required... while some distros don't
> > >> need/have bash. /bin/sh should be enough.
> > >
> > > *nod*  bash is in many ways a worse requirement than perl.  strict posix
> > > /bin/sh + awk + sed would be nicest, but if that's too much work perl
> > > seems reasonable.
> >
> > well, bash is not worse as bash is trivial to cross-compile to run on a
> > constrained sandbox and perl is a nightmare, but I agree bash should be
> > avoided too.
> >
> > I think the $(( ... )) bash-ism can be replaced with a simple .c helper
> > toy.
> 
> No, $[ ] is the bashism, $(( )) is susv3:
> http://www.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06_04
(Continue reading)

Rob Landley | 2 Jan 13:56 2009
Picon

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

On Friday 02 January 2009 03:49:34 Christoph Hellwig wrote:
> On Fri, Jan 02, 2009 at 10:26:37AM +0100, Arkadiusz Miskiewicz wrote:
> > On Friday 02 of January 2009, Rob Landley wrote:
> > > Before 2.6.25 (specifically git
> > > bdc807871d58285737d50dc6163d0feb72cb0dc2 ) building a Linux kernel
> > > never required perl to be installed on the build system.  (Various
> > > development and debugging scripts were written in perl and python and
> > > such, but they weren't involved in actually building a kernel.)
> > > Building a kernel before 2.6.25 could be done with a minimal system
> > > built from gcc, binutils, bash, make, busybox, uClibc, and the Linux
> > > kernel, and nothing else.
> >
> > And now bash is going to be required... while some distros don't
> > need/have bash. /bin/sh should be enough.
>
> *nod*  bash is in many ways a worse requirement than perl.  strict posix
> /bin/sh + awk + sed would be nicest, but if that's too much work perl
> seems reasonable.

The scripts should work with dash (modulo the one that needs 64 bit math, 
which dash only provides on a 64 bit host), or with busybox ash (which can 
provide 64 bit math on 32 bit hosts just like bash can).  I'll explicitly 
retest both of those when I respin the patches in the <strike>morning</strike> 
afternoon.

(And yes I thought about writing my own arbitrary precision arithmetic shell 
functions, but it really didn't seem worth the complexity since the only 32 
bit Linux distros I've seen that install dash also install bash by default.  I 
just put in a test for 32 bit math so it can spot it and fail, on the off 
chance you're running a 32 bit host with dash after explicitly uninstalling 
(Continue reading)

Theodore Tso | 2 Jan 15:04 2009
Picon
Picon

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

On Fri, Jan 02, 2009 at 06:56:31AM -0600, Rob Landley wrote: 
> That said, how is bash _worse_ than perl?  (Where's the second
> implementation of perl?  Even Python had jython, but perl
> has... what?  The attempt to rebase on Parrot went down in
> flames...)

(1) bash implies POSIX extensions; perl is actually quite portable.

(2) There are distributions that install with perl by default but not
bash; they use dash for speed reasons.

Sounds like though modulo dealing with 64-bit arithmetic, your patches
are mostly dash/POSIX.2 comformant, so you're probably mostly good on
that front once you address the 32/64-bit issues.  I'd also suggest
explicitly add a reminder to the shell scripts' comments to avoid
bashisms for maximum portability, to remind developers in the future
who might try to change the shell scripts to watch out for portability
issues.

							- Ted

Jamie Lokier | 3 Jan 04:22 2009

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

Theodore Tso wrote:
> perl is actually quite portable.

Portability aside, Perl has another fun issue.  The number of times
I've had a Perl script break when copied to a newer system which had a
newer version of Perl is... noticable.

> I'd also suggest explicitly add a reminder to the shell scripts'
> comments to avoid bashisms for maximum portability, to remind
> developers in the future who might try to change the shell scripts
> to watch out for portability issues.

You can force Bash into POSIX mode if that's helpful.

-- Jamie
Rob Landley | 4 Jan 03:23 2009
Picon

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

On Friday 02 January 2009 08:04:09 Theodore Tso wrote:
> Sounds like though modulo dealing with 64-bit arithmetic, your patches
> are mostly dash/POSIX.2 comformant, so you're probably mostly good on
> that front once you address the 32/64-bit issues.  I'd also suggest
> explicitly add a reminder to the shell scripts' comments to avoid
> bashisms for maximum portability, to remind developers in the future
> who might try to change the shell scripts to watch out for portability
> issues.

I changed the scripts to start with #!/bin/sh and retested under dash.

If scripts say #!/bin/sh when they actually need bash, or say #!/bin/bash when 
they work with dash, that should probably be fixed.

Rob
Mark Miller | 2 Jan 11:02 2009

Re: PATCH [0/3]: Simplify the kernel build by removing perl.


On Jan 2, 2009, at 3:26 AM, Arkadiusz Miskiewicz wrote:

> On Friday 02 of January 2009, Rob Landley wrote:
>> Before 2.6.25 (specifically git  
>> bdc807871d58285737d50dc6163d0feb72cb0dc2 )
>> building a Linux kernel never required perl to be installed on the  
>> build
>> system.  (Various development and debugging scripts were written in  
>> perl
>> and python and such, but they weren't involved in actually building a
>> kernel.) Building a kernel before 2.6.25 could be done with a minimal
>> system built from gcc, binutils, bash, make, busybox, uClibc, and  
>> the Linux
>> kernel, and nothing else.
>
> And now bash is going to be required... while some distros don't  
> need/have
> bash. /bin/sh should be enough.

Which distros only have /bin/sh which do not have Perl? I'm honestly  
curious.

> -- 
> Arkadiusz Miśkiewicz        PLD/Linux Team
> arekm / maven.pl            http://ftp.pld-linux.org/

--
Mark Miller
mark <at> mirell.org
(Continue reading)

Mark Miller | 2 Jan 11:03 2009

Re: PATCH [0/3]: Simplify the kernel build by removing perl.


On Jan 2, 2009, at 4:02 AM, Mark Miller wrote:

>
> On Jan 2, 2009, at 3:26 AM, Arkadiusz Miskiewicz wrote:
>
>> On Friday 02 of January 2009, Rob Landley wrote:
>>> Before 2.6.25 (specifically git  
>>> bdc807871d58285737d50dc6163d0feb72cb0dc2 )
>>> building a Linux kernel never required perl to be installed on the  
>>> build
>>> system.  (Various development and debugging scripts were written  
>>> in perl
>>> and python and such, but they weren't involved in actually  
>>> building a
>>> kernel.) Building a kernel before 2.6.25 could be done with a  
>>> minimal
>>> system built from gcc, binutils, bash, make, busybox, uClibc, and  
>>> the Linux
>>> kernel, and nothing else.
>>
>> And now bash is going to be required... while some distros don't  
>> need/have
>> bash. /bin/sh should be enough.
>
> Which distros only have /bin/sh which do not have Perl? I'm honestly  
> curious.

That is, *do* have Perl. Typo there.

(Continue reading)

Rob Landley | 2 Jan 12:13 2009
Picon

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

On Friday 02 January 2009 03:26:37 Arkadiusz Miskiewicz wrote:
> On Friday 02 of January 2009, Rob Landley wrote:
> > Before 2.6.25 (specifically git bdc807871d58285737d50dc6163d0feb72cb0dc2
> > ) building a Linux kernel never required perl to be installed on the
> > build system.  (Various development and debugging scripts were written in
> > perl and python and such, but they weren't involved in actually building
> > a kernel.) Building a kernel before 2.6.25 could be done with a minimal
> > system built from gcc, binutils, bash, make, busybox, uClibc, and the
> > Linux kernel, and nothing else.
>
> And now bash is going to be required... while some distros don't need/have
> bash. /bin/sh should be enough.
>
> Heh,

I believe all three scripts run under dash and busybox ash.  (The timeconst.sh 
one needs 64 bit math which dash only provides on 64 bit hosts, which is a 
regression from Red Hat 9 in 2003 by the way.  Busybox ash can also provide 64 
bit math on 32 bit hosts, and the script should run with that just fine if you 
haven't got bash and that's what your "sh" in the path is.)

The makefiles execute those scripts via CONFIG_SHELL, not via the #!/blah line 
at the start, so it's largely irrelevant what gets put there anyway.  If you 
haven't got bash installed it'll use "sh", which should work with dash on a 64 
bit host or with busybox ash.  (That's why that one file has a test to make 
sure 64 bit math _does_ work.  The only Linux development environment I'm 
aware of where that test would trigger is if you use a 32 bit ubuntu and go 
out of your way to _uninstall_ bash.  Even Cygwin uses bash.)

Beyond that, "find linux -type f | xargs grep bin/bash | wc" comes up with 38 
(Continue reading)

Matthieu CASTET | 2 Jan 17:04 2009

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

Rob Landley a écrit :
> On Friday 02 January 2009 03:26:37 Arkadiusz Miskiewicz wrote:
>> On Friday 02 of January 2009, Rob Landley wrote:
>>
>> Heh,
> 
> I believe all three scripts run under dash and busybox ash.  (The timeconst.sh 
> one needs 64 bit math which dash only provides on 64 bit hosts, which is a 
> regression from Red Hat 9 in 2003 by the way.
With dash 0.5.4-12 (from debian sid), I seems I got the 64 bit math for
32 bit hosts :
$ uname -m
i686
$ dash -c 'echo $((1<<32))'
4294967296

Matthieu
Rob Landley | 3 Jan 20:46 2009
Picon

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

On Friday 02 January 2009 10:04:08 Matthieu CASTET wrote:
> Rob Landley a écrit :
> > On Friday 02 January 2009 03:26:37 Arkadiusz Miskiewicz wrote:
> >> On Friday 02 of January 2009, Rob Landley wrote:
> >>
> >> Heh,
> >
> > I believe all three scripts run under dash and busybox ash.  (The
> > timeconst.sh one needs 64 bit math which dash only provides on 64 bit
> > hosts, which is a regression from Red Hat 9 in 2003 by the way.
>
> With dash 0.5.4-12 (from debian sid), I seems I got the 64 bit math for
> 32 bit hosts :
> $ uname -m
> i686
> $ dash -c 'echo $((1<<32))'
> 4294967296

Cool.

The "relatively recent" 32 bit image I have lying around for testing purposes 
is xubuntu 7.10, and when dash was first introduced into ubuntu it had 
_buckets_ of bugs.  (If you backgrounded a task with & and then hit ctrl-z on 
the command line, it would suspend the backgrounded task.  It was Not Ready 
for Prime Time in a big way.)  Lack of 64 bit math could easily be one more.  
(It _is_ a regression vs Red Hat 9.)

The dash in ubuntu 8.10 seems to have a lot of the more obvious problems 
worked out.  Good to know. :)

(Continue reading)

Sam Ravnborg | 3 Jan 21:10 2009

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

> 
> I'll fix this and resubmit, it just wasn't ready last night.  (If the merge 
> window is closing soon I could resubmit the other two with Sam's suggestions 
> and resubmit this one next time 'round, but it was only a couple days to write 
> in the first place, once I finally figured out what the perl version was 
> trying to _do_...)

For kbuild only fixes and trivial stuff will be merged until next merge window.
Neither of the three patches fall into that category.

With respect to your three patches the plan is to:
- add the updated timeconst patch to kbuild-next
- add the updated cpu-feature patch to kbuild-next

- the patch touching headers_install will not be merged.
  The way forward for headers_install is to combine the
  unifdef feature and the header modifications.
  And this must be in a single program that can process
  all headers in one go so the install process becomes so fast
  that we do not worry about if it was done before or not.
  Then we can avoid all the .* files in the directory
  where we isntall the headers.
  The program is a prime candidate for a small C program
  and I hope someone can take the challenge to write it.

  Migrating from perl to shell does not help us here
  and the shell version was less readable than the perl version.

	Sam
(Continue reading)

H. Peter Anvin | 3 Jan 21:50 2009

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

Sam Ravnborg wrote:
> With respect to your three patches the plan is to:
> - add the updated timeconst patch to kbuild-next

If you add this, you take the responsibility for the breakages that will
occur.  The reason his patch is "simpler" is because he removes the
arbitrary-precision arithmetic, and simply hopes that the system
utilities that he uses uses an integer size which happens to be big enough.

	-hpa

--

-- 
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.

Rob Landley | 4 Jan 02:47 2009
Picon

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

On Saturday 03 January 2009 14:50:43 H. Peter Anvin wrote:
> Sam Ravnborg wrote:
> > With respect to your three patches the plan is to:
> > - add the updated timeconst patch to kbuild-next
>
> If you add this, you take the responsibility for the breakages that will
> occur.  The reason his patch is "simpler" is because he removes the
> arbitrary-precision arithmetic, and simply hopes that the system
> utilities that he uses uses an integer size which happens to be big enough.

Actually once I _noticed_ that ADJ32 could overflow (none of the other values 
can), it was fairly easy to prevent it from doing so.  I added some comments 
analyzing the pathological case (HZ 1, assuming nobody's interested in 
HZ>1000000).

> 	-hpa

Rob
Rob Landley | 4 Jan 02:45 2009
Picon

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

On Saturday 03 January 2009 14:10:59 Sam Ravnborg wrote:
> > I'll fix this and resubmit, it just wasn't ready last night.  (If the
> > merge window is closing soon I could resubmit the other two with Sam's
> > suggestions and resubmit this one next time 'round, but it was only a
> > couple days to write in the first place, once I finally figured out what
> > the perl version was trying to _do_...)
>
> For kbuild only fixes and trivial stuff will be merged until next merge
> window. Neither of the three patches fall into that category.

*shrug*  I poke my head into kernel development every few months, and have 
just enough familiarity with it to remember that "changes go in during the 
merge window".  Seemed a good time to post 'em.

> With respect to your three patches the plan is to:
> - add the updated timeconst patch to kbuild-next
> - add the updated cpu-feature patch to kbuild-next
>
> - the patch touching headers_install will not be merged.
>   The way forward for headers_install is to combine the
>   unifdef feature and the header modifications.

Since you're turning down an existing patch in favor of a theoretical patch, I 
assume you have plans to do this yourself?

>   And this must be in a single program that can process
>   all headers in one go so the install process becomes so fast
>   that we do not worry about if it was done before or not.
>   Then we can avoid all the .* files in the directory
>   where we isntall the headers.
(Continue reading)

Sam Ravnborg | 4 Jan 09:09 2009

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

On Sat, Jan 03, 2009 at 07:45:34PM -0600, Rob Landley wrote:
> > With respect to your three patches the plan is to:
> > - add the updated timeconst patch to kbuild-next
> > - add the updated cpu-feature patch to kbuild-next
> >
> > - the patch touching headers_install will not be merged.
> >   The way forward for headers_install is to combine the
> >   unifdef feature and the header modifications.
> 
> Since you're turning down an existing patch in favor of a theoretical patch, I 
> assume you have plans to do this yourself?

If noone else beats me I will do so - yes.
> 
> >   And this must be in a single program that can process
> >   all headers in one go so the install process becomes so fast
> >   that we do not worry about if it was done before or not.
> >   Then we can avoid all the .* files in the directory
> >   where we isntall the headers.
> 
> What if they run out of disk space halfway through writing a file and thus it 
> creates a short file (or a 0 length file where the dentry was created but no 
> blocks could be allocated for the write)?

Then they fail and make will know. Then may leave a file or 100
but it still failed. At next run everything will be done right
assuming the culprint has been fixed.

> I can try to make the shell version more readable, and more powerful.  It's 
> already noticeably faster than the perl version.  I have no objections to 
(Continue reading)

Rob Landley | 4 Jan 21:19 2009
Picon

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

On Sunday 04 January 2009 02:09:31 Sam Ravnborg wrote:
> On Sat, Jan 03, 2009 at 07:45:34PM -0600, Rob Landley wrote:
> > Since you're turning down an existing patch in favor of a theoretical
> > patch, I assume you have plans to do this yourself?
>
> If noone else beats me I will do so - yes.

Ok.

> > >   And this must be in a single program that can process
> > >   all headers in one go so the install process becomes so fast
> > >   that we do not worry about if it was done before or not.
> > >   Then we can avoid all the .* files in the directory
> > >   where we isntall the headers.
> >
> > What if they run out of disk space halfway through writing a file and
> > thus it creates a short file (or a 0 length file where the dentry was
> > created but no blocks could be allocated for the write)?
>
> Then they fail and make will know. Then may leave a file or 100
> but it still failed. At next run everything will be done right
> assuming the culprint has been fixed.

Ok, so the important thing is propagating failures up to the exit code, then?

When making this patch I hit a problem that the exit code of "unifdef" seems 
to depend on whether it found anything to remove within the file it was 
processing, so when I changed the caller to actually care about its exit code 
it spontaneously aborted.

(Continue reading)

Robert Hancock | 4 Jan 01:44 2009
Picon

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

Rob Landley wrote:
> For the record, the reason I can't just pregenerate all these suckers on a 
> system that's got an arbitrary precision calculator (ala dc) and then just 
> ship the resulting header files (more or less the what the first version of 
> that first patch did) is that some architectures (arm omap and and arm at91) 
> allow you to enter arbitrary HZ values in kconfig.  (Their help text says that 
> in many cases values that aren't powers of two won't work, but nothing 
> enforces this.)  So if we didn't have the capability to dynamically generate 
> these, you could enter a .config value that would break the build.

Is there a good reason that these archs allow you enter arbitrary HZ 
values? The use case for using custom HZ values at all nowadays seems 
fairly low now that dynticks is around (if that arch supports it 
anyway), let alone being able to specify wierd obscure values for it. 
Especially if nothing can ensure that all values it allows will actually 
result in a functional kernel..
David Brownell | 4 Jan 02:39 2009
Picon

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

On Saturday 03 January 2009, Robert Hancock wrote:
> Rob Landley wrote:
> > ... some architectures (arm omap and and arm at91) 
> > allow you to enter arbitrary HZ values in kconfig.  (Their help text says that 
> > in many cases values that aren't powers of two won't work, but nothing 
> > enforces this.)
> 
> Is there a good reason that these archs allow you enter arbitrary HZ 
> values?

Power-of-two can be desirable when using a 32 KiHz oscillator, because
other values accumulate rounding errors ... you can't make 100 Hz, or
250 Hz, or 300 Hz, or 1000 Hz, by a binary division of 32 KiHz.

Other values were supported to help work around stupid software making
bad assumptions about HZ.  IMO, enforcing power-of-two would be better;
that software breaks with dyntick anyway, and needs fixing.

> The use case for using custom HZ values at all nowadays seems  
> fairly low now that dynticks is around (if that arch supports it 
> anyway),

A better argument would be that GENERIC_TIME exists (and works
on OMAP and AT91), which avoids some flavors of rounding error.
ISTR those CONFIG_HZ options predate GENERIC_TIME support.

However, the issue remains that most kernel times are measured in
jiffies not ktime_t -- they're easier and more efficient, all
those 64-bit multiplies can hurt on ARM (32-bit, non-GHz) -- so
it's still good to be able to ensure that jiffies-centric logic
(Continue reading)

Rob Landley | 4 Jan 04:05 2009
Picon

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

On Saturday 03 January 2009 18:44:58 Robert Hancock wrote:
> Rob Landley wrote:
> > For the record, the reason I can't just pregenerate all these suckers on
> > a system that's got an arbitrary precision calculator (ala dc) and then
> > just ship the resulting header files (more or less the what the first
> > version of that first patch did) is that some architectures (arm omap and
> > and arm at91) allow you to enter arbitrary HZ values in kconfig.  (Their
> > help text says that in many cases values that aren't powers of two won't
> > work, but nothing enforces this.)  So if we didn't have the capability to
> > dynamically generate these, you could enter a .config value that would
> > break the build.
>
> Is there a good reason that these archs allow you enter arbitrary HZ
> values?

Not that I've noticed, no.  But you should ask Thomas Gleixner about that 
about that, I'm not a domain expert...

> The use case for using custom HZ values at all nowadays seems
> fairly low now that dynticks is around (if that arch supports it
> anyway), let alone being able to specify wierd obscure values for it.

And high performance event timers.  A kernel can have more than one time 
source these days...

Rob
Rob Landley | 4 Jan 02:32 2009
Picon

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

On Friday 02 January 2009 10:04:08 Matthieu CASTET wrote:
> Rob Landley a écrit :
> > On Friday 02 January 2009 03:26:37 Arkadiusz Miskiewicz wrote:
> >> On Friday 02 of January 2009, Rob Landley wrote:
> >>
> >> Heh,
> >
> > I believe all three scripts run under dash and busybox ash.  (The
> > timeconst.sh one needs 64 bit math which dash only provides on 64 bit
> > hosts, which is a regression from Red Hat 9 in 2003 by the way.
>
> With dash 0.5.4-12 (from debian sid), I seems I got the 64 bit math for
> 32 bit hosts :
> $ uname -m
> i686
> $ dash -c 'echo $((1<<32))'
> 4294967296
>
>
> Matthieu

Alas, my attempt to install a 32 bit version of xubuntu 8.10 under qemu hung 
at "Scanning files: 15%", and has been there for an hour now.  I'll have to 
take your word for it.  (All three scripts work fine under 64 bit dash.)

I encountered one bug in busybox, which I pinged that list about, but 
otherwise busybox ash works on 'em all too.

Rob
(Continue reading)

Paul Mundt | 2 Jan 10:50 2009

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

On Fri, Jan 02, 2009 at 02:07:28AM -0600, Rob Landley wrote:
> Before 2.6.25 (specifically git bdc807871d58285737d50dc6163d0feb72cb0dc2 ) 
> building a Linux kernel never required perl to be installed on the build 
> system.  (Various development and debugging scripts were written in perl and 
> python and such, but they weren't involved in actually building a kernel.)  
> Building a kernel before 2.6.25 could be done with a minimal system built from 
> gcc, binutils, bash, make, busybox, uClibc, and the Linux kernel, and nothing 
> else.  (Embedded developers creating clean cross compile environments that 
> won't leak bits of the host system into the target, or bootstrapping native 
> development environments to run on target hardware or under emulators, tend to 
> care about this sort of thing.)
> 
> The perl checkin for 2.6.25 was the camel's nose under the tent flap, and 
> since then two more instances of perl have shown up in the core kernel build.  
> This patch series removes the three required to build a basic kernel for qemu 
> for the targets I've tested (arm, mips, powerpc, sparc, m68k, sh4, and of 
> course x86 and x86-64), replacing them with shell scripts.

Misguided rhetoric aside, what does this actually accomplish? If folks
add meaningful tools in to the kernel that require python, and it is
generally regarded as being fairly ubiquitous, I can't imagine there
being any substantiated objections against it.

Your main reasons against inclusion of perl seem to be that there is no
realistic expectation for target systems that will be self-hosting will
have perl included, or the inherent complexity in maintaining a coherent
cross compiling environment. Both of these things are issues with your
own environment, and in no way are these representative of the embedded
development community at large.

(Continue reading)

Mark Miller | 2 Jan 11:32 2009

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

On Jan 2, 2009, at 3:50 AM, Paul Mundt wrote:

> On Fri, Jan 02, 2009 at 02:07:28AM -0600, Rob Landley wrote:
>> Before 2.6.25 (specifically git  
>> bdc807871d58285737d50dc6163d0feb72cb0dc2 )
>> building a Linux kernel never required perl to be installed on the  
>> build
>> system.  (Various development and debugging scripts were written in  
>> perl and
>> python and such, but they weren't involved in actually building a  
>> kernel.)
>> Building a kernel before 2.6.25 could be done with a minimal system  
>> built from
>> gcc, binutils, bash, make, busybox, uClibc, and the Linux kernel,  
>> and nothing
>> else.  (Embedded developers creating clean cross compile  
>> environments that
>> won't leak bits of the host system into the target, or  
>> bootstrapping native
>> development environments to run on target hardware or under  
>> emulators, tend to
>> care about this sort of thing.)
>>
>> The perl checkin for 2.6.25 was the camel's nose under the tent  
>> flap, and
>> since then two more instances of perl have shown up in the core  
>> kernel build.
>> This patch series removes the three required to build a basic  
>> kernel for qemu
>> for the targets I've tested (arm, mips, powerpc, sparc, m68k, sh4,  
(Continue reading)

Paul Mundt | 2 Jan 11:57 2009

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

On Fri, Jan 02, 2009 at 04:32:42AM -0600, Mark Miller wrote:
> On Jan 2, 2009, at 3:50 AM, Paul Mundt wrote:
> >Misguided rhetoric aside, what does this actually accomplish? If folks
> >add meaningful tools in to the kernel that require python, and it is
> >generally regarded as being fairly ubiquitous, I can't imagine there
> >being any substantiated objections against it.
> 
> And if the said meaningful tools introduce complex dependencies, then  
> there should be an open discussion as to why exactly we need those  
> tools as opposed to a simpler implementation.
> 
Complex is relative, something that is fairly ubiquitious can hardly be
labelled as complex, regardless of whether historically people have had
issues with that dependency in certain spaces. In any event, simplifying
things is always good, but this open discussion thing is pure fancy,
since when was the kernel a democracy?

> >Your main reasons against inclusion of perl seem to be that there is
> >no realistic expectation for target systems that will be self-hosting
> >will have perl included, or the inherent complexity in maintaining a
> >coherent cross compiling environment. Both of these things are issues
> >with your own environment, and in no way are these representative of
> >the  embedded development community at large.
> 
> I feel that if you attempted to look for discussions on "cross- 
> compiling perl", you will meet with a variety of complaints on what a  
> nightmare it is to do so in a sandboxed environment.
> 
I've had to deal with cross compiling perl for over a decade, in all of
its various forms, in all manner of embedded applications, so please tell
(Continue reading)

Mark Miller | 2 Jan 13:11 2009

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

On Jan 2, 2009, at 4:57 AM, Paul Mundt wrote:

> On Fri, Jan 02, 2009 at 04:32:42AM -0600, Mark Miller wrote:
>> On Jan 2, 2009, at 3:50 AM, Paul Mundt wrote:
>>> Misguided rhetoric aside, what does this actually accomplish? If  
>>> folks
>>> add meaningful tools in to the kernel that require python, and it is
>>> generally regarded as being fairly ubiquitous, I can't imagine there
>>> being any substantiated objections against it.
>>
>> And if the said meaningful tools introduce complex dependencies, then
>> there should be an open discussion as to why exactly we need those
>> tools as opposed to a simpler implementation.
>>
> Complex is relative, something that is fairly ubiquitious can hardly  
> be
> labelled as complex, regardless of whether historically people have  
> had
> issues with that dependency in certain spaces. In any event,  
> simplifying
> things is always good, but this open discussion thing is pure fancy,
> since when was the kernel a democracy?

I'm ignoring the bait.

>>> Your main reasons against inclusion of perl seem to be that there is
>>> no realistic expectation for target systems that will be self- 
>>> hosting
>>> will have perl included, or the inherent complexity in maintaining a
>>> coherent cross compiling environment. Both of these things are  
(Continue reading)

Rob Landley | 2 Jan 13:44 2009
Picon

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

On Friday 02 January 2009 03:50:23 Paul Mundt wrote:
> On Fri, Jan 02, 2009 at 02:07:28AM -0600, Rob Landley wrote:
> > The perl checkin for 2.6.25 was the camel's nose under the tent flap, and
> > since then two more instances of perl have shown up in the core kernel
> > build. This patch series removes the three required to build a basic
> > kernel for qemu for the targets I've tested (arm, mips, powerpc, sparc,
> > m68k, sh4, and of course x86 and x86-64), replacing them with shell
> > scripts.
>
> Misguided rhetoric aside, what does this actually accomplish? If folks
> add meaningful tools in to the kernel that require python, and it is
> generally regarded as being fairly ubiquitous, I can't imagine there
> being any substantiated objections against it.

I think bloat-o-meter is a marvelous tool, and I'm a big fan of python.  But I 
don't think you shouldn't have to run that to compile a kernel either, largely 
because not needing it for the first 17 years or so implied living without 
this requirement could be done, sustainably even.

There's a difference between a development workstation and a dedicated build 
system.  Requiring you to install X11 plus qt on the headless build server 
cranking out nightly snapshots in order to run the configure stage of the 
kernel build would be silly.  But this is not an argument for ripping out 
"make xconfig" from the kernel.

Spot the difference?

> Your main reasons against inclusion of perl seem to be that there is no
> realistic expectation for target systems that will be self-hosting will
> have perl included, or the inherent complexity in maintaining a coherent
(Continue reading)

Wookey | 2 Jan 18:25 2009

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

On 2009-01-02 18:50 +0900, Paul Mundt wrote:
> Your main reasons against inclusion of perl seem to be that there is no
> realistic expectation for target systems that will be self-hosting will
> have perl included, or the inherent complexity in maintaining a coherent
> cross compiling environment. Both of these things are issues with your
> own environment, and in no way are these representative of the embedded
> development community at large.

It may well be true that most embedded people cross-build kernels and
use native perl on a fat build box, but there are plenty of situations
where being able to build kernels without perl is useful. Given the
simplicitly of these patches I can't see any reason not to put them
in, and appreciate Rob's work on this.

And if cross-building perl is really easy, as some in this thread
claim, can someone fix the Debian packages to do it, because that
would be really useful (it appears to be of similar complexity to
cross-building gcc, requiring a 2-stage self-referential build). 

Wookey
--

-- 
Principal hats:  Balloonz - Toby Churchill - Aleph One - Debian
http://wookware.org/
Sam Ravnborg | 2 Jan 19:01 2009

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

Hi Wookey.

> Given the
> simplicitly of these patches I can't see any reason not to put them
> in

Please do NOT do the mistake and think this the same thing.

Rob's patch simplyfy the timecost stuff - and will be applied on
this merit alone assuming comments will be addressed.

But the serie rased anohter topic: shall we ban use of perl
for generating a kernel.
And this is what primary is discussed and the outcome of
that discussion will not prevent patches that stands on their
own to be applied.

	Sam
H. Peter Anvin | 2 Jan 20:27 2009

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

Sam Ravnborg wrote:
> Hi Wookey.
> 
>> Given the
>> simplicitly of these patches I can't see any reason not to put them
>> in
> 
> Please do NOT do the mistake and think this the same thing.
> 
> Rob's patch simplyfy the timecost stuff - and will be applied on
> this merit alone assuming comments will be addressed.
> 
> But the serie rased anohter topic: shall we ban use of perl
> for generating a kernel.
> And this is what primary is discussed and the outcome of
> that discussion will not prevent patches that stands on their
> own to be applied.
> 

My personal opinion on this is that this is ridiculous.  Given that you
need gcc, binutils, make etc. to build the kernel, and this is more than
inherent, you have to have a pretty bloody strangely constrained system
to disallow Perl, which is as close to a standard Unix utility you can
get without making it into SuS.

The only *real* motivation I have seen for this is a system that as far
I can tell is nothing other than a toy, specifically designed to show
off how little you need to build the kernel.  In other words, it's not a
practical application, it's a show-off art piece.

(Continue reading)

Rob Landley | 4 Jan 02:35 2009
Picon

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

On Friday 02 January 2009 13:27:45 H. Peter Anvin wrote:
> Sam Ravnborg wrote:
> > Hi Wookey.
> >
> >> Given the
> >> simplicitly of these patches I can't see any reason not to put them
> >> in
> >
> > Please do NOT do the mistake and think this the same thing.
> >
> > Rob's patch simplyfy the timecost stuff - and will be applied on
> > this merit alone assuming comments will be addressed.
> >
> > But the serie rased anohter topic: shall we ban use of perl
> > for generating a kernel.
> > And this is what primary is discussed and the outcome of
> > that discussion will not prevent patches that stands on their
> > own to be applied.
>
> My personal opinion on this is that this is ridiculous.  Given that you
> need gcc, binutils, make etc. to build the kernel,

I believe Intel's icc builds the kernel, and tinycc previously built a subset 
of the kernel.  The pcc and llvm/clang projects are getting close to being 
able to build the kernel.  Ever since c99 came out, lots of gcc-isms with c99 
equivalents have been swapped over, most of the rest is testing.

> and this is more than
> inherent, you have to have a pretty bloody strangely constrained system
> to disallow Perl, which is as close to a standard Unix utility you can
(Continue reading)

Rob Landley | 3 Jan 20:48 2009
Picon

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

On Friday 02 January 2009 12:01:34 Sam Ravnborg wrote:
> But the serie rased anohter topic: shall we ban use of perl
> for generating a kernel.

I dunno about "ban", but every time somebody adds perl to the "hot path" of 
the kernel build it breaks my build system, and I write a removal patch 
anyway.  I have to maintain them anyway, so I might as well try to push 'em 
upstream.  (I posted the first patch in this series twice before, once for 25 
and then an updated version to the linux-embedded list for .26.)

I didn't discover this topic independently.  Somebody pinged me about it on 
freenode back in February, and several other people sent me private email 
about it, and it's been previously raised on several other mailing lists (such 
as busybox and uclibc ones).

Unfortunately, most of the embedded developers I know aren't subscribed to 
linux-kernel.  (You either do kernel development, or you do everything else.  
It's not really feasible to keep up with the guts of the kernel and uClibc and 
busybox and gcc and qemu and the current offerings of three different hardware 
vendors and whatever userspace application the board's supposed to run and 
your build system and what INSANE things your EEdiot electrical engineer 
decided to miswire this time and fighting off marketing's desire to switch 
everything over to WinCE because they can get their entire advertising budget 
subsidized and there's a trade show next week we're not READY for...  Not only 
can none of 'em read a meaningful subset of linux-kernel anymore, but if you 
disappear into your own little niche for nine months, when you pop back up the 
kernel's all different and sometimes even the patch submission policy's 
migrated a bit.  Heck, I'm three months behind reading the LWN kernel page 
myself, and that's no substitute for kernel-traffic, RIP...)

(Continue reading)

klaasjan gm | 8 Jan 14:13 2009
Picon

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

Rob and to whom it may concern,

> I didn't discover this topic independently.  Somebody pinged me about it on
> freenode back in February, and several other people sent me private email
> about it, and it's been previously raised on several other mailing lists (such
> as busybox and uclibc ones).
>
> Unfortunately, most of the embedded developers I know aren't subscribed to
> linux-kernel.

FWIW,

> Hopefully the cc: to linux-embedded is helping get more embedded guys involved
> in the discussion than just me. :)

Having some experience with cross-compiling the kernel, and some of the basic
toolsets:
I'm with you on this one (and testing the waters at linux-embedded
while I'm at it)

Regards,
Klaasjan
Christian Gagneraud | 8 Jan 16:04 2009
Picon

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

klaasjan gm wrote:
> Rob and to whom it may concern,
> 
>> I didn't discover this topic independently.  Somebody pinged me about it on
>> freenode back in February, and several other people sent me private email
>> about it, and it's been previously raised on several other mailing lists (such
>> as busybox and uclibc ones).
>>
>> Unfortunately, most of the embedded developers I know aren't subscribed to
>> linux-kernel.
> 
> FWIW,
> 
>> Hopefully the cc: to linux-embedded is helping get more embedded guys involved
>> in the discussion than just me. :)
> 
> Having some experience with cross-compiling the kernel, and some of the basic
> toolsets:
> I'm with you on this one (and testing the waters at linux-embedded
> while I'm at it)

+1

Chris
> 
> Regards,
> Klaasjan
> --
> To unsubscribe from this list: send the line "unsubscribe linux-embedded" in
> the body of a message to majordomo <at> vger.kernel.org
(Continue reading)

Wolfgang Denk | 3 Jan 15:59 2009
Picon
Picon

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

Dear Paul Mundt,

In message <20090102095023.GA28078 <at> linux-sh.org> you wrote:
>
> Your main reasons against inclusion of perl seem to be that there is no
> realistic expectation for target systems that will be self-hosting will
> have perl included, or the inherent complexity in maintaining a coherent
> cross compiling environment. Both of these things are issues with your
> own environment, and in no way are these representative of the embedded
> development community at large.

I'm  not  sure  how  representative  for  the  "embedded  development
community at large" your statement is.

Just to add a data point to the statistice, I do agree with Rob's
opinion about needing Per to build the Linux kernel.

All other arguments aside, natively compiling the Linux kernel  on  a
target  board, especially with root file system mounted over NFS, has
always been an excellent stress test (you may even call it regression
test) for Linux running on some target hardware.

Losing this ability just because  some  scripts  are  implemented  in
language  FOO when plain shell will do as well is something we should
try to avoid.

Of course, YMMV.

Best regards,

(Continue reading)

Leon Woestenberg | 3 Jan 23:54 2009
Picon

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

Hello all,

On Fri, Jan 2, 2009 at 9:07 AM, Rob Landley <rob <at> landley.net> wrote:
> Before 2.6.25 (specifically git bdc807871d58285737d50dc6163d0feb72cb0dc2 )
> building a Linux kernel never required perl to be installed on the build
> system.  (Various development and debugging scripts were written in perl and
> python and such, but they weren't involved in actually building a kernel.)
> Building a kernel before 2.6.25 could be done with a minimal system built from
> gcc, binutils, bash, make, busybox, uClibc, and the Linux kernel, and nothing
> else.  (Embedded developers creating clean cross compile environments that
>

I agree with Rob that the amount of required dependencies should be
kept to a minimum.

If we only use 0.5% of a certain language (or: dependent package),
then rather implement
that 0.5% in the existing language.

Dependencies very quickly become dependency hell. If A requires B,
then A also inherits all
(future) requirements of B, etc. etc.

In my daily software development work with Linux and GNU software in
general, 10% of it is spent fighting/removing these extremely "thin"
or false depencies, so that it is usuable in embedded devices.

Regards,
--

-- 
Leon
(Continue reading)

H. Peter Anvin | 4 Jan 00:03 2009

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

Leon Woestenberg wrote:
> 
> I agree with Rob that the amount of required dependencies should be
> kept to a minimum.
> 
> If we only use 0.5% of a certain language (or: dependent package),
> then rather implement that 0.5% in the existing language.
> 
> Dependencies very quickly become dependency hell. If A requires B,
> then A also inherits all (future) requirements of B, etc. etc.
> 
> In my daily software development work with Linux and GNU software in
> general, 10% of it is spent fighting/removing these extremely "thin"
> or false depencies, so that it is usuable in embedded devices.
> 

First of all, I largely consider this a joke.  All real-life embedded
kernel builds take place on hosted platforms; anything else seems to be
done "just because it can be done", as a kind of show-off art project.
Cute, but hardly worth impeding the rest of the kernel community for.

We're not talking about general platform dependencies here, but build
dependencies for the kernel.  A platform that can build the kernel is
not a small platform.

Second of all, these patches are not fullworthy replacements.  The
original patch using bc had less dependencies, but bc failed on some
platforms, mysteriously.  The new patches have *more* environmental
dependencies than that ever did.

(Continue reading)

Leon Woestenberg | 4 Jan 01:37 2009
Picon

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

Hello,

On Sun, Jan 4, 2009 at 12:03 AM, H. Peter Anvin <hpa <at> zytor.com> wrote:
>> Dependencies very quickly become dependency hell. If A requires B,
>> then A also inherits all (future) requirements of B, etc. etc.
>>
>> In my daily software development work with Linux and GNU software in
>> general, 10% of it is spent fighting/removing these extremely "thin"
>> or false depencies, so that it is usuable in embedded devices.
>>
>
> First of all, I largely consider this a joke.  All real-life embedded
> kernel builds take place on hosted platforms; anything else seems to be
> done "just because it can be done", as a kind of show-off art project.
> Cute, but hardly worth impeding the rest of the kernel community for.
>
Let me explain why it is not a joke for me, although yes I agree it's
a funny way of how software engineering works.

My argument on thin dependencies indeed mostly holds for run-time
dependencies (to reduce size) but also for build dependency (to reduce
complexity)*.

In general the right version of the right tool is not available on the
build host.

If I cross-build 30 packages all of which need a build-host-native
perl during their build, consider the chance of these packages
building with the one version of perl that lives on the system. It's
near 0% for the average mix of packages.
(Continue reading)

Rob Landley | 4 Jan 03:53 2009
Picon

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

On Saturday 03 January 2009 18:37:12 Leon Woestenberg wrote:
> Hello,
>
> On Sun, Jan 4, 2009 at 12:03 AM, H. Peter Anvin <hpa <at> zytor.com> wrote:
> >> Dependencies very quickly become dependency hell. If A requires B,
> >> then A also inherits all (future) requirements of B, etc. etc.
> >>
> >> In my daily software development work with Linux and GNU software in
> >> general, 10% of it is spent fighting/removing these extremely "thin"
> >> or false depencies, so that it is usuable in embedded devices.
> >
> > First of all, I largely consider this a joke.  All real-life embedded
> > kernel builds take place on hosted platforms; anything else seems to be
> > done "just because it can be done", as a kind of show-off art project.
> > Cute, but hardly worth impeding the rest of the kernel community for.
>
> Let me explain why it is not a joke for me, although yes I agree it's
> a funny way of how software engineering works.
>
> My argument on thin dependencies indeed mostly holds for run-time
> dependencies (to reduce size) but also for build dependency (to reduce
> complexity)*.

I usually just point to the gnucash 1.6 release as where this sort of thing 
leads if you ignore it long enough:
http://lwn.net/2001/0614/

These days, a more modern example is the way that after even the gentoo folks 
gave up on trying to build openoffice (and shipped prebuilt binaries of it in 
their "build everything from source code" OS), Open Office's own developers 
(Continue reading)

Markus Heidelberg | 4 Jan 04:38 2009
Picon

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

Rob Landley, 04.01.2009:
> On Saturday 03 January 2009 18:37:12 Leon Woestenberg wrote:
> > My argument on thin dependencies indeed mostly holds for run-time
> > dependencies (to reduce size) but also for build dependency (to reduce
> > complexity)*.
> 
> I usually just point to the gnucash 1.6 release as where this sort of thing 
> leads if you ignore it long enough:
> http://lwn.net/2001/0614/
> 
> These days, a more modern example is the way that after even the gentoo folks 
> gave up on trying to build openoffice (and shipped prebuilt binaries of it in 
> their "build everything from source code" OS), Open Office's own developers 
> described that project "profoundly sick" and "stagnating" ( 
> http://developers.slashdot.org/article.pl?sid=08/12/28/0124230 ).

Now that you mention this the second time, I have to ask where you have
this information from. Since I use Gentoo, I was always able to compile
OpenOffice (version 1, 2 and now 3) myself.

At the same time, it was always possible to use prebuilt packages as an
alternative - the same way as it is possible for a few other packages
(Firefox, Thunderbird, Seamonkey, maybe more). But AFAIK compiling from
source is still the preferred method.

Markus

Rob Landley | 4 Jan 05:57 2009
Picon

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

On Saturday 03 January 2009 21:38:13 Markus Heidelberg wrote:
> Rob Landley, 04.01.2009:
> Now that you mention this the second time, I have to ask where you have
> this information from. Since I use Gentoo, I was always able to compile
> OpenOffice (version 1, 2 and now 3) myself.

The gentoo panel last OLS.  (Either a BOF or a tutorial, I don't remember 
which.)  It's still _possible_ to build it from source, but they created a 
separate "openoffice-bin" package for the sole purpose of _not_ compiling it 
from source, and it's what they recommend installing.

> At the same time, it was always possible to use prebuilt packages as an
> alternative - the same way as it is possible for a few other packages
> (Firefox, Thunderbird, Seamonkey, maybe more). But AFAIK compiling from
> source is still the preferred method.

Apparently not for Open Office.

First hit googling "gentoo openoffice install":
http://grokdoc.net/index.php/Gentoo-OpenOffice.org

The next two hits are bug tracker entries, and the one after that:
http://www.linuxforums.org/forum/gentoo-linux-help/71086-installing-
openoffice-question.html

Contains this cut and paste from emerge output:

These are the packages that would be merged, in order:

Calculating dependencies
(Continue reading)

Rob Landley | 4 Jan 03:06 2009
Picon

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

On Saturday 03 January 2009 17:03:11 H. Peter Anvin wrote:
> Leon Woestenberg wrote:
> > I agree with Rob that the amount of required dependencies should be
> > kept to a minimum.
> >
> > If we only use 0.5% of a certain language (or: dependent package),
> > then rather implement that 0.5% in the existing language.
> >
> > Dependencies very quickly become dependency hell. If A requires B,
> > then A also inherits all (future) requirements of B, etc. etc.
> >
> > In my daily software development work with Linux and GNU software in
> > general, 10% of it is spent fighting/removing these extremely "thin"
> > or false depencies, so that it is usuable in embedded devices.
>
> First of all, I largely consider this a joke.

Yes, I've noticed this.  The fact multiple other people do _not_ consider this 
a joke doesn't seem to have sunk in yet.

> All real-life embedded
> kernel builds take place on hosted platforms; anything else seems to be
> done "just because it can be done", as a kind of show-off art project.
> Cute, but hardly worth impeding the rest of the kernel community for.
>
> We're not talking about general platform dependencies here, but build
> dependencies for the kernel.  A platform that can build the kernel is
> not a small platform.

The kernel didn't need perl to build until 2.6.25.  For 17 years, this 
(Continue reading)

H. Peter Anvin | 4 Jan 03:14 2009

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

Rob Landley wrote:
> 
>> The new patches have *more* environmental
>> dependencies than that ever did.
> 
> Could you please be a little more specific?
> 

In this case, you're assuming that every version of every shell this is
going to get involved with is going to do math correctly with the
requisite precision, which is nowhere guaranteed, I'm pretty sure.

>> Third, if someone actually cares to do it right, I have a smallish
>> bignum library at http://git.zytor.com/?p=lib/pbn.git;a=summary that
>> might be a starting point.
> 
> This doesn't _need_ bignum support.  It maxes out around 72 bits and the 
> _result_ can't use more than about $SHIFT bits because you're dividing by the 
> amount you shifted, so just chop off the bottom 32 bits, do a normal 64 bit 
> division on the top (it has to fit), and then do the same division on the 
> appropriate shifted remainder, and combine the results.  This is easy because 
> when the shift _is_ 32 bits or more, the bottom 32 bits all have to be zeroes 
> so you don't even have to mask and add, just shift the remainder left 32 bits 
> so you can continue the divide.
> 
> Pulling out perl isn't always a good alternative to thinking about the 
> problem.
> 

Neither is open-coding a bignum operation instead of relying on an
(Continue reading)

Rob Landley | 4 Jan 07:29 2009
Picon

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

On Saturday 03 January 2009 20:14:44 H. Peter Anvin wrote:
> Rob Landley wrote:
> >> The new patches have *more* environmental
> >> dependencies than that ever did.
> >
> > Could you please be a little more specific?
>
> In this case, you're assuming that every version of every shell this is
> going to get involved with is going to do math correctly with the
> requisite precision, which is nowhere guaranteed, I'm pretty sure.

Well, SUSv3 requires that the shell support signed long arithmetic:
http://www.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06_04

And the LP64 standard says that on 64 bit systems, long must be 64 bit:
http://www.unix.org/whitepapers/64bit.html

Now the potential weakness there is that on 32 bit systems, shells might only 
support 32 bit math instead of 64 bit math.  (You'll notice I have a test for 
that.)  However, bash has supported 64 bit math on 32 bit systems since at 
least 2003.  (I keep a Red Hat 9 test image around because it had 50% market 
share when it shipped, so the majority of "old" Linux systems still in use 
_are_ RH9 or similar.  It also has the oldest gcc version the kernel still 
claims to build under.)  Busybox ash can also support 64 bit math on 32 bit 
hosts, and I just confirmed that the dash in the 32 bit xubuntu 8.10 supports 
64 bit math as well.

(It would also be possible to do a similar overflow avoiding trick to do the 
lot entirely in 32 bit math, but given that the three main shells in use on 
Linux _do_ support 64 bit math on 32 bit hosts and are _required_ to support 
(Continue reading)

Pádraig Brady | 15 Jan 15:32 2009

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

Rob Landley wrote:
> Implementing something by hand isn't _always_ a good alternative, sure.  That 
> would be the "thinking about the problem" part.  In this instance, avoiding 
> overflow is trivial.  (If 1<<-1 didn't wrap around, it wouldn't even need the 
> if statement.)

I don't think this affects your script but it's worth noting
that both bash and ksh use arithmetic rather than logical shift
for the >> operator.

Now arithmetic shift is not useful on 2's compliment machines,
and moreover it's compiler dependent as to whether arithmetic
or logical shift is done for >>. Therefore to increase usefulness
and decrease ambiguity, shells really should only shift unsigned
variables internally.

I know the opengroup spec says to use signed ints, but I think
that is intended to disambiguate input and output, rather than defining
internal operations. This is correct I think since the POSIX spec says
you can even use floating point internally if you like.

I asked the bash maintainer who said he would need clarification
from the austin group (CC'd) before changing anything.

cheers,
Pádraig.
Pádraig Brady | 15 Jan 17:18 2009

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

Pádraig Brady wrote:
> Rob Landley wrote:
>> Implementing something by hand isn't _always_ a good alternative, sure.  That 
>> would be the "thinking about the problem" part.  In this instance, avoiding 
>> overflow is trivial.  (If 1<<-1 didn't wrap around, it wouldn't even need the 
>> if statement.)
> 
> I don't think this affects your script but it's worth noting
> that both bash and ksh use arithmetic rather than logical shift
> for the >> operator.
> 
> Now arithmetic shift is not useful on 2's compliment machines,
> and moreover it's compiler dependent as to whether arithmetic
> or logical shift is done for >>. Therefore to increase usefulness
> and decrease ambiguity, shells really should only shift unsigned
> variables internally.
> 
> I know the opengroup spec says to use signed ints, but I think
> that is intended to disambiguate input and output, rather than defining
> internal operations. This is correct I think since the POSIX spec says
> you can even use floating point internally if you like.
> 
> I asked the bash maintainer who said he would need clarification
> from the austin group (CC'd) before changing anything.

I should have given a few examples sorry.

Sample output from bash with the patch below:

  $ printf "%x\n" $((0x8000000000000000>>1))
(Continue reading)

Jamie Lokier | 4 Jan 03:36 2009

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

Rob Landley wrote:
> This doesn't _need_ bignum support.  It maxes out around 72 bits and
> the _result_ can't use more than about $SHIFT bits because you're
> dividing by the amount you shifted, so just chop off the bottom 32
> bits, do a normal 64 bit division on the top (it has to fit), and
> then do the same division on the appropriate shifted remainder, and
> combine the results.  This is easy because when the shift _is_ 32
> bits or more, the bottom 32 bits all have to be zeroes so you don't
> even have to mask and add, just shift the remainder left 32 bits so
> you can continue the divide.
> 
> Pulling out perl isn't always a good alternative to thinking about
> the problem.

Related query:

Does the Perl script being replaced use 64-bit arithmetic?  Because
many Perl installations only do 32-bit arithmetic.

If the Perl version works in 32-bit arithmetic, why does the shell
version not do the same thing?

-- Jamie
H. Peter Anvin | 4 Jan 03:39 2009

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

Jamie Lokier wrote:
> 
> Related query:
> 
> Does the Perl script being replaced use 64-bit arithmetic?  Because
> many Perl installations only do 32-bit arithmetic.
> 
> If the Perl version works in 32-bit arithmetic, why does the shell
> version not do the same thing?
> 

The Perl version uses Math::BigInt, a Perl standard module (with a
canned-values fallback for ancient or minimal Perl installations) to do
arbitrary precision arithmetic.

The original version also produced constants that could be used with
64-bit values, but since gcc doesn't support 128-bit arithmetic on
32-bit platforms (gcc *does* support 128-bit arithmetic on 64-bit
platforms) we didn't end up using it and removed them, although the code
to generate them can still be activated.

	-hpa

--

-- 
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.

H. Peter Anvin | 4 Jan 03:43 2009

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

H. Peter Anvin wrote:
> Jamie Lokier wrote:
>> Related query:
>>
>> Does the Perl script being replaced use 64-bit arithmetic?  Because
>> many Perl installations only do 32-bit arithmetic.
>>
>> If the Perl version works in 32-bit arithmetic, why does the shell
>> version not do the same thing?
>>
> 
> The Perl version uses Math::BigInt, a Perl standard module (with a
> canned-values fallback for ancient or minimal Perl installations) to do
> arbitrary precision arithmetic.
> 
> The original version also produced constants that could be used with
> 64-bit values, but since gcc doesn't support 128-bit arithmetic on
> 32-bit platforms (gcc *does* support 128-bit arithmetic on 64-bit
> platforms) we didn't end up using it and removed them, although the code
> to generate them can still be activated.
> 

I should point out that we really *should* use same kind of techniques
on 64 bits as well.  Even though the likelihood of overflow is much less
there (and the use of the LCD reduces it further) it is nonzero.
However, the places that were most seriously affected were all operating
on 32-bit input (int), and therefore the overflow-free 64-bit code never
got written.

	-hpa
(Continue reading)

Paul Mundt | 4 Jan 04:06 2009

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

On Sat, Jan 03, 2009 at 08:06:47PM -0600, Rob Landley wrote:
> On Saturday 03 January 2009 17:03:11 H. Peter Anvin wrote:
> > Leon Woestenberg wrote:
> > > I agree with Rob that the amount of required dependencies should be
> > > kept to a minimum.
> > >
> > > If we only use 0.5% of a certain language (or: dependent package),
> > > then rather implement that 0.5% in the existing language.
> > >
> > > Dependencies very quickly become dependency hell. If A requires B,
> > > then A also inherits all (future) requirements of B, etc. etc.
> > >
> > > In my daily software development work with Linux and GNU software in
> > > general, 10% of it is spent fighting/removing these extremely "thin"
> > > or false depencies, so that it is usuable in embedded devices.
> >
> > First of all, I largely consider this a joke.
> 
> Yes, I've noticed this.  The fact multiple other people do _not_ consider this 
> a joke doesn't seem to have sunk in yet.
> 
Let's look at the rationale presented so far in this thread:

	1 - Being able to build the kernel natively on a constrained
	target is useful, regardless of whether it is being used for
	regression/stress testing or for headers installation or whatever
	else.

	2 - Cross-compiling perl is hard.

(Continue reading)

Leon Woestenberg | 4 Jan 11:23 2009
Picon

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

Hello,

On Sun, Jan 4, 2009 at 4:06 AM, Paul Mundt <lethal <at> linux-sh.org> wrote:
> Let's look at the rationale presented so far in this thread:
>
>        1 - Being able to build the kernel natively on a constrained
>        target is useful, regardless of whether it is being used for
>        regression/stress testing or for headers installation or whatever
>        else.
>
>        2 - Cross-compiling perl is hard.
>
>        3 - Some oddly constrained target distributions manage to ship
>        with an entire toolchain yet fail to provide any implementation
>        of perl.
>
>        4 - It wasn't required before.
>
> If there is anything I missed, feel free to add it to the list. It was
> difficult to extract even those 4 from the ranting.
>

2 is not hard.

5. Tool *version* dependency is hard to get right. When cross-building
30 software packages all requiring native perl, we probably need to
build a few versions of perl (native), one for each set of packages.

Regards,
--

-- 
(Continue reading)

Vladimir Dronnikov | 4 Jan 17:22 2009
Picon

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

> Let's look at the rationale presented so far in this thread:
>
>        1 - Being able to build the kernel natively on a constrained
>        target is useful, regardless of whether it is being used for
>        regression/stress testing or for headers installation or whatever
>        else.
>
>        2 - Cross-compiling perl is hard.
>
>        3 - Some oddly constrained target distributions manage to ship
>        with an entire toolchain yet fail to provide any implementation
>        of perl.
>
>        4 - It wasn't required before.

OK, let's see.

> If you have enough space and CPU power and
> complete build environment to crunch away at the kernel for stress
> testing natively, you can do the same with building perl and negating
> point #2.

It seems you meant "If you have enough space ... for stress testing
natively, you _MUST ALSO_ do the same with building perl  _TO JUST_
negate point #2". From this point of view your further arguments are
obvious.

>
> #2 is another byproduct of your environment and generally a non-issue.
>
(Continue reading)

Leon Woestenberg | 4 Jan 11:23 2009
Picon

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

Hello,

On Sun, Jan 4, 2009 at 4:06 AM, Paul Mundt <lethal <at> linux-sh.org> wrote:
> Let's look at the rationale presented so far in this thread:
>
>        1 - Being able to build the kernel natively on a constrained
>        target is useful, regardless of whether it is being used for
>        regression/stress testing or for headers installation or whatever
>        else.
>
>        2 - Cross-compiling perl is hard.
>
>        3 - Some oddly constrained target distributions manage to ship
>        with an entire toolchain yet fail to provide any implementation
>        of perl.
>
>        4 - It wasn't required before.
>
> If there is anything I missed, feel free to add it to the list. It was
> difficult to extract even those 4 from the ranting.
>

2 is not hard.

5. Tool *version* dependency is hard to get right. When cross-building
30 software packages all requiring native perl, we probably need to
build a few versions of perl (native), one for each set of packages.

Regards,
--

-- 
(Continue reading)

Mike Frysinger | 8 Jan 14:29 2009
Picon

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

On Sun, Jan 4, 2009 at 05:23, Leon Woestenberg wrote:
> On Sun, Jan 4, 2009 at 4:06 AM, Paul Mundt <lethal <at> linux-sh.org> wrote:
>> Let's look at the rationale presented so far in this thread:
>>
>>        2 - Cross-compiling perl is hard.
>
> 2 is not hard.

i dont know where you're getting this mythical version of perl, but
anything beyond micro-perl is a pita.  and micro-perl is practically
worthless.  maybe for kernel/
-mike
Bernd Petrovitsch | 11 Jan 13:45 2009
Picon

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

On Son, 2009-01-04 at 11:23 +0100, Leon Woestenberg wrote:
[...]
> On Sun, Jan 4, 2009 at 4:06 AM, Paul Mundt <lethal <at> linux-sh.org> wrote:
[...]

I'm ignoring the "cross-compile perl" issue - haven't tried it for
years.

> 5. Tool *version* dependency is hard to get right. When cross-building
> 30 software packages all requiring native perl, we probably need to
> build a few versions of perl (native), one for each set of packages.

perl is IMHO special (and quite different to others - including
especially autotools): perl5 is used widely enough so that "one somewhat
recent version" should cover all of 30 software packages.
The hard part are the CPAN modules and their versions which are really a
PITA.
As long as you don't use modules from CPAN, "perl5" should be specific
enough.

	Bernd
--

-- 
Firmix Software GmbH                   http://www.firmix.at/
mobil: +43 664 4416156                 fax: +43 1 7890849-55
          Embedded Linux Development and Services

Mark A. Miller | 12 Jan 04:36 2009

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

On Sun, Jan 11, 2009 at 6:45 AM, Bernd Petrovitsch <bernd <at> firmix.at> wrote:
>
> On Son, 2009-01-04 at 11:23 +0100, Leon Woestenberg wrote:
> [...]
> > On Sun, Jan 4, 2009 at 4:06 AM, Paul Mundt <lethal <at> linux-sh.org> wrote:
> [...]
>
> I'm ignoring the "cross-compile perl" issue - haven't tried it for
> years.
>
> > 5. Tool *version* dependency is hard to get right. When cross-building
> > 30 software packages all requiring native perl, we probably need to
> > build a few versions of perl (native), one for each set of packages.
>
> perl is IMHO special (and quite different to others - including
> especially autotools): perl5 is used widely enough so that "one somewhat
> recent version" should cover all of 30 software packages.
> The hard part are the CPAN modules and their versions which are really a
> PITA.
> As long as you don't use modules from CPAN, "perl5" should be specific
> enough.
>
>        Bernd
> --
> Firmix Software GmbH                   http://www.firmix.at/
> mobil: +43 664 4416156                 fax: +43 1 7890849-55
>          Embedded Linux Development and Services

Actually, something that has amused me during this discussion, is that
right now, the latest stable Perl (5.8.8) does not compile correctly
(Continue reading)

H. Peter Anvin | 12 Jan 06:11 2009

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

Mark A. Miller wrote:
> 
> Actually, something that has amused me during this discussion, is that
> right now, the latest stable Perl (5.8.8) does not compile correctly
> on a uclibc host...
>

The latest stable Perl is 5.10.0, and the latest of the 5.8 series is 5.8.9.

	-hpa

--

-- 
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.

Mark A. Miller | 12 Jan 06:23 2009

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

On Sun, Jan 11, 2009 at 11:11 PM, H. Peter Anvin <hpa <at> zytor.com> wrote:
> Mark A. Miller wrote:
>>
>> Actually, something that has amused me during this discussion, is that
>> right now, the latest stable Perl (5.8.8) does not compile correctly
>> on a uclibc host...
>>
>
> The latest stable Perl is 5.10.0, and the latest of the 5.8 series is 5.8.9.
>
>        -hpa
>
> --
> H. Peter Anvin, Intel Open Source Technology Center
> I work for Intel.  I don't speak on their behalf.

My mistake. However,
http://perl5.git.perl.org/perl.git/blob_plain/HEAD:/hints/linux.sh
still has the issue, specifically:

if test -L /lib/libc.so.6; then
    libc=`ls -l /lib/libc.so.6 | awk '{print $NF}'`
    libc=/lib/$libc
fi

So, my version was incorrect, yet the problem still exists. I've got a
patch, need to submit it, yet Perl *does not compile* on a uclibc
target *as is*.

And this is why we should avoid adding new tools to build the kernel,
(Continue reading)

Paul Mundt | 12 Jan 09:20 2009

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

On Sun, Jan 11, 2009 at 09:36:58PM -0600, Mark A. Miller wrote:
> Actually, something that has amused me during this discussion, is that
> right now, the latest stable Perl (5.8.8) does not compile correctly
> on a uclibc host, which is typically what you want for embedded
> systems, which is why you'd bother to cross compile. (Albeit I was
> doing this natively under QEMU with a gcc/uclibc toolchain).
> 
> I'll have a patch submitted upstream shortly, but basically the
> hints/linux.sh assumes *obviously* you're linking to glibc. I've made
> that less libc dependent, looking for either glibc or uclibc.
> 
There are plenty that ship with glibc, too. What you "want" for embedded
systems depends entirely on the application for the device, not general
hand-wavy assertions. We (Renesas) ship BSPs on both precisely because of
this reason, and eglibc will probably factor in at some point later on
too.

> So without patching Perl, by adding it to the kernel configuration,
> it's broken being able to compile the kernel on most embedded
> architectures.
> 
This again has nothing to do with the kernel and everything to do with
your distribution. I use perl on uclibc natively just fine, it is
possible there are patches that have not been merged upstream, but this
is again an entirely separate issue.

You seem to be confusing the fact that people who build distributions and
people who use them are one and the same, whereas "most" embedded
developers are going to be using pre-built distributions provided with
their reference hardware, and locking it down during productization. The
(Continue reading)

Mark A. Miller | 12 Jan 10:18 2009

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

On Mon, Jan 12, 2009 at 2:20 AM, Paul Mundt <lethal <at> linux-sh.org> wrote:
> On Sun, Jan 11, 2009 at 09:36:58PM -0600, Mark A. Miller wrote:
>> Actually, something that has amused me during this discussion, is that
>> right now, the latest stable Perl (5.8.8) does not compile correctly
>> on a uclibc host, which is typically what you want for embedded
>> systems, which is why you'd bother to cross compile. (Albeit I was
>> doing this natively under QEMU with a gcc/uclibc toolchain).
>>
>> I'll have a patch submitted upstream shortly, but basically the
>> hints/linux.sh assumes *obviously* you're linking to glibc. I've made
>> that less libc dependent, looking for either glibc or uclibc.
>>
> There are plenty that ship with glibc, too. What you "want" for embedded
> systems depends entirely on the application for the device, not general
> hand-wavy assertions. We (Renesas) ship BSPs on both precisely because of
> this reason, and eglibc will probably factor in at some point later on
> too.
>
>> So without patching Perl, by adding it to the kernel configuration,
>> it's broken being able to compile the kernel on most embedded
>> architectures.
>>
> This again has nothing to do with the kernel and everything to do with
> your distribution. I use perl on uclibc natively just fine, it is
> possible there are patches that have not been merged upstream, but this
> is again an entirely separate issue.
>
> You seem to be confusing the fact that people who build distributions and
> people who use them are one and the same, whereas "most" embedded
> developers are going to be using pre-built distributions provided with
(Continue reading)

Mark A. Miller | 12 Jan 10:18 2009

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

On Mon, Jan 12, 2009 at 2:20 AM, Paul Mundt <lethal <at> linux-sh.org> wrote:
> On Sun, Jan 11, 2009 at 09:36:58PM -0600, Mark A. Miller wrote:
>> Actually, something that has amused me during this discussion, is that
>> right now, the latest stable Perl (5.8.8) does not compile correctly
>> on a uclibc host, which is typically what you want for embedded
>> systems, which is why you'd bother to cross compile. (Albeit I was
>> doing this natively under QEMU with a gcc/uclibc toolchain).
>>
>> I'll have a patch submitted upstream shortly, but basically the
>> hints/linux.sh assumes *obviously* you're linking to glibc. I've made
>> that less libc dependent, looking for either glibc or uclibc.
>>
> There are plenty that ship with glibc, too. What you "want" for embedded
> systems depends entirely on the application for the device, not general
> hand-wavy assertions. We (Renesas) ship BSPs on both precisely because of
> this reason, and eglibc will probably factor in at some point later on
> too.
>
>> So without patching Perl, by adding it to the kernel configuration,
>> it's broken being able to compile the kernel on most embedded
>> architectures.
>>
> This again has nothing to do with the kernel and everything to do with
> your distribution. I use perl on uclibc natively just fine, it is
> possible there are patches that have not been merged upstream, but this
> is again an entirely separate issue.
>
> You seem to be confusing the fact that people who build distributions and
> people who use them are one and the same, whereas "most" embedded
> developers are going to be using pre-built distributions provided with
(Continue reading)

Paul Mundt | 12 Jan 10:41 2009

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

On Mon, Jan 12, 2009 at 03:18:53AM -0600, Mark A. Miller wrote:
> On Mon, Jan 12, 2009 at 2:20 AM, Paul Mundt <lethal <at> linux-sh.org> wrote:
> 
> Paul:
> I initially wrote a rather details response to your e-mail. But
> instead, I shall quote a previous e-mail of yours:
> 
> > I will repeat again that no one has provided a
> > _single_ reason for why they are unable to provide perl within their
> > constrained environment. Until that happens, this entire thread is a
> > joke.
> 
> And I did so. And you have disregarded it. That makes me question the
> logic of your fervent vehemence against such "Perl is perhaps not a
> good idea in the kernel build infrastructure" people like myself.
> 
You have done no such thing. You have provided an example as to why you
personally find perl objectionable, and in your previous mail you even
noted that you have patches for perl to fix the build issues, so there is
no fundamental reason why you are _unable_ to provide perl in your
environment. It all comes down to the fact you don't want to be bothered
to put the effort in to getting perl setup in your environment, which
quite frankly is no one's problem but your own.

Personally I think perl (and python for that matter) is a terrible
language and I wouldn't use it for anything of consequence, but again,
that is my personal opinion and has nothing to do with regards to the
build system and whether it was the better tool for the job as perceived
by the people that elected to implemented infrastructure using it. I
choose not to use it for my own projects, but I have no qualms with those
(Continue reading)

Mark A. Miller | 12 Jan 11:03 2009

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

On Mon, Jan 12, 2009 at 3:41 AM, Paul Mundt <lethal <at> linux-sh.org> wrote:

> I will repeat, there has not been a single coherent argument against what
> makes perl inherently incapable of being supported.

You're right, this thread is worthless with that particular mindset,
Paul. Not, perhaps that the tool in question is brittle, and prone to
potentially break on more architectures than the current make and C
code infrastructure, no, your stance is, unless Perl *cannot* run on
that particular architecture and environment, it has a valid place in
the kernel because it was chosen by certain developers.

And you're right, I did patch around Perl in order to get it to build
under a MIPSEL uclibc environment.

But yes, this particular thread with you *is* worthless, because it's
an argument who's stance is not worth fighting against because of it's
flawed premise.

--

-- 
Mark A. Miller
mark <at> mirell.org
Rob Landley | 12 Jan 18:56 2009
Picon

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

On Monday 12 January 2009 03:41:22 Paul Mundt wrote:
> Personally I think perl (and python for that matter) is a terrible
> language and I wouldn't use it for anything of consequence, but again,
> that is my personal opinion and has nothing to do with regards to the
> build system and whether it was the better tool for the job as perceived
> by the people that elected to implemented infrastructure using it. I
> choose not to use it for my own projects, but I have no qualms with those
> that do.

Apparently you have qualms with people who chose to reimplement the perl bits 
in one of the languages kernel developers already needed to know this time 
last year (shell, C, make).

> The kernel does not need to provide justification for every change that
> goes on as long as there is a reasonable attempt not to break things for
> other people.

I submitted a change, you insisted that I justify it to your satisfaction.  
That pretty much summarizes your participation in this thread.

> The onus is (and always has been) on you to demonstrate why
> perl is an unreasonable dependency to push on embedded developers, and
> you have failed utterly at demonstrating this in any sort of coherent
> fashion.

Large additional dependencies without benefit are unreasonable.  My primary 
objection to perl is that it happens to be an additional large dependency 
without a correspondingly large benefit.

Your position is not internally consistent.  There was no need to scrutinize 
(Continue reading)

Alan Cox | 12 Jan 19:04 2009
Picon

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

> I didn't say it was incapable of being supported.  We're _capable_ of 
> reimplementing the entire kernel in perl 

Which perl. What minor release, what day of the week syntax.

Ask anyone in the distribution business about the joy of perl and you can
listen to the screams for hours.

Perl5 has no formal grammar and you cannot tell what perl of the week
does and perl of last week doesn't do.

That makes it a bad candidate for our toolchain dependencies.

Alan
--
   "I don't want world domination if it means I have to deal with more
	people like this" - Mike Wangsmo
Mark A. Miller | 12 Jan 11:03 2009

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

On Mon, Jan 12, 2009 at 3:41 AM, Paul Mundt <lethal <at> linux-sh.org> wrote:

> I will repeat, there has not been a single coherent argument against what
> makes perl inherently incapable of being supported.

You're right, this thread is worthless with that particular mindset,
Paul. Not, perhaps that the tool in question is brittle, and prone to
potentially break on more architectures than the current make and C
code infrastructure, no, your stance is, unless Perl *cannot* run on
that particular architecture and environment, it has a valid place in
the kernel because it was chosen by certain developers.

And you're right, I did patch around Perl in order to get it to build
under a MIPSEL uclibc environment.

But yes, this particular thread with you *is* worthless, because it's
an argument who's stance is not worth fighting against because of it's
flawed premise.

--

-- 
Mark A. Miller
mark <at> mirell.org
Paul Mundt | 12 Jan 11:34 2009

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

On Mon, Jan 12, 2009 at 04:03:32AM -0600, Mark A. Miller wrote:
> On Mon, Jan 12, 2009 at 3:41 AM, Paul Mundt <lethal <at> linux-sh.org> wrote:
> > I will repeat, there has not been a single coherent argument against what
> > makes perl inherently incapable of being supported.
> 
> You're right, this thread is worthless with that particular mindset,
> Paul. Not, perhaps that the tool in question is brittle, and prone to
> potentially break on more architectures than the current make and C
> code infrastructure, no, your stance is, unless Perl *cannot* run on
> that particular architecture and environment, it has a valid place in
> the kernel because it was chosen by certain developers.
> 
Nonsense. I singled out that point because that was the one you were
replying to in the first place. I itemized the objections in this thread
earlier on and attempted to indicate why they were not applicable in this
context, and asked people to add to it if anything had been overlooked.
If you want to play semantics, do it somewhere else.

If the tool is brittle and constantly breaking, we will see bug reports,
and re-evaluate the support position. This hasn't happened to date in the
context of the kernel build system, so there is no point in even bringing
this up.

Anyways, given that you haven't contributed anything to the kernel and
are therefore perhaps unfamiliar with how things work, I attempted to
show you why the kernel made the decision it did and what it would take
to change that. You have from the beginning only wanted to focus on idle
semantics and refused to re-evaluate your own position on what precisely
it is you find to be problematic in the first place.

(Continue reading)

Peter Korsgaard | 12 Jan 09:27 2009
Picon

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

>>>>> "Mark" == Mark A Miller <mark <at> mirell.org> writes:

 Mark> And for H. Peter Anvin, before you refer to such uses as compiling the
 Mark> kernel under a native environment as a "piece of art", please be aware
 Mark> that the mainstream embedded development environment, buildroot, is
 Mark> also attempting to utilize QEMU for a "sanity check" on the
 Mark> environment.

That's for verifying that the rootfs'es actually work, not for
building stuff.

--

-- 
Bye, Peter Korsgaard
Rob Landley | 12 Jan 18:45 2009
Picon

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

On Monday 12 January 2009 02:27:32 Peter Korsgaard wrote:
> >>>>> "Mark" == Mark A Miller <mark <at> mirell.org> writes:
>
>  Mark> And for H. Peter Anvin, before you refer to such uses as compiling
> the Mark> kernel under a native environment as a "piece of art", please be
> aware Mark> that the mainstream embedded development environment,
> buildroot, is Mark> also attempting to utilize QEMU for a "sanity check" on
> the Mark> environment.
>
> That's for verifying that the rootfs'es actually work, not for
> building stuff.

Not in my case.

Rob
Vladimir Dronnikov | 4 Jan 17:22 2009
Picon

Re: PATCH [0/3]: Simplify the kernel build by removing perl.

> Let's look at the rationale presented so far in this thread:
>
>        1 - Being able to build the kernel natively on a constrained
>        target is useful, regardless of whether it is being used for
>        regression/stress testing or for headers installation or whatever
>        else.
>
>        2 - Cross-compiling perl is hard.
>
>        3 - Some oddly constrained target distributions manage to ship
>        with an entire toolchain yet fail to provide any implementation
>        of perl.
>
>        4 - It wasn't required before.

OK, let's see.

> If you have enough space and CPU power and
> complete build environment to crunch away at the kernel for stress
> testing natively, you can do the same with building perl and negating
> point #2.

It seems you meant "If you have enough space ... for stress testing
natively, you _MUST ALSO_ do the same with building perl  _TO JUST_
negate point #2". From this point of view your further arguments are
obvious.

>
> #2 is another byproduct of your environment and generally a non-issue.
>
(Continue reading)

Rob Landley | 4 Jan 02:24 2009
Picon

PATCH [0/3]: Simplify the kernel build by removing perl v2

Here's an updated set of patches to remove use of perl from the kernel build's 
"hot path" (roughly defined as "make allnoconfig; make; make 
headers_install").

This update incorporates feedback from Sam Ravnborg, Ted Tso, Joe Perches, 
Ingo Oeser, and others.  It also fixes an integer overflow error in the first 
patch, and all three scripts have been retested under dash.
Rob Landley | 4 Jan 02:27 2009
Picon

PATCH [1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh (v2)

From: Rob Landley <rob <at> landley.net>

Replace kernel/timeconst.pl with kernel/timeconst.sh.  The new shell script
is much simpler, about 1/4 the size, and runs on Red Hat 9 from 2003.

It requires a shell which can do 64 bit math, such as bash, busybox ash,
or dash running on a 64 bit host.

Changes from previous version:

Redo ADJ32 math to avoid integer overflow for small HZ sizes (such as 24 or
122).  In the pathological case (HZ=1) both versions now produce
USEC_TO_HZ_ADJ32 of 0x7ffff79c842fa.  (See source comments for details.)

Also expand usage message, add error message when no 64 bit math available in
shell (and suggest some shells that support it), add whitespace around
operators in equations, added underscores before __KERNEL_TIMECONST_H, change
makefile so script is responsible for creating output file, make script delete 
output file on error, change shebang to #!/bin/sh and test with dash and bash.

Signed-off-by: Rob Landley <rob <at> landley.net>
---

 kernel/Makefile     |    4 
 kernel/timeconst.pl |  378 ------------------------------------------
 kernel/timeconst.sh |  149 ++++++++++++++++
 3 files changed, 151 insertions(+), 380 deletions(-)

--- linux-2.6.28/kernel/timeconst.pl	2008-12-24 17:26:37.000000000 -0600
+++ /dev/null	2008-11-21 04:46:41.000000000 -0600
(Continue reading)

David Vrabel | 4 Jan 03:48 2009

Re: PATCH [1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh (v2)

Rob Landley wrote:
> From: Rob Landley <rob <at> landley.net>
> 
> Replace kernel/timeconst.pl with kernel/timeconst.sh.  The new shell script
> is much simpler, about 1/4 the size, and runs on Red Hat 9 from 2003.
> 
> It requires a shell which can do 64 bit math, such as bash, busybox ash,
> or dash running on a 64 bit host.

I use Ubuntu (hence dash) on 32 bit systems so I think this needs to 
work with dash on 32 bit hosts.

David
Rob Landley | 4 Jan 21:21 2009
Picon

Re: PATCH [1/3]: Replace kernel/timeconst.pl with kernel/timeconst.sh (v2)

On Saturday 03 January 2009 20:48:21 David Vrabel wrote:
> Rob Landley wrote:
> > From: Rob Landley <rob <at> landley.net>
> >
> > Replace kernel/timeconst.pl with kernel/timeconst.sh.  The new shell
> > script is much simpler, about 1/4 the size, and runs on Red Hat 9 from
> > 2003.
> >
> > It requires a shell which can do 64 bit math, such as bash, busybox ash,
> > or dash running on a 64 bit host.
>
> I use Ubuntu (hence dash) on 32 bit systems so I think this needs to
> work with dash on 32 bit hosts.

I have a qemu/images directory full of various OS images for testing purposes.

I just fired up my jeos 7.10 image to make sure that even the most stripped-
down version of Ubuntu ("just enough operating system) still installs bash by 
default, and it does.  (It doesn't install a development toolchain, but it 
does install bash.)

I also installed a 32 bit xubuntu 8.10 image (which took 4 hours for some 
reason, and which also has bash), and explicitly tested its 32-bit 
"/bin/dash", and that did 64-bit math too.  So current versions of dash do 
offer 64 bit math on 32 bit platforms.

> David

Rob
(Continue reading)

Rob Landley | 4 Jan 02:28 2009
Picon

PATCH [2/3]: Remove perl from make headers_install.

From: Rob Landley <rob <at> landley.net>

Remove perl from make headers_install by replacing a perl script (doing
a simple regex search and replace) with a smaller and faster shell script
implementation.  The new shell script is a single for loop calling sed and
piping its output through unifdef to produce the target file.

Changes from previous version: Added help text and a check for the right
number of arguments.  Removed unused ARCH input from script and makefile
(the makefile incorporates ARCH into INDIR, so the script doesn't care),
fixed a whitespace mistake in the makefile pointed out by Sam Ravnborg,
changed the shebang to #!/bin/sh and tested under bash and dash.

put_changelog_here

Signed-off-by: Rob Landley <rob <at> landley.net>
---

 scripts/Makefile.headersinst |    6 ++--
 scripts/headers_install.pl   |   46 ---------------------------------
 scripts/headers_install.sh   |   36 +++++++++++++++++++++++++
 3 files changed, 39 insertions(+), 49 deletions(-)

diff -ruN linux-2.6.28/scripts/headers_install.sh linux-2.6.28-new/scripts/headers_install.sh
--- linux-2.6.28/scripts/headers_install.sh	1969-12-31 18:00:00.000000000 -0600
+++ linux-2.6.28-new/scripts/headers_install.sh	2009-01-02 22:35:17.000000000 -0600
 <at>  <at>  -0,0 +1,36  <at>  <at> 
+#!/bin/sh
+
+if [ $# -lt 2 ]
(Continue reading)

Rob Landley | 4 Jan 02:29 2009
Picon

PATCH [3/3]: Convert kernel/cpu/mkcapflags.pl to kernel/cpu/mkcapflags.sh

From: Rob Landley <rob <at> landley.net>

Convert kernel/cpu/mkcapflags.pl to kernel/cpu/mkcapflags.sh.

This script generates kernel/cpu/capflags.c from include/asm/cpufeature.h.

Changes from last time: changed shebang to #!/bin/sh and tested under bash
and dash.

Signed-off-by: Rob Landley <rob <at> landley.net>
---

 arch/x86/kernel/cpu/Makefile      |    4 +--
 arch/x86/kernel/cpu/mkcapflags.pl |   32 ----------------------------
 arch/x86/kernel/cpu/mkcapflags.sh |   28 ++++++++++++++++++++++++
 3 files changed, 30 insertions(+), 34 deletions(-)

diff -ruN linux-2.6.28/arch/x86/kernel/cpu/Makefile linux-2.6.28-new/arch/x86/kernel/cpu/Makefile
--- linux-2.6.28/arch/x86/kernel/cpu/Makefile	2008-12-24 17:26:37.000000000 -0600
+++ linux-2.6.28-new/arch/x86/kernel/cpu/Makefile	2009-01-02 01:10:00.000000000 -0600
 <at>  <at>  -23,10 +23,10  <at>  <at> 
 obj-$(CONFIG_X86_LOCAL_APIC) += perfctr-watchdog.o

 quiet_cmd_mkcapflags = MKCAP   $ <at> 
-      cmd_mkcapflags = $(PERL) $(srctree)/$(src)/mkcapflags.pl $< $ <at> 
+      cmd_mkcapflags = $(CONFIG_SHELL) $(srctree)/$(src)/mkcapflags.sh $< $ <at> 

 cpufeature = $(src)/../../include/asm/cpufeature.h

 targets += capflags.c
(Continue reading)


Gmane