headers
Jérémie Courrèges-Anglas | 20 Aug 2012 21:53

bug#12242: Emacs 24.2 RC1 build fails on OpenBSD

Hi.

Sorry for being late.  The RC1 archive fails to build on
OpenBSD/i386 -current (post 5.2) and OpenBSD/macppc -stable (5.1).
Emacs 24.1 was building and running properly.

I use
$ ./configure --without-x && gmake
[snip]
EMACSLOADPATH=/home/jca/src/emacs-24.2/leim/../lisp LC_ALL=C ../src/emacs -batch
--no-site-file --no-site-lisp  -l /home/jca/src/emacs-24.2/leim/../lisp/international/qua
il \
  -f batch-byte-compile-if-not-done quail/CCDOSPY.el quail/Punct.el quail/QJ.el quail/SW.el
quail/TONEPY.el quail/4Corner.el quail/ARRAY30.el quail/ECDICT.el quail/ETZY.e
l quail/Punct-b5.el quail/PY-b5.el quail/QJ-b5.el quail/ZOZY.el quail/tsang-b5.el
quail/quick-b5.el quail/tsang-cns.el quail/quick-cns.el quail/PY.el quail/ZIRANMA.el qua
il/CTLau.el quail/CTLau-b5.el
if [ x`(cd /home/jca/src/emacs-24.2/leim && /bin/pwd)` = x`(/bin/pwd)` ] ; then \
  EMACSLOADPATH=/home/jca/src/emacs-24.2/leim/../lisp LC_ALL=C ../src/emacs -batch
--no-site-file --no-site-lisp -l /home/jca/src/emacs-24.2/leim/../lisp/international/qu
ail \
    --eval "(update-leim-list-file \".\")" ; \
else \
  EMACSLOADPATH=/home/jca/src/emacs-24.2/leim/../lisp LC_ALL=C ../src/emacs -batch
--no-site-file --no-site-lisp -l /home/jca/src/emacs-24.2/leim/../lisp/international/qu
ail \
    --eval "(update-leim-list-file \".\" \"/home/jca/src/emacs-24.2/leim\")" ; \
fi
Updating /home/jca/src/emacs-24.2/leim/leim-list.el ...
Checking /home/jca/src/emacs-24.2/leim/quail/PY.el ...
(Continue reading)

Permalink | Reply |
headers
Eli Zaretskii | 21 Aug 2012 05:03
Picon

bug#12242: Emacs 24.2 RC1 build fails on OpenBSD

> From: jca <at> wxcvbn.org (Jérémie Courrèges-Anglas)
> Date: Mon, 20 Aug 2012 21:53:16 +0200
> 
> Checking /home/jca/src/emacs-24.2/leim/quail/cyril-jis.el ...
> Fatal error (11)Segmentation fault (core dumped) 
> gmake[1]: *** [leim-list.el] Error 139
> gmake[1]: Leaving directory `/home/jca/src/emacs-24.2/leim'
> gmake: *** [leim] Error 2
> $ 
> [...]
> #0  free_bloc (bloc=0x1b8fa80) at ralloc.c:719
> 719       if (heap->first_bloc == bloc)
> (gdb) bt
> #0  free_bloc (bloc=0x1b8fa80) at ralloc.c:719
> [...]
> Using git bisect, I was able to track the build error up to the
> 573c8f870aa68b8c5937524e1a4db645026a3240 git commit id:
> 
>      Backport: Really fix bug #11519, by fixing the last change in ralloc.c.
> 
>       src/ralloc.c (r_alloc_inhibit_buffer_relocation): Fix stupid
>       thinko
>       in the logic of incrementing and decrementing the value of
>       use_relocatable_buffers.
> 
> Sorry for not providing the hg commit id.
> Reverting that commit actually makes rc1 build again, but I have not
> tried to find a proper fix.

I think I see the problem and will fix it later today.
(Continue reading)

Permalink | Reply |
headers
Eli Zaretskii | 21 Aug 2012 18:49
Picon

bug#12242: Emacs 24.2 RC1 build fails on OpenBSD

> Date: Tue, 21 Aug 2012 06:03:37 +0300
> From: Eli Zaretskii <eliz <at> gnu.org>
> Cc: 12242 <at> debbugs.gnu.org
> 
> I think I see the problem and will fix it later today.

Actually, no, I don't think I see the problem.  What does the
following GDB command print in frame #0?

  (gdb) p *heap

Also, could you please run Emacs under GDB, and when it crashes, type
the following commands, which define a new command "heaps", and then
run it?  Like this:

  (gdb) define heaps
  Type commands for definition of "heaps".
  End with a line saying just "end".
  >set $a = first_heap
  >while $a
   >p *$a
   >set $a = $a->next
   >end
  >end
  (gdb) heaps

Please post here everything that GDB prints as result.

Thanks.
(Continue reading)

Permalink | Reply |
headers
Jérémie Courrèges-Anglas | 21 Aug 2012 21:23

bug#12242: Emacs 24.2 RC1 build fails on OpenBSD

Eli Zaretskii <eliz <at> gnu.org> writes:

>> Date: Tue, 21 Aug 2012 06:03:37 +0300
>> From: Eli Zaretskii <eliz <at> gnu.org>
>> Cc: 12242 <at> debbugs.gnu.org
>> 
>> I think I see the problem and will fix it later today.
>
> Actually, no, I don't think I see the problem.  What does the
> following GDB command print in frame #0?
>
>   (gdb) p *heap
>
> Also, could you please run Emacs under GDB, and when it crashes, type
> the following commands, which define a new command "heaps", and then
> run it?  Like this:
>
>   (gdb) define heaps
>   Type commands for definition of "heaps".
>   End with a line saying just "end".
>   >set $a = first_heap
>   >while $a
>    >p *$a
>    >set $a = $a->next
>    >end
>   >end
>   (gdb) heaps
>
> Please post here everything that GDB prints as result.
(Continue reading)

Permalink | Reply |
headers
YAMAMOTO Mitsuharu | 22 Aug 2012 04:35
Picon

bug#12242: Emacs 24.2 RC1 build fails on OpenBSD

>>>>> On Tue, 21 Aug 2012 19:49:52 +0300, Eli Zaretskii <eliz <at> gnu.org> said:

>> Date: Tue, 21 Aug 2012 06:03:37 +0300 From: Eli Zaretskii
>> <eliz <at> gnu.org> Cc: 12242 <at> debbugs.gnu.org
>> 
>> I think I see the problem and will fix it later today.

> Actually, no, I don't think I see the problem.

I took a look at ralloc.c a bit, and I thought that the variable
`use_relocatable_buffers' is not designed to be changed temporarily in
the first place.

I think we should use r_alloc_freeze/r_alloc_thaw instead of
r_alloc_inhibit_buffer_relocation calls.  In that case, we have to
estimate how much memory can be malloc'ed while the relocation is
frozen, so that we can pass an appropriate number to r_alloc_freeze.

				     YAMAMOTO Mitsuharu
				mituharu <at> math.s.chiba-u.ac.jp
Permalink | Reply |
headers
Eli Zaretskii | 22 Aug 2012 05:02
Picon

bug#12242: Emacs 24.2 RC1 build fails on OpenBSD

> Date: Wed, 22 Aug 2012 11:35:26 +0900
> From: YAMAMOTO Mitsuharu <mituharu <at> math.s.chiba-u.ac.jp>
> Cc: jca <at> wxcvbn.org,
> 	12242 <at> debbugs.gnu.org
> 
> I took a look at ralloc.c a bit, and I thought that the variable
> `use_relocatable_buffers' is not designed to be changed temporarily in
> the first place.

Why not?  Can you tell what led you to that conclusion?

My reading of the code is that inhibiting relocation just means that
ralloc.c always asks for more memory when it cannot find a large
enough block in the existing heaps.
Permalink | Reply |
headers
YAMAMOTO Mitsuharu | 22 Aug 2012 05:13
Picon

bug#12242: Emacs 24.2 RC1 build fails on OpenBSD

>>>>> On Wed, 22 Aug 2012 06:02:06 +0300, Eli Zaretskii <eliz <at> gnu.org> said:

>> Date: Wed, 22 Aug 2012 11:35:26 +0900 From: YAMAMOTO Mitsuharu
>> <mituharu <at> math.s.chiba-u.ac.jp> Cc: jca <at> wxcvbn.org,
>> 12242 <at> debbugs.gnu.org
>> 
>> I took a look at ralloc.c a bit, and I thought that the variable
>> `use_relocatable_buffers' is not designed to be changed temporarily
>> in the first place.

> Why not?  Can you tell what led you to that conclusion?

> My reading of the code is that inhibiting relocation just means that
> ralloc.c always asks for more memory when it cannot find a large
> enough block in the existing heaps.

For example, `virtual_break_value' is not updated in that case.  It
can lead to an inconsistent state as reported if r_alloc_sbrk is
called with a positive argument via malloc when
use_relocatable_buffers <= 0, and then it is called with a negative
argument via free when use_relocatable_buffers > 0.

				     YAMAMOTO Mitsuharu
				mituharu <at> math.s.chiba-u.ac.jp
Permalink | Reply |
headers
Eli Zaretskii | 22 Aug 2012 18:58
Picon

bug#12242: Emacs 24.2 RC1 build fails on OpenBSD

> Date: Wed, 22 Aug 2012 12:13:27 +0900
> From: YAMAMOTO Mitsuharu <mituharu <at> math.s.chiba-u.ac.jp>
> Cc: 	jca <at> wxcvbn.org,
> 	12242 <at> debbugs.gnu.org
> 
> >>>>> On Wed, 22 Aug 2012 06:02:06 +0300, Eli Zaretskii <eliz <at> gnu.org> said:
> 
> >> Date: Wed, 22 Aug 2012 11:35:26 +0900 From: YAMAMOTO Mitsuharu
> >> <mituharu <at> math.s.chiba-u.ac.jp> Cc: jca <at> wxcvbn.org,
> >> 12242 <at> debbugs.gnu.org
> >> 
> >> I took a look at ralloc.c a bit, and I thought that the variable
> >> `use_relocatable_buffers' is not designed to be changed temporarily
> >> in the first place.
> 
> > Why not?  Can you tell what led you to that conclusion?
> 
> > My reading of the code is that inhibiting relocation just means that
> > ralloc.c always asks for more memory when it cannot find a large
> > enough block in the existing heaps.
> 
> For example, `virtual_break_value' is not updated in that case.  It
> can lead to an inconsistent state as reported if r_alloc_sbrk is
> called with a positive argument via malloc when
> use_relocatable_buffers <= 0, and then it is called with a negative
> argument via free when use_relocatable_buffers > 0.

I see your point.

Unfortunately, using r_alloc_freeze/r_alloc_thaw doesn't seem to be
(Continue reading)

Permalink | Reply |
headers
YAMAMOTO Mitsuharu | 23 Aug 2012 01:12
Picon

bug#12242: Emacs 24.2 RC1 build fails on OpenBSD

>>>>> On Wed, 22 Aug 2012 19:58:12 +0300, Eli Zaretskii <eliz <at> gnu.org> said:

>>> My reading of the code is that inhibiting relocation just means
>>> that ralloc.c always asks for more memory when it cannot find a
>>> large enough block in the existing heaps.
>> 
>> For example, `virtual_break_value' is not updated in that case.  It
>> can lead to an inconsistent state as reported if r_alloc_sbrk is
>> called with a positive argument via malloc when
>> use_relocatable_buffers <= 0, and then it is called with a negative
>> argument via free when use_relocatable_buffers > 0.

> I see your point.

Sorry, I noticed that the senario I gave above was actually bogus.
Typically free will call r_alloc_sbrk(0) and check the return value
with respect to the area to be reclaimed before calling r_alloc_sbrk
with a negative argument.

Now I don't have a concrete senario to conclude that it is wrong to
change use_relocatable_buffers temporarily.  I'm really sorry if my
previous posts confused you.

				     YAMAMOTO Mitsuharu
				mituharu <at> math.s.chiba-u.ac.jp
Permalink | Reply |
headers
Eli Zaretskii | 23 Aug 2012 18:06
Picon

bug#12242: Emacs 24.2 RC1 build fails on OpenBSD

> Date: Thu, 23 Aug 2012 08:12:37 +0900
> From: YAMAMOTO Mitsuharu <mituharu <at> math.s.chiba-u.ac.jp>
> Cc: YAMAMOTO Mitsuharu <mituharu <at> math.s.chiba-u.ac.jp>,
> 	Stefan Monnier <monnier <at> iro.umontreal.ca>,
> 	Chong Yidong <cyd <at> gnu.org>,
> 	Kenichi Handa <handa <at> m17n.org>,
> 	jca <at> wxcvbn.org,
> 	12242 <at> debbugs.gnu.org
> 
> >>>>> On Wed, 22 Aug 2012 19:58:12 +0300, Eli Zaretskii <eliz <at> gnu.org> said:
> 
> >>> My reading of the code is that inhibiting relocation just means
> >>> that ralloc.c always asks for more memory when it cannot find a
> >>> large enough block in the existing heaps.
> >> 
> >> For example, `virtual_break_value' is not updated in that case.  It
> >> can lead to an inconsistent state as reported if r_alloc_sbrk is
> >> called with a positive argument via malloc when
> >> use_relocatable_buffers <= 0, and then it is called with a negative
> >> argument via free when use_relocatable_buffers > 0.
> 
> > I see your point.
> 
> Sorry, I noticed that the senario I gave above was actually bogus.
> Typically free will call r_alloc_sbrk(0) and check the return value
> with respect to the area to be reclaimed before calling r_alloc_sbrk
> with a negative argument.
> 
> Now I don't have a concrete senario to conclude that it is wrong to
> change use_relocatable_buffers temporarily.  I'm really sorry if my
(Continue reading)

Permalink | Reply |
headers
Chong Yidong | 23 Aug 2012 16:24
Picon

bug#12242: Emacs 24.2 RC1 build fails on OpenBSD

Eli Zaretskii <eliz <at> gnu.org> writes:

> Unfortunately, using r_alloc_freeze/r_alloc_thaw doesn't seem to be
> workable in practice, either.  I tried to use it, and the best patch I
> could come up with (got me through several bootstraps with different
> compiler switches) is below.  It is (a) butt-ugly, and (b) very
> fragile, for the reasons I explain below.
>
> So it sounds like the only practical way out of this mess is to step
> back one notch and talk about bug #11519, which was the cause for
> inhibiting relocations while maybe_unify_char is in progress.  At the
> time, Handa-san promised to work on removing unify_char, but I guess
> that job is not yet done, since it isn't even on the trunk.  Ken'ichi,
> what would it take to do this now?

I don't think it is feasible to rework maybe_unify_char, and test that
fix, and still have a reasonably timely 24.2 release.

Let us consider outright reversion of 108048.  What would be the effect?
Given a choice between a bug (Bug#11519) which is also in 24.1, and
failure-to-compile on OpenBSD as a new regression, the former is far
preferable, unless you can point out some consequence that is more
serious that the discussion in Bug#11519 implies.

Also, if we revert 108048, should we revert 108020 (which is in 24.1) as
well?  Could you explain what problem 108020 causes which 108048 is
supposed to fix?  Does it have symptoms?

Thanks.
(Continue reading)

Permalink | Reply |
headers
Eli Zaretskii | 23 Aug 2012 18:16
Picon

bug#12242: Emacs 24.2 RC1 build fails on OpenBSD

> From: Chong Yidong <cyd <at> gnu.org>
> Cc: YAMAMOTO Mitsuharu <mituharu <at> math.s.chiba-u.ac.jp>,  Stefan Monnier
<monnier <at> iro.umontreal.ca>,  Kenichi Handa <handa <at> m17n.org>,  jca <at> wxcvbn.org,  12242 <at> debbugs.gnu.org
> Date: Thu, 23 Aug 2012 22:24:19 +0800
> 
> Let us consider outright reversion of 108048.  What would be the effect?
> Given a choice between a bug (Bug#11519) which is also in 24.1, and
> failure-to-compile on OpenBSD as a new regression, the former is far
> preferable, unless you can point out some consequence that is more
> serious that the discussion in Bug#11519 implies.

Reverting 108048 (and 108020, see below) will cause trouble to any
system that uses ralloc.c, including (evidently) OpenBSD.  The fact
that the problem was originally found only during bootstrap on Windows
is just a coincidence.  It can strike at any time that regexp search
is used in a text that includes characters which need unification.
It's a bomb waiting to explode.  So I don't recommend that.

Of course, Emacs 24.1 was shipped with that bug, so ...

Can I have a 1-day grace to try debugging the OpenBSD crash?  Jérémie
generously gave me a login on the machine where it happened, and I'd
like a chance to try debugging it.

> Also, if we revert 108048, should we revert 108020 (which is in 24.1) as
> well?  Could you explain what problem 108020 causes which 108048 is
> supposed to fix?  Does it have symptoms?

108020 had a logic flaw in the incrementing and decrementing the flag
that inhibits relocation.  Due to that flaw, the inhibition was not
(Continue reading)

Permalink | Reply |
headers
Chong Yidong | 24 Aug 2012 05:25
Picon

bug#12242: Emacs 24.2 RC1 build fails on OpenBSD

Eli Zaretskii <eliz <at> gnu.org> writes:

> The fact that the problem was originally found only during bootstrap
> on Windows is just a coincidence.  It can strike at any time that
> regexp search is used in a text that includes characters which need
> unification.
>
> Of course, Emacs 24.1 was shipped with that bug, so ...

The fact that the problem showed up so late in the 24.1 pretest, and has
been with us since Emacs 23, indicates that the symptoms are rather rare
in practice.  So reverting seems like the least bad solution.

> Can I have a 1-day grace to try debugging the OpenBSD crash?  Jérémie
> generously gave me a login on the machine where it happened, and I'd
> like a chance to try debugging it.

OK.  Good luck, and thanks for all your efforts with this bug.
Permalink | Reply |
headers
Eli Zaretskii | 24 Aug 2012 10:46
Picon

bug#12242: Emacs 24.2 RC1 build fails on OpenBSD

> From: Chong Yidong <cyd <at> gnu.org>
> Cc: mituharu <at> math.s.chiba-u.ac.jp,  monnier <at> iro.umontreal.ca,  handa <at> m17n.org,  jca <at> wxcvbn.org,  12242 <at> debbugs.gnu.org
> Date: Fri, 24 Aug 2012 11:25:26 +0800
> 
> > Can I have a 1-day grace to try debugging the OpenBSD crash?  Jérémie
> > generously gave me a login on the machine where it happened, and I'd
> > like a chance to try debugging it.
> 
> OK.  Good luck, and thanks for all your efforts with this bug.

I committed a fix for this as emacs-24 branch revision 108120.  It is
somewhat of a phenomenological nature, because I could not actually
catch the entire sequence of calls and actions leading to the crash,
which might have allowed me to fix the root cause where it happens, if
possible.  (It turns out OpenBSD doesn't have hardware watchpoints
support in GDB, which makes catching references to variables painfully
slow, certainly when a deadline is looming.)

I did see that the crash happens because a 'heap' structure recorded
in a memory control block maintained by ralloc.c refers to addresses
that aren't managed as part of the linked list of 'heap' structures.
It is therefore wrong to dereference such bogus 'heap' structures to
update them.  The crash happens because 'heap' pointed to memory that
was beyond our break point; however, I found that we dereference such
bogus 'heap's in more places, and survive that only because, by sheer
luck, they are still within our address space.  (ralloc.c does not
relinquish memory to the system until it has enough excess memory to
justify that.)

The solution I checked in is not to dereference 'heap' pointers that
(Continue reading)

Permalink | Reply |

Gmane