Discussion of Emacs support for multi-directional text

Martin Duerst | 22 Nov 02:30

Picon

Re: state of BIDI within stable GNU emacs (UTF8) nikud

At 13:53 05/11/21, Eli Zaretskii wrote:

 >I once wrote here the reason: Pango, like fribidi, is a batch-mode
 >reordering library: you hand it a buffer with text and it returns it
 >reordered for display.
 >
 >By contrast, Emacs needs a reordering function that could be called
 >repeatedly, and will on every call return the next character in the
 >visual order.  This is because the Emacs display engine walks the
 >buffer one character at a time and decides how to display it based on
 >data structures built by the application.

It looks to me like you could just use fribidi or so, with an
additional layer that caches the batch-mode results and returns
them one-by-one. You'd want to cache quite a bit of stuff even
in a character walking implementation, I guess, although there
are probably some shortcuts you can make (haven't thought about
that yet in detail).

 >I wrote such a sequential version of bidi reordering, and it was
 >integrated into redisplay on a special branch of Emacs CVS.  But no
 >one had time to debug it and expand it, unfortunately.

Can you give a definitive pointer (ideally URI) to that branch?
Thanks!

Regards,    Martin.

Eli Zaretskii | 22 Nov 05:32

Picon

Re: state of BIDI within stable GNU emacs (UTF8) nikud

> Date: Tue, 22 Nov 2005 10:30:06 +0900
> From: Martin Duerst <duerst <at> it.aoyama.ac.jp>
> Cc: emacs-bidi <at> gnu.org
> 
> It looks to me like you could just use fribidi or so, with an
> additional layer that caches the batch-mode results and returns
> them one-by-one.

This was considered.  Such a caching would be a performance hit, and
it also requires significant changes in the logic and structure of the
Emacs display code.

>  >I wrote such a sequential version of bidi reordering, and it was
>  >integrated into redisplay on a special branch of Emacs CVS.  But no
>  >one had time to debug it and expand it, unfortunately.
> 
> Can you give a definitive pointer (ideally URI) to that branch?

The branch name is emacs-bidi.  The instructions to use the Emacs CVS
can be found here:

   http://savannah.gnu.org/cvs/?group=emacs

Gregg Reynolds | 28 Nov 18:29

Re: state of BIDI within stable GNU emacs (UTF8) nikud

Eli Zaretskii wrote:
>>Date: Tue, 22 Nov 2005 10:30:06 +0900
...
>> >I wrote such a sequential version of bidi reordering, and it was
>> >integrated into redisplay on a special branch of Emacs CVS.  But no
>> >one had time to debug it and expand it, unfortunately.
>>
>>Can you give a definitive pointer (ideally URI) to that branch?
> 
> The branch name is emacs-bidi.  The instructions to use the Emacs CVS
> can be found here:
> 
>    http://savannah.gnu.org/cvs/?group=emacs

emacs-bidi Not found.  Nor is it in the "Browse Sources Repository" 
link.  Any other place to look?

Michael Welsh Duggan | 28 Nov 22:03

Picon

Re: state of BIDI within stable GNU emacs (UTF8) nikud

Gregg Reynolds <gar <at> arabink.com> writes:

> Eli Zaretskii wrote:
>>>Date: Tue, 22 Nov 2005 10:30:06 +0900
> ...
>>> >I wrote such a sequential version of bidi reordering, and it was
>>> >integrated into redisplay on a special branch of Emacs CVS.  But no
>>> >one had time to debug it and expand it, unfortunately.
>>>
>>>Can you give a definitive pointer (ideally URI) to that branch?
>> The branch name is emacs-bidi.  The instructions to use the Emacs
>> CVS
>> can be found here:
>>    http://savannah.gnu.org/cvs/?group=emacs
>
> emacs-bidi Not found.  Nor is it in the "Browse Sources Repository"
> link.  Any other place to look?

Without actually knowing, I am assuming the branch tag is
emacs-bidi, as can be seen at 
http://savannah.gnu.org/cgi-bin/viewcvs/emacs/emacs/?only_with_tag=emacs-bidi

You should be able to check it out with 
cvs -d :ext:anoncvs <at> savannah.gnu.org:/cvsroot/emacs co emacs -r emacs-bidi

Apologies if I have rendered any incorrect advice.

--

-- 
Michael Welsh Duggan
(md5i <at> cs.cmu.edu)

(Continue reading)

Eli Zaretskii | 29 Nov 05:32

Picon

Re: state of BIDI within stable GNU emacs (UTF8) nikud

> From: Michael Welsh Duggan <md5i <at> cs.cmu.edu>
> Cc: Eli Zaretskii <eliz <at> gnu.org>,  emacs-bidi <at> gnu.org
> Date: Mon, 28 Nov 2005 16:03:58 -0500
> 
> >> The branch name is emacs-bidi.  The instructions to use the Emacs
> >> CVS
> >> can be found here:
> >>    http://savannah.gnu.org/cvs/?group=emacs
> >
> > emacs-bidi Not found.  Nor is it in the "Browse Sources Repository"
> > link.  Any other place to look?
> 
> Without actually knowing, I am assuming the branch tag is emacs-bidi

I thought I said precisely that, see above.

Gregg Reynolds | 29 Nov 01:11

Re: state of BIDI within stable GNU emacs (UTF8) nikud

Michael Welsh Duggan wrote:
...
>Without actually knowing, I am assuming the branch tag is
> emacs-bidi, as can be seen at 
> http://savannah.gnu.org/cgi-bin/viewcvs/emacs/emacs/?only_with_tag=emacs-bidi
> 
> You should be able to check it out with 
> cvs -d :ext:anoncvs <at> savannah.gnu.org:/cvsroot/emacs co emacs -r emacs-bidi
> 

Tried that.  Result:

$ cvs -d :ext:anoncvs <at> savannah.gnu.org:/cvsroot/emacs co emacs -r emacs-bidi
cvs server: cannot find module `-r' - ignored
cvs server: cannot find module `emacs-bidi' - ignored
cvs [checkout aborted]: cannot expand modules

However, I can see it at

http://savannah.gnu.org/cgi-bin/viewcvs/emacs/emacs/?only_with_tag=emacs-bidi

So what am I doing wrong with CVS?

thx,

-g

Michael Welsh Duggan | 29 Nov 02:29

Picon

Re: state of BIDI within stable GNU emacs (UTF8) nikud

Gregg Reynolds <gar <at> arabink.com> writes:

> Michael Welsh Duggan wrote:
> ...
>>Without actually knowing, I am assuming the branch tag is
>> emacs-bidi, as can be seen at
>> http://savannah.gnu.org/cgi-bin/viewcvs/emacs/emacs/?only_with_tag=emacs-bidi
>> You should be able to check it out with cvs -d
>> :ext:anoncvs <at> savannah.gnu.org:/cvsroot/emacs co emacs -r emacs-bidi
>> 
>
> Tried that.  Result:
>
> $ cvs -d :ext:anoncvs <at> savannah.gnu.org:/cvsroot/emacs co emacs -r emacs-bidi
> cvs server: cannot find module `-r' - ignored
> cvs server: cannot find module `emacs-bidi' - ignored
> cvs [checkout aborted]: cannot expand modules
>
> However, I can see it at
>
> http://savannah.gnu.org/cgi-bin/viewcvs/emacs/emacs/?only_with_tag=emacs-bidi
>
> So what am I doing wrong with CVS?

That's what I get for not trying things out before posting them.  The
correct incantation is:

cvs -d :ext:anoncvs <at> savannah.gnu.org:/cvsroot/emacs co -r emacs-bidi emacs 

--

--

(Continue reading)

Gregg Reynolds | 29 Nov 03:10

Re: state of BIDI within stable GNU emacs (UTF8) nikud

Michael Welsh Duggan wrote:
...
> That's what I get for not trying things out before posting them.  The
> correct incantation is:
> 
> cvs -d :ext:anoncvs <at> savannah.gnu.org:/cvsroot/emacs co -r emacs-bidi emacs 
> 

Ahh so.  That did the trick.

However, compiling on FreeBSD 6.00, I end up with stuff like the 
following.  I just now did this, so I haven't really investigated as I 
should, so I post this just in case somebody already knows what my 
problem is.  I mean, my problem with compiling. ;-:)

-g

wd=/home/gar/dev/emacs-bidi/emacs/lisp; subdirs=`find $wd -type d 
-print`;  for\
  file in $subdirs; do  case $file in */Old | */RCS | */CVS | */CVS/* | 
*/=* | *\
/obsolete | */term ) ;;  *) wins="$wins $file" ;;  esac;  done;  echo 
Directori\
es: $wins;  LC_ALL=C ../src/emacs -batch --no-site-file --multibyte -l 
cus-dep \
--eval '(setq generated-custom-dependencies-file 
"/home/gar/dev/emacs-bidi/emac\
s/lisp/cus-load.el")' -f custom-make-dependencies $wins
Directories: /home/gar/dev/emacs-bidi/emacs/lisp 
/home/gar/dev/emacs-bidi/emacs\

(Continue reading)

Eli Zaretskii | 29 Nov 05:46

Picon

Re: state of BIDI within stable GNU emacs (UTF8) nikud

> Date: Mon, 28 Nov 2005 20:10:32 -0600
> From: Gregg Reynolds <gar <at> arabink.com>
> Cc: emacs-bidi <at> gnu.org
> 
> However, compiling on FreeBSD 6.00, I end up with stuff like the 
> following.  I just now did this, so I haven't really investigated as I 
> should, so I post this just in case somebody already knows what my 
> problem is.

It's hard to say, since you didn't show the commands you used to build
Emacs; please always do that when you report problems.

> ../src/emacs: not found

This means the real problem is elsewhere: the Emacs binary was not
built for some reason.  Look for some compilation or link error
earlier in the session transcript.

Gregg Reynolds | 29 Nov 12:16

Re: state of BIDI within stable GNU emacs (UTF8) nikud

Eli Zaretskii wrote:
>>Date: Mon, 28 Nov 2005 20:10:32 -0600
>>From: Gregg Reynolds <gar <at> arabink.com>
>>Cc: emacs-bidi <at> gnu.org
>>
>>However, compiling on FreeBSD 6.00, I end up with stuff like the 
>>following.  I just now did this, so I haven't really investigated as I 
>>should, so I post this just in case somebody already knows what my 
>>problem is.
> 
> 
> It's hard to say, since you didn't show the commands you used to build
> Emacs; please always do that when you report problems.
> 
Sorry 'bout that.  I ran "make bootstrap" after cvs'ing the emacs-bidi 
branch.

Turns out that function "bidi_dump_cached_states" in bidi.c prints to 
stderr, but stdio is only included #ifdef TEST_STANDALONE.  Since the 
function doesn't seem to be used, I just #if 0'ed it out and "make 
bootstrap" succeeded.

Regarding TEST_STANDALONE, I don't see any make targets or anything 
obvious for doing that.  Do you have any tools/test stuff remaining for 
this?

What would you think about moving your TEST_STANDALONE code out of 
bidi.c and in to a separate driver program?  Also, would it make sense 
to use a logging framework like Log4c (http://log4c.sourceforge.net/) 
instead of simulate_display?

(Continue reading)

Eli Zaretskii | 29 Nov 22:32

Picon

Re: state of BIDI within stable GNU emacs (UTF8) nikud

> Date: Tue, 29 Nov 2005 05:16:43 -0600
> From: Gregg Reynolds <gar <at> arabink.com>
> CC: emacs-bidi <at> gnu.org
> 
> > It's hard to say, since you didn't show the commands you used to build
> > Emacs; please always do that when you report problems.
> > 
> Sorry 'bout that.  I ran "make bootstrap" after cvs'ing the emacs-bidi 
> branch.

Thanks for working on this.

> Turns out that function "bidi_dump_cached_states" in bidi.c prints to 
> stderr, but stdio is only included #ifdef TEST_STANDALONE.  Since the 
> function doesn't seem to be used, I just #if 0'ed it out and "make 
> bootstrap" succeeded.

I'd prefer a cleaner solution, one that doesn't disable the standalone
configuration.  How about if stdio was included unconditionally? it
cannot hurt, I think.

Also, if you can, please post context diffs of the changes you
suggest, as that makes fixing the problems in the CVS much easier.

> Regarding TEST_STANDALONE, I don't see any make targets or anything 
> obvious for doing that.  Do you have any tools/test stuff remaining for 
> this?

Yes, I do: that's how I tested and debugged the code outside Emacs.
But I don't see any need to make it part of Emacs, since that testing

(Continue reading)

Eli Zaretskii | 29 Nov 22:55

Picon

Re: state of BIDI within stable GNU emacs (UTF8) nikud

> Date: Tue, 29 Nov 2005 23:32:20 +0200
> From: Eli Zaretskii <eliz <at> gnu.org>
> Cc: emacs-bidi <at> gnu.org
> 
> > Turns out that function "bidi_dump_cached_states" in bidi.c prints to 
> > stderr, but stdio is only included #ifdef TEST_STANDALONE.  Since the 
> > function doesn't seem to be used, I just #if 0'ed it out and "make 
> > bootstrap" succeeded.
> 
> I'd prefer a cleaner solution, one that doesn't disable the standalone
> configuration.  How about if stdio was included unconditionally? it
> cannot hurt, I think.

I took a look at the code and it came back to me: The reason
bidi_dump_cached_states function appears to be unused is because it's
supposed to be called from within GDB, whenever you are debugging the
algorithm and want to see its internal state at some arbitrary point.
So including stdio.h unconditionally would be a good solution, since
this function could be useful for debugging the algorithm inside Emacs
as well.

TIA

Gregg Reynolds | 30 Nov 01:13

Re: state of BIDI within stable GNU emacs (UTF8) nikud

Eli Zaretskii wrote:
...
>>Turns out that function "bidi_dump_cached_states" in bidi.c prints to 
>>stderr, but stdio is only included #ifdef TEST_STANDALONE.  Since the 
>>function doesn't seem to be used, I just #if 0'ed it out and "make 
>>bootstrap" succeeded.
> 
> I'd prefer a cleaner solution, one that doesn't disable the standalone
> configuration.  How about if stdio was included unconditionally? it
> cannot hurt, I think.

Probably wouldn't hurt, I guess.  Link-wise, it would presumable not 
lead to code bloat, right?  (I'm a little rusty on this kind of stuff - 
it's been 5+ years since I did serious systems-level programming.)  If 
it would use the same object code used elsewhere in emacs, why not do it?
> 
> Also, if you can, please post context diffs of the changes you
> suggest, as that makes fixing the problems in the CVS much easier.

Will do, if and when I have something useful.  The #if 0 was really just 
to see if it would compile.  Long-term something cleaner would 
definitely be better.  (I suppose I also have to sign some kind of 
papers.  Probably not a problem with my employer, but the timing is a 
bit delicate at the moment.  I'm not sure that "Hey, I want to work on 
free software!" is the right thing to say at the moment.  ;)
> 
> 
>>Regarding TEST_STANDALONE, I don't see any make targets or anything 
>>obvious for doing that.  Do you have any tools/test stuff remaining for 
>>this?

(Continue reading)

Eli Zaretskii | 30 Nov 05:33

Picon

Re: state of BIDI within stable GNU emacs (UTF8) nikud

> Date: Tue, 29 Nov 2005 18:13:58 -0600
> From: Gregg Reynolds <gar <at> arabink.com>
> CC: emacs-bidi <at> gnu.org
> 
> I suppose I also have to sign some kind of papers.  Probably not a
> problem with my employer, but the timing is a bit delicate at the
> moment.

We only need papers for changes longer than 15 lines.

If you cannot sign papers, then just explaining (in English) where is
the buggy code and how, in principle, it should be changed, will do;
someone else who already signed papers (such as myself) can then code
the required change.

> Hmm.  I wonder if we couldn't come up with stand-alone test stuff that 
> would establish the capability of the code w/r/t emacs processing first, 
> and then do in-context emacs testing.  Speculative question.

I thought about this at the time, and decided it would require a
significant effort that wasn't worth it.  One big problem is that I
don't think I know enough about the redisplay code to be able to
specify all the different circumstances under which the low-level
iterator code (into which my bidi code is plugged) is called.  It's
better to let Emacs speak for itself 

> Just for the sake of clarity and hygiene.  In general, if I'm reading a 
> chunk of code, I'd rather not be distracted by code that really should 
> be in a distinct test harness.  Plus, would your bidi code not be of use 
> to anybody outside of emacs?  Looks to me like it might be a useful

(Continue reading)

Martin Duerst | 30 Nov 07:44

Picon

Re: state of BIDI within stable GNU emacs (UTF8) nikud

At 13:33 05/11/30, Eli Zaretskii wrote:

 >I built many small test cases which use every feature.  To that, I
 >added a test case for every bug I've found during debugging.  As for
 >embedding, it's sufficient to test several levels, no need to test all
 >61 of them.

To some extent, this sounds as if the code is actually ready for
practical use as long as it's not for terribly important stuff
where loss of data would hurt seriously.

Is that understanding okay?

Regards,   Martin.

Eli Zaretskii | 30 Nov 21:48

Picon

Re: state of BIDI within stable GNU emacs (UTF8) nikud

> Date: Wed, 30 Nov 2005 15:44:16 +0900
> From: Martin Duerst <duerst <at> it.aoyama.ac.jp>
> Cc: emacs-bidi <at> gnu.org
> 
>  >I built many small test cases which use every feature.  To that, I
>  >added a test case for every bug I've found during debugging.  As for
>  >embedding, it's sufficient to test several levels, no need to test all
>  >61 of them.
> 
> To some extent, this sounds as if the code is actually ready for
> practical use as long as it's not for terribly important stuff
> where loss of data would hurt seriously.
> 
> Is that understanding okay?

Well, yes and no.

Yes, because it was extensively tested as a standalone application.

No, because, when it is called by the display engine, it's quite
possible that the initial conditions (i.e., the place where redisplay
starts to walk the buffer) is something other than what I had in mind
when I designed and coded the algorithm, which could easily cause the
code to become confused, since the Unicode algorithm needs to resync
at the paragraph/line boundaries, find the first strong directional
character, etc.  I think someone told me that the bidi-enabled version
of Emacs crashes fairly quickly after you turn on the bidi display
flag.  (The crashes are actually deliberate calls to `abort' when the
code finds itself in a situation that wasn't supposed to happen, and
Emacs always auto-saves everything before it dies, so you aren't

(Continue reading)

Martin Duerst | 1 Dec 10:14

Picon

Re: state of BIDI within stable GNU emacs (UTF8)nikud

At 05:48 05/12/01, Eli Zaretskii wrote:

 >So some real debugging is, in fact, in order.  The good news are that
 >the bidi reordering code is not invoked unless you turn on the bidi
 >display flag (the buffer-specific variable enable-bidi-display, which
 >is nil by default).

Okay, from your explanations, it sounds as if the following
would work: Do some edits with enable-bidi-display nil, then
switch that on to trigger display, then switch it off again,
and continue editing. Not exactly sure I'd like to work like
this, I just want to know whether that would avoid the bugs.

Also, have you considered calling whatever corrresponds to
cntl-L internally instead of abort when you find an inconsistent
situation, or log that inconsistency in some kind of 'dump'?

Regards,   Martin.

Eli Zaretskii | 2 Dec 08:52

Picon

Re: state of BIDI within stable GNU emacs (UTF8)nikud

> Date: Thu, 01 Dec 2005 18:14:13 +0900
> From: Martin Duerst <duerst <at> it.aoyama.ac.jp>
> Cc: gar <at> arabink.com, emacs-bidi <at> gnu.org
> 
> At 05:48 05/12/01, Eli Zaretskii wrote:
> 
>  >So some real debugging is, in fact, in order.  The good news are that
>  >the bidi reordering code is not invoked unless you turn on the bidi
>  >display flag (the buffer-specific variable enable-bidi-display, which
>  >is nil by default).
> 
> Okay, from your explanations, it sounds as if the following
> would work: Do some edits with enable-bidi-display nil, then
> switch that on to trigger display, then switch it off again,
> and continue editing.

No, what I meant was that, should some grave bugs be uncovered that
could cause you lose work, you could use the editor with the bidi
display flag switched off until the bug is fixed or until you save
your work and exit (whichever happens first).

> Also, have you considered calling whatever corrresponds to
> cntl-L internally instead of abort when you find an inconsistent
> situation, or log that inconsistency in some kind of 'dump'?

It's not clear to me that C-l will fix the display, but that's one
known way of dealing with display problems.

However, the calls to `abort' are intentional: they make sure that
bugs will not go unnoticed.  As I wrote, Emacs will autosave before it

(Continue reading)

Eli Zaretskii | 28 Nov 20:50

Picon

Re: state of BIDI within stable GNU emacs (UTF8) nikud

> Date: Mon, 28 Nov 2005 11:29:03 -0600
> From: Gregg Reynolds <gar <at> arabink.com>
> CC: emacs-bidi <at> gnu.org
> 
> > The branch name is emacs-bidi.  The instructions to use the Emacs CVS
> > can be found here:
> > 
> >    http://savannah.gnu.org/cvs/?group=emacs
> 
> emacs-bidi Not found.

What CVS command(s) you used?  The branch _is_ there, believe me, I
can see it from my machine.

Return

Return to gmane.emacs.bidi.

Search Archive

Language

Change language

Options

Current view: Threads only / Showing only 20 lines / Not hiding cited text.
Change to All messages, whole messages, or hide cited text.

Post a message
NNTP Newsgroup
Classic Gmane web interface
XML

XML

RSS Feed
List Information

About Gmane