Discussion of the Scintilla free source code editing component

Otto Wyss | 2 Sep 2004 20:52

Discussing another editor/edit-component concept

My wyoEditor ("http://freshmeat.net/projects/wyoeditor/") has moved up
within the top 25 editor projects at sourceforge.net, reaching 2% of
publicity. Well wyoEditor is now at the point where further improving
isn't reasonably possible because it would mean to implement its own
lexer and much other code duplication. So I think it's time to do a
review and start a discussion about another editor/edit-component concept.

I know there is already Sinkworld and some ideas how to make a better
component but as far as I know it doesn't address any of my issues.
Still I'd appreciate if you would independently discuss my concept.

First we have to look at what users expect from an editor and for what
typical tasks they use an editor:
- viewing and editing simple texts like config files or plain text notes
- programming, coding in any language
- viewing and editing structure text (html, xml)
- writing wysiwyg text (text processing)
- etc.

It now would be fantastic if all these where possible through a set of
components which are able to closely interact. I see about the following components:
- edit-component: Just base viewing and editing functionality
- lexer-component: Full lexing including identifier classification
- display-component: Transforming text into nice display
- wysiwyg-component(s): Ditto above but for text processing formats
- spelling-component(s): Ditto lexer but for natural languages

I envision an editor where it's possible to spell check any text or
comments when programming or where syntax descriptions in a document are
colourised. There are many more uses of such combinations. On the other

(Continue reading)

headers

Neil Hodgson | 12 Sep 2004 09:19

Re: Discussing another editor/edit-component concept

Otto Wyss:

> The main problem is to create now an API between all these components
> which is well suited for any future component. This is IMO only possible
> if these components interact through tokens. A token is essentially only
> a distinct set of consecutive chars with identical properties (i.e.
> style). The word "class", the operator "+" or the brace "[" are tokens.

   You should think about situations where there are different viewpoints on
what a token is. In a HTML document the HTML view may treat all of embedded
script as a single token but from the point of view of embedded JavaScript,
it is a snippet of a code made up of many tokens. Some annotations
(spelling, error, URL, nesting level, revision) may include multiple tokens
and parts of tokens. If you are going to perform incremental lexing (or
other incremental analysis) then the final token in the styled region may be
incomplete and need special handling.

   Neil

headers

Otto Wyss | 14 Sep 2004 20:29

Re: Discussing another editor/edit-component concept

Neil Hodgson wrote:
> 
> Otto Wyss:
> 
> > The main problem is to create now an API between all these components
> > which is well suited for any future component. This is IMO only possible
> > if these components interact through tokens. A token is essentially only
> > a distinct set of consecutive chars with identical properties (i.e.
> > style). The word "class", the operator "+" or the brace "[" are tokens.
> 
>    You should think about situations where there are different viewpoints on
> what a token is. In a HTML document the HTML view may treat all of embedded
> script as a single token but from the point of view of embedded JavaScript,
> it is a snippet of a code made up of many tokens. Some annotations
> (spelling, error, URL, nesting level, revision) may include multiple tokens
> and parts of tokens. If you are going to perform incremental lexing (or
> other incremental analysis) then the final token in the styled region may be
> incomplete and need special handling.
> 
I don't think this is a big problem. IMO a html lexer should treat any
consecutive embedded code as one token and leave it to the lexer of the
embedded code. Just look once how wyoEditor can handle all the 100 html
styles in 32 style entries through indirection (sort of incremental).
Okay it might not be as easy as I think but IMO it should be possible to
handle everything this way.

I don't think that annotations can conflict since I can't think of a
resonable sample. It doesn't make sense to spell check an URL and an
error usually apply to the full token. I may overlook something but IMO
there's always a way around it. Nobody expects from a compiler to go on

(Continue reading)

headers

Carsten | 5 Sep 2004 20:43

Re: Discussing another editor/edit-component concept

Hi Otto,

your article is very interesting. I like your clear concept.

Do you know colorer? This is a lexer component, you can find it on 
colorer.sourceforge.net . It has many features and support many 
languages. This component seems to be the ideally lexer component for 
your new editor.

Regards,

Carsten