Discussing the Comprehensive Common Lisp Archive Net

Gary King | 8 Oct 2008 16:12

rfc - delay the parsing of system components

I just finished a rough cut of an ASDF overall that makes three  
largish changes:

1. delays the parsing of system components until needed by traverse.  
In particular, this means that the :components clause of defsystem  
form happens after that system's dependencies have been loaded rather  
than when the ASD file itself is loaded.

2. rewrites traverse as a set of methods that immediately call perform  
(rather than building up a list of perform actions that  are then  
called in operate).

3. does away with :do-first and moves what it did into :in-order-to

The main motivation for change #1 is to be able to define new  
operations and source file classes in a system's dependencies and have  
them used in that systems definitions; the main motivation#2 was #1:  
it does no good to delay parsing of the defsystem form if traverse has  
to complete the parse before anything else happens!. As for #3, while  
doing three things at once is just very human (and besides, it made #2  
easier).

I've got several hours of work left cleaning up loose ends and moving  
restarts and with-compilation-units around into better places.

What I'd appreciate is any high-level feedback about the idea of these  
changes and whether or not I've missed some horrible interaction that  
nullifies the whole direction. I hope to post the suggested changes  
later in the week.

(Continue reading)

Richard M Kreuter | 8 Oct 2008 17:12

Re: rfc - delay the parsing of system components

Gary King writes:
> I just finished a rough cut of an ASDF overall that makes three  
> largish changes:
> 
> 1. delays the parsing of system components until needed by traverse.  
> In particular, this means that the :components clause of defsystem  
> form happens after that system's dependencies have been loaded rather  
> than when the ASD file itself is loaded.
>
> 2. rewrites traverse as a set of methods that immediately call perform  
> (rather than building up a list of perform actions that  are then  
> called in operate).

In the plan-before-execute design that ASDF's always had, methods that
implement pieces of the plan construction protocol (OPERATION-DONE-P,
COMPONENT-DEPENDS-ON, et al.) can rely on being executed before any
steps are carried out in the current run.  Since the protocol doesn't
have any solid semantics, extensions' methods can do anything, really;
executing them mid-way through execution can expose them to conditions
that are arbitrarily unlike what such methods have been written to
reason about.

> 3. does away with :do-first and moves what it did into :in-order-to
>
> The main motivation for change #1 is to be able to define new  
> operations and source file classes in a system's dependencies and have  
> them used in that systems definitions; 

The idiomatic solution for this is to load ASDF extensions in an
EVAL-WHEN at the beginning of a .asd file.  But maybe what you're after

(Continue reading)

Gary King | 13 Oct 2008 02:08

Re: rfc - delay the parsing of system components

Hi Richard, et. al.

Thanks for your comments and thoughts.

> In the plan-before-execute design that ASDF's always had, methods that
> implement pieces of the plan construction protocol (OPERATION-DONE-P,
> COMPONENT-DEPENDS-ON, et al.) can rely on being executed before any
> steps are carried out in the current run.

This is true but I have my doubts that it will matter. Methods like  
COMPONENT-DEPENDS-ON shouldn't really have anything to do with the  
environment (either X depends on Y or it doesn't). Methods like  
operation-done-p should depend on a component and its children (not  
its parents) and thus would not be altered by the proposed change.

> The idiomatic solution for this is to load ASDF extensions in an
> EVAL-WHEN at the beginning of a .asd file.  But maybe what you're  
> after
> is representing the fact that system FOO needs ASDF extension BAR,  
> which
> isn't explicit with the EVAL-WHEN solution.  If that's what you  
> need, it
> might suffice to add a slot for this to SYSTEM and to modify
> PARSE-COMPONENT-FORM to hoist out the slot's initarg and do the LOAD- 
> OP
> on those extensions before looping over the components.

It's true that my proposal isn't required to make ASDF jump through  
the hoops I want it to. IMHO, the eval-when solution is ugly and while  
adding another slot is alright, it seems to me that I should be able

(Continue reading)

Christophe Rhodes | 13 Oct 2008 08:33

Re: rfc - delay the parsing of system components

Gary King <gwking <at> metabang.com> writes:

>> I think #2 is an at-least-in-theory incompatible change.  It might not
>> break anything in the wild, but that can't be known in advance.  It
>> might also lead to a better defsystem than ASDF, but I'm not sure that
>> ASDF should be breaking compatibility with itself at this point.
>
> Hmmm, evolve or die <smile>.

No problem with that, but biological taxonomists have learnt that it's
useful to give mutually-incompatible species different names.  Some
day, computer programmers might see the point of that, too.

(In other words: please do make code easier to write, but please also
give the resulting software a different name.)

Cheers,

Christophe

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/

Gary King | 13 Oct 2008 14:51

Re: rfc - delay the parsing of system components

>> Hmmm, evolve or die <smile>.
>
> No problem with that, but biological taxonomists have learnt that it's
> useful to give mutually-incompatible species different names.  Some
> day, computer programmers might see the point of that, too.
>
> (In other words: please do make code easier to write, but please also
> give the resulting software a different name.)

The question reduces to whether or not this is a different species. I  
don't think it is. Richard correctly points out that my proposal might  
cause problems in the "wild" but there is no evidence for this  
(yet...). To my mind, I've kept the same syntax and ASDF does the same  
operations in the same order (at least in as much as ASDF ever did the  
same things in the same order...). The only change is that instead of  
planning everything before executing, ASDF would now execute as it  
went. I don't see this as incompatible. Do you?

thanks,
--
Gary Warren King, metabang.com
Cell: (413) 559 8738
Fax: (206) 338-4052
gwkkwg on Skype * garethsan on AIM

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/

(Continue reading)

Christophe Rhodes | 13 Oct 2008 15:08

Re: rfc - delay the parsing of system components

Gary King <gwking <at> metabang.com> writes:

> The question reduces to whether or not this is a different species. I
> don't think it is. Richard correctly points out that my proposal might
> cause problems in the "wild" but there is no evidence for this
> (yet...). To my mind, I've kept the same syntax and ASDF does the same
> operations in the same order (at least in as much as ASDF ever did the
> same things in the same order...). The only change is that instead of
> planning everything before executing, ASDF would now execute as it
> went. I don't see this as incompatible. Do you?

Yes, I do, because (as I understand it) it means that performing
operations on components now wraps operations on subcomponents, rather
than happening afterwards.

If this causes no difficulties in the wild, then it also costs you
very little to call this "GHJK", provide an in-the-wild-compatible
"ASDF" package, and advertise 99.9% (or whatever) ASDF-compatibility,
without the horrible broken bits and with additional capabilities.

The cost of not doing so is, in my experience, huge: over time,
references to "ASDF"-the-software accumulate, and suddenly no-one
knows whether a particular reference applies to before or after your
change, or both.

I don't see why you even _want_ to keep the same name, frankly.

Best,

Christophe

(Continue reading)

Nikodemus Siivola | 13 Oct 2008 21:16

Re: rfc - delay the parsing of system components

On Mon, Oct 13, 2008 at 4:08 PM, Christophe Rhodes <csr21 <at> cantab.net> wrote:

> I don't see why you even _want_ to keep the same name, frankly.

I agree. "plan, then execute" is one of the major (mis?)features of
ASDF. While I have often wanted it, calling a system that behaves like
that ASDF is just confusing.

While the change may be practically backwards-incompatible, it is a
major headache to deal with when people start actually using it --
since system depending on it will *not* work on older ASDF versions.

YASDF, ASDF2, ASDF.REC, GKSDF, whatever -- and I *like* the idea, but
not calling it ASDF.

Cheers,

 -- Nikodemus

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/

Daniel Barlow | 13 Oct 2008 22:27

Re: rfc - delay the parsing of system components

"Nikodemus Siivola" <nikodemus <at> random-state.net> writes:

> On Mon, Oct 13, 2008 at 4:08 PM, Christophe Rhodes <csr21 <at> cantab.net> wrote:
>
>> I don't see why you even _want_ to keep the same name, frankly.
>
> I agree. "plan, then execute" is one of the major (mis?)features of
> ASDF. While I have often wanted it, calling a system that behaves like
> that ASDF is just confusing.

Excuse me if I'm about to rehash old ground.

I cannot entirely remember the reasons for plan-then-execute, but asdf
was changed to work that way very early in its history and there must
have been some kind of justification at the time.  One point is that if
you want to write operation-done-p methods which compare file dates
then it's much easier to think about if you're comparing the dates
from *before* you started work than from some arbitrary point halfway
through the operation - it's also safer in the face of clock drift.

KMP does the the same thing in his "Description of Large Systems"
paper, though doesn't give much justification for it beyond "Because a
system can be asked to produce a plan for an operation such as
compilation without actually performing the operation, it is possible
to write programs which inspect the plan, possibly optimizing it or
presenting it for for interactive approval, before executing it. "

http://www.nhplace.com/kent/Papers/Large-Systems.html

A lot of this was driven by IRC discussion with Kevin Rosenberg - I

(Continue reading)

Gary King | 14 Oct 2008 02:06

Re: rfc - delay the parsing of system components

OK. I hear what everyone is saying and appreciate the feedback. I'd  
like to back up (or out!) a bit and reconsider my goals:

* I don't want to replace ASDF; it;s good, it works, it's in use;  
adding a new one just muddies the waters.
* I do want to make it easier to use extensions (new operations and  
file types) in a system file.

My first plan was the proposal to delay parsing and muck with  
traverse / perform. Here is two others:

1. add keyword

* As in #0, Delay the parsing of system components (so that these can  
come later)
* Add a new keyword (as RIchard Kreuter suggested) that loads  
extensions and which is processed "outside" of and before traverse/ 
perform. Let's call it :requires
* An operation will first process the requires form, then expand the  
component forms, then run the existing traverse/perform

2. Like #1 only without the new keyword.

* ASDF behaves as it does now except that a systems :depends-on is  
processed before and separately from the processing of its components.

As I see it, #2 this keeps the syntax the same but runs a higher risk  
of breaking things whereas #1 adds syntax but can cause no breakage.

Both of these lose the ability to (easily) wrap restarts around

(Continue reading)

Richard M Kreuter | 14 Oct 2008 16:57

A plurality of defsystems [was Re: rfc - delay the parsing of system components ]

Gary King writes:

> * I don't want to replace ASDF; it;s good, it works, it's in use;
> adding a new one just muddies the waters.

For my part, I don't see any reason why the Lisp community should
constrain itself to having exactly one defsystem facility.  ASDF is
useful, but it has shortcomings, and since it's never clear which of
ASDF's details are intended or accident, and in any case which details
are important to anybody, ISTM that it would probably be a good thing if
alternative defsystems could co-exist.

So I wonder how hard it would be to introduce an interface that
insulated users and programs from the details of how a system's
construction and loading happens to be implemented.  Something like the
following seems as though it would suffice:

--
BUILD-SYSTEM <NAME> &key &allow-other-keys                   [Function]

Call successive functions in *SYSTEM-BUILDER-HOOKS* using <NAME>, the
list of keywords, and :ALLOW-OTHER-KEYS T, until one returns true, and
signals an error if no function in *SYSTEM-BUILDER-HOOKS* returns true.

*SYSTEM-BUILDER-HOOKS*                                       [Variable]

A list of designators for functions that take a string designator and a
list of keywords.  When called, each function will attempt to build a
system identified by the string designator and possibly some of the
keyword arguments, and return true if such a system been built, or NIL

(Continue reading)

Gary King | 14 Oct 2008 23:59

Re: A plurality of defsystems [was Re: rfc - delay the parsing of system components ]

Hi Richard,

How meta! (pun intended)

On it's face, I think that your idea is great.  I do, however, have  
two concerns that revolve around your statement that:

> For my part, I don't see any reason why the Lisp community should
> constrain itself to having exactly one defsystem facility.  ASDF is
> useful, but it has shortcomings, and since it's never clear which of
> ASDF's details are intended or accident, and in any case which details
> are important to anybody, ISTM that it would probably be a good  
> thing if
> alternative defsystems could co-exist.

Sometimes, I think Lisp is nothing but a plethora of ways to do the  
same thing. Maybe it's only because I'm not familiar with the  
communities, but AFAIK, if one wants to install something in Ruby, you  
look for a GEM; if you want to get something for Python, you go to  
CPAN, etc. Lisp tends to make things much harder because we keep  
inventing things. Each new defsystem adds another barrier to entry  
because it requires more setup and thinking. Choice is good; too much  
choice is paralyzing.

To my mind, ASDF has _won_. It is the default and most Lisps are  
bundled with a version of it. I think that we should be improving it,  
not replacing it. The last thing we (as a Lisp community) need is  
another defsystem -- even a 99% compatible one.

My concerns therefore are:

(Continue reading)

GP lisper | 15 Oct 2008 01:44

Re: A plurality of defsystems [was Re: rfc - delay the parsing of system components ]

   From: Gary King <gwking <at> metabang.com>
   Date: Tue, 14 Oct 2008 17:59:59 -0400

   Lisp tends to make things much harder because we keep inventing
   things. Each new defsystem adds another barrier to entry because it
   requires more setup and thinking. Choice is good; too much choice
   is paralyzing.

I must agee with this, having felt this way many times.  However, I
can also point at UFFI and CFFI as a somewhat similar situation to
what is going on in ASDF.  The names are similar, the functions are
similar, and the latter one improved upon the former.

I also know the confusion factor from 'same-name but incompatible with
the past'.  Pick a rather similar name, and give the ideas the test of
time.  People do beat pathways to code that moves them in a needed
direction.  Frankly worrying about the name, rather than the code
doesn't seem productive.

r

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/

Richard M Kreuter | 14 Oct 2008 15:08

Re: rfc - delay the parsing of system components

Gary King writes:

> 1. add keyword
> 
> * As in #0, Delay the parsing of system components (so that these can  
> come later)
> * Add a new keyword (as RIchard Kreuter suggested) that loads  
> extensions and which is processed "outside" of and before traverse/ 
> perform. Let's call it :requires
> * An operation will first process the requires form, then expand the  
> component forms, then run the existing traverse/perform

I didn't think my suggestion through.  One way to use an ASDF extension
is to include package-qualified symbols in the DEFSYSTEM form, and so
it's not generally possible to read a DEFSYSTEM form before the relevant
extensions are loaded.  Example:

(defsystem foo
   :requires "my-asdf-stuff"
   :components ((:file "foo" :class my-asdf-stuff:whizbang-source-file)))

So my suggestion won't work for all cases.  Sorry about that.

--
Richard

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world

(Continue reading)

Robert Goldman | 14 Oct 2008 03:14

Re: rfc - delay the parsing of system components

Gary King wrote:
> OK. I hear what everyone is saying and appreciate the feedback. I'd  
> like to back up (or out!) a bit and reconsider my goals:
> 
> * I don't want to replace ASDF; it;s good, it works, it's in use;  
> adding a new one just muddies the waters.
> * I do want to make it easier to use extensions (new operations and  
> file types) in a system file.
> 
> My first plan was the proposal to delay parsing and muck with  
> traverse / perform. Here is two others:
> 
> 1. add keyword
> 
> * As in #0, Delay the parsing of system components (so that these can  
> come later)
> * Add a new keyword (as RIchard Kreuter suggested) that loads  
> extensions and which is processed "outside" of and before traverse/ 
> perform. Let's call it :requires

Nitpicky suggestion --- let's call it something else to avoid confusing
with the (deprecated) CL notion of require.

> * An operation will first process the requires form, then expand the  
> component forms, then run the existing traverse/perform

While we're messing with this, any chance of getting something which
wraps around the whole operation on a component so that we can, e.g.,
impose dynamic bindings around the entire load-op as applied to a
system?  I suppose that's off-topic, but the current overhaul seems like

(Continue reading)

Return

Return to gmane.lisp.cclan.general.

Search Archive

Language

Change language

Options

Current view: Threads only / Showing only 20 lines / Not hiding cited text.
Change to All messages, whole messages, or hide cited text.

Post a message
NNTP Newsgroup
Classic Gmane web interface
XML

XML

RSS Feed
List Information

About Gmane