gmane.editors.j.devel

Russ McManus | 14 Aug 16:28

behavior of DIRECTORY with wild pathnames


I have added a function of my own devising to
asdf:*system-definition-search-functions* that relies on DIRECTORY
doing something smart with wild pathnames.

I have no idea whether this is standard or not.  However I have
certainly noticed a difference between the behavior of abcl and sbcl
in this case.

In abcl on my system I get this:

CL-USER> (directory #P"/home/users/russm/work/lisp/*/*.asd")
NIL

But in sbcl on my system I get this:

CL-USER> (directory #P"/home/users/russm/work/lisp/*/*.asd")
(#P"/home/users/russm/work/lisp/asdf/cclan.asd"
 #P"/home/users/russm/work/lisp/cl-containers/cl-containers-test.asd"
 #P"/home/users/russm/work/lisp/cl-containers/cl-containers.asd"
 #P"/home/users/russm/work/lisp/cl-ppcre-1.3.2/cl-ppcre-test.asd"
 #P"/home/users/russm/work/lisp/cl-ppcre-1.3.2/cl-ppcre.asd"
 #P"/home/users/russm/work/lisp/cl-who-0.11.0/cl-who.asd"
 #P"/home/users/russm/work/lisp/clsql-4.0.3/clsql-aodbc.asd"
 #P"/home/users/russm/work/lisp/clsql-4.0.3/clsql-db2.asd"
 #P"/home/users/russm/work/lisp/clsql-4.0.3/clsql-mysql.asd"
 #P"/home/users/russm/work/lisp/clsql-4.0.3/clsql-odbc.asd"
 #P"/home/users/russm/work/lisp/clsql-4.0.3/clsql-oracle.asd"
 #P"/home/users/russm/work/lisp/clsql-4.0.3/clsql-postgresql-socket.asd"
 #P"/home/users/russm/work/lisp/clsql-4.0.3/clsql-postgresql.asd"

(Continue reading)

headers

Erik Huelsmann | 14 Aug 17:05

Re: behavior of DIRECTORY with wild pathnames

On Thu, Aug 14, 2008 at 4:30 PM, Russ McManus <russell_mcmanus <at> yahoo.com> wrote:
>
> I have added a function of my own devising to
> asdf:*system-definition-search-functions* that relies on DIRECTORY
> doing something smart with wild pathnames.
>
> I have no idea whether this is standard or not.  However I have
> certainly noticed a difference between the behavior of abcl and sbcl
> in this case.
>
> In abcl on my system I get this:
>
> CL-USER> (directory #P"/home/users/russm/work/lisp/*/*.asd")
> NIL
>
> But in sbcl on my system I get this:
>
> CL-USER> (directory #P"/home/users/russm/work/lisp/*/*.asd")
> (#P"/home/users/russm/work/lisp/asdf/cclan.asd"
>  #P"/home/users/russm/work/lisp/cl-containers/cl-containers-test.asd"
>  #P"/home/users/russm/work/lisp/cl-containers/cl-containers.asd"
>  #P"/home/users/russm/work/lisp/cl-ppcre-1.3.2/cl-ppcre-test.asd"
>  #P"/home/users/russm/work/lisp/cl-ppcre-1.3.2/cl-ppcre.asd"
>  #P"/home/users/russm/work/lisp/cl-who-0.11.0/cl-who.asd"
>  #P"/home/users/russm/work/lisp/clsql-4.0.3/clsql-aodbc.asd"
>  #P"/home/users/russm/work/lisp/clsql-4.0.3/clsql-db2.asd"
>  #P"/home/users/russm/work/lisp/clsql-4.0.3/clsql-mysql.asd"
>  #P"/home/users/russm/work/lisp/clsql-4.0.3/clsql-odbc.asd"
>  #P"/home/users/russm/work/lisp/clsql-4.0.3/clsql-oracle.asd"
>  #P"/home/users/russm/work/lisp/clsql-4.0.3/clsql-postgresql-socket.asd"

(Continue reading)

headers

Ville Voutilainen | 14 Aug 17:51

Re: behavior of DIRECTORY with wild pathnames

On Thu, Aug 14, 2008 at 6:05 PM, Erik Huelsmann <ehuels <at> gmail.com> wrote:
> On Thu, Aug 14, 2008 at 4:30 PM, Russ McManus <russell_mcmanus <at> yahoo.com> wrote:
>> I have added a function of my own devising to
>> asdf:*system-definition-search-functions* that relies on DIRECTORY
>> doing something smart with wild pathnames.
> >From my reading of the spec this is exactly the required behaviour.
> So, a fix would be much appreciated! Any ideas on how you will
> approach the issue?

I did a quick analysis of this. directory invokes wild-pathname-p (in
pathnames.lisp), which in turn invokes
%wild-pathname-p, which is implemented in java in Pathname.java. This
further invokes
coerceToPathname in Lisp.java, which then creates a Pathname object by
casting or converting.
Pathname constructor (once again in Pathname.java) just invokes an
init function.

Then, in init, the path is actually split up. It checks whether the
path is relative (by looking
whether it starts with . or ..), handles bang paths for jar files,
expands home directory on
unix platforms, and then finally finds the last occurrence of a path
separator. Then it looks
at a file separator (.) and then tries to locate a single * with a
String.equals function. Looks
a bit fishy to me, for matching wildcard paths..

But, there's hope - init invokes parseDirectory, where wildcards seem
to be handled correctly,

(Continue reading)

headers

Erik Huelsmann | 14 Aug 18:45

Re: behavior of DIRECTORY with wild pathnames

On Thu, Aug 14, 2008 at 5:51 PM, Ville Voutilainen
<ville.voutilainen <at> gmail.com> wrote:
> On Thu, Aug 14, 2008 at 6:05 PM, Erik Huelsmann <ehuels <at> gmail.com> wrote:
>> On Thu, Aug 14, 2008 at 4:30 PM, Russ McManus <russell_mcmanus <at> yahoo.com> wrote:
>>> I have added a function of my own devising to
>>> asdf:*system-definition-search-functions* that relies on DIRECTORY
>>> doing something smart with wild pathnames.
>> >From my reading of the spec this is exactly the required behaviour.
>> So, a fix would be much appreciated! Any ideas on how you will
>> approach the issue?
>
> I did a quick analysis of this. directory invokes wild-pathname-p (in
> pathnames.lisp), which in turn invokes
> %wild-pathname-p, which is implemented in java in Pathname.java. This
> further invokes
> coerceToPathname in Lisp.java, which then creates a Pathname object by
> casting or converting.
> Pathname constructor (once again in Pathname.java) just invokes an
> init function.
>
> Then, in init, the path is actually split up. It checks whether the
> path is relative (by looking
> whether it starts with . or ..), handles bang paths for jar files,
> expands home directory on
> unix platforms, and then finally finds the last occurrence of a path
> separator. Then it looks
> at a file separator (.) and then tries to locate a single * with a
> String.equals function. Looks
> a bit fishy to me, for matching wildcard paths..
>

(Continue reading)

headers

Ville Voutilainen | 14 Aug 19:15

Re: behavior of DIRECTORY with wild pathnames

> Thanks for the analysis. I came to my conclusion by reading about #p
> being equal to #.(parse-namestring ..); this means that the problem
> might be in PARSE-NAMESTRING. Or, at least, the solution should affect
> both PARSE-NAMESTRING and #p.

parse-namestring also invokes the coerceToPathname function at least
in one place, which once again invokes the pathname construction.

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/

headers

Erik Huelsmann | 14 Aug 19:19

Re: behavior of DIRECTORY with wild pathnames

On Thu, Aug 14, 2008 at 6:45 PM, Erik Huelsmann <ehuels <at> gmail.com> wrote:
> On Thu, Aug 14, 2008 at 5:51 PM, Ville Voutilainen
> <ville.voutilainen <at> gmail.com> wrote:
>> On Thu, Aug 14, 2008 at 6:05 PM, Erik Huelsmann <ehuels <at> gmail.com> wrote:
>>> On Thu, Aug 14, 2008 at 4:30 PM, Russ McManus <russell_mcmanus <at> yahoo.com> wrote:
>>>> I have added a function of my own devising to
>>>> asdf:*system-definition-search-functions* that relies on DIRECTORY
>>>> doing something smart with wild pathnames.
>>> >From my reading of the spec this is exactly the required behaviour.
>>> So, a fix would be much appreciated! Any ideas on how you will
>>> approach the issue?
>>
>> I did a quick analysis of this. directory invokes wild-pathname-p (in
>> pathnames.lisp), which in turn invokes
>> %wild-pathname-p, which is implemented in java in Pathname.java. This
>> further invokes
>> coerceToPathname in Lisp.java, which then creates a Pathname object by
>> casting or converting.
>> Pathname constructor (once again in Pathname.java) just invokes an
>> init function.
>>
>> Then, in init, the path is actually split up. It checks whether the
>> path is relative (by looking
>> whether it starts with . or ..), handles bang paths for jar files,
>> expands home directory on
>> unix platforms, and then finally finds the last occurrence of a path
>> separator. Then it looks
>> at a file separator (.) and then tries to locate a single * with a
>> String.equals function. Looks
>> a bit fishy to me, for matching wildcard paths..

(Continue reading)

headers

Ville Voutilainen | 17 Aug 20:45

Re: behavior of DIRECTORY with wild pathnames

On Thu, Aug 14, 2008 at 8:19 PM, Erik Huelsmann <ehuels <at> gmail.com> wrote:
> Based on your remarks I did some research of my own. However, I have
> no time to fix it now. Again, contributions welcome.
> The problem is in directory.lisp, in the fragment below:
>
>    (if (wild-pathname-p pathname)
>        (let ((namestring (directory-namestring pathname)))
>          (when (and namestring (> (length namestring) 0))
>            #+windows
>            (let ((device (pathname-device pathname)))
>              (when device
>                (setq namestring (concatenate 'string device ":" namestring))))
>            (let ((entries (list-directory namestring))
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> Here the code assumes that if the namestring is wild, then the
> directory-namestring isn't. Obviously in our example, that's
> incorrect. Neither does the spec require it to be.

Could you try the attached patch? It was a bit painful to get
together, I'm no lisp guru
and abcl build is not lightning fast. It seems to work correctly but
due to lack of
time I haven't been able to test it that much. It works for a couple
of scenarios,
like in my test function

(defun test-directory ()
  (format t "~a~%" (directory
"/home/ville/projects/lisp/abcl/directory/*/*/*.txt"))
  (format t "~a~%" (directory

(Continue reading)

headers

Erik Huelsmann | 20 Aug 21:47

Re: behavior of DIRECTORY with wild pathnames

Index: directory.lisp
===================================================================
--- directory.lisp	(revision 11286)
+++ directory.lisp	(working copy)
@@ -1,6 +1,7 @@
 ;;; directory.lisp
 ;;;
 ;;; Copyright (C) 2004-2007 Peter Graves
+;;; Copyright (C) 2008 Ville Voutilainen
 ;;; $Id: directory.lisp,v 1.7 2007-03-15 18:56:28 piso Exp $
 ;;;
 ;;; This program is free software; you can redistribute it and/or
@@ -28,6 +29,33 @@
                    :type nil
                    :version nil)))

Hi Ville,

Thanks for another patch! You mentioned not being an experienced Lisp Hacker. In
my comments I'm going to assume you want to learn more about the
different options
you could have used to solve a problem.

+(defun list-directories-with-wildcards (pathname)
+  (let* ((result ())
+	 (directory (pathname-directory pathname))
+	 (first-wild (position-if #'wild-p directory))
+	 (non-wild (or (and first-wild
+			    (subseq directory 0 first-wild)) directory))
+	 (wild (and first-wild (subseq directory first-wild)))

(Continue reading)

headers

Ville Voutilainen | 20 Aug 23:07

Re: behavior of DIRECTORY with wild pathnames

On Wed, Aug 20, 2008 at 10:47 PM, Erik Huelsmann <ehuels <at> gmail.com> wrote:

First of all, thanks for the advice, it's always nice to have this kind of help.

Then to the rebuttals. :)

> Regarding both WILD and NON-WILD, I don't believe the AND operator is guaranteed
> to return the last element, so, best would be to use IF instead of OR, testing
> the FIRST-WILD condition as the IF test.

The hyperspec (http://www.lispworks.com/documentation/HyperSpec/Body/m_and.htm)
disagrees. It claims that the (result of the) last form is returned.

About consing: I wanted to get the algorithm to work, performance was secondary.
I am aware of destructive functions (nconc et al) but I chose to avoid them for
the time being. I'd really like to have profiling data before I start
worrying about
append vs. nconc, for example. Even more so since this is an i/o operation.
I'm not against optimizing it, even if just for not bogging the cpu too much -
I just doubt whether the whole runtime of directory listing is affected much
by consing.

> To that extent, it uses NCONC: the destructive cousin of APPEND.
> APPEND creates a copy
> of all the lists it's being passed. So, for the same reason as the one

Not all of them, I suppose - append is permitted to not copy the last list
given to it.

> I have a question: does your code *only* expand the directories at

(Continue reading)

headers

Ville Voutilainen | 24 Aug 20:19

Re: behavior of DIRECTORY with wild pathnames

So, I have a brand-new patch. Attached, as usual. Some remarks follow.

On Thu, Aug 21, 2008 at 12:07 AM, Ville Voutilainen
<ville.voutilainen <at> gmail.com> wrote:
>> Regarding both WILD and NON-WILD, I don't believe the AND operator is guaranteed
>> to return the last element, so, best would be to use IF instead of OR, testing

I haven't touched this part of the patch.

So, the patch
1) uses nthcdr and nbutlast for the wild/non-wild split
2) uses mapcan
3) uses nconc in the function given to mapcan. I created a separate
helper, because I don't like flets or lambdas in the middle of code.
We have only one list given to mapcan, thus the function used by mapcan
takes only one param. I needed it to take three so I create and return the
function bound to parameters of the creator function
(create-directory-recurser).

>> I have a question: does your code *only* expand the directories at
>> higher levels, or
>> does it also expand the files there? If it also expands the files
>> (only to be removed
>> later because they don't match), this too may be done more efficiently
>> - reducing the
>> need to instantiate objects.

This may still need some additional work, Erik, can you describe
how to do this, in general? I haven't spent much energy on this, but
it's something that cannot be done just by looking at the file names

(Continue reading)

headers

Ville Voutilainen | 25 Aug 22:09

Re: behavior of DIRECTORY with wild pathnames

>>> I have a question: does your code *only* expand the directories at
>>> higher levels, or
>>> does it also expand the files there? If it also expands the files
> This may still need some additional work, Erik, can you describe
> how to do this, in general? I haven't spent much energy on this, but

Well, I have a new version that does not attempt recursion into
non-directories (and thus does not attempt to list contents of
non-directories). I'm not sure if that's sufficient, but here's yet
another version of the patch. This last modification was done
by removing the pointless concatenation of file-namestring
before recursion, it's nil for directories. So now I just check
the file-namestring and don't recurse at all if it exists.

Any more issues? For me, it looks like this patch is done.

-VJV-

Attachment (directory-abcl-patch3): application/octet-stream, 2058 bytes

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/

_______________________________________________
armedbear-j-devel mailing list
armedbear-j-devel <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/armedbear-j-devel

(Continue reading)