gmane.comp.lang.scala

Michael | 8 Oct 00:30

Scala interpreter: classpath vs. class loader

Currently the Scala interpreter/compiler needs a string classpath in order 
to resolve imports. However, this is a problem in an OSGi environment where 
such a classpath is not available but classes should rather be loaded from 
the class loader. 

I took a look at the Scala sources. As it seems, Scala currently rebuilds 
the directory structure of the classpath in memory. Using class loaders this 
is not possible since classes can only be loaded on the fly from a given 
class name. There is no way to enumerate the classes of a class loader. 

Is this pre-loading a strict requirement for the Scala compiler or can it 
cope with on the fly loading? That is, does the Scala compiler need to 
enumerate available classes? I figure that could be the case for resolving
implicits and on demand imports. If this is the case, we might be out of 
options for OSGi environments - which is a petty. If not, what would be a 
good starting point to add support for loading classes from a class loader? 
Could it even be done in a compiler plugin? (I dont think so but I might 
be wrong). Any pointers appreciated.

Michael

headers

David MacIver | 8 Oct 01:00

Re: Scala interpreter: classpath vs. class loader

On Tue, Oct 7, 2008 at 11:30 PM, Michael <michid <at> gmail.com> wrote:
> Currently the Scala interpreter/compiler needs a string classpath in order
> to resolve imports. However, this is a problem in an OSGi environment where
> such a classpath is not available but classes should rather be loaded from
> the class loader.
>
> I took a look at the Scala sources. As it seems, Scala currently rebuilds
> the directory structure of the classpath in memory. Using class loaders this
> is not possible since classes can only be loaded on the fly from a given
> class name. There is no way to enumerate the classes of a class loader.
>
> Is this pre-loading a strict requirement for the Scala compiler or can it
> cope with on the fly loading? That is, does the Scala compiler need to
> enumerate available classes? I figure that could be the case for resolving
> implicits and on demand imports. If this is the case, we might be out of
> options for OSGi environments - which is a petty. If not, what would be a
> good starting point to add support for loading classes from a class loader?
> Could it even be done in a compiler plugin? (I dont think so but I might
> be wrong). Any pointers appreciated.

This sounds related to problems I had with trying to add classes to
the interpreter classpath at runtime - scalac's caching behaviour
meant that although the classes were being added the interpreter could
not pick them up. At the time Martin said that the way the compiler
was structured made it very difficult to change this behaviour. I
suspect similar applies to your case too unfortunately.

headers

Sean McDirmid | 8 Oct 07:35

Re: Scala interpreter: classpath vs. class loader

Actually, we have some of the same problems in the Eclipse plugin.
I've made some effort to hack symbol loaders, purging classes causing
them to reload when possible. My success hasn't been great. I'm
guessing that we'll have to rewrite symbol loaders before we can have
good inter-project dependency invalidation.

Sean

On Wed, Oct 8, 2008 at 7:00 AM, David MacIver <david.maciver <at> gmail.com> wrote:
> On Tue, Oct 7, 2008 at 11:30 PM, Michael <michid <at> gmail.com> wrote:
>> Currently the Scala interpreter/compiler needs a string classpath in order
>> to resolve imports. However, this is a problem in an OSGi environment where
>> such a classpath is not available but classes should rather be loaded from
>> the class loader.
>>
>> I took a look at the Scala sources. As it seems, Scala currently rebuilds
>> the directory structure of the classpath in memory. Using class loaders this
>> is not possible since classes can only be loaded on the fly from a given
>> class name. There is no way to enumerate the classes of a class loader.
>>
>> Is this pre-loading a strict requirement for the Scala compiler or can it
>> cope with on the fly loading? That is, does the Scala compiler need to
>> enumerate available classes? I figure that could be the case for resolving
>> implicits and on demand imports. If this is the case, we might be out of
>> options for OSGi environments - which is a petty. If not, what would be a
>> good starting point to add support for loading classes from a class loader?
>> Could it even be done in a compiler plugin? (I dont think so but I might
>> be wrong). Any pointers appreciated.
>
> This sounds related to problems I had with trying to add classes to

(Continue reading)

headers

Michael | 8 Oct 09:36

Re: Scala interpreter: classpath vs. class loader

Sean McDirmid <sean.mcdirmid <at> gmail.com> writes:

> 
> Actually, we have some of the same problems in the Eclipse plugin.
> I've made some effort to hack symbol loaders, purging classes causing
> them to reload when possible. My success hasn't been great. I'm
> guessing that we'll have to rewrite symbol loaders before we can have
> good inter-project dependency invalidation.
> 

Purging and refreshing (existing) classes is one thing. There might be a more
fundamental issues with discovering new classes which are not explicitly named
in the source: 

import foo._

class Bar {
  def m(implicit x: Any) {...}
}

I think here the compiler has to search through all classes in foo._ in order to
find an implicit argument to pass for x. Java class loaders do not posses such a
discovery mechanism. The compiler currently resorts to traversing the file
system's directory structure directly. 

If I'm not mistaken, this is a blocker for the interpreter to work in an OSGi
environment!?

Michael

(Continue reading)

headers

Sean McDirmid | 8 Oct 09:45

Re: Re: Scala interpreter: classpath vs. class loader

No, there is nothing very fundamental about it: PackageSymbolLoader
scans the directory when it is created to see what class files are
there. Then thats it, it will never re-scan the directory. This should
be fairly easy to fix: just write a new PackageSymbolLoader class that
doesn't create a digest of the directory on load, instead when a class
is requested, just use the file system to find it! Or if the file is
not in the digest, just look in the file system again. Ok, this is
really easy to fix, its not so related to my much harder problem.

You should fix PackageSymbolLoader and submit a patch.

On Wed, Oct 8, 2008 at 3:36 PM, Michael <michid <at> gmail.com> wrote:
> Sean McDirmid <sean.mcdirmid <at> gmail.com> writes:
>
>>
>> Actually, we have some of the same problems in the Eclipse plugin.
>> I've made some effort to hack symbol loaders, purging classes causing
>> them to reload when possible. My success hasn't been great. I'm
>> guessing that we'll have to rewrite symbol loaders before we can have
>> good inter-project dependency invalidation.
>>
>
> Purging and refreshing (existing) classes is one thing. There might be a more
> fundamental issues with discovering new classes which are not explicitly named
> in the source:
>
> import foo._
>
> class Bar {
>  def m(implicit x: Any) {...}

(Continue reading)

headers

Michael | 8 Oct 10:10

Re: Scala interpreter: classpath vs. class loader

> No, there is nothing very fundamental about it: PackageSymbolLoader
> scans the directory when it is created to see what class files are
> there. Then thats it, it will never re-scan the directory. This should
> be fairly easy to fix: just write a new PackageSymbolLoader class that
> doesn't create a digest of the directory on load, instead when a class
> is requested, just use the file system to find it! Or if the file is
> not in the digest, just look in the file system again. Ok, this is
> really easy to fix, its not so related to my much harder problem.

Well but there is no file system and no directories to scan from within an OSGi
environment. The only means to get a class is from a class loader. That's it. 

So my question boils down to: does the Scala compiler need to know about what
classes are available to it a priori? Or can it live within an environment where
the only means to get a class is by explicitly asking for it?

See my earlier example on resolving implicits and general imports for why I
think this might be relevant.

Michael

headers

Sean McDirmid | 8 Oct 10:24

Re: Re: Scala interpreter: classpath vs. class loader

So you create a new SymbolLoader that wraps a classloader and
determinse demand determines whether or not a class exists. I don't
think this is so hard if you are willing to define and install your
own symbol loader.

On Wed, Oct 8, 2008 at 4:10 PM, Michael <michid <at> gmail.com> wrote:

> Well but there is no file system and no directories to scan from within an OSGi
> environment. The only means to get a class is from a class loader. That's it.
>
> So my question boils down to: does the Scala compiler need to know about what
> classes are available to it a priori? Or can it live within an environment where
> the only means to get a class is by explicitly asking for it?
>
> See my earlier example on resolving implicits and general imports for why I
> think this might be relevant.
>
> Michael
>
>
>

headers

Johannes Rudolph | 8 Oct 10:28

Re: Scala interpreter: classpath vs. class loader

On Wed, Oct 8, 2008 at 12:30 AM, Michael <michid <at> gmail.com> wrote:
> Currently the Scala interpreter/compiler needs a string classpath in order
> to resolve imports. However, this is a problem in an OSGi environment where
> such a classpath is not available but classes should rather be loaded from
> the class loader.
I think the scala compiler uses its own Symbolloader and doesn't rely
on the Java class loader architecture. It needs to do so to load scala
signatures and similar which is not exposed by Java reflection
mechanisms.

--

-- 
Johannes

-----------------------------------------------
Johannes Rudolph
http://virtual-void.net

headers

Michael | 8 Oct 10:47

Re: Scala interpreter: classpath vs. class loader

Johannes Rudolph <johannes.rudolph <at> googlemail.com> writes:

> 
> On Wed, Oct 8, 2008 at 12:30 AM, Michael <michid <at> gmail.com> wrote:
> > Currently the Scala interpreter/compiler needs a string classpath in order
> > to resolve imports. However, this is a problem in an OSGi environment where
> > such a classpath is not available but classes should rather be loaded from
> > the class loader.
> I think the scala compiler uses its own Symbolloader and doesn't rely
> on the Java class loader architecture. It needs to do so to load scala
> signatures and similar which is not exposed by Java reflection
> mechanisms.

Yes that's what I figured. So the question is whether it is possible to wrap a
Java class loader into a symbol loader for Scala. I fear not since the symbol
loader might need to discover what classes are available while the java class
loader cannot do that.

Michael

headers

Stepan Koltsov | 8 Oct 11:14

Re: Scala interpreter: classpath vs. class loader

Yes, Scala Java reflection is not enough to get symbol information,
but paths to Jar files can be found by analyzing current classloader.

Spring Framework does so to find (for example) all classes
implementing some interface in current classloader.

S.

On Wed, Oct 8, 2008 at 12:28, Johannes Rudolph
<johannes.rudolph <at> googlemail.com> wrote:
> On Wed, Oct 8, 2008 at 12:30 AM, Michael <michid <at> gmail.com> wrote:
>> Currently the Scala interpreter/compiler needs a string classpath in order
>> to resolve imports. However, this is a problem in an OSGi environment where
>> such a classpath is not available but classes should rather be loaded from
>> the class loader.
> I think the scala compiler uses its own Symbolloader and doesn't rely
> on the Java class loader architecture. It needs to do so to load scala
> signatures and similar which is not exposed by Java reflection
> mechanisms.

headers

Michael | 8 Oct 14:10

Re: Scala interpreter: classpath vs. class loader

> Yes, Scala Java reflection is not enough to get symbol information,
> but paths to Jar files can be found by analyzing current classloader.
> 
> Spring Framework does so to find (for example) all classes
> implementing some interface in current classloader.

Ok that's what I expected. Hacking the class loader is an option but introduces
dependencies on the implementation. 

Thanks,
Michael

headers

Stepan Koltsov | 8 Oct 22:07

Re: Re: Scala interpreter: classpath vs. class loader

On Wed, Oct 8, 2008 at 16:10, Michael <michid <at> gmail.com> wrote:
>> Yes, Scala Java reflection is not enough to get symbol information,
>> but paths to Jar files can be found by analyzing current classloader.
>>
>> Spring Framework does so to find (for example) all classes
>> implementing some interface in current classloader.
>
> Ok that's what I expected. Hacking the class loader is an option but introduces
> dependencies on the implementation.

Actually almost all classloaders in all JVMs are URLClassLoaders.
java.net.URLClassLoader has public getURLs() method. So Spring
classloading is implemented without knowledge of any Java
implementations, AFAIR.

S.

headers

Michael | 9 Oct 14:10

Re: Scala interpreter: classpath vs. class loader

Stepan Koltsov <stepan.koltsov <at> gmail.com> writes:

> Actually almost all classloaders in all JVMs are URLClassLoaders.
> java.net.URLClassLoader has public getURLs() method. So Spring
> classloading is implemented without knowledge of any Java
> implementations, AFAIR.

Hmmm yes but not so with application servers or within OSGi. But you are 
right, it seems that at the moment it is the only option to get the 
classpath from the class loader. I will try to to this for Apache 
Felix then.

Michael

headers

Michael Nascimento | 13 Oct 19:41

Re: Re: Scala interpreter: classpath vs. class loader

I guess the compiler still needs access to the actual byte[] for the
Class, so it can read the extra information for Scala signatures
contained in the bytecode.

If that happens to be the case, the best option I can think of is to
write an agent so it can intercept class loading and grab the byte
array for every class.

Regards,
Michael Nascimento Santos
https://genesis.dev.java.net/
https://jsr-310.dev.java.net/

On Thu, Oct 9, 2008 at 10:10 AM, Michael <michid <at> gmail.com> wrote:
> Stepan Koltsov <stepan.koltsov <at> gmail.com> writes:
>
>> Actually almost all classloaders in all JVMs are URLClassLoaders.
>> java.net.URLClassLoader has public getURLs() method. So Spring
>> classloading is implemented without knowledge of any Java
>> implementations, AFAIR.
>
> Hmmm yes but not so with application servers or within OSGi. But you are
> right, it seems that at the moment it is the only option to get the
> classpath from the class loader. I will try to to this for Apache
> Felix then.
>
> Michael
>
>
>

(Continue reading)