NPE in ClassFinder#findClass

[chrisichris]

With the latest yeti from git hub I get a NullPointerException when compiling.

Because with the yeti from 2 weeks ago the same codebase compiles nice, I don't realy know the reason for the NPE. Maybe (and hopefully) the stacktrace is enough info to see what is happening, (the full code is here: https://github.com/chrisichris/yvertx . If this does not help I will of course try to isulate it.

Thanks,

Christian 

 [java] java.lang.NullPointerException

 [java]     at yeti.lang.compiler.ClassFinder.findClass(ClassFinder.java:132)

 [java]     at yeti.lang.compiler.ClassFinder.readClass(ClassFinder.java:189)

 [java]     at yeti.lang.compiler.JavaType.resolve(JavaType.java:534)

 [java]     at yeti.lang.compiler.JavaType.resolve(JavaType.java:902)

 [java]     at yeti.lang.compiler.JavaType.resolveConstructor(JavaType.java:921)

 [java]     at yeti.lang.compiler.YetiAnalyzer.analyze(YetiAnalyzer.java:132)

 [java]     at yeti.lang.compiler.YetiAnalyzer.analyze(YetiAnalyzer.java:202)

 [java]     at yeti.lang.compiler.YetiAnalyzer.lambda(YetiAnalyzer.java:1158)

 [java]     at yeti.lang.compiler.YetiAnalyzer.lambda(YetiAnalyzer.java:1155)

 [java]     at yeti.lang.compiler.YetiAnalyzer.lambdaBind(YetiAnalyzer.java:1116)

 [java]     at yeti.lang.compiler.YetiAnalyzer.singleBind(YetiAnalyzer.java:813)

 [java]     at yeti.lang.compiler.YetiAnalyzer.analSeq(YetiAnalyzer.java:1033)

 [java]     at yeti.lang.compiler.YetiAnalyzer.analyze(YetiAnalyzer.java:91)

 [java]     at yeti.lang.compiler.YetiAnalyzer.toCode(YetiAnalyzer.java:1794)

 [java]     at yeti.lang.compiler.Compiler.compile(Compiler.java:467)

 [java]     at yeti.lang.compiler.YetiTypeVisitor.getType(TypeAttr.java:500)

 [java]     at yeti.lang.compiler.YetiAnalyzer.toCode(YetiAnalyzer.java:1738)

 [java]     at yeti.lang.compiler.Compiler.compile(Compiler.java:467)

 [java]     at yeti.lang.compiler.YetiTypeVisitor.getType(TypeAttr.java:500)

 [java]     at yeti.lang.compiler.YetiAnalyzer.toCode(YetiAnalyzer.java:1738)

 [java]     at yeti.lang.compiler.Compiler.compile(Compiler.java:467)

 [java]     at yeti.lang.compiler.Compiler.compileAll(Compiler.java:194)

 [java]     at yeti.lang.compiler.eval$compileYetiFiles$._0(eval.yeti:334)

 [java]     at yeti.lang.compiler.eval$compileYetiFiles$.apply(eval.yeti:325)

 [java]     at code._0(<>:10)

 [java]     at code.apply(<>)

 [java]     at yeti.lang.compiler.eval._1(eval.yeti:98)

 [java]     at yeti.lang.compiler.eval.execClass(eval.yeti:76)

 [java]     at yeti.lang.compiler.eval$evaluateYetiCode$._0(eval.yeti:484)

 [java]     at yeti.lang.compiler.eval$evaluateYetiCode$.apply(eval.yeti:460)

 [java]     at yeti.lang.Fun2_.apply(Unknown Source)

 [java]     at yeti.lang.compiler.yeti.main(yeti.yeti:188)

[Madis Janson]

On Wed, 13 Feb 2013, chrisichris wrote:

> With the latest yeti from git hub I get a NullPointerException when
> compiling.
> Because with the yeti from 2 weeks ago the same codebase compiles nice, I
> don't realy know the reason for the NPE. Maybe (and hopefully) the
> stacktrace is enough info to see what is happening, (the full code is here:
> https://github.com/chrisichris/yvertx . If this does not help I will of
> course try to isulate it.

Thank you for reporting it, should be fixed now.

The "outer shell" of compiler dealing with sources and classpaths has been
quite extensibly modified lately, so any testing is welcome. Additionally,
the whole 0.9.7+ development has relatively many changes (including binary
module interface incompatibility with 0.9.7 and earlier versions), which
also makes any testing useful. I'll write another mail sometime,
describing more thoroughly the interface change (it was needed to
make the module class interface independent of module implementation,
thus avoiding accidental breakages in future).

> �[java] java.lang.NullPointerException

[chrisichris]

Thanks for the fix works great now.

And of course I am happy to test it. Especially if the fixes come so fast and yeti improves.

Regarding binary comapitbility: It is obviously necessary in an evolving language (and I welcome all your changes). However I have seen binary incompatibility in scala and this was realy annoying, when there are already apis in use. Wouldn't it be better to switch to a "run from source model" like scripting languages ie clojure? I mean that (potential) yeti apis are not compiled to jars, but rather the source .yeti files are jared up and yeti compiles them on the fly. Source seems to be more compatible and if not at least I as a user can fix it myself, get better errormsgs etc. And only optionally compile it to java-classes in the filesystem. I know I have asked this question before, but I have seen this binary incompatibility in scala and I realy did not like it. 

On Wednesday, February 13, 2013 4:00:52 PM UTC+1, Madis Janson wrote:

On Wed, 13 Feb 2013, chrisichris wrote:

> With the latest yeti from git hub I get a NullPointerException when
> compiling.
> Because with the yeti from 2 weeks ago the same codebase compiles nice, I
> don't realy know the reason for the NPE. Maybe (and hopefully) the
> stacktrace is enough info to see what is happening, (the full code is here:
> https://github.com/chrisichris/yvertx . If this does not help I will of
> course try to isulate it.

Thank you for reporting it, should be fixed now.

The "outer shell" of compiler dealing with sources and classpaths has been
quite extensibly modified lately, so any testing is welcome. Additionally,
the whole 0.9.7+ development has relatively many changes (including binary
module interface incompatibility with 0.9.7 and earlier versions), which
also makes any testing useful. I'll write another mail sometime,
describing more thoroughly the interface change (it was needed to
make the module class interface independent of module implementation,
thus avoiding accidental breakages in future).

> �[java] java.lang.NullPointerException
> �[java] � � at
> yeti.lang.compiler.ClassFinder.findClass(ClassFinder.java:132)
> �[java] � � at
> yeti.lang.compiler.ClassFinder.readClass(ClassFinder.java:189)
> �[java] � � at yeti.lang.compiler.JavaType.resolve(JavaType.java:534)

[Madis Janson]

On Wed, 13 Feb 2013, chrisichris wrote:

> Thanks for the fix works great now.
> And of course I am happy to test it. Especially if the fixes come so fast
> and yeti improves.
>
> Regarding binary comapitbility: It is obviously necessary in an evolving
> language (and I welcome all your changes). However I have seen binary
> incompatibility in scala and this was realy annoying, when there are already
> apis in use. Wouldn't it be better to switch to a "run from source model"
> like scripting languages ie clojure?

The new binary class interface should be stable. The problem was with
optimization of module-structure fields access. Now there is simple static
method generated for each field, if the module type is structure, and it
shouldn't break unless field is removed, or the field type changed
fundamentally. In these cases source compatibility would probably be also
broken. You can list those methods for example with the javap tool to see
how it's done:

$ javap -classpath yeti.jar -public yeti.xml
Compiled from "xml.yeti"
public class yeti.xml extends java.lang.Object{
public static synchronized java.lang.Object eval();
public static yeti.lang.Fun xmlParse();
public static yeti.lang.Fun xmlElement();
public static yeti.lang.Fun xmlByPath();
public static yeti.lang.Fun xmlWrite();
public static void init();
}

Only non-obvious thing here is when you have field named 'eval' - these
are given $ as suffix. How the module internally optimises the field
access is hidden from the JVM level public interface (currently most of
them return public static _ field belonging to the class implementing the
corresponding function).

> I mean that (potential) yeti apis are not compiled to jars, but rather
> the source .yeti files are jared up and yeti compiles them on the fly.
> Source seems to be more compatible and if not at least I as a user can
> fix it myself, get better errormsgs etc. And only optionally compile it
> to java-classes in the filesystem. I know I have asked this question
> before, but I have seen this binary incompatibility in scala and I realy
> did not like it.

Dynamic compilation comes with some performance penalty on loading and
memory use, and is not possible everywhere. This doesn't mean that loading
that kind of source jars shouldn't be possible in future, but having
binary compatibility as much as possible is also useful. The issue with
maven plugin highlighted the problem with previous implementation for me,
and I wanted to fix it before 1.0, so it had to be broken in 0.9.7+.

[chrisichris]

Thanks for the detailed explanation and that you make the changes now so that further binary incompatibilities are not that necessary anymore. I just wanted to point out that this was realy something I did not like with scala and IMO could turn people away.

Thanks also for mentioning the maven-plugin I have to update this one. (maybe I should wait for the next stable yeti version 0.9.8?, because of the incompaitibility with the old stable version - or should I update immidiately to current vs 0.9.7+.

Finally I think I have found another bug:

The compiler does not seem to find compiled modules with a simple name (ie module yvertx; or module foo;) if they are on the Thread Context ClassLoader . Modules which have a packaged name (ie module yeb.async; module par.foo;) are found with no problem.

Thanks for the realy great work

[Madis]

On Wed, 13 Feb 2013, chrisichris wrote:

> Thanks for the detailed explanation and that you make the changes now so
> that further binary incompatibilities are not that necessary anymore. I just
> wanted to point out that this was realy something I did not like with scala
> and IMO could turn people away.
> Thanks also for mentioning the maven-plugin I have to update this one.
> (maybe I should wait for the next stable yeti version 0.9.8?, because of the
> incompaitibility with the old stable version - or should I update
> immidiately to current vs 0.9.7+.

When built against current 0.9.7+, the code should work with 0.9.8 (I
hope at least), but it might be easier to version maven dependencies,
when 0.9.8 is out. In essence, do whatever is easier. ;)

> Finally I think I have found another bug:
>
> The compiler does not seem to find compiled modules with a simple name (ie
> module yvertx; or module foo;) if they are on the Thread Context ClassLoader
> . Modules which have a packaged name (ie module yeb.async; module par.foo;)
> are found with no problem.

Should be fixed now.

The compiler searches module in following order:

1. Sourcepath: foo/bar/baz.yeti
2. Classpath: foo/bar/baz.class
3. Sourcepath: baz.yeti

When the package is root (empty), the step 3 is identical to first, so it
should be skipped. The bug was, that step 2 was skipped also.

[chrisichris]

Thanks for the fix and explanation.

When built against current 0.9.7+, the code should work with 0.9.8 (I
hope at least), but it might be easier to version maven dependencies,
when 0.9.8 is out. In essence, do whatever is easier. ;)

For now I have put the latest yeti (0.9.7+) under 0.9.8-SNAPSHOT in my maven-repo. The old 0.9.7-SNAPSHOT before the changes is still there (and the older 0.9.7 stable anyway). I think this is more consistent with maven-versioning of projects I have seen where development versions are named like the next release but with -SNAPSHOT. However I am not sure about that and have to check what to do with the + so. Anyway I thought for now 0.9.8-SNAPSHOT is fine also to mark new maven plugin etc.

 

Then I went on to update the maven project however the YetiTask (in c/eval.yeti) did not find my sources when used from maven (through the maven ant plugin)

I changed the code to use canonical file names in the YetiTask#execute() and that works.

around line 644 in eval.yeti I replaced

files = this#getDirectoryScanner(src)#getIncludedFiles();

with

        files = 

            (import java.io.File;

            ds = this#getDirectoryScanner(src);

            (ds#getIncludedFiles() as list<string>) 

                |> map do f: 

                    (new File(ds#getBasedir(),f is string))#getCanonicalPath()

            done);

I hope this is a good fix and maybe you could add this.

Thanks,

Christian

[Madis]

On Thu, 14 Feb 2013, chrisichris wrote:

> Then I went on to update the maven project however the YetiTask (in c/eval.yeti) did not find my
> sources when used from maven (through the maven ant plugin)
>
> I changed the code to use canonical file names in the YetiTask#execute() and that works.
>
> around line 644 in eval.yeti I replaced
>
> files = this#getDirectoryScanner(src)#getIncludedFiles();
>
> with
>
> � � � � files =�
> � � � � � � (import java.io.File;
> � � � � � � ds = this#getDirectoryScanner(src);
> � � � � � � (ds#getIncludedFiles() as list<string>)�
> � � � � � � � � |> map do f:�
> � � � � � � � � � � (new File(ds#getBasedir(),f is string))#getCanonicalPath()
> � � � � � � done);
>
> I hope this is a good fix and maybe you could add this.

I'm just guessing, that #getAbsolutePath() and even #getPath() would be
enough? These are little bit cheaper, and Yeti later finds canonical path
anyway - the problem seems to be here, that basedir was not added in
current code.

[Madis]

Did it now with getPath(), check whether it works.

[chrisichris]

rigth getPath() works here as well, however there is nothing in github yet. 

On Thursday, February 14, 2013 2:08:44 PM UTC+1, Madis wrote:

On Thu, 14 Feb 2013, Madis wrote:

> On Thu, 14 Feb 2013, chrisichris wrote:
>
>> Then I went on to update the maven project however the YetiTask (in
>> c/eval.yeti) did not find my
>> sources when used from maven (through the maven ant plugin)
>>
>> I changed the code to use canonical file names in the YetiTask#execute()
>> and that works.
>>
>> around line 644 in eval.yeti I replaced
>>
>> files = this#getDirectoryScanner(src)#getIncludedFiles();
>>
>> with
>>

>> � � � � files =�
>> � � � � � � (import java.io.File;
>> � � � � � � ds = this#getDirectoryScanner(src);
>> � � � � � � (ds#getIncludedFiles() as list<string>)�
>> � � � � � � � � |> map do f:�
>> � � � � � � � � � � (new File(ds#getBasedir(),f is
>> string))#getCanonicalPath()
>> � � � � � � done);

[chrisichris]

Your changes from github work great thanks

On Thursday, February 14, 2013 2:08:44 PM UTC+1, Madis wrote:

On Thu, 14 Feb 2013, Madis wrote:

> On Thu, 14 Feb 2013, chrisichris wrote:
>
>> Then I went on to update the maven project however the YetiTask (in
>> c/eval.yeti) did not find my
>> sources when used from maven (through the maven ant plugin)
>>
>> I changed the code to use canonical file names in the YetiTask#execute()
>> and that works.
>>
>> around line 644 in eval.yeti I replaced
>>
>> files = this#getDirectoryScanner(src)#getIncludedFiles();
>>
>> with
>>

>> � � � � files =�
>> � � � � � � (import java.io.File;
>> � � � � � � ds = this#getDirectoryScanner(src);
>> � � � � � � (ds#getIncludedFiles() as list<string>)�
>> � � � � � � � � |> map do f:�
>> � � � � � � � � � � (new File(ds#getBasedir(),f is
>> string))#getCanonicalPath()

>> � � � � � � done);