Generated code

[Sarvi Shanmugham]

I've been reading through the blog post at

https://blogs-pyjeon.rhcloud.com/?p=302

and hence researching some of the complaints there and had a questions

I tried compiling the following piece of code

cat test1.py 

def hello(i):

    i=i*2

    return i

print hello(1)

Which gave the following

sarvi-mac:tmp sarvi$ ../bin/pyjscompile test1.py 

/* start module: test1 */

$pyjs['loaded_modules']['test1'] = function (__mod_name__) {

if($pyjs['loaded_modules']['test1']['__was_initialized__']) return $pyjs['loaded_modules']['test1'];

var $m = $pyjs['loaded_modules']['test1'];

$m['__repr__'] = function() { return '<module: test1>'; };

$m['__was_initialized__'] = true;

if ((__mod_name__ === null) || (typeof __mod_name__ == 'undefined')) __mod_name__ = 'test1';

$m['__name__'] = __mod_name__;

$m['hello'] = function(i) {

var $mul2,$mul1;

i = (typeof ($mul1=i)==typeof ($mul2=2) && typeof $mul1=='number'?

$mul1*$mul2:

$p['op_mul']($mul1,$mul2));

return i;

};

$m['hello']['__name__'] = 'hello';

$m['hello']['__bind_type__'] = 0;

$m['hello']['__args__'] = [null,null,['i']];

$p['printFunc']([$m['hello'](1)], 1);

return this;

}; /* end test1 */

/* end module: test1 */

Question 1:

     The simple i=i*2 seems to have a complex translation. Can't it just be i=i*2 in java script and handle the error when it happens?

Question 2:

   The whole idea of using dictionaries to store functions seems inefficient for access. Is there an alternative approach using say

prototype.bin() or other approaches to define and access these functions that people have tried?

Sarvi

[Lex Berezhny]

On Sun, Oct 20, 2013 at 10:24 PM, Sarvi Shanmugham <sarv...@gmail.com> wrote:

Question 1:

     The simple i=i*2 seems to have a complex translation. Can't it just be i=i*2 in java script and handle the error when it happens?

Python allows you to overload the multiplication operator. We have to check at runtime if the objects being multiplied have overloaded multiplication.

Also, native strings in Python overload the multiplication operator, that's why you can do:

print '='*10

 

Question 2:

   The whole idea of using dictionaries to store functions seems inefficient for access. Is there an alternative approach using say

prototype.bin() or other approaches to define and access these functions that people have tried?

 In javascript objects/dictionaries are one and the same, so one can't be more efficient than the other.

>> a = {}

Object {}

>> a.foo = 3

3

>> a['foo']

3

If you want to hack on pyjs internals you're definitely going to have to learn JavaScript.

 - lex

[Sarvi Shanmugham]

On Sunday, October 20, 2013 8:33:40 PM UTC-7, Lex Berezhny wrote:

On Sun, Oct 20, 2013 at 10:24 PM, Sarvi Shanmugham <sarv...@gmail.com> wrote:

Question 1:

     The simple i=i*2 seems to have a complex translation. Can't it just be i=i*2 in java script and handle the error when it happens?

Python allows you to overload the multiplication operator. We have to check at runtime if the objects being multiplied have overloaded multiplication.

Also, native strings in Python overload the multiplication operator, that's why you can do:

print '='*10

I get this. 

But considering Javascript is JITed and highly optimized including unrolling static loops and function calls if I understand right,

wouldn't the following 

i=p.op_mul(i,2)

be faster than the current

i = (typeof ($mul1=i)==typeof ($mul2=2) && typeof $mul1=='number'?

$mul1*$mul2:

$p['op_mul']($mul1,$mul2));

as long as op_mul() can handle numbers as well?

And much more readable too?

 

 

Question 2:

   The whole idea of using dictionaries to store functions seems inefficient for access. Is there an alternative approach using say

prototype.bin() or other approaches to define and access these functions that people have tried?

 In javascript objects/dictionaries are one and the same, so one can't be more efficient than the other.

>> a = {}

Object {}

>> a.foo = 3

3

>> a['foo']

3

Yeah, thats sorta why I asked the question. Just a beginnner but this how I understood java script objects and classes

and I thought this entire generated code segmented would have a been much more readable as objects instead of dictionary objects.

And whats with the $ notation. From what I am reading this is just jQuery convention and not required.

I presume we do pyjamas because we love python and hence readable code.

Yet the generated Javascript seems less so, though from what I understand there more readable alternatives.

So was just trying to understand what the thinking was. 

If you want to hack on pyjs internals you're definitely going to have to learn JavaScript.

Yup. Pretty much what I have been doing right now. :-)

Still much ways to go.

Sarvi:

PS: Has this blog post been discussed on the list. Did any changes/action items come out of it?

 

 - lex

[Lex Berezhny]

On Mon, Oct 21, 2013 at 1:40 AM, Sarvi Shanmugham <sarv...@gmail.com> wrote:

I get this. 

But considering Javascript is JITed and highly optimized including unrolling static loops and function calls if I understand right,

wouldn't the following 

i=p.op_mul(i,2)

be faster than the current

i = (typeof ($mul1=i)==typeof ($mul2=2) && typeof $mul1=='number'?

$mul1*$mul2:

$p['op_mul']($mul1,$mul2));

as long as op_mul() can handle numbers as well?

And much more readable too?

Unrolling requires work. If you have a lot of equations that means for every operator you will need the JS engine to unroll. There is no way "unrolling" something would be faster than var1*var2, especially not on the first pass. Also, with unrolling a function there is some book keeping required: what if $op.op_mul is changed to a different function at runtime (not that it would be) but the JS engine has to take that possibility into account.

I'm not necessarily against what you are suggesting. I think in the grand scheme of things readability is more important. If you truly need something to be very performant than you can always just write that part in pure JS.

 

Yeah, thats sorta why I asked the question. Just a beginnner but this how I understood java script objects and classes

and I thought this entire generated code segmented would have a been much more readable as objects instead of dictionary objects.

There may have been a reason for doing it that way but I do not know.

 

And whats with the $ notation. From what I am reading this is just jQuery convention and not required.

This has nothing to do with jQuery or conventions.

$ allows you to have private implementation related variables. Since in Python you cannot have variables that start with $ that means we can prefix generated/implementation specific variables with a $ and never worry about clashing.

Think of stuff like:

a, b = (b, a)

Above is not possible in JavaScript without a temporary variable. In pyjs this temporary variable would have a $ prefixed.

I presume we do pyjamas because we love python and hence readable code.

Yet the generated Javascript seems less so, though from what I understand there more readable alternatives.

So was just trying to understand what the thinking was. 

I think a lot of attention wasn't paid to this because there are a lot of other more important things missing from pyjs that need attention.

 

Sarvi:

PS: Has this blog post been discussed on the list. Did any changes/action items come out of it?

I don't remember if we discussed it or not but I can give you my take on a few of the bullet points:

Browser Detection: He's right about the gwt/pyjamas stuff but this is irrelevant to pyjs core. As far as I know only one version of the compiled Python is generated. The Pyjamas/GWT library does have multiple versions per browser. Given that we plan to rip that out of pyjs core into it's own repository it will be for somebody else to deal with :-)

Bloat and Boilerplate Hell: Without specifics this can't really be addressed. But my preference would be to rewrite pyjs from scratch with an eye on less boilerplate. I've started this to some degree and can send you what I have so far if you're interested.

Debugging: Theoretically if it works in Python it should work in compiled JS too - if it doesn't, that's a bug in pyjs and not in your code. I've actually had very few issues with this (as long as I stayed within the limitation of pyjs). So I can't relate to Mr. Tsepkov on that point.

Python is not Java, DOM is not a Desktop: We are taking the pyjamas/gwt stuff out of pyjs repository. This will encourage people to write their own frameworks/widget libraries for pyjs.

JavaScript has its Strengths: Complaining mostly about the GWT stuff in pyjs.

My impression from his article is that he relied too much on the widget library/gwt framework that comes with pyjs.

I think pyjs can hit the sweet spot if you're willing to experiment with it and not relying too much on the gwt framework.

You have access to the DOM and events in pyjs just as you do in JavaScript. In the article he implied that this was not the case.

 - lex

[Sarvi Shanmugham]

I tried to hack a bit to simplify the generated code to see how it can be done and this is what I got. 

I left the $ alone for the reasons Lex mentioned. Turned the dictionary accesses into object notation and 

reduced some of the repetitive code into functions that do the same code. 

From what I tested of inlining, JITs should take care of inlining these with no performance impact.

and it was relatively simple, repetitive but sizable changes to the translate_proto.py code to do this.

/* start module: test1 */

$pyjs.loaded_modules.test1 = function (__mod_name__) {

        if ($pymdecorate(this,'test1',__mod_name__)) return $m

        $m.hello = function(i) {

                return multiply(multiply(multiply(i,i),2),25);

        };

        $pyfdecorate($m.hello,'hello',0,[null,null,['i']]);

        return this;

}; /* end test1 */

/* end module: test1 */

The result would from what I can see would be much simpler, readable and a lot less code

Would something like this be accepted if I work on finishing this?

Sarvi

[Sarvi Shanmugham]

Just a bit of statistic I calculated based on this hacking

cat test1.py | wc -l   # The python testcase

      24

../../pyjsold/lpython/bin/pyjscompile test1.py | wc -l         # As is today

     131

cat t.js | wc -l

      39

I understand it may not extrapolate exactly for everything. 

Just shows there is much that can be improved in that direction.

Sarvi

[Kees Bos]

On Tue, 2013-10-22 at 20:29 -0700, Sarvi Shanmugham wrote:
> I tried to hack a bit to simplify the generated code to see how it can
> be done and this is what I got.
> I left the $ alone for the reasons Lex mentioned. Turned the
> dictionary accesses into object notation and

That looks cleaner, but inhibits the use of closure compiler, since that
will mangle object attributes (at high optimization).

> reduced some of the repetitive code into functions that do the same
> code.
> From what I tested of inlining, JITs should take care of inlining
> these with no performance impact.
> and it was relatively simple, repetitive but sizable changes to the
> translate_proto.py code to do this.

Did you do performance comparisons, on different engines, probably
(arguably) including IE8 for those that still use XP and refuse to
install something different? It used to matter a lot whether you use
function calls or inlined (<test>?<cond1>:<cond2>).

> /* start module: test1 */
> $pyjs.loaded_modules.test1 = function (__mod_name__) {
> if ($pymdecorate(this,'test1',__mod_name__)) return $m
>
>
>
> $m.hello = function(i) {
> return multiply(multiply(multiply(i,i),2),25);
> };
> $pyfdecorate($m.hello,'hello',0,[null,null,['i']]);
>
>
> return this;
> }; /* end test1 */
>
>
>
>
> /* end module: test1 */
> The result would from what I can see would be much simpler, readable
> and a lot less code
> Would something like this be accepted if I work on finishing this?

Depends... on the speed of the result. Since this is only meant to
generate simpler/cleaner/smaller code (which are for _me_ minor issues
compared to speed), this might not be way to go in _my_ opinion.

My guess (when I see this generated code) is that it will run about 5
times slower than what's currently generated (with --strict option). But
please, prove me wrong :-)


- Kees

[Sarvi Shanmugham]

On Tuesday, October 22, 2013 9:42:19 PM UTC-7, Kees Bos wrote:

On Tue, 2013-10-22 at 20:29 -0700, Sarvi Shanmugham wrote:
> I tried to hack a bit to simplify the generated code to see how it can
> be done and this is what I got.
> I left the $ alone for the reasons Lex mentioned. Turned the
> dictionary accesses into object notation and

That looks cleaner, but inhibits the use of closure compiler, since that
will mangle object attributes (at high optimization).

would you refering me to something that explains this? I am relative noob with this stuff.

> reduced some of the repetitive code into functions that do the same
> code.
> From what I tested of inlining, JITs should take care of inlining
> these with no performance impact.
> and it was relatively simple, repetitive but sizable changes to the
> translate_proto.py code to do this.

Did you do performance comparisons, on different engines, probably
(arguably) including IE8 for those that still use XP and refuse to
install something different? It used to matter a lot whether you use
function calls or inlined (<test>?<cond1>:<cond2>).

expalined below. 

> /* start module: test1 */
> $pyjs.loaded_modules.test1 = function (__mod_name__) {
>         if ($pymdecorate(this,'test1',__mod_name__)) return $m
>
>
>
>         $m.hello = function(i) {
>                 return multiply(multiply(multiply(i,i),2),25);
>         };
>         $pyfdecorate($m.hello,'hello',0,[null,null,['i']]);
>
>
>         return this;
> }; /* end test1 */
>
>
>
>
> /* end module: test1 */
> The result would from what I can see would be much simpler, readable
> and a lot less code
> Would something like this be accepted if I work on finishing this?

Depends... on the speed of the result. Since this is only meant to
generate simpler/cleaner/smaller code (which are for _me_ minor issues
compared to speed), this might not be way to go in _my_ opinion.

My guess (when I see this generated code) is that it will run about 5
times slower than what's currently generated (with --strict option). But
please, prove me wrong :-)

I found this interesting site that measures exactly this :-)

Compare the Orange and Blue graphs for each browser for what we need here.

Every browser shows quite comparable performance, infact almost equal between inlining and using a function.

Except, now I hate IE even more, IE. Even the latest one sucks at this. :-)) 

Sarvi

- Kees

[Sarvi Shanmugham]

For got to provide a link to the performance comparison site

http://jsperf.com/function-call-vs-inline

Sarvi

[Kees Bos]

On Tue, 2013-10-22 at 22:21 -0700, Sarvi Shanmugham wrote:
>
>
> On Tuesday, October 22, 2013 9:42:19 PM UTC-7, Kees Bos wrote:
> On Tue, 2013-10-22 at 20:29 -0700, Sarvi Shanmugham wrote:
> > I tried to hack a bit to simplify the generated code to see
> how it can
> > be done and this is what I got.
> > I left the $ alone for the reasons Lex mentioned. Turned
> the
> > dictionary accesses into object notation and
>
> That looks cleaner, but inhibits the use of closure compiler,
> since that
> will mangle object attributes (at high optimization).
> would you refering me to something that explains this? I am relative
> noob with this stuff.
>

It's somewhere in the docs or command line help. This is the issue (from
memory) after running closure compiler:
foo.something gets compressed to foo.XX
foo['something'] becomes foo.something

[Sarvi Shanmugham]

Thanks for the pointer.  I did a little bit more reading at

https://developers.google.com/closure/compiler/docs/api-tutorial3

A couple of things.

   1. Is foo.something --> foo.XX really a problem. From what I can tell as long as its usage is consistent across the code we should be fine.

       Also does all the other javascript frameworks use foo['something'] to avoid this issue? 

       Infact the compiler advanced optimization documentation suggests suggests the opposite of what you are suggesting. 

       Consistent dot notation Vs quoted strings as a solution here 

        https://developers.google.com/closure/compiler/docs/api-tutorial3#propnames

   2. Closure advanced optimization does the inlining during compilation and hence the performance should be comparable on all browsers.

       I tried the following code at

function dumult(x,y) {

    return x*y;

}

function multiply(x,y) {

    return (typeof x == typeof y && typeof x=='number')?x*y:dumult(x,y);

}

function hello(name) {

var i;

var j;

i=10;

j=20;

i=multiply(i,j);

alert('Hello, ' + name + i);

}

hello('New user');

It got optimized to, Wow!! :-))

var a;a=10;a*=20;alert("Hello, New user"+a); 

Another complaint I've read about pyjamas is size of generated code

https://blogs-pyjeon.rhcloud.com/?p=302

And I am reading about closure tool doing dead code removal

Shouldn't that be trimming simple applications to the minimum code?

I am guessing people have tried this. What is the feedback?

Should closure be integrated into the pyjs compile/build process?

Sarvi

[Kees Bos]

Nope. It's a blocking issue. How does hasattr(foo, 'something') now that
'something' should in fact be 'XX'?

Impressive, but probably only possible since the compiler knows
_beforehand_ that i = 10 and j = 20 and dumult(x,y) _is_ x*y. Try
changing dumult to:
function dumult(x,y) {return x*y+1;}

> Another complaint I've read about pyjamas is size of generated code
> https://blogs-pyjeon.rhcloud.com/?p=302

For him. Indeed. He doesn't need python, but something with the syntax
like python, that remotely behaves like python, but is primarily
javascript. BTW. I think that speed was his most important issue with
pyjamas. (Graphpad is a highly interactive app and needs very fast
response times, which is quite different from a standard
form-based-lookalike app).

>
> And I am reading about closure tool doing dead code removal
> Shouldn't that be trimming simple applications to the minimum code?
> I am guessing people have tried this. What is the feedback?

It saves primarily space. It didn't speed-up the code execution (much).
Well, a year ago or something like that. Might be different now.

> Should closure be integrated into the pyjs compile/build process?

Yes. But optionally, and needs lots of testing (since all modules will
be compiled separately, doing all-in-one-run would make it easier to
have consistent code compression).

[Sarvi Shanmugham]

Checked some of the things you raise.

From what I read and experimented as long as 'something' is stored in a global dictionary it is not optmized

window['hello'] = hello;

this can be avoided by simply generating an extern.js file with these and using them during compilation. 

Here is closure generated code I got after doing it 

var pyjs={apple:{}};pyjs.apple.test=5;pyjs.orange={};pyjs.orange.test=10;pyjs.name="apple";pyjs.b="test";window.a=pyjs;function c(d){var b;b=10;b="number"==typeof b&&"number"==typeof b?20*b:20*b+1*pyjs[pyjs.name][pyjs.tname];alert("Hello, "+d+b)}window.hello=c;c("New user");

This required // @js_externs var pyjs;pyjs.apple;pyjs.orange;function hello(name){};

Well I did and got the following. This included the js_externs and so specifying hello() as an extern did not impact inlining.

var a;a=10;a="number"==typeof a&&"number"==typeof a?20*a:20*a+1;alert("Hello, New user"+a); 

[Kees Bos]

On Tue, 2013-10-22 at 22:22 -0700, Sarvi Shanmugham wrote:
> For got to provide a link to the performance comparison site
> http://jsperf.com/function-call-vs-inline
>

I noticed yesterday (I executed the tests on the url above), that
depending on the browser history the outcome may differ. On a fresh
startup of firefox 24 the difference between function calls and in line
was almost non-existent. However, when I first tried the same url, the
inline code was about 60 times faster than function calls. Apart from
that, the speed of inline calls was with a fresh restart about four
times faster.

This kind of behavior makes it somewhat difficult to interpret
'benchmarking' test results.