Importing Qt source: soon (TM)

[Ariya Hidayat]

This is about http://code.google.com/p/phantomjs/issues/detail?id=226.

In a day or two, I'll carry out the import. I've been testing it for a
while and seems that it works really well.

Several things you need to be aware of:

The repository size will explode. The actual check out grows by ~100
MB, see https://picasaweb.google.com/103266860731871773002/Screenshots#5710629690310389138.
Note that I carefully exclude Qt stuff which we would never need. I'm
sure we can trim this further but so far it is quite optimal.

The build steps will be different. I'm longing for the universal step
of "./configure && make", but for now we have to be happy with just
"./build.sh". Note that it is still Mac only. Also I haven't
incorporated all the recent refactoring on the deploy/build* scripts
yet. Help will be appreciated.

Because we don't rely on system-installed Qt anymore, the build time
increases significantly. It is still about 30 minutes on a modern
machine with 4 parallel compile jobs. I expect that "normal" users
should not worry much about this. If you contribute to PhantomJS,
expect the initial build to be quite long but further subsequent
changes often only imply incremental compile and not full rebuild.

Also, the import is kinda semi-automatic: importer script (which I
will include of course) and manual patches apply (what we already use
in deploy/). I'm doing it for 4.8.0 source tree but once 4.8.1 is out,
I expect the re-import/update/patches reapply to be easy.

Feedback is welcomed!

--
Ariya Hidayat, http://ariya.ofilabs.com

[Jon Leighton]

On 19/02/12 00:10, Ariya Hidayat wrote:
> This is about http://code.google.com/p/phantomjs/issues/detail?id=226.

Hi Ariya,

After the import, how straightforward will it be to patch Qt so we can
get proper Javascript stack traces? This is a feature I am dying for and
will gladly assist if I can :) Is it realistic for phantomjs to support
this in 1.5?

Cheers,
Jon

--
http://jonathanleighton.com/

[Nicolas Perriault]

On Tue, Feb 28, 2012 at 19:05, Jon Leighton <j...@jonathanleighton.com> wrote:

> After the import, how straightforward will it be to patch Qt so we can get
> proper Javascript stack traces? This is a feature I am dying for and will
> gladly assist if I can :) Is it realistic for phantomjs to support this in
> 1.5?

+1000^1000 (for what its worth)

Seriously, the phantomjs native environment is hardly debuggable
without proper stack traces atm :/

++

--
Nicolas Perriault
http://www.akei.com/ — http://prendreuncafe.com/
Mobile: +33 660 92 08 67 — Skype: nperriault

[Ariya Hidayat]

Sorry for the delay. I have imported Qt source tree, however the build
system is not adjusted yet to take this into account.

> After the import, how straightforward will it be to patch Qt so we can get
> proper Javascript stack traces? This is a feature I am dying for and will
> gladly assist if I can :) Is it realistic for phantomjs to support this in
> 1.5?

I haven't investigated this bug yet (is there an issue filed for it
already?). Consider JavaScriptCore always has a sane stack trace, I
wonder if it is just a matter of missing hook somewhere.

Thanks!

Regards,

[Jon Leighton]

There is this: http://code.google.com/p/phantomjs/issues/detail?id=166
(although it refers to jasmine despite it being more general than that).

I squinted at the qt/webkit source code the other night out of
curiosity. I think QtWebKit doesn't pass the file details into WebKit,
and I think I identified how I can fix this, so I will try it when I get
a chance. That will hopefully solve the file/line problem, and I don't
know if that'll help with the full stack trace, but one step at a time.

I'll keep trying as I am really keen to get this sorted, but if anyone
wants to help don't let me discourage you...!

--
http://jonathanleighton.com/

[Ariya Hidayat]

The build process has been modified. From now on, please run:

./build.sh

This works both on Linux and Mac OS X. It will trigger static build on Mac OS X.

[Ariya Hidayat]

If you want to help with the Linux build (testing it on various
distributions), that'll be appreciated.

I don't hook headless feature via QPA yet, but will do in the next few
days. It is planned to the standard default so there is no need to X11
for this Linux build anymore.

Note that static build does not work yet, please track
http://code.google.com/p/phantomjs/issues/detail?id=413 if you are
interested in.

[Ariya Hidayat]

The details about the import:

It's done using tools/import-qt.sh script. There is a white list of
files to be imported, thus our 'src/qt' directory is not as gigantic
as a typical vanilla Qt from Nokia.

Further build streamlining is still possible, please track
http://code.google.com/p/phantomjs/issues/detail?id=414.

I've applied our custom Qt patches to src/qt. Whenever we import the
next version of Qt, e.g. 4.8.1, it's actually a matter of
cherry-picking the previous commits. However, it's still a good idea
to keep those patches around. I will likely move deploy/qt-patches to
src/qt/patches.

Note that deploy/ won't be needed anymore, except for
package-linux-dynamic.sh script. Jon, feel free to beat me in
adjusting the script so that it works with the new build process.

Comments and questions are welcomed.

[Ariya Hidayat]

Final note: Windows build is not tackled yet, I'll try to get it
working by next week.

Overall, in the (near) future hopefully the build steps will be
better, see http://code.google.com/p/phantomjs/issues/detail?id=414.

For better build workflow (incremental build, installation,
uninstallation), the ideal step will be:

./configure
make

and with targets allowing both:

make install
make uninstall

Thanks!

Regards,

[Ariya Hidayat]

> Note that deploy/ won't be needed anymore, except for
> package-linux-dynamic.sh script. Jon, feel free to beat me in
> adjusting the script so that it works with the new build process.

deploy/ has been cleaned-up.

Only deploy/package-linux-dynamic.sh is the remaining (relevant)
script, it's been adjusted properly.

[Milian Wolff]

I notice that the build*.sh scripts in deploy/ are still there, yet the patches have been removed. Either the former should be removed as well (since without patches they don't give you anything), or the latter be reintroduced.

[Ariya Hidayat]

Good catch, I've removed them. Thanks for the notice?

[Alessandro Portale]

Hi,

did anyone or does anyone plan to try out the PhantomJS+Qt build under MSVC on Windows? I just gave it a try.

Apparently, a few more adjustments to src.pro need to be done. I cannot deactivate compilation of svg and active-qt via configure parameter, and therefore I need to modify src.pro. (Windows comes with a separate configure.exe, which is feature-wise unfortunately different from the shell-script based configure tool)

First thing I am planning to do however is creating a preconfig.cmd, the Windows pendant of preconfig.sh.

Cheers,

Alessandro

[Jon Leighton]

Hey,

Great work on this! Building is now much more pleasant.

I've been trying it out today, and it seems that I need to set
LD_LIBRARY_PATH=src/qt/lib in order to get it to link correctly. Is this
expected?

[Ariya Hidayat]

Hi Alessandro,

Thanks for helping out. I myself struggle with using MinGW so any MSVC
build fixes will be appreciated.

I create http://code.google.com/p/phantomjs/issues/detail?id=424 to
house any related Windows build issues.

I think it's fine to modify src.pro (or any other files for that
matter). For my upcoming X-less tweaks I even need to change the
configure script because it was not flexible enough.

Thanks!

Regards,

[Ariya Hidayat]

Hi Jon,

> I've been trying it out today, and it seems that I need to set
> LD_LIBRARY_PATH=src/qt/lib in order to get it to link correctly. Is this
> expected?

That's weird, I did not have to do that. Because main PhantomJS is
built via src/qt/bin/qmake, I think it should figure out by itself
that the library is in src/qt/lib.

Any further pointers to the issue?

[Jon Leighton]

On 13/03/12 04:08, Ariya Hidayat wrote:
>> I've been trying it out today, and it seems that I need to set
>> LD_LIBRARY_PATH=src/qt/lib in order to get it to link correctly. Is this
>> expected?
>
> That's weird, I did not have to do that. Because main PhantomJS is
> built via src/qt/bin/qmake, I think it should figure out by itself
> that the library is in src/qt/lib.

I am not sure what it was, but it seems to no longer be an issue for me
at the moment - so please disregard.

--
http://jonathanleighton.com/

[Ariya Hidayat]

> There is this: http://code.google.com/p/phantomjs/issues/detail?id=166
> (although it refers to jasmine despite it being more general than that).

Hi Jon, did you get any further on this issue? I'd be happy to help.

[Jon Leighton]

On 14/03/12 01:58, Ariya Hidayat wrote:
>> There is this: http://code.google.com/p/phantomjs/issues/detail?id=166
>> (although it refers to jasmine despite it being more general than that).
>
> Hi Jon, did you get any further on this issue? I'd be happy to help.

I can successfully obtain a backtrace, but I have been stuck on getting
it to report to a callback on the actual page.

I finally figured out last night that the problem is like this:

page = require('webpage').create()
page.onError = function() { ... }
// main page context
page.evaluate(function() {
// error in the subpage gets reported to the debugger of the
// main page, so it doesn't find the correct callback
missing_reference
})

Probably the pragmatic solution will be to implement some sort of kludge
fix to keep track of subpage references in page.evaluate(). FWIW the
following works fine:

page.evaluate(function() {
setTimeout(function() { missing_reference }, 0)
})

So it seems to be just when we are inside nested
QWebFrame::evaluateJavaScript calls that there is a problem.

Here is my initial code for getting the backtraces out:
https://github.com/jonleighton/phantomjs/commit/34ffe4df5585aba31475d25e2439d897fab4dab8

I have some more code around reporting it back to the QWebPage, but
given the above problem it's currently littered with printf statements,
so I will try to clean it up later and push it.

I also have a question: we probably need some way to refer to the main
page, in order to attach callbacks. Something like:

phantom.main.onError = function() { ... }

Do you have any thoughts on this API?

Anyway, I am actively working on this as I am really keen to get it into
the 1.5 release. I'll appreciate a thorough code review when it's ready
as I have very little C++ knowledge.

Cheers

--
http://jonathanleighton.com/

[Nicolas Perriault]

On Wed, Mar 14, 2012 at 10:25, Jon Leighton <j...@jonathanleighton.com> wrote:

> Do you have any thoughts on this API?

I may be missing something here and be a bit off topic, but the
problem doesn't only relate to the page. A simple script like this
one, where one could expect an error backtrace being available and
shown just won't:

function foo() { throw new Error('foo'); }
function bar() { foo(); }
bar();

Expected, like using v8:

$ v8 t.js
t.js:2: Error: foo
throw new Error('foo');
^
Error: foo
at foo (t.js:2:11)
at bar (t.js:6:5)
at t.js:9:1

Obtained with phantomjs 1.4.1:

$ phantomjs t.js
undefined:0 Error: foo

This is a quite serious issue, scripts are currently hardly debuggable
without proper stack traces being available.

[Jon Leighton]

On 14/03/12 10:02, Nicolas Perriault wrote:
> function foo() { throw new Error('foo'); }
> function bar() { foo(); }
> bar();

Maybe I wasn't clear, but this will work (when I've finished my patch).
I can definitely make this produce a backtrace.

Regarding my previous email, on further reflection I think I am
mistaken. Consider this example:

page = require('webpage').create()

page.evaluate(function a() {
oh_no
})

That should actually produce a trace like:

test.js:3 in a()
test.js:2

I.e. it should include line(s) from the call stack outside the
page.evaluate().

And the trace should be reported to the "main" page (phantom.onError
callback or whatever - which by default would print to STDERR).

However this:

page = require('webpage').create()

page.evaluate(function a() {
setTimeout(function b() {
oh_no
}, 10)
})

should probably produce a trace like:

test.js:4 in b()

And since the error no longer occurs during the synchronous flow of the
main script, it should be reported to page.onError, rather than
phantom.onError).

So, looking at it like this I think I was confused and there is actually
not an issue, so I will just carry on hacking to hook this up...

--
http://jonathanleighton.com/

[Jon Leighton]

On 14/03/12 10:02, Nicolas Perriault wrote:

> This is a quite serious issue, scripts are currently hardly debuggable
> without proper stack traces being available.

I did some more work tonight. Please see here:
https://github.com/ariya/phantomjs/pull/221

--
http://jonathanleighton.com/

[Ariya Hidayat]

Some more details (and the history) in my blog post:
http://ariya.ofilabs.com/2012/03/the-evolution-of-phantomjs-build-workflow.html.