Turpial 2.0 Performance

[Wil Alvarez]

Hi everybody,

Turpial 2.0 has been fully tested out the last days and we discover some performance issues. William Cabrera reported a segment fault after several hours using Turpial, so we studied this weird behavior (is not common to see segment faults in python) and after some tracebacks we observed that Turpial was consuming a lot of memory and when reached 900MB of RAM the segment fault closed the application.

I did a lot of analysis about memory consumption and memory profiling using [1] [2] [3] [4] and discovered the problem, tweets were keeping in memory (in an array) and in HTML view ad infinitum, making the memory grows and grows until the OS decided to close the application. You can see details in attached file heapy1.log, there are observed 40655 strings objects and a lot of non freed objects (dicts, tuples, etc) with alive references. I fixed the issue and tested again and Turpial was working pretty well and with no memory leaks and just using a little bit more than 90MB of RAM.

I remembered that Turpial 1.6.x used like 50~60MB of RAM so I decided to run some new tests and tried to improve a couple of critical sections of code but the consumption was around 90MB yet. I did whatever crossed my mind and, according test results (details in attached file heapy2.log), Turpial base memory usage decrease from 14MB to 300KB, however system monitor still said that it was around 90MB. Finally, I decided to test just the HTML render engine (Webkit and all that stuff) so I removed all libturpial functionality (even imports) and tested just HTML (like a kind of browser) and for my surprise memory consumption was around 50MB. This means that only Webkit uses around 50MB of RAM and Turpial base functionality is around 40MB, that is mora then less 90MB in total.

So, here we are... Turpial is using almost 100MB of RAM while heavy applications like Banshee are using 60MB and gnome-shell uses 120MB. For my this is a really hard situation, I mean, should we sacrifice performance in favor of functionality, beauty, cool effects and the same interface in all platforms (HTML) or should we think about another approach (maybe gtk3, or gtk2 + clutter, whatever else)?

Please I would like to read your feedback.

Regards

[1] http://docs.python.org/library/hotshot.html

[2] http://docs.python.org/library/pdb.html

[3] http://www.smira.ru/wp-content/uploads/2011/08/heapy.html

[4] http://www.pixelbeat.org/scripts/ps_mem.py

--
“Yo construyo Soberanía, uso Software Libre”
Wil A. Alvarez
Linux Counter #415026

Debian Counter #259

http://turpial.org.ve
http://damncorner.blogspot.com/

[José Rafael Torres]

Hi guys

As you all might know already I'm not a dev but ill give you my
opinion from my user point of view.

I'm using Canaima and other GNU/Linux distributions since couple of
years but mainly since almost 1 year and i have to say that Iḿ totally
into this for several reasons but mainly cause of what i can do with
this compared with privative OS, so i think desktop applications for
regular users are more appreciated if don't eat up all of your
memory/resources.

Iḿ not in to fancy rainbow aplications with a lot of graphics, I'm
using turpial 1.6.6 an love it as it is cause of his light interface,
of course I'm expecting the new features but most of it i hope it can
be done without put to much beauty - hair style that consume memory,
lol...

Ill like to see what others think

Best Regards

José Torres
http://www.redradioypc.com/?Contacto

[Wil Alvarez]

Yeap, I would like to read opinions of others too to discuss and make a good decision. It's a tough compromise between resources and benefits

[Eleazar Meza]

While as a Gnome 3 user I'm not a big fan of webkit-desktop-integrated apps (Does that even make sense?), I'm aware of the effort Wil and the rest of the devs involved have put into making the new webkit ui a reality, it's a great achievement, turpial is now way more usable than ever before. Webkit is a powerful tool and it will allows us to do a lot more with relative little effort (GTK2 hell? hello?). Also, since there's a lot of people out tuere with webapp programming skills, we're making it easier for those developers to help us out if they wan to ;)

But that power comes at a cost, the fucker sucks memmory like hell, as far as I know the rendering is not hardware accelerated, every fancy javascript transition will suck a considerable amount of cpu if we go nuts with the eyecandy and since webkit is basically a white canvas, we have to come up with UI guidelines of our own in order to build a consistent ui.

I'm all for native widgets (Qt4, Gtk3), they're simply the right way to do desktop integrated apps, the lastest versions available are powerful enough to give us all the tools we need to make the things webkit is doing right now (of course not all the things, only a tiny subset, but that's good, that's all we need). So what's wrong with this approach? It takes time, it is hard, we can't just get stucked with gtk2 until we're ready to use modern widgets.

So what's my advice? Let's use webkit in the meantime, but let's make it count, trimming things where possible and avoid the trap of going nuts with the eyecandy just because it's easy to do so.

I'll be working with Wil to clean up the html/css code and maybe blow up the horizontal dock (hey it was my idea in the firstplace, I drew it on a piece of paper almost 3 years ago) it's an outdated design, trust me ;)

[Wil Alvarez]

El 19 de febrero de 2012 14:13, Eleazar Meza <meza.e...@gmail.com> escribió:

While as a Gnome 3 user I'm not a big fan of webkit-desktop-integrated apps

I prefer native apps too but Webkit is letting us close the gap between gtk2 and gtk3, and as you said, there are more people with HTML skills than with native frameworks skills.

The main effort on this Webkit development was putting all together and make it work inside the desktop in a seamless way but I'm not closed to migrate to a gtk3/qt4 (or even cocoa) native implementation. I just want the best performance with the best UI, that's the tough compromise what I was talking before.

But that power comes at a cost, the fucker sucks memmory like hell, as far as I know the rendering is not hardware accelerated, every fancy javascript transition will suck a considerable amount of cpu if we go nuts with the eyecandy and since webkit is basically a white canvas, we have to come up with UI guidelines of our own in order to build a consistent ui.

Yeap, as uncle Ben said: "With great power comes great responsability". The question is: We are not using all this HTML fancy features currently, it worth to have such a big webkit machinery to render simple HTML pages just with some javascript code and smooth transitions? How much (in resources term) are we willing to pay to use HTML features that probably we can achieve with gtk3/qt4? Obviously, HTML has advantages but resources consumption is a critical issue that we need to discuss. I just can't imagine Turpial running in my netbook and sucking 100MB of RAM, I mean, I could but it should really worth, Turpial should be an amazing app, that's the point.

So what's my advice? Let's use webkit in the meantime, but let's make it count, trimming things where possible and avoid the trap of going nuts with the eyecandy just because it's easy to do so.

We must optimize A LOT of things with Turpial/libturpial to get a decent memory consumption under webkit. I've been tested disabling a lot of webkit features (obviously not needed) and I decreased the consumption from 115MB to 89MB (under similar cincurnstances).

I'll be working with Wil to clean up the html/css code and maybe blow up the horizontal dock (hey it was my idea in the firstplace, I drew it on a piece of paper almost 3 years ago) it's an outdated design, trust me ;)

Another thing I would like to point out is this: Last night, restorer (I don't know his real name) told me by IRC that he tested the new Turpial 2.0 UI and he wasn't excited. After that comment he shows me this pic [1] and told me that old interface was compact and readable. I took the time to analyze the pic and here are my conclusions:

* The new interface is far away simpler

* We gain a lot of space with HTML but we show less tweets

* We show less tweets but each tweet has more useful information (fav mark, protected users mark, retweeted mark, etc)

* Probably we can hack the new interface to gain a little bit more of space

I don't know if this conclusions are enough to answer restorer's questions but we can discuss about it.

So, to make a decision we need to start working, that's why I'll be on #turpial IRC channel at freenode this thursday, 21/02/2012 at 10.00 am VE, 11.30 AR with Eleazar discussing about all this stuff and try to make the best decision. Everybody is invited to come and discuss with us about all this topics, just let us know if you want to participate

Cheers

[1] http://imgur.com/bod1M

[Wil Alvarez]

Ok, sorry I had a little mistake in my previous post, we're going to be working on #turpial IRC tuesday, TUESDAY  21/02/2012 not thursday ;)

That's it. Bye

[Eleazar Meza]

HOYGANS, actual users, HOYGAN devs, actual devs, you're all invited to show up.

Bring it, bitch.

[Matthew Hielscher]

On 02/19/2012 10:34 AM, Wil Alvarez wrote:
> Another thing I would like to point out is this: Last night, restorer (I don't know his real name) told me by IRC that he tested the new Turpial 2.0 UI and he wasn't excited. After that comment he shows me this pic [1] and told me that old interface was compact and readable. I took the time to analyze the pic and here are my conclusions:
>
> * The new interface is far away simpler
> * We gain a lot of space with HTML but we show less tweets
> * We show less tweets but each tweet has more useful information (fav mark, protected users mark, retweeted mark, etc)
> * Probably we can hack the new interface to gain a little bit more of space
>
> I don't know if this conclusions are enough to answer restorer's questions but we can discuss about it.
>

> [1] http://imgur.com/bod1M

restorer/WasabiFlux/Matt here,

I do think the interface is still better in several ways than any other Twitter client I've tried. I do support keeping with HTML because of the potential for making layout preferences a moot point with customizable themes. The things I don't like about the look of it now:

- The Fav, Retweet, Quote, Reply, Delete buttons take up space whether they're visible or not.
- The username is now on a separate line from the tweet text.
- I preferred the previous font and size, and the spacing is less compact.

Obviously, those are all matters of preference in presentation. Someone else might like the new layout better than the old one, and that's fine. I only ask that it be customizable to a high degree later, which is why I think HTML with themes is the way to go if we can work out the memory usage.

> So, to make a decision we need to start working, that's why I'll be on #turpial IRC channel at freenode this [TUESDAY], 21/02/2012 at 10.00 am VE, 11.30 AR with Eleazar discussing about all this stuff and try to make the best decision. Everybody is invited to come and discuss with us about all this topics, just let us know if you want to participate

I'll try to be there. That's pretty early for me here in California.

-Matt

[Eleazar Meza]

On Sun, Feb 19, 2012 at 2:33 PM, Matthew Hielscher <mhiel...@gmail.com> wrote:
> On 02/19/2012 10:34 AM, Wil Alvarez wrote:

> ...The things I don't like about the look of it now:

>
> - The Fav, Retweet, Quote, Reply, Delete buttons take up space whether
> they're visible or not.
> - The username is now on a separate line from the tweet text.
> - I preferred the previous font and size, and the spacing is less compact.
>
> Obviously, those are all matters of preference in presentation. Someone else
> might like the new layout better than the old one, and that's fine. I only
> ask that it be customizable to a high degree later, which is why I think
> HTML with themes is the way to go if we can work out the memory usage.
>

> I'll try to be there. That's pretty early for me here in California.
>
> -Matt

Matt and Jose Rafael Torres: You're giving exactly the kind of
feedback we need to hear, honest and egotistic crap free.

I really appreciate you guys helping us. Even if you don't consider
developers yourselves, I'd say you are, simply because you're putting
your time to help us build a better software, just like any of us are
;).

If someone doesn't like something, say it out loud, if you choose to
bitch on the process, demand things using an authoritarian tone,
disrespect someone, and HOYGANIng in general, we'll still read your
critics (we need them) but don't get mad if you feel ignored haha.

Best regards

PS: Matt: Want some venezuelan coffee? It might help ;)

[José Rafael Torres]

thanks Eleazar
hmnn... i have to say after seen the picture that i like more the
space use of the previous UI, i dont know, i read that ubuntu users
can install now tweetdeck, and i think turpial will be able to beat
that if offer the same or even better features with less resource
consumption... who knows maybe turpial end up being split in two
versions: a lite version that even mobile users can install on android
and a full version for powerfull desktops, lol...

in other words: give me back my old interface with new features but
light as hell or ill be mad!!! (joke :D)

[Edwind Richzendy Contreras Soto]

I've always liked light applications, but I also love the functional applications, I tried developing turpial 2.0 and I definitely like what I see.

To be honest, I'm not the best person to put me in favor of the old gtk interface, because with 4GB of ram in my laptop I do not mind losing in turpial 100mb if you really get the best possible user experience.

However I can say giving a technical argument:

The old interface can be lighter, but also results in an anchor that prevents turpial progress, it is difficult to port to other platforms turpial because of it, I think if you keep using webkit, perhaps the first release is a monster resource hog but I have faith that if you have a large application that calls the attention of other developers (windows, mac) and facilitates the work on it, very soon see more and more users and contributors who knows, maybe get some lose some turpial megs of consumption using tools that are available may not today.

That people are using HootSuite for their desktop client? the interface is amazing, maybe we could get some ideas from there

 

[Wil Alvarez]

Richy,

Definitely, gtk2 is limited but we are talking of a new gtk3 interface. However we must admit the power of webkit and HTML, I think everything is about what you said: "get the best possible user experience".

I would love to see what HootSuite offers, and see how can we implement the best of it. Anyone using HootSuite already that can show us those features?

[José Rafael Torres]

with this argument ill have to turn back to myself and agree with
Richzendy... it might help with turpial progress if attract more
developers, even i can make some coding on HTML / CSS.

about HootSuite, for me its not just the interface (http://
omploader.org/vY3U5eg) or the features (http://hootsuite.com/about)
but the Hootlet (http://help.hootsuite.com/entries/131320-what-is-the-
hootlet) that let you publish/schedule to any of the social accounts
from any site...


On 19 feb, 17:05, Edwind Richzendy Contreras Soto

[Carlos Guerrero]

The 2.0 UI can be easily fixed to take all the best from 1.0 that it's missing right now, it's just matter of a HTML template.

I believe most of us have faith in all the advantages that webkit brings to Turpial, rather than the RAM consumption (wich we must also struggle to be as less as possible), anyway, see you all at #turpial IRC, TUESDAY  21/02/2012  :)

Best Regards.

2012/2/19 José Rafael Torres <great...@gmail.com>

--
Atte:
Carlos A. Guerrero M.
http://www.carlosguerrero.com

[Sebastián Magrí]

What if we try (again) to migrate to gtk3/pygobject while keeping
webkit, now using gobject introspection, and compare its performance.

I believe this will be needed sooner or later and maybe over the way
we will find things we can replace to improve the performance...

Regards.

2012/2/19 Carlos Guerrero <guerrer...@gmail.com>:

--
Sebastián Ramírez Magrí
http://sebasmagri.alwaysdata.net/

[Eleazar Meza]

2012/2/19 Sebastián Magrí <sebas...@gmail.com>:

> What if we try (again) to migrate to gtk3/pygobject while keeping
> webkit, now using gobject introspection, and compare its performance.

Interesting, it'd be nice to try that, maybe, just maybe, gtk3's
webkit is not that much of a resource hog with pygobject binding
things in a more streamlined way.... maybe as in 'I'm 99% sure it
won't make that much of a difference', but I could be wrong, I'd love
to be wrong on this one actually :D

I'll cook up a tiny little hello world test with webkit on gtk2 and
gtk3 when I get the time and post the results here if someone else
doesn't beat me to it.

Nos vemos en el espejo

[Sebastián Magrí]

2012/2/19 Eleazar Meza <meza.e...@gmail.com>:

I also think that it will not make a big difference. Anyway, as I
said, I believe that sooner or later we will need to migrate to
gtk3/pygobject, with or without webkit.

[Edwind Richzendy Contreras Soto]

2012/2/19 Wil Alvarez <wil.al...@gmail.com>

Richy,

Definitely, gtk2 is limited but we are talking of a new gtk3 interface. However we must admit the power of webkit and HTML, I think everything is about what you said: "get the best possible user experience".

I would love to see what HootSuite offers, and see how can we implement the best of it. Anyone using HootSuite already that can show us those features?

my mistake, I was confused with hootsuite, the program is _Hotot_[1], my wife uses it and I've seen him use that is very functional (I never tried it, I'm use turpial: D), is in most repositories of distributions, so it is easy to prove.

I propose to examine the way it is constructed and analyzed some of its features, compare turpial Hotot with gtk, is like comparing web 1.0 html web 2.0 with ajax


[1] http://xenodesystems.blogspot.com/2012/02/hotot-el-mejor-cliente-de-twitter.html