Suggestion to solve #329534 [Firefox:Places]-Live bookmarks load way too agressive (lock up browser) [Win]
p3ngwin
*#329534 [Firefox:Places]-Live bookmarks load way too agressive (lock up
browser) [Win]* <https://bugzilla.mozilla.org/show_bug.cgi?id=329534>
as an example of how bad it can be, i have about 60+ RSS feeds using
live-bookmarks in firefox, using nightlies. when the time comes to
update the RSS feeds, firefox locks up for seconds and is TOTALLY
unresponsive until the RSS feeds are updated. the disk gets thrashed to
oblivion and you get to hear the joy of you computer screaming in place
of a pleasant browsing experience.
it doesn't matter what interval yo have set for the update frequency, i
have mine set to update every 10 minutes as i like information as
"real-time" as possible, you can imagine what over 60 RSS feeds can do
to firefox updating like that!
the solution i believe is quite simple.
a simple bit of realtime maths that the browser needs to do that makes
the whole process more dynamic with less "hard coded" numbers". instead
of forcing firefox to update ALL RSS feeds at the same time, offset each
RSS feed update so they are the same frequency but NOT the synchronised.
example:
60 RSS feeds updated ALL together, every 10 mins =dead firefox.
better example:
whatever frequency is set for the update interval is DIVIDED by the
number of RSS feeds.
*E.G. 60 feeds/10minutes=update one feed every 10 secs*
each feed is updated ONE feed at a time, yet they all have the same
frequency. just update the next one on the list of feeds every 10 secs.
even extreme cases like my 60 RSS feeds every 10 minutes........then
EACH individual RSS feed would be updating ONCE every 10 seconds!
while still all being updated every 10 minutes, just not all simultaneously.
and to cap off the elegance of this solution to hard coded rules and
inflexibility, make firefox re-calculate the NEW division every time the
user makes/deletes an RSS feed. simple!
Enrico Weigelt
Hi,
> as an example of how bad it can be, i have about 60+ RSS feeds using
> live-bookmarks in firefox, using nightlies. when the time comes to
> update the RSS feeds, firefox locks up for seconds and is TOTALLY
> unresponsive until the RSS feeds are updated.
there're many other reasons for moz getting totally unresponsive
for some time, ie. tab actions when having more then about 20 ones
opened, accessing large mailboxes, rendering large and complex pages,
bulk history access, horribly broken plugins, etc, etc, etc.
> better example:
> whatever frequency is set for the update interval is DIVIDED by the
> number of RSS feeds.
> *E.G. 60 feeds/10minutes=update one feed every 10 secs*
> each feed is updated ONE feed at a time, yet they all have the same
> frequency. just update the next one on the list of feeds every 10 secs.
this still produce hangs, just short ones but many.
Why not just moving out the whole scanning stuff into an separate
server process, ie. connected via 9P ?
An simple way would be putting the whole RSS rerieval and parsing
into its own synthetic filesystem and present the content in an
very simple hierachical structure.
For example the per-feed structure could look like this:
./
subscription/
url
interval
last-scan
next-scan
status
head/
title
link
description
language
ttl
image-url
image-link
image/
url
link
description
items/
<list of per-items structure>
And the per-item structure:
./
title
description
guid
guid-permalink
published-date
seen-date
attributes/
<list of attributes>
The main tree could look like that:
./
subscriptions/
<list of per-subscription structure>
control/
status
The server handles the whole subscription management and data
retrieval. Mozilla's just one client who calls the feed server
and presents the stuff to the user.
Writing 9P applications is faily simple. For example libixp
and libmixp provide simple API for 9P clients (libmixp already
supports ninep URI's). And libmixpsrv is an easy-to-use library
for writing tiny 9P servers. Both are lightweight, portable,
thread-safe and written in plain-C.
cu
--
---------------------------------------------------------------------
Enrico Weigelt == metux IT service - http://www.metux.de/
---------------------------------------------------------------------
Please visit the OpenSource QM Taskforce:
http://wiki.metux.de/public/OpenSource_QM_Taskforce
Patches / Fixes for a lot dozens of packages in dozens of versions:
http://patches.metux.de/
---------------------------------------------------------------------
p3ngwin
suggested idea?
the way i thought of my idea was of "spreading" a load over a time
period asynchronously, yet maintaining the same frequency for the RSS feeds.
creating a separate process....hmmm. if that would free up the browser
so that the feeds are updated in a way that interrupts the browser
less/not at all, then brilliant. but surely this STILL means that
Firefox is doing a load of things at the same time? i'm not sure on the
method you explained as it was WAY over my head, i don't know about
server stuff and the API's you mentioned :(
just that it sounded like moving the feeds updating to another process,
which i understand to mean that it will happen in the background and not
be "joined" to firefox as much, hence not impacting the main browsing
experience as much.
my thought is that still this would mean that even though the feeds
updating is "separated" a bit, that it doesn't change the fact that
firfox is wanting to do too much at once when updating the feeds, hence
my idea to "spread the load over time", whilst maintain the frequency
per feed.
perhaps you can simplify your idea for someone like me to understand?
then i'll have a better ideas of the 2 ideas and how they relate to the
problem.
thanks
CJ JUSZCZAK
Enrico Weigelt
Hi,
> would creating a seperate server process reduce the hangs better
> than my suggested idea?
Yes, because that server isn't bound to mozilla's insane codeflow,
allows very easy coding because it's living in it's own process
and will most likely use threads.
Of course your idea might find its home within that server :)
(but that will be no longer in mozilla's scope ;-P)
> creating a separate process....hmmm. if that would free up the
> browser so that the feeds are updated in a way that interrupts
> the browser less/not at all, then brilliant.
Yes, if the server is sanely coded, ie. using separate threads
per connections and crawlers, clients won't be interrupted by
each other or the crawling (besides the fact that your machine
might be overloaded at all ;-P).
> but surely this STILL means that Firefox is doing a load of
> things at the same time?
FF and other processes consuming resources, as usual.
With the new server we've just separated things, makes it much
easier to handle and heavily reduces complexity. So we also
save resources :)
The load reduction is still another issue, but no longer in
scope of mozilla. If you don't have too many (ie. thousands of)
feeds, it (IMHO) won't be bad looping through all reached the
rescan-timeout and maybe sleep a few seconds between scans.
> i don't know about server stuff and the API's you mentioned :(
You might subscribe to the 9fans list:
https://lists.cse.psu.edu/mailman/listinfo/9fans
> just that it sounded like moving the feeds updating to another process,
> which i understand to mean that it will happen in the background and not
> be "joined" to firefox as much, hence not impacting the main browsing
> experience as much.
Exactly. If the server's coded well, it won't affect more than
any other process on the system.
> perhaps you can simplify your idea for someone like me to understand?
> then i'll have a better ideas of the 2 ideas and how they relate to the
> problem.
Well, the optimal solution seems to be an combination of both ideas:
a) the external (multithreaded) server will reduce code complexity
very much and remove one of the many lockup points in mozilla.
b) an clever balancing as you suggested can prevent excessive load peaks.
p3ngwin
people that need to know? :)
Boris Zbarsky
> Yes, because that server isn't bound to mozilla's insane codeflow,
Which part is insane?
> allows very easy coding because it's living in it's own process
> and will most likely use threads.
Mozilla does use threads, you realize? ;)
-Boris
Boris Zbarsky
> ok, sounds like we have a 2 stage plan, now how do we propose it to the
> people that need to know? :)
The number one best way is to create a patch implementing the idea.
-Boris
p3ngwin
crap i'm no coder :(
what's the next best way to get their attention?
Boris Zbarsky
> what's the next best way to get their attention?
Who is this "they" you're speaking of?
-Boris
Enrico Weigelt
> Enrico Weigelt wrote:
> > Yes, because that server isn't bound to mozilla's insane codeflow,
>
> Which part is insane?
In fact, almost evrything ;-P
I don't want to go into details now - I'll grab out several
points one by one and start different threads for that.
Once thing is the streams api/classes: far too complex.
Maybe it had been historically an good decision, but today I
see no value in the whole async+notification stuff.
Why can't it just (almost) the same semantics like the postix/
unix style filesystem interface plus some additional functionality
(ie. metadata handling) ?
> > allows very easy coding because it's living in it's own process
> > and will most likely use threads.
>
> Mozilla does use threads, you realize? ;)
Yes, but obviously not in an very good way, since we still have
many, many blockings.
Let's take some example each viewer should entirely run in its
own thread, and the GUI (maybe including the step of painting
the current visible viewer(s)) should get it's own. If the user
clicks on some link, the triggered action (ie. loading some page,
running some jscript, etc) should be sent to the viewer's thread
which will do all the necessary things and maybe send back some
message to the gui thread that something should be repainted.
The fact that even when simultaniously loading 20 pages the number
of threads/processes is still about 8 tells me that it's NOT
done the way I just described ;-P
Enrico Weigelt
Well, you should learn it ;-P
Trying to code within mozilla is probably not a good stating
point for learning the basics, but you might join us @9fans
list and help us in the lib(m)ixp project (it's fairly simple) :)
Enrico Weigelt
> ok, sounds like we have a 2 stage plan, now how do we propose it to the
> people that need to know? :)
hmm, maybe write some (good) paper and spread it around the world ? ;-O
Boris Zbarsky
> Maybe it had been historically an good decision, but today I
> see no value in the whole async+notification stuff.
> Why can't it just (almost) the same semantics like the postix/
> unix style filesystem interface
Because that interface blocks on I/O operations. That's reasonably acceptable
with a local filesystem (though applications that need high I/O throughput end
up using different, async APIs anyway), but when your I/O is has latency
measured in seconds (as with HTTP), that's just not acceptable.
Put another way, once you start loading an image linked from an HTML document,
you want to be able to do other things while the network coughs up the bits.
With a blocking API you can't. One could do a non-blocking API with polling,
but at that point a non-blocking API with notifications is clearly better.
> Let's take some example each viewer should entirely run in its
> own thread
This has been discussed many times in the past, and I suggest you read those
discussions. To summarize, the viewers and the GUI can all touch each other and
they use the DOM APIs to do so, which means that putting them on different
threads would impose synchronization overhead on those APIs.
Put another way, it would be possible to do what you propose, but there would
obviously be a noticeable performance impact. So you'd definitely get
across-the-board slowdown. You might eliminate the freeze issues, but you might
not. For example, if the UI and the page are on different threads, and the UI
needs information from the page and is waiting on a lock while the page has
started a rendering pass, you get a freeze as now. Threads are not a panacea.
All that said, this might be worth trying. It's a huge undertaking with obvious
costs and possibly no reward, which is why no one's done it yet.
> and the GUI (maybe including the step of painting
> the current visible viewer(s)) should get it's own.
You do realize that said painting would have to touch data on all the threads in
question, right?
-Boris
p3ngwin
it's a mailing list about Plan9 from Bell Labs....but WHAT IS IT??