Greasemonkey GM_setValue and GM_getValue -- VERY SLOW

[Daniel Wynalda]

I have a script that is run by several thousand people all over the world.  It modifies Facebook pages when playing a game (providing coordination information helping game play).  

I have found that some users using the script running current Firefox and Greasemonkey 3.6 or greater are seeing significant slowdowns when running while others are not. 

The problem appears to be related to the response time of GM_getValue and GM_setValue for the more current versions of Greasemonkey.   

Has the method for saving GM_setValue and GM_getValue data changed in Greasemonkey?

The slowdown the users are experiencing immediately goes away if switching back to Greasemonkey 2.2.1 (but this requires going back to at least Firefox 43, to be compatible with the Greasemonkey version).

We are seeing speeds three times as fast accessing data on Greasemonkey 2.2 compared to Greasemonkey 3.6 or Greasemonkey 3.7.

We've resorted to moving some of the data out of GM_setValue locations into localStorage locations and other checks we've just skipped to reduce the delays. 

Any suggestions as to what might make GM_getValue and GM_setValue *VERY* slow on some profiles and not others would be greatly appreciated.  We are not sure if large amounts of data would be a detriment (meaning 10K instead of 1K total size -- not megabytes) or if something else could be making the profile very slow.   It is evident on the machines running slow that the profile is being accessed on disk over and over again causing significant latency waiting for the disk to read/write constantly.  

[Anthony Lieuallen]

On Fri, Feb 26, 2016 at 10:23 AM, Daniel Wynalda <dwyn...@gmail.com> wrote:

Has the method for saving GM_setValue and GM_getValue data changed in Greasemonkey?

Yes.  Compatibility with multi-process Firefox forced significant changes on our part.  No coincidence that the 2.x/3.x barrier is what you've noticed, given the 3.0 release notes ( http://wiki.greasespot.net/Version_history#3.0_2 ).  The scripts operate in the child where the content is, but only the parent can read/write from the disk.  The forced restriction on the (purely in-memory) preference store also didn't help.

 

Any suggestions as to what might make GM_getValue and GM_setValue *VERY* slow on some profiles and not others would be greatly appreciated.  ...  It is evident on the machines running slow that the profile is being accessed on disk over and over again causing significant latency waiting for the disk to read/write constantly.

You already have more data than we do.   Can you find a pattern?  Hardware?  Operating system and/or version?  Etc?

We've resorted to moving some of the data out of GM_setValue locations into localStorage locations and other checks we've just skipped to reduce the delays.

If localStorage works, why are you not relying on that completely?

[Daniel Wynalda]

localStorage seems to reduce the disk usage but some of the data I must store must be available cross domain.  This is where I'm stuck using the Greasemonkey storage values.   The script I wrote works on a number of domains (specifically facebook, the amazon hosting domain for the game, and my own domains -- such that it allows data to be kept and organized and allows the various people using the software to communicate with each other "around" the facebook/amazon link.   Ie:  Coordination that can be done together with people they don't even know...)

I'm glad to have your confirmation that something changed that could cause this.  Oddly I have not been able to get data that positively identifies what might be making some machines slow.   I'll have I7 machines running very fast hardware and network connections dragging like crazy and I'll have reasonable i5 machines working without delay almost as fast as version 2.2.     

The thing that is getting me is that some people will be running along really fast and then all of a sudden it's like the brake got put on the browser.  It will run slow for all actions and continue to do so even if the browser is closed completely, machine is restarted and cache is cleared.   It definitely has something to do with the browser profile storage (at least that's my theory) because we can see the hard drive being thrashed on the machines in question.

The slowdown is definitely exacerbated on virtual machines and machines where the profile is stored over a network or on a slower hard drive.  

Thanks for your input.  I'm going to keep researching.   The main reason I asked the question is I wondered if there was a limit on the amount of storage (total data) that could be kept with GM_setValue/GM_getValue and if when it hits a specific threshold it starts keeping the data via another method.  

I notice that old versions of Greasemonkey kept the setValue data in the area where about:config in firefox could see them.  Now they aren't visible.  Are there any tools that can be used to identify what is stored in the GM_getValue and GM_setValue storage area?   I'd like to see if there's something unique about the data being stored for specific people (perhaps a runaway loop storing a ton of data or something)

[Anthony Lieuallen]

On Sat, Feb 27, 2016 at 7:42 PM, Daniel Wynalda <dwyn...@gmail.com> wrote:

I notice that old versions of Greasemonkey kept the setValue data in the area where about:config in firefox could see them.  Now they aren't visible.  Are there any tools that can be used to identify what is stored in the GM_getValue and GM_setValue storage area?   I'd like to see if there's something unique about the data being stored for specific people (perhaps a runaway loop storing a ton of data or something)

Normally digging into internals like this isn't supported, but for debugging it's reasonable.  In the http://wiki.greasespot.net/Scripts_directory where your script is installed, there should be a .db file.  It's a SQLite database, and that's where the preferences are now stored.

[spirit]

W dniu niedziela, 28 lutego 2016 01:42:12 UTC+1 użytkownik Daniel Wynalda napisał:

I notice that old versions of Greasemonkey kept the setValue data in the area where about:config in firefox could see them.  Now they aren't visible.  Are there any tools that can be used to identify what is stored in the GM_getValue and GM_setValue storage area?   I'd like to see if there's something unique about the data being stored for specific people (perhaps a runaway loop storing a ton of data or something)

You are right, I notice the same speed problem when jump form older GM (store data in prefs.js) to new GM (store data in SQLite). I save a lot of data URL for future processing and remember that after update GM then this new storage system was worse. The only positive was not rubbish prefs.js. But really if you have developed a script that works on multiple tabs (in the same time) then all this new Firefox version is "crap", release before 2 o 3 years works better (especially when it comes to UI responsiveness or network operations). Even turn on e10s does not help,  what really scares me.