Is yielding on Zotero.initializationPromise enough for Zotero.Items.getAsync to work?

[Emiliano Heyns]

I'm getting transient errors where Zotero.Items.getAsync yields without error, but if I call getField on that item sometimes (early in the startup but after Zotero.initializationPromise yields) I get "Item data not loaded and field 'extra' not set for item".

[Emiliano Heyns]

On Friday, August 18, 2017 at 5:59:34 PM UTC+2, Emiliano Heyns wrote:

I'm getting transient errors where Zotero.Items.getAsync yields without error, but if I call getField on that item sometimes (early in the startup but after Zotero.initializationPromise yields) I get "Item data not loaded and field 'extra' not set for item".

I just found https://groups.google.com/d/msg/zotero-dev/naAxXIbpDhU/iSLpXo-UBQAJ . What must I do after getAsync to ensure all the item data has been loaded? 

[Emiliano Heyns]

I need creators, itemdata (fields) and tags to be loaded for a given item, to be specific. 

[Dan Stillman]

A couple different issues here:

- You shouldn't do anything other than UI modification before Zotero.uiReadyPromise — otherwise you'll be negatively affecting Zotero load time. And really it's best to wait for schemaUpdatePromise, which needs to finish before various dialogs/operations work properly. (initializationPromise only existed for Firefox mode and will likely be removed or made private at some point.)

- Once uiReadyPromise is done, all data in the currently selected library will be loaded, and you don't need to use getAsync() on it. If you need to interact with objects in a potentially unloaded library, you have to 1) use getAsync() and 2) call the asynchronous item.load* (e.g., loadItemData, loadCreators(), loadTags(), or just loadAllData()) methods yourself for the data you need. (What I said earlier about getAsync() being enough isn't true, though in practice it often doesn't matter because people will have either clicked on the library or synced (which loads all data in all synced libraries).)

[Emiliano Heyns]

On Friday, August 18, 2017 at 11:19:38 PM UTC+2, Dan Stillman wrote:

A couple different issues here:

- You shouldn't do anything other than UI modification before Zotero.uiReadyPromise — otherwise you'll be negatively affecting Zotero load time. And really it's best to wait for schemaUpdatePromise, which needs to finish before various dialogs/operations work properly. (initializationPromise only existed for Firefox mode and will likely be removed or made private at some point.)

That may put me in a bit of a bind; there is BBT initialisation work I would want done before the UI becomes active but which needs DB access, so waiting for uiReadyPromise would seem counter to that goal. schemaUpdatePromise reliably takes between 10 and 30 seconds to resolve on the systems I work with (CircleCI containers, my 2015 Macbook Air, and a 2017 Zbook 15 G3), and from what I can tell users can start to interact well before that, which would give me an uncertain starting state.

- Once uiReadyPromise is done, all data in the currently selected library will be loaded, and you don't need to use getAsync() on it. If you need to interact with objects in a potentially unloaded library, you have to 1) use getAsync() and 2) call the asynchronous item.load* (e.g., loadItemData, loadCreators(), loadTags(), or just loadAllData()) methods yourself for the data you need.

Out of curiosity, in which cases would Zotero itself call getAsync but not follow it immediately with item.load*?

 

(What I said earlier about getAsync() being enough isn't true, though in practice it often doesn't matter because people will have either clicked on the library or synced (which loads all data in all synced libraries).)

Right, that's what I was aiming to refer to; you mentioned that getAsync is not just an async get. 

[Dan Stillman]

On 8/19/17 10:55 AM, Emiliano Heyns wrote:

On Friday, August 18, 2017 at 11:19:38 PM UTC+2, Dan Stillman wrote:

A couple different issues here:

- You shouldn't do anything other than UI modification before Zotero.uiReadyPromise — otherwise you'll be negatively affecting Zotero load time. And really it's best to wait for schemaUpdatePromise, which needs to finish before various dialogs/operations work properly. (initializationPromise only existed for Firefox mode and will likely be removed or made private at some point.)

That may put me in a bit of a bind; there is BBT initialisation work I would want done before the UI becomes active but which needs DB access, so waiting for uiReadyPromise would seem counter to that goal. schemaUpdatePromise reliably takes between 10 and 30 seconds to resolve on the systems I work with (CircleCI containers, my 2015 Macbook Air, and a 2017 Zbook 15 G3), and from what I can tell users can start to interact well before that, which would give me an uncertain starting state.

Well, again, various Zotero dialogs and operations involving translators or styles won't work until schemaUpdatePromise is resolved, so that's clearly the priority. And since schemaUpdatePromise can include schema updates, you don't even know that the DB is in an up-to-date state until that's done.

I'm not sure what you mean by an uncertain starting state. Everything you do should just be dependent on that having been resolved, the same way that various dialogs in Zotero wait for that. (You can, e.g., display a "Loading…" line if some action is performed before then.)

But also, for me, on a 2013 rMBP, schemaUpdatePromise is consistently resolved 5 seconds after uiReadyPromise (which is the only time that really matters for this, because displaying data takes precedence over absolutely everything). It's fairly easy to run up against that as a developer, testing a specific function, but there's not a lot a user is going to do in 5 seconds. More than likely they're opening Zotero to save an item, so the priority is having translators be ready to use.

- Once uiReadyPromise is done, all data in the currently selected library will be loaded, and you don't need to use getAsync() on it. If you need to interact with objects in a potentially unloaded library, you have to 1) use getAsync() and 2) call the asynchronous item.load* (e.g., loadItemData, loadCreators(), loadTags(), or just loadAllData()) methods yourself for the data you need.

Out of curiosity, in which cases would Zotero itself call getAsync but not follow it immediately with item.load*?

It generally would, but not necessarily all data. E.g., it might just get item data to display a title. We could consider having getAsync() automatically load all data, but we'd have to review all the uses of it.

[Emiliano Heyns]

On Monday, August 21, 2017 at 3:46:13 PM UTC+2, Dan Stillman wrote:

On 8/19/17 10:55 AM, Emiliano Heyns wrote:

On Friday, August 18, 2017 at 11:19:38 PM UTC+2, Dan Stillman wrote:

A couple different issues here:

- You shouldn't do anything other than UI modification before Zotero.uiReadyPromise — otherwise you'll be negatively affecting Zotero load time. And really it's best to wait for schemaUpdatePromise, which needs to finish before various dialogs/operations work properly. (initializationPromise only existed for Firefox mode and will likely be removed or made private at some point.)

That may put me in a bit of a bind; there is BBT initialisation work I would want done before the UI becomes active but which needs DB access, so waiting for uiReadyPromise would seem counter to that goal. schemaUpdatePromise reliably takes between 10 and 30 seconds to resolve on the systems I work with (CircleCI containers, my 2015 Macbook Air, and a 2017 Zbook 15 G3), and from what I can tell users can start to interact well before that, which would give me an uncertain starting state.

Well, again, various Zotero dialogs and operations involving translators or styles won't work until schemaUpdatePromise is resolved, so that's clearly the priority. And since schemaUpdatePromise can include schema updates, you don't even know that the DB is in an up-to-date state until that's done.

Ah. OK.

 

I'm not sure what you mean by an uncertain starting state. Everything you do should just be dependent on that having been resolved, the same way that various dialogs in Zotero wait for that. (You can, e.g., display a "Loading…" line if some action is performed before then.)

Where/how could I best display this? I currently use Zotero.ProgressWindow(), but sometimes that popup lingers longer than I'd want when I had overestimated the time something would take, or it would be gone before the action is finished in the reverse case.

 

But also, for me, on a 2013 rMBP, schemaUpdatePromise is consistently resolved 5 seconds after uiReadyPromise (which is the only time that really matters for this, because displaying data takes precedence over absolutely everything). It's fairly easy to run up against that as a developer, testing a specific function, but there's not a lot a user is going to do in 5 seconds. More than likely they're opening Zotero to save an item, so the priority is having translators be ready to use.

I'd agree, but on my ZBook G3 the time between  uiReadyPromise and schemaUpdatePromise is 12 seconds. Between those I see no BBT activity (although abviously not every line of code is logged) but I do see a lot of log lines like

SELECT IA.itemID FROM itemAttachments IA NATURAL JOIN items I LEFT JOIN itemData ID ON (IA.itemID=ID.itemID AND fieldID=1) LEFT JOIN itemDataValues IDV ON (ID.valueID=IDV.valueID) WHERE parentItemID=? AND linkMode NOT IN (?) AND IA.itemID NOT IN (SELECT itemID FROM deletedItems) ORDER BY contentType='application/pdf' DESC, value=? DESC, dateAdded ASC [2, 3, '']

and then a lot like

 REPLACE INTO translatorCache VALUES (?, ?, ?)

could I be causing those? I defer the installation of my translators and reinit until schemaUpdatePromise resolves, so that should not be the cause.

 

Out of curiosity, in which cases would Zotero itself call getAsync but not follow it immediately with item.load*?

It generally would, but not necessarily all data. E.g., it might just get item data to display a title. We could consider having getAsync() automatically load all data, but we'd have to review all the uses of it.

Got it. 

 

[Emiliano Heyns]

On Monday, August 21, 2017 at 4:37:43 PM UTC+2, Emiliano Heyns wrote:

Well, again, various Zotero dialogs and operations involving translators or styles won't work until schemaUpdatePromise is resolved, so that's clearly the priority. And since schemaUpdatePromise can include schema updates, you don't even know that the DB is in an up-to-date state until that's done.

Ah. OK.

 

I'm not sure what you mean by an uncertain starting state. Everything you do should just be dependent on that having been resolved, the same way that various dialogs in Zotero wait for that. (You can, e.g., display a "Loading…" line if some action is performed before then.)

What I meant here is that e.g. the user could have started interacting with the DB before my cache is filled. There are parts of my cache I load from disk during startup -- if the user has changed the reference before my cache loads, the cache will be out of sync with the DB. I listen for notifiers to invalidate my cache, but that only works when the cache is in sync before those notifications come in. But a display of a "loading..." line can ameliorate that potential problem.

[Dan Stillman]

On 8/21/17 4:37 PM, Emiliano Heyns wrote:

 

I'm not sure what you mean by an uncertain starting state. Everything you do should just be dependent on that having been resolved, the same way that various dialogs in Zotero wait for that. (You can, e.g., display a "Loading…" line if some action is performed before then.)

Where/how could I best display this? I currently use Zotero.ProgressWindow(), but sometimes that popup lingers longer than I'd want when I had overestimated the time something would take, or it would be gone before the action is finished in the reverse case.

It depends on your UI, but if you do use Zotero.ProgressWindow(), you can close the popup manually.

 I'd agree, but on my ZBook G3 the time between  uiReadyPromise and schemaUpdatePromise is 12 seconds. Between those I see no BBT activity (although abviously not every line of code is logged) but I do see a lot of log lines like

SELECT IA.itemID FROM itemAttachments IA NATURAL JOIN items I LEFT JOIN itemData ID ON (IA.itemID=ID.itemID AND fieldID=1) LEFT JOIN itemDataValues IDV ON (ID.valueID=IDV.valueID) WHERE parentItemID=? AND linkMode NOT IN (?) AND IA.itemID NOT IN (SELECT itemID FROM deletedItems) ORDER BY contentType='application/pdf' DESC, value=? DESC, dateAdded ASC [2, 3, '']

You'll see one of these for each item with an attachment in view — that's updating the attachment presence indicators and is triggered automatically by the tree.

and then a lot like

 REPLACE INTO translatorCache VALUES (?, ?, ?)

could I be causing those? I defer the installation of my translators and reinit until schemaUpdatePromise resolves, so that should not be the cause.

You shouldn't be seeing those for normal startups in an existing data directory — those only happen when Zotero is updated or for a new data directory. I'm planning to try to speed those up, but if you're seeing those, something is wrong, and that's definitely why it's taking so long for you.

[Emiliano Heyns]

On Monday, August 21, 2017 at 5:17:06 PM UTC+2, Dan Stillman wrote:

On 8/21/17 4:37 PM, Emiliano Heyns wrote:

 

I'm not sure what you mean by an uncertain starting state. Everything you do should just be dependent on that having been resolved, the same way that various dialogs in Zotero wait for that. (You can, e.g., display a "Loading…" line if some action is performed before then.)

Where/how could I best display this? I currently use Zotero.ProgressWindow(), but sometimes that popup lingers longer than I'd want when I had overestimated the time something would take, or it would be gone before the action is finished in the reverse case.

It depends on your UI, but if you do use Zotero.ProgressWindow(), you can close the popup manually.

I did not know that. Do you know a place in the Zotero code that does this offhand? 

 

You shouldn't be seeing those for normal startups in an existing data directory — those only happen when Zotero is updated or for a new data directory. I'm planning to try to speed those up, but if you're seeing those, something is wrong, and that's definitely why it's taking so long for you.

Ah, that would explain a lot -- my tests move a snapshot data dir in place and set extensions.zotero.dataDir before Zotero starts up, but Zotero must detect that the data dir has changed another way. Do you know how this detection works?

[Emiliano Heyns]

On Monday, August 21, 2017 at 5:25:36 PM UTC+2, Emiliano Heyns wrote:

It depends on your UI, but if you do use Zotero.ProgressWindow(), you can close the popup manually.

I did not know that. Do you know a place in the Zotero code that does this offhand? 

I think I've found this.

 

You shouldn't be seeing those for normal startups in an existing data directory — those only happen when Zotero is updated or for a new data directory. I'm planning to try to speed those up, but if you're seeing those, something is wrong, and that's definitely why it's taking so long for you.

Ah, that would explain a lot -- my tests move a snapshot data dir in place and set extensions.zotero.dataDir before Zotero starts up, but Zotero must detect that the data dir has changed another way. Do you know how this detection works?

 

I've gone through the code but the only thing I could find that looked relevant was "lastDataDir'. But that isn't set in my test profile before the move, unless I'm missing where it lives.

[Emiliano Heyns]

On Monday, August 21, 2017 at 4:56:08 PM UTC+2, Emiliano Heyns wrote:

 

I'm not sure what you mean by an uncertain starting state. Everything you do should just be dependent on that having been resolved, the same way that various dialogs in Zotero wait for that. (You can, e.g., display a "Loading…" line if some action is performed before then.)

What I meant here is that e.g. the user could have started interacting with the DB before my cache is filled. There are parts of my cache I load from disk during startup -- if the user has changed the reference before my cache loads, the cache will be out of sync with the DB. I listen for notifiers to invalidate my cache, but that only works when the cache is in sync before those notifications come in. But a display of a "loading..." line can ameliorate that potential problem.

Another such issue I just stumbled upon -- I display the citation key in the reference list and in the reference details, and these get loaded (and are ready for user interaction) well before schemaUpdatePromise clears and I can start loading the keys from the DB. I can show empty cells until that is all done, but that means I have to change to a different collection and nack to have the citekeys show in the reference list.

[Dan Stillman]

You might not need to worry about those promises at all. Take a look at what's done for the hasAttachment indicators in itemTreeView::getImageSrc(). The tree methods are called synchronously, but the getImageSrc() call kicks of an asynchronous check for file presence, and when that's ready it invalidates the cell, so those get automatically loaded as the tree is rendered and scrolled.

[Emiliano Heyns]

In which case I'd have to monkey-patch getCellText to behave in a similar way for the citekey field, correct? Because I currently only monkey-patch getField and let getCellText do its regular thing. 

[Emiliano Heyns]

On Thursday, August 24, 2017 at 1:51:56 AM UTC+2, Dan Stillman wrote:

How come that the references display well before schemaUpdatePromise resolves BTW? Doesn't that mean that the library is loaded before it resolves? Wouldn't that mean the DB is ready before schemaUpdatePromise resolves?

[Emiliano Heyns]

On Monday, August 21, 2017 at 3:46:13 PM UTC+2, Dan Stillman wrote:

On 8/19/17 10:55 AM, Emiliano Heyns wrote:

On Friday, August 18, 2017 at 11:19:38 PM UTC+2, Dan Stillman wrote:

A couple different issues here:

- You shouldn't do anything other than UI modification before Zotero.uiReadyPromise — otherwise you'll be negatively affecting Zotero load time. And really it's best to wait for schemaUpdatePromise, which needs to finish before various dialogs/operations work properly. (initializationPromise only existed for Firefox mode and will likely be removed or made private at some point.)

That may put me in a bit of a bind; there is BBT initialisation work I would want done before the UI becomes active but which needs DB access, so waiting for uiReadyPromise would seem counter to that goal. schemaUpdatePromise reliably takes between 10 and 30 seconds to resolve on the systems I work with (CircleCI containers, my 2015 Macbook Air, and a 2017 Zbook 15 G3), and from what I can tell users can start to interact well before that, which would give me an uncertain starting state.

Well, again, various Zotero dialogs and operations involving translators or styles won't work until schemaUpdatePromise is resolved, so that's clearly the priority. And since schemaUpdatePromise can include schema updates, you don't even know that the DB is in an up-to-date state until that's done.

I'm not sure what you mean by an uncertain starting state. Everything you do should just be dependent on that having been resolved, the same way that various dialogs in Zotero wait for that. (You can, e.g., display a "Loading…" line if some action is performed before then.)

But also, for me, on a 2013 rMBP, schemaUpdatePromise is consistently resolved 5 seconds after uiReadyPromise (which is the only time that really matters for this, because displaying data takes precedence over absolutely everything). It's fairly easy to run up against that as a developer, testing a specific function, but there's not a lot a user is going to do in 5 seconds. More than likely they're opening Zotero to save an item, so the priority is having translators be ready to use.

I see in schema.js that updateSchema first does a schema upgrade if necessary but then at the end waits for updatebundledfiles and optionally waits for the updateFromRepository -- but the DB would be ready before that, right? Would it be possible to get a promise I can yield on just before that? Or is the time spent in updateBundledFiles not really significant?