How does the Zotero architecture work?

[Luke Tully]

I've been slowly working me way through trying to understand how the standalone client comes together, but so far I've been unable to find documentation on main topics like debugging which is what I might normally use to dig deeper from and entry point.

Are there any posts or articles about this that I'm missing? Do I just need to review XUL architecture? Any suggestions on where to get started would certainly be welcomed.

[Dan Stillman]

On 1/22/18 6:52 PM, Luke Tully wrote:

I've been slowly working me way through trying to understand how the standalone client comes together, but so far I've been unable to find documentation on main topics like debugging which is what I might normally use to dig deeper from and entry point.

Are there any posts or articles about this that I'm missing? Do I just need to review XUL architecture? Any suggestions on where to get started would certainly be welcomed.

You can use the debugger in the Firefox dev tools.

From https://www.zotero.org/support/dev/client_coding/building_the_standalone_client:

A -debugger flag will run the Mozilla DevTools server, which you can connect to from the Firefox DevTools by using the “Connect…” option and pointing to port 6100. You'll first need to enable browser chrome/add-on and remote debugging in the settings (gear icon) pane of the Firefox Web Console and pass -t to the dir_build script.

(You'll want to use the build_and_run script [1], which is an easier interface to dir_build, if you're building Zotero from source.)

But Bluebird promises tend to make using a debugger fairly difficult (and I don't believe you can blackbox scripts in the current Fx dev tools, or at least in remote mode), so just adding Zotero.debug() lines may be easier. (I rarely bother with the debugger personally and use Zotero.debug() almost exclusively, since any code change requires restarting Zotero, and that requires reconnecting from the dev tools, but the debugger might be useful for exploration.) For trying out UI modifications, the Inspector is extremely useful.

In terms of architecture, there's not really much documentation, though https://www.zotero.org/support/dev/client_coding/javascript_api has some example code. But basically, most UI entry points are in zoteroPane.js, and the core, non-window code is in 'xpcom'. The data layer (item saving, etc.) is in 'xpcom/data'. The left/middle panes are collectionTreeView.js and itemTreeView.js. The translation architecture is…complicated. Translators run in their own sandbox and can be developed with Scaffold [2] without building Zotero from source.

Happy to give additional guidance if you say more about the parts you're interested in.

- Dan


[1] https://www.zotero.org/support/dev/client_coding/building_the_standalone_client#helper_script
[2] https://www.zotero.org/support/dev/translators/scaffold

[Duncan Paterson]

I m in the same boat as the OP, but am trying to debug something in jurism. Just understanding if the errors I m encountering stem from Jurism or are also present in the main zotero code is turning out a lot more challenging then I'd like. 

Looking at travisCi logs from zotero/zotero i noticed the same warnings on my branch that appear on zotero/zotero master related to addons.xpi, e.g. https://travis-ci.org/zotero/zotero/jobs/331993473#L5564

the crucial difference my code exits with status 1 at this point, while zotero/zotero keeps running. Could you explain whats going on here with addons.xpi?

I'm somewhat at a loss to understand what the two firefox versions (54 and 52, why not 57 and is zotero actually supporting FF-ESR?) matrix is actually simulating. I regularly see failures in one matrix but not the other. I thought Zotero was moving away from Firefox, which browser interactions are tested with selenium?

Lastly, what's the deal with "Possibly unhandled rejection:" should i even bother trying to hunt these down? 

[Emiliano Heyns]

None of my answers can be seen as authoritative, but here's my take:

On Tuesday, January 23, 2018 at 5:33:13 PM UTC+1, Duncan Paterson wrote:

I m in the same boat as the OP, but am trying to debug something in jurism. Just understanding if the errors I m encountering stem from Jurism or are also present in the main zotero code is turning out a lot more challenging then I'd like. 

Looking at travisCi logs from zotero/zotero i noticed the same warnings on my branch that appear on zotero/zotero master related to addons.xpi, e.g. https://travis-ci.org/zotero/zotero/jobs/331993473#L5564

the crucial difference my code exits with status 1 at this point, while zotero/zotero keeps running. Could you explain whats going on here with addons.xpi?

I'm somewhat at a loss to understand what the two firefox versions (54 and 52, why not 57 and is zotero actually supporting FF-ESR?

This may be related to the fact that those two versions still supported unsigned XPCOM extensions. 57 does not. Not sure why both 52 and 54 are tested, but it has been the case before that for different OSes different FF versions were used for the standalone client (I don't know why though).

 

) matrix is actually simulating. I regularly see failures in one matrix but not the other. I thought Zotero was moving away from Firefox, which browser interactions are tested with selenium?

Zotero will be moving away from Firefox, but currently it's still wholly FF-based. Platform-wise, not much has changed between Zotero 4 and 5 (I think the Linux version git an upgrade from an ancient FF version to 52 or 54). The architectural change inside Zotero itself (the aync push) is separate from this. When the move to Electron happens we will have to see how much will change again, but anything that uses XUL or XBL will have their work cut out for them I'd venture to guess (and I would be one of those).

 

Lastly, what's the deal with "Possibly unhandled rejection:" should i even bother trying to hunt these down? 

It's the promise-equivalent of an exception being thrown but not handled; unfortunately, unhandled rejections carry very little information on what happened, or where. Some of these are fairly benign, but because it's so hard to tell whether they actually are benign, I try to hunt these down and convert these into logged exceptions where I can.

[Duncan Paterson]

Hi Emiliano, 

running tests locally via ./test/runtests.sh -x /local-path/firefox I see a possibly unhandled exceptions related to an `unset this._creator` which happens to be the last error message before my jurism decided to go on strike, and that started me on this whole journey. 

https://travis-ci.org/duncdrum/zotero/jobs/332801866#L5768

So maybe this one is not so ignorable. 

Electron would be easier for me to deal with, but if I understand you correctly Standalone is just a firefox system pretending to be something else. So I should keep testing with 54 (although ERS would suit me well at this point) . I d still like to know if the add-on related warnings are something that others are seeing as well, that can safely be ignored, or if there is something i can try to address them. It looks to me that they are related to the fact that I run tests on FF54 but I might be totally off here. 

[Emiliano Heyns]

On Wednesday, January 24, 2018 at 6:44:40 PM UTC+1, Duncan Paterson wrote:

Hi Emiliano, 

running tests locally via ./test/runtests.sh -x /local-path/firefox I see a possibly unhandled exceptions related to an `unset this._creator` which happens to be the last error message before my jurism decided to go on strike, and that started me on this whole journey. 

https://travis-ci.org/duncdrum/zotero/jobs/332801866#L5768

So maybe this one is not so ignorable. 

For sure that one seems important to track down, but there's quite a bit of other errors there too. I don't know what damage those do along the way -- I've seen in my own tests (unrelated to these -- different tests, different tech) that one problem would down the line cause others. Although in my case they would usually fail more immediately. I have never ran the Zotero test suite, so I really don't know what it's supposed to do, but "ignorable errors" goes against the idea of an automated test suite for me.

 

Electron would be easier for me to deal with, but if I understand you correctly Standalone is just a firefox system pretending to be something else.

I think that is a correct description. It's not the only app to do that -- I think Thunderbird would fit that description. Not entirely, but close enough. But yeah, Zotero standalone is basically a stripped-down Firefox; if you go into Preferences -- Advanced -- About:config, you'll just see the same as you see in FF, the extensions screen is exactly the same, etc. And it uses XUL, XPCOM, XBL, etc.

I have no idea whether Electron will be easier to deal with; it will be easier in some senses (for one, I could just straight-up use npm modules instead of webpacking them), but the only thing I know about the Electron version is that it'd be react based, and as bad as XUL is, I think I prefer it over react. 

 

So I should keep testing with 54 (although ERS would suit me well at this point) .

That I don't know. I strictly test with formally released versions myself, so I download the full release for my tests, but IIRC, both the Zotero team and Juris-M build a parallel version that is loadable into FF 54 and test against that. I could be wrong, but that's what I remember.

 

I d still like to know if the add-on related warnings are something that others are seeing as well, that can safely be ignored, or if there is something i can try to address them. It looks to me that they are related to the fact that I run tests on FF54 but I might be totally off here. 

I couldn't say, sorry. 

[Duncan Paterson]

but "ignorable errors" goes against the idea of an automated test suite for me

amen to that 

[Dan Stillman]

On 1/25/18 3:39 AM, Duncan Paterson wrote:

but "ignorable errors" goes against the idea of an automated test suite for me

amen to that

The actual tests in the test suite are obviously meant to pass. If those are passing, you're in good shape.

The rapid UI changes during the tests result in some spurious errors in the console output, and those can be ignored.

Certain tests can also fail intermittently due to race conditions (more so on local computers with unpredictable load than on Travis), which is unfortunate but very hard to combat in the current architecture and ultimately not that big a deal. (When zotero/zotero shows "build failing", that's almost always the reason. We restart the run and it almost always goes through.)

[Duncan Paterson]

The actual tests in the test suite are obviously meant to pass. If those are passing, you're in good shape.

but I m not. The unhandled exception reported in a passing travis builds, has the exact error message that killed sync in my jurism client. Now this could be accidental, or it could indicate that there needs to be another test. 

The rapid UI changes during the tests result in some spurious errors in the console output, and those can be ignored.

Would the addons.xpi related errors fall into this category, what about the series of GLib-GObject-CRITICAL **? (These are also present in current zotero/zotero master)

 

Certain tests can also fail intermittently due to race conditions (more so on local computers with unpredictable load than on Travis), which is unfortunate but very hard to combat in the current architecture and ultimately not that big a deal. (When zotero/zotero shows "build failing", that's almost always the reason. We restart the run and it almost always goes through.)

Yeah I noticed that restart tests works wonders :-) I m still curious why there are 1100  local test, but only 1093 on travis?

[Frank Bennett]

For those who drive by this thread, I've built some instructions on the Juris-M build process, available in the top-level README files of the Juris-M forks of zotero, zotero-build, and zotero-standalone-build (the last is a work in progress, but will be up soon).

I confess that preparing the instructions has been a journey of discovery. When I built the initial release of Juris-M 5.0, I ended up replicating most of the full Zotero release steps (with various files reflected in AWS S3 buckets, etc), and never fully understood how the basic build process worked. Documenting local build steps has been like taking off a pair of tight shoes.

Apologies to Duncan that I didn't get to this sooner.