JSX-ified zoteroPane?

[Emiliano Heyns]

Is zoteroPane.xul already jsx-ified? I do an overlay of itemPane.xul which injects columns into <treecols id="zotero-items-columns-header">, and that still works, but I cannot find the treecols in any of the Zotero xul sources. I also cannot find the string "zotero-items-columns-header" at all in the Zotero sources.

[adoma...@gmail.com]

Hi Emiliano,

Yes, we've recently merged the HTML tree which replaced the XUL tree into master. I'm working on a migration guide for plugin developers. It shouldn't be too complicated, but some modifications will be necessary. We'll have more details today or tomorrow and push a build on the dev zotero channel. We want to release this ASAP for the Beta channel to get as much testing as possible before release on the stable channel since it's quite a big overhaul. The HTML tree will not be released on the stable channel at least until October.

[adoma...@gmail.com]

See https://github.com/scitedotai/scite-zotero-plugin/pull/26

I believe the process for you will be very similar, given that the scite plugin seems to have borrowed a lot of ideas from your plugin. Sorry I didn't modify BBT, but there's no development instructions in the README.md and it's a much bigger and more intimidating plugin to work with. If you have any questions shoot me a message here and we'll try to respond ASAP. Dan will release a build on the dev channel later today.

Adomas

[Emiliano Heyns]

On Thursday, August 26, 2021 at 3:48:11 PM UTC+2 adoma...@gmail.com wrote:

See https://github.com/scitedotai/scite-zotero-plugin/pull/26

I already see a problem I might run into.

You can find the new tree interface at chrome/content/zotero/itemTree.jsx. It is imported for monkey-patching by calling require('zotero/itemTree').

I use a bundler to package in npm modules. I will see if I can convince it to not replace some "require" calls, but my code does a ton of require calls for npm modules.

I also use overlays to "start" BBT. Must I migrate to a restartless plugin? I've tried that before and it was looking to be a substantial project. 

I take it it won't be possible to have a single source be compatible with the current and the upcoming release? When I ported to Zotero 5 having to keep two very divergent source trees working was a significant hurdle.

 

I believe the process for you will be very similar, given that the scite plugin seems to have borrowed a lot of ideas from your plugin.

It looks like it was scaffolded by an older version of my yeoman generator.

 

Sorry I didn't modify BBT, but there's no development instructions in the README.md and it's a much bigger and more intimidating plugin to work with.

I don't blame you.

 

If you have any questions shoot me a message here and we'll try to respond ASAP. Dan will release a build on the dev channel later today.

What branch does the current zotero sit on? I need to look in the source sometimes to find out how things work.

[adoma...@gmail.com]

On Thursday, 26 August 2021 at 19:15:53 UTC+3 Emiliano Heyns wrote:

I use a bundler to package in npm modules. I will see if I can convince it to not replace some "require" calls, but my code does a ton of require calls for npm modules.

I ran into the same issue when converting the scite plugin as I am not too familiar with webpack config. I found a more clean solution, but if worst comes to worst you can do it with `eval('require("zotero/itemTree")')`.

 

I also use overlays to "start" BBT. Must I migrate to a restartless plugin? I've tried that before and it was looking to be a substantial project. 

 

Scite does too and it didn't create problems.

I take it it won't be possible to have a single source be compatible with the current and the upcoming release? When I ported to Zotero 5 having to keep two very divergent source trees working was a significant hurdle.

Yes, but it shouldn't require divergent sources here. You can put the old patching code under `if(typeof Zotero.ItemTreeView != "undefined")` and the new code in the else clause. 

You can keep the column spec in the overlay files for the XUL tree, and you'll specify the columns for the HTML tree in the monkey-patch.

If you have any questions shoot me a message here and we'll try to respond ASAP. Dan will release a build on the dev channel later today.

What branch does the current zotero sit on? I need to look in the source sometimes to find out how things work.

Dan might be able to answer this better. The Beta channel keeps up with the master branch mostly, but is not released for every commit, especially now as we have a significant

portion of our users using it for the new PDF reader and we want to make sure it's stable.

Doing this conversion and seeing the amount of monkey-patching that is required for extension developers has also lead to some discussions internally about the fact that for certain common plugin actions

such as editing the item tree, item pane, adding context menu items and perhaps some others we shouldn't expect plugin developers to monkey-patch methods as that is quite dangerous and may break our own

UI if done incorrectly, or as a result of a change to our own interfaces and also is too complicated for plugin developers. We'd like to have dedicated APIs for at least the most common extension needs,

under Zotero.Extension or similar. This is not going to happen in the coming months, but we'd like to say that the current way to extend the item tree is temporary and we'll have a more robust solution.

Also to update the future technology roadmap. The plans to move to Electron are currently on hold. We'd first like to update to at least Firefox 78 ESR and eventually a current Firefox build. That is not happening for the

Zotero 6 release (the PDF reader) but is the nearest change that will have an overall impact on the architecture of Zotero. Most importantly, Firefox 78 no longer supports XUL overlays, but there will probably be more

implications for extension developers. We hope to address some of them with a new Extension API too. We'll have more info on this as we are closer to the transition.

[Emiliano Heyns]

On Thursday, August 26, 2021 at 11:22:36 PM UTC+2 adoma...@gmail.com wrote:

On Thursday, 26 August 2021 at 19:15:53 UTC+3 Emiliano Heyns wrote:

I use a bundler to package in npm modules. I will see if I can convince it to not replace some "require" calls, but my code does a ton of require calls for npm modules.

I ran into the same issue when converting the scite plugin as I am not too familiar with webpack config. I found a more clean solution, but if worst comes to worst you can do it with `eval('require("zotero/itemTree")')`.

I'm using esbuild these days, and I have found you can declare modules as external, and it will just leave those require's as-is. 

 

I also use overlays to "start" BBT. Must I migrate to a restartless plugin? I've tried that before and it was looking to be a substantial project. 

 

Scite does too and it didn't create problems.

But with the change to ESR 78, I will need to move to restartless, since you mention that that no longer has XUL, correct?

I take it it won't be possible to have a single source be compatible with the current and the upcoming release? When I ported to Zotero 5 having to keep two very divergent source trees working was a significant hurdle.

Yes, but it shouldn't require divergent sources here. You can put the old patching code under `if(typeof Zotero.ItemTreeView != "undefined")` and the new code in the else clause. 

You can keep the column spec in the overlay files for the XUL tree, and you'll specify the columns for the HTML tree in the monkey-patch.

OK. This change is probably less disruptive than asyncification.

Doing this conversion and seeing the amount of monkey-patching that is required for extension developers has also lead to some discussions internally about the fact that for certain common plugin actions

such as editing the item tree, item pane, adding context menu items and perhaps some others we shouldn't expect plugin developers to monkey-patch methods as that is quite dangerous and may break our own

UI if done incorrectly, or as a result of a change to our own interfaces and also is too complicated for plugin developers. We'd like to have dedicated APIs for at least the most common extension needs,

under Zotero.Extension or similar. This is not going to happen in the coming months, but we'd like to say that the current way to extend the item tree is temporary and we'll have a more robust solution.

That sounds good.

Also to update the future technology roadmap. The plans to move to Electron are currently on hold. We'd first like to update to at least Firefox 78 ESR and eventually a current Firefox build. That is not happening for the

Zotero 6 release (the PDF reader) but is the nearest change that will have an overall impact on the architecture of Zotero. Most importantly, Firefox 78 no longer supports XUL overlays, but there will probably be more

implications for extension developers. We hope to address some of them with a new Extension API too. We'll have more info on this as we are closer to the transition.

My main worry is that my time for large projects has really shrunk over the past few years; something like the port to Zotero 5 would be harder to allocate time for now, and that already was a massive undertaking for me at the time. I think my main hurdle then was that I couldn't chunk the work into parts I could reasonably oversee, finish, and test.

[adoma...@gmail.com]

On Friday, 27 August 2021 at 11:09:44 UTC+3 Emiliano Heyns wrote:

But with the change to ESR 78, I will need to move to restartless, since you mention that that no longer has XUL, correct?

AFAIK Firefox 78 had removed a lot of old extension loading code (even Firefox 68 had removed certain things like the addon page that Dan had to add back manually) and I'm not even sure what the mechanism for loading extensions will be then. Either way you will indeed have to convert to something similar to a restartless extension and certainly won't be able to use overlays to inject your UI. It might be worth investigating converting to a restartless extension now as it's likely that some form of that code will be usable going forward.

 

My main worry is that my time for large projects has really shrunk over the past few years; something like the port to Zotero 5 would be harder to allocate time for now, and that already was a massive undertaking for me at the time. I think my main hurdle then was that I couldn't chunk the work into parts I could reasonably oversee, finish, and test.

We're always aware that any architectural changes we make to Zotero also impacts our plugin developers and we try to minimize that impact. But XUL deprecation is absolutely beyond our control and we will inevitably have to move away from all XUL technologies. Hopefully as we transition to HTML UIs they will be more in our control and remain stable, even if we have to switch underlying platforms from mozilla to electron or other.

That said, Zotero 5 asyncification created rather holistic changes to our own codebase across the board, which meant a difficult transition for plugin developers too. The move from XUL to HTML will hopefully be less painful as we don't expect any big data-layer code changes and the UI is largely decoupled from business logic. It helps that UI components lend themselves to self-isolation and easier natural work chunking. We also hope to have at least some basic extension APIs by then, which will help with the transition further. On the other hand it is likely that you will have to have multiple branches for some transition period for the old and new architecture.

I'm not sure if what I wrote above is entirely comforting in terms of ensuring that porting will be easy, but I hope it at least removes some uncertainty.

[Dan Stillman]

A beta with the new HTML tree is available now. If you switched to a
build on the 'dev' channel, you should switch to the beta, since the
'dev' channel isn't maintained.

- Dan

[Emiliano Heyns]

I realize a move away from xul was inevitable. It's good too know there is a viable path to single-branch development; I simply lack the time to keep two potentially divergent versions running. I'll hold off on restartless until I have the new UI stuff going for the same reason - I can't really handle two parallel projects right now, and I don't want to marry the two goals, because that increases the risk that they both don't get met.

I must admit some trepidation about the work to come. My previous stint at react (unrelated to Zotero) was an incredibly frustrating experience, although that will be partly addressed by Zotero putting the infrastructure in place

The other is just the nature of porting - doing work just to remain in position. Not a complaint. I just feel somewhat overwhelmed.

[Dan Stillman]

On 8/27/21 8:50 PM, Emiliano Heyns wrote:
> I realize a move away from xul was inevitable. It's good too know
> there is a viable path to single-branch development; I simply lack the
> time to keep two potentially divergent versions running. I'll hold off
> on restartless until I have the new UI stuff going for the same reason
> - I can't really handle two parallel projects right now, and I don't
> want to marry the two goals, because that increases the risk that they
> both don't get met.
>
> I must admit some trepidation about the work to come. My previous
> stint at react (unrelated to Zotero) was an incredibly frustrating
> experience, although that will be partly addressed by Zotero putting
> the infrastructure in place
> The other is just the nature of porting - doing work just to remain in
> position. Not a complaint. I just feel somewhat overwhelmed.

Just to echo Adomas, while upcoming changes — most notably XUL overlay
removal — will require some work, asyncification really was a special
case, due to its viral nature within a codebase. So don't use that as
any sort of benchmark for what to expect here.

Unlike the async changes, which avoided UI freezing but otherwise
weren't particularly rewarding, these changes will enable all sorts of
future functionality. The items list has already gained a fun new
feature and some bug fixes with this change, and there are many things
that we and plugin developers will now be able to do that weren't
possible before (e.g., showing search results in the middle pane). So
hopefully it won't just be work to remain in position.

(Also, minor terminology note: the change here was to an HTML tree. It's
partially React-based, with some custom performance optimizations, but
"JSX" isn't really relevant.)

[Emiliano Heyns]

On Saturday, August 28, 2021 at 7:12:20 AM UTC+2 Dan Stillman wrote:

Just to echo Adomas, while upcoming changes — most notably XUL overlay
removal — will require some work, asyncification really was a special
case, due to its viral nature within a codebase. So don't use that as
any sort of benchmark for what to expect here.

Right, I try to keep that in mind.

 

Unlike the async changes, which avoided UI freezing but otherwise
weren't particularly rewarding, these changes will enable all sorts of
future functionality. The items list has already gained a fun new
feature and some bug fixes with this change, and there are many things
that we and plugin developers will now be able to do that weren't
possible before (e.g., showing search results in the middle pane). So
hopefully it won't just be work to remain in position.

I'll try to keep that in mind too.

(Also, minor terminology note: the change here was to an HTML tree. It's
partially React-based, with some custom performance optimizations, but
"JSX" isn't really relevant.)

Right, correct, I meant React, and because Zotero will have made many of the choices for me, most of this will not apply. But I still haven't really internalized the workings of React, and I feel some (largely self-imposed) pressure to get this done quickly, as the first reports are already rolling in, and I don't have much time the coming weeks.

[Emiliano Heyns]

The asyncification also had an immense benefit - background exports would have been literally inconceivable to me had I not gotten so familiar with async, and I'd never have gotten there if I hadn't been forced to.

On Saturday, August 28, 2021 at 7:12:20 AM UTC+2 Dan Stillman wrote:

[Emiliano Heyns]

Could there be some central place where knowledge/info about this transition is gathered? That would help me a lot.

On Saturday, August 28, 2021 at 7:12:20 AM UTC+2 Dan Stillman wrote:

[Dan Stillman]

On 8/29/21 4:00 AM, Emiliano Heyns wrote:
> Could there be some central place where knowledge/info about this
> transition is gathered? That would help me a lot.

Just ask your questions here.

[Emiliano Heyns]

Is the item list now XUL-free in 6.0+? Since my ZoteroPane overlay still loads, something must still be happening in zoteroPane.xul.

[Dan Stillman]

On 5/11/22 11:37 AM, Emiliano Heyns wrote:
> Is the item list now XUL-free in 6.0+? Since my ZoteroPane overlay
> still loads, something must still be happening in zoteroPane.xul.

- XUL isn't going away completely anytime soon — it's even still present
in some forms in Firefox 101.

- The items list in Zotero 6 is now mostly HTML. (Context menus and
tooltips are still XUL.)

- XUL overlays are indeed going away, and are removed in Firefox 101.
But they're still present in Firefox 60, so an overlay will still work
for anything it matches. I would strongly recommend starting to adjust
your code not to use them, though.

[Emiliano Heyns]

I suspected as much. How do I get notified of the zoteropane/item pane being loaded/unloaded?

[Emiliano Heyns]

I've just looked at scite and it looks like it still uses XUL for overlay & bootup? Is there an example for a plugin that modifies the zoteropane and itempane and that doesn't use XUL itself?

On Thursday, August 26, 2021 at 11:22:36 PM UTC+2 adoma...@gmail.com wrote:

[Dan Stillman]

On 5/15/22 11:07 AM, Emiliano Heyns wrote:
> I've just looked at scite and it looks like it still uses XUL for
> overlay & bootup? Is there an example for a plugin that modifies the
> zoteropane and itempane and that doesn't use XUL itself?

For the items list, it looks like Scite just has backwards compatibility
with the XUL tree, which could be removed at this point, so the overlay
part isn't particularly relevant for that. It does appear to use an
overlay to add a tab to the item pane. But it should be able to just
load code in a bootstrap script on window load and do all the same
stuff, after waiting for `Zotero.Schema.schemaUpdatePromise`.

Zutilo is an example of a "bootstrapped" extension that uses
bootstrap.js. That's of course still a kind of legacy Firefox extension,
and we may not be able to support that exact mechanism in Firefox 102
(where we're still figuring out the plugin plan), but the basic idea
would be the same — you'd run a script on window load, wait for a
promise or event, and do what you need to do.

[Emiliano Heyns]

How do I get notified the itemlist is loaded/unloaded, and of when a new item is shown in the itempane?

[adoma...@gmail.com]

There's not really a way to "get notified" when collections change or when different items are shown in the item pane. We have a notification system in Zotero that issues events for item data change and then the UI reacts on that. This is done to at least partially decouple the data flow from the UI and make the data flow more one-directional. I'm wondering what you want to do and whether there's a better way to do it than to get notified about UI changes. If there isn't, you'd have to monkey patch ZoteroItemPane.viewItem and ZoteroPane.itemsView.changeCollectionTreeRow.

[Emiliano Heyns]

For the item view I need to display the corresponding citekey. 

--
You received this message because you are subscribed to a topic in the Google Groups "zotero-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/zotero-dev/yi4olucA_vY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to zotero-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/zotero-dev/7e54c6ea-ba9f-4c4a-837d-a283bb7130f4n%40googlegroups.com.

[adoma...@gmail.com]

How are you currently doing it?

[Emiliano Heyns]

During the overlay-load of itempane.xul I monkey-patch ZoteroItemPane.viewItem

[Emiliano Heyns]

As far as I can tell, ZoteroItemPane is a variable that's local to the item pane, so I don't know how to get to it to monkey-patch it when not using the overlay.

[Emiliano Heyns]

I have my hands free for 4 days and I really want to get started on restartless, but I need to know

• How to run code when the itempane shows a new item
  
• How to run code when the zoteropane loads/unloads, other than from the bootstrap methods; on macos, it is possible to close the last window while zotero keeps running, so when a new zoteropane is unloaded, I need to do listener cleanup, and when it is shown again, I need to re-do UI mods and listeners.
  

[Emiliano Heyns]

Would really love to get started on this, but I need a solution to get notified of these two events. For both, I also need a way to access `document`, although for the latter I think I can get it from getActiveZoteroPane. Does "active" here mean there could also be other, non-active szotero panes?