Zotero Programming Consulting Projects & Training

[Russ McBride]

Hi,

I've got a handful of Zotero customizations that I need for my persona workflow that all require better integration between Zotero and the finder (on a Mac) and various Mac applications (other PDF apps, LiquidText, Scrivener). I'd also like to learn to "develop" with Zotero myself so I'm looking for someone to walk me through what they've created in each case.

$100 per each hour of work which would include an additional 15min Zoom'ing with me tutoring me on what you did.

We could start with this first project: I would like to be able to select a number of entries in Zotero, be able to right click, and have an option to "create aliases of PDFs", and drag that collection of PDF aliases into the Mac Finder. Other means of accomplishing a mass drag of PDF aliases would be o.k. too (I store my PDF attachments in DropBox for Zotero in a DropBox folder).

There are others but this is a good entry project. Let me know if you're interested.

Thanks!

Russ

[Emiliano Heyns]

In the entry level project you describe the interaction with MacOS is going to be harder than the interaction with Zotero. Selecting entries and reacting to a custom menu item is simple, but I wouldn't knnow how to bring up a finder that has files in multiple folders selected and has pressed cmd-option on your behalf.

[Abe Jellinek]

It would probably be easier to have it prompt for a destination folder and then create all the aliases in that folder, no Finder interaction or dragging involved.

On Jul 30, 2022, at 12:08 PM, Emiliano Heyns <emilian...@iris-advies.com> wrote:

In the entry level project you describe the interaction with MacOS is going to be harder than the interaction with Zotero. Selecting entries and reacting to a custom menu item is simple, but I wouldn't knnow how to bring up a finder that has files in multiple folders selected and has pressed cmd-option on your behalf.

--
You received this message because you are subscribed to the Google Groups "zotero-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zotero-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/zotero-dev/f02ec945-6f74-48cf-bcbe-88df97f32f2bn%40googlegroups.com.

[Emiliano Heyns]

That would indeed be much simpler. 

You received this message because you are subscribed to a topic in the Google Groups "zotero-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/zotero-dev/rpY4zNBzdIU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to zotero-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/zotero-dev/5F7D9BE0-2656-45F0-A2C8-6BFA2FC36E1A%40berkeley.edu.

[Russ McBride]

Yeah, application and Mac OS control is difficult and unpredictable in that not all apps have an accessible (AppleScript/Javascript) API (e.g., Scrivener, LiquidText) but though it's more limited, everything can be controlled through simulated mouse clicks & drags using AppleScript, Javascript, and Python (and probably other languages) using "accessibility". I'd prefer to keep as much as possible in Javascript since that's the easiest way to access Zotero locally (I only know AppleScript, Python, Ruby, and a little Swift). In Python I can use PyAutoGUI for this and it works well. There are often also limited commands that can be issued through some language-agnostic, application-specific URIs.

First a comment and then I should explain the broader project goal.

Comment: 

"Collections' in Zotero that contain "sub-collections" do not behave intuitively like folders. If I have collection B inside collection A and do a search for an entry that is in collection B but not A while collection A is selected, then I get no results. Collections do not act like containers so sub-collection entries do not show up in searches (unless I'm missing something obvious). I consider this to be counterintuitive enough to be considered a bug (feel free to correct me here).

The Broader Goal: 

I've got a Zotero library with all my PDFs stored on DropBox. I know that Zotero storage is the greatest thing since sliced bread and makes collaboration using "Group"s possible but for many of us the Zotero folder structure is a deal-breaker because it imposes a non-human-readable intermediary folder layer between us and the PDFs that we need to access using a variety of different apps (in my case-- PDF Expert, PDF Reader Pro, LiquidText, and sometimes Preview or Skim and others). PDFs are a nightmare of non-standardized properties, libraries, and app features, but that's the default these days so we have to deal with it. Each PDF app has something it can do that others can't so you have to pick each for the job that you need.

Every research project needs a different selection of PDFs organized into various subfolders. And these need to be accessible by my collaborators. I can't re-organize the folder structure of the original PDFs since that would screw up Zotero's links to those PDFs each time if I changed their location constantly. The solution, then, is to use aliases. I can put aliases anywhere and organize them however I want. And my collaborators can easily gain access to the original PDF through the organized folders of aliases. 

I might have a Zotero collection, "Fish Paper", with general entries about fish, then a sub-collection called, "Fish Mammals" that contains entries about whales and dolphins, and another sub-collection named, "Aquatic Serpents" with entries about eels and sea snakes. (I'm making this up--I know nothing about marine biology.) The goal is to mirror the Zotero collection and sub-collection structure with the same folder and sub-folder OS file structure filled with aliases of the PDFs from those Zotero collections. 

So, I would click on the top-level collection in Zotero--"Fish Paper", a dialog from Zotero would pop-up asking me where I want the mirrored aliases to go, and then a new top-level folder with the same name "Fish Paper" would be created with the subfolders and all the PDF aliases in the right places—the new "Fish Paper" folder would contain all the alias from the PDFs in the Zotero entries for the "Fish Paper" collection. The "Fish Mammals" would contain all the PDF aliases from its entries", an the same for "Aquatic Serpents".

Pseudo-Code:

[Zotero Javascript]

-- repeat for each collection starting at the top: get the name of the collection and a list of all the PDF paths in that collection

(error out if more than one collection has been selected by the user) 

[Mac Javascript/AppleScript]

-- repeat for each saved Zotero collection and PDF paths list: create the file folder, create an alias of each PDF using its path, and place the alias in the folder

(if the top-level folder already exists, delete it without asking and rebuild the entire folder tree)

Phase 2--

Watch that top-level Zotero collection and re-run the code whenever there is any change to any part of that collection or anything beneath it. This keeps the Zotero collection (and sub-collections) all in sync with the folder aliases.

My apologies for the lengthy explanation but when I'm coding I find it helpful to have lots of detail. 

==============================================================================

I have no idea how to do the Zotero part of the pseudo code. I also have no idea how to set up the combined Javascript/Applescript code so that it can call Zotero. If the two can't be combined, then then the first Javascript script can juts past control to a second AppleScript-only script.

If the Zotero part is easy for someone buy they're not comfortable with the Mac OS side I can do that (or show you, it's easy). You can look for some basics here:  https://developer.apple.com/library/archive/documentation/LanguagesUtilities/Conceptual/MacAutomationScriptingGuide/index.html#//apple_ref/doc/uid/TP40016239-CH56-SW1

For example, to bring up a dialog selecting a OS folder (in Javascript):

1. var app = Application.currentApplication()
2. app.includeStandardAdditions = true
4. var outputFolder = app.chooseFolder({
5. withPrompt: "Please select an output folder:"
6. })
7. outputFolder
8. // Result: Path("/Users/yourUserName/Desktop")

Getting a path (in Javascript):

1. var path = Path("/Users/yourUserName/Desktop/My File.txt")
2. var string = path.toString()
3. string
4. // Result: "/Users/yourUserName/Desktop/My File.txt"

To make some aliases (in AppleScript):  https://stackoverflow.com/questions/22799334/creating-a-finder-alias-using-applescript

After all that, if someone is still interested :-) let me know. I'm flexible on collaborating, paying for just the Zotero part, paying for the whole thing, or just learning together!

Cheers!

Russ 

[Dan Stillman]

On 7/31/22 2:35 AM, Russ McBride wrote:
> I know that Zotero storage is the greatest thing since sliced bread
> and makes collaboration using "Group"s possible but for many of us the
> Zotero folder structure is a deal-breaker because it imposes a
> non-human-readable intermediary folder layer between us and the PDFs
> that we need to access using a variety of different apps (in my case--
> PDF Expert, PDF Reader Pro, LiquidText, and sometimes Preview or Skim
> and others).

Just to be clear, the idea here is that Zotero already provides much
more advanced and flexible organization than the OS, with collections,
tags, full-text search, etc. So you find things in Zotero and, if you
want to open them externally, you right-click → Show File and do
whatever you want with the file. For some apps/websites (e.g., Gmail)
you can even just drag the attachment straight from Zotero.

Or if you don't need the collections, you can create a virtual folder in
your OS with a flat list of all PDFs within 'storage', searchable by
filename (creator, title, year) or content.

If that doesn't work for you for whatever reason, you can of course go
through the trouble of trying to do this, but I just wanted to push back
on the idea that this is somehow necessary to use files in Zotero in
other apps. (And if the problem is that your collaborators don't use
Zotero, clearly the easiest option is to convince them to do so!)

[Emiliano Heyns]

On Sunday, July 31, 2022 at 8:35:33 AM UTC+2 rus...@gmail.com wrote:

Yeah, application and Mac OS control is difficult and unpredictable in that not all apps have an accessible (AppleScript/Javascript) API (e.g., Scrivener, LiquidText) but though it's more limited, everything can be controlled through simulated mouse clicks & drags using AppleScript, Javascript,

This is not generally true. Not all javascript environments are equal -- what javascript can do in its own world is (largely), but how it reaches outside that environment is constrained, and in the case of the XPCOM environment that Zotero offers, simulating mouseclicks is not on the table to my knowledge. If you're taking what can be done in javascript from what can be done in node, these two environments are very much not the same,

 

and Python (and probably other languages) using "accessibility".

Which is because these environments have ways to use native-code libraries. That is not possible in Zotero without some significant effort, and knowledge of XPCOM, the documentation for which is incomplete, unmaintained, and deprecated as far as Mozilla is concerned.

 

I'd prefer to keep as much as possible in Javascript since that's the easiest way to access Zotero locally (I only know AppleScript, Python, Ruby, and a little Swift). In Python I can use PyAutoGUI for this and it works well. There are often also limited commands that can be issued through some language-agnostic, application-specific URIs.

The most realistic way of reaching outside Zotero is by running shell executables, redirecting their output (not always possible), and parsing that output. This is a pretty fragile connection to the outside workd.

 

First a comment and then I should explain the broader project goal.

Comment: 

"Collections' in Zotero that contain "sub-collections" do not behave intuitively like folders. If I have collection B inside collection A and do a search for an entry that is in collection B but not A while collection A is selected, then I get no results. Collections do not act like containers so sub-collection entries do not show up in searches (unless I'm missing something obvious). I consider this to be counterintuitive enough to be considered a bug (feel free to correct me here).

View -> Show items from subcollections

 

The Broader Goal:

So, I would click on the top-level collection in Zotero--"Fish Paper", a dialog from Zotero would pop-up asking me where I want the mirrored aliases to go, and then a new top-level folder with the same name "Fish Paper" would be created with the subfolders and all the PDF aliases in the right places—the new "Fish Paper" folder would contain all the alias from the PDFs in the Zotero entries for the "Fish Paper" collection. The "Fish Mammals" would contain all the PDF aliases from its entries", an the same for "Aquatic Serpents".

This is possible without the drag-as alias bits. The plugin would just pop up a folder picker and store aliases of the selected PDFs there.

 

Pseudo-Code:

[Zotero Javascript]

-- repeat for each collection starting at the top: get the name of the collection and a list of all the PDF paths in that collection

(error out if more than one collection has been selected by the user) 

Not hard to do.

 

[Mac Javascript/AppleScript]

-- repeat for each saved Zotero collection and PDF paths list: create the file folder, create an alias of each PDF using its path, and place the alias in the folder

(if the top-level folder already exists, delete it without asking and rebuild the entire folder tree)

Doable.

 

Phase 2--

Watch that top-level Zotero collection and re-run the code whenever there is any change to any part of that collection or anything beneath it. This keeps the Zotero collection (and sub-collections) all in sync with the folder aliases.

Doable.

 

I have no idea how to do the Zotero part of the pseudo code. I also have no idea how to set up the combined Javascript/Applescript code so that it can call Zotero. If the two can't be combined, then then the first Javascript script can juts past control to a second AppleScript-only script.

It should be possible to just create the alias file from javascript: https://en.wikipedia.org/wiki/Alias_(Mac_OS)#File_structure

For example, to bring up a dialog selecting a OS folder (in Javascript):

In MacOS JXA javascript.

After all that, if someone is still interested :-) let me know. I'm flexible on collaborating, paying for just the Zotero part, paying for the whole thing, or just learning together!

The coming two weeks I have limited availability, I don't know how much of a hurry you're in. I'm willing to explore this with you, but have you done any programming before? What you're proposing isn't a big project but I'd venture to guess that effort is going to lie between a day to a week of effort to get this done and wrapped while explaining to you what's going on, depending on how much explanation you need and how many changes you want while we're doing this (and that does happen most of the time in my experience).

[Dan Stillman]

On 7/31/22 3:22 AM, Emiliano Heyns wrote:
> It should be possible to just create the alias file from javascript:
> https://en.wikipedia.org/wiki/Alias_(Mac_OS)#File_structure

No, that's about the aliases on classic Mac OS. Modern macOS just uses
symlinks, which can't be created from Zotero without calling out to `ln
-s` (or perhaps with great effort using js-ctypes).

[Russ McBride]

Hey Dan,

On Sunday, July 31, 2022 at 1:57:14 PM UTC+7 Dan Stillman wrote:

Just to be clear, the idea here is that Zotero already provides much
more advanced and flexible organization than the OS, with collections,
tags, full-text search, etc. So you find things in Zotero and, if you
want to open them externally, you right-click → Show File and do
whatever you want with the file. For some apps/websites (e.g., Gmail)
you can even just drag the attachment straight from Zotero.

Or if you don't need the collections, you can create a virtual folder in
your OS with a flat list of all PDFs within 'storage', searchable by
filename (creator, title, year) or content.

If that doesn't work for you for whatever reason, you can of course go
through the trouble of trying to do this, but I just wanted to push back
on the idea that this is somehow necessary to use files in Zotero in
other apps. (And if the problem is that your collaborators don't use
Zotero, clearly the easiest option is to convince them to do so!)

I'm happy to recommend Zotero to my colleagues after years with BibDesk and EndNote! (Fun fact-- I was a grad student in the same department with John MacFarlane when he created Pandoc... though I didn't help with that at all).

I would also be happy to use Zotero storage rather than swimming upstream against it using external storage. Doing so would solve the collaboration issue because I could just use online "Groups" (although I wish there was a way to turn any collection into a group so the group mirrored the collection... maybe there is--?). The big obstacle for me, and I gather, lots of others, is accessing the PDFs from other apps. Yes, I can do a "show file", and then "open with...". But I can't do that on my iPad so I have to start with LiquidText and work in the other direction and I don't see a way to do that easily with Zotero PDF storage system. Please correct me if I'm overlooking something!

Among other things, I need to:

-- open an PDF or bunch of PDFs easily and quickly from any PDF app and save annotations back to the PDF

-- provide access to those PDFs in an organized file hierarchy to my collaborators

-- create a selection of a PDF as an image in the annotations file for that PDF (I can only do this in LiquidText, not any other PDF too, as far as I know)

-- export all annotations (which LT is the best at because I can also export the workspace and images, and Zotero does a pretty good job of)

-- have deep links into sections of the PDF with some kind of URI or "Hook" (which Zotero provides as nice URIs, and LT also does but with more options to link to the images).

-- Build out other workflow features that will often require easily grabbing a local PDF programmatically through Zotero (this seems straightforward with the Javascript API provided)  

I'm all ears if Zotero there is a way to the first three things somehow that's easier than what I am proposing to do with aliases, or if there is an easy way to select a bunch of random Zotero entries and makes aliases to all the PDFs in those selected entries.

Thanks!

Russ

 

[Emiliano Heyns]

On Sunday, July 31, 2022 at 12:35:39 PM UTC+2 rus...@gmail.com wrote:

Among other things, I need to:

-- open an PDF or bunch of PDFs easily and quickly from any PDF app and save annotations back to the PDF

These are not hard assuming the app has a commend line option to open one or more PDFs.

 

-- provide access to those PDFs in an organized file hierarchy to my collaborators

As Dan pointed out, this is what Zotero groups do. If you want to have this outside Zotero, you're looking at a fragile solution where you'd have to keep an on-disk folder structure in sync while users are free to change both folder structure and file content without restriction. This seems like a recipe for data loss to me.

 

-- create a selection of a PDF as an image in the annotations file for that PDF (I can only do this in LiquidText, not any other PDF too, as far as I know)

Technically possible but it would be a fair bit of work.

 

-- have deep links into sections of the PDF with some kind of URI or "Hook" (which Zotero provides as nice URIs, and LT also does but with more options to link to the images).

So this seems well handled by Zotero?

 

-- Build out other workflow features that will often require easily grabbing a local PDF programmatically through Zotero (this seems straightforward with the Javascript API provided)  

This is too generic a request to address in any detail. It is easy enough to get programmatic access to Zotero content, I have two plugins that build REST-endpoints on the built-in Zotero web-server that can do whatever you want with Zotero content.

[Emiliano Heyns]

On Sunday, July 31, 2022 at 12:49:01 PM UTC+2 Emiliano Heyns wrote:

On Sunday, July 31, 2022 at 12:35:39 PM UTC+2 rus...@gmail.com wrote:

Among other things, I need to:

-- open an PDF or bunch of PDFs easily and quickly from any PDF app and save annotations back to the PDF

These are not hard assuming the app has a commend line option to open one or more PDFs.

Sorry -- I misread "from" as "with". "with" is easy. I don't have an easy solution for "from". I have dabbled with implementing a webdav service on the built-in http server but got stuck on header handling.

[Dan Stillman]

On 7/31/22 6:35 AM, Russ McBride wrote:

-- create a selection of a PDF as an image in the annotations file for that PDF (I can only do this in LiquidText, not any other PDF too, as far as I know)

You're aware of image annotations in Zotero 6?

https://www.zotero.org/blog/zotero-6/

[Russ McBride]

Dan, I didn't see that little gem of image annotations. Awesome! Also like that it automatically attaches a citation to each note. Nice! I can do more of my annotating in Zotero.

Emilian--

Dan is right that it would be better to go with the flow of Zotero and use Zotero storage. But I'm still stuck with the core problem of how to easily accessing my Zotero PDFs from other locations with other apps (mostly PDF readers/annotators). 

Is it possible to directly access Zotero PDFs stored with Zotero Storage on the Zotero servers using WebDAV or something?  If not, then I'm back to my original solution which is to keep my PDFs on DropBox or Google Drive and then build have a program that keeps my collections in sync with an identical folder hierarchy of alias/symbolic links (also on DropBox or GDrive) that point back to the originals. One initial obstacle is that Dropbox doesn't seem to respect symbolic links or aliases and GDrive will make them but only if done through their API (not a big deal since this will all happen programmatically, I guess--just annoying that I can't manually make aliases or symbolic links in the operating system).

Calls to doing so (in python or Node.js)  here: https://developers.google.com/drive/api/guides/shortcuts#python

PyZotero also looks cool... but probably out of date.

--russ

 

[Dan Stillman]

On 8/3/22 3:55 AM, Russ McBride wrote:

But I'm still stuck with the core problem of how to easily accessing my Zotero PDFs from other locations with other apps (mostly PDF readers/annotators)

I'm not sure what you mean by "other locations", but I did suggest how to make the files generally available to the rest of your system, by creating a smart folder in your OS that displays all PDFs within your 'storage' directory. You won't have the collection hierarchy, but the files will all be named based on the parent item metadata, and you can also search by content.

[Emiliano Heyns]

If you're going with gdrive shortcuts I'd recommend using the zotero api for a headless tool rather than doing this in a plugin. 

--
You received this message because you are subscribed to a topic in the Google Groups "zotero-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/zotero-dev/rpY4zNBzdIU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to zotero-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/zotero-dev/9604bad5-7273-473c-9273-eb122d25e95bn%40googlegroups.com.

[Stephan Hügel]

> PyZotero also looks cool... but probably out of date.

It probably isn't.

[Russ McBride]

@Stephan, thanks for the awesome work! It's up to date even though the last release was spring of 2019?

@Dan & @Emilan-- I think that's the first time I've ever tried a Mac smart folder. Works great... on the laptop. The bummer is that it's not recognizable when stored on DropBox or GDrive from other remote (iOS, iPadOS) apps because neither DropBox nor Google Drive respect the symbolic links created by the smart folder. I still like your idea of having my cake and eating it, i.e., using Zotero Storage and then coding up a hierarchy of GDrive Shortcuts to the PDFs that mirrors my Zotero Collections structure. It seems like these could be kept in sync with PyZotero and the GDrive python API.

My minor worry is that someone might make annotations through, e.g., Zotero PDF while I might be making annotations through, e.g., PDF Expert accessed through GDrive. But I'm usually in teams of 2-3 and this is a small concern relative to the large-scale benefits here.

--russ

[Stephan Hügel]

> @Stephan, thanks for the awesome work! It's up to date even though the last release was spring of 2019?

The latest release was on the 5th of July 2022: https://pypi.org/project/pyzotero/

Feature and bugfix releases are frequent: https://pypi.org/project/pyzotero/#history