Secure way to update Zotero add-ons

[jl]

Hi,

what is the best way to manage updates to Zotero add-ons today? Over time, I basically went through two different update processes.

1. Zotfile used to be listed as a Firefox add-on managed on the platform managed by Mozilla.

2. At some point, I moved to a self-hosted 'zotfile-update.rdf'. I sign the rdf file with McCoy (link) and new versions are hosted by GitHub.

This process has two problems: a) McCoy is dead and will stop working when I install macOS Catalina. b) Hosting on GitHub is not optimal and I don't think you are supposed to do that. I really need a solution for a.

So here are my questions: a) What is the best way to host and update Zotero add-ons today? b) What is the best way to transition from my current approach to a new system. I am worried that as soon as McCoy stops working I can't push updates to users anymore.

Thanks!

[Dan Stillman]

You can definitely stop using McCoy. Simply using HTTPS for the RDF file and the XPI obviates the need for both updateKey in install.rdf and updateHash in the update manifest.

For hosting, I think most people just use GitHub. E.g., see Zutilo:

https://github.com/willsALMANJ/Zutilo/commit/aab98cde7774b5e8701abcc5228417b052575a98

I would personally just put it in an Amazon S3 bucket for a bit more control (including, probably, setting Content-Type and/or Content-Disposition to prevent Firefox from trying to install it, until we do https://github.com/zotero/zotero/issues/1379) . I suspect the data transfer costs would be minimal, but you could do some quick calculations.

[jl]

Done. Thanks! That is great

[Emiliano Heyns]

I would personally just put it in an Amazon S3 bucket for a bit more control (including, probably, setting Content-Type and/or Content-Disposition to prevent Firefox from trying to install it, until we do https://github.com/zotero/zotero/issues/1379) . I suspect the data transfer costs would be minimal, but you could do some quick calculations.

You can set content-type for GH release assets. I have my xpis as assets, and some script that manage the update.rdf along with it. Agreed that S3 is simpler because you can have fixed paths there (required for the update.rdf).

[Will S]

On Friday, November 1, 2019 at 9:47:36 AM UTC-4, Emiliano Heyns wrote:

You can set content-type for GH release assets. I have my xpis as assets

Hmm, I don't see a practical difference between the xpi files attached your BBT GitHub releases and those attached to my Zutilo releases. In both cases, when I click on the link in Firefox, it tries to install the xpi as a Firefox add-on. If I try to do "Save link as...", I get a message that an unknown error has occurred. After I download the xpi's outside of Firefox, `file --mime-type` gives me application/zip for both. For Zutilo, I set the Content-Type header to application/xpi-install when I upload using the GitHub API.

> Hosting on GitHub is not optimal and I don't think you are supposed to do that

I don't think it is true that you are not supposed to host releases on GitHub. Releases are a standard feature of GitHub (e.g. https://help.github.com/en/github/administering-a-repository/creating-releases). I do agree that hosting on a site you fully control would give more control, and possibly be more reliable (trusting GitHub not to change their features vs. trusting yourself not to forget to renew a domain or something like that).

[Dan Stillman]

On 11/4/19 10:34 AM, Will S wrote:
> For Zutilo, I set the Content-Type header to application/xpi-install
> when I upload using the GitHub API.

You don't want to do that, since these are no longer Firefox extensions
and you don't want Firefox to try to install them. (It's possible
Firefox would still try to install an extension with a different
Content-Type if it had an .xpi extension, but hopefully not.)

We should probably just pick a MIME type for Zotero plugins, and a
future version of the Zotero Connector could prompt to install those
into the client. application/vnd.zotero.plugin?

[Emiliano Heyns]

On Monday, November 4, 2019 at 4:34:14 PM UTC+1, Will S wrote:

On Friday, November 1, 2019 at 9:47:36 AM UTC-4, Emiliano Heyns wrote:

You can set content-type for GH release assets. I have my xpis as assets

Hmm, I don't see a practical difference between the xpi files attached your BBT GitHub releases and those attached to my Zutilo releases. In both cases, when I click on the link in Firefox, it tries to install the xpi as a Firefox add-on. If I try to do "Save link as...", I get a message that an unknown error has occurred. After I download the xpi's outside of Firefox, `file --mime-type` gives me application/zip for both. For Zutilo, I set the Content-Type header to application/xpi-install when I upload using the GitHub API.

Save link as works for me without issue. It's how I've always done it.

 

> Hosting on GitHub is not optimal and I don't think you are supposed to do that

I don't think it is true that you are not supposed to host releases on GitHub. Releases are a standard feature of GitHub (e.g. https://help.github.com/en/github/administering-a-repository/creating-releases). I do agree that hosting on a site you fully control would give more control, and possibly be more reliable (trusting GitHub not to change their features vs. trusting yourself not to forget to renew a domain or something like that).

Reliability hasn't been an issue for me, and as long as you control update.rdf, you can move fairly easily.

That's not to say a separate site couldn't have benefits (for one, you need a stable location for update.rdf, and that's awkward on GH), and the simplest S3 bucket will do, but GH releases have worked just fine for me, and I don't see what else they would be useful for -- it is a supported feature that you can attach your own assets, the site allows it, there's an API endpoint to do it.

[Emiliano Heyns]

On Monday, November 4, 2019 at 6:19:28 PM UTC+1, Dan Stillman wrote:

On 11/4/19 10:34 AM, Will S wrote:
> For Zutilo, I set the Content-Type header to application/xpi-install
> when I upload using the GitHub API.

You don't want to do that, since these are no longer Firefox extensions
and you don't want Firefox to try to install them. (It's possible
Firefox would still try to install an extension with a different
Content-Type if it had an .xpi extension, but hopefully not.)

I don't think it would.

 

We should probably just pick a MIME type for Zotero plugins, and a
future version of the Zotero Connector could prompt to install those
into the client. application/vnd.zotero.plugin?

That would be a trivial change for me.

[Emiliano Heyns]

On Monday, November 4, 2019 at 7:11:05 PM UTC+1, Emiliano Heyns wrote:

On Monday, November 4, 2019 at 4:34:14 PM UTC+1, Will S wrote:

On Friday, November 1, 2019 at 9:47:36 AM UTC-4, Emiliano Heyns wrote:

You can set content-type for GH release assets. I have my xpis as assets

Hmm, I don't see a practical difference between the xpi files attached your BBT GitHub releases and those attached to my Zutilo releases. In both cases, when I click on the link in Firefox, it tries to install the xpi as a Firefox add-on. If I try to do "Save link as...", I get a message that an unknown error has occurred. After I download the xpi's outside of Firefox, `file --mime-type` gives me application/zip for both. For Zutilo, I set the Content-Type header to application/xpi-install when I upload using the GitHub API.

Save link as works for me without issue. It's how I've always done it.

Right-click, save-as also works for me on the Zutilo release. No error, and `file zutilo.xpi` says `Zip archive data, at least v2.0 to extract`, as I'd expect

[Emiliano Heyns]

On Monday, November 4, 2019 at 7:59:42 PM UTC+1, Emiliano Heyns wrote:

On Monday, November 4, 2019 at 7:11:05 PM UTC+1, Emiliano Heyns wrote:

On Monday, November 4, 2019 at 4:34:14 PM UTC+1, Will S wrote:

On Friday, November 1, 2019 at 9:47:36 AM UTC-4, Emiliano Heyns wrote:

You can set content-type for GH release assets. I have my xpis as assets

Hmm, I don't see a practical difference between the xpi files attached your BBT GitHub releases and those attached to my Zutilo releases. In both cases, when I click on the link in Firefox, it tries to install the xpi as a Firefox add-on. If I try to do "Save link as...", I get a message that an unknown error has occurred. After I download the xpi's outside of Firefox, `file --mime-type` gives me application/zip for both.

That is expected. The content-type is between the browser and the server. Once the bytes hit the disk, that information is lost. Each OS has its own way to associate extensions with mimetypes, but what the server said at time of download is not part of that process.

[Will S]

> Right-click, save-as also works for me on the Zutilo release. No error, and `file zutilo.xpi` says `Zip archive data, at least v2.0 to extract`, as I'd expect

The behavior I see must be system specific then. I see an "unknown error" on Linux, and another user who was also using Linux contacted me about the error as well.

> You don't want to do that, since these are no longer Firefox extensions
> and you don't want Firefox to try to install them. (It's possible
> Firefox would still try to install an extension with a different
> Content-Type if it had an .xpi extension, but hopefully not.) 

The fact that Emiliano and I see the same behavior for Zutilo and BBT makes me think that GitHub does not do anything with this Content-Type header. For now I could change it application/zip (nothing else in the list linked to from the GitHub documentation seems appropriate https://developer.github.com/v3/repos/releases/#upload-a-release-asset).

[Emiliano Heyns]

On Thursday, November 7, 2019 at 8:18:00 PM UTC+1, Will S wrote:

> Right-click, save-as also works for me on the Zutilo release. No error, and `file zutilo.xpi` says `Zip archive data, at least v2.0 to extract`, as I'd expect

The behavior I see must be system specific then. I see an "unknown error" on Linux, and another user who was also using Linux contacted me about the error as well.

I've just tried on Ubuntu 19.10, and I can right-click download without problem in firefox (60.8.0esr if that matters). Download shows as zip file, as expected.

 

The fact that Emiliano and I see the same behavior for Zutilo and BBT makes me think that GitHub does not do anything with this Content-Type header. For now I could change it application/zip (nothing else in the list linked to from the GitHub documentation seems appropriate https://developer.github.com/v3/repos/releases/#upload-a-release-asset).

When downloading the content-type shows as application/x-xpinstall for both Zutilo and BBT. I don't know what you set it to at the time of upload; mine was always uploaded as application/x-xpinstall, so I can't judge whether GH overrides this.GH links to a list of media types but the docs do not state that it's limited to a well-known set.

[Dan Stillman]

On 11/7/19 2:18 PM, Will S wrote:
> > You don't want to do that, since these are no longer Firefox extensions
> > and you don't want Firefox to try to install them. (It's possible
> > Firefox would still try to install an extension with a different
> > Content-Type if it had an .xpi extension, but hopefully not.)
>
> The fact that Emiliano and I see the same behavior for Zutilo and BBT
> makes me think that GitHub does not do anything with this Content-Type
> header. For now I could change it application/zip (nothing else in the
> list linked to from the GitHub documentation seems appropriate
> https://developer.github.com/v3/repos/releases/#upload-a-release-asset).

I'm not sure what you mean by "GitHub does not do anything with this
Content-Type header". Both Zutilo and BBT are currently uploaded as
application/x-xpinstall, which is why Firefox offers to install them
both. GitHub serves the file with the type that you specify — that's the
thing it does with the header.

The list GitHub links to is just a list of registered types, but there's
no need to specify something from there.

If there are no objections, let's go with application/vnd.zotero.plugin,
and we'll look into adding installation from the Zotero Connector. I've
also confirmed that Firefox does not offer to install an XPI with
another type, so this will immediately put an end to the confusing
"appears to be corrupt" message for Firefox users.

[Dan Stillman]

On 11/7/19 2:44 PM, Dan Stillman wrote:
> If there are no objections, let's go with
> application/vnd.zotero.plugin, and we'll look into adding installation
> from the Zotero Connector.

Issue created: https://github.com/zotero/zotero-connectors/issues/297

[Emiliano Heyns]

On Thursday, November 7, 2019 at 8:44:23 PM UTC+1, Dan Stillman wrote:

If there are no objections, let's go with application/vnd.zotero.plugin,
and we'll look into adding installation from the Zotero Connector. I've
also confirmed that Firefox does not offer to install an XPI with
another type, so this will immediately put an end to the confusing
"appears to be corrupt" message for Firefox users.

Done. My next release will use this mimetype; before Zotero support, this should just prompt users to download the plugin rather than install, but that's already an improvement.

[Will S]

On Thursday, November 7, 2019 at 2:44:23 PM UTC-5, Dan Stillman wrote:

I'm not sure what you mean by "GitHub does not do anything with this
Content-Type header". Both Zutilo and BBT are currently uploaded as
application/x-xpinstall, which is why Firefox offers to install them
both. GitHub serves the file with the type that you specify — that's the
thing it does with the header.

Ah, okay. I think everything makes sense to me now. When Emiliano said that he uploaded his xpis as assets, I thought he meant he was using an asset Content-Type instead of application/xpi-install. So then the fact that I saw the same behavior for BBT and Zutilo would have meant that the Content-Type was not doing anything. However, since he was also using application/xpi-install, now it makes sense that I see the same behavior for both. I can change the Content-Type to application/vnd.zotero.plugin and hope that it fixes the Firefox issues for me.

[Brenton Wiernik]

Would `application/vnd.zoteroplugin` (without the second dot) or `application/vnd.zotero` be better to accommodate the Windows convention of only one file extension?

[Emiliano Heyns]

Would that mean we're also using that as a file extension rather than xpi?

[Dan Stillman]

On 12/3/19 10:12 AM, Brenton Wiernik wrote:
> Would `application/vnd.zoteroplugin` (without the second dot) or
> `application/vnd.zotero` be better to accommodate the Windows
> convention of only one file extension?

The media type is totally separate from the file extension. The file
extension can still be .zoteroplugin.

vnd.zotero.plugin is consistent with other media types (e.g.,
application/vnd.openxmlformats-officedocument.wordprocessingml.document,
application/vnd.oasis.opendocument.text).

[Brenton Wiernik]

Okay, thanks for the clarification. Forgive my ignorance, how exactly do you specify the content_type header when serving releases through github?

[Emiliano Heyns]

On Wednesday, December 4, 2019 at 12:03:16 AM UTC+1, Brenton Wiernik wrote:

Okay, thanks for the clarification. Forgive my ignorance, how exactly do you specify the content_type header when serving releases through github?

You can just specify it as a header when you're uploading the asset, eg https://gist.github.com/stefanbuck/ce788fee19ab6eb0b4447a85fc99f447

[Will S]

For what it's worth -- I changed the Content-Type header to application/vnd.zotero.plugin, but Firefox still offers to install the xpi when I click on the link and right-click->"Save link as" still results in an unknown error. Since other people don't see this, I think it is OS dependent. I see this on Fedora.

[Dan Stillman]

On 12/6/19 12:25 PM, Will S wrote:
> For what it's worth -- I changed the Content-Type header to
> application/vnd.zotero.plugin, but Firefox still offers to install the
> xpi when I click on the link and right-click->"Save link as" still
> results in an unknown error. Since other people don't see this, I
> think it is OS dependent. I see this on Fedora.

If you mean Zutilo 3.3.3, that's still being served with Content-Type:
application/x-xpinstall. (You can see in the Network pane in browser
devtools.)

[jl]

Hi!

I ran into a problem with my last update. Can someone help me out with this? Zotero throws this error when I try to update zotfile:

Here is the GitHub repro: https://github.com/jlegewie/zotfile

[JavaScript Error: "XML Parsing Error: prefix not bound to a namespace

Location: moz-nullprincipal:{e9287aed-7214-9f4c-bcde-f6ecfa7fa532}

Line Number 4, Column 5:" {file: "moz-nullprincipal:{e9287aed-7214-9f4c-bcde-f6ecfa7fa532}" line: 4 column: 5 source: "    <RDF:Description about="urn:mozilla:extension:zot...@columbia.edu">"}]

Also what are the changes I should make to follow the convention about content type and file extension? 

Thanks!

[Emiliano Heyns]

On Saturday, February 1, 2020 at 3:45:38 PM UTC+1, jl wrote:

Hi!

I ran into a problem with my last update. Can someone help me out with this? Zotero throws this error when I try to update zotfile:

Here is the GitHub repro: https://github.com/jlegewie/zotfile

[JavaScript Error: "XML Parsing Error: prefix not bound to a namespace

Location: moz-nullprincipal:{e9287aed-7214-9f4c-bcde-f6ecfa7fa532}

Line Number 4, Column 5:" {file: "moz-nullprincipal:{e9287aed-7214-9f4c-bcde-f6ecfa7fa532}" line: 4 column: 5 source: "    <RDF:Description about="urn:mozilla:extension:zot...@columbia.edu">"}]

I’d guess your missing a xmlns:RDF="http://www.w3.org/1999/02/22-rdf-syntax-ns#” 

 

Also what are the changes I should make to follow the convention about content type and file extension? 

That depends on how you’re hosting the XPI. If they live in GH releases (looks like it), you can specify the content-type when you upload using the GH API, which is how my own extensions get to GH. I don’t see a way to specify the content-type in the web-ui for releases.

[jl]

Thanks. "xmlns:RDF" is line 2 in zotfile-update.rdf though...

[Emiliano Heyns]

Bad guess then 😉. What's the full link to the rdf file? 

--
You received this message because you are subscribed to a topic in the Google Groups "zotero-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/zotero-dev/ryzC01j8IhY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to zotero-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/zotero-dev/fe9b47d0-74c8-4aa7-83cc-f84e0db48f0a%40googlegroups.com.

[jl]

Here you go:

https://github.com/jlegewie/zotfile/blob/master/zotfile-update.rdf

On Sunday, February 2, 2020 at 8:23:13 AM UTC-5, Emiliano Heyns wrote:

Bad guess then 😉. What's the full link to the rdf file? 

To unsubscribe from this group and all its topics, send an email to zoter...@googlegroups.com.

[Emiliano Heyns]

Aren't namespace prefixes case sensitive? You have both rdf and RDF. 

To unsubscribe from this group and all its topics, send an email to zotero-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/zotero-dev/51b6bf10-b0e6-4db9-a71a-05a8df13214a%40googlegroups.com.

[Will S]

On Friday, December 6, 2019 at 2:29:42 PM UTC-5, Dan Stillman wrote:

If you mean Zutilo 3.3.3, that's still being served with Content-Type:
application/x-xpinstall. (You can see in the Network pane in browser
devtools.)

Yes, but I don't understand why. I should be uploading it with "Content-Type: application/vnd.zotero.plugin". My script for uploading is:

https://github.com/willsALMANJ/release.py/blob/master/release.py#L251

https://github.com/willsALMANJ/Zutilo/blob/master/deploy/config.yaml#L7

I will try to debug it carefully the next time I make a release. Perhaps I am missing something.

[Emiliano Heyns]

On Sunday, February 2, 2020 at 9:09:33 PM UTC+1, Emiliano Heyns wrote:

Aren't namespace prefixes case sensitive? You have both rdf and RDF. 

That's the problem with the RDF. On line 4 you have "RDF:Description", that should be "rdf:Description"

WRT the content-type: it looks like GH ignores the content-type set during upload. Mine are also still served out with "Content-Type: application/x-xpinstall"