Custom citation keys?

[Erik Hetzner]

Hi,

I asked about this the other day, & I am sure you are all tired of
this. But this feature has been in the works for 4 years:

https://www.zotero.org/trac/ticket/763

Is this stalled because there is a difficulty with engineering the
feature, or because nobody has implemented it?

If this is answered somewhere, I am unable to find it. Thanks for any
help that others can provide here!

best, Erik

[Avram Lyon]

Erik,

On Fri, Aug 26, 2011 at 4:42 AM, Erik Hetzner <e...@e6h.org> wrote:
> I asked about this the other day, & I am sure you are all tired of
> this. But this feature has been in the works for 4 years:
>
>  https://www.zotero.org/trac/ticket/763
>
> Is this stalled because there is a difficulty with engineering the
> feature, or because nobody has implemented it?

This is not an unreasonable request, and there is agreement that
something should be done. The most recent discussion for Zotero in
particular is at https://github.com/ajlyon/zotero-bits/issues/24,
where Bruce and Simon expressed support for the idea.

There has been discussion of somewhat related questions on the CSL
mailing lists (probably XBIBLIO, but maybe it was bug traffic on the
schema bug tracker).

On the Zotero side, we're waiting for a new field and an
implementation of the generation / templating system for the keys.

Avram

[Bruce D'Arcus]

So for sake of argument, if Erik wanted to implement this now, he
could, or couldn't?

Like Erik and others, I personally consider this a priority.

Bruce

[Rintze Zelle]

In https://github.com/ajlyon/zotero-bits/issues/24, the proposal is to use a single CSL variable (the existing "citation-label") for key storage. I just realized that this would be limiting if separate keys are required for a) BibTeX referencing and b) label styles. If people need to convert between different label styles (e.g. "Doe2009" and "Doe09"), that would conflict with the objective of having stable BibTeX keys. So maybe we need multiple variables.

Rintze

[Bruce D'Arcus]

No, because we have no idea how many different forms there would be.

I think we need to treat that case (style-specific key formats) as
different. One solution:

There is a label, but it's not editable; rather, one can simply choose
the key format to use.

So you're using a style foo.csl that requires one style of key, and
you choose that from a preference. When browsing your library, you can
see that key in a column, and when you export a bibtex/mods/ris file,
you get that value.

Saves labor too.

Bruce

[Simon]

It's difficulty in engineering the feature. Adding a new field to items would require that we sync that field. While I don't know the sync code, Dan has posted on the issue of sync changes at https://groups.google.com/d/msg/zotero-dev/qWQElPUaPes/POwJXuBXXvUJ.

An alternative would be to make a non-editable key that is defined by a user-specified template. As long as the template doesn't need to sync, I believe we could do this without a sync cutoff, although it would need a DB upgrade. However, I'm not sure this second solution makes sense if the eventual goal is to have editable keys.

Simon

[Bruce D'Arcus]

So perhaps ...

... we ought to first establish whether an editable, syncable, key is important?

I say it's not, but that, at least at some point, it would be nice for
the key template could be synced.

So imagine a UI config called "label key template" that had a handful
of default options initially, but then could be easily extended to
include an "other" option where users could create their own.

How feasible and sensible would that be?

Bruce

[Richard Karnesky]

> ... we ought to first establish whether an editable, syncable, key is important?
>
> I say it's not

I think it is very important. The only real disadvantages seems to be
that it is hard and will take longer to implement this way, but it
seems to be the right thing to do because (i) it is the only way to
keep legacy keys and (ii) it is the only way I see to have sane
stability to keys that need disambiguation.

(i) mostly impacts LaTeX/BibTeX users and a lot of support for having
more configurable keys is driven from these users.

(ii) may be a bit more subtle. Many computer-generated label-
generating schemes make very short labels that only use an author and
the year (indeed, the examples in this thread are structured that
way). But this leads to collisions that must be disambiguated.
Sometimes, some strings of the title are used (as in BibTeX.js). But
this doesn't prevent all collisions & makes for longer/more tedious
keys. More often, a number or letter is appended to the key. Perhaps
this token can be based off of the item creation date in a completely
automatic system. But it still would not be stable: Zotero allows the
removal of items from the database, so removing items before the most
recent one would cause re-numbering/re-lettering of all subsequent
entries.

> So imagine a UI config called "label key template" that had a handful
> of default options initially, but then could be easily extended to
> include an "other" option where users could create their own.

By "other" do you mean another option for a template for auto-
generated label keys or to instruct it to use a newly added field that
is a manually entered key or both?

If we eventually go to a system that does have user-customizable keys
for the two reasons above, I'm with Simon in questioning the utility
of having a label template field that might be wholly deprecated in
the future.

--Rick

[Bruce D'Arcus]

On Fri, Aug 26, 2011 at 2:12 PM, Richard Karnesky <karn...@gmail.com> wrote:
>> ... we ought to first establish whether an editable, syncable, key is important?
>>
>> I say it's not
>
> I think it is very important.  The only real disadvantages seems to be
> that it is hard and will take longer to implement this way,

There are two disadvantages:

1) for Zotero devs (what Simon said)

2) for users who have to maintain these keys (add them, check for
conflicts, etc.)

> but it seems to be the right thing to do because (i) it is the only way to
> keep legacy keys

This is true.

> and (ii) it is the only way I see to have sane
> stability to keys that need disambiguation.

Maybe. But I'm still curious how you would propose to deal with the
case I mentioned of people having label styles that require particular
key formats?

> (i) mostly impacts LaTeX/BibTeX users and a lot of support for having
> more configurable keys is driven from these users.

This may be true now, but I'm not sure I'd want to base an
implementation decision based on the assumption that it will always be
true.

In my own personal case, I want keys for use with pandoc and a variety
of output formats (but mostly RIS).

> (ii) may be a bit more subtle.  Many computer-generated label-
> generating schemes make very short labels that only use an author and
> the year (indeed, the examples in this thread are structured that
> way).  But this leads to collisions that must be disambiguated.
> Sometimes, some strings of the title are used (as in BibTeX.js).  But
> this doesn't prevent all collisions & makes for longer/more tedious
> keys.  More often, a number or letter is appended to the key.  Perhaps
> this token can be based off of the item creation date in a completely
> automatic system.  But it still would not be stable: Zotero allows the
> removal of items from the database, so removing items before the most
> recent one would cause re-numbering/re-lettering of all subsequent
> entries.

Good point on an important detail. But still, I would hope we don't
expect a user with 5000 items in their db to manage this themselves?

>> So imagine a UI config called "label key template" that had a handful
>> of default options initially, but then could be easily extended to
>> include an "other" option where users could create their own.
>
> By "other" do you mean another option for a template for auto-
> generated label keys

This: a user-created template.

Bruce

> or to instruct it to use a newly added field that
> is a manually entered key or both?
>
> If we eventually go to a system that does have user-customizable keys
> for the two reasons above, I'm with Simon in questioning the utility
> of having a label template field that might be wholly deprecated in
> the future.
>
> --Rick
>

> --
> You received this message because you are subscribed to the Google Groups "zotero-dev" group.
> To post to this group, send email to zoter...@googlegroups.com.
> To unsubscribe from this group, send email to zotero-dev+...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/zotero-dev?hl=en.
>
>

[Richard Karnesky]

> There are two disadvantages:
>
> 1) for Zotero devs (what Simon said)
>
> 2) for users who have to maintain these keys (add them, check for
> conflicts, etc.)

I don't think that (2) is a big disadvantage, mostly because of the
answer to the next question you propose.

> Maybe. But I'm still curious how you would propose to deal with the
> case I mentioned of people having label styles that require particular
> key formats?

The same way JabRef and others handle it: provide a tool to overwrite
keys of selected items using that preferred format. Allowing people
custom labels does not mean there shouldn't also be tools to handle
key (re-)generation, duplicate handling, etc.

> > (i) mostly impacts LaTeX/BibTeX users and a lot of support for having
> > more configurable keys is driven from these users.
>
> This may be true now, but I'm not sure I'd want to base an
> implementation decision based on the assumption that it will always be
> true.

That's fine. The question is: do you use some reference management
outside of zotero that requires labels to survive the import/export
round trip? Most BibTeX/LaTeX users probably want this, but they are
not necessarily the only people who do.


--Rick

[Erik Hetzner]

Hi,

Thanks Avram, Bruce, Simon, Richard and Rintze for your answers to my
question.

Perhaps it will help if I give my use case.

With Frank Bennett, I have been working on extensions for docutils to
support citations generated by Zotero in reStructuredText (reST)
documents. This is in some ways quite similar to the work done in
pandoc using citeproc-hs, except that we interact directly with
Zotero, [1] in much the same way as the Word/OpenOffice plugins work
(as I understand; I have never used these plugins).

Users can insert citations into their reST documents using a syntax
that looks like:

see @3A8VMRJ5 p. 12

Now, for the time being we are using Zotero keys for this (these look
like 3A8VMRJ5). It would obviously be better to use:

see @Cohen2008 p. 12

because these keys are entered directly by the user into a text
document. Clearly it would be nice to have keys that can be remembered
by ordinary people.

Now, our tools interact directly with Zotero. So when the user is
generating a ODT file from the reST source, they type:

rst2odt mysource.rst

and the rst2odt script communicates with Zotero when generating the
opendocument file.

The solution I imagine would look like this:

1. Every Zotero item has a human readable key field, editable by the
user.
2. This key is unique in the database schema.
3. Each item is given a unique, generated key when it is added to
the user’s library.
4. The key is generated by a user-chosen function (e.g. LastnameYear).
5. The above functions will have a solution if the key is not unique
initially; this could be as simple as appending a 1 after the end
until the key is unique.
6. After initial key generation, Zotero never touches the key except
on user input.
7. Users can change the key to whatever they want; uniqueness is
enforced by rejecting any change until the key is unique.
8. Users may have the option of re-generating all keys according to a
new function, but they will lose their customization.

But I imagine that I am missing some other use cases that have come up
before.

best, Erik

1. This may or may not be a mistake; it does tie the document to a
particular user’s library, which bothers me.

[David Sanson]

> It would obviously be better to use:
>
> see @Cohen2008 p. 12
> 

> because these keys are entered directly by the user into a text
> document.

And because the whole point of rst (or markdown) is that it be human readable plain text.

If Zotero's database supported both editable keys and standard Zotero keys, one could avoid tying the document to a particular user's library by providing for a way to generate a mapping from the user editable keys to the Zotero keys, a way of including that in the rst or markdown document (perhaps as an appendix or a metadata block), and a way of allowing the parser to use those mappings. Or one could just provide a tool for generating an "archival" version of the document, swapping out the editable keys with Zotero keys.

David

[Erik Hetzner]

Hi all,

At Thu, 1 Sep 2011 08:57:53 -0700 (PDT),

David Sanson wrote:
>
> And because the whole point of rst (or markdown) is that it be human
> readable plain text.

That too; although it raises a slight problem for my zotero4rst code
in that the human-readable rst file is tied to a user’s Zotero
library, which is not human readable. I am considering solutions to
this problem.

> If Zotero's database supported both editable keys and standard Zotero keys,
> one could avoid tying the document to a particular user's library by
> providing for a way to generate a mapping from the user editable keys to the
> Zotero keys, a way of including that in the rst or markdown document
> (perhaps as an appendix or a metadata block), and a way of allowing the
> parser to use those mappings. Or one could just provide a tool for
> generating an "archival" version of the document, swapping out the editable
> keys with Zotero keys.

I solved this problem for the time being by using a mapping file to
map human readable keys to Zotero library keys:

item1 = ZBZQ4KMP

Another issue is that it is not easy to get ahold of these library
keys. You can use the emacs support of zotero-plain, but this is a bit
of a hack.

I am still looking for a solution to the human readable key problem. I
think my solution that I laid out earlier is reasonable, and if
anybody has feedback I would love to hear it.

In the meantime, there has been a huge overhaul of the docutils/rst
support in zotero-plain; please do check it out if you get a chance!
We now support almost everything that pandoc+citeproc-hs does, but we
read the citations directly from a Zotero instance. Here is the pandoc
example document, copied to zotero4rst:

https://bitbucket.org/egh/zotero-plain/raw/tip/example/example.rst
https://bitbucket.org/egh/zotero-plain/raw/tip/example/example.html
https://bitbucket.org/egh/zotero-plain/raw/tip/example/example.pdf
https://bitbucket.org/egh/zotero-plain/raw/tip/example/example.odt

best, Erik

[David Sanson]

> Another issue is that it is not easy to get ahold of these library keys.

I managed to hack up working key completion in vim from a local instance of zotero in vim using pygnotero and modifying the code here:

http://www.cogsci.nl/blog/tutorials/97-writing-a-command-line-zotero-client-in-9-lines-of-code

If someone wants to take a look, you can find the relevant bit of code (an unholy mixture of python and vimscript) here:

https://github.com/vim-pandoc/vim-pandoc/blob/zotero/plugin/pandoc.vim#L287

The way it works in vim is that you type something like `@sanson` and it searches Zotero for any entries that contain 'sanson', and returns a popup each result with the key, the first author, and the title. You select what you want from the popup, and vim replaces `sanson` with the selected key, e.g., `@EXIVRISR`. Using pygnotero, getting the key was fairly easy (though I have no idea how robust this is, and would be happier using an official API). One nice thing about it is that it does not require that the user interact with Zotero at all, so she can stay in the flow of her writing.

I haven't found any way to get zotero to feed me a bibliography that contains these keys, either via manual export or on request from the command line (I haven't looked at mozrepl), so I can't use pandoc and citeproc-hs to process the resulting citations.

My ultimate goal is to offer a uniform experience for pandoc authors using a static bibliography database in one of the formats supported by citeproc-hs and pandoc authors using Zotero....

David

[Stephan Hügel]

I haven't found any way to get zotero to feed me a bibliography that contains these keys, either via manual export or on request from the command line (I haven't looked at mozrepl), so I can't use pandoc and citeproc-hs to process the resulting citations.

This is a problem, but the Read server API can almost do it: You can pass a 'content' parameter when making a request for an item, and you can pass a 'style' parameter along with it for 'bib'-type content. I've even added a 'subset' method to pyzotero, so you can retrieve an arbitrary selection of non-adjacent items using their keys: see here. I'd have to adapt it a little bit so that it remembers any url parameters that have been set, but that's (probably) not a big deal.

The problem (for this use case, anyway) is that the bibliography entry is returned as a HTML fragment in an atom doc, as opposed to a string, or JSON, which makes it a pain to embed as rst. I don't personally fancy stripping off all the HTML. It might be worth asking Dan/Simon if there's a way to get 'raw' bib data back.

[Dan Stillman]

On 9/2/11 5:04 PM, Stephan H�gel wrote:
> The problem (for this use case, anyway) is that the bibliography entry
> is returned as a HTML fragment in an atom doc, as opposed to a string,
> or JSON, which makes it a pain to embed as rst. I don't personally
> fancy stripping off all the HTML. It might be worth asking Dan/Simon
> if there's a way to get 'raw' bib data back.

What do you mean by 'raw' bib data? If you mean BibTeX (or other
standard formats supported by the Zotero client), that's coming soon.
There's also content=json for raw Zotero fields.

You can also use format=bib to get just an HTML bibliography, but then
you don't get keys.

[Stephan Hügel]

Dan: My point regarding the 'bib'-style returned HTML bibliographies was that they work perfectly, but only if you want to embed HTML, which isn't the case for David. But maybe I'm overthinking this; I know that there's nothing to prevent someone building a bibliography now, using returned item data (i.e. the result of item() API calls, whether it's extracted from the atom doc or the JSON), and formatting it appropriately on the client side. But that feels like re-inventing the wheel (considering the amount of work that's gone into e.g. BibLaTeX implementations of Chicago and MLA, and more importantly, Zotero's CSL system), and it'll be a pain for anything other than the most basic bib formats anyway.

It's probably more sensible to wait on whatever you guys do, and use the server API.

[Dan Stillman]

OK, but I'm still not sure what you're asking for from us, then. A
plain-text 'bib' mode (which citeproc-js supports, I believe), and
styles that require rich-text formatting are just lossy? I'm just trying
to figure out what you meant by "'raw' bib data".

[David Sanson]

I poked around in the server api. The closest I got was 

https://api.zotero.org/users/<userid>/items/key=<api key>&format=atom&content=json

That delivered an atom feed with all the items in my zotero database, with some json embedded for each one. I don't think that is the sort of thing that citeproc-hs can handle without further processing. I assume I'd need to extract the embedded json for each item and combine them into a single json representation. 

What would be nice is a way to get a pure json representation of the database. And it would be nice to be able to get this both locally and from the server. Sounds like that sort of thing might be in the works, which is great.

And looks like I definitely need to take a look at pyzotero...

David

[Stephan Hügel]

Ah, right. I meant something like the HTML-formatted bibliography that's currently returned, but not HTML. Maybe a JSON representation of a citation in whatever format is specified. I don't know anything about citeproc-js (I've only dealt with the server API), so maybe I'm missing something.

[Bruce D'Arcus]

On Fri, Sep 2, 2011 at 6:04 PM, Stephan Hügel <ursc...@gmail.com> wrote:

> Ah, right. I meant something like the HTML-formatted bibliography that's
> currently returned, but not HTML. Maybe a JSON representation of a citation
> in whatever format is specified.

That doesn't make any sense. If by "whatever format" you mean citation
format, then you need something that's presentation-oriented like
HTML. If by that phrase you mean the bib format (bibtex, ris, mods,
etc.) then it makes no sense to talk of JSON.

Can you try again?

Bruce

[Stephan Hügel]

David,

Pyzotero will enable you to get an entire library representation as python objects (lists of dicts, each dict representing an item). You can easily retrieve an arbitrary collection of items using it, and (presumably?) feed that collection to citeproc from there. I don't know anything about citeproc itself, but feel free to ask if there's something you'd like to see in Pyzotero.

[Kieren Diment]

I have some code that does something that from when I was messing around with annotated bibliographies.

var z = Zotero;
var report = new Array;
var collection= z.getActiveZoteroPane().getSelectedCollection();
var items = collection.getChildItems();
for (var i=0;i<items.length;i++) {
item = items[i];
var qc = z.QuickCopy; // uses the default selected citation output style
var cite = qc.getContentFromItems(new Array(item),
z.Prefs.get("export.quickCopy.setting"));
var type = z.ItemTypes.getName(item.itemTypeID);
var date = z.Date.formatDate(z.Date.strToDate(item.getField('date')),'y')
var abs = item.getField('abstractNote');
report.push({ 'citation': cite, 'date': date, 'abstract': abs, 'type': type });
}
return (report.toSource());

[Stephan Hügel]

I mean an abstract representation of a citation. Using HTML to present e.g. an MLA-formatted citation is fine, but only if you want to render HTML. If you want to output a citation as RST, or Markdown, or Pandoc, starting with HTML is a pain – you're going backwards from the end-result. Starting with a JSON representation of a citation would be far easier to work with.

[Dan Stillman]

content=json is a 1:1 mapping of Zotero fields, meant primarily for use
with the write API but usable by anything that wants lossless
transportation of Zotero data. It has nothing to do with CSL or
citeproc-*. But if there's demand, we can trivially offer citeproc-js
JSON too, which I believe is shaping up to be the standard input format
for the various CSL implementations.

[Dan Stillman]

On 9/2/11 6:17 PM, Stephan H�gel wrote:
> I mean an abstract representation of a citation. Using HTML to present
> e.g. an MLA-formatted citation is fine, but only if you want to render
> HTML. If you want to output a citation as RST, or Markdown, or Pandoc,

> starting with HTML is a pain � you're going backwards from the

> end-result. Starting with a JSON representation of a citation would be

> far easier to work with. --

You're going to have to provide an example (and explain how it differs
from content=json or citeproc-js JSON). I still don't have any idea what
you mean.

[Bruce D'Arcus]

On Fri, Sep 2, 2011 at 6:17 PM, Stephan Hügel <ursc...@gmail.com> wrote:
> I mean an abstract representation of a citation. Using HTML to present e.g.
> an MLA-formatted citation is fine, but only if you want to render HTML. If
> you want to output a citation as RST, or Markdown, or Pandoc, starting with
> HTML is a pain – you're going backwards from the end-result.

Well, not all HTML is created equal. Certainly some representations of
HTML are easy enough to convert to other formats? Indeed, the very
design of CSL is based on that notion (given that it borrows its
styling language design from CSS).

> Starting with a
> JSON representation of a citation would be far easier to work with.

On the not-all-representation-are-equal theme, and to Dan's point,
what kinds of "JSON representation"? Are you talking about something
like pandoc's JSON representation of its internal model?

Bruce

[Bruce D'Arcus]

if this is easy for you to do, I'd like to see this.

Bruce

[Erik Hetzner]

At Fri, 2 Sep 2011 19:11:45 -0400,

Bruce D'Arcus wrote:
>
> On Fri, Sep 2, 2011 at 6:17 PM, Stephan Hügel <ursc...@gmail.com> wrote:
> > I mean an abstract representation of a citation. Using HTML to present e.g.
> > an MLA-formatted citation is fine, but only if you want to render HTML. If
> > you want to output a citation as RST, or Markdown, or Pandoc, starting with
> > HTML is a pain – you're going backwards from the end-result.
>
> Well, not all HTML is created equal. Certainly some representations of
> HTML are easy enough to convert to other formats? Indeed, the very
> design of CSL is based on that notion (given that it borrows its
> styling language design from CSS).

For what it’s worth, I use HTML output from citeproc-js to generate a
rst node tree. It’s not a lot of code, once you have an HTML parser.

best, Erik

[David Sanson]

Yes, this would be very useful.

David

[Stephan Hügel]

I've updated Pyzotero, so that get_subset() now remembers URL parameters. I tested it with content='bib' and style='mla', and a list of 3 items, and the result is here. You can pass a list of up to 50 item IDs. The returned list items are unicode strings, so you'll have to encode them back to UTF-8 (for example) at some point before output.

[Erik Hetzner]

Hi David,

At Fri, 2 Sep 2011 08:55:48 -0700 (PDT),

David Sanson wrote:
>
> > Another issue is that it is not easy to get ahold of these library keys.
>
> I managed to hack up working key completion in vim from a local instance of
> zotero in vim using pygnotero and modifying the code here:
>
>

> […]

This is very cool! As an emacs user I must disdain vimscript, but I
may port my emacs tools for Zotero to pygnotero, it looks easier to
deal with than a MozRepl hack.

> I haven't found any way to get zotero to feed me a bibliography that
> contains these keys, either via manual export or on request from the command
> line (I haven't looked at mozrepl), so I can't use pandoc and citeproc-hs to
> process the resulting citations.
>
> My ultimate goal is to offer a uniform experience for pandoc authors using a
> static bibliography database in one of the formats supported by citeproc-hs
> and pandoc authors using Zotero....

Yes, that is my goal too. Right now documents created with zotero4rst
are tied to a zotero library; ideally they should a) be processable by
anyone with the right tools and b) completely human readable even
without the right tools. I am looking at using citeproc-node to
support generation of citations without Zotero.

best, Erik