Improving BibTeX export
spartan
http://forums.zotero.org/discussion/17847/improving-bibtex-export/
I love zotero so much. But got disturbed by the poor support of
bibtex. So I spent quite some time to improve the functionality of the
bibtex export. It should be very useful for those who use bibtex/latex
a lot, in particular for scientists like me. But I don't know where I
should post my modified codes.
Here is the brief list of improvements from my hard work:
1. much more flexible (user control) bibtex key generation: from
author names, initials, title, journal, year, volume, pages ,...etc
2. the suffix for key collisions can be numeric or alphabetic
3. key format strings can be read from prefs
(extention.zotero.bibtex....)
4. attachments like pdf files can be exported with real path links
only. now you can easily get the resulting bibtex file work with other
external applications like JabRef and Mendeley.
5. unicode conversions for greek letters
6. user specified field like callNumber to export pre-stored keys
... and other improvements.
It's been working pretty well for me. But I'd like to share it with
your guys.
Please drop me a note if somebody knows how to get these improvements
added to the next release of zotero.
spartan
Avram Lyon
On Thu, May 5, 2011 at 7:53 AM, spartan <spart...@gmail.com> wrote:
> I love zotero so much. But got disturbed by the poor support of
> bibtex. So I spent quite some time to improve the functionality of the
> bibtex export. It should be very useful for those who use bibtex/latex
> a lot, in particular for scientists like me. But I don't know where I
> should post my modified codes.
Please post the code to http://gist.github.com/ and post a link to the
list, so the other BibTeX devs can take a look.
Avram
spartan
Public Clone URL: git://gist.github.com/956623.git
The readme file is attached below for your convenience:
===============
For the better of the research world that enjoys Zotero, Latex/BibTeX,
JabRef, Mendeley, SciPlore MindMapping, etc.
updated: May 4, 2011
************************
* spart...@gmail.com *
************************
===============
List of files:
reame -- this file
bibtex_js_and_path.gz -- tarball of everything
BibTeXTan.js -- to replace the translator BibTeX.js
BibTeXTan.patch -- patch file relative to BibTeX.js of zotero v2.1.6
BibTeXKeyOnly.js -- new translator for quickcopy / drag&drop export of
cite keys
BibTeXTanKeyOnly.patch -- patch file relative to BibTeXTan.js
item_local.js -- to replace content/zotero/xpcom/translation/
item_local.js
item_local.patch -- patch file to that of zotero v2.1.6
translate.js -- to replace content/zotero/xpcom/translation/
translate.js
translate.patch -- patch file to that of zotero v2.1.6
===============
New Features and Improvements on BibTeX Export:
1. very flexible bibtex key format definition using the fields of
creators,
title, journal, year, volume, pages. including options of lower/upper
cases,
initials, lengths, n-th author/word picking, etc. it is also easy to
extend
to include new fields for key generation.
2. to resolve possible key collisions, users can specify their own
suffix
either alphabetic or numeric.
3. a special field (e.g., callNumber or shortTitle) can be dedicated
for export
of keys either pre-assigned or stored that are better done manually.
Until
zotero provides a dedicated BibTeX key field or some type of local id
field,
this is still a very appealing hack for most latex/bibtex users.
4. attachments like pdf files can be exported as file links only. The
local
real path is stored in the file field. No need to export the large
files
themselves. Keeping two copies on the machine is not good, in
particular
for people who like to make comments on pdf files. This is also making
zotero
cooperating better with other external applications like JabRef and
SciPlore.
The real file export still works as expected if the option of
exportFileData is
set to true. Only a small change in item_local.js is needed for such a
benefit.
5. unicode conversions for greek letters.
6. the strings of controlling the 1,2,3 features are easily modified
through
browser prefs: extensions.zotero.bibtexKeyFormat,
extensions.zotero.bibtexKeyCollisionFormat,
extensions.zotero.bibtexKeyField.
No need to touch the translator itself for user's own definition. No
formats
will be lost after upgrade. All of these only require a small
modification in
translate.js. It also opens up access to prefs for all the other
translators
which could be potentially very useful for others as well.
7. A Cite Key only stripped down translator is provided as well in a
similar way
that Andrew Leifer did. This is very convenient for quickCopy or
drag&drop in
latex editing.
8. some other minor improvements for bibtex export such as better
treatment for
latex special characters like $,\,_,^, etc.
===============
How Key Format String is defined?
var citeKeyFormat = Zotero.getPrefs("bibtexKeyFormat") ?
Zotero.getPrefs("bibtexKeyFormat") : "%au%yr%ti";
1. A general matching element is % followed by a two-letter label:
for
matching field and specifying the lower/upper case for 1st/remaining
letters.
2. Then it can be optionally followed by a number for the maximum
length in
number of characters. (default=0: means all characters)
3. Last it can be also optionally followed by a number within curly
brackets
for selecting the n-th word of the field (start from 0-th, the
default).
4. The special repitive 2-letter labels like %TT, %aa, and %JJ are to
group
the first letters only of surnames/words in corresponding fields.
5. Separators/indicators including +,-,:,., etc, and all letters and
numbers
can be used in between the elements.
Some examples:
%au4%yr2%tt2 %AU_%yr_%TI %Au4-%Au4{1}:%JJ%yr:Vol%vo:Pg%pg-%pg{1}
%Au4{1} = first 4 letters of 2nd creator's surname (first letter
capitalized)
%ti6{2} = first 6 letters of 3rd word in title (all lowercase)
%AA4 = initials of first 4 creators' surnames (uppercase)
%tt6 = first letters of first 6 words in title (lowercase)
%JJ5 = first letters of first 5 words in journal name (uppercase)
%yr = %yr4 = four-digit year, %yr2 = two-digit year
%vo = volume number
%pg = first page number
%pg{1} = last page number
===============
Collision Format: %a for alphabetic suffix, %n for numeric suffix
var citeKeyCollisionFormat =
Zotero.getPrefs("bibtexKeyCollisionFormat") ?
Zotero.getPrefs("bibtexKeyCollisionFormat") : "-%n";
===============
Specified Key Field: e.g. callNumber for using the stored key
var citeKeyField = Zotero.getPrefs("bibtexKeyField"); //e.g.
callNumber
Avram Lyon
wanted to point out the two changes to Zotero core code that you're
proposing:
1. Zotero.getPrefs(..), a new function in the sandbox that allows
access to arbitrary preferences from the Zotero section of the user's
prefs (patch to translate.js)
2. Expose the local path of attachments through the "path" attribute
on the attachment object (patch to item_local.js)
I'm a little concerned about the first one, since I don't see a
justification for exposing all the user's preferences to all
translators. This also has some implications for the future
portability of the translator to a server-side or in-connector
translator sandbox. Couldn't we just keep BibTeX key editing internal
to the file and have people edit a constant there?
The second proposed change seems quite reasonable.
- Avram
> --
> You received this message because you are subscribed to the Google Groups "zotero-dev" group.
> To post to this group, send email to zoter...@googlegroups.com.
> To unsubscribe from this group, send email to zotero-dev+...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/zotero-dev?hl=en.
>
>
Richard Karnesky
> creators,
> title, journal, year, volume, pages. including options of lower/upper
> cases,
> initials, lengths, n-th author/word picking, etc. it is also easy to
> extend
> to include new fields for key generation.
>
> 2. to resolve possible key collisions, users can specify their own
> suffix
> either alphabetic or numeric.
>
> 3. a special field (e.g., callNumber or shortTitle) can be dedicated
> for export
> of keys either pre-assigned or stored that are better done manually.
> Until
> zotero provides a dedicated BibTeX key field or some type of local id
> field,
> this is still a very appealing hack for most latex/bibtex users.
> 4. attachments like pdf files can be exported as file links only. The
> local
> real path is stored in the file field. No need to export the large
> files
> themselves. Keeping two copies on the machine is not good, in
> particular
> for people who like to make comments on pdf files. This is also making
> zotero
> cooperating better with other external applications like JabRef and
> SciPlore.
> The real file export still works as expected if the option of
> exportFileData is
> set to true. Only a small change in item_local.js is needed for such a
> benefit.
or SciPlore. Have you? What specific issues were you running into?
This implementation differs from the file export of other
translators. Current implementation also allows you to share both the
bibtex file and attachments with other bibtex users who use other
machines. I think exportFileData should be obeyed. I don't know if I
disagree with the option of exporting a file that has links to
zotero's copy of files, but I don't think this implementation is what
we want right now.
> 5. unicode conversions for greek letters.
have one-way conversions right now. A lot of users of greek use, at
minimum, babel. Many use inputenc. Export would work as-is for
that. (And most greek-specific or international-specific tools would
work even better.) Don't know the right answer for this: I'm not sure
of any other program that transliterates greek into tex entities. Do
you have examples?
> 8. some other minor improvements for bibtex export such as better
> treatment for
> latex special characters like $,\,_,^, etc.
Preferably with an example (containing input text, output of the two
BibTeX translators, and how LaTeX/BibTeX would mis-render them)?
Thanks again for the patch!
--Rick
Edouard
this :
if(item.patentNumber) {
writeField("note", item.patentNumber);
writeField("howpublished", "Patent");
}
As suggested here :http://see-out.com/sandramau/bibpat.html
What do you think ?
Edouard.
spartan
for point 1:
The reason for the new function Zotero.getPrefs() is to easily store
user's
setting somewhere. So they don't lose their setting from the next
upgrade.
Users really hate to remember their complicated settings in somewhere
else
like a sticky note and to manually edit the file (e.g. BibTeX.js in
this case)
when the next upgrade of zotero happens. If it can be done some other
way
like getOption() and the GUI, it would be fantastic. But I don't know
how to
achieve that.
If people are really concerned about exposing all zotero-related
preferences
to all translators. We can define a more specific function like,
"getBibtexPrefs":function(translate, pref) {
...
return Zotero.Prefs.get("bibtex"+pref);
}
In this case, only bibtex related preferences get exposed. What do you
think?
-spartan
Avram Lyon
On Thu, May 5, 2011 at 8:19 PM, spartan <spart...@gmail.com> wrote:
[..]
> If people are really concerned about exposing all zotero-related
> preferences
> to all translators. We can define a more specific function like,
>
> "getBibtexPrefs":function(translate, pref) {
> ...
> return Zotero.Prefs.get("bibtex"+pref);
> }
>
> In this case, only bibtex related preferences get exposed. What do you
> think?
I agree with your intent here, and I hope we can find a workable
solution. If the key generation settings are going to be in the Zotero
preferences, I think we should probably handle it as a
non-BibTeX-specific subtree of the preferences:
"getPref":function(translate, pref) {
...
return Zotero.Prefs.get("translator."+pref);
}
We can imagine more uses for such preferences for other translators as
well, although I think we'd have to be very careful about what uses we
allow in translators that ship with Zotero. The added freedom might
prove useful for internal-use translators being created for specific
users' or institutions' workflows.
I flagged this not because I disagree with it, but rather because
changes to the core client go through a different review process, and
I wanted to make sure that the core devs took a look at these specific
proposed changes and weighed in on them.
Avram
spartan
> > 4. attachments like pdf files can be exported as file links only. The
> > local
> > real path is stored in the file field. No need to export the large
> > files
> > themselves. Keeping two copies on the machine is not good, in
> > particular
> > for people who like to make comments on pdf files. This is also making
> > zotero
> > cooperating better with other external applications like JabRef and
> > SciPlore.
> > The real file export still works as expected if the option of
> > exportFileData is
> > set to true. Only a small change in item_local.js is needed for such a
> > benefit.
>
> File exports should already work with JabRef. Haven't tested Mendeley
> or SciPlore. Have you? What specific issues were you running into?
> This implementation differs from the file export of other
> translators. Current implementation also allows you to share both the
> bibtex file and attachments with other bibtex users who use other
> machines. I think exportFileData should be obeyed. I don't know if I
> disagree with the option of exporting a file that has links to
> zotero's copy of files, but I don't think this implementation is what
> we want right now.
as
expected. What I did was to have the native path of the file saved in
the 'file' field of bibtex when people set exportFileData = false. In
this
way when you play the generated bibtex file using some external
applications you will have access to the attachments that still reside
in the zotero dataDir. It is not a good idea in general to keep two
copies of attachment data. Especially when people do a lot of
comments on those pdf files. It will be hard to keep the copies synced
all the time. The real file copying function is not lost when you
set exportFileData = true. It is the same as before so you can still
share your data in the same way with friends,etc.
>
> > 5. unicode conversions for greek letters.
>
> These are in mathmode, which is probably not ideal. You also only
> have one-way conversions right now. A lot of users of greek use, at
> minimum, babel. Many use inputenc. Export would work as-is for
> that. (And most greek-specific or international-specific tools would
> work even better.) Don't know the right answer for this: I'm not sure
> of any other program that transliterates greek into tex entities. Do
> you have examples?
the corresponding websites using unicode. The unicode conversion tool
is already in the translator. What I did was to put additional lines
for
the very common greek letters used widely in science. for example,
the greek alpha -> $\alpha$, the capital greek GAMMA -> $\Gamma$.
The resulting bibtex file will be more compatible with latex.
>
> > 8. some other minor improvements for bibtex export such as better
> > treatment for
> > latex special characters like $,\,_,^, etc.
>
> Again: can you explain the issues you're trying to address?
> Preferably with an example (containing input text, output of the two
> BibTeX translators, and how LaTeX/BibTeX would mis-render them)?
escape
some very common latex special characters like $ and \. When these
letters are in the fields like title, abstract, you want to keep it as
it is.
for example, it is very common $\alpha$ appears in title. The current
BibTeX.js will convert it to \$\\alpha\$ which is unwanted. These
kinds
of things are better kept untouched. The one situation that we do need
to escape a $ sign is, for cases like $1,000.00 which is considered in
the new code.
spartan
> > "getBibtexPrefs":function(translate, pref) {
> > ...
> > return Zotero.Prefs.get("bibtex"+pref);
> > }
>
> > In this case, only bibtex related preferences get exposed. What do you
> > think?
>
> I agree with your intent here, and I hope we can find a workable
> solution. If the key generation settings are going to be in the Zotero
> preferences, I think we should probably handle it as a
> non-BibTeX-specific subtree of the preferences:
>
> "getPref":function(translate, pref) {
> ...
> return Zotero.Prefs.get("translator."+pref);
>
> }
>
> We can imagine more uses for such preferences for other translators as
> well, although I think we'd have to be very careful about what uses we
> allow in translators that ship with Zotero. The added freedom might
> prove useful for internal-use translators being created for specific
> users' or institutions' workflows.
>
> I flagged this not because I disagree with it, but rather because
> changes to the core client go through a different review process, and
> I wanted to make sure that the core devs took a look at these specific
> proposed changes and weighed in on them.
good way to store user settings.
A little bit off the topic, but...The lack of strong and better
support of bibtex/latex is the only thing that prevents a lot of
scientific researchers like me to fully embrace zotero. I hope that
the situation will get better and better. I guess that the latex users
are a pretty large group of zotero users. I see zotero posts related
to bibtex or latex everywhere. Unfortunately most of these guys
are so busy with their research projects. Not many of them can
devote their time to improve the bibtex support in zotero. I am not
a programmer but a normal busy researcher. I got so frustrated in
the situation. So I dived in and learned javascript myself and
started modifying the codes. I hope more researchers will join
in and make the bibtex support the best part of zotero.
-spartan
>
> Avram
Richard Karnesky
of the item types that originally documented in 'BibTexing', but it
is tremendously popular. JabRef & BibLaTeX use it, as do a ton
of .bst styles.
--Rick
Richard Karnesky
(1) don't export information about attachments
(2) copy attachments and export information about them
(3) export information about the local copies.
Current behavior is (1) or (2). Your proposed behavior is (2) or
(3). I'm not sure if this result is intuitive based on the "Export
Files" option. Perhaps, at minimum, this should be rephrased?
File export is not unique to BibTeX & behavior (3) does not happen in
other translators that can export files. Should it? Do people value
having option (1)? It has the advantage of privacy & also being able
to import the file onto another system without the brittle local
paths. Should we therefore allow any of these three choices?
Or perhaps BibTeX is special: it is more likely to be used by a
researcher on his own machine, as it is not only a data exchange
format, but a format that is used in document preparation.
As I said: this raises issues that should be discussed....
> > > 5. unicode conversions for greek letters.
in 'refbase', which I co-lead (though Matthias Steffens was the author
of that code). I've patched that section of code within Zotero.
After double-checking, I am only slight embarrassed to see that I
forgot that 'refbase' also exports greek characters in math mode. If
it is good enough for that, it is good enough for this :-). (But I'll
also note that Elsevier, Springer, AIP, and many others do NOT do
this).
HOWEVER: you only added entries to the mapping table & not to the
reverse table. This means import/export is not reversible. We should
add entries to the other table too.
> > > 8. some other minor improvements for bibtex export such as better
> > > treatment for
> > > latex special characters like $,\,_,^, etc.
users really want it. I've helped them implement something similar in
their personal copies of the translator. But it runs with a very
different philosophy than Zotero is made with.
For the clarity of others: the proposed change effects export by
allowing BibTeX users to use BibTeX-specific markup that will be
preserved on export. Encouraging this markup would break sharing of
the references with anything that is not BibTeX, as people would write
LaTeX entities into zotero fields, rather than using the what-you-say-
is-what-you-get-everywhere UTF-8 equivalents.
Yes, if you write 'α' in Zotero, you should get '$\alpha$' in BibTeX.
Patching the mapping tables would get you that. However, people
should not write '$\alpha$' directly in a Zotero field! While
accepting this change would make Zotero more expressive to LaTeX/
BibTeX users, it wouldn't help people who use e.g. the word processor
plugins or html output or other ways of using references. It'd make
references less shareable.
The right way to fix this is to address:
https://www.zotero.org/trac/ticket/439
See:
http://forums.zotero.org/discussion/5324/bibtex-and-greek-characters/
and many others for philosophical discussion.
--Rick
Bruce D'Arcus
>> > > 8. some other minor improvements for bibtex export such as better
>> > > treatment for
>> > > latex special characters like $,\,_,^, etc.
>
> I am opposed to this change. I do realize that some other BibTeX
> users really want it. I've helped them implement something similar in
> their personal copies of the translator. But it runs with a very
> different philosophy than Zotero is made with.
>
> For the clarity of others: the proposed change effects export by
> allowing BibTeX users to use BibTeX-specific markup that will be
> preserved on export. Encouraging this markup would break sharing of
> the references with anything that is not BibTeX, as people would write
> LaTeX entities into zotero fields, rather than using the what-you-say-
> is-what-you-get-everywhere UTF-8 equivalents.
Yes, definitely agree with Rick's position here, and strongly against
in any way treating bibtex as special for purposes of data entry (use
UTF8) or key/label generation (which is relevant for a variety of
formats).
Bruce
spartan
On May 5, 2:47 pm, Richard Karnesky <karne...@gmail.com> wrote:
> > > > 4. attachments like pdf files can be exported as file links only.
>
> As I see it, there are three possibilities:
> (1) don't export information about attachments
> (2) copy attachments and export information about them
> (3) export information about the local copies.
>
> Current behavior is (1) or (2). Your proposed behavior is (2) or
> (3). I'm not sure if this result is intuitive based on the "Export
> Files" option. Perhaps, at minimum, this should be rephrased?
>
> File export is not unique to BibTeX & behavior (3) does not happen in
> other translators that can export files. Should it? Do people value
> having option (1)? It has the advantage of privacy & also being able
> to import the file onto another system without the brittle local
> paths. Should we therefore allow any of these three choices?
>
> Or perhaps BibTeX is special: it is more likely to be used by a
> researcher on his own machine, as it is not only a data exchange
> format, but a format that is used in document preparation.
>
> As I said: this raises issues that should be discussed....
can put it in a user setting like bibtexAttachmentsLocalPath in the
preferences. Maybe even set it to false as default. Then users will
have
control if they want to export the local path information or not.
> > > > 5. unicode conversions for greek letters.
>
> Let me note that the Unicode<->LaTeX entity transliteration originated
> in 'refbase', which I co-lead (though Matthias Steffens was the author
> of that code). I've patched that section of code within Zotero.
> After double-checking, I am only slight embarrassed to see that I
> forgot that 'refbase' also exports greek characters in math mode. If
> it is good enough for that, it is good enough for this :-). (But I'll
> also note that Elsevier, Springer, AIP, and many others do NOT do
> this).
>
> HOWEVER: you only added entries to the mapping table & not to the
> reverse table. This means import/export is not reversible. We should
> add entries to the other table too.
>
> > > > 8. some other minor improvements for bibtex export such as better
> > > > treatment for
> > > > latex special characters like $,\,_,^, etc.
>
> I am opposed to this change. I do realize that some other BibTeX
> users really want it. I've helped them implement something similar in
> their personal copies of the translator. But it runs with a very
> different philosophy than Zotero is made with.
>
> For the clarity of others: the proposed change effects export by
> allowing BibTeX users to use BibTeX-specific markup that will be
> preserved on export. Encouraging this markup would break sharing of
> the references with anything that is not BibTeX, as people would write
> LaTeX entities into zotero fields, rather than using the what-you-say-
> is-what-you-get-everywhere UTF-8 equivalents.
>
> should not write '$\alpha$' directly in a Zotero field! While
> accepting this change would make Zotero more expressive to LaTeX/
> BibTeX users, it wouldn't help people who use e.g. the word processor
> plugins or html output or other ways of using references. It'd make
> references less shareable.
in zotero community. Please forgive me if I did not use more polite
words here or there because I am not a native english speaker.
Physical sciences all have long history for research and literature.
Not all of the documents are in modern or fancy formats. But so
much of the information is good enough and found its way getting
into my reference database and I am sure, also, a lot of other
scientists'. Are we asking all the scientific researchers who are
using zotero and bibtex to manually modify their databases to
conform "zotero's standard"? Those latex-specific
markups are scattered everywhere in these people's databases.
It would be a hack of job for individual users to convert those to
other markups appealing to office word users. And the info was
not created by the actual users but their ancestors. However,
zotero wants to punish these users because they include such
information in their databases. That is ironic even for the sake
of demoting the use of latex markups in zotero.
On the other hand, users of the bibtex translator probably use
latex editors more than office word. They do not need their
databases to look appealing to other guys. When they do
share their information, they share it most likely with people
of similar minds. Is there any reason that a office word user
would use the bibtex exporter? I guess not. So why don't
we just let the bibtex exporter do what is best for most of
the latex/bibtex users?
Somehow the spirit around zotero is not right. It should be
more for a cooperative, compatible, user-friendly one. It will
be really disappointing if it becomes a policing tool to
tell people what to use and what not.
Please forgive me if my words offend some people.
-spartan
> The right way to fix this is to address:
> https://www.zotero.org/trac/ticket/439
>
> See:
> http://forums.zotero.org/discussion/5324/bibtex-and-greek-characters/
> and many others for philosophical discussion.
>
> --Rick
Bruce D'Arcus
...
> Somehow I am feeling a little too much hatred towards bibtex/latex
> in zotero community. Please forgive me if I did not use more polite
> words here or there because I am not a native english speaker.
I have no "hatred" towards that community.
I do get annoyed when people present it as the only significant game
in town though, and don't consider the wider world that Zotero is
attempting to be compatible with (a variety of other formats, the web,
word-processors, etc.).
> Physical sciences all have long history for research and literature.
> Not all of the documents are in modern or fancy formats. But so
> much of the information is good enough and found its way getting
> into my reference database and I am sure, also, a lot of other
> scientists'. Are we asking all the scientific researchers who are
> using zotero and bibtex to manually modify their databases to
> conform "zotero's standard"? Those latex-specific
> markups are scattered everywhere in these people's databases.
So use a BibTeX-dedicated solution?
I have no problem if, for example, Zotero were to add math support in
ways that would allow round-tripping to LaTeX. But that's a different
matter.
Rick has more direct experience working with BibTeX, though, as well
as understanding of Zotero, so I'd tend to defer to him on the
details.
[...]
> Somehow the spirit around zotero is not right. It should be
> more for a cooperative, compatible, user-friendly one. It will
> be really disappointing if it becomes a policing tool to
> tell people what to use and what not.
It really depends on where you stand: "cooperative, compatible,
user-friendly" for whom, after all? Just BibTeX users?
Bruce
adamsmith
there is certainly no hostility towards bibtex/latex: Rick, for
example, is a physicist and, I believe, authors mainly in LaTeX. I
can't weigh in on the details, but on a broader note, there is a very
strong commitment in the open source community to standards: Standards
facilitate data sharing, they facilitate interoperability, they
facilitate upgrades, they help dealing with unforseen new use cases
etc. There is a good reason that people strongly defend them - that
doesn't mean they don't want LaTex suitable solutions, they just don't
want them as ideosyncratic (quasi-) hacks.
spartan
I am sorry if I used some harsh words. But frankly I don't see how the
current practice in zotero will help promote standard. Say, in a
reference item, $\alpha$ got embedded in the title. In the current
BibTeX.js, all the special characters get intentionally escaped. So in
the exported bibtex file, the title will include garbage like \$\\alpha
\$
which can not be correctly treated in any bibtex/latex editor. It is
not useful for any word application either. If this piece of ill
formated
information got shared around, for example, imported back to another
zotero database, it will not get corrected. If it got exported again,
it
will become \\\$\\\\alpha\\\$. It could grow into a monster garbage if
such things go into cycle and cycle again. No sane importer/exporter
later can really correct this problem easily. I don't see how such a
piece of junk is useful for any other users (e.g., who use office
word).
So I view this practice as a pure punishment to whoever use this
piece of reference and the guys we shared it with.
On the contrary, things will be dramatically different if we do the
export in the correct way. Let's use the above example again. The
markup piece $\alpha$ should go untouched into the resulting
bibtex file for a sensible exporter. Any bibtex/latex applications
will deal with the file nicely. If later on the item got imported back
into the same or another database through user-reorganizing or
sharing with colleagues, the good things happen: the bibtex-
standard markup piece $\alpha$ will be converted into a correct
unicode character of the greek alpha. Suddenly everything
starts to follow zotero standard. No matter how many times it
got imported/exported either via bibtex or any other format, the
information will be correctly interpreted. Not only are the latex
users happy, but also are other users who have to import the
same item. See, gradually we can convert old non-standard
markups circulating in many databases into standard-obeying
new format. You can not achieve this using the former
punishing practice.
-spartan
spartan
> I do get annoyed when people present it as the only significant game
> in town though, and don't consider the wider world that Zotero is
> attempting to be compatible with (a variety of other formats, the web,
> word-processors, etc.).
in
the last post, the current practice does not make the existing
databases more compatible with other formats. So why punish
the latex users without real benefits for other formats?
>
> > Physical sciences all have long history for research and literature.
> > Not all of the documents are in modern or fancy formats. But so
> > much of the information is good enough and found its way getting
> > into my reference database and I am sure, also, a lot of other
> > scientists'. Are we asking all the scientific researchers who are
> > using zotero and bibtex to manually modify their databases to
> > conform "zotero's standard"? Those latex-specific
> > markups are scattered everywhere in these people's databases.
>
> So use a BibTeX-dedicated solution?
zotero
in collecting information, I seriously will go for it. I don't know
any.
That is why I love zotero but got frustrated with its bibtex support.
I just learned javascript myself and spent a lot of my spare time
implementing all these changes.
>
> I have no problem if, for example, Zotero were to add math support in
> ways that would allow round-tripping to LaTeX. But that's a different
> matter.
>
> Rick has more direct experience working with BibTeX, though, as well
> as understanding of Zotero, so I'd tend to defer to him on the
> details.
>
> [...]
>
> > Somehow the spirit around zotero is not right. It should be
> > more for a cooperative, compatible, user-friendly one. It will
> > be really disappointing if it becomes a policing tool to
> > tell people what to use and what not.
>
> It really depends on where you stand: "cooperative, compatible,
> user-friendly" for whom, after all? Just BibTeX users?
BibTeX.js, aren't we? Why can't we make it follow bibtex
rules. It is not that I am asking for the changes in other format
translators to favor bibtex markups.
>
> Bruce
Avram Lyon
> I have no objections to the support for other formats. As I mentioned
> in
> the last post, the current practice does not make the existing
> databases more compatible with other formats. So why punish
> the latex users without real benefits for other formats?
>> It really depends on where you stand: "cooperative, compatible,
>> user-friendly" for whom, after all? Just BibTeX users?
>
> This really puzzles me. We are talking about modifying
> BibTeX.js, aren't we? Why can't we make it follow bibtex
> rules. It is not that I am asking for the changes in other format
> translators to favor bibtex markups.
The concern is that decisions like handling of LaTeX's math-mode
delimiters or character entities have pretty major consequences for
data exchange. Upon import, $\alpha$ should definitely be converted to
the Greek letter in its Unicode representation. Similarly,
superscripted and subscripted characters and many other symbols have
Unicode representations, and we can and should make the proper
conversions to convert to them when importing BibTeX files. This is
particularly important because BibTeX is a common data interchange
format-- we have site translators like the Google Scholar translator
that rely on BibTeX, and many people import large quantities of items
using the format. These may be people who never actually use
BibTeX/LaTeX for bibliographies and documents, or they could be people
who intend to export back to BibTeX at a later date.
Most of the (La)TeX markup that occurs in bibliographic data is
limited to individual symbols -- these can be stored as Unicode within
Zotero, and imported and exported correctly. Full-fledged math support
is only rarely needed, particularly outside of abstracts. To enter the
individual symbols from within Zotero, the current system requires
that users enter them as Unicode-- probably by using the system
character palette. While this is probably not terribly convenient,
it's nothing that couldn't be streamlined by a small plugin to convert
(La)TeX names into the correct Unicode symbols.
If we set up a situation where the recommended workflow is to mark up
bibliographic data in (La)TeX ways, users who take this approach will
lose the full flexibility of Zotero and may no longer be able to use
their data in any non-BibTeX context. No export to RIS, (possibly) no
OpenURL locator lookups, no use of the server API, particularly its
formatted citations, no use of the local integration plugins, from
word processor plugins to emacs org-mode integration.
All that said, I do think we could do better in our handling of
math-mode upon export to BibTeX. The arguments above do not imply that
we actively do not support handling what the user clearly intends to
do-- if we can devise a handling for $...$ that respects user
intentions on export, and doesn't escape the lot of it, but still
doesn't risk biting users who just happen to use dollar signs a lot,
then I think we should put it into the BibTeX export function.
Regards,
Avram
Dan Stillman
> Say, in a reference item, $\alpha$ got embedded in the title.
The rest of your argument depends on this assumption, but $\alpha$ is
"garbage" to all non-LaTeX-using members of the Zotero ecosystem. It
shouldn't be there to begin with, and it's a bug if it is. Import
translators should be fixed to ensure that it doesn't end up in a Zotero
field. Batch editing should allow people to fix that data.
Our philosophy on this is pretty well documented by this point, in this
thread and elsewhere. You may feel that BibTeX-using Zotero users "do
not need their databases to look appealing to other guys", but that's
contrary to Zotero's goals. This isn't a "punitive" policy, and it's in
no way specific to BibTeX. Adding support to export translators for
format-specific markup in Zotero fields encourages the entry of bad data
in Zotero fields.
(And, really, it's a little silly to keep suggesting that properly
rendered characters are something that only MS Word users are interested
in.)
spartan
Thanks for your elaboration. But I felt that you may get my point
better if you read my reply to adam smith earlier.
Your and other devs' concerns are that support of proper bibtex
markup export will promote the more convenient non-standard use
of bibtex markups. My point is that it is more likely the opposite.
At least for me, I use zotero because it is a really powerful tool
to gather reference data automatically from other places
especially the web. I seldom manually input anything in the
standard fields like title, abstract. I doubt people do manual input
in these fields a lot. People would go to other reference manager
if they enjoy doing these kind of manual input. The only field that
I do manual input and I imagine others probably do as well is the
note which should not affect the discussion here. The bibtex
markups appeared in many people's zotero databases are
most probably NOT coming from these individual end users. They
came from various web resources either current or in old or
ancient times which most likely have nothing to do with zotero.
So why punish the innocent end users for 'crimes' others commit?
If we do as I proposed, an export and then an import will
automatically convert non-standard databases back to conform
the unicode standard. The current practice will keep those
non-standard data for ever unless the end users suddenly
become diligent to manually correct them all.
-spartan
> The concern is that decisions like handling of LaTeX's math-mode
> delimiters or character entities have pretty major consequences for
> data exchange. Upon import, $\alpha$ should definitely be converted to
> the Greek letter in its Unicode representation. Similarly,
> superscripted and subscripted characters and many other symbols have
> Unicode representations, and we can and should make the proper
> conversions to convert to them when importing BibTeX files. This is
> particularly important because BibTeX is a common data interchange
> format-- we have site translators like the Google Scholar translator
> that rely on BibTeX, and many people import large quantities of items
> using the format. These may be people who never actually use
> BibTeX/LaTeX for bibliographies and documents, or they could be people
> who intend to export back to BibTeX at a later date.
you
can correct it to the proper greek letter? When these non-latex users
import the bib file, they'll get the junk as well into their
databases.
>
> Most of the (La)TeX markup that occurs in bibliographic data is
> limited to individual symbols -- these can be stored as Unicode within
> Zotero, and imported and exported correctly. Full-fledged math support
> is only rarely needed, particularly outside of abstracts. To enter the
> individual symbols from within Zotero, the current system requires
> that users enter them as Unicode-- probably by using the system
> character palette. While this is probably not terribly convenient,
> it's nothing that couldn't be streamlined by a small plugin to convert
> (La)TeX names into the correct Unicode symbols.
To me that is really defeating the purpose of using zotero.
>
> If we set up a situation where the recommended workflow is to mark up
> bibliographic data in (La)TeX ways, users who take this approach will
> lose the full flexibility of Zotero and may no longer be able to use
> their data in any non-BibTeX context. No export to RIS, (possibly) no
> OpenURL locator lookups, no use of the server API, particularly its
> formatted citations, no use of the local integration plugins, from
> word processor plugins to emacs org-mode integration.
standard one and ready for all the uses you mentioned. But currently
the non-standard data will stay there until manual corrections which
can be a huge task.
As I said, zotero users very unlikely input the title, abstract etc
manually. We don't use it to write new papers, do we? We use it
to gather published information from various places automatically.
People would abandon zotero long ago if they had to do such input.
On the other hand, I don't see zotero's intentional demotion of
bibtex will force all the websites to abandon latex markups.
I don't see either that a proper export of latex markups will make
all the websites favor latex markups.
Personally, I think it is also good for other translators to preserve
their own markups ( I have no knowledge of it) when they do
exporting. Just like the case if we translate an English book or
article that contain a little bit Russian phases to Russian, we
keep the original existing Russian phase as it is. It is hard to
imagine that we trash those Russian phases in translation due
to the non-standard in an English book. If the whole world starts
to speak in Russian because of that, then so be it.
(Just use Russian as an example, I have great respect for Russian)
spartan
On May 6, 1:58 am, Dan Stillman <dstill...@zotero.org> wrote:
> On 5/5/11 11:04 PM, spartan wrote:
>
> > Say, in a reference item, $\alpha$ got embedded in the title.
>
> The rest of your argument depends on this assumption, but $\alpha$ is
> "garbage" to all non-LaTeX-using members of the Zotero ecosystem. It
> shouldn't be there to begin with, and it's a bug if it is. Import
> translators should be fixed to ensure that it doesn't end up in a Zotero
> field. Batch editing should allow people to fix that data.
you
and me but from various websites in the histrory. You don't want
people
to throw their databases away because of bugs inside. A current
importer
can not fix things already in the databases. My suggested approach
actually can fix the bugs by an export and an import.
>
> Our philosophy on this is pretty well documented by this point, in this
> thread and elsewhere. You may feel that BibTeX-using Zotero users "do
> not need their databases to look appealing to other guys", but that's
> contrary to Zotero's goals. This isn't a "punitive" policy, and it's in
> no way specific to BibTeX. Adding support to export translators for
> format-specific markup in Zotero fields encourages the entry of bad data
> in Zotero fields.
use
zotero? Or are you saying that zotero's practice here really have big
impact on many websites? The only encouragement I can see is for
a casual individual user if he/she does do a lot of manual input.
Which
is really not a big deal since zotero is not targeted for such people.
Frank Bennett
It seems that everyone is in favor of normalized source, on both sides
of a Zotero/BibTeX (or any other) conversion. Your correspondents in
this discussion are only opposed to the implicit normalization of text
by the tools that move data across the Zotero/BibTeX divide.
Personally, I would have to agree, based on some recent personal
experiences with the Zotero citation formatter (of which I am the
perpetrator in chief). Users do all sorts of strange and wonderful
things in their data, and records that work swimmingly in one context
(say, when talking to BibTeX) can cause consternation and havoc in
others. So all Dan is suggesting is that data coming into Zotero
(including "historical" BibTeX records) should be normalized at the
entry gate, where (relatively) safe assumptions about the intention
behind various patterns in the text. The emphasis differs from your
own initial position, but the aim is the same, and there is a
carefully considered logic behind taking that approach. (It's
certainly not a strategy for demolishing anything or punishing anyone
-- that's outside the scope of the Zotero mission statement :)
Frank
Olivier Cailloux
Many thanks to spartan for his improvements to the bibtex exporter which I look forward to see integrated! (Naturally, after reaching a consensus in the discussion with the main zotero devs.) Thanks also for this thread that raises some interesting questions.
I am a zotero user and bibtex user, bibtex being my primary source (almost: only source) for actually using the zotero data when I write papers.
Although I would be very happy to see better bibtex support in zotero, I do agree with the zotero team that general support of bibtex-formatted entries in the zotero db such as $\alpha$ is not the way to go, at least on the long term. However, we do have a problem (as mentioned already by others) in that zotero does not accept maths in its fields. E.g. $\frac{a}{b}$. We can dream of a very good and general solution for maths support in zotero, but AFAIK that's not going to happen anytime soon. In the meantime, let us try to think about an acceptable solution for allowing minimal maths support in zotero.
My suggestion (out of the top of my head so may be stupid): use <latex> </latex> markups. On bibtex import, $..$ would be changed to <latex>..</latex> when there is no alternative (such as for $\frac{a}{b}$). On export, changed back. For better compatibility with other formats, bibtex import should preferably be transformed into pure Unicode than into <latex> code when possible (such as for $\alpha$).
I agree that these <latex> tags would currently be bibtex specific and it's not very satisfactory compared to the general zotero goals. However, there is no theoretical reason other zotero exporters could not support them, e.g. someone could write latex-to-html transform exporter for zotero (I guess such tools already exist for other softwares), latex-to-word, etc.
This proposal, although imperfect, seems to me more satisfactory than the current situation, where $\frac{a}{b}$ gets transformed into something that nobody is happy with, be them HTML, Word, or bibtex format users.
Just my 2 cents.
Olivier
spartan
"Non-Doing". But it actually intentionally implemented a piece
of code to make the exported bib file even messier. The current
entrance normalization procedure (i.e. import) is far from
complete. So often, latex or semi-latex like markups easily
go inside our databases. Instead of export it as it is so we can
find better solution to normalized it through re-import later, the
current implementation makes it irrecoverable.
Just like, say, the US is too crowded now so the government
wants to promote the use of trains for transportation. It will be
a reasonable measure if they build better railroads and make
more trains, etc. But it will be ourageous if they start to dig
holes on the normal roads to prevent people from driving their
cars. Sadly such kind of policing behavior is what zotero
treats bibtex now.
spartan
On May 6, 10:50 am, Olivier Cailloux <olivier.caill...@ecp.fr> wrote:
> Hello,
> Many thanks to spartan for his improvements to the bibtex exporter which I look forward to see integrated! (Naturally, after reaching a consensus in the discussion with the main zotero devs.) Thanks also for this thread that raises some interesting questions.
participate in the discussions. Without our users' own push, nothing
will change the bibtex situation in zotero.
> I am a zotero user and bibtex user, bibtex being my primary source (almost: only source) for actually using the zotero data when I write papers.
> Although I would be very happy to see better bibtex support in zotero, I do agree with the zotero team that general support of bibtex-formatted entries in the zotero db such as $\alpha$ is not the way to go, at least on the long term. However, we do have a problem (as mentioned already by others) in that zotero does not accept maths in its fields. E.g. $\frac{a}{b}$. We can dream of a very good and general solution for maths support in zotero, but AFAIK that's not going to happen anytime soon. In the meantime, let us try to think about an acceptable solution for allowing minimal maths support in zotero.
markups are good or not in zotero entries. It is how we deal with the
markups that are already in or may penetrate in the data. The current
practice is using a special escaping implementation in BibTeX.js to
make the resulting bib file unusable for any user at all. The devs'
stand is hoping that this will prevent normal users from abusing these
markups in their data entries. My argument is that this is an unfair
policing procedure and removing it actually make it possible get back
to the databases in standard format so we don't lose these valueable
entries.
> My suggestion (out of the top of my head so may be stupid): use <latex> </latex> markups. On bibtex import, $..$ would be changed to <latex>..</latex>when there is no alternative(such as for $\frac{a}{b}$). On export, changed back. For better compatibility with other formats, bibtex import should preferably be transformed into pure Unicode than into <latex> code when possible (such as for $\alpha$).
well.
> This proposal, although imperfect, seems to me more satisfactory than the current situation, where $\frac{a}{b}$ gets transformed into something thatnobodyis happy with, be them HTML, Word, or bibtex format users.
Yeah, the current implementation is just trashing those non-standard
entries even when users explicitly want to export it in bib format. It
does not really make any users happy include the one using HTML word
etc.
Bruce D'Arcus
> I could understand it if zotero's stand on this issue is based on
> "Non-Doing". But it actually intentionally implemented a piece
> of code to make the exported bib file even messier. The current
> entrance normalization procedure (i.e. import) is far from
> complete. So often, latex or semi-latex like markups easily
> go inside our databases. Instead of export it as it is so we can
> find better solution to normalized it through re-import later, the
> current implementation makes it irrecoverable.
>
> Just like, say, the US is too crowded now so the government
> wants to promote the use of trains for transportation. It will be
> a reasonable measure if they build better railroads and make
> more trains, etc. But it will be ourageous if they start to dig
> holes on the normal roads to prevent people from driving their
> cars. Sadly such kind of policing behavior is what zotero
> treats bibtex now.
This thread is getting off the rails, and I suggest you tone down the
rhetoric. It serves no useful purpose in advancing your aims. Frank
tried to explain the issues in a polite way, and this is how you
respond?
You understand the requirements outlined by various people in this
thread (basically, to normalize text content around UTF8, and HTML),
so how about working on a concrete proposal to improve things based on
those requirements, rather than complaining with overheated rhetoric?
If you can't do that, then I suggest you drop it.
And many of what you seem to derisively call "devs" in this thread are
users too.
Bruce
Avram Lyon
BibTeX import needs to be improved, so that common *TeX conventions
are correctly converted into their proper equivalents in the wider
world-- the $\alpha$ example is a good one. No one is opposed to this.
Such an improvement will allow these symbols to be used cleanly within
Zotero and also to round-trip back to BibTeX.
The contentious behavior is not on import, but on export. When
exporting to BibTeX, Zotero currently makes the assumption that the
user wants the present field contents to not be interpreted as marked
up using *TeX. Therefore, Zotero escapes backslashes, dollar signs,
etc. There is little doubt that some such escaping is necessary in any
potential BibTeX translator.
The controversial part is that Zotero provides no special handling for
field contents that is already marked up with *TeX, and it escapes it
as with everything else. This is actually a reasonable practice, but
it's certainly a pain for anyone who is trying to preserve the markup
while still keeping items in Zotero.
I think there is room for a BibTeX translator that supports both
escaping- and non-escaping export, with the behavior controlled by a
preference exposed as spartan suggests, or by a variable set in the
translator.
The default behavior, however, and perhaps the only behavior possible
without editing the translator (if only to uncomment a line, say),
should be escaping output. The tenet of data portability matters.
There are also proposed changes to the attachment path exposure, which
have not been properly discussed, mainly because of the separate
dispute over Zotero's escaping procedures on export to BibTeX.
In future messages, spartan, please keep in mind that you are dealing
with a number of people with extensive experience with BibTeX and a
wide variety of systems-- I certainly cut my teeth on LaTeX and
understand the appeal of its expressiveness. The members of this list
are all very dedicated to making researchers' lives (in many cases
their own) easier. And that's about all-- no hidden agenda, no
discrimination.
Avram
Olivier Cailloux
proposed patch to bibtex export melt. Even discarding some of the
changes because of the disagreement, there are interesting added
functionalities. I hope someone from zotero is still willing to look
into it and spartan is still willing to (marginally) commit suggested
changes to make the patch integrate the code base.
2) Let me insist on my proposal and elaborate it a bit further. I know
it does not address exactly the discussion between spartan and others,
but this debate is a good chance to think more generally about a better
bibtex life into zotero. And I'd even go as far as saying that my
suggestion might solve the problem that raised the disagreement of
spartan with the zotero team. I hope this is not considered as thread
hijacking as it is really related to the current discussion.
Here is a suggestion to better integrate bibtex support (and more
generally latex support, or math writing support), into zotero.
A) On bibtex import into zotero. If a pair of $ signs is detected,
either 1) (preferably) transform the content into pure Unicode text if
possible. Example: $\alpha$ is transformed into "α", unicode alpha. 2)
(if no text alternative) transform the content into
<latex>content</latex> markup. Example: $\frac{a}{b}$ would be
transformed into <latex>\frac{a}{b}</latex>.
B) On bibtex export from zotero. Obviously, change back the <latex> and
</latex> tags into dollar signs. Quote dollar signs from zotero (because
these must be considered as real dollar signs).
C) (For more general support of maths.) Transform other translators to
account for these <latex> tags. E.g. transforming latex content into
HTML should be possible by re-using some latex-to-html program.
D) Problem of the currently existing entries that have latex code
enclosed into dollar signs (e.g. non-transformed entries imported from
bibtex). A plugin to zotero (or other appropriate mechanism as suggested
by the zotero team) should be written in order to batch-transform these
entries. Problem: not every $ pairs must be transformed into <latex>
tags. Probably ask the user case-by-case for confirmation. It should
thus be possible to run the batch on a subset of entries to permit
correcting only the specific subset of entries the user is interested in
from a large db.
E) Problem of people wanting to use the strings "<latex>" and "</latex>"
in an entry without giving them a specific meaning. Does that exist?
Advantages of the proposal over the current situation:
1) The handling of bibtex imported entries containing latex code is
better for bibtex users, not worst for other ones (in the current
situation they are anyway unhappy with this unexploited and
unexploitable — let us pretend that word exists — latex code inside
their database).
2) Opens a path for better handling of bibtex imported entries, and more
generally of fields containing math, through the writing of appropriate
exporters, even for non bibtex users.
3) The zotero database becomes better because the latex code is clearly
recognizable a) by users, visually (easier to spot problems after
import); b) by automatic tools: easier to batch-change these entries if
e.g. the user wants to get rid of them, if zotero team comes up with a
bright idea on how to deal with mathematics, if writing a dedicated GUI,
etc.
Advantages of the proposal over spartan's proposal:
1) The user may use dollar signs where he wants in zotero to really mean
"dollar sign" and assume these will be exported accordingly to bibtex or
any format (i.e. as a dollar sign rendered in the resulting document,
thus as a quoted dollar sign in the bibtex source file).
2) Latex code is clearly recognizable from zotero (see point 3 here above).
Problems linked to the proposal:
1) ... well, someone has to implement it.
2) ... and I won’t do it.
... But at least having the specs and trying to agree on it makes us
move forward, I feel (when someone has time to implement something for
better bibtex support, he then has some hints about how to do it).
3) A simple and very concrete proposal to spartan (provided zotero team
agrees).
As suggested by I think Avram: spartan, add a user preference that asks
for dollar signs to be exported to bibtex as-is. The preference is by
default to export quoted dollar signs. Add a simple check into your
proposed patch to ensure the user really wants to export dollar signs to
bibtex as-is. Then I think everybody is happy.
Comments?
Olivier
spartan
two seem to be the most knowledgeable as well. I really appreciate
your feedbacks. I apologize if my words offend people here. But I was
just conveying my feel as a user that shared by many other latex users
in this community. From now on, I'll keep myself focused on the real
topic: how should we treat latex markups on BibTeX export?
On May 6, 11:41 am, Avram Lyon <ajl...@gmail.com> wrote:
> Let me be perfectly clear-- there is no policing going on. The current
> BibTeX import needs to be improved, so that common *TeX conventions
> are correctly converted into their proper equivalents in the wider
> world-- the $\alpha$ example is a good one. No one is opposed to this.
> Such an improvement will allow these symbols to be used cleanly within
> Zotero and also to round-trip back to BibTeX.
to a standard format in zotero. But keep in mind, the current import
mechanisms are far from perfect. Not only the latex markups but also
others like html tags are still very easily getting into people's
databases.
>
> The contentious behavior is not on import, but on export. When
> exporting to BibTeX, Zotero currently makes the assumption that the
> user wants the present field contents to not be interpreted as marked
> up using *TeX. Therefore, Zotero escapes backslashes, dollar signs,
> etc. There is little doubt that some such escaping is necessary in any
> potential BibTeX translator.
will see it way more often in latex markups than any other uses in a
field like title and abstract. One of the sensible use of $ sign as in
$1,000.00 has already been taken into account in my code. The change
is just one single line.
This is even more damaging when you consider the current
implementation is on bibtex export. I guess, most of people using
bibtex export are either to use it with other bibtex applications or
to prepare latex papers. I could not imagine any other format (html,
word) lovers will do bibtex export a lot.
I am not objecting to some escaping. In fact my implementation keeps
certain escaping. But my point is escaping all is much worse than
escaping nothing.
>
> The controversial part is that Zotero provides no special handling for
> field contents that is already marked up with *TeX, and it escapes it
> as with everything else. This is actually a reasonable practice, but
> it's certainly a pain for anyone who is trying to preserve the markup
> while still keeping items in Zotero.
from day one when they were born, then this is not really a pain at
all for anybody. But the reality hits. In bibtex export, it does bad
for much more bibtex export users than good.
>
> I think there is room for a BibTeX translator that supports both
> escaping- and non-escaping export, with the behavior controlled by a
> preference exposed as spartan suggests, or by a variable set in the
> translator.
hope the other devs listen to this. I am glad that you are not closing
the door to the possiblity of some non-escaping export. Thanks.
>
> The default behavior, however, and perhaps the only behavior possible
> without editing the translator (if only to uncomment a line, say),
> should be escaping output. The tenet of data portability matters.
will endanger data portability. A proper export will at least make the
existing data usable for latex users and possibly re-import it back in
standard zotero format. The blind escaping basically mess up the
bibtex export making it unusable to anybody.
>
> There are also proposed changes to the attachment path exposure, which
> have not been properly discussed, mainly because of the separate
> dispute over Zotero's escaping procedures on export to BibTeX.
>
> In future messages, spartan, please keep in mind that you are dealing
> with a number of people with extensive experience with BibTeX and a
> wide variety of systems-- I certainly cut my teeth on LaTeX and
> understand the appeal of its expressiveness. The members of this list
> are all very dedicated to making researchers' lives (in many cases
> their own) easier. And that's about all-- no hidden agenda, no
> discrimination.
for your hard work to make zotero such a wonderful piece of tool. As
an outsider, my humbe opinions may provide help in a different
perspective.
>
> Avram
>
Bruce D'Arcus
For LaTeX-friendly math display on the web, MathJax seems to be a popular option these days. You might want to look into whether that might provide a path forward?
spartan
On May 6, 2:24 pm, Olivier Cailloux <olivier.caill...@ecp.fr> wrote:
> 1) I hope that the heat generated in this thread will not make the
> proposed patch to bibtex export melt. Even discarding some of the
> changes because of the disagreement, there are interesting added
> functionalities. I hope someone from zotero is still willing to look
> into it and spartan is still willing to (marginally) commit suggested
> changes to make the patch integrate the code base.
>
> 2) Let me insist on my proposal and elaborate it a bit further. I know
> it does not address exactly the discussion between spartan and others,
> but this debate is a good chance to think more generally about a better
> bibtex life into zotero. And I'd even go as far as saying that my
> suggestion might solve the problem that raised the disagreement of
> spartan with the zotero team. I hope this is not considered as thread
> hijacking as it is really related to the current discussion.
>
> Here is a suggestion to better integrate bibtex support (and more
> generally latex support, or math writing support), into zotero.
> A) On bibtex import into zotero. If a pair of $ signs is detected,
> either 1) (preferably) transform the content into pure Unicode text if
> possible. Example: $\alpha$ is transformed into "α", unicode alpha. 2)
for greek letters. As Rick pointed out I forgot to implement it in the
reverse mapping table. So this is not a problem.
> (if no text alternative) transform the content into
> <latex>content</latex> markup. Example: $\frac{a}{b}$ would be
> transformed into <latex>\frac{a}{b}</latex>.
topic. There may be more controversies.
> B) On bibtex export from zotero. Obviously, change back the <latex> and
> </latex> tags into dollar signs. Quote dollar signs from zotero (because
> these must be considered as real dollar signs).
implemented.
> C) (For more general support of maths.) Transform other translators to
> account for these <latex> tags. E.g. transforming latex content into
> HTML should be possible by re-using some latex-to-html program.
> D) Problem of the currently existing entries that have latex code
> enclosed into dollar signs (e.g. non-transformed entries imported from
> bibtex). A plugin to zotero (or other appropriate mechanism as suggested
> by the zotero team) should be written in order to batch-transform these
> entries. Problem: not every $ pairs must be transformed into <latex>
> tags. Probably ask the user case-by-case for confirmation. It should
> thus be possible to run the batch on a subset of entries to permit
> correcting only the specific subset of entries the user is interested in
> from a large db.
agaist the philosophy of a lot of the devs here except Avram. I don't
see in near future that this will be done by the zotero team.
> E) Problem of people wanting to use the strings "<latex>" and "</latex>"
> in an entry without giving them a specific meaning. Does that exist?
really affect everybody.
am not capable to implement this. Nor is it possible to get most of
the devs agree shortly.
>
> 3) A simple and very concrete proposal to spartan (provided zotero team
> agrees).
> As suggested by I think Avram: spartan, add a user preference that asks
> for dollar signs to be exported to bibtex as-is. The preference is by
> default to export quoted dollar signs. Add a simple check into your
> proposed patch to ensure the user really wants to export dollar signs to
> bibtex as-is. Then I think everybody is happy.
be the only one who sort of agree on such a compromise. We'll still
have to wait to see if other members of the team will loosen up their
philosophy and agree on this.
>
> read more »
Erik Hetzner
Bruce D'Arcus wrote:
>
> For LaTeX-friendly math display on the web, MathJax seems to be a popular
> option these days. You might want to look into whether that might provide a
> path forward?
On a related note, it might be worth distinguishing between:
a) BibTeX junk that is a legacy of pre-unicode days and TeX-specific
hacks (e.g. all those silly curly braces, -- for – (en-dash), etc.)
b) math, in which nothing has caught on like (La)TeX notation. It
would be silly, in my opinion, to prevent users from including TeX
notation for math in abstracts (or even titles?), esp. with products
like MathJax providing an alternate rendering method for TeX math
notation.
(If somebody already made this point, my apologies for the inadvertent
plagiarism, but I haven’t been able to read all the text.)
best, Erik
Dan Stillman
No, that's a fine distinction.
Keeping that in mind, ASCIIMathML.js
(http://www1.chapman.edu/~jipsen/mathml/asciimath.html) looks
interesting, given that Firefox supports MathML rendering. It's
basically aiming for a user-friendly graphing calculator notation, and
it also supports standard LaTeX notation.
Advantages:
1) We could (in theory—I haven't looked to see if there's support for
this or if we'd have to write it) import LaTeX into a simpler notation
that could be clearer for people who don't know LaTeX and that looks
less like gibberish in places that don't handle LaTeX explicitly.
2) There's also a PHP port, which would let us use it on the server.
3) The ASCII notation could be displayed on the web for browsers that
don't support MathML (without requiring MathJax, which may be a bit heavy).
Downsides:
1) It's basically just a different arbitrary format, though it's at
least aiming to be as simple and intuitive as possible, and it seems to
have a fairly wide adoption in wikis and other tools.
2) Given that it supports LaTeX notation, it doesn't really address our
core concerns with privileging LaTeX in Zotero data fields, but, as Erik
notes, if math simply can't be represented as Unicode, there may not be
a better alternative.
Dan Stillman
> Advantages:
>
> that could be clearer for people who don't know LaTeX and that looks
> less like gibberish in places that don't handle LaTeX explicitly.
>
> 2) There's also a PHP port, which would let us use it on the server.
>
> 3) The ASCII notation could be displayed on the web for browsers that
> don't support MathML (without requiring MathJax, which may be a bit
> heavy).
>
> Downsides:
>
> 1) It's basically just a different arbitrary format, though it's at
> least aiming to be as simple and intuitive as possible, and it seems
> to have a fairly wide adoption in wikis and other tools.
>
> 2) Given that it supports LaTeX notation, it doesn't really address
> our core concerns with privileging LaTeX in Zotero data fields, but,
> as Erik notes, if math simply can't be represented as Unicode, there
> may not be a better alternative.
I also haven't checked whether there's something to convert their ASCII
notation to LaTeX directly for use in the BibTeX export, though given
that it produces MathML we could get there one way or another.
spartan
unicode part) in fields like title and abstract, I see nothing but
some sort of LaTeX notation will prevail like Erik and Dan did. Both
MathJax and ASCIIMathML as suggested by Bruce and Dan, respectively,
are good options. They both support latex notations. I actually used
ASCIIMathML with tinymce in zotero for note field, it worked pretty
well for me. Should such a support happen in the future, the blind
escaping in bibtex export will look really bad.
-spartan
> > a) BibTeX junk that is a legacy of pre-unicode days and TeX-specific
> > hacks (e.g. all those silly curly braces, -- for – (en-dash), etc.)
>
> > b) math, in which nothing has caught on like (La)TeX notation. It
>
Dan Stillman
> This is interesting. If zotero will implement math-support (non-
> unicode part) in fields like title and abstract, I see nothing but
> some sort of LaTeX notation will prevail like Erik and Dan did. Both
> MathJax and ASCIIMathML as suggested by Bruce and Dan, respectively,
> are good options. They both support latex notations. I actually used
> ASCIIMathML with tinymce in zotero for note field, it worked pretty
> well for me. Should such a support happen in the future, the blind
> escaping in bibtex export will look really bad.
If we implemented ASCIIMathML, the parsed math would obviously be
exported as proper LaTeX. That doesn't mean the escaping wouldn't remain
for non-math LaTeX (i.e., things for which there are Unicode equivalents).
It's good to hear that ASCIIMathML has worked for you before. Seems
promising.
spartan
earlier. Suppose I have a data entry with latex markup like $...$.
When I export it to a bib file, I'll get something like \$..(more \
\)...\$. If I share it with another person, he import it in to his
database and then export it to share with others, the entry will
become something like \\\$..(much more \\)...\\\$. The more sharing or
cycles it goes, the messier it will get. Only the original version can
be adapted easily by new math support or even unicode support. All the
others just basically get trashed.
For the opposite approach, the exported bib file keeps it the same as
$...$. When the next guy imports the shared item, it suddenly become
standard math format supported in zotero. Any further sharing follow
exactly what zotero would like it to do. No data got trashed.
Everybody should be happy.
Erik Hetzner
To be honest I am not sure why Zotero would support ASCIIMathML
instead of TeX math. Everybody already knows TeX math.
By the way, given my previous remarks about distinguishing between
math/non-math in BibTeX, I would like to point out that $\alpha$ is a
tricky case. Is the user using $\alpha$ as a workaround because the
were not able to enter α (U+03B1 GREEK SMALL LETTER ALPHA), perhaps in
a paper about the greek alphabet? Or did they want 𝛼 (U+1D6FC,
MATHEMATICAL ITALIC SMALL ALPHA)? In which case should it be imported
as that unicode character or as <tex>$\alpha$</tex>? An importer will
always need to make some guesses about user intention, as will an
exporter.
best, Erik
Dan Stillman
> You will understand my point better if you take a look at my example
> earlier. Suppose I have a data entry with latex markup like $...$.
> When I export it to a bib file, I'll get something like \$..(more \
> \)...\$. If I share it with another person, he import it in to his
> database and then export it to share with others, the entry will
> become something like \\\$..(much more \\)...\\\$. The more sharing or
> cycles it goes, the messier it will get. Only the original version can
> be adapted easily by new math support or even unicode support. All the
> others just basically get trashed.
>
> For the opposite approach, the exported bib file keeps it the same as
> $...$. When the next guy imports the shared item, it suddenly become
> standard math format supported in zotero. Any further sharing follow
> exactly what zotero would like it to do. No data got trashed.
> Everybody should be happy.
You seem to be under the impression that we don't understand you. I
assure you we do. There's no need to keep repeating this.
I'm telling you that, if we implemented ASCIIMathML, parsed math
obviously wouldn't be escaped when exporting to BibTeX. The same would
quite possibly not be true�for the reasons we've outlined�for anything
that didn't go through the math parser.
Dan Stillman
> At Fri, 06 May 2011 17:16:52 -0400,
> Dan Stillman wrote:
>> On 5/6/11 4:45 PM, spartan wrote:
>>> This is interesting. If zotero will implement math-support (non-
>>> unicode part) in fields like title and abstract, I see nothing but
>>> some sort of LaTeX notation will prevail like Erik and Dan did. Both
>>> MathJax and ASCIIMathML as suggested by Bruce and Dan, respectively,
>>> are good options. They both support latex notations. I actually used
>>> ASCIIMathML with tinymce in zotero for note field, it worked pretty
>>> well for me. Should such a support happen in the future, the blind
>>> escaping in bibtex export will look really bad.
>> If we implemented ASCIIMathML, the parsed math would obviously be
>> exported as proper LaTeX. That doesn't mean the escaping wouldn't remain
>> for non-math LaTeX (i.e., things for which there are Unicode equivalents).
>>
>> It's good to hear that ASCIIMathML has worked for you before. Seems
>> promising.
> To be honest I am not sure why Zotero would support ASCIIMathML
> instead of TeX math. Everybody already knows TeX math.
Well, mainly because TeX math looks like code, and ASCIIMathML looks
(more) like math. The graphing calculator notation supported by
ASCIIMathML seems to be the better analogue to the Unicode
representations—i.e., natural, non-code-like representations—of other
LaTeX symbols that we're trying to use. And while I'll certainly defer
to others on this, it doesn't seem that "everybody" knows TeX math, or
else graphing calculators would use it and ASCIIMathML wouldn't be as
popular as it seems to be. Given spartan's example, it seems even people
who know TeX math appreciate the simpler syntax in some contexts.
I'd like to hear other opinions on this, though.
spartan
It is a nice plugin for TinyMCE which is the editor zotero uses for
note. It should be easy to implement and all the fields (e.g., title,
abstract, note) supporting math will be consistent. Here are some
links:
The main website: http://www1.chapman.edu/~jipsen/mathml/asciimath.html
The lastest ASCIIMathML.js: http://mathcs.chapman.edu/~jipsen/mathml/ASCIIMathML.js
demo for ASCIIMathML in TinyMCE: http://www.imathas.com/editordemo/demo.html
spartan
===============
New Updates:
updated: May 7, 2011
1. unicode conversion of greek letters are also put in the reverse
mapping
table per Rick Karnesky's suggestion.
2. getPref() can now access only the subtree preferences prefixed with
'translator.' based on the discussion with Avram Lyon. This is a
modification
for one of the core files, translate.js.
3. export of the local path information of attachments is optional now
to
address Rick Karnesky's concern. The new preference setting of a
boolean
extensions.zotero.translator.bibtexExportLocalFilePath is used.
The default is no ( i.e., same as before).
4. escaping of Latex markups is controlled via a new preference
setting of
extensions.zotero.translator.bibtexPreserveLatex. The default is not
preserving, i.e., escaping markups the same way as before.
5. use (last name, first name) format for creator's name output to
remove
possible ambiguity of the current format (first name last name).
Dan Stillman
> 2. getPref() can now access only the subtree preferences prefixed with
> 'translator.' based on the discussion with Avram Lyon. This is a
> modification
> for one of the core files, translate.js.
>
> 3. export of the local path information of attachments is optional now
> to
> address Rick Karnesky's concern. The new preference setting of a
> boolean
> extensions.zotero.translator.bibtexExportLocalFilePath is used.
> The default is no ( i.e., same as before).
>
> 4. escaping of Latex markups is controlled via a new preference
> setting of
> extensions.zotero.translator.bibtexPreserveLatex. The default is not
> preserving, i.e., escaping markups the same way as before.
Thanks for your work on this, but we're not going to just pref anything
that's controversial.
- We have a possible solution (ASCIIMathML) to the escaping issue.
- The local path issue needs to be discussed further.
- It's not clear that we need to expose prefs to translators if the
above two issues are addressed without prefs. But this merits further
discussion as well.
Olivier Cailloux
> On 5/8/11 1:39 AM, spartan wrote:
>> 2. getPref() can now access only the subtree preferences prefixed with
>> 'translator.' based on the discussion with Avram Lyon. This is a
>> modification
>> for one of the core files, translate.js.
>>
>> 3. export of the local path information of attachments is optional now
>> to
>> address Rick Karnesky's concern. The new preference setting of a
>> boolean
>> extensions.zotero.translator.bibtexExportLocalFilePath is used.
>> The default is no ( i.e., same as before).
>>
>> 4. escaping of Latex markups is controlled via a new preference
>> setting of
>> extensions.zotero.translator.bibtexPreserveLatex. The default is not
>> preserving, i.e., escaping markups the same way as before.
>
> Thanks for your work on this, but we're not going to just pref
> anything that's controversial.
>
> - We have a possible solution (ASCIIMathML) to the escaping issue.
You mention that other possibilities, possibly (or probably) better,
exist to solve the problem spartan is addressing here. True. But it�s
not a reasonably small change to ask to spartan to implement, it�s a
completely different path. He mentioned he won't do it (that�s
understandable), and AFAIU we are not close to find someone to do it. So
that suggestion amounts to leave the issue unresolved.
So to recap, spartan worked to improve the bibtex support. He proposed a
solution, and, after some (though) discussions he changed his proposal
to include most reasonable (in terms of required amount of work)
suggestions that he were offered. Importantly, he added a pref to make
sure that the default is as before and if the user really wants no
quote, he can. This means that spartan's proposal is a superset of the
current functionalities. Now it appears that the offered improvement
will perhaps not make it into zotero because some possibly better
solution that noone in the near future is going to implement exist?
Let me be clear: I favor (as my previous posts showed) thinking about a
better approach to solve that math problem. But I also favor being
realistic about the time, amount of work, probability of finding someone
to do it, etc., issues about that better approach. I�d suggest
separating long term and short term here.
>
> - The local path issue needs to be discussed further.
>
> - It's not clear that we need to expose prefs to translators if the
> above two issues are addressed without prefs. But this merits further
> discussion as well.
It seems reasonable to me to have per-translator preferences. Most
probably other translators could use these, e.g. because of other
formats ambiguities that the user could have to specify how to solve.
Olivier
Bruce D'Arcus
<olivier....@ecp.fr> wrote:
...
>> - The local path issue needs to be discussed further.
>>
>> - It's not clear that we need to expose prefs to translators if the above
>> two issues are addressed without prefs. But this merits further discussion
>> as well.
>
> It seems reasonable to me to have per-translator preferences. Most probably
> other translators could use these, e.g. because of other formats ambiguities
> that the user could have to specify how to solve.
So if other translators can use preferences for key generation and
paths, say, then why are the preferences specific to one translator?
I'm guessing this would be Dan's question at least.
Bruce
Avram Lyon
>>> - The local path issue needs to be discussed further.
>>>
>>> - It's not clear that we need to expose prefs to translators if the above
>>> two issues are addressed without prefs. But this merits further discussion
>>> as well.
>>
>> It seems reasonable to me to have per-translator preferences. Most probably
>> other translators could use these, e.g. because of other formats ambiguities
>> that the user could have to specify how to solve.
>
> So if other translators can use preferences for key generation and
> paths, say, then why are the preferences specific to one translator?
I had suggested at the very start of this discussion that the
preferences say something like "translator", not "bibtex". We'd only
be talking about a subtree of the prefs, if getPref is added to the
sandbox. If the key generation preference is shared one or more of
file renaming, citation labels (like [JAbD99]), and BibTeX export,
then we should probably put it in a different part of the prefs and
give it its own function in the sandbox.
- Avram
spartan
On May 8, 12:29 pm, Dan Stillman <dstill...@zotero.org> wrote:
>
> Thanks for your work on this, but we're not going to just pref anything
> that's controversial.
>
> - We have a possible solution (ASCIIMathML) to the escaping issue.
>
> - The local path issue needs to be discussed further.
>
> - It's not clear that we need to expose prefs to translators if the
> above two issues are addressed without prefs. But this merits further
> discussion as well.
proposed, the bibtex key format string, collision format string, and
manual key field need access to prefs. I don't see why these strings
can not use prefs just like the way the format string for renaming
attachment files is doing now.
Bruce D'Arcus
Again, I don't know what Dan had in mind in his comment, but my point
would that be that the key and path issues are not specific to BibTeX:
they're relevant to most output formats.
Bruce
Peter Gerdes
Often the title will be stored in various databases (mathscinet tho it's hard to search for symbols) as the TeX markup and a unicode encoded version would no longer allow people to search by the specific markup they saved (extra escapes etc..). While you can export the bibliography to bibtex and render it in your citation list correctly even if unicode conversion is done you can no longer copy the title name out of zotero and send it to a colleague in a text email (many academics still use non-unicode terminals).
Ultimately, of course, you have to preserve and display some latex markup for complicated mathematical expressions (short of rolling in mathml support or something). So given that the TeX marked up title is effectively regarded as being the title amoung mathematicians I think it's fallacious to argue that conversion is necessery to allow interoperability with other zotero users. The TeX markup is just part of the title and if you don't understand it it's just like not understanding a word in the title.
---
Don't get me wrong I like the idea of converting symbols as a default preference but I just felt the arguments about interoperability weren't quite convincing.
Bruce D'Arcus
> I'm not going to weigh in directly on the contentiousness in this thread but
> I wanted to take issue with an implicit assumption that seems to be
> permeating this discussion: Namely that the REAL title of an article with
> mathematical markup is the result of replacing the markup with appropriate
> symbols.
>
> Often the title will be stored in various databases (mathscinet tho it's
> hard to search for symbols) as the TeX markup and a unicode encoded version
> would no longer allow people to search by the specific markup they saved
> (extra escapes etc..). While you can export the bibliography to bibtex and
> render it in your citation list correctly even if unicode conversion is done
> you can no longer copy the title name out of zotero and send it to a
> colleague in a text email (many academics still use non-unicode terminals).
>
> Ultimately, of course, you have to preserve and display some latex markup
> for complicated mathematical expressions (short of rolling in mathml support
> or something). So given that the TeX marked up title is effectively
> regarded as being the title amoung mathematicians I think it's fallacious to
> argue that conversion is necessery to allow interoperability with other
> zotero users. The TeX markup is just part of the title and if you don't
> understand it it's just like not understanding a word in the title.
I am willing to accept the notion that mathematical expressions are a
kind of structured inline data of sorts. What I object to is
privileging TeX as the only way to represent that data.
To back up, my concerns are in part about the user-facing display
issues, but also about data portability across different formats
(HTML, RTF, Word, etc.).
Certainly you'd prefer to have the HTML output of your math-inclusive
titles be properly rendered as a math expression, than raw TeX?
Certainly it's feasible that in some data exchange contexts it might
make sense to have those expressions encoded as MathML rather than
TeX?
Those are the sorts of issues that are behind my own position on this.
Perhaps MathJax is worth looking at in the future, as it seems to be
able to a) properly represent math expressions, and b) convert among
TeX and MathML.
Bruce
Alexander Brensing
i recently installed a version of your BibTeX_Tan.js edited by Robin (BibTex_RobinTan.js attached). I further edited it and now I got a bug.
My problem is that I can not translate Zotero entries with a number as first character in the title. With the original BibTeX.js-file it works but with our version it does not.
I hope you can help me by identifying the error.
best Alex
Am Donnerstag, 5. Mai 2011 05:53:07 UTC+2 schrieb spartan:
This is a re-post of my post at zotero-forum:
http://forums.zotero.org/discussion/17847/improving-bibtex-export/
I love zotero so much. But got disturbed by the poor support of
bibtex. So I spent quite some time to improve the functionality of the
bibtex export. It should be very useful for those who use bibtex/latex
a lot, in particular for scientists like me. But I don't know where I
should post my modified codes.
Here is the brief list of improvements from my hard work:
1. much more flexible (user control) bibtex key generation: from
author names, initials, title, journal, year, volume, pages ,...etc
2. the suffix for key collisions can be numeric or alphabetic
3. key format strings can be read from prefs
(extention.zotero.bibtex....)
4. attachments like pdf files can be exported with real path links
only. now you can easily get the resulting bibtex file work with other
external applications like JabRef and Mendeley.
5. unicode conversions for greek letters
6. user specified field like callNumber to export pre-stored keys
... and other improvements.
It's been working pretty well for me. But I'd like to share it with
your guys.
Please drop me a note if somebody knows how to get these improvements
added to the next release of zotero.
spartan