Very large Zotero library for testing?

[Robin Wilson]

Hi all,

I've spent the first part of the Christmas holidays doing a major re-write of my AutoZotBib extension for Zotero that will automatically update a BibTex file whenever anything in the Zotero library changes. The major issue with the previous version was that I could cause Zotero to run *very* slowly, particularly when running it on a large library. This was mainly because the old version was very naive and exported the entire library to Bibtex format each time anything changed, rather than just updating the bit that changed! I've now put some far better logic in here, and I'd like to test it on a huge Zotero library to see what the performance is like.

So, my question: Is there a very large Zotero library available anywhere for testing purposes? The ideal would be a library that is used for performance-testing by the Zotero development team - does something like that exist? If not, does anyone have a very large library they could send me? I could create a large library for testing purposes myself (with a for loop to add lots and lots of entries using semi-random choices for various fields - but that won't be very realistic).

Happy Christmas (for tomorrow, or the day after - depending what time zone you're in),

Robin

[Kieren Diment]

How large is "very large"? Should it contain fulltext and notes?. Would it be helpful to have a few people email you a zotero export of 100 or so references so you can merge them into one library? Could you download a sample of data from proquest, ebschost, pubmed, ISI etc?

> --
> You received this message because you are subscribed to the Google Groups "zotero-dev" group.
> To post to this group, send email to zoter...@googlegroups.com.
> To unsubscribe from this group, send email to zotero-dev+...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/zotero-dev?hl=en.
>

[Robin Wilson]

Hi,

Basically the larger the better (I was thinking thousands of items if possible).

I would be happy to merge various exports etc if that seems to be the best way to do it.

I'll investigate how I can get loads of references from places like ISI/WoK - I was really just checking to see if a large Zotero library for testing purposes existed already that I could just use without having to bother to create one myself (lazy I know!). I suppose the other issue with importing lots of items from ISI/WoK is that it wouldn't necessarily be like a 'real world' large Zotero library (in terms of how the items were added, the types of items, the range of attachments etc).

Cheers,

Robin

[adamsmith]

Robin,
what's the status of the "new" AutoZotBib? I'm giving a presentation to a group of science/engineering types next Wednesday and I'd like to recommend it, but obviously only if its in at least a beta-ish state.
Sebastian

[Aurimas Vinckevicius]

This reminds me. I have recently created an approximately 56,000 item library in Endnote and exported it as a RIS file. It is not an extremely well balanced library, since it is all journal articles from PubMed with no notes, attachments, or keywords, but it's a large library nonetheless.

One downside is that currently there is a bug in Zotero that makes Zotero halt during a large import, so I have not successfully imported this library yet. It appears to progressively slow down and get stuck after about 8,000 items or so. I'm looking into this issue, but in the mean time you can probably import the whole library in chunks.

The RIS file is ~32MB and can be downloaded from https://www.dropbox.com/s/sifgr55ybjlvg82/test.ris

Hope this is helpful

To view this discussion on the web visit https://groups.google.com/d/msg/zotero-dev/-/OEg3pIpSZN8J.

[Robin Wilson]

Hi Sebasian,

Unfortunately it's not quite ready yet. It's almost working but I've run into a bug in Zotero which stops it working fully. Instead of fixing the bug, I've been discussing with the main Zotero authors about how best to implement that bit of code differently (see the thread on this forum about the extraData issue) - and I'm waiting to find out when that will be done. Once that's done it should be fairly easy to update my code to use the new method, do a bit more testing, and then release.

Sorry about the delay - feel free to post in the other thread if you're interested in an ETA for the bug fix for Zotero itself,

Cheers,

Robin

To view this discussion on the web visit https://groups.google.com/d/msg/zotero-dev/-/OEg3pIpSZN8J.

[Dominik K.]

Hi Robin and Sebastian,

did you hear about the Zotero autoexporting plugin by Robert Kühn [1]?
It provides the same functionality to Zotero or Robert has the same goal
as you, Robin, but in a more global manner (exporting to all available
Zotero export formats, and other things [2]).

@Robin: Unfortunately, it seems that its also coded in the same way as
yours before was that the whole library will be exported after a change
in it, which really slows down the plugin. So, maybe you're interested
to join your efforts with Robert to code and maintain one autoexport
plugin with synthesized (merged) and improved functionality?


Cheers,

Dominik


[1] http://rokdd.de/b/zotero-autoexport
[2] "the main features are:

trigger export by a timer for example every 120 minutes
trigger the export by changes in your zotero database
setup the format options (and the translator)
splitting exported files into collections and subcollections [upon
1.0.6]
postprocessing with an external script like a batch file for ftp
upload" -> [1]

[rokdo]

Hello together

after nearly one year I found this posting now.. :) The problem was that I did not found the AutoZotBib at the starting point of my addon. But I have some more features but not so well adjusted for Bibtex. I try to make the addon every time more powerful and fitting to its user - but this needs feedback from everybody. The source code is still available at github so if someone want to contribute feel free, because I have a lack of time too. I just setup an issue in my tracker to test with huge databases. But my plugin is able to export only certain collections / groups / saved searches and so it does not export the whole library, for example only current projects.

So if someone of you have an issue or an idea or something else let me know

Cheers

Robert