Anticipated performance problems working with a Zotero library of 500K+ references

[Sally Kleinfeldt]

Hello Zotero devs - we are scoping out a project that would use the Zotero API to index the items in a Zotero Library for display and search in the Plone CMS. Our client has 100's of thousands of references that they are considering managing in a single public Zotero library organized in decade and year collections. Are they likely to encounter any performance problems working with a library that large?

Thank you!

[Dan Stillman]

Almost all API requests are paged, so the library size shouldn't matter
too much if you're just interacting with the API, but you'd want to do a
lot of local caching [1] to avoid having to repeatedly make a huge
number of small requests.

Not clear to me whether you're saying you would use the desktop app or
the web library as well. We know that some people do manage to use the
desktop app with hundreds of thousands of items, though we don't
currently do much testing with libraries of that size.

If something isn't performing well, you can let us know and we can try
to improve it.

- Dan

[1] https://www.zotero.org/support/dev/web_api/v3/basics#caching

[Hugh Paterson III]

Using zotero desktop I imported 300,000 entries from bibtex. It took me 11 hours on an IBM Thinkpad T-530 running Ubuntu 16.04. Zotero was sluggish, and I encounter sluggish ness at 8-10k entries.  I find the degree of sluggishness is related to the free ram. But I have not done any formal testing. In my import zotero was not the only application running. My machine had/has 16gb of ram. 

- Hugh 

--
You received this message because you are subscribed to the Google Groups "zotero-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zotero-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/zotero-dev/010001807f75ccf5-790cd70e-5a73-47b2-a2eb-fd6c0af00d80-000000%40email.amazonses.com.

--

All the best,
-Hugh

Sent from my iPhone

[Emiliano Heyns]

Import is incredibly slow for large datasets. You could try using the API and syncing the data in, that would at least skip the overhead of the import translator.

[Sally Kleinfeldt]

Thank you so much Dan, Hugh and Emiliano! Yes we weren't concerned about the API due to the pagination, just about the editing experience - both desktop app and web interface will be used. References will be imported in smallish batches.We'll report here if we run into trouble but hopefully not. It sounds like we may not have many more than 150K references after all, whew.

Sally