[gpapers] 14 new revisions pushed by marcelCo...@gmail.com on 2012-03-25 22:09 GMT

3 views

Skip to first unread message

codesite...@google.com

unread,

Mar 25, 2012, 6:10:06 PM3/25/12

to gpapers...@googlegroups.com

14 new revisions:

Revision: 6eb35ba9807a
Author: Marcel Stimberg <marcel...@gmail.com>
Date: Sun Mar 4 15:07:37 2012
Log: Some more leftover changes from the GTK3 conversion
http://code.google.com/p/gpapers/source/detail?r=6eb35ba9807a

Revision: 2297aa561675
Author: Marcel Stimberg <marcel...@gmail.com>
Date: Sun Mar 4 13:24:20 2012
Log: continued refactoring, added JSTOR search
http://code.google.com/p/gpapers/source/detail?r=2297aa561675

Revision: 0c02fe87175e
Author: Marcel Stimberg <marcel...@gmail.com>
Date: Sun Mar 4 13:25:39 2012
Log: Remove unused code
http://code.google.com/p/gpapers/source/detail?r=0c02fe87175e

Revision: 529f4a23df3f
Author: Marcel Stimberg <marcel...@gmail.com>
Date: Sun Mar 4 13:55:35 2012
Log: Use GObject.timeout_add instead of a dedicated thread and sleep
for re...
http://code.google.com/p/gpapers/source/detail?r=529f4a23df3f

Revision: 5334b0ee3ac8
Author: Marcel Stimberg <marcel...@gmail.com>
Date: Sun Mar 4 14:28:26 2012
Log: remove unused code, document code
http://code.google.com/p/gpapers/source/detail?r=5334b0ee3ac8

Revision: ca332885ee92
Author: Marcel Stimberg <marcel...@gmail.com>
Date: Sun Mar 4 14:35:06 2012
Log: small unicode-related changes
http://code.google.com/p/gpapers/source/detail?r=ca332885ee92

Revision: deb9b798e357
Author: Marcel Stimberg <marcel...@gmail.com>
Date: Sun Mar 4 15:40:52 2012
Log: Remove some code duplication
http://code.google.com/p/gpapers/source/detail?r=deb9b798e357

Revision: 00b5d40f5931
Author: Marcel Stimberg <marcel...@gmail.com>
Date: Sun Mar 18 12:00:26 2012
Log: progressing towards an again usable import...
http://code.google.com/p/gpapers/source/detail?r=00b5d40f5931

Revision: 1556a9466b05
Author: Marcel Stimberg <marcel...@gmail.com>
Date: Sat Mar 24 05:36:26 2012
Log: small changes for Gtk3
http://code.google.com/p/gpapers/source/detail?r=1556a9466b05

Revision: a8df07bccb73
Author: Marcel Stimberg <marcel...@gmail.com>
Date: Sun Mar 25 12:12:42 2012
Log: make sure the GUI updates on database changes
http://code.google.com/p/gpapers/source/detail?r=a8df07bccb73

Revision: 18cdb1a44680
Author: Marcel Stimberg <marcel...@gmail.com>
Date: Sun Mar 25 15:07:36 2012
Log: work around some libsoup bugs...
http://code.google.com/p/gpapers/source/detail?r=18cdb1a44680

Revision: 4eed7c76daf3
Author: Marcel Stimberg <marcel...@gmail.com>
Date: Sun Mar 25 14:15:48 2012
Log: re-add the "busy notifier"
http://code.google.com/p/gpapers/source/detail?r=4eed7c76daf3

Revision: de69e230ca2f
Author: Marcel Stimberg <marcel...@gmail.com>
Date: Sun Mar 25 14:18:44 2012
Log: remove some unused code
http://code.google.com/p/gpapers/source/detail?r=de69e230ca2f

Revision: 8c79c339d8cd
Author: Marcel Stimberg <marcel...@gmail.com>
Date: Sun Mar 25 14:49:56 2012
Log: make document import more or less work
http://code.google.com/p/gpapers/source/detail?r=8c79c339d8cd

==============================================================================
Revision: 6eb35ba9807a
Author: Marcel Stimberg <marcel...@gmail.com>
Date: Sun Mar 4 15:07:37 2012
Log: Some more leftover changes from the GTK3 conversion
http://code.google.com/p/gpapers/source/detail?r=6eb35ba9807a

Modified:
/gPapers.py
/ui.xml

=======================================
--- /gPapers.py Thu Feb 23 10:47:00 2012
+++ /gPapers.py Sun Mar 4 15:07:37 2012
@@ -326,7 +326,7 @@
dialog.set_markup('Import URL...\n\nEnter the URL you would
like to import:')
entry = Gtk.Entry()
entry.set_activates_default(True)
- dialog.vbox.pack_start(entry, True)
+ dialog.vbox.add(entry)
dialog.set_default_response(Gtk.ResponseType.OK)
dialog.show_all()
response = dialog.run()
@@ -351,7 +351,7 @@
dialog.set_markup('Import via DOI...\n\nEnter the DOI name
(e.g., 10.1000/182) you would like to import:')
entry = Gtk.Entry()
entry.set_activates_default(True)
- dialog.vbox.pack_start(entry, True)
+ dialog.vbox.add(entry)
dialog.set_default_response(Gtk.ResponseType.OK)
dialog.show_all()
response = dialog.run()
@@ -420,7 +420,7 @@
scrolledwindow = Gtk.ScrolledWindow()
scrolledwindow.add(entry)
scrolledwindow.set_property('height-request', 300)
- dialog.vbox.pack_start(scrolledwindow, True)
+ dialog.vbox.add(scrolledwindow)
dialog.set_default_response(Gtk.ResponseType.OK)
dialog.show_all()
response = dialog.run()
=======================================
--- /ui.xml Thu Feb 23 10:47:00 2012
+++ /ui.xml Sun Mar 4 15:07:37 2012
@@ -1,6 +1,141 @@
<?xml version="1.0" encoding="UTF-8"?>
<interface>

+ <object class="GtkUIManager" id="uimanager1">
+ <child>
+ <object class="GtkActionGroup" id="actiongroup1">
+ <child>
+ <object class="GtkAction" id="menuitem_files">
+ <property name="name">menuitem_files</property>
+ <property name="label" translatable="yes">_File</property>
+ </object>
+ </child>
+ <child>
+ <object class="GtkAction" id="menuitem_import_files">
+ <property name="name">menuitem_import_files</property>
+ <property name="label" translatable="yes">Import
Files…</property>
+ </object>
+ </child>
+ <child>
+ <object class="GtkAction" id="menuitem_import_directory">
+ <property name="name">menuitem_import_directory</property>
+ <property name="label" translatable="yes">Import
Directory…</property>
+ </object>
+ </child>
+ <child>
+ <object class="GtkAction" id="menuitem_import_url">
+ <property name="name">menuitem_import_url</property>
+ <property name="label" translatable="yes">Import
URL…</property>
+ </object>
+ </child>
+ <child>
+ <object class="GtkAction" id="menuitem_import_doi">
+ <property name="name">menuitem_import_doi</property>
+ <property name="label" translatable="yes">Import
DOI…</property>
+ </object>
+ </child>
+ <child>
+ <object class="GtkAction" id="menuitem_import_bibtex">
+ <property name="name">menuitem_import_bibtex</property>
+ <property name="label" translatable="yes">Import
BibTex…</property>
+ </object>
+ </child>
+ <child>
+ <object class="GtkAction" id="menuitem_quit">
+ <property name="stock_id">gtk-quit</property>
+ <property name="name">menuitem_quit</property>
+ </object>
+ </child>
+ <child>
+ <object class="GtkAction" id="menuitem_edit">
+ <property name="name">menuitem_edit</property>
+ <property name="label" translatable="yes">_Edit</property>
+ </object>
+ </child>
+ <child>
+ <object class="GtkAction" id="menuitem_cut">
+ <property name="stock_id">gtk-cut</property>
+ <property name="name">menuitem_cut</property>
+ </object>
+ </child>
+ <child>
+ <object class="GtkAction" id="menuitem_copy">
+ <property name="stock_id">gtk-copy</property>
+ <property name="name">menuitem_copy</property>
+ </object>
+ </child>
+ <child>
+ <object class="GtkAction" id="menuitem_paste">
+ <property name="stock_id">gtk-paste</property>
+ <property name="name">menuitem_paste</property>
+ </object>
+ </child>
+ <child>
+ <object class="GtkAction" id="menuitem_delete">
+ <property name="stock_id">gtk-delete</property>
+ <property name="name">menuitem_delete</property>
+ </object>
+ </child>
+ <child>
+ <object class="GtkAction" id="menuitem_view">
+ <property name="name">menuitem4</property>
+ <property name="label" translatable="yes">_View</property>
+ </object>
+ </child>
+ <child>
+ <object class="GtkAction" id="menuitem_author_graph">
+ <property name="name">menuitem_author_graph</property>
+ <property name="label" translatable="yes">Author
Graph...</property>
+ </object>
+ </child>
+ <child>
+ <object class="GtkAction" id="menuitem_paper_graph">
+ <property name="name">menuitem_paper_graph</property>
+ <property name="label" translatable="yes">Paper
Graph...</property>
+ </object>
+ </child>
+ <child>
+ <object class="GtkAction" id="menuitem_help">
+ <property name="name">menuitem_help</property>
+ <property name="label" translatable="yes">_Help</property>
+ </object>
+ </child>
+ <child>
+ <object class="GtkAction" id="menuitem_about">
+ <property name="stock_id">gtk-about</property>
+ <property name="name">menuitem_about</property>
+ </object>
+ </child>
+ </object>
+ </child>
+ <ui>
+ <menubar name="menubar1">
+ <menu action="menuitem_files">
+ <menuitem action="menuitem_import_files"/>
+ <menuitem action="menuitem_import_directory"/>
+ <menuitem action="menuitem_import_url"/>
+ <menuitem action="menuitem_import_doi"/>
+ <menuitem action="menuitem_import_bibtex"/>
+ <separator/>
+ <menuitem action="menuitem_quit"/>
+ </menu>
+ <menu action="menuitem_edit">
+ <menuitem action="menuitem_cut"/>
+ <menuitem action="menuitem_copy"/>
+ <menuitem action="menuitem_paste"/>
+ <menuitem action="menuitem_delete"/>
+
+ </menu>
+ <menu action="menuitem_view">
+ <menuitem action="menuitem_author_graph"/>
+ <menuitem action="menuitem_paper_graph"/>
+ </menu>
+ <menu action="menuitem_help">
+ <menuitem action="menuitem_about"/>
+ </menu>
+ </menubar>
+ </ui>
+ </object>
<object class="GtkWindow" id="main_window">
<property name="width_request">800</property>
<property name="height_request">600</property>
@@ -19,7 +154,7 @@
<property name="can_focus">False</property>
<property name="events">GDK_POINTER_MOTION_MASK |
GDK_POINTER_MOTION_HINT_MASK | GDK_BUTTON_PRESS_MASK |
GDK_BUTTON_RELEASE_MASK</property>
<child>
- <object class="GtkMenuBar" id="menubar1">
+ <object class="GtkMenuBar" id="menubar1"
constructor="uimanager1">
<property name="visible">True</property>
<property name="can_focus">False</property>
<property name="events">GDK_POINTER_MOTION_MASK |
GDK_POINTER_MOTION_HINT_MASK | GDK_BUTTON_PRESS_MASK |
GDK_BUTTON_RELEASE_MASK</property>
@@ -743,140 +878,5 @@
</child>
</object>
</child>
- </object>
- <object class="GtkUIManager" id="uimanager1">
- <child>
- <object class="GtkActionGroup" id="actiongroup1">
- <child>
- <object class="GtkAction" id="menuitem_files">
- <property name="name">menuitem_files</property>
- <property name="label" translatable="yes">_File</property>
- </object>
- </child>
- <child>
- <object class="GtkAction" id="menuitem_import_files">
- <property name="name">menuitem_import_files</property>
- <property name="label" translatable="yes">Import
Files…</property>
- </object>
- </child>
- <child>
- <object class="GtkAction" id="menuitem_import_directory">
- <property name="name">menuitem_import_directory</property>
- <property name="label" translatable="yes">Import
Directory…</property>
- </object>
- </child>
- <child>
- <object class="GtkAction" id="menuitem_import_url">
- <property name="name">menuitem_import_url</property>
- <property name="label" translatable="yes">Import
URL…</property>
- </object>
- </child>
- <child>
- <object class="GtkAction" id="menuitem_import_doi">
- <property name="name">menuitem_import_doi</property>
- <property name="label" translatable="yes">Import
DOI…</property>
- </object>
- </child>
- <child>
- <object class="GtkAction" id="menuitem_import_bibtex">
- <property name="name">menuitem_import_bibtex</property>
- <property name="label" translatable="yes">Import
BibTex…</property>
- </object>
- </child>
- <child>
- <object class="GtkAction" id="menuitem_quit">
- <property name="stock_id">gtk-quit</property>
- <property name="name">menuitem_quit</property>
- </object>
- </child>
- <child>
- <object class="GtkAction" id="menuitem_edit">
- <property name="name">menuitem_edit</property>
- <property name="label" translatable="yes">_Edit</property>
- </object>
- </child>
- <child>
- <object class="GtkAction" id="menuitem_cut">
- <property name="stock_id">gtk-cut</property>
- <property name="name">menuitem_cut</property>
- </object>
- </child>
- <child>
- <object class="GtkAction" id="menuitem_copy">
- <property name="stock_id">gtk-copy</property>
- <property name="name">menuitem_copy</property>
- </object>
- </child>
- <child>
- <object class="GtkAction" id="menuitem_paste">
- <property name="stock_id">gtk-paste</property>
- <property name="name">menuitem_paste</property>
- </object>
- </child>
- <child>
- <object class="GtkAction" id="menuitem_delete">
- <property name="stock_id">gtk-delete</property>
- <property name="name">menuitem_delete</property>
- </object>
- </child>
- <child>
- <object class="GtkAction" id="menuitem_view">
- <property name="name">menuitem4</property>
- <property name="label" translatable="yes">_View</property>
- </object>
- </child>
- <child>
- <object class="GtkAction" id="menuitem_author_graph">
- <property name="name">menuitem_author_graph</property>
- <property name="label" translatable="yes">Author
Graph...</property>
- </object>
- </child>
- <child>
- <object class="GtkAction" id="menuitem_paper_graph">
- <property name="name">menuitem_paper_graph</property>
- <property name="label" translatable="yes">Paper
Graph...</property>
- </object>
- </child>
- <child>
- <object class="GtkAction" id="menuitem_help">
- <property name="name">menuitem_help</property>
- <property name="label" translatable="yes">_Help</property>
- </object>
- </child>
- <child>
- <object class="GtkAction" id="menuitem_about">
- <property name="stock_id">gtk-about</property>
- <property name="name">menuitem_about</property>
- </object>
- </child>
- </object>
- </child>
- <ui>
- <menubar name="menubar1">
- <menu action="menuitem_files">
- <menuitem action="menuitem_import_files"/>
- <menuitem action="menuitem_import_directory"/>
- <menuitem action="menuitem_import_url"/>
- <menuitem action="menuitem_import_doi"/>
- <menuitem action="menuitem_import_bibtex"/>
- <separator/>
- <menuitem action="menuitem_quit"/>
- </menu>
- <menu action="menuitem_edit">
- <menuitem action="menuitem_cut"/>
- <menuitem action="menuitem_copy"/>
- <menuitem action="menuitem_paste"/>
- <menuitem action="menuitem_delete"/>
-
- </menu>
- <menu action="menuitem_view">
- <menuitem action="menuitem_author_graph"/>
- <menuitem action="menuitem_paper_graph"/>
- </menu>
- <menu action="menuitem_help">
- <menuitem action="menuitem_about"/>
- </menu>
- </menubar>
- </ui>
</object>
</interface>

==============================================================================
Revision: 2297aa561675
Author: Marcel Stimberg <marcel...@gmail.com>
Date: Sun Mar 4 13:24:20 2012
Log: continued refactoring, added JSTOR search
http://code.google.com/p/gpapers/source/detail?r=2297aa561675

Modified:
/gPapers.py
/importer/__init__.py
/importer/google_scholar.py
/importer/jstor.py
/importer/pubmed.py

=======================================
--- /gPapers.py Sun Mar 4 15:07:37 2012
+++ /gPapers.py Sun Mar 4 13:24:20 2012
@@ -216,6 +216,11 @@
def import_citation_via_bibtex(bibtex):
importer.update_paper_from_bibtex_html(None, bibtex)

+def paper_info_received(paper):
+ paper.save()
+ importer.import_citation(paper.import_url, paper=paper,
+ callback=main_gui.refresh_middle_pane_search)
+
def import_citation_via_middle_top_pane_row(row):
# id, authors, title, journal, year, rating, abstract, icon,
import_url, doi, created, updated, empty_str, pubmed_id

@@ -230,22 +235,14 @@
pubmed_id = row[13]
data = row[14]
provider = row[15]
- paper, created = importer.get_or_create_paper_via(id=paper_id, doi=doi,
- pubmed_id=pubmed_id,
-
import_url=import_url,
- title=title,
data=data,
- provider=provider)
-
- if title: paper.title = title
- if abstract: paper.abstract = abstract
- if import_url: paper.import_url = import_url
- if doi: paper.doi = doi
- if pubmed_id: paper.pubmed_id = pubmed_id
-
- paper.save()
- if created:
- log_debug('Paper created: %s' % paper)
- importer.import_citation(paper.import_url, paper=paper,
callback=main_gui.refresh_middle_pane_search)
+ if doi and not import_url:
+ import_url = 'http://dx.doi.org/' + doi
+ importer.get_or_create_paper_via(id=paper_id, doi=doi,
+ pubmed_id=pubmed_id,
+ import_url=import_url,
+ title=title, data=data,
+ provider=provider,
+ callback=paper_info_received)

def import_document(filename, data=None):
@@ -258,6 +255,7 @@
try:
log_info('Importing paper: %s' % filename)
md5_hexdigest = get_md5_hexdigest_from_data(data)
+ # FIXME
paper, created =
importer.get_or_create_paper_via(full_text_md5=md5_hexdigest)
if created:
#paper.title = filename
@@ -286,6 +284,18 @@
def row_from_dictionary(info, provider=None):
assert info is not None

+ created = info.get('created')
+ updated = info.get('updated')
+ try:
+ created = created.strftime('%Y-%m-%d')
+ except:
+ pass
+ try:
+ updated = updated.strftime('%Y-%m-%d')
+ except:
+ pass
+
+
row = (
info.get('id', -1), # paper id
pango_escape(', '.join([author for author in
info.get('authors', [])])), # authors
@@ -297,8 +307,8 @@
info.get('icon'), # icon
info.get('import_url'), # import_url
info.get('doi'), # doi
- info.get('created'), # created
- info.get('updated'), # updated
+ created, # created
+ updated, # updated
'', # empty_str
info.get('pubmed_id'), # pubmed_id
info.get('data'),
@@ -433,7 +443,7 @@
dialog.destroy()

def __init__(self):
- self.provider = {'pubmed' : pubmed.PubMedSearch(),
+ self.search_providers = {'pubmed' : pubmed.PubMedSearch(),
'google_scholar' :
google_scholar.GoogleScholarSearch(),
'jstor' : jstor.JSTORSearch()}
self.displayed_paper = None
@@ -551,7 +561,7 @@
playlist, created = Playlist.objects.get_or_create(
title='search: %s' %
self.ui.get_object('middle_pane_search').get_text(),

search_text=self.ui.get_object('middle_pane_search').get_text(),
- parent=self.provider[liststore[row][4]]
+ parent=self.search_providers[liststore[row][4]]
)
if created: playlist.save()
self.refresh_left_pane()
@@ -573,7 +583,7 @@
if liststore[row][4] != 'local':
# For the external search providers: clear cache
text = self.ui.get_object('middle_pane_search').get_text()
- self.provider[liststore[row][4]].clear_cache(text)
+ self.search_providers[liststore[row][4]].clear_cache(text)

self.last_middle_pane_search_string = None

@@ -914,7 +924,7 @@
('highest rated',
FAVORITE_ICON, -4, False, 'local'))

- for _, provider in self.provider.iteritems():
+ for _, provider in self.search_providers.iteritems():
# FIXME: Load them in the providers
provider_icon =
GdkPixbuf.Pixbuf.new_from_file(os.path.join(RUN_FROM_DIR,
'icons',
@@ -983,13 +993,18 @@
if liststore[row][4] == 'local':
self.current_middle_top_pane_refresh_thread_ident =
thread.start_new_thread(self.refresh_middle_pane_from_my_library, (True,))
else:
- self.current_middle_top_pane_refresh_thread_ident =
thread.start_new_thread(lambda :
self.refresh_middle_pane_from_external(self.provider[liststore[row][4]]),
())
+ def error_callback(data1, data2):
+ print 'Error callback, received data1: ', data1
+ print 'Error callback, received data2: ', data2
+
+
self.search_providers[liststore[row][4]].search(self.ui.get_object('middle_pane_search').get_text(),
+
self.refresh_middle_pane_from_external, error_callback)

self.select_middle_top_pane_item(self.ui.get_object('middle_top_pane').get_selection())

def init_middle_top_pane(self):
middle_top_pane = self.ui.get_object('middle_top_pane')
- # id, authors, title, journal, year, rating, abstract, icon,
import_url, doi, created, updated, empty_str, pubmed_id, data, search
provider
+ # id, authors, title, journal, year, rating, abstract, icon,
import_url, doi, created, updated, empty_str, pubmed_id, data, search
search_providers
self.middle_top_pane_model = Gtk.ListStore(int, str, str, str,
str, int, str, GdkPixbuf.Pixbuf, str, str, str, str, str, str, object,
object)
middle_top_pane.set_model(self.middle_top_pane_model)

middle_top_pane.get_selection().set_mode(Gtk.SelectionMode.MULTIPLE)
@@ -1441,7 +1456,8 @@
# description.append( ref.line )

#self.ui.get_object('paper_information_pane').get_buffer().set_text( '\n'.join(description)
)

- if liststore[rows[0]][8]:
+ if liststore[rows[0]][8] or liststore[rows[0]][9]:
+ log_debug('URL or DOI exists')
button = Gtk.ToolButton(stock_id=Gtk.STOCK_HOME)
button.set_tooltip_text('Open this URL in your browser...')
button.connect('clicked', lambda x:
desktop.open(liststore[rows[0]][8]))
@@ -1954,42 +1970,64 @@
liststore.set_value(self.left_pane_model.get_iter((0,)), 0, 'My
Library (%i)' %
Paper.objects.count())
Gdk.threads_leave()

- def refresh_middle_pane_from_external(self, search_provider):
- log_debug('Starting external search in %s' % search_provider)
- search_text =
self.ui.get_object('middle_pane_search').get_text().strip()
- if not search_text: return
- self.active_threads[ thread.get_ident() ] = 'searching %s...
(%s)' % (search_provider, search_text)
+ def refresh_middle_pane_from_external(self, search_info, results):
+ '''
+ This callback is called when an external search has finished and
+ returned results. `search_info` has to be a tuple of a search
search_providers
+ label and the original search string, `results` is a list of
dictionaries
+ containing the search results, e.g.
+ [{'authors' : ['Author A', 'Author B], 'title': 'A title'},
+ {'authors' : ['Author C'], 'title': 'Another title'}]
+
+ Before the search results are actually displayed in the GUI, it is
+ checked whether the same search search_providers is still active
and the same
+ search string is still given. This should avoid situations where a
+ previous search result arrives later and would otherwise overwrite
the
+ results of a later search.
+ '''
+
+ label, search_string = search_info
+ search_provider = self.search_providers[label]
+ log_info('Received results for %s: %s' % (search_provider.name,
+ search_string))
+
+ liststore, row =
self.ui.get_object('left_pane_selection').get_selected()
+ current_label = liststore[row][4]
+ current_search_text =
self.ui.get_object('middle_pane_search').get_text().strip()
+
+ if label != current_label or search_string != current_search_text:
+ log_info('Results are no longer wanted.')
+ return
+
+ log_info('Results are still wanted, processing further...')
+
rows = []
- try:
- papers = search_provider.search(search_text)
- for info in papers:
- try:
- unique_key = search_provider.unique_key()
- kwds = {unique_key : info.get(unique_key)}
- existing_paper = Paper.objects.filter(**kwds)
- if not existing_paper:
- raise Paper.DoesNotExist()
- existing_paper = existing_paper[0]
- info['id'] = existing_paper.id
- info['created'] = existing_paper.created
- info['updated'] = existing_paper.updated
- if existing_paper.full_text and \
-
os.path.isfile(existing_paper.full_text.path):
- info['icon'] =
self.ui.get_object('middle_top_pane').\
- render_icon(Gtk.STOCK_DND,
- Gtk.IconSize.MENU)
- except Paper.DoesNotExist:
- pass
-
- # Add information to table
- rows.append(row_from_dictionary(info, search_provider))
-
-
self.update_middle_top_pane_from_row_list_if_we_are_still_the_preffered_thread(rows)
- except:
- traceback.print_exc()
-
- if self.active_threads.has_key(thread.get_ident()):
- del self.active_threads[ thread.get_ident() ]
+ for info in results:
+ try:
+ unique_key = search_provider.unique_key
+ kwds = {unique_key : info.get(unique_key)}
+ existing_paper = Paper.objects.filter(**kwds)
+ if not existing_paper:
+ raise Paper.DoesNotExist()
+ existing_paper = existing_paper[0]
+ info['id'] = existing_paper.id
+ info['created'] = existing_paper.created
+ info['updated'] = existing_paper.updated
+ if existing_paper.full_text and \
+ os.path.isfile(existing_paper.full_text.path):
+ info['icon'] = self.ui.get_object('middle_top_pane').\
+ render_icon(Gtk.STOCK_DND,
+ Gtk.IconSize.MENU)
+ except Paper.DoesNotExist:
+ pass
+
+ # Add information to table
+ rows.append(row_from_dictionary(info, search_provider))
+
+ self.middle_top_pane_model.clear()
+ for row in rows:
+ self.middle_top_pane_model.append(row)
+

class AuthorEditGUI:
def __init__(self, author_id, callback_on_save=None):
=======================================
--- /importer/__init__.py Wed Feb 22 16:03:16 2012
+++ /importer/__init__.py Sun Mar 4 13:24:20 2012
@@ -23,8 +23,10 @@

import gi
from gi.repository import GObject
+from gi.repository import Gdk
from gi.repository import Gtk
from gi.repository import Pango
+from gi.repository import Soup

from pyPdf import PdfFileReader

@@ -40,8 +42,8 @@
p_whitespace = re.compile('[\s]+')
p_doi = re.compile('doi *: *(10.[a-z0-9]+/[a-z0-9.]+)', re.IGNORECASE)

-# set our google scholar prefs (cookie-based)
-thread.start_new_thread(openanything.fetch,
('http://scholar.google.com/scholar_setprefs?num=100&scis=yes&scisf=4&submit=Save+Preferences',))
+soup_session = Soup.SessionAsync()
+

def latex2unicode(s):
"""
@@ -63,23 +65,28 @@
# TODO: expand this to really work
return s.replace('\c{s}', u's')

+
def _decode_htmlentities(string):
entity_re = re.compile("&(#?)(\d{1,5}|\w{1,8});")
return entity_re.subn(_substitute_entity, string)[0]

+
def html_strip(s):
if isinstance(s, BeautifulSoup.Tag):
s = ''.join([ html_strip(x) for x in s.contents ])
return _decode_htmlentities(p_whitespace.sub(' ',
str(s).replace(' ', ' ').strip()))

+
def pango_escape(s):
return
s.replace('&', '&').replace('>', '>').replace('<', '<')

+
def get_md5_hexdigest_from_data(data):
m = md5.new()
m.update(data)
return m.hexdigest()

+
def _substitute_entity(match):
ent = match.group(2)
if match.group(1) == "#":
@@ -92,6 +99,8 @@
else:
return match.group()

+
+#TODO: Remove all GUI code from this module
def _should_we_reimport_paper(paper):
Gdk.threads_enter()
dialog = Gtk.MessageDialog(type=Gtk.MessageType.QUESTION,
buttons=Gtk.ButtonsType.OK_CANCEL, flags=Gtk.DialogFlags.MODAL)
@@ -104,7 +113,8 @@
Gdk.threads_leave()
return response == Gtk.ResponseType.OK

-def get_or_create_paper_via(id=None, doi=None, pubmed_id=None,
+
+def get_or_create_paper_via(callback, id=None, doi=None, pubmed_id=None,
import_url=None, title=None,
full_text_md5=None,
data=None, provider=None):
"""Tries to look up a paper by various forms of id, from most specific
to
@@ -112,7 +122,7 @@
data and search_provider have to be specified."""
#print id, doi, pubmed_id, import_url, title, full_text_md5
paper = None
- created = False
+
if id >= 0:
try: paper = Paper.objects.get(id=id)
except: pass
@@ -156,16 +166,24 @@

if not paper:
# it looks like we haven't seen this paper before...
- if title == None: title = ''
- if doi == None: doi = ''
- if pubmed_id == None: pubmed_id = ''
- if import_url == None: import_url = ''
- paper = Paper.objects.create(doi=doi, pubmed_id=pubmed_id,
import_url=import_url, title=title)
- created = True
- if provider:
- provider.import_paper(data, paper=paper)
-
- return paper, created
+ if not doi:
+ doi = ''
+ if not pubmed_id:
+ pubmed_id = ''
+ if not import_url:
+ import_url = ''
+ if not title:
+ title = ''
+ paper = Paper.objects.create(doi=doi, pubmed_id=pubmed_id,
+ import_url=import_url, title=title)
+
+ if provider:
+ # Get the paper from the search provider
+ provider.import_paper_after_search(data, paper=paper,
callback=callback)
+ else:
+ callback(paper)
+
+

def update_paper_from_bibtex_html(paper, html):
@@ -193,6 +211,7 @@

# create our paper if not provided for us
if not paper:
+ # FIXME
paper, created =
get_or_create_paper_via(doi=bibtex.get('doi'), title=bibtex.get('title'))
if created:
log_info('creating paper: %s' % str(paper))
@@ -244,8 +263,8 @@

return paper

-#TODO: Refactor import_pdf into a new function
-
+
+#TODO: Refactor import_pdf into a new function
def import_citation(url, paper=None, callback=None):

log_info('Importing URL: %s' % url)
@@ -280,6 +299,7 @@

if not paper:
md5_hexdigest = get_md5_hexdigest_from_data(data)
+ # FIXME
paper, created =
get_or_create_paper_via(full_text_md5=md5_hexdigest)
if created:
if not filename:
@@ -322,6 +342,7 @@
if active_threads.has_key(thread.get_ident()):
del active_threads[ thread.get_ident() ]

+
def _import_google_scholar_citation(params, paper=None):
log_info('downloading google scholar citation: %s' % params['url'])
try:
@@ -350,6 +371,7 @@
p_html_a = re.compile("<a [^>]+>" , re.IGNORECASE)
p_html_a_href = re.compile('''href *= *['"]([^'^"]+)['"]''' ,
re.IGNORECASE)

+
def _import_unknown_citation(data, orig_url, paper=None):

# soupify
@@ -406,7 +428,6 @@
return import_citation(pdf_link)

-
def find_and_attach_pdf(paper, urls, visited_urls=set()):

# search for a PDF linked directly
@@ -471,3 +492,208 @@
continue
if promising_links: print promising_links
if find_and_attach_pdf(paper, list(promising_links),
visited_urls=visited_urls): return
+
+
+def update_paper_from_dictionary(paper_info, paper):
+ assert paper is not None
+ # Journal
+ if 'journal' in paper_info:
+ #TODO: Volume etc
+ if 'issue' in paper_info:
+ source, created =
Source.objects.get_or_create(name=paper_info['journal'],
+
issue=paper_info['issue'])
+ else:
+ source, created =
Source.objects.get_or_create(name=paper_info['journal'])
+ if created:
+ source.save()
+ paper.source = source
+
+ if 'pages' in paper_info:
+ paper.source_pages = paper_info['pages']
+
+ # Authors
+ if 'authors' in paper_info:
+ for author in paper_info['authors']:
+ author_obj, created = Author.objects.get_or_create(name=author)
+ paper.authors.add(author_obj)
+ if created:
+ author_obj.save()
+ # Simple attributes
+ attributes = ['title', 'abstract', 'year']
+
+ for attr in attributes:
+ if attr in paper_info:
+ paper.__setattr__(attr, paper_info[attr])
+
+ paper.save()
+ return paper
+
+class WebSearchProvider(object):
+ '''
+ Base class for all web search providers, i.e. websites or web APIs that
+ return a number of search results for a search string.
+
+ Implementation of new search providers should derive from this class
and
+ have to provide the following attributes as class attributes:
+ * `name`: A human readable name that is used in the left column of the
GUI,
+ e.g. "Google Scholar".
+ * `label`: A simple name that is used for saving searches to the
database,
+ e.g. "gscholar"
+ * `icon` : The name of an icon (expected in the `icons` subdirectory
+ currently) used in the left column of the GUI, e.g.
+ "favicon_google.ico". If no icon name is provided, a
standard
+ icon is used.
+ * `unique_keys`: Defining under what circumstances a search result
should be
+ considered a duplicate of an existing paper in the
+ database. For a PubMED search, for example, this
should be
+ 'pubmed_id'. If the unique key is not set, 'doi' is
used
+ as a default.
+
+
+ A search provider has to provide the following methods:
+ * `search_async(self, search_string, callback)`
+ This method receives a search string from the GUI and should call the
+ callback with a list of search results where each single result is a
+ dictionary containing all the information that could be fetched from
the
+ webpage, e.g.:
+ [{'title': 'A paper title', 'authors': ['Author A', 'Author B']},
+ {'title': 'Another paper', 'authors': ['Author C'],
+ 'import_url': 'http://example.com/paper.pdf'}]
+ In addition, each paper can also contain arbitrary additional data as
the
+ value for a 'data' key. This could for example be used to save the full
+ HTML code of a search result (which might be useful for an import of
this
+ paper) as opposed to only the extracted information.
+ This method should not block but use the :class:`AsyncSoupSession`
object
+ `importer.soup_session` for getting the information.
+
+ * `import_paper_after_search(self, data, paper, callback,
error_callback)`
+ This method receives the `data` (if any) previously returned from the
+ :method:`parse_website_response` method and a :class:`Paper` object,
already
+ filled with the information previously returned from the search. It
should
+ call the callback when it is finished processing (which may include
getting
+ more information from webpages -- in this case
the :class:`AsyncSoupSession`
+ object `importer.soup_session` should be used to asynchronously fetch
the
+ page(s). The callback function has to be called with the
same :class:`Paper`
+ object (with any additional information now available filled in,
generated
+ :class:`Author` objects, etc.) as the first argument. Optionally, a
list of
+ URL strings can be given as the second argument, these URLs will be
used
+ in the order they are given, i.e. if fetching the document from the
first
+ one is not successful, the second one will be tried etc.
+
+ The :method:`__init__` method of the subclass has to call the
+ :method:`__init__` method of the superclass.
+ '''
+
+ unique_key = 'doi'
+
+ def __init__(self):
+ # Remember previous search results so that no new search is
necessary.
+ # Useful especially if switching between libraries/searches in the
left
+ # pane
+ self.search_cache = {}
+
+ def __str__(self):
+ return self.name
+
+ def clear_cache(self, text):
+ if text in self.search_cache:
+ del self.search_cache[text]
+
+ def search(self, search_string, callback, error_callback):
+ if not search_string:
+ return []
+
+ if search_string in self.search_cache:
+ return self.search_cache[search_string]
+
+ try:
+ self.search_async(search_string, callback, error_callback)
+ except Exception as ex:
+ error_callback(ex, None)
+
+ def import_paper_after_search(self, data, paper, callback):
+ raise NotImplementedError()
+
+ def search_async(self, search_string, callback, error_callback):
+ raise NotImplementedError()
+
+class SimpleWebSearchProvider(WebSearchProvider):
+ '''
+ Convenience class for web searches that do a single request to a
website for
+ a search and do not have to perform additional web requests to get more
+ detailed info for a paper chosen for import.
+
+ Such web search providers need only provide three functions:
+ * prepare_search_message
+ * parse_response
+ * fill_in_paper_info
+
+ '''
+
+ def __init__(self):
+ WebSearchProvider.__init__(self)
+
+ def search_async(self, search_string, callback, error_callback):
+ try:
+ # Call the method defined in the subclass
+ message = self.prepare_search_message(search_string)
+
+ def my_callback(session, message, user_data):
+ self.response_received(message, callback, error_callback,
+ user_data)
+
+ # Provide a tuple of label and search string as `user_data` to
the
+ # callback -- this way it is clear to what search a
result/error belongs
+ soup_session.queue_message(message, my_callback, (self.label,
search_string))
+ except Exception as ex:
+ error_callback(ex, search_string)
+
+ def response_received(self, message, callback, error_callback,
+ user_data):
+ '''
+ Will be called when the server returns a response.
+ '''
+ log_info('Received a response for %s' % str(user_data))
+ if message.status_code == Soup.KnownStatusCode.OK:
+ #try:
+ callback(user_data,
self.parse_response(message.response_body.data))
+ #except Exception as ex:
+ # error_callback(ex, user_data)
+ else:
+ error_callback(message.status_code, user_data)
+
+ def import_paper_after_search(self, data, paper, callback):
+ try:
+ callback(self.fill_in_paper_info(data, paper))
+ except Exception as ex:
+ log_error(str(ex))
+
+ #
--------------------------------------------------------------------------
+ # Methods to overwrite in sub classes
+ #
--------------------------------------------------------------------------
+ def prepare_search_message(self, search_string):
+ raise NotImplementedError()
+
+ def parse_response(self, response):
+ raise NotImplementedError()
+
+ def fill_in_paper_info(self, data, paper=None):
+ raise NotImplementedError()
+
+
+class WebRessourceProvider(object):
+ '''
+ In addition to searching, a :class:`WebRessourceProvider` can provide
+ information about a single paper, e.g. a PubMED provider can
+ provide paper data given a PubMED ID. A method gives information about
its
+ capabilities:
+ * `provides_infos_for_attribute(self, attribute)`
+ `attribute` is a string like 'pubmed_id'
+
+ If the class return True for any attribute in the previous method
(otherwise
+ the class is pretty useless...), it has to provide another method:
+ * `get_infos_for_attribute(self, paper, attribute, value, callback,
error_callback)`
+ '''
+
+ def provides_infos_for_attribute(self, attribute):
+ return False
=======================================
--- /importer/google_scholar.py Tue Feb 21 11:03:48 2012
+++ /importer/google_scholar.py Sun Mar 4 13:24:20 2012
@@ -1,125 +1,108 @@
-import urllib, traceback, thread
-
-from BeautifulSoup import BeautifulSoup
+import urllib, hashlib, random, traceback
+
+from gi.repository import Soup # @UnresolvedImport
+
+import BeautifulSoup
from django.template import defaultfilters

import openanything
from logger import log_debug, log_info, log_error
-from importer import html_strip, update_paper_from_bibtex_html,
find_and_attach_pdf
-
-BASE_URL = 'http://scholar.google.com'
+from importer import *
+
+BASE_URL = 'http://scholar.google.com/'

-class GoogleScholarSearch(object):
+class GoogleScholarSearch(SimpleWebSearchProvider):
+
+ name = 'Google Scholar'
+ label = 'google_scholar'
+ icon = 'favicon_google.ico'
+ unique_key = 'import_url'

def __init__(self):
- # set our google scholar prefs (cookie-based)
- #FIXME: This is blocking...
- log_debug('Google Scholar: Trying to set a cookie so that we get
100 results and bibtex links')
- #FIXME: Does not seem to work...
- params = openanything.fetch(BASE_URL + \
- '/scholar_setprefs?num=100&scis=yes&scisf=4&submit=Save+Preferences')
- log_debug('Google Scholar: Returned status: %d' % params['status'])
-
- self.name = 'Google Scholar'
- self.label = 'google_scholar'
- self.icon = 'favicon_google.ico'
-
- # Remember previous search results so that no new search is
necessary.
- # Useful especially if switching between libraries/searches in the
left
- # pane
- self.search_cache = {}
-
- def clear_cache(self, text):
- if text in self.search_cache:
- del self.search_cache[text]
-
- def unique_key(self):
- return 'import_url'
-
- def search(self, search_text):
- '''
- Returns a list of dictionaries: The PUBMED results for the given
search
- query
- '''
- if not search_text:
- return []
-
- if search_text in self.search_cache:
- return self.search_cache[search_text]
-
+ SimpleWebSearchProvider.__init__(self)
+
+ # create a "google ID" used later for setting our preferences
instead
+ # of using a cookie
+ self.google_id = hashlib.md5(str(random.random())).hexdigest()[:16]
+
+ def prepare_search_message(self, search_string):
+
+ uri_string = BASE_URL + 'scholar?' + urllib.urlencode({'q':
search_string})
+ message = Soup.Message.new(method='GET',
+ uri_string=uri_string)
+ log_info('Starting google scholar request with uri_string="%s"' %
uri_string)
+ # This tells Google Scholar to return links to BibTeX
+ message.request_headers.append('Cookie',
+ 'GSP=ID=%s:CF=4' % self.google_id)
+ return message
+
+ def parse_response(self, response):
+ node = BeautifulSoup.BeautifulSoup(response,
+
convertEntities=BeautifulSoup.BeautifulSoup.HTML_ENTITIES)
papers = []
- try:
- query = BASE_URL + '/scholar?q=%s' % \
- defaultfilters.urlencode(search_text)
- log_info('Starting google scholar search for query "%s"' %
query)
- params = openanything.fetch(query)
- if params['status'] == 200 or params['status'] == 302:
- node = BeautifulSoup(params['data'],
-
convertEntities=BeautifulSoup.HTML_ENTITIES)
- for result in node.findAll('div', attrs={'class': 'gs_r'}):
- paper = {}
- #print '==========================================='
-
- try:
- title_node = result.findAll('h3',
-
attrs={'class': 'gs_rt'})[0]
- #Can be a link or plain text
- title_link = title_node.findAll('a')
- if title_link:
- log_debug('title_link: %s' % \
- title_link[0].prettify())
- paper['title'] = title_link[0].string
- paper['import_url'] = title_link[0]['href']
- else:
- paper['title'] = title_node.string
- paper['import_url'] = ''
-
- if not paper['import_url'].startswith('http'):
- paper['import_url'] = BASE_URL +
paper['import_url']
-
+ for result in node.findAll('div', attrs={'class': 'gs_r'}):
+ paper = {}
+ try:
+ title_node = result.findAll('h3',
+ attrs={'class': 'gs_rt'})[0]
+ #Can be a link or plain text
+ title_link = title_node.findAll('a')
+ if title_link:
+ log_debug('title_link: %s' % \
+ title_link[0].prettify())
+ paper['title'] = title_link[0].string
+ paper['import_url'] = title_link[0]['href']
+ else:
+ paper['title'] = title_node.string
+ paper['import_url'] = ''
+
+ if not paper['import_url'].startswith('http'):
+ paper['import_url'] = BASE_URL + paper['import_url']
+
+ try:
+ author_journal_publisher = result.findAll('div',
+ attrs={'class': 'gs_a'})[0]
+ log_debug('Author string: %s' % \
+ str(author_journal_publisher.text))
+ authors, journal, publisher = \
+ author_journal_publisher.text.split(' - ')
+ paper['authors'] = authors.split(',')
+ journal_year = journal.split(',')
+ if len(journal_year) == 2:
+ paper['journal'] = journal_year[0]
+ paper['year'] = journal_year[1]
+ elif len(journal_year) == 1: # might be a year or a
journal
try:
- author_journal_publisher =
result.findAll('div',
-
attrs={'class' : 'gs_a'})[0]
- log_debug('Author string: %s' % \
-
str(author_journal_publisher.text))
- authors, journal, publisher = \
-
author_journal_publisher.text.split(' - ')
- paper['authors'] = authors.split(',')
- journal_year = journal.split(',')
- if len(journal_year) == 2:
- paper['journal'] = journal_year[0]
- paper['year'] = journal_year[1]
- elif len(journal_year) == 1: # might be a year
or a journal
- try:
- paper['year'] = int(journal_year[0])
- except ValueError:
- paper['journal'] = journal_year[0]
- paper['publisher'] = publisher
- except:
- pass
-
- try:
- paper['abstract'] =
html_strip(result.findAll('div', attrs='gs_rs')[0].text)
- except:
- pass
-
- # Also attach the html data so it can be used
later for
- # importing the document
- paper['data'] = result
- except:
- traceback.print_exc()
- papers.append(paper)
-
- self.search_cache[search_text] = papers
- return papers
- else:
- log_error('Google scholar replied with error code %d for
query: %s' % \
- (params['status'], search_text))
- except:
- traceback.print_exc()
-
- def import_paper(self, data, paper=None):
+ paper['year'] = str(int(journal_year[0]))
+ except ValueError:
+ paper['journal'] = journal_year[0]
+ paper['publisher'] = publisher
+ except:
+ pass
+
+ try:
+ paper['abstract'] = html_strip(result.findAll('div',
+
attrs='gs_rs')[0].text)
+ except:
+ pass
+
+ # Also attach the html data so it can be used later for
+ # importing the document
+ paper['data'] = result
+ except:
+ traceback.print_exc()
+ papers.append(paper)
+
+ return papers
+
+ def _got_bibtex(self, message, paper, callback):
+ if message.status_code == Soup.KnownStatusCode.OK:
+ paper = update_paper_from_bibtex_html(paper,
+
message.request_body.data)
+ callback(paper)
+
+ def import_paper_after_search(self, data, paper, callback):
log_info('Trying to import google scholar citation "%s"' %
paper.title)
try:
citations = data.findAll('div', {'class': 'gs_fl'})[0]
@@ -127,10 +110,15 @@
for link in citations.findAll('a'):
log_debug('Link: %s' % str(link))
if link['href'].startswith('/scholar.bib'):
- log_debug('Found BibTex link: ', link['href'])
- data_bibtex = openanything.fetch(BASE_URL +
link['href'])
- if data_bibtex['status'] == 200 or
data_bibtex['status'] == 302:
- paper = update_paper_from_bibtex_html(paper,
data_bibtex['data'])
+ log_debug('Found BibTex link: %s' % link['href'])
+
+ def bibtex_callback(session, message, user_data):
+ self._got_bibtex(message, paper, callback)
+
+ message = Soup.Message.new(method='GET',
+ uri_string=BASE_URL +
link['href'])
+ soup_session.queue_message(message, bibtex_callback,
None)
+
return
link = data.findAll('div', {'class': 'gs_ggs gs_fl'})[0]
find_and_attach_pdf(paper, urls=[x['href'] for x in
link.findAll('a') ])
@@ -138,6 +126,3 @@
return paper
except:
traceback.print_exc()
-
- def __str__(self):
- return "Google Scholar"
=======================================
--- /importer/jstor.py Tue Feb 21 11:03:48 2012
+++ /importer/jstor.py Sun Mar 4 13:24:20 2012
@@ -1,14 +1,92 @@
-class JSTORSearch(object):
+from gi.repository import Soup
+from BeautifulSoup import BeautifulStoneSoup
+from importer import SimpleWebSearchProvider, update_paper_from_dictionary
+from gPapers.models import Paper
+
+from logger import *
+
+QUERY_STRING = 'http://dfr.jstor.org/sru/?version=1.1&' + \
+ 'operation=searchRetrieve&query=%(query)s&' + \
+ 'maximumRecords=%(max)d&' + \
+ 'recordSchema=info:srw/schema/srw_jstor'
+
+
+class JSTORSearch(SimpleWebSearchProvider):
+
+ name = 'JSTOR'
+ label = 'jstor'
+ icon = 'favicon_jstor.ico'

def __init__(self):
- self.name = 'JSTOR'
- self.label = 'jstor'
- self.icon = 'favicon_jstor.ico'
- self.search_cache = {}
-
- def clear_cache(self, text):
- if text in self.search_cache:
- del self.search_cache[text]
-
- def search(self, search_text):
- return [] #Not implemented yet
+ SimpleWebSearchProvider.__init__(self)
+ # TODO: Make this configurable
+ self.max_results = 20
+
+ def prepare_search_message(self, search_string):
+ return Soup.Message.new(method='GET',
+ uri_string=QUERY_STRING % {'query':
search_string,
+ 'max' :
self.max_results})
+
+ def _parse_result(self, result):
+ '''
+ Parses a single result (a "swr:recorddata" in JSTOR's XML) and
returns
+ a dictionary with the info (title, authors, etc.).
+ '''
+ paper = {}
+ authors = []
+ for author in result.findAll('jstor:author'):
+ if author:
+ authors.append(author.string)
+ if authors:
+ paper['authors'] = authors
+
+ # define mappings from the JSTOR names to our own keywords
+ mappings = {'title': 'jstor:title',
+ 'doi': 'jstor:id',
+ 'abstract': 'jstor:abstract',
+ 'journal': 'jstor:journaltitle',
+ 'volume': 'jstor:volume',
+ 'issue': 'jstor:issue',
+ 'year': 'jstor:year',
+ 'pages': 'jstor:pagerange',
+ 'publisher': 'jstor:publisher'
+ }
+
+ for our_key, jstor_key in mappings.items():
+ try:
+ paper[our_key] = result.find(jstor_key).string
+ except:
+ pass
+
+ # The year is a string like "YEAR: 2012" in JSTOR
+ if 'year' in paper:
+ try:
+ int(paper['year'])
+ except ValueError:
+ paper['year'] = paper['year'][-5:]
+
+ return paper
+
+ def parse_response(self, response):
+ parsed = BeautifulStoneSoup(response)
+ papers = []
+ for result in parsed.find('srw:records').findAll('srw:record'):
+ result = result.find('srw:recorddata')
+ log_debug('Single result: %s' % result.prettify())
+
+ paper = self._parse_result(result)
+
+ log_debug('JSTOR paper info: %s' % str(paper))
+
+ # Add the full data, useful for later import
+ paper['data'] = result
+ papers.append(paper)
+
+ return papers
+
+ def fill_in_paper_info(self, data, paper=None):
+ if not paper:
+ paper = Paper()
+ paper_info = self._parse_result(data)
+
+ return update_paper_from_dictionary(paper_info, paper)
=======================================
--- /importer/pubmed.py Tue Feb 21 11:03:48 2012
+++ /importer/pubmed.py Sun Mar 4 13:24:20 2012
@@ -1,11 +1,13 @@
import urllib

import BeautifulSoup
+from gi.repository import Soup # @UnresolvedImport

from logger import log_debug, log_info, log_error
from BeautifulSoup import BeautifulStoneSoup

from gPapers.models import *
+from importer import WebSearchProvider, soup_session,
update_paper_from_dictionary

BASE_URL = 'http://eutils.ncbi.nlm.nih.gov/entrez/eutils/'
ESEARCH_QUERY = 'esearch.fcgi?db=pubmed&term=%s&usehistory=y'
@@ -13,156 +15,152 @@
EFETCH_QUERY = 'efetch.fcgi?db=pubmed&id=%s&retmode=xml'

-class PubMedSearch(object):
+class PubMedSearch(WebSearchProvider):
+
+ name = 'PubMed'
+ label = 'pubmed'
+ icon = 'favicon_pubmed.ico'
+ unique_key = 'pubmed_id'

def __init__(self):
- self.search_cache = {}
- self.name = 'PubMed'
- self.label = 'pubmed'
- self.icon = 'favicon_pubmed.ico'
-
- def unique_key(self):
- return 'pubmed_id'
-
- def clear_cache(self, text):
- if text in self.search_cache:
- del self.search_cache[text]
-
- def search(self, search_text):
- '''
- Returns a list of dictionaries: The PUBMED results for the given
search
- query
- '''
- if not search_text:
- return [] # Do not make empty queries
-
- if search_text in self.search_cache:
- return self.search_cache[search_text]
-
- # First do a query only for ids that is saved on the server
- log_debug('Starting Pubmed query for string "%s"' % search_text)
- query = BASE_URL + ESEARCH_QUERY % urllib.quote_plus(search_text)
- response = urllib.urlopen(query)
- if not (response.getcode() == 200 or response.getcode() == 302):
- log_error('Pubmed replied with error code %d for query: %s' % \
- (response.getcode(), query))
- #TODO: Show a dialog or handle it differently?
- return []
-
- parsed_response = BeautifulSoup.BeautifulStoneSoup(response)
-
- # Check wether there were any hits at all
- if int(parsed_response.esearchresult.count.string) == 0:
- log_info('No hits for search string "%s"' % search_text)
- self.search_cache[search_text] = []
- return []
-
- web_env = parsed_response.esearchresult.webenv.string
- query_key = parsed_response.esearchresult.querykey.string
- response.close()
-
- # Download the summaries
- log_debug('Continuing Pubmed query (downloading summaries)')
- query = BASE_URL + ESUMMARY_QUERY % (query_key, web_env)
- response = urllib.urlopen(query)
- if not (response.getcode() == 200 or response.getcode() == 302):
- log_error('Pubmed replied with error code %d for query: %s' % \
- (response.getcode(), query))
- parsed_response = BeautifulSoup.BeautifulStoneSoup(response)
-
- # get information for all documents
- documents = parsed_response.esummaryresult.findAll('docsum')
- papers = []
- for document in documents:
- info = {}
- # Extract information
- info['pubmed_id'] = str(document.id.string)
-
- doi = document.findAll('item', {'name': 'doi'})
- if doi:
- info['doi'] = doi[0].string
- info['import_url'] = 'http://dx.doi.org/' + info['doi']
-
- info['title'] = document.findAll('item',
- {'name': 'Title'})[0].string
- info['authors'] = [str(author.string) for author in \
- document.findAll('item',
- {'name': 'Author'})]
- info['journal'] = document.findAll('item',
-
{'name': 'FullJournalName'})[0].string
-
- pubdate = document.findAll('item', {'name': 'PubDate'})
- if pubdate and pubdate[0]:
- info['year'] = pubdate[0].string[:4]
-
- #TODO: Retrieve abstract
-
- papers.append(info)
-
- self.search_cache[search_text] = papers
- return papers
-
- def import_paper(self, data, paper=None):
- assert paper.pubmed_id
- log_info('Trying to import pubmed citation with id %s' %
paper.pubmed_id)
- query = BASE_URL + EFETCH_QUERY % paper.pubmed_id
- try:
- response = urllib.urlopen(query)
- if not (response.getcode() == 200 or response.getcode() ==
302):
- log_error('Pubmed replied with error code %d for
query: %s' % \
- (response.getcode(), query))
- return paper
- parsed_response = BeautifulStoneSoup(response)
-
+ WebSearchProvider.__init__(self)
+
+ def _ids_received(self, message, callback, error_callback, user_data):
+ if not message.status_code == Soup.KnownStatusCode.OK:
+ error_callback('Pubmed replied with error code %d.' %
message.status_code, user_data)
+ else:
+ parsed_response =
BeautifulSoup.BeautifulStoneSoup(message.response_body.data)
+
+ # Check wether there were any hits at all
+ if int(parsed_response.esearchresult.count.string) == 0:
+ log_info('No hits for search string "%s"' % user_data[1])
+ return # Nothing to do anymore
+
+ # Continue with a second request asking for the summaries
+ web_env = parsed_response.esearchresult.webenv.string
+ query_key = parsed_response.esearchresult.querykey.string
+ log_debug('Continuing Pubmed query (downloading summaries)')
+ query = BASE_URL + ESUMMARY_QUERY % (query_key, web_env)
+
+ message = Soup.Message.new(method='GET', uri_string=query)
+
+ def mycallback(session, message, user_data):
+ self._summaries_received(message, callback, error_callback,
+ user_data)
+
+ soup_session.queue_message(message, mycallback, user_data)
+
+ def _summaries_received(self, message, callback, error_callback,
user_data):
+ if not message.status_code == Soup.KnownStatusCode.OK:
+ error_callback('Pubmed replied with error code %d.' %
message.status_code, user_data)
+ else:
+ parsed_response =
BeautifulSoup.BeautifulStoneSoup(message.response_body.data)
+
+ # get information for all documents
+ documents = parsed_response.esummaryresult.findAll('docsum')
+ papers = []
+ for document in documents:
+ info = {}
+ # Extract information
+ info['pubmed_id'] = str(document.id.string)
+
+ doi = document.findAll('item', {'name': 'doi'})
+ if doi:
+ info['doi'] = doi[0].string
+ info['import_url'] = 'http://dx.doi.org/' + info['doi']
+
+ info['title'] = document.findAll('item',
+
{'name': 'Title'})[0].string
+ info['authors'] = [str(author.string) for author in \
+ document.findAll('item',
+
{'name': 'Author'})]
+ info['journal'] = document.findAll('item',
+
{'name': 'FullJournalName'})[0].string
+
+ pubdate = document.findAll('item', {'name': 'PubDate'})
+ if pubdate and pubdate[0]:
+ info['year'] = pubdate[0].string[:4]
+
+ #TODO: Retrieve abstract
+
+ papers.append(info)
+
+ callback(user_data, papers)
+
+ def search_async(self, search_text, callback, error_callback):
+ '''
+ Returns a list of dictionaries: The PUBMED results for the given
search
+ query
+ '''
+
+ # First do a query only for ids that is saved on the server
+ log_debug('Starting Pubmed query for string "%s"' % search_text)
+ query = BASE_URL + ESEARCH_QUERY % urllib.quote_plus(search_text)
+ message = Soup.Message.new(method='GET', uri_string=query)
+
+ def mycallback(session, message, user_data):
+ self._ids_received(message, callback, error_callback,
user_data)
+
+ soup_session.queue_message(message, mycallback, (self.label,
search_text))
+
+ def _paper_info_received(self, message, callback, paper, user_data):
+ if not message.status_code == Soup.KnownStatusCode.OK:
+ log_error('Pubmed replied with error code %d for paper_info
with id: %s' % \
+ (message.status_code, user_data[1]))
+ else:
+ parsed_response =
BeautifulStoneSoup(message.response_body.data)
+ paper_info = {}
# Journal
try:
journal = parsed_response.findAll('journal')[0]
- name = journal.findAll('title')[0].text
+ paper_info['journal'] = journal.findAll('title')[0].text
log_debug('Journal name: %s' % name)
try:
- issue = journal.findAll('issue')[0].text
+ paper_info['issue'] = journal.findAll('issue')[0].text
except:
- issue = ''
+ pass
log_debug('Journal issue: %s' % issue)
- #TODO: Volume etc
- source, created = Source.objects.get_or_create(name=name,
- issue=issue)
- if created:
- source.save()
- paper.source = source
- paper.source_pages =
parsed_response.findAll('medlinepgn')[0].text
- log_debug('Pages: %s' % paper.source_pages)
+
+ paper_info['pages'] =
parsed_response.findAll('medlinepgn')[0].text
+ log_debug('Pages: %s' % paper_info.source_pages)
except Exception as ex:
pass

# Title and abstract
try:
- paper.title =
parsed_response.findAll('articletitle')[0].text
- log_debug('Title: %s' % paper.title)
- paper.abstract =
parsed_response.findAll('abstracttext')[0].text
- log_debug('Abstract: %s' % paper.abstract)
+ paper_info['title'] =
parsed_response.findAll('articletitle')[0].text
+ log_debug('Title: %s' % paper_info.title)
+ paper_info['abstract'] =
parsed_response.findAll('abstracttext')[0].text
+ log_debug('Abstract: %s' % paper_info.abstract)
except Exception as ex:
pass

# Authors
try:
+ all_authors = []
authors = parsed_response.findAll('author')
for author in authors:
author_name = author.forename.text + ' ' + \
author.lastname.text
log_debug('\tAuthor: %s' % author_name)
- author_obj, created =
Author.objects.get_or_create(name=author_name)
- paper.authors.add(author_obj)
- if created:
- author_obj.save()
+ all_authors.append(author_name)
+ if all_authors:
+ paper_info['authors'] = all_authors
except Exception as ex:
traceback.print_stack()

- except Exception as ex:
- log_error('Downloading paper with id %s failed: %s' % \
- (paper.pubmed_id, ex))
-
- return paper
-
- def __str__(self):
- return "PubMed"
+ update_paper_from_dictionary(paper_info, paper)
+
+ callback(paper)
+
+ def import_paper_after_search(self, data, paper, callback):
+ assert paper.pubmed_id
+ log_info('Trying to import pubmed citation with id %s' %
paper.pubmed_id)
+ query = BASE_URL + EFETCH_QUERY % paper.pubmed_id
+ message = Soup.Message.new(method='GET', uri_string=query)
+
+ def mycallback(session, message, user_data):
+ self._paper_info_received(message, callback, paper, user_data)
+
+ soup_session.queue_message(message, mycallback, (self.label,
+ paper.pubmed_id))

==============================================================================
Revision: 0c02fe87175e
Author: Marcel Stimberg <marcel...@gmail.com>
Date: Sun Mar 4 13:25:39 2012
Log: Remove unused code
http://code.google.com/p/gpapers/source/detail?r=0c02fe87175e

Modified:
/gPapers.py

=======================================
--- /gPapers.py Sun Mar 4 13:24:20 2012
+++ /gPapers.py Sun Mar 4 13:25:39 2012
@@ -273,14 +273,6 @@
if paper:
paper.delete()

-def show_html_error_dialog(code):
- Gdk.threads_enter()
- error = Gtk.MessageDialog(type=Gtk.MessageType.ERROR,
buttons=Gtk.ButtonsType.OK, flags=Gtk.DialogFlags.MODAL)
- error.connect('response', lambda x, y: error.destroy())
- error.set_markup('Unable to Search External Repository\n\nHTTP
Error code: %d' % code)
- error.run()
- Gdk.threads_leave()
-
def row_from_dictionary(info, provider=None):
assert info is not None

==============================================================================
Revision: 529f4a23df3f
Author: Marcel Stimberg <marcel...@gmail.com>
Date: Sun Mar 4 13:55:35 2012
Log: Use GObject.timeout_add instead of a dedicated thread and sleep
for regularly checking the search box
http://code.google.com/p/gpapers/source/detail?r=529f4a23df3f

Modified:
/gPapers.py

=======================================
--- /gPapers.py Sun Mar 4 13:25:39 2012
+++ /gPapers.py Sun Mar 4 13:55:35 2012
@@ -290,7 +290,8 @@

row = (
info.get('id', -1), # paper id
- pango_escape(', '.join([author for author in
info.get('authors', [])])), # authors
+ pango_escape(', '.join([author for author in
+ info.get('authors', [])])), # authors
pango_escape(info.get('title') or ''), # title
pango_escape(info.get('journal') or ''), # journal
info.get('year'), # year
@@ -444,7 +445,13 @@
self.main_window = self.ui.get_object('main_window')
self.main_window.connect("delete-event", lambda x, y: sys.exit(0))
self.init_menu()
+ # Save content of last search to avoid generating too many
+ # search events
+ self.last_middle_pane_search_string = ''
+ self.last_middle_pane_performed_search = ''
+ self.last_middle_pane_search_time = 0.0
self.init_search_box()
+
self.init_left_pane()
self.init_my_library_filter_pane()
self.init_middle_top_pane()
@@ -507,7 +514,9 @@
self.ui.get_object('menuitem_about').connect('activate',
self.show_about_dialog)

def init_search_box(self):
- thread.start_new_thread(self.watch_middle_pane_search, ())
+ # Check the search box for changes every 0.5 seconds
+ GObject.timeout_add(500, lambda x:
self.middle_pane_search_changed(),
+ None)

self.ui.get_object('refresh_middle_pane_search').connect('clicked', lambda
x: self.refresh_middle_pane_search())
self.ui.get_object('clear_middle_pane_search').connect('clicked',
lambda x: self.clear_all_search_and_filters())
self.ui.get_object('save_smart_search').connect('clicked', lambda
x: self.save_smart_search())
@@ -579,18 +588,23 @@

self.last_middle_pane_search_string = None

- #FIXME: Use signal instead -- and refactor
- def watch_middle_pane_search(self):
- self.last_middle_pane_search_string = ''
- while True:
- if self.last_middle_pane_search_string == None or
self.ui.get_object('middle_pane_search').get_text() !=
self.last_middle_pane_search_string:
- self.last_middle_pane_search_string =
self.ui.get_object('middle_pane_search').get_text()
- log_debug('middle search pane string changed')
- #Gdk.threads_enter()
-
#self.ui.get_object('left_pane').get_selection().emit('changed')
- GObject.idle_add(self.select_left_pane_item,
self.ui.get_object('left_pane_selection'))
- #Gdk.threads_leave()
- time.sleep(1)
+
+ def middle_pane_search_changed(self):
+ search_text = self.ui.get_object('middle_pane_search').get_text()
+ # Only initiate a search if the string did not change since the
last
+ # check
+ if search_text != self.last_middle_pane_search_string:
+ log_debug('Search text changed, doing nothing')
+ self.last_middle_pane_search_string = search_text
+ else:
+ self.last_middle_pane_search_string = search_text
+ # The search text was stable and we did not search for this
text yet
+ if self.last_middle_pane_performed_search != search_text:
+ log_debug('Search text changed, initiating search')
+ self.last_middle_pane_performed_search = search_text
+
self.select_left_pane_item(self.ui.get_object('left_pane_selection'))
+
+ return True # we do want repeated calls of the timeout

def init_left_pane(self):
'''

==============================================================================
Revision: 5334b0ee3ac8
Author: Marcel Stimberg <marcel...@gmail.com>
Date: Sun Mar 4 14:28:26 2012
Log: remove unused code, document code
http://code.google.com/p/gpapers/source/detail?r=5334b0ee3ac8

Modified:
/gPapers.py
/importer/__init__.py

=======================================
--- /gPapers.py Sun Mar 4 13:55:35 2012
+++ /gPapers.py Sun Mar 4 14:28:26 2012
@@ -18,14 +18,9 @@
# with this program; if not, write to the Free Software Foundation,
Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.

-import commands, dircache, getopt, math, os, re, string, sys, thread,
threading, time, traceback, urllib
+import commands, math, os, sys, thread, threading, time, traceback
import mimetypes
-from datetime import date, datetime, timedelta
-from time import strptime
-
-import BeautifulSoup
-
-import sqlite3
+from datetime import datetime, timedelta

from logger import *
log_level_debug()
@@ -44,14 +39,12 @@

# GUI imports
try:
- import gi
from gi.repository import GObject
from gi.repository import Gdk
from gi.repository import Gtk
from gi.repository import GdkPixbuf
from gi.repository import Pango
from gi.repository import Poppler
- import cairo
GObject.threads_init()
except:
traceback.print_exc()
@@ -66,17 +59,17 @@

import settings
import desktop, openanything
-from django.template import defaultfilters
+
os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'
import django.core.management
django.core.management.setup_environ(settings)
from django.db.models import Q
-from django.template import defaultfilters
+
import deseb
from gPapers.models import *
import importer
-from importer import pango_escape
-from importer import html_strip, pango_escape, get_md5_hexdigest_from_data
+
+from importer import pango_escape, get_md5_hexdigest_from_data
from importer import pubmed, google_scholar, jstor

NOTE_ICON =
GdkPixbuf.Pixbuf.new_from_file(os.path.join(RUN_FROM_DIR, 'icons', 'note.png'))
@@ -1838,18 +1831,6 @@
Organization.objects.get(id=id).delete()
self.refresh_my_library_filter_pane()

- def
update_middle_top_pane_from_row_list_if_we_are_still_the_preffered_thread(self,
rows):
- middle_top_pane = self.ui.get_object('middle_top_pane')
- for column in middle_top_pane.get_columns():
- column.set_sort_indicator(False)
- if self.current_middle_top_pane_refresh_thread_ident ==
thread.get_ident():
- Gdk.threads_enter()
- self.middle_top_pane_model.clear()
- for row in rows:
- self.middle_top_pane_model.append(row)
- middle_top_pane.columns_autosize()
- Gdk.threads_leave()
-
def refresh_middle_pane_from_my_library(self,
refresh_library_filter_pane=True):
self.active_threads[ thread.get_ident() ] = 'searching local
library...'
try:
@@ -1930,7 +1911,7 @@
for paper in papers:
authors = []
for author in paper.authors.order_by('id'):
- authors.append(str(author.name))
+ authors.append(unicode(author.name))
if paper.full_text and
os.path.isfile(paper.full_text.path):
icon =
self.ui.get_object('middle_top_pane').render_icon(Gtk.STOCK_DND,
Gtk.IconSize.MENU)
else:
@@ -1962,7 +1943,9 @@
None,
None
))
-
self.update_middle_top_pane_from_row_list_if_we_are_still_the_preffered_thread(rows)
+ self.middle_top_pane_model.clear()
+ for row in rows:
+ self.middle_top_pane_model.append(row)
self.refresh_my_library_count()
except:
traceback.print_exc()
@@ -1970,11 +1953,10 @@
del self.active_threads[ thread.get_ident() ]

def refresh_my_library_count(self):
- Gdk.threads_enter()
selection = self.ui.get_object('left_pane_selection')
liststore, rows = selection.get_selected()
- liststore.set_value(self.left_pane_model.get_iter((0,)), 0, 'My
Library (%i)' %
Paper.objects.count())
- Gdk.threads_leave()
+ liststore.set_value(self.left_pane_model.get_iter((0,)), 0,
+ 'My Library (%i)' % Paper.objects.count())

def refresh_middle_pane_from_external(self, search_info, results):
'''
=======================================
--- /importer/__init__.py Sun Mar 4 13:24:20 2012
+++ /importer/__init__.py Sun Mar 4 14:28:26 2012
@@ -100,20 +100,6 @@
return match.group()

-#TODO: Remove all GUI code from this module
-def _should_we_reimport_paper(paper):
- Gdk.threads_enter()
- dialog = Gtk.MessageDialog(type=Gtk.MessageType.QUESTION,
buttons=Gtk.ButtonsType.OK_CANCEL, flags=Gtk.DialogFlags.MODAL)
- #dialog.connect('response', lambda x,y: dialog.destroy())
- dialog.set_markup('This paper already exists in your local
library:\n\n"%s"\n(imported on %s)\n\nShould we continue the import,
updating/overwriting the previous entry?' % (paper.title,
str(paper.created.date())))
- dialog.set_default_response(Gtk.ResponseType.OK)
- dialog.show_all()
- response = dialog.run()
- dialog.destroy()
- Gdk.threads_leave()
- return response == Gtk.ResponseType.OK
-
-
def get_or_create_paper_via(callback, id=None, doi=None, pubmed_id=None,
import_url=None, title=None,
full_text_md5=None,
data=None, provider=None):
@@ -342,32 +328,6 @@
if active_threads.has_key(thread.get_ident()):
del active_threads[ thread.get_ident() ]

-
-def _import_google_scholar_citation(params, paper=None):
- log_info('downloading google scholar citation: %s' % params['url'])
- try:
- log_debug('parsing...')
- soup = BeautifulSoup.BeautifulSoup(params['data'])
-
- # search for bibtex link
- def f(paper):
- for a in soup.findAll('a'):
- for c in a.contents:
- if str(c).lower().find('bibtex') != -1:
- log_debug('found bibtex link: %s' % a)
- params_bibtex =
openanything.fetch('http://scholar.google.com' + a['href'])
- if params_bibtex['status'] == 200 or
params_bibtex['status'] == 302:
- paper = update_paper_from_bibtex_html(paper,
params_bibtex['data'])
- return
- f(paper)
-
- find_and_attach_pdf(paper, urls=[ x['href'] for x in
soup.findAll('a', onmousedown=True) ])
-
- log_info('imported paper: %s' % str(paper))
- return paper
- except:
- traceback.print_exc()
-
p_html_a = re.compile("<a [^>]+>" , re.IGNORECASE)
p_html_a_href = re.compile('''href *= *['"]([^'^"]+)['"]''' ,
re.IGNORECASE)

@@ -495,6 +455,14 @@

def update_paper_from_dictionary(paper_info, paper):
+ '''
+ Adds all information from a `paper_info` dictionary to the given
+ :class:`gPapers.model.Paper` object, creating and saving
+ :class:`gPapers.model.Author` and :class:`gPapers.model.Source` objects
+ if necessary.
+
+ Returns the `paper` object.
+ '''
assert paper is not None
# Journal
if 'journal' in paper_info:

==============================================================================
Revision: ca332885ee92
Author: Marcel Stimberg <marcel...@gmail.com>
Date: Sun Mar 4 14:35:06 2012
Log: small unicode-related changes
http://code.google.com/p/gpapers/source/detail?r=ca332885ee92

Modified:
/gPapers.py
/gPapers/models.py

=======================================
--- /gPapers.py Sun Mar 4 14:28:26 2012
+++ /gPapers.py Sun Mar 4 14:35:06 2012
@@ -1753,7 +1753,7 @@

def delete_papers(self, paper_ids):
papers = Paper.objects.in_bulk(paper_ids).values()
- paper_list_text = '\n'.join([ ('"%s"' % str(paper.title))
for paper in papers ])
+ paper_list_text = '\n'.join([ ('"%s"' %
unicode(paper.title)) for paper in papers ])
dialog = Gtk.MessageDialog(type=Gtk.MessageType.QUESTION,
buttons=Gtk.ButtonsType.YES_NO, flags=Gtk.DialogFlags.MODAL)
dialog.set_markup('Really delete the following %s?\n\n%s\n\n' %
(humanize_count(len(papers), 'paper', 'papers', places= -1),
paper_list_text))
dialog.set_default_response(Gtk.ResponseType.NO)
@@ -1762,7 +1762,7 @@
dialog.destroy()
if response == Gtk.ResponseType.YES:
for paper in papers:
- log_info('deleting paper: %s' % str(paper))
+ log_info('deleting paper: %s' % unicode(paper))
paper.delete()
self.refresh_middle_pane_search()

=======================================
--- /gPapers/models.py Thu Feb 16 14:37:40 2012
+++ /gPapers/models.py Sun Mar 4 14:35:06 2012
@@ -232,7 +232,9 @@
list_display = ('id', 'doi', 'title')

def __unicode__(self):
- return 'Paper<%i: %s>' % (self.id, ' '.join([str(self.doi),
str(self.title), str(self.authors.all())]))
+ return 'Paper<%i: %s>' % (self.id, ' '.join([unicode(self.doi),
+ unicode(self.title),
+
unicode(self.authors.all())]))

def pretty_string(self):
return '[' + ', '.join([ author.name for author in
self.get_authors_in_order() ]) + '] ' + self.title
@@ -255,7 +257,7 @@
list_display =
('id', 'referencing_paper', 'line_from_referencing_paper', 'doi_from_referencing_paper', 'referenced_paper', 'line_from_referenced_paper', 'doi_from_referenced_paper')

def __unicode__(self):
- return 'Reference<%s,%s>' % (str(self.referencing_paper),
str(self.referenced_paper))
+ return 'Reference<%s,%s>' % (unicode(self.referencing_paper),
unicode(self.referenced_paper))

class Bookmark(models.Model):

==============================================================================
Revision: deb9b798e357
Author: Marcel Stimberg <marcel...@gmail.com>
Date: Sun Mar 4 15:40:52 2012
Log: Remove some code duplication
http://code.google.com/p/gpapers/source/detail?r=deb9b798e357

Modified:
/gPapers.py

=======================================
--- /gPapers.py Sun Mar 4 14:35:06 2012
+++ /gPapers.py Sun Mar 4 15:40:52 2012
@@ -1776,60 +1776,63 @@
except:
traceback.print_exc()

- def delete_playlist(self, id):
- dialog = Gtk.MessageDialog(type=Gtk.MessageType.QUESTION,
buttons=Gtk.ButtonsType.YES_NO, flags=Gtk.DialogFlags.MODAL)
- dialog.set_markup('Really delete this document collection?')
+ def delete_object(self, text, obj, update_function):
+ '''
+ Asks for confirmation before deleting an object and calling an
update
+ function (e.g. :method:`refresh_left_pane`). Is called by the
+ more specific methods like :method:`delete_playlist` etc.
+ '''
+ dialog = Gtk.MessageDialog(type=Gtk.MessageType.QUESTION,
+ buttons=Gtk.ButtonsType.YES_NO,
+ flags=Gtk.DialogFlags.MODAL)
+ dialog.set_markup(text)
dialog.set_default_response(Gtk.ResponseType.NO)
dialog.show_all()
response = dialog.run()
dialog.destroy()
if response == Gtk.ResponseType.YES:
- Playlist.objects.get(id=id).delete()
- self.refresh_left_pane()
+ obj.delete()
+ update_function()
+
+ def delete_playlist(self, id):
+ '''
+ Ask for confirmation before deleting a document collection
(playlist).
+ '''
+ obj = Playlist.objects.get(id=id)
+ self.delete_object('Really delete this document collection?', obj,
+ self.refresh_left_pane)

def delete_author(self, id):
- dialog = Gtk.MessageDialog(type=Gtk.MessageType.QUESTION,
buttons=Gtk.ButtonsType.YES_NO, flags=Gtk.DialogFlags.MODAL)
- dialog.set_markup('Really delete this author?')
- dialog.set_default_response(Gtk.ResponseType.NO)
- dialog.show_all()
- response = dialog.run()
- dialog.destroy()
- if response == Gtk.ResponseType.YES:
- Author.objects.get(id=id).delete()
- self.refresh_my_library_filter_pane()
+ '''
+ Ask for confirmation before deleting an author.
+ '''
+ obj = Author.objects.get(id=id)
+ self.delete_object('Really delete this author?', obj,
+ self.refresh_my_library_filter_pane())

def delete_bookmark(self, id):
- dialog = Gtk.MessageDialog(type=Gtk.MessageType.QUESTION,
buttons=Gtk.ButtonsType.YES_NO, flags=Gtk.DialogFlags.MODAL)
- dialog.set_markup('Really delete this bookmark?')
- dialog.set_default_response(Gtk.ResponseType.NO)
- dialog.show_all()
- response = dialog.run()
- dialog.destroy()
- if response == Gtk.ResponseType.YES:
- Bookmark.objects.get(id=id).delete()
- self.update_bookmark_pane_from_paper(self.displayed_paper)
+ '''
+ Ask for confirmation before deleting a bookmark.
+ '''
+ obj = Bookmark.objects.get(id=id)
+ self.delete_object('Really delete this bookmark?', obj,
+ lambda :
self.update_bookmark_pane_from_paper(self.displayed_paper))

def delete_source(self, id):
- dialog = Gtk.MessageDialog(type=Gtk.MessageType.QUESTION,
buttons=Gtk.ButtonsType.YES_NO, flags=Gtk.DialogFlags.MODAL)
- dialog.set_markup('Really delete this source?')
- dialog.set_default_response(Gtk.ResponseType.NO)
- dialog.show_all()
- response = dialog.run()
- dialog.destroy()
- if response == Gtk.ResponseType.YES:
- Source.objects.get(id=id).delete()
- self.refresh_my_library_filter_pane()
+ '''
+ Ask for confirmation before deleting a source (e.g. a journal)
+ '''
+ obj = Source.objects.get(id=id)
+ self.delete_object('Really delete this source?', obj,
+ self.refresh_my_library_filter_pane)

def delete_organization(self, id):
- dialog = Gtk.MessageDialog(type=Gtk.MessageType.QUESTION,
buttons=Gtk.ButtonsType.YES_NO, flags=Gtk.DialogFlags.MODAL)
- dialog.set_markup('Really delete this organization?')
- dialog.set_default_response(Gtk.ResponseType.NO)
- dialog.show_all()
- response = dialog.run()
- dialog.destroy()
- if response == Gtk.ResponseType.YES:
- Organization.objects.get(id=id).delete()
- self.refresh_my_library_filter_pane()
+ '''
+ Ask for confirmation before deleting an organization.
+ '''
+ obj = Organization.objects.get(id=id)
+ self.delete_object('Really delete this organization?', obj,
+ self.refresh_my_library_filter_pane)

def refresh_middle_pane_from_my_library(self,
refresh_library_filter_pane=True):
self.active_threads[ thread.get_ident() ] = 'searching local
library...'

==============================================================================
Revision: 00b5d40f5931
Author: Marcel Stimberg <marcel...@gmail.com>
Date: Sun Mar 18 12:00:26 2012
Log: progressing towards an again usable import...
http://code.google.com/p/gpapers/source/detail?r=00b5d40f5931

Added:
/importer/bibtex.py
/importer/pdf_file.py
Modified:
/gPapers.py
/gPapers/models.py
/importer/__init__.py
/importer/google_scholar.py
/importer/jstor.py
/importer/pubmed.py

=======================================
--- /dev/null
+++ /importer/bibtex.py Sun Mar 18 12:00:26 2012
@@ -0,0 +1,218 @@
+# coding: utf8
+import re
+from logger import log_info, log_debug
+import pprint
+""" Pyparsing parser for BibTeX files
+
+A standalone parser using pyparsing.
+
+pyparsing has a simple and expressive syntax so the grammar is easy to
read and
+write.
+
+Matthew Brett 2010
+Simplified BSD license
+"""
+
+from pyparsing import (Regex, Suppress, ZeroOrMore, Group, Optional,
Forward,
+ SkipTo, CaselessLiteral, Dict)
+
+
+class Macro(object):
+ """ Class to encapsulate undefined macro references """
+ def __init__(self, name):
+ self.name = name
+ def __repr__(self):
+ return 'Macro("%s")' % self.name
+ def __eq__(self, other):
+ return self.name == other.name
+ def __ne__(self, other):
+ return self.name != other.name
+
+
+# Character literals
+LCURLY = Suppress('{')
+RCURLY = Suppress('}')
+LPAREN = Suppress('(')
+RPAREN = Suppress(')')
+QUOTE = Suppress('"')
+COMMA = Suppress(',')
+AT = Suppress('@')
+EQUALS = Suppress('=')
+HASH = Suppress('#')
+
+
+def bracketed(expr):
+ """ Return matcher for `expr` between curly brackets or parentheses """
+ return (LPAREN + expr + RPAREN) | (LCURLY + expr + RCURLY)
+
+
+# Define parser components for strings (the hard bit)
+chars_no_curly = Regex(r"[^{}]+")
+chars_no_curly.leaveWhitespace()
+chars_no_quotecurly = Regex(r'[^"{}]+')
+chars_no_quotecurly.leaveWhitespace()
+# Curly string is some stuff without curlies, or nested curly sequences
+curly_string = Forward()
+curly_item = Group(curly_string) | chars_no_curly
+curly_string << LCURLY + ZeroOrMore(curly_item) + RCURLY
+# quoted string is either just stuff within quotes, or stuff within
quotes, within
+# which there is nested curliness
+quoted_item = Group(curly_string) | chars_no_quotecurly
+quoted_string = QUOTE + ZeroOrMore(quoted_item) + QUOTE
+
+# Numbers can just be numbers. Only integers though.
+number = Regex('[0-9]+')
+
+# Basis characters (by exclusion) for variable / field names. The
following
+# list of characters is from the btparse documentation
+any_name = Regex('[^\s"#%\'(),={}]+')
+
+# btparse says, and the test bibs show by experiment, that macro and field
names
+# cannot start with a digit. In fact entry type names cannot start with a
digit
+# either (see tests/bibs). Cite keys can start with a digit
+not_digname = Regex('[^\d\s"#%\'(),={}][^\s"#%\'(),={}]*')
+
+# Comment comments out to end of line
+comment = (AT + CaselessLiteral('comment') +
+ Regex("[\s{(].*").leaveWhitespace())
+
+# The name types with their digiteyness
+not_dig_lower = not_digname.copy().setParseAction(
+ lambda t: t[0].lower())
+macro_def = not_dig_lower.copy()
+macro_ref = not_dig_lower.copy().setParseAction(lambda t :
Macro(t[0].lower()))
+field_name = not_dig_lower.copy()
+# Spaces in names mean they cannot clash with field names
+entry_type = not_dig_lower.setResultsName('entry type')
+cite_key = any_name.setResultsName('cite key')
+# Number has to be before macro name
+string = (number | macro_ref | quoted_string |
+ curly_string)
+
+# There can be hash concatenation
+field_value = string + ZeroOrMore(HASH + string)
+field_def = Group(field_name + EQUALS + field_value)
+entry_contents = Dict(ZeroOrMore(field_def + COMMA) + Optional(field_def))
+
+# Entry is surrounded either by parentheses or curlies
+entry = (AT + entry_type +
+ bracketed(cite_key + COMMA + entry_contents))
+
+# Preamble is a macro-like thing with no name
+preamble = AT + CaselessLiteral('preamble') + bracketed(field_value)
+
+# Macros (aka strings)
+macro_contents = macro_def + EQUALS + field_value
+macro = AT + CaselessLiteral('string') + bracketed(macro_contents)
+
+# Implicit comments
+icomment = SkipTo('@').setParseAction(lambda t : t.insert(0, 'icomment'))
+
+# entries are last in the list (other than the fallback) because they have
+# arbitrary start patterns that would match comments, preamble or macro
+definitions = Group(comment |
+ preamble |
+ macro |
+ entry |
+ icomment)
+
+# Start symbol
+bibfile = ZeroOrMore(definitions)
+
+
+def parse_str(str):
+ return bibfile.parseString(str)
+
+if __name__ == '__main__':
+ # Run basic test
+ txt = """
+@article{10.1371/journal.pone.0020409,
+ author = {Tkačik, Gašper AND Garrigan, Patrick AND Ratliff, Charles
AND Milčinski, Grega AND Klein, Jennifer M. AND Seyfarth, Lucia H. AND
Sterling, Peter AND Brainard, David H. AND Balasubramanian, Vijay},
+ journal = {PLoS ONE},
+ publisher = {Public Library of Science},
+ title = {Natural {I}mages from the Birthplace of the Human Eye},
+ year = {2011},
+ month = {06},
+ volume = {6},
+ url = {http://dx.doi.org/10.1371%2Fjournal.pone.0020409},
+ pages = {e20409},
+ abstract = {
+ Here we introduce a database of calibrated natural images
publicly available through an easy-to-use web interface. Using a Nikon D70
digital SLR camera, we acquired about <inline-formula><inline-graphic
xmlns:xlink="http://www.w3.org/1999/xlink"
xlink:href="info:doi/10.1371/journal.pone.0020409.e001" mimetype="image"
xlink:type="simple"/></inline-formula> six-megapixel images of Okavango
Delta of Botswana, a tropical savanna habitat similar to where the human
eye is thought to have evolved. Some sequences of images were captured
unsystematically while following a baboon troop, while others were designed
to vary a single parameter such as aperture, object distance, time of day
or position on the horizon. Images are available in the raw RGB format and
in grayscale. Images are also available in units relevant to the physiology
of human cone photoreceptors, where pixel values represent the expected
number of photoisomerizations per second for cones sensitive to long (L),
medium (M) and short (S) wavelengths. This database is distributed under a
Creative Commons Attribution-Noncommercial Unported license to facilitate
research in computer vision, psychophysics of perception, and visual
neuroscience.
+ },
+ number = {6},
+ doi = {10.1371/journal.pone.0020409}
+}
+"""
+ pp = pprint.PrettyPrinter(indent=4)
+
+ pp.pprint(mydict)
+
+p_bibtex = re.compile('[@][a-z]+[\s]*{(.*)}', re.IGNORECASE | re.DOTALL)
+
+def latex2unicode(s):
+ """
+ * \`{o} produces a grave accent
+ * \'{o} produces an acute accent
+ * \^{o} produces a circumflex
+ * \"{o} produces an umlaut or dieresis
+ * \H{o} produces a long Hungarian umlaut
+ * \~{o} produces a tilde
+ * \c{c} produces a cedilla
+ * \={o} produces a macron accent (a bar over the letter)
+ * \b{o} produces a bar under the letter
+ * \.{o} produces a dot over the letter
+ * \d{o} produces a dot under the letter
+ * \u{o} produces a breve over the letter
+ * \v{o} produces a "v" over the letter
+ * \t{oo} produces a "tie" (inverted u) over the two letters
+ """
+ # TODO: expand this to really work
+ return s
+
+
+def paper_info_from_bibtex(data):
+
+ # ieee puts s in their bibtex
+ data = data.replace(' ', '\n')
+
+ paper_info = {}
+
+ result = parse_str(data)
+ bibtex = {}
+ # FIXME: This does not handle special cases well...
+ for i, r in enumerate(result[0][2:]):
+ bibtex[r[0].lower()] = ''.join([str(r_part) for r_part in r[1:]])
+
+ # fix for ACM's doi retardedness
+ if bibtex.get('doi', '').startswith('http://dx.doi.org/'):
+ bibtex['doi'] = bibtex['doi'][ len('http://dx.doi.org/'): ]
+ if bibtex.get('doi', '').startswith('http://doi.acm.org/'):
+ bibtex['doi'] = bibtex['doi'][ len('http://doi.acm.org/'): ]
+
+ # Mappings from BibTeX to our keys
+ # TODO: Handle more fields
+ mappings = {'doi': 'doi',
+ 'url': 'import_url',
+ 'title': 'title',
+ 'pages': 'pages',
+ 'abstract': 'abstract',
+ 'journal': 'journal',
+ 'year': 'year',
+ 'publisher': 'publisher'}
+
+ for bibtex_key, our_key in mappings.items():
+ if bibtex_key in bibtex:
+ log_debug('Have key %s' % bibtex_key)
+ paper_info[our_key] = bibtex[bibtex_key]
+
+ # TODO: Handle editors, etc.?
+ if 'author' in bibtex:
+ if ' AND ' in bibtex['author']:
+ paper_info['authors'] = bibtex['author'].split(' AND ')
+ else:
+ paper_info['authors'] = bibtex['author'].split(' and ')
+
+ paper_info['bibtex'] = data
+ log_info('imported paper_info: %s\nFrom bibtex: %s' %
(str(paper_info), str(bibtex)))
+
+ return paper_info
=======================================
--- /dev/null
+++ /importer/pdf_file.py Sun Mar 18 12:00:26 2012
@@ -0,0 +1,86 @@
+import os
+
+from gi.repository import Soup # @UnresolvedImport
+from gi.repository import Gio # @UnresolvedImport
+
+from pdfminer.pdfparser import PDFParser, PDFDocument, resolve1
+
+from logger import log_debug, log_info, log_error
+from importer import RessourceProvider
+
+
+class PdfFileRessourceProvider(RessourceProvider):
+
+ def provides_infos_for_file(self, filename, content_type):
+ return content_type == 'application/pdf'
+
+ def get_infos_for_file(self, filename, callback, error_callback):
+ # TODO: Use async GIO methods?
+ fp = open(filename, 'rb')
+ # Create a PDF parser object associated with the file object.
+ parser = PDFParser(fp)
+ # Create a PDF document object that stores the document structure.
+ doc = PDFDocument()
+ # Connect the parser and document objects.
+ parser.set_document(doc)
+ doc.set_parser(parser)
+ # Initialize
+ doc.initialize()
+ # Extract the metadata
+ for xref in doc.xrefs:
+ info_ref = xref.trailer.get('Info')
+ if info_ref:
+ info = resolve1(info_ref)
+
+ paper_info = {}
+ if info:
+ authors = info.get('Author')
+ if authors:
+ if ';' in authors:
+ author_list = authors.split(';')
+ elif ' AND ' in authors:
+ author_list = authors.split(' AND ')
+ elif ',' in authors:
+ #FIXME: This cuts 'LastName, FirstName' in two...
+ author_list = authors.split(',')
+ else:
+ author_list = [authors]
+
+ paper_info['authors'] = author_list
+ title = info.get('Title')
+ if title:
+ paper_info['title'] = title
+
+ #TODO: Additional metadata?
+ #TODO: What about embedded BibTeX (as done by JabRef)?
+
+ #TODO: Extract text
+
+ callback(paper_info)
+
+ def get_document_for_file(self, filename, callback, error_callback):
+ #TODO: Copy the file to our PDF directory
+ return filename
+
+if __name__ == '__main__':
+ import sys
+ pdf_ressource = PdfFileRessourceProvider()
+ if len(sys.argv) > 1:
+ for arg in sys.argv[1:]:
+ if pdf_ressource.provides_infos_for_file(arg):
+ print 'Infos for %s' % arg
+
+ def print_all(paper_info):
+ for key, value in paper_info.items():
+ print key, ':', value
+
+ paper_info = pdf_ressource.get_infos_for_file(arg,
+ print_all,
+ None)
+
+
+
+
+
+
+
=======================================
--- /gPapers.py Sun Mar 4 15:40:52 2012
+++ /gPapers.py Sun Mar 18 12:00:26 2012
@@ -20,9 +20,11 @@

import commands, math, os, sys, thread, threading, time, traceback
import mimetypes
-from datetime import datetime, timedelta
+from datetime import datetime, timedelta, date

from logger import *
+from importer import bibtex
+
log_level_debug()

RUN_FROM_DIR = os.path.abspath(os.path.dirname(sys.argv[0])) + '/'
@@ -64,6 +66,7 @@
import django.core.management
django.core.management.setup_environ(settings)
from django.db.models import Q
+from django.template import defaultfilters

import deseb
from gPapers.models import *
@@ -280,7 +283,6 @@
except:
pass

-
row = (
info.get('id', -1), # paper id
pango_escape(', '.join([author for author in
@@ -304,10 +306,101 @@
return row

+def paper_from_dictionary(paper_info, paper=None):
+ '''
+ Adds all information from a `paper_info` dictionary to the given
+ :class:`gPapers.model.Paper` object, creating and saving
+ :class:`gPapers.model.Author` and :class:`gPapers.model.Source` objects
+ if necessary.
+
+ Returns the `paper` object.
+ '''
+
+ log_debug('paper_from_dictionary received: %s' % str(paper_info))
+
+ if paper is None:
+ paper = Paper.objects.create()
+ if paper_info is None:
+ paper_info = {}
+
+ # Journal
+ if 'journal' in paper_info:
+ #TODO: Volume etc
+ if 'issue' in paper_info:
+ source, created =
Source.objects.get_or_create(name=paper_info['journal'],
+
issue=paper_info['issue'])
+ else:
+ source, created =
Source.objects.get_or_create(name=paper_info['journal'])
+
+ paper.source = source
+
+ if 'year' in paper_info:
+ paper.source.publication_date = date(int(paper_info['year']),
1, 1)
+
+ paper.source.location = paper_info.get('location', '')
+ paper.source.save()
+
+ if 'pages' in paper_info:
+ paper.source_pages = paper_info['pages']
+
+ # Authors
+ if 'authors' in paper_info:
+ for author in paper_info['authors']:
+ author_obj, created = Author.objects.get_or_create(name=author)
+ paper.authors.add(author_obj)
+ if created:
+ author_obj.save()
+
+ # Simple attributes
+ attributes = ['title', 'abstract', 'doi']
+
+ for attr in attributes:
+ log_debug('Checking if %s is in paper_info' % attr)
+ if attr in paper_info:
+ log_debug('%s is in paper_info' % attr)
+ paper.__setattr__(attr, paper_info[attr])
+
+ paper.save()
+
+ return paper
+
+
class MainGUI:

current_middle_top_pane_refresh_thread_ident = None
active_threads = {}
+
+ def document_imported(self, paper_info, paper_data, user_data):
+ '''
+ Should be called after a paper is imported. `paper_info` is a
+ dictionary with document metadata, `paper_data` is the PDF itself.
+ '''
+
+ if user_data in importer.active_threads:
+ del importer.active_threads[user_data]
+
+ if paper_data is None and paper_info is None:
+ # FIXME: This should be handled via an error callback
+ return
+
+ log_debug('Calling paper_from_dictionary for %s' % str(paper_info))
+ paper = paper_from_dictionary(paper_info)
+ log_debug('paper_from_dictionary returned %s' % str(paper))
+ if paper_data is not None:
+ #TODO: What is a good filename? Make this configurable?
+ if paper.doi:
+ filename = 'doi_' + paper.doi
+ elif paper.pubmed_id:
+ filename = 'pubmed_' + paper.pubmed_id
+ else:
+ filename = 'internal_id_' + str(paper.id)
+
+ filename = defaultfilters.slugify(filename) + '.pdf'
+ log_debug('Saving paper to "%s"' % filename)
+ paper.save_file(filename, paper_data)
+ log_debug('Paper saved')
+
+ self.refresh_middle_pane_from_my_library()

def import_url_dialog(self, o):
'''
@@ -328,9 +421,9 @@
response = dialog.run()
if response == Gtk.ResponseType.OK:
url = entry.get_text()
- threading.Thread(target=importer.import_citation,
- args=(url, None,
-
main_gui.refresh_middle_pane_search)).start()
+ importer.active_threads[url] = 'Importing URL'
+ importer.import_from_url(url, self.document_imported)
+
dialog.destroy()

def import_doi_dialog(self, o):
@@ -344,7 +437,8 @@
buttons=Gtk.ButtonsType.OK_CANCEL,
flags=Gtk.DialogFlags.MODAL)
#dialog.connect('response', lambda x,y: dialog.destroy())
- dialog.set_markup('Import via DOI...\n\nEnter the DOI name
(e.g., 10.1000/182) you would like to import:')
+ dialog.set_markup('Import via DOI...\n\nEnter the DOI name '
+ '(e.g., 10.1000/182) you would like to import:')
entry = Gtk.Entry()
entry.set_activates_default(True)
dialog.vbox.add(entry)
@@ -353,9 +447,9 @@
response = dialog.run()
if response == Gtk.ResponseType.OK:
url = 'http://dx.doi.org/' + entry.get_text().strip()
- threading.Thread(target=importer.import_citation,
- args=(url, None,
-
main_gui.refresh_middle_pane_search)).start()
+ importer.active_threads[url] = 'Importing DOI'
+ importer.import_from_url(url, self.document_imported)
+
dialog.destroy()

def import_file_dialog(self, o):
@@ -421,11 +515,25 @@
dialog.show_all()
response = dialog.run()
if response == Gtk.ResponseType.OK:
+
text_buffer = entry.get_buffer()
- bibtex = text_buffer.get_text(text_buffer.get_start_iter(),
- text_buffer.get_end_iter())
- threading.Thread(target=import_citations_via_bibtexs,
- args=(bibtex,)).start()
+ bibtex_data =
text_buffer.get_text(text_buffer.get_start_iter(),
+ text_buffer.get_end_iter(),
False)
+ paper_info = bibtex.paper_info_from_bibtex(bibtex_data)
+
+ url = paper_info.get('import_url')
+ if not url:
+ url = paper_info.get('doi')
+ if url:
+ url = 'http://dx.doi.org/' + url
+
+ if url:
+ importer.import_from_url(url, self.document_imported,
+ paper_info=paper_info)
+ else:
+ self.document_imported(paper_info, paper_data=None,
+ user_data=None)
+
dialog.destroy()

def __init__(self):
@@ -462,7 +570,7 @@

treeview_running_tasks =
self.ui.get_object('treeview_running_tasks')
# thread_id, text
- self.treeview_running_tasks_model = Gtk.ListStore(int, str)
+ self.treeview_running_tasks_model = Gtk.ListStore(str, str)
treeview_running_tasks.set_model(self.treeview_running_tasks_model)

renderer = Gtk.CellRendererText()
@@ -475,24 +583,26 @@
thread.start_new_thread(self.watch_busy_notifier, ())

def watch_busy_notifier(self):
- while True:
- try:
- if len(self.active_threads):
- self.treeview_running_tasks_model.clear()
- for x in self.active_threads.items():
- self.treeview_running_tasks_model.append(x)
- if not self.busy_notifier_is_running:
-
self.ui.get_object('busy_notifier').set_from_file(os.path.join(RUN_FROM_DIR, 'icons', 'process-working.gif'))
- self.busy_notifier_is_running = True
- self.ui.get_object('treeview_running_tasks').show()
- else:
- if self.busy_notifier_is_running:
-
self.ui.get_object('busy_notifier').set_from_file(os.path.join(RUN_FROM_DIR, 'icons', 'blank.gif'))
- self.busy_notifier_is_running = False
- self.ui.get_object('treeview_running_tasks').hide()
- except:
- traceback.print_exc()
- time.sleep(1)
+ pass
+ # FIXME
+# while True:
+# try:
+# if len(self.active_threads):
+# self.treeview_running_tasks_model.clear()
+# for x in self.active_threads.items():
+# self.treeview_running_tasks_model.append(x)
+# if not self.busy_notifier_is_running:
+#
self.ui.get_object('busy_notifier').set_from_file(os.path.join(RUN_FROM_DIR, 'icons', 'process-working.gif'))
+# self.busy_notifier_is_running = True
+#
self.ui.get_object('treeview_running_tasks').show()
+# else:
+# if self.busy_notifier_is_running:
+#
self.ui.get_object('busy_notifier').set_from_file(os.path.join(RUN_FROM_DIR, 'icons', 'blank.gif'))
+# self.busy_notifier_is_running = False
+#
self.ui.get_object('treeview_running_tasks').hide()
+# except:
+# traceback.print_exc()
+# time.sleep(1)

def init_menu(self):
@@ -823,7 +933,14 @@

self.source_filter_model.clear()
for source in Source.objects.order_by('name'):
- self.source_filter_model.append((source.id, source.name,
source.issue, source.location, source.publisher, source.publication_date))
+ if source.publication_date:
+ publication_date =
source.publication_date.strftime('%Y-%m-%d')
+ else:
+ publication_date = ''
+ self.source_filter_model.append((source.id, source.name,
+ source.issue, source.location,
+ source.publisher,
+ publication_date))

def init_bookmark_pane(self):
@@ -1459,18 +1576,27 @@
log_debug('URL or DOI exists')
button = Gtk.ToolButton(stock_id=Gtk.STOCK_HOME)
button.set_tooltip_text('Open this URL in your browser...')
- button.connect('clicked', lambda x:
desktop.open(liststore[rows[0]][8]))
+ url = liststore[rows[0]][8]
+ if not url:
+ url = 'http://dx.doi.org/' + liststore[rows[0]][9]
+ button.connect('clicked', lambda x: desktop.open(url))
paper_information_toolbar.insert(button, -1)
if paper:
button = Gtk.ToolButton(stock_id=Gtk.STOCK_REFRESH)
button.set_tooltip_text('Re-add this paper to your
library...')
button.connect('clicked', lambda x:
fetch_citation_via_middle_top_pane_row(liststore[rows[0]]))
paper_information_toolbar.insert(button, -1)
- else:
- button = Gtk.ToolButton(stock_id=Gtk.STOCK_ADD)
- button.set_tooltip_text('Add this paper to your
library...')
- button.connect('clicked', lambda x:
fetch_citation_via_middle_top_pane_row(liststore[rows[0]]))
- paper_information_toolbar.insert(button, -1)
+
+ source = liststore[rows[0]][15]
+ log_debug('Source: %s' % source)
+ if not paper and source != 'local': # This is a search result
+ button = Gtk.ToolButton(stock_id=Gtk.STOCK_ADD)
+ button.set_tooltip_text('Add this paper to your
library...')
+ provider = self.search_providers[source]
+ button.connect('clicked',
+ lambda x:
provider.import_paper_after_search(liststore[rows[0]][14],
+
self.document_imported))
+ paper_information_toolbar.insert(button, -1)

if paper:
importable_references = set()
@@ -1885,7 +2011,7 @@

filter_liststore, filter_rows =
self.ui.get_object('source_filter').get_selection().get_selected_rows()
q = None
- for row in filter_rows:
+ for treepath in filter_rows:
row = filter_liststore[treepath]
if q == None: q = Q(source__id=row[0])
else: q = q | Q(source__id=row[0])
@@ -1893,7 +2019,7 @@

filter_liststore, filter_rows =
self.ui.get_object('organization_filter').get_selection().get_selected_rows()
q = None
- for row in filter_rows:
+ for treepath in filter_rows:
row = filter_liststore[treepath]
if q == None: q = Q(organizations__id=row[0])
else: q = q | Q(organizations__id=row[0])
@@ -1922,7 +2048,7 @@
if paper.source:
journal = paper.source.name
if paper.source.publication_date:
- pub_year = paper.source.publication_date.year
+ pub_year = str(paper.source.publication_date.year)
else:
pub_year = ''
else:
@@ -2013,7 +2139,7 @@
pass

# Add information to table
- rows.append(row_from_dictionary(info, search_provider))
+ rows.append(row_from_dictionary(info, search_provider.label))

self.middle_top_pane_model.clear()
for row in rows:
@@ -2682,6 +2808,7 @@

if __name__ == "__main__":

+ Gtk.init(sys.argv)
MEDIA_ROOT = settings.MEDIA_ROOT

log_info('using database at %s' % MEDIA_ROOT)
=======================================
--- /gPapers/models.py Sun Mar 4 14:35:06 2012
+++ /gPapers/models.py Sun Mar 18 12:00:26 2012
@@ -15,11 +15,12 @@
# with this program; if not, write to the Free Software Foundation,
Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.

-import md5, os, re, traceback
+import hashlib, os, re, traceback
from django.db import models
import django.core.files.base
import desktop, pyPdf

+from logger import log_debug

p_doi = re.compile('doi *: *(10.[a-z0-9]+/[a-z0-9.]+)', re.IGNORECASE)

@@ -158,10 +159,14 @@
updated = models.DateTimeField(auto_now=True)

def save_file(self, filename, raw_contents, save=True):
- m = md5.new()
- m.update(raw_contents)
- self.full_text_md5 = m.hexdigest()
- self.full_text.save(filename,
django.core.files.base.ContentFile(raw_contents), save)
+ log_debug('Generating md5 sum')
+ self.full_text_md5 = hashlib.md5(raw_contents).hexdigest()
+ log_debug('Saving file content')
+ self.full_text.save(filename,
+
django.core.files.base.ContentFile(raw_contents),
+ save)
+ self.save()
+ log_debug('Extracting information from PDF')
try: self.extract_document_information_from_pdf()
except: traceback.print_exc()

=======================================
--- /importer/__init__.py Sun Mar 4 14:28:26 2012
+++ /importer/__init__.py Sun Mar 18 12:00:26 2012
@@ -23,6 +23,7 @@

import gi
from gi.repository import GObject
+from gi.repository import Gio
from gi.repository import Gdk
from gi.repository import Gtk
from gi.repository import Pango
@@ -30,42 +31,21 @@

from pyPdf import PdfFileReader

-from gPapers.models import *
+#from gPapers.models import *
from django.template import defaultfilters
import BeautifulSoup, openanything

from logger import *
+import bibtex

active_threads = None

-p_bibtex = re.compile('[@][a-z]+[\s]*{([^<]*)}', re.IGNORECASE | re.DOTALL)
p_whitespace = re.compile('[\s]+')
p_doi = re.compile('doi *: *(10.[a-z0-9]+/[a-z0-9.]+)', re.IGNORECASE)

soup_session = Soup.SessionAsync()

-def latex2unicode(s):
- """
- * \`{o} produces a grave accent
- * \'{o} produces an acute accent
- * \^{o} produces a circumflex
- * \"{o} produces an umlaut or dieresis
- * \H{o} produces a long Hungarian umlaut
- * \~{o} produces a tilde
- * \c{c} produces a cedilla
- * \={o} produces a macron accent (a bar over the letter)
- * \b{o} produces a bar under the letter
- * \.{o} produces a dot over the letter
- * \d{o} produces a dot under the letter
- * \u{o} produces a breve over the letter
- * \v{o} produces a "v" over the letter
- * \t{oo} produces a "tie" (inverted u) over the two letters
- """
- # TODO: expand this to really work
- return s.replace('\c{s}', u's')
-
-
def _decode_htmlentities(string):
entity_re = re.compile("&(#?)(\d{1,5}|\w{1,8});")
return entity_re.subn(_substitute_entity, string)[0]
@@ -277,7 +257,8 @@
#That didn't work, try to find a filename in the query
string
query = urlparse.parse_qs(parsed_url.query)
for key in query:
- if os.path.splitext(query[key])[1].lower() == '.pdf':
+ print key, query[key]
+ if os.path.splitext(query[key][0])[1].lower()
== '.pdf':
filename = query[key].lower() # found a .pdf name
break

@@ -454,53 +435,200 @@
if find_and_attach_pdf(paper, list(promising_links),
visited_urls=visited_urls): return

-def update_paper_from_dictionary(paper_info, paper):
+def determine_content_type(filename):
'''
- Adds all information from a `paper_info` dictionary to the given
- :class:`gPapers.model.Paper` object, creating and saving
- :class:`gPapers.model.Author` and :class:`gPapers.model.Source` objects
- if necessary.
-
- Returns the `paper` object.
+ Determines the content type for a file. Returns either a string like
+ 'application/pdf' or None if the type could not be determined. For
files
+ with a file extension, this function normally does only look at the
+ filename, but if this is not sufficient for determining the
content-type,
+ the file can also be opened.
'''
- assert paper is not None
- # Journal
- if 'journal' in paper_info:
- #TODO: Volume etc
- if 'issue' in paper_info:
- source, created =
Source.objects.get_or_create(name=paper_info['journal'],
-
issue=paper_info['issue'])
+
+ # Try the fast content type first
+ gfile = Gio.file_new_for_path(filename)
+ info = gfile.query_info(Gio.FILE_ATTRIBUTE_STANDARD_FAST_CONTENT_TYPE,
+ Gio.FileQueryInfoFlags.NONE, None)
+ content_type =
info.get_attribute_as_string(Gio.FILE_ATTRIBUTE_STANDARD_FAST_CONTENT_TYPE)
+
+ # If that did not work, try the full content-type determination
+ if Gio.content_type_is_unknown(content_type):
+ info = gfile.query_info(Gio.FILE_ATTRIBUTE_STANDARD_CONTENT_TYPE,
+ Gio.FileQueryInfoFlags.NONE, None)
+ content_type =
info.get_attribute_as_string(Gio.FILE_ATTRIBUTE_STANDARD_CONTENT_TYPE)
+
+ if Gio.content_type_is_unknown(content_type):
+ content_type = None
+
+ return content_type
+
+
+def import_from_url(url, callback, paper_info=None, paper_data=None):
+ '''
+ Searches the given url (asynchronously) for a PDF and/or metadata
+ (currently it will only look for BibTeX data). If the URL is a standard
+ HTML page, it will be searched for potential links, then
+ :function:`import_from_urls` will be called with this link. Finally,
the
+ callback is called with the `paper_info` (a dictionary) and
`paper_data`
+ (binary data)
+ as an argument.
+ '''
+
+ def data_received(session, message, user_data):
+ if not message.status_code == Soup.KnownStatusCode.OK:
+ # FIXME: Use error handler here
+ log_warn('URL %s responded with error code %d' % (user_data,
+
message.status_code))
+ callback(None, None, user_data)
+ return
+
+ content_type = message.response_headers.get_content_type()[0]
+ orig_url = message.get_uri().to_string(False)
+
+ if message.response_body.data:
+ # Heuristic: BibTeX data starts with a @
+ first_letter = message.response_body.data.strip()[0]
+ data = message.response_body.flatten().data
else:
- source, created =
Source.objects.get_or_create(name=paper_info['journal'])
- if created:
- source.save()
- paper.source = source
-
- if 'pages' in paper_info:
- paper.source_pages = paper_info['pages']
-
- # Authors
- if 'authors' in paper_info:
- for author in paper_info['authors']:
- author_obj, created = Author.objects.get_or_create(name=author)
- paper.authors.add(author_obj)
- if created:
- author_obj.save()
- # Simple attributes
- attributes = ['title', 'abstract', 'year']
-
- for attr in attributes:
- if attr in paper_info:
- paper.__setattr__(attr, paper_info[attr])
-
- paper.save()
- return paper
+ first_letter = None
+ data = None
+
+ log_debug('Received content type %s for URI %s' % (content_type,
+
message.get_uri()))
+
+ if content_type == 'application/pdf':
+ callback(None, data, user_data)
+ elif (content_type == 'text/x-bibtex' or first_letter == '@') and
paper_info is None:
+ paper_info = {'bibtex': data}
+ callback(paper_info, None, user_data)
+ elif content_type == 'text/html':
+ log_debug('Searching page for links')
+ parsed = BeautifulSoup.BeautifulSoup(data)
+
+ # Check all links
+ urls = []
+ for a in parsed.findAll('a'):
+ if not a.has_key('href'):
+ continue
+ href = a['href']
+ if href.lower() == orig_url.lower(): #this is were we came
from...
+ continue
+
+ # Check for PDF links
+ if href.lower().endswith('.pdf'):
+ urls.append(href)
+ log_debug('found potential PDF link: %s' % href)
+ else:
+ for c in a.contents:
+ c = str(c).lower()
+ if 'pdf' in c:
+ urls.append(href)
+ log_debug('found potential PDF link: %s' %
href)
+ break
+
+ # Check for BibTeX links
+ if href.lower().endswith('.bib'):
+ urls.append(href)
+ log_debug('found potential BibTeX link: %s' % href)
+ else:
+ for c in a.contents:
+ c = str(c).lower()
+ if 'bibtex' in c:
+ urls.append(href)
+ log_debug('found potential BibTeX link: %s' %
href)
+ break
+ if urls: # we found at least something...
+ #Combine the base URL with the PDF link (necessary for
relative URLs)
+ urls = [urlparse.urljoin(orig_url, url) for url in urls]
+ log_debug('Calling import_from_urls with %s' % str(urls))
+ import_from_urls(urls, callback, user_data)
+ else:
+ log_warn('Nothing found...')
+ callback(paper_info, paper_data, user_data)
+ else:
+ log_warn('Do not know what to do with content type %s of
URL %s' % (content_type, orig_url))
+ callback(paper_info, paper_data, user_data)
+
+ try:
+ message = Soup.Message.new(method='GET', uri_string=url)
+ except TypeError:
+ message = None
+
+ if message:
+ soup_session.queue_message(message, data_received, url)
+ else:
+ callback(paper_info, paper_data, url)
+
+
+def _import_from_urls(urls, callback, user_data, paper_info=None,
paper_data=None):
+ '''
+ Searches the given urls (asynchronously) for a PDF and/or metadata
+ (currently it will only look for BibTeX data). When either all URLs
have
+ been searched or both metadata and PDF have been found, the callback is
+ called with the `paper_info` (a dictionary) and `paper_data` (binary
data)
+ as an argument.
+ '''
+ log_debug('_import_from_urls')
+ if not urls or (paper_info and paper_data):
+ callback(paper_info, paper_data, user_data)
+ return
+
+ url = urls.pop()
+
+ def data_received(session, message, user_data):
+ content_type = message.response_headers.get_one('content-type')
+ if message.response_body.data:
+ # Heuristic: BibTeX data starts with a @
+ first_letter = message.response_body.data.strip()[0]
+ log_debug('First letter of Body is: %s' % first_letter)
+ data = bytearray(message.response_body.flatten().data)
+ else:
+ first_letter = None
+ data = None
+ log_debug('Received content type %s for URI %s' % (content_type,
+
message.get_uri()))
+
+ if content_type == 'application/pdf' and not paper_data:
+ _import_from_urls(urls, callback, user_data,
paper_info=paper_info,
+ paper_data=data)
+ elif (content_type == 'text/x-bibtex' or first_letter == '@') and
not paper_info:
+ # TODO: Convert bibtex data
+ new_paper_info = bibtex.paper_info_from_bibtex(data)
+ _import_from_urls(urls, callback, user_data,
paper_info=new_paper_info,
+ paper_data=paper_data)
+ else: # Continue searching for usable data
+ _import_from_urls(urls, callback, user_data,
paper_info=paper_info,
+ paper_data=paper_data)
+
+ message = Soup.Message.new(method='GET', uri_string=url)
+ soup_session.queue_message(message, data_received, user_data)
+
+
+def import_from_urls(urls, callback, user_data):
+ '''
+ Searches the given urls (asynchronously) for a PDF and/or metadata
+ (currently it will only look for BibTeX data). When either all URLs
have
+ been searched or both metadata and PDF have been found, the callback is
+ called with the `paper_info` (a dictionary) and `paper_data` (binary
data)
+ as an argument.
+ '''
+ if urls is None:
+ callback(None, None, user_data)
+
+ log_info(('Starting to look for PDF and/or metadata '
+ 'from %d possible URLs' % len(urls)))
+
+ def _import_from_urls_finished(paper_info, paper_data, user_data):
+ log_debug('_import_from_urls_finished')
+ callback(paper_info, paper_data, user_data)
+
+ _import_from_urls(urls, _import_from_urls_finished, user_data)
+

class WebSearchProvider(object):
'''
Base class for all web search providers, i.e. websites or web APIs that
return a number of search results for a search string.
-
+
Implementation of new search providers should derive from this class
and
have to provide the following attributes as class attributes:
* `name`: A human readable name that is used in the left column of the
GUI,
@@ -533,7 +661,7 @@
paper) as opposed to only the extracted information.
This method should not block but use the :class:`AsyncSoupSession`
object
`importer.soup_session` for getting the information.
-
+
* `import_paper_after_search(self, data, paper, callback,
error_callback)`
This method receives the `data` (if any) previously returned from the
:method:`parse_website_response` method and a :class:`Paper` object,
already
@@ -547,7 +675,7 @@
URL strings can be given as the second argument, these URLs will be
used
in the order they are given, i.e. if fetching the document from the
first
one is not successful, the second one will be tried etc.
-
+
The :method:`__init__` method of the subclass has to call the
:method:`__init__` method of the superclass.
'''
@@ -590,7 +718,7 @@
Convenience class for web searches that do a single request to a
website for
a search and do not have to perform additional web requests to get more
detailed info for a paper chosen for import.
-
+
Such web search providers need only provide three functions:
* prepare_search_message
* parse_response
@@ -612,7 +740,8 @@

# Provide a tuple of label and search string as `user_data` to
the
# callback -- this way it is clear to what search a
result/error belongs
- soup_session.queue_message(message, my_callback, (self.label,
search_string))
+ soup_session.queue_message(message, my_callback, (self.label,
+
search_string))
except Exception as ex:
error_callback(ex, search_string)

@@ -624,44 +753,38 @@
log_info('Received a response for %s' % str(user_data))
if message.status_code == Soup.KnownStatusCode.OK:
#try:
- callback(user_data,
self.parse_response(message.response_body.data))
+ callback(user_data,
+ self.parse_response(message.response_body.data))
#except Exception as ex:
# error_callback(ex, user_data)
else:
error_callback(message.status_code, user_data)

- def import_paper_after_search(self, data, paper, callback):
+ def import_paper_after_search(self, data, callback):
try:
- callback(self.fill_in_paper_info(data, paper))
+ callback(self.fill_in_paper_info(data), None, self.label)
except Exception as ex:
log_error(str(ex))

- #
--------------------------------------------------------------------------
+ #
-------------------------------------------------------------------------
# Methods to overwrite in sub classes
- #
--------------------------------------------------------------------------
+ #
-------------------------------------------------------------------------
def prepare_search_message(self, search_string):
raise NotImplementedError()

def parse_response(self, response):
raise NotImplementedError()

- def fill_in_paper_info(self, data, paper=None):
+ def fill_in_paper_info(self, data):
raise NotImplementedError()

-
-class WebRessourceProvider(object):
- '''
- In addition to searching, a :class:`WebRessourceProvider` can provide
- information about a single paper, e.g. a PubMED provider can
- provide paper data given a PubMED ID. A method gives information about
its
- capabilities:
- * `provides_infos_for_attribute(self, attribute)`
- `attribute` is a string like 'pubmed_id'
-
- If the class return True for any attribute in the previous method
(otherwise
- the class is pretty useless...), it has to provide another method:
- * `get_infos_for_attribute(self, paper, attribute, value, callback,
error_callback)`
- '''
-
- def provides_infos_for_attribute(self, attribute):
- return False
+if __name__ == '__main__':
+ log_level_debug()
+
+ def print_all(paper_info, paper_data):
+ print 'paper_info', paper_info
+ print 'paper_data', len(paper_data)
+ Gtk.main_quit()
+
+
import_from_url('http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0020409',
print_all)
+ Gtk.main()
=======================================
--- /importer/google_scholar.py Sun Mar 4 13:24:20 2012
+++ /importer/google_scholar.py Sun Mar 18 12:00:26 2012
@@ -7,7 +7,8 @@

import openanything
from logger import log_debug, log_info, log_error
-from importer import *
+from bibtex import paper_info_from_bibtex
+from importer import SimpleWebSearchProvider, html_strip, soup_session

BASE_URL = 'http://scholar.google.com/'

@@ -96,14 +97,18 @@

return papers

- def _got_bibtex(self, message, paper, callback):
+ def _got_bibtex(self, message, callback, user_data):
if message.status_code == Soup.KnownStatusCode.OK:
- paper = update_paper_from_bibtex_html(paper,
-
message.request_body.data)
- callback(paper)
-
- def import_paper_after_search(self, data, paper, callback):
- log_info('Trying to import google scholar citation "%s"' %
paper.title)
+ bibtex_data = message.request_body.data
+ log_debug('Received BibTeX data:\n%s' % bibtex_data)
+ paper_info = paper_info_from_bibtex(bibtex_data)
+ else:
+ log_error('google scholar got status code %d' %
message.status_code)
+ paper_info = None
+ callback(paper_info, None, user_data)
+
+ def import_paper_after_search(self, data, callback):
+ log_info('Trying to import google scholar citation')
try:
citations = data.findAll('div', {'class': 'gs_fl'})[0]
log_debug('Citations: %s' % str(citations))
@@ -113,16 +118,15 @@
log_debug('Found BibTex link: %s' % link['href'])

def bibtex_callback(session, message, user_data):
- self._got_bibtex(message, paper, callback)
+ self._got_bibtex(message, callback, user_data)

message = Soup.Message.new(method='GET',
uri_string=BASE_URL +
link['href'])
- soup_session.queue_message(message, bibtex_callback,
None)
-
- return
- link = data.findAll('div', {'class': 'gs_ggs gs_fl'})[0]
- find_and_attach_pdf(paper, urls=[x['href'] for x in
link.findAll('a') ])
-
- return paper
+ message.request_headers.append('Cookie',
+ 'GSP=ID=%s:CF=4' % self.google_id)
+ soup_session.queue_message(message, bibtex_callback,
+ self.label)
+ #FIXME: Google scholar does not always seem to include
the
+ # URL in the bibtex data -- in this case add a
link
except:
traceback.print_exc()
=======================================
--- /importer/jstor.py Sun Mar 4 13:24:20 2012
+++ /importer/jstor.py Sun Mar 18 12:00:26 2012
@@ -1,6 +1,6 @@
from gi.repository import Soup
from BeautifulSoup import BeautifulStoneSoup
-from importer import SimpleWebSearchProvider, update_paper_from_dictionary
+from importer import SimpleWebSearchProvider
from gPapers.models import Paper

from logger import *
@@ -84,9 +84,7 @@

return papers

- def fill_in_paper_info(self, data, paper=None):
- if not paper:
- paper = Paper()
+ def fill_in_paper_info(self, data):
paper_info = self._parse_result(data)

- return update_paper_from_dictionary(paper_info, paper)
+ return paper_info
=======================================
--- /importer/pubmed.py Sun Mar 4 13:24:20 2012
+++ /importer/pubmed.py Sun Mar 18 12:00:26 2012
@@ -6,8 +6,7 @@
from logger import log_debug, log_info, log_error
from BeautifulSoup import BeautifulStoneSoup

-from gPapers.models import *
-from importer import WebSearchProvider, soup_session,
update_paper_from_dictionary
+from importer import WebSearchProvider, soup_session

BASE_URL = 'http://eutils.ncbi.nlm.nih.gov/entrez/eutils/'
ESEARCH_QUERY = 'esearch.fcgi?db=pubmed&term=%s&usehistory=y'
@@ -26,6 +25,7 @@
WebSearchProvider.__init__(self)

def _ids_received(self, message, callback, error_callback, user_data):
+ message.response_body.flatten()
if not message.status_code == Soup.KnownStatusCode.OK:
error_callback('Pubmed replied with error code %d.' %
message.status_code, user_data)
else:
@@ -51,6 +51,7 @@
soup_session.queue_message(message, mycallback, user_data)

def _summaries_received(self, message, callback, error_callback,
user_data):
+ message.response_body.flatten()
if not message.status_code == Soup.KnownStatusCode.OK:
error_callback('Pubmed replied with error code %d.' %
message.status_code, user_data)
else:
@@ -63,7 +64,9 @@
info = {}
# Extract information
info['pubmed_id'] = str(document.id.string)
-
+ # This is needed for retrieving the paper in
+ # import_paper_after_search
+ info['data'] = info['pubmed_id']
doi = document.findAll('item', {'name': 'doi'})
if doi:
info['doi'] = doi[0].string
@@ -101,12 +104,14 @@
def mycallback(session, message, user_data):
self._ids_received(message, callback, error_callback,
user_data)

- soup_session.queue_message(message, mycallback, (self.label,
search_text))
-
- def _paper_info_received(self, message, callback, paper, user_data):
+ soup_session.queue_message(message, mycallback, (self.label,
+ search_text))
+
+ def _paper_info_received(self, message, callback, user_data):
if not message.status_code == Soup.KnownStatusCode.OK:
log_error('Pubmed replied with error code %d for paper_info
with id: %s' % \
(message.status_code, user_data[1]))
+ paper_info = None
else:
parsed_response =
BeautifulStoneSoup(message.response_body.data)
paper_info = {}
@@ -114,12 +119,10 @@
try:
journal = parsed_response.findAll('journal')[0]
paper_info['journal'] = journal.findAll('title')[0].text
- log_debug('Journal name: %s' % name)
try:
paper_info['issue'] = journal.findAll('issue')[0].text
except:
pass
- log_debug('Journal issue: %s' % issue)

paper_info['pages'] =
parsed_response.findAll('medlinepgn')[0].text
log_debug('Pages: %s' % paper_info.source_pages)
@@ -147,20 +150,17 @@
if all_authors:
paper_info['authors'] = all_authors
except Exception as ex:
- traceback.print_stack()
-
- update_paper_from_dictionary(paper_info, paper)
-
- callback(paper)
-
- def import_paper_after_search(self, data, paper, callback):
- assert paper.pubmed_id
- log_info('Trying to import pubmed citation with id %s' %
paper.pubmed_id)
- query = BASE_URL + EFETCH_QUERY % paper.pubmed_id
+ pass
+
+ callback(paper_info, None, user_data)
+
+ def import_paper_after_search(self, pubmed_id, callback):
+ log_info('Trying to import pubmed citation with id %s' % pubmed_id)
+ query = BASE_URL + EFETCH_QUERY % pubmed_id
message = Soup.Message.new(method='GET', uri_string=query)

def mycallback(session, message, user_data):
- self._paper_info_received(message, callback, paper, user_data)
-
- soup_session.queue_message(message, mycallback, (self.label,
- paper.pubmed_id))
+ self._paper_info_received(message, callback, user_data)
+
+ soup_session.queue_message(message, mycallback,
+ (self.label, pubmed_id))

==============================================================================
Revision: 1556a9466b05
Author: Marcel Stimberg <marcel...@gmail.com>
Date: Sat Mar 24 05:36:26 2012
Log: small changes for Gtk3
http://code.google.com/p/gpapers/source/detail?r=1556a9466b05

Modified:
/gPapers.py
/ui.xml

=======================================
--- /gPapers.py Sun Mar 18 12:00:26 2012
+++ /gPapers.py Sat Mar 24 05:36:26 2012
@@ -2269,7 +2269,9 @@
def save(self):
self.author.name = self.ui.get_object('entry_name').get_text()
text_buffer = self.ui.get_object('notes').get_buffer()
- self.author.notes =
text_buffer.get_text(text_buffer.get_start_iter(),
text_buffer.get_end_iter())
+ self.author.notes =
text_buffer.get_text(text_buffer.get_start_iter(),
+
text_buffer.get_end_iter(),
+ False)
self.author.rating =
round(self.ui.get_object('rating').get_value())
self.author.save()
org_ids = set()
@@ -2680,13 +2682,18 @@
self.paper.doi = self.ui.get_object('entry_doi').get_text()
self.paper.import_url =
self.ui.get_object('entry_import_url').get_text()
text_buffer = self.ui.get_object('textview_abstract').get_buffer()
- self.paper.abstract =
text_buffer.get_text(text_buffer.get_start_iter(),
text_buffer.get_end_iter())
+ self.paper.abstract =
text_buffer.get_text(text_buffer.get_start_iter(),
+
text_buffer.get_end_iter(),
+ False)
text_buffer = self.ui.get_object('textview_bibtex').get_buffer()
- self.paper.bibtex =
text_buffer.get_text(text_buffer.get_start_iter(),
text_buffer.get_end_iter())
+ self.paper.bibtex =
text_buffer.get_text(text_buffer.get_start_iter(),
+
text_buffer.get_end_iter(),
+ False)
self.paper.rating = round(self.ui.get_object('rating').get_value())
self.paper.read_count =
self.ui.get_object('spinbutton_read_count').get_value()
new_file_name =
self.ui.get_object('filechooserbutton').get_filename()
- if new_file_name and self.paper.full_text and new_file_name !=
self.paper.full_text.path:
+ if new_file_name and (not self.paper.full_text or
+ new_file_name != self.paper.full_text.path):
try:
ext = new_file_name[ new_file_name.rfind('.') + 1: ]
except:
@@ -2695,7 +2702,9 @@
self.paper.save_file(full_text_filename,
open(new_file_name, 'r').read())

self.paper.authors.clear()
- self.authors_model.foreach(lambda model, path, iter:
self.paper.authors.add(Author.objects.get(id=model.get_value(iter, 0))))
+
+ for row in self.authors_model:
+ self.paper.authors.add(Author.objects.get(id=row[0]))

self.paper.save()
self.edit_dialog.destroy()
=======================================
--- /ui.xml Sun Mar 4 15:07:37 2012
+++ /ui.xml Sat Mar 24 05:36:26 2012
@@ -225,7 +225,7 @@
<property name="can_focus">False</property>
<property name="events">GDK_POINTER_MOTION_MASK |
GDK_POINTER_MOTION_HINT_MASK | GDK_BUTTON_PRESS_MASK |
GDK_BUTTON_RELEASE_MASK</property>
<property name="toolbar_style">icons</property>
- <property name="icon_size">0</property>
+ <property name="icon-size">1</property>
</object>
<packing>
<property name="expand">False</property>
@@ -533,7 +533,7 @@
<property
name="can_focus">False</property>
<property
name="events">GDK_POINTER_MOTION_MASK | GDK_POINTER_MOTION_HINT_MASK |
GDK_BUTTON_PRESS_MASK | GDK_BUTTON_RELEASE_MASK</property>
<property
name="toolbar_style">icons</property>
- <property name="icon_size">0</property>
+ <property name="icon-size">1</property>
</object>
<packing>
<property
name="expand">False</property>
@@ -686,7 +686,7 @@
<property
name="can_focus">False</property>
<property
name="events">GDK_POINTER_MOTION_MASK | GDK_POINTER_MOTION_HINT_MASK |
GDK_BUTTON_PRESS_MASK | GDK_BUTTON_RELEASE_MASK</property>
<property
name="toolbar_style">icons</property>
- <property
name="icon_size">0</property>
+ <property
name="icon-size">1</property>
</object>
<packing>
<property
name="expand">False</property>
@@ -750,7 +750,7 @@
<property name="visible">True</property>
<property name="can_focus">False</property>
<property name="events">GDK_POINTER_MOTION_MASK |
GDK_POINTER_MOTION_HINT_MASK | GDK_BUTTON_PRESS_MASK |
GDK_BUTTON_RELEASE_MASK</property>
- <property name="icon_size">0</property>
+ <property name="icon-size">1</property>
<child>
<object class="GtkToolButton" id="toolbutton1">
<property
name="use_action_appearance">False</property>

==============================================================================
Revision: a8df07bccb73
Author: Marcel Stimberg <marcel...@gmail.com>
Date: Sun Mar 25 12:12:42 2012
Log: make sure the GUI updates on database changes
http://code.google.com/p/gpapers/source/detail?r=a8df07bccb73

Modified:
/gPapers.py

=======================================
--- /gPapers.py Sat Mar 24 05:36:26 2012
+++ /gPapers.py Sun Mar 25 12:12:42 2012
@@ -67,6 +67,8 @@
django.core.management.setup_environ(settings)
from django.db.models import Q
from django.template import defaultfilters
+from django.db.models.signals import post_save, post_delete
+from django.dispatch import receiver

import deseb
from gPapers.models import *
@@ -561,6 +563,11 @@
self.init_bookmark_pane()
self.init_pdf_preview_pane()
self.refresh_left_pane()
+ # make sure the GUI updates on database changes
+ def receiver_wrapper(sender, **kwargs):
+ self.handle_library_updates()
+ post_save.connect(receiver_wrapper, sender=Paper, weak=False)
+ post_delete.connect(receiver_wrapper, sender=Paper, weak=False)
self.main_window.show()

def init_busy_notifier(self):
@@ -1001,6 +1008,14 @@
width = treeview.get_column(1).get_width() - 16
treeview.get_column(1).get_cells()[0].set_property('wrap-width',
width)

+ def handle_library_updates(self):
+ selection = self.ui.get_object('left_pane_selection')
+ liststore, row = selection.get_selected()
+ if liststore[row][4] == 'local':
+ # Re-select to force a refresh
+
self.select_left_pane_item(self.ui.get_object('left_pane_selection'))
+ self.refresh_my_library_count()
+
def refresh_left_pane(self):
# FIXME: These should not be loaded again and again
NEVER_READ_ICON =
GdkPixbuf.Pixbuf.new_from_file(os.path.join(RUN_FROM_DIR,

==============================================================================
Revision: 18cdb1a44680
Author: Marcel Stimberg <marcel...@gmail.com>
Date: Sun Mar 25 15:07:36 2012
Log: work around some libsoup bugs...
http://code.google.com/p/gpapers/source/detail?r=18cdb1a44680

Modified:
/importer/__init__.py
/importer/pubmed.py

=======================================
--- /importer/__init__.py Sun Mar 18 12:00:26 2012
+++ /importer/__init__.py Sun Mar 25 15:07:36 2012
@@ -481,13 +481,16 @@
callback(None, None, user_data)
return

- content_type = message.response_headers.get_content_type()[0]
+ log_debug('Response received (status code OK)')
+
+ content_type =
message.response_headers.get_one('content-type').split(';')[0]
+ log_debug('Content type determined')
orig_url = message.get_uri().to_string(False)

if message.response_body.data:
- # Heuristic: BibTeX data starts with a @
- first_letter = message.response_body.data.strip()[0]
- data = message.response_body.flatten().data
+ data = message.response_body.flatten().get_data()
+ # Heuristic: BibTeX data starts with a @
+ first_letter = data.strip()[0]
else:
first_letter = None
data = None
@@ -496,10 +499,9 @@

message.get_uri()))

if content_type == 'application/pdf':
- callback(None, data, user_data)
- elif (content_type == 'text/x-bibtex' or first_letter == '@') and
paper_info is None:
- paper_info = {'bibtex': data}
- callback(paper_info, None, user_data)
+ callback(paper_info, data, user_data)
+ elif (content_type == 'text/x-bibtex' or first_letter == '@') and
not paper_info:
+ callback(bibtex.paper_info_from_bibtex(data), paper_data,
user_data)
elif content_type == 'text/html':
log_debug('Searching page for links')
parsed = BeautifulSoup.BeautifulSoup(data)
@@ -550,11 +552,14 @@

try:
message = Soup.Message.new(method='GET', uri_string=url)
- except TypeError:
+ log_debug('Message generated')
+ except TypeError as ex:
+ log_error(str(ex))
message = None

if message:
soup_session.queue_message(message, data_received, url)
+ log_debug('Message queued')
else:
callback(paper_info, paper_data, url)

@@ -580,7 +585,7 @@
# Heuristic: BibTeX data starts with a @
first_letter = message.response_body.data.strip()[0]
log_debug('First letter of Body is: %s' % first_letter)
- data = bytearray(message.response_body.flatten().data)
+ data = message.response_body.flatten().get_data()
else:
first_letter = None
data = None
=======================================
--- /importer/pubmed.py Sun Mar 18 12:00:26 2012
+++ /importer/pubmed.py Sun Mar 25 15:07:36 2012
@@ -25,11 +25,12 @@
WebSearchProvider.__init__(self)

def _ids_received(self, message, callback, error_callback, user_data):
- message.response_body.flatten()
+
if not message.status_code == Soup.KnownStatusCode.OK:
error_callback('Pubmed replied with error code %d.' %
message.status_code, user_data)
else:
- parsed_response =
BeautifulSoup.BeautifulStoneSoup(message.response_body.data)
+ response_data = message.response_body.flatten().get_data()
+ parsed_response =
BeautifulSoup.BeautifulStoneSoup(response_data)

# Check wether there were any hits at all
if int(parsed_response.esearchresult.count.string) == 0:
@@ -51,11 +52,11 @@
soup_session.queue_message(message, mycallback, user_data)

def _summaries_received(self, message, callback, error_callback,
user_data):
- message.response_body.flatten()
if not message.status_code == Soup.KnownStatusCode.OK:
error_callback('Pubmed replied with error code %d.' %
message.status_code, user_data)
else:
- parsed_response =
BeautifulSoup.BeautifulStoneSoup(message.response_body.data)
+ response_data = message.response_body.flatten().get_data()
+ parsed_response =
BeautifulSoup.BeautifulStoneSoup(response_data)

# get information for all documents
documents = parsed_response.esummaryresult.findAll('docsum')

==============================================================================
Revision: 4eed7c76daf3
Author: Marcel Stimberg <marcel...@gmail.com>
Date: Sun Mar 25 14:15:48 2012
Log: re-add the "busy notifier"
http://code.google.com/p/gpapers/source/detail?r=4eed7c76daf3

Modified:
/gPapers.py
/importer/__init__.py

=======================================
--- /gPapers.py Sun Mar 25 12:12:42 2012
+++ /gPapers.py Sun Mar 25 14:15:48 2012
@@ -379,7 +379,7 @@
'''

if user_data in importer.active_threads:
- del importer.active_threads[user_data]
+ del importer.active_threads[str(user_data)]

if paper_data is None and paper_info is None:
# FIXME: This should be handled via an error callback
@@ -587,30 +587,25 @@
treeview_running_tasks.append_column(column)

make_all_columns_resizeable_clickable_ellipsize(treeview_running_tasks.get_columns())

- thread.start_new_thread(self.watch_busy_notifier, ())
+ GObject.timeout_add_seconds(1, self.watch_busy_notifier)

def watch_busy_notifier(self):
- pass
- # FIXME
-# while True:
-# try:
-# if len(self.active_threads):
-# self.treeview_running_tasks_model.clear()
-# for x in self.active_threads.items():
-# self.treeview_running_tasks_model.append(x)
-# if not self.busy_notifier_is_running:
-#
self.ui.get_object('busy_notifier').set_from_file(os.path.join(RUN_FROM_DIR, 'icons', 'process-working.gif'))
-# self.busy_notifier_is_running = True
-#
self.ui.get_object('treeview_running_tasks').show()
-# else:
-# if self.busy_notifier_is_running:
-#
self.ui.get_object('busy_notifier').set_from_file(os.path.join(RUN_FROM_DIR, 'icons', 'blank.gif'))
-# self.busy_notifier_is_running = False
-#
self.ui.get_object('treeview_running_tasks').hide()
-# except:
-# traceback.print_exc()
-# time.sleep(1)
-
+ if len(self.active_threads):
+ self.treeview_running_tasks_model.clear()
+ for x in self.active_threads.items():
+ self.treeview_running_tasks_model.append(x)
+ if not self.busy_notifier_is_running:
+
self.ui.get_object('busy_notifier').set_from_file(os.path.join(RUN_FROM_DIR, 'icons', 'process-working.gif'))
+ self.busy_notifier_is_running = True
+ self.ui.get_object('treeview_running_tasks').show()
+ else:
+ if self.busy_notifier_is_running:
+
self.ui.get_object('busy_notifier').set_from_file(os.path.join(RUN_FROM_DIR, 'icons', 'blank.gif'))
+ self.busy_notifier_is_running = False
+ self.ui.get_object('treeview_running_tasks').hide()
+
+ # Assure that this function gets called again
+ return True

def init_menu(self):
self.ui.get_object('menuitem_quit').connect('activate', lambda x:
sys.exit(0))
@@ -643,24 +638,6 @@
about.connect('response', lambda x, y: about.destroy())
about.show()

- def check_for_updates(self):
- parent_self = self
- class UpdateThread(threading.Thread):
- def run(self):
- parent_self.active_threads[ thread.get_ident() ]
= 'checking for updates...'
- output = commands.getoutput('svn update')
- Gdk.threads_enter()
- dialog = Gtk.MessageDialog(type=Gtk.MessageType.INFO,
buttons=Gtk.ButtonsType.OK)
- dialog.connect('response', lambda x, y: dialog.destroy())
- dialog.set_markup('Output from
SVN:\n\n%s\n\n(restart for changes to take effect)' %
(pango_escape(output)))
- dialog.show_all()
- response = dialog.run()
- Gdk.threads_leave()
- if parent_self.active_threads.has_key(thread.get_ident()):
- del parent_self.active_threads[ thread.get_ident() ]
- t = UpdateThread()
- t.start()
-
def clear_all_search_and_filters(self):
self.ui.get_object('middle_pane_search').set_text('')
self.ui.get_object('author_filter').get_selection().unselect_all()
@@ -1976,7 +1953,7 @@
self.refresh_my_library_filter_pane)

def refresh_middle_pane_from_my_library(self,
refresh_library_filter_pane=True):
- self.active_threads[ thread.get_ident() ] = 'searching local
library...'
+ self.active_threads[ str(thread.get_ident()) ] = 'searching local
library...'
try:
rows = []
my_library_filter_pane =
self.ui.get_object('my_library_filter_pane')
@@ -2093,8 +2070,8 @@
self.refresh_my_library_count()
except:
traceback.print_exc()
- if self.active_threads.has_key(thread.get_ident()):
- del self.active_threads[ thread.get_ident() ]
+ if self.active_threads.has_key(str(thread.get_ident())):
+ del self.active_threads[ str(thread.get_ident()) ]

def refresh_my_library_count(self):
selection = self.ui.get_object('left_pane_selection')
=======================================
--- /importer/__init__.py Sun Mar 25 15:07:36 2012
+++ /importer/__init__.py Sun Mar 25 14:15:48 2012
@@ -235,7 +235,7 @@

log_info('Importing URL: %s' % url)

- active_threads[ thread.get_ident() ] = 'importing: ' + url
+ active_threads[ str(thread.get_ident()) ] = 'importing: ' + url
try:
response = urllib.urlopen(url)
if response.getcode() != 200 and response.getcode() != 302:
@@ -306,8 +306,8 @@
error.set_markup('No Paper Found\n\nThe given URL does not
appear to contain or link to any PDF files. (perhaps you have it buy it?)
Try downloading the file and adding it using "File >>
Import..."\n\n%s' % pango_escape(url))
error.run()
Gdk.threads_leave()
- if active_threads.has_key(thread.get_ident()):
- del active_threads[ thread.get_ident() ]
+ if active_threads.has_key(str(thread.get_ident())):
+ del active_threads[str(thread.get_ident())]

p_html_a = re.compile("<a [^>]+>" , re.IGNORECASE)
p_html_a_href = re.compile('''href *= *['"]([^'^"]+)['"]''' ,
re.IGNORECASE)

==============================================================================
Revision: de69e230ca2f
Author: Marcel Stimberg <marcel...@gmail.com>
Date: Sun Mar 25 14:18:44 2012
Log: remove some unused code
http://code.google.com/p/gpapers/source/detail?r=de69e230ca2f

Modified:
/gPapers.py

=======================================
--- /gPapers.py Sun Mar 25 14:15:48 2012
+++ /gPapers.py Sun Mar 25 14:18:44 2012
@@ -192,28 +192,6 @@

main_gui.refresh_middle_pane_search()

-def import_citations_via_bibtexs(bibtexs):
-
- log_info('Starting BibTex import')
-
- a_bibtex = []
-
- for line in bibtexs.split('\n'):
- if not line: continue
- if len(a_bibtex) and line.strip().startswith('@'):
- # found new bibtex start
- import_citation_via_bibtex('\n'.join(a_bibtex))
- a_bibtex = []
- a_bibtex.append(line)
- if len(a_bibtex):
- # import last found bibtex
- import_citation_via_bibtex('\n'.join(a_bibtex))
-
- main_gui.refresh_middle_pane_search()
-
-def import_citation_via_bibtex(bibtex):
- importer.update_paper_from_bibtex_html(None, bibtex)
-
def paper_info_received(paper):
paper.save()
importer.import_citation(paper.import_url, paper=paper,

==============================================================================
Revision: 8c79c339d8cd
Author: Marcel Stimberg <marcel...@gmail.com>
Date: Sun Mar 25 14:49:56 2012
Log: make document import more or less work
http://code.google.com/p/gpapers/source/detail?r=8c79c339d8cd

Modified:
/gPapers.py
/gPapers/models.py
/importer/__init__.py
/importer/pdf_file.py

=======================================
--- /gPapers.py Sun Mar 25 14:18:44 2012
+++ /gPapers.py Sun Mar 25 14:49:56 2012
@@ -23,7 +23,7 @@
from datetime import datetime, timedelta, date

from logger import *
-from importer import bibtex
+from importer import bibtex, pdf_file

log_level_debug()

@@ -231,11 +231,13 @@
try:
log_info('Importing paper: %s' % filename)
md5_hexdigest = get_md5_hexdigest_from_data(data)
- # FIXME
- paper, created =
importer.get_or_create_paper_via(full_text_md5=md5_hexdigest)
+ log_info('get/create paper with MD5: %s' % md5_hexdigest)
+ paper, created =
Paper.objects.get_or_create(full_text_md5=md5_hexdigest)
if created:
- #paper.title = filename
-# help(paper.full_text)
+ log_info('Trying to get some metadata from PDF')
+ paper_info = pdf_file.get_paper_info_from_pdf(data)
+ if paper_info:
+ paper_from_dictionary(paper_info, paper=paper)

paper.save_file(defaultfilters.slugify(os.path.split(filename)[1].replace('.pdf', ''))
+ '.pdf', data)
if not data:
paper.import_url = params['url']
@@ -543,7 +545,8 @@
self.refresh_left_pane()
# make sure the GUI updates on database changes
def receiver_wrapper(sender, **kwargs):
- self.handle_library_updates()
+ # Make sure this gets only called from the main thread
+ GObject.idle_add(lambda x : self.handle_library_updates, ())
post_save.connect(receiver_wrapper, sender=Paper, weak=False)
post_delete.connect(receiver_wrapper, sender=Paper, weak=False)
self.main_window.show()
@@ -651,7 +654,7 @@
text = self.ui.get_object('middle_pane_search').get_text()
self.search_providers[liststore[row][4]].clear_cache(text)

- self.last_middle_pane_search_string = None
+ self.last_middle_pane_search_string = ''

def middle_pane_search_changed(self):
=======================================
--- /gPapers/models.py Sun Mar 18 12:00:26 2012
+++ /gPapers/models.py Sun Mar 25 14:49:56 2012
@@ -186,6 +186,7 @@
self.read_count = self.read_count + 1
self.save()

+ # FIXME: This is obsolete, see importer.pdf_file
def extract_document_information_from_pdf(self, force_overwrite=False):
"""will overwrite the extracted_text and page_count fields, and
the title if the title is empty"""
if self.full_text and os.path.isfile(self.full_text.path):
@@ -215,8 +216,7 @@
author.save()
self.authors.add(author)
except:
- traceback.print_exc()
- # also has: doc_info['/Author'], doc_info['/ModDate'],
doc_info['/CreationDate']
+ pass

# extract the actual text
stdin, stdout = os.popen4('ps2txt "%s"' % self.full_text.path)
=======================================
--- /importer/__init__.py Sun Mar 25 14:15:48 2012
+++ /importer/__init__.py Sun Mar 25 14:49:56 2012
@@ -20,6 +20,7 @@
from time import strptime
from htmlentitydefs import name2codepoint as n2cp
import urllib, urlparse
+import hashlib

import gi
from gi.repository import GObject
@@ -62,7 +63,7 @@

def get_md5_hexdigest_from_data(data):
- m = md5.new()
+ m = hashlib.md5()
m.update(data)
return m.hexdigest()

=======================================
--- /importer/pdf_file.py Sun Mar 18 12:00:26 2012
+++ /importer/pdf_file.py Sun Mar 25 14:49:56 2012
@@ -1,86 +1,50 @@
-import os
-
-from gi.repository import Soup # @UnresolvedImport
-from gi.repository import Gio # @UnresolvedImport
+from io import BytesIO

from pdfminer.pdfparser import PDFParser, PDFDocument, resolve1

from logger import log_debug, log_info, log_error
-from importer import RessourceProvider

-class PdfFileRessourceProvider(RessourceProvider):
-
- def provides_infos_for_file(self, filename, content_type):
- return content_type == 'application/pdf'
-
- def get_infos_for_file(self, filename, callback, error_callback):
- # TODO: Use async GIO methods?
- fp = open(filename, 'rb')
- # Create a PDF parser object associated with the file object.
- parser = PDFParser(fp)
- # Create a PDF document object that stores the document structure.
- doc = PDFDocument()
- # Connect the parser and document objects.
- parser.set_document(doc)
- doc.set_parser(parser)
- # Initialize
- doc.initialize()
- # Extract the metadata
- for xref in doc.xrefs:
- info_ref = xref.trailer.get('Info')
- if info_ref:
- info = resolve1(info_ref)
-
- paper_info = {}
- if info:
- authors = info.get('Author')
- if authors:
- if ';' in authors:
- author_list = authors.split(';')
- elif ' AND ' in authors:
- author_list = authors.split(' AND ')
- elif ',' in authors:
- #FIXME: This cuts 'LastName, FirstName' in two...
- author_list = authors.split(',')
- else:
- author_list = [authors]
-
- paper_info['authors'] = author_list
- title = info.get('Title')
- if title:
- paper_info['title'] = title
-
- #TODO: Additional metadata?
- #TODO: What about embedded BibTeX (as done by JabRef)?
-
- #TODO: Extract text
-
- callback(paper_info)
-
- def get_document_for_file(self, filename, callback, error_callback):
- #TODO: Copy the file to our PDF directory
- return filename
-
-if __name__ == '__main__':
- import sys
- pdf_ressource = PdfFileRessourceProvider()
- if len(sys.argv) > 1:
- for arg in sys.argv[1:]:
- if pdf_ressource.provides_infos_for_file(arg):
- print 'Infos for %s' % arg
-
- def print_all(paper_info):
- for key, value in paper_info.items():
- print key, ':', value
-
- paper_info = pdf_ressource.get_infos_for_file(arg,
- print_all,
- None)
-
-
-
-
-
-
-
+def get_paper_info_from_pdf(data):
+ fp = BytesIO(data)
+ # Create a PDF parser object associated with the file object.
+ parser = PDFParser(fp)
+ # Create a PDF document object that stores the document structure.
+ doc = PDFDocument()
+ # Connect the parser and document objects.
+ parser.set_document(doc)
+ doc.set_parser(parser)
+ # Initialize
+ doc.initialize()
+ # Extract the metadata
+ for xref in doc.xrefs:
+ info_ref = xref.trailer.get('Info')
+ if info_ref:
+ info = resolve1(info_ref)
+
+ paper_info = {}
+ if info:
+ authors = info.get('Author')
+ if authors:
+ if ';' in authors:
+ author_list = authors.split(';')
+ elif ' AND ' in authors:
+ author_list = authors.split(' AND ')
+ elif ',' in authors:
+ #FIXME: This cuts 'LastName, FirstName' in two...
+ author_list = authors.split(',')
+ else:
+ author_list = [authors]
+
+ paper_info['authors'] = author_list
+ title = info.get('Title')
+ if title:
+ paper_info['title'] = title
+
+ #TODO: Additional metadata?
+ #TODO: What about embedded BibTeX (as done by JabRef)?
+
+ #TODO: Extract text
+ #TODO: Find doi
+
+ return paper_info

Reply all

Reply to author

Forward

0 new messages