Delta?

[pascal]

Hi

I'd like to fetch only delta changes from a repository, that is
changes that happened since my last check.
How could i do this?

I just posted another question in the group regarding "sort order" of
search results. If i can get the search results sorted by mDate, then
i can use this.

First run:
1. Find all Items matching my query, sort by mDate ASC
2. Loop over all these items, memorize the latest mDate

On subsequent runs:
1. Find all Items matching my query with an mDate > memorized mDate
2. ....

why is sorting important:
if a run can not be finished because of an error, then i need to
continue at the highest mDate. if result list is unsorted, then this
does not work.

[Matthew Zumwalt]

For something this simple, you can just use Fedora's BasicSearch api, which is what you hit when you go to fedora/objects.  You can pass these queries directly to Fedora via the RubyFedora classes (in the Fedora:: namespace instead of the ActiveFedora:: namespace).

Both solr and basicsearch support searching by date ranges for any date fields.  See the corresponding documentation to get the syntax.

Matt Zumwalt

MediaShelf, LLC

http://www.yourmediashelf.com

--

You received this message because you are subscribed to the Google Groups "ActiveFedora / Ruby + Fedora Commons" group.
To post to this group, send email to active...@googlegroups.com.
To unsubscribe from this group, send email to active-fedor...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/active-fedora?hl=.

[pascal]

Yes but as mentioned i need sorting. And as you mentioned in the other
post: basic search does not support sorting.
So i need to get SOLR up and running.

On Nov 13, 8:57 pm, Matthew Zumwalt <matt.zumw...@yourmediashelf.com>
wrote:

> For something this simple, you can just use Fedora's BasicSearch api,  
> which is what you hit when you go to fedora/objects.  You can pass  
> these queries directly to Fedora via the RubyFedora classes (in the  
> Fedora:: namespace instead of the ActiveFedora:: namespace).
>
> Both solr and basicsearch support searching by date ranges for any  
> date fields.  See the corresponding documentation to get the syntax.
>
> Matt Zumwalt

> MediaShelf, LLChttp://www.yourmediashelf.com

[Matthew Zumwalt]

Huh.  I thought basicsearch did support sorting.  Either way, yes.  Solr will treat you well.  This is, or course, assuming that you're using an ActiveFedora app to create the objects?   If not, you will have to find a way to index the objects and metadata into solr.  You can either use ActiveFedora to write your own utility that does this or you can wire up Fedora GSearch to handle it for you (a bit of a hairy install...)

Matt Zumwalt

MediaShelf, LLC

http://www.yourmediashelf.com

[pascal]

Well i could not find an option. at least not in the doc. Perhaps i
need to read the source code or ask in the fedora group.

http://www.fedora-commons.org/confluence/display/FCR30/Basic+Search

Objects in Fedora are created and managed from outside my application.


On Nov 13, 9:29 pm, Matthew Zumwalt <matt.zumw...@yourmediashelf.com>
wrote:

> Huh.  I thought basicsearch did support sorting.  Either way, yes.  
> Solr will treat you well.  This is, or course, assuming that you're  
> using an ActiveFedora app to create the objects?   If not, you will  
> have to find a way to index the objects and metadata into solr.  You  
> can either use ActiveFedora to write your own utility that does this  
> or you can wire up Fedora GSearch to handle it for you (a bit of a  
> hairy install...)
>
> Matt Zumwalt

[Matthew Zumwalt]

As long as the metadata you need to index is pretty straightforward, writing an indexer with activefedora is darn easy.  You can run it as a command-line rebuilder utility or slap a JMS/Stomp listener on it and you've got a tool to update solr in real time when anything changes in Fedora.

Matt Zumwalt

MediaShelf, LLC

http://www.yourmediashelf.com

[pascal]

Do i get this right?

- I setup SOLR to work with fedora
- I index pid and mDate in Solr, trough either activefedora or JMS
(JMS preferred since no polling involved)
- I use activefedora to query for deltas in SOLR index and receive a
list of pids
- I use activefedora to get objects from fedora repository by pid
found
- I'm happy
-?


Btw: I checked the source code, it doesnt look like basic search
supports sorting:
// disabled sorting: see bug 78
// queryText.append(" ORDER BY");
// queryText.append(resultFieldsString);

I could not find a bug description.....


pascal


On Nov 13, 10:13 pm, Matthew Zumwalt <matt.zumw...@yourmediashelf.com>
wrote:

> As long as the metadata you need to index is pretty straightforward,  
> writing an indexer with activefedora is darn easy.  You can run it as  
> a command-line rebuilder utility or slap a JMS/Stomp listener on it  
> and you've got a tool to update solr in real time when anything  
> changes in Fedora.
>
> Matt Zumwalt

[Matthew Zumwalt]

On Nov 13, 2009, at 4:05 PM, pascal wrote:

>
> Do i get this right?

Almost

>
> - I setup SOLR to work with fedora

See the activefedora installation docs for ideas about how to set up
solr.

> - I index pid and mDate in Solr, trough either activefedora or JMS
> (JMS preferred since no polling involved)

Write your updater -- the thing that reads objects from fedora and
pushes chosen metadata into solr -- using activefedora.
Trigger the updater however you see fit. The easiest option is to
manually run it from the command line. Slightly more complicated, but
still really easy, add a JMA/Stomp listener that triggers the indexer
when fedora sends out JMS messages about changes.

> - I use activefedora to query for deltas in SOLR index and receive a
> list of pids

You don't need activefedora for this. Go directly to solr using rsolr
or solr-ruby (or whatever solr client library you want).

> - I use activefedora to get objects from fedora repository by pid
> found

Yes.

> - I'm happy

Happiness comes from within.

[pascal]

Hi


I implemented it without SOLR by just using the JMS (embedded AMQ)
notifications.
It was a bit a pain (as you said) to get it running: some
springframework classes are required to get the stomp protocol
working. So i had to add the jar file to the classpath.
After that, it was pretty easy: activemessaging+stomp gem installed,
configured the queue and wrote some XPATH expressions to parse the
message.

Now things work without polling. Nice!

The only thing missing is to configure AMQ with username/password (i
dont know how to do this with the embedded AMQ broker) to "protect"
the queue.

Thanks for the hints!

Pascal



On Nov 13, 11:58 pm, Matthew Zumwalt <matt.zumw...@yourmediashelf.com>