sorting

[Joris Janssens]

Is there a way to get results from a query returned in a sorted or differently ordered fashion?

Thanks

[Péter Király]

Hi Joris,

not yet, unfortunatelly. We talked about this feature, so maybe there
will be sorting options in the future.
Can you, please, give us some possibilities you would make utilize in the API?

Regards,
Péter

2013/9/24 Joris Janssens <jo...@packed.be>:

> Is there a way to get results from a query returned in a sorted or
> differently ordered fashion?
>
> Thanks
>

> --
> You received this message because you are subscribed to the Google Groups
> "Europeana API forum" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to europeanaAPI...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.



--
Péter Király
software developer

Europeana - http://europeana.eu
eXtensible Catalog - http://eXtensibleCatalog.org

[Andrés Viedma]

I think it would be great an option to retrieve results sorted by "popularity", ie, number of visits to the record.

2013/9/24 Péter Király <kiru...@gmail.com>

[Mia]

Jumping in, we really could have used sorting by weight (of query
term, not the physical object!) or record completeness for
Serendip-o-matic.

Cheers, Mia
--------------------------------------------
http://openobjects.org.uk/
http://twitter.com/mia_out
I mostly use this address for list mail and don't check it daily; use
my open.ac.uk address for personal email

[Gordea Sergiu]

Hi Mia,

I don't really understand your suggestion...
Of course, solr sorts the objects according to the field weightings defined in the schema ...

BR,

Sergiu

[Mia]

That's what I get for jumping in too quickly then...

--------------------------------------------
http://openobjects.org.uk/
http://twitter.com/mia_out
I mostly use this address for list mail and don't check it daily; use
my open.ac.uk address for personal email

[Ian Davis]

Solr also allows sorting by fields in the index so this could be exposed
in the Europeana API

http://wiki.apache.org/solr/CommonQueryParameters#sort

[Péter Király]

2013/9/24 Ian Davis <m...@iandavis.com>:

> Solr also allows sorting by fields in the index so this could be exposed
> in the Europeana API

We can make use of Solr's sorting by field feature. Right now the sort
is based on basic relevancy -- I mean, we haven't add any fancy thing.
Untill we introduce a sort parameter, you can apply the Solr boosting
operator on the query terms, which will modify the order of elements
in the search result (adding a given field more weight). For example:
what:peter^80 who:peter will give different result than what:peter
who:peter^80.

(I would like to mention, that I am only a developer, not a decision
maker on what features should the API include...)

Regards,
Péter

[Gordea Sergiu]

That is true ... and it would be nice to have this at hand in the API even if I don't know which would be the concrete motivation for ordering by a particular column.

Maybe ordering by title would be useful for representing responses of very restrictive queries (e.g. the ones returning less than 100 results).
(title is almost unique)

Otherwise, the facet filtering seems to be more useful in conjunction with other metadata fields.
(the most of the other fields contain values that appear in many documents )

Best regards,

[Gordea Sergiu]

Peter said: (I would like to mention, that I am only a developer, not a decision maker on what features should the API include...)

Therefore I suggest to think of concrete usecases and in order to demonstrate the added value of any feature requested through the forum.

I think that the "popularity" sorting is something that makes sense even if it introduces a sort of bias in the search... If the popularity is not the default sorting, that the bias will be very low.

BR,

Sergiu


-----Original Message-----
From: europe...@googlegroups.com [mailto:europe...@googlegroups.com] On Behalf Of Péter Király
Sent: Dienstag, 24. September 2013 16:07
To: europe...@googlegroups.com
Subject: Re: {Europeana API forum} sorting

2013/9/24 Ian Davis <m...@iandavis.com>:
> Solr also allows sorting by fields in the index so this could be
> exposed in the Europeana API

We can make use of Solr's sorting by field feature. Right now the sort is based on basic relevancy -- I mean, we haven't add any fancy thing.
Untill we introduce a sort parameter, you can apply the Solr boosting operator on the query terms, which will modify the order of elements in the search result (adding a given field more weight). For example:
what:peter^80 who:peter will give different result than what:peter who:peter^80.

Regards,
Péter

--
Péter Király
software developer

Europeana - http://europeana.eu
eXtensible Catalog - http://eXtensibleCatalog.org

[Péter Király]

Popularity would be quite hard to implement, because it requires both
some advanced Solr feature (and Solr administration feature as well),
and an extraction tool which would regularly report an up to date
pupularity chart in Solr consumable format. But there are a couple of
other things which would be ideal to start with, such as what Sergiu
mentioned title, or author, facet and aggregated fields (who, what) --
existing, and fairly populated fields.

Regads,
Péter

2013/9/24 Gordea Sergiu <Sergiu...@ait.ac.at>:

[Ian Davis]

I think completeness would be a good field to allow sorting on.

[Gordea Sergiu]

:))

It was one used as main contributor to the score, but the results were quite bad ...
Again ... an completion facet in conjunction with the relevance score is more useful ...

Solr 4 should support query time joins... which would be the appropriate solution for popularity based search:
http://lucenerevolution.org/sites/default/files/grouping-and-joining_0.pdf

I mean .. this will be similar to two indexes, one holding the metadata and one small holding the popularity scores.
Indexing popularity scores shouldn't affect the metadata index... This way you could update the popularity scores monthly ...

Of course there is some work to do for implementing and maintaining this functionality, but it might be worth...

[Aldo Gordillo]

I also think it would be very useful for some cases to have the option of sorting the results randomly.

I don't mean to retrieve 100 records and then sorting randomly, I mean to randomly retrieve 100 records, this way each request will return different results. 

Although this thread was posted about 2 years ago, the sorting feature is not available yet (at least, is not included in the documentation).

I think it would be a very usefull feature. Sort by relevance/popularity/random/completeness as measure or metadata quality makes sense for me.

For building the popularity metric I would suggest to take into account several aspects like visits, views (of the original source), and the times the item has been saved by the users. 

Looking forward for this feature...

Greetings,

Aldo

[James Morley]

Hi Aldo & others

This thread is from before I joined Europeana, but I agree with your comments and sorting is something I raised internally a while ago.  As far as I understand it can be difficult to implement as it is very resource intensive, and with a collection of what is now 46 million records, and getting continuously larger, we have to be very careful.  But your email has prompted me to ask our (rather overstretched!) tech team again, and see what their thoughts are.

It's an interesting comment about popularity too. Even before I joined I was thinking around issues of 'popularity' on platforms like Europeana - http://www.catchingtherain.com/2014/02/sentiment-analysis-for-cultural-collection-objects-aka-how-to-identify-the-good-stuff/. As I mention in that piece, Flickr have something they call 'interestingness' which works on some kind of mystery algorithm based on views, comments, favourites, and even negative factors like if it has been posted to huge numbers of groups.

It's certainly something to explore and I'd love to think that we can at least try some of these things out, but I'm afarid I can't make any promises right now.

Best,

James

---

From: europe...@googlegroups.com [europe...@googlegroups.com] on behalf of Aldo Gordillo [iamc...@gmail.com]
Sent: 29 October 2015 16:29
To: Europeana API forum
Subject: {Europeana API forum} Re: sorting

--
Visit Europeana Labs for API Documentation, Open Datasets, and our Apps Showcase - http://labs.europeana.eu
---

You received this message because you are subscribed to the Google Groups "Europeana API forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to europeanaAPI...@googlegroups.com.

To post to this group, send email to europe...@googlegroups.com.
Visit this group at http://groups.google.com/group/europeanaAPI.
For more options, visit https://groups.google.com/d/optout.

James Morley

Creative Industries Community Developer

 

T:
M: +44 (0)77 1336 0563
E: James....@europeana.eu

Skype: jamesinealing

 

Be part of Europe's online cultural movement - join the Europeana Network Association: http://bit.ly/NetworkAssociation

 #AllezCulture!

Disclaimer: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. If you are not the named addressee you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system.