Figuring out what to measure

[Jody DeRidder]

One of the challenges I run into when performing user studies is in determining what exactly I'm measuring, so I know what it is that needs to be modified as a result.  When I look at an online digital library, I'm thinking about:

• the effectiveness and speed of the search engine
  
• the encoded information on the website
  
• the item-level or collection-level metadata
  
• the actual content
  
• the way the content is displayed and accessed
  
• the browse functionality (an interplay with the metadata and usually the website encoding)
  
• the refining and sorting capabilities (which are an interplay of the system with the metadata)
  
• additional services built on top (like citation extraction, mail/download selection, etc.)
  

But all this is a mix when you're doing the study itself, and you have to be able to suss out the meaning of your results, in a way that tells you what you can and should do to improve the digital library.    That makes measurement additionally complex.

And measurement already isn't all that clear.  I'm very grateful to those who spell out in their research how exactly they implement measurements (such as what exemplifies "efficiency", "effectiveness", "satisfaction," "learnability" and so forth) -- though often I find I cannot follow the exact same steps in my own system.

I'm hoping we can determine -- together -- the best way to measure these and other aspects of usability, in a way that helps us not only understand but repeat findings.   AND I would like for us to sort out how it's best to measure all the different aspects of digital library systems.  I think the usual "usability" measures (mentioned above) leave out assessment of some of the aspects in those bullet points above.  Yet a complete and useful evaluation of a digital library, to me, would certainly include all these aspects, and more.

Your thoughts?

Jody L DeRidder

Head, Digital Services

University of Alabama Libraries

[joyce.chapman]

I think the most common types of assessment with digital libraries are usability testing and the use of web analytics for really basic content analysis (what stuff is getting looked at the most?) I agree that usability testing often returns results that are a bit of a jumble in regards to what exactly was tested: the interface? The content? The metadata? If we don't consciously plan studies to specifically address these different areas, we aren't going to be able to tease apart results very well. In particular from the list you mention, I think metadata often gets the short end of the stick in digital library evaluation. Which is interesting because libraries put so much time and energy into metadata creation. I've worked on a few studies that specifically evaluated digital library metadata[1]  and each time we had little success finding evidence of similar work or literature around the topics. 

When you say you hope we can find a way to measure "these" aspects of usability, do you mean the bulleted list or the second list ("effectiveness, efficiency, satisfaction, learnability")? 

It seems like one way to get started would be to look at what's already out there, compare all the methodologies people are using, and match up what research and methodologies are being used to address which issues (though a lot of it may address multiple issues at once). We might see patterns in how people address different aspects of usability or value. This would of course be a monumental task and I have no idea how we would go about doing it. It would be great if we as a community had some sort of digital library assessment clearing house where we could all upload information about research that's been done (methodology and results)... and if I'm dreaming big, I'll keep dreaming big and add datasets to that list! It would be so great if we could download and use each other's data as well as see methodology and results. 

I may have veered slightly off topic there from what you were talking about Jody, but there are some of the ideas banging around in my head at 8:30am on a Tuesday.

[1] For example, this study looked at different levels of metadata for digital images https://staff.lib.ncsu.edu/confluence/display/MNC/Evaluating+the+effectiveness+of+manual+metadata+enhancements+for+digital+images

This study looked at GIS researcher's search and metadata needs: https://www.academia.edu/1156104/Juggling_Points_and_Polygons_GIS_Researchers_Metadata_and_Search_Needs

This study looked at cost and value of different elements of metadata in archival finding aids https://www.academia.edu/1715499/_User_Feedback_and_the_user_experience_New_ideas_for_analyzing_ROI_for_metadata._In_ALTCS_MIG_panel_Converging_Metadata_Standards_in_Cultural_Institutions_Apples_and_Oranges._

[Chelcie Rowell]

Just piping up to comment that a systematic review of digital library assessment literature — admittedly exhausting — actually sounds achievable.

[Jody DeRidder]

I think you're right, Chelcie.  It *is* achievable;  there really isn't *that* much out there.  There are a handful of articles already that cite almost everything available at the time of publication.

And I agree with you too, Joyce:  determining what metadata users actually want and need is critical to leveraging our resources effectively, yet there is almost NOTHING out there yet on this.  For example, I've been really surprised at the indicators I've seen that at least experienced researchers don't value subject headings, because they are interpretive and often vague.  

To answer your question, Joyce:  I think we need to figure out the best way to measure ALL those things.  :-)   Some of us could start from the bottom up, while others start from the top down.  By that I mean, from the ground up, some of us could identify what it would take to measure the effectiveness of metadata, or the content, or the search engine.  And from the top down, others could focus on identifying the best way to measure effectiveness, efficiency, satisfaction...

Jody L. DeRidder

Head, Digital Services
University of Alabama Libraries

Tuscaloosa, AL 35487
Phone: 205.348.0511

"Hope lies in dreams, in imagination, and in the courage of those who dare to make dreams into reality."
--Jonas Salk

> Just piping up to comment that a systematic review of digital library
> assessment literature — admittedly exhausting — actually sounds
> achievable.
>
> On Tuesday, May 13, 2014 8:39:54 AM UTC-4, joyce.chapman wrote:
>>
>> I think the most common types of assessment with digital libraries are
>> usability testing and the use of web analytics for really basic content
>> analysis (what stuff is getting looked at the most?) I agree that
>> usability
>> testing often returns results that are a bit of a jumble in regards to
>> what
>> exactly was tested: the interface? The content? The metadata? If we
>> don't
>> consciously plan studies to specifically address these different areas,
>> we
>> aren't going to be able to tease apart results very well. In particular

>> from the list you mention, I think *metadata* often gets the short end

>> of
>> the stick in digital library evaluation. Which is interesting because
>> libraries put so much time and energy into metadata creation. I've
>> worked
>> on a few studies that specifically evaluated digital library metadata[1]
>> and each time we had little success finding evidence of similar work or
>> literature around the topics.
>>
>> When you say you hope we can find a way to measure "these" aspects of
>> usability, do you mean the bulleted list or the second list
>> ("effectiveness, efficiency, satisfaction, learnability")?
>>
>> It seems like one way to get started would be to look at what's already
>> out there, compare all the methodologies people are using, and match up
>> what research and methodologies are being used to address which issues
>> (though a lot of it may address multiple issues at once). We might see
>> patterns in how people address different aspects of usability or value.
>> This would of course be a monumental task and I have no idea how we
>> would
>> go about doing it. It would be great if we as a community had some sort
>> of
>> digital library assessment clearing house where we could all upload
>> information about research that's been done (methodology and results)...

>> and if I'm dreaming big, I'll keep dreaming big and add *datasets* to

>> that list! It would be so great if we could download and use each
>> other's
>> data as well as see methodology and results.
>>
>> I may have veered slightly off topic there from what you were talking
>> about Jody, but there are some of the ideas banging around in my head at
>> 8:30am on a Tuesday.
>>
>> [1] For example, this study looked at different levels of metadata for
>> digital images
>> https://staff.lib.ncsu.edu/confluence/display/MNC/Evaluating+the+effectiveness+of+manual+metadata+enhancements+for+digital+images
>>
>> This study looked at GIS researcher's search and metadata needs:
>> https://www.academia.edu/1156104/Juggling_Points_and_Polygons_GIS_Researchers_Metadata_and_Search_Needs
>>
>> This study looked at cost and value of different elements of metadata in
>> archival finding aids
>> https://www.academia.edu/1715499/_User_Feedback_and_the_user_experience_New_ideas_for_analyzing_ROI_for_metadata._In_ALTCS_MIG_panel_Converging_Metadata_Standards_in_Cultural_Institutions_Apples_and_Oranges._
>>
>>>
>>>
>>
>> On Monday, May 12, 2014 3:52:06 PM UTC-4, Jody DeRidder wrote:
>>>
>>> One of the challenges I run into when performing user studies is in
>>> determining what exactly I'm measuring, so I know what it is that needs
>>> to
>>> be modified as a result. When I look at an online digital library, I'm
>>> thinking about:
>>>

>>> - the effectiveness and speed of the search engine
>>> - the encoded information on the website
>>> - the item-level or collection-level metadata
>>> - the actual content
>>> - the way the content is displayed and accessed
>>> - the browse functionality (an interplay with the metadata and
>>> usually the website encoding)
>>> - the refining and sorting capabilities (which are an interplay of

>>> the system with the metadata)

>>> - additional services built on top (like citation extraction,

> --
> You received this message because you are subscribed to the Google Groups
> "Digital Library Assessment" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to digital-library-ass...@googlegroups.com.
> To post to this group, send email to
> digital-libra...@googlegroups.com.
> Visit this group at
> http://groups.google.com/group/digital-library-assessment.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/digital-library-assessment/ce9a2967-4d30-47ae-bc2a-70f2ae404987%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

[Chelcie Rowell]

Tangentially related to Joyce's interest in cost/benefit analyses of metadata creation, I've been intrigued by this recent thought piece in the ASIST Bulletin by Jane Greenberg, Metadata Capital: Raising Awareness, Exploring a New Concept, which asks "Can we recognize metadata as an asset? Where and how might we begin an inquiry on this topic? And more precisely, can we quantify the value of metadata in some way that will allow us to gather empirical evidence as to its value?"

Jody, it strikes me that the information retrieval world has grappled with metrics for both higher order aspects of a system such as "efficiency" and lower order concepts of a system such as effectiveness of the browse functionality, e.g. how previewing the results of a search can help users to reformulate queries on the fly. I'm afraid I'm muddying the waters, but I guess my question is would/should a systematic review of digital library assessment that matches which methodologies are being used to evaluate which aspects of usability and value — would this systematic review zoom out from the library world to the information retrieval world?

A few more questions about metrics — What metrics can be used to evaluate the system? What metrics can be used to evaluate its usability from the perspective of the user? How do we interpret data from system evaluations and user studies? For example:

• Do batch and user evaluations give the same results? (ACM SIGIR '00)
  
• Why batch and user evaluations do not give the same results (ACM SIGIR '01)
  

> email to digital-library-assessment+unsub...@googlegroups.com.

[Jody DeRidder]

Hi Chelcie.  Thank you for the interesting links!

I don't think you'll find any argument from this crowd that metadata (in general) is an asset, as content cannot be located, managed or supported effectively without it.  I think the question is:  *which* metadata is most valuable, and which is not?  We cannot effectively leverage our resources without clarifying this question.  The cost of metadata creation is high;  it only makes sense to assess what is worth the cost.

Yes, of course, much has been explored in terms of information retrieval; it is, after all, the basis of librarianship.  We aren't trying to reinvent the wheel here, but I am hoping we will build upon what's already mapped out, with a greater focus on user needs and how to practically respond to needs with shrinking resources.

For example, if we were to find out that browse functionality is useless to a specific target audience, we could drop that development in digital libraries for whom that audience is a primary focus.  If we determine that the only metadata a particular audience wants is creator, date, location, and type, along with transcription of any text, then we know to focus on that for our efforts, as opposed to other metadata.    

If we find our target audience is not using our digital library because: 

a) they don't know it exists, 

b) it doesn't have the content they want, 

c) we failed to OCR or transcribe all the text so they can find what they need, 

d) we failed to provide useful browse functionality or refining/limiting options, e) they can't extract what they find when they find it ---   well, then, we know what to focus on. 

Without assessment, we cannot leverage our increasingly limited resources to provide the service our users need.

So the first question, in my mind, is:  what do our target audiences actually need?

Following that is:  how best do we provide it?

--jody

[Chelcie Rowell]

Thank you, Jodi. Your trenchant questions are incredibly helpful to someone so new to the field — like me!