2014 DLF Forum session?

[Jody DeRidder]

I'm wondering how best to move this effort forward, and what would be the best focus for a session on DL assessment at the upcoming forum.

Please vote on your preferences, and if you have a case study or findings you'd like to share that may be helpful, please note that as well.

A possible framework might be something like this:

1)     1) Make a list of primary user groups

•         Identify likely overlapping needs
  
•         Identify likely specific needs
  
•          Do studies to verify /correct theoretical needs specifications
  

2)     2) Divvy up basic areas for interface evaluation (ex:  search, refinement/limiting, browse, results lists, item access, information extraction, database content, online findability)

-          Determine best methods to evaluate each one across a range of interfaces, and provide guidelines & references

3)     3)  Make list of benefits measurements

•           Identify methods for collection and interpretation
•           Weigh pros / cons and applicability of each (for different user groups?)

4)     4)  Costs 

•         Make list of measurements it’s feasible to include
  
•         Collect information from multiple institutions about time for digitization for different types of material on different types of capture equipment
  
•          Analyze results of collected information
  
•         Build reusable online matrix for estimating costs (and noting limitations)
  

What do you think?  What is missing here, and what should be changed?

Please respond to the poll so we can develop or provide the most useful gathering at this year's forum.

Thank you!

--jody

[Rita Van Duinen]

Hello!  How many of you submitted proposals to DLF?  We want to start the conversation with you soon on building out some curriculum offerings through CLIR/DLF.

[Joyce Chapman]

Jody DeRidder, Nettie Lagace, Ho Jung Yoo and I submitted an assessment-related proposal together.

(jlder...@ua.edu, nla...@niso.org, hjs...@ucsd.edu)

Joyce

--
You received this message because you are subscribed to the Google Groups "Digital Library Assessment" group.
To unsubscribe from this group and stop receiving emails from it, send an email to digital-library-ass...@googlegroups.com.
To post to this group, send email to digital-libra...@googlegroups.com.
Visit this group at http://groups.google.com/group/digital-library-assessment.

To view this discussion on the web visit https://groups.google.com/d/msgid/digital-library-assessment/87dd1f2b-efa9-41e4-b397-99758b8dd127%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Jody DeRidder]

Our session proposal was accepted!  And the program committee has asked that we continue the conversation with attendees over lunch as well.

I hope you will be able to join us!  Here's more about what we'll be presenting:

Title: Moving Forward with Digital Library Assessment

Description: Assessment is a necessity: in the face of diminishing resources and tremendous demand for online access to research materials, most of us are pressed to justify our need for funding, and seek to maximize our resources to best serve user needs and provide a strong return on investment. There are multiple facets to assessment: evaluating interfaces, content, benefits, and impact, comparing online services to user needs, and measuring costs. Building off of two sessions at last year’s conference, this session will address several of these facets in an effort to identify next steps toward developing best practices and guidelines. An important emerging area for evaluating impact of digital libraries is the use of altmetrics; Lagace will report on NISO’s current community effort to evaluate and standardize various forms of alternative impact measurements. Chapman will discuss a developing framework to assist institutions in developing viable cost estimates for proposed digitization projects. DeRidder will report on a qualitative study of faculty researchers utilizing a broad array of online primary source interfaces that identified gaps in services and unmet needs, and Yoo will present a case study evaluating the usability of the UCSD Digital Collections through user interactions with the website. Following these presentations, we will engage the audience in small group discussions about what steps should follow in efforts to develop best practices and guidelines for digital library assessment in key areas, and how best to proceed.

Session Presenters:

• Joyce Chapman, Consultant for Communications and Data Analysis, State Library of North Carolina (soon to be the Assessment Coordinator at Duke University Libraries!)
  
• Jody DeRidder, Head of Digital Services, University of Alabama Libraries
  
• Nettie Lagace, Associate Director for Programs, National Information Standards Organization
  
• Ho Jung Yoo, Technical Analyst & SuHui Ho, Digital User Experience Librarian, UC San Diego Library
  

Please plan to join in the conversation:  bring your considerable expertise and thoughts to the table!

--jody

[Rita Van Duinen]

Congratulations!  We are looking forward to your session and the conversation that follows!

[Jody DeRidder]

Thanks to all of you who helped make our DLF session on assessment such a success!  We had some great discussions, and the notes we took are available here. 

For those who weren't present, we split up into 3 groups, each focusing on one of these topics:  user studies, costs, and altmetrics.  The questions posed to each group were these:

• What are the critical aspects that we need to address?

• What are the next steps we can take?

About 30 people signed up to continue this conversation, and to try to make some targeted headway on each of these topics over the next year.  Potential goals may include:

• Establishing a central location for collecting and developing best practices, guidelines, and resource links (I'm hoping we can get DLF to host a wiki site for us)
  
• Developing use case scenarios for altmetrics, building on what NISO has developed thus far
  
• Help develop meausers for peer review to minimize gaming of altmetrics
  
• Further development of Joyce Chapman's digitization cost estimation calculator, with more data submitted from other institutions
  
• Develop a white paper addressing the current status of user studies and directions for developing best practices/guidelines
  
• Research into some of the issues that still need to be resolved
  
• Possibly developing planning grant proposals for further development
  

Would assisting us in these or other goals help further your career?  Whatever we do should benefit all of us.

To target our conversations, I'm collecting emails of those who are willing to actually work on moving us forward in the next few month, breaking them down into 4 categories at the moment: altmetrics, analytics, costs, and user studies.  These will become informal working groups, I think, and if the lists become too large, I'll set up separate google groups for them.

If you would like to be in one of these working groups -- or would like to establish another, around a different topic!  -- please let me know.

Thanks!

--jody

Jody DeRidder

Head, Digital Services

University of Alabama

jo...@jodyderidder.com

[Stacy Konkiel]

Jody,

This all looks great! Please add me to the interest list for the altmetrics section. I also want to share some thoughts on the altmetrics aspect of the work the group might possibly do, and what was discussed at the forum:

• NISO use cases for altmetrics: we should absolutely contribute, but only in the context of digital library (and IR) assessment. There are enough possibilities for other ways librarians can use altmetrics that it makes the head spin and the gut clench to think about possible scope creep. Whatever we end up doing, keeping it tightly focused on use cases in the context of digital libraries is of utmost importance. There are some good gems for use cases in the "altmetrics in DCs and & IRs" white paper we circulated [1], and I encourage you to reuse much of that, so you're not reinventing the wheel. By no means should we get involved with articulating faculty use cases--that's up to faculty to do, and what NISO got that big fat grant to facilitate :) Same for institution-level use cases. At most, we should be helping NISO make connections with people at our institutions, and then stepping back and letting them work together, without our involvement. (After all, do we really need more meetings on our calendars? :))
• "Help develop meausers for peer review to minimize gaming of altmetrics": I think this is outside of the scope of this group and the DLF, for the most part. Gaming is something many are worried about, but few altmetrics aggregators have encountered to date. That's because altmetrics aren't central to the rewards and incentives systems in scholarship (yet), so no one's bothering to spend time doing it (yet). On the other hand, if you were to come up with proposals as to how we should mitigate gaming (both intentional and unintentional) in *digital libraries,* it'd be within the group's scope. (As for what peer review has to do with that, I'm unclear and would welcome more info.)
• Outreach to non-DL librarians about altmetrics: also outside of the scope of the DLF and this group. Raising awareness within DLs, though, is totally something to aim for.
• Sentiment analysis of mentions and metrics: this is another issue that altmetrics researchers are discussing, but no one's figured out. I think it's outside of the scope of the DLF community to try to crack that nut; rather, we should keep an eye on developments from altmetrics aggregators and apply any technology that appears down the pike. And until then, providing people access to the underlying qualitative data, so they can decide for themselves whether a mention is good, bad, or inconsequential, is important.
• Metric audit-ability, link rot, etc: on the DLF session notes, someone mentioned that linkrot exists, and asked if we're expected to start backing up the data we collect (say, if a paper is mentioned in a news article or blog, and then the link to that resource breaks). Absolutely not. What we need instead is a system like COUNTER that can certify that such web transactions happened, instead.
• "It needs to be obvious what is not included as well": why? Aside from a disclaimer that the metrics aren't comprehensive, I don't think we need to go overboard with information. 
• "Uses vary, so should the measures vary for the use?  For example, the kinds of information sought for tenure and promotion purposes might be different from the kinds of information sought for relevance ranking for discovery systems.": we should only concern ourselves with articulating what measures are necessary for *our* uses, and leave it to faculty to sort out what's useful for P&T, etc. (Of course, if they ask us to begin including certain metrics in our IRs that'll help them in their pursuit of P&T, we should definitely work with them to make that happen.) Iterative implementation, beginning with small features that meet our own needs, is key to keeping deployment from being bogged down by too many stakeholders' needs at once. And it'll keep us focused on articulating and championing the needs that we know best: our own.

I also want to gently remind folks that altmetrics = Altmetric.com. (There seems to be some conflation on the DLF Forum notes from the assessment session, which are overall really great!) They're an altmetrics aggregator like PlumX and Impactstory (for which I work), one that often works with IRs but is not the *only* third-party solution. So, keep that in mind when writing/discussing the use of altmetrics in your DL or IR. :) Altmetrics overall are comprised of more sources, and can track more kinds of content, than Altmetric.com currently does.

All best,

Stacy Konkiel

[1] https://docs.google.com/document/d/1uVn0BsY4RAZN7gNI3Jnbp9UiFkiQ5UDBAzceM-WoKiE/edit

--
You received this message because you are subscribed to the Google Groups "Digital Library Assessment" group.
To unsubscribe from this group and stop receiving emails from it, send an email to digital-library-ass...@googlegroups.com.
To post to this group, send email to digital-libra...@googlegroups.com.
Visit this group at http://groups.google.com/group/digital-library-assessment.

To view this discussion on the web visit https://groups.google.com/d/msgid/digital-library-assessment/7c70f4e1-9722-4441-ad24-75e79af52963%40googlegroups.com.

[Stacy Konkiel]

Typo!

In my previous email, I incorrectly wrote that "altmetrics = Altmetric.com," when the opposite is true. Altmetric.com is only one service that delivers altmetrics. :) The field of altmetrics as a whole is much more comprehensive.

Stacy