Your manager needs your help :)
Sebastian Kruk
as some of you might be aware - I am pretty close to finishing my PhD.
The last thing I need is evaluation of semantic and social features in
JeromeDL.
I have put together a evaluation apparatus, where JeromeDL can be
compared with DSpace in a series of tests.
For the last 2 weeks, together with Kasia, we were trying to encourage
her group of 23 students to help us in the evaluation.
The problem is , that after the second round of questions we have only
5 of them who answered them - which put me in a very depressing mode.
Since some of you have already helped me with evaluation of MBB and
Treemaps/Tagclouds, I have decided to ask you for the help once again.
The evaluation scenario would be as follows:
The evaluation procedure will consist of 6 consecutive phases. Phases
2, 4, and 5 should have tasks which adhere to three types of user
goals: resource seeking, navigational, and informational. With enough
subjects we could mix the questions - and determine which part of the
S3B life-cycle delivers the best results to given type of goal.
Introductory tutorial to the digital library system.
Duration: 1 week
Task: Try out each feature of the digital libraries
Objective: Subjects get acquainted with the system they are going to
use.
Two tutorials (one for each group of subjects) will be available for
one week day. Each subject will be given credentials to access the
digital library he/she has been appointed to. Subjects will be
encouraged to use each of the features delivered by given library. We
should mention that the better they will get to know the system, the
more likely they will have less problems in answering questions
prepared in later phases of the evaluation. We will monitor which
features of each system, and how heavily, were used.
Searching for the answer
Duration: 45 min (anytime within given 1 day)
Task: Answer one question
Objective: Perform baseline comparison between JeromeDL and DSpace
search features.
During this phase the only useful search features in JeromeDL will
be SQE (with only partial information required by the algorithm), NLQ,
plus filtering (TTM and Exhibit) and browsing components (MBB).
Networking (JeromeDL only)
Duration: 1 day
Task: Link to other users of the library
Objective: Introduce social network connections between the users of
the JeromeDL library
The next day after the phase 2 subjects using JeromeDL system will be
asked to try to network with their friends, and other users of the
system. This will enable SSCF feature in the next phase
Collaborative search and browsing
Duration: 45 minutes (anytime within given 1 day)
Task: Answer two questions
Objective: Compare collaborative search and browsing features in
JeromeDL with baseline results from DSpace
During this phase subjects should appreciate the collaborative
filtering and browsing features provided by SSCF and MBB. They should
be encourage to put resources into their bookmarks, annotate them,
blog them (SSCF-blog feature), and share their MBB queries. Users will
also be encouraged to subscribe to categories maintained by their
friends.
Recommendations
Duration: 45 min (anytime within given 1 day)
Task: Answer one question.
Objective: Evaluate recommendations features in JeromeDL
In this phase recommendations feature in JeromeDL and in SSCF will be
enabled. This phase will unroll access to all search, filter, and
browsing features in JeromeDL.
I have attached evaluation questionnaires (there will be 2 more - but
cannot show them before the evaluation is completed).
Please let me know if could help me with this evaluation. I think we
could start just after Christmas (with one week for playing with
JeromeDL and DSpace), and spend some time for this 3 days of
evaluation at the very beginning of Jan 2008.
I am also open to modify the schedule to suit you better - e.g. to
have 12h only for each task, or to do Networking together with
Searching, or for example, to allow to do Searching, Collaboration and
Recommendations in sequence within couple of days, without clear
division of 24h for each task.
And ... there will be prices as well - this time we have "Eagle-eyed
Searcher Competition" (ref. http://blog.corrib.org/?p=23).
Best, and Merry Christmas,
Sebastian