Use of web analytics in design/evaluation of distance education

35 views
Skip to first unread message

George Siemens

unread,
Sep 9, 2010, 4:11:34 PM9/9/10
to learning...@googlegroups.com
Hi,


I see learning analytics developing somewhat along these lines/stages:

Stage 1: Incorporation of analytics methods and activities from other fields (web, primarily). This involves a basic analysis of what's happening - access time, # of hits, frequency of logins, etc.

Stage 2: Web analytics with social network analysis (this is what SNAPP does) - using logins/time/and interaction as a premise for analytics. Who is logging in? How often? Who is posting? Who talks to whom?

Stage 3: Distributed network analysis (capturing interaction across multiple platforms and networks).

Stage 4: Social integration: recommender systems - LA system begins to integrate social activities of people in a course or program to provide additional learning resources (think amazon, group tagging, digg, etc). See Handler & Berners-Lee's paper "From the semantic web to social machines" in Artificial Intelligence (2010).

Stage 5: Semantic/linked data. Data needs to be more intelligent in order to be meaningful (I tossed this idea out to the conference steering committee, but most didn't like the notion of intelligent data). Data that is well-defined can be repurposed in appropriate situations and contexts. As advocates of the semantic web state, semantic data is unambiguous data. Intelligent data serves as an important basis for personalization (more on that below).

Stage 6: Knowledge analytics - this isn't quite the right word, but at this stage, through analytics, educators will be able to get a sense of whether or not learners are mastering subject areas and where key deficiencies exist. At this stage we are seeing an emphasis on "what is being talked about". 

Stage 7: Intervention/Personalization/Adaptation - this is technically outside the realm of learning analytics because this stage provides personalization and adaptation to learners (content and social) or intervention of the part of the educator/school/system. However, the relationship is iterative - what we learn through analytics influences how and what we personalize/adapt...which in term feeds back into the system for additional data for analysis.

Stage 8: Physical/virtual worlds analytics: our personal life recorder captures what we do and say and who we interact with in order to provide a more holistic analysis of our learning and development.

Stage 9: Full integration: Analytics bringing together all aspects of what we do on a daily basis, drawing connections between email interactions, web search, physical interactions, "internet of things", etc to provide not only an analysis of what we're doing, but a full person knowledge analysis, offering to assist us in forming connections between ideas and concepts that we have not yet explicitly integrated in our activities. Predictive modelling serves a role here as well. Basically, we are "known" by the LA system and we enter a symbiotic relationship with technology where both entities (human and technology) do what each does best. 

These are not sequential stages - there will be overlap and simultaneous development. But each stage will need a degree of maturity before the next stage can develop fully. A few simultaneous activities (occurring at most/all stages):

1. Better methods of analysis so we can extract better understanding from the interactions of learners with content and each other.
2. Easier to use tools (Gephi, Gridbase)
3. Better, more meaningful visualization techniques
4. Development of predictive/probabilistic models...as well as inferences made by the LA system based on data we have provided. For example, if we haven't explicitly stated certain views or perspectives (political, for example), the system will be able to guess based on our existing social network and explicit content we've provided.

On a slightly related note, learning analytics should not only be available to the educator or institution. Learners should be able to see exactly what others see. Analytics should assist learners in better managing their own learning...

George

David Wiley

unread,
Sep 9, 2010, 6:26:29 PM9/9/10
to learning...@googlegroups.com
Consolidating thoughts triggered by a few emails here.

I think the reason learning analytics is emerging now, as opposed to
say 10 years ago, is the incredible ease and very low cost with which
we can now capture very large amounts of data about learners and their
activities. These increases in ease / decreases in cost are best
enabled when learners engage in their learning activities online -
when learning activities are technology-mediated, they can also be
technologically recorded. To me, the analysis of manually created,
expensive-to-gather data (like classroom observations) doesn't fall in
the category of learning analytics - else we would simply have renamed
the field previously known as "educational research" as "learning
analytics," because the analysis of any education-related data would
qualify. I think this new subset of educational research deserves a
new label because it deals with something new - the huge amounts of
learner activity data so easily and inexpensively captured when those
activities are mediated by technology.

I will also freely admit that I have exactly 0% faith in full
automated instructional approaches, like "intelligent" tutoring
systems, and that to my mind the best purpose of learning analytics
approaches is to provide actionable information to one or more people,
whether they are acting in the role of teacher or learner at the
moment. George uses the phrase "extract better understanding from the
interactions of learners with content and each other." One legitimate
way of viewing these interactions is as actions, or to use a more
loaded term, behaviors. I believe we all know what we call approaches
to learning (or analysis) that focus exclusively on observable
behaviors. Michael Atkisson and I will be submitting a paper to the LA
conference on the unhappy possibility of learning analytics turning
into a sort of "Behaviorism 2.0." Traditional behaviorist work relied
exclusively on the analysis of observable behaviors, ignoring
unobservables. At least they could directly observe behaviors! The
learning analytics approach is abstracted away even further from the
unobservables, using a significantly impoverished behavior vocabulary
that includes only keyboard and mouse events. It seems absolutely
critical to me that the results of LA can provide only a portion of
the data necessary for making decisions, and that it must be a human
with more subtle meaning-making capabilities that ultimately acts on
the data coming out of LA.

D

Leigh Blackall

unread,
Sep 9, 2010, 6:52:26 PM9/9/10
to learning...@googlegroups.com
Good logical points I think David.

David wrote: " to my mind the best purpose of learning analytics approaches is to provide actionable information to one or more people, whether they are acting in the role of teacher or learner at the moment."

By way of an easy concrete example I experienced yesterday:
  1. A teacher I help with using social media, is using a web cam and youtube to put out "mini lectures" to his students.
  2. Before we started using Youtube, we attempted to load the same sorts of videos to Moodle with various problems that lead us to Youtube instead.
  3. Each week we load a new video to Youtube as unlisted - meaning only people with the URL can see the video.
  4. Youtube gives a variety of data on each video, both to the teacher and the people studying his course. Most notably in this example is the Number of Views.
  5. I showed this teacher the various points of data that Youtube gives, and he noticed that the Number of Views were considerably less than the number of students he has, leading us into a very productive discussion on various ways to improve engagement, one idea involving a negotiation with the students based on their ability to see the number of views also. (A bit like early comments in this group about only 10% of the people here being active.)
  6. The point here is, accessing that data is VERY simple. Especially compared to Moodle.. that ease of access, and the actual data has resulted in very positive actions in my view. I doubt this teacher would ever use a special and more complicated tool for getting so-called Learning Analytics.
Perhaps that's too simple an example for most here.. sorry if so, just wanted to support what David was saying.
--
--
Leigh Blackall
+61(0)404561009
skype - leigh_blackall
leighb...@gmail.com
Website

Prashant Kaw

unread,
Sep 10, 2010, 5:30:22 PM9/10/10
to Learning Analytics
Gents -

This is an extremely stimulating discussion and group! @George, I
really like how you have broken down Learning Analytics into micro
segments. I agree there is no sequential order, in fact there cannot
be as created an integrated solution is a mammoth task. In fact each
stage is potentially an industry in itself.

@David - While we are not building a system to automate instruction
(even though the late Arthur C. Clarke would like us to) we are
building tools that will easily synthesize data and create actionable
information so instructors can make data driven decisions. My company
Socrato.com is focused on Stages 6 (Knowledge Analytics) and 7
(Intervention/Personalization/Adaptation). FYI - I come from a Web/
Marketing Analytics background.

Check out www.websitegrader.com or yotta.com or compete.com. These
are tools that analyze websites and provide recommendations to improve
them in various areas like improved load times/performance, increased
traffic, better search rankings. We are taking a lesson from this
space and hope to provide intelligent recommendations to students and
teachers through test diagnostics and performance reports. These
reports are meant as aids and not replacement for instructors.

@leigh - interesting observation on the YouTube videos. Great ideas
for intervening and ensuring students are watching the information.
One thought is understanding how YouTube counts a view. Only if a
video is watched through to the very end is it considered a view. So
if there is 3 or 5 second splash screen at the end of the video and a
student surfs away or exits the browser that is not counted as a
view. YouTube is a lot better than moodle but it is still not perfect
and that is just the case with all analytics - the nuances.

What is the margin of error that will be tolerable in education. In
business one can get by with directional data. I'm curious what kind
of a stomach for directional data there is in the education community
because not tracking tool is perfect.

Cheers,
Prashant / @prashantkaw @socrato


On Sep 9, 6:52 pm, Leigh Blackall <leighblack...@gmail.com> wrote:
> Good logical points I think David.
>
> David wrote: " to my mind the best purpose of learning analytics approaches
>
> is to provide actionable information to one or more people, whether they are
> acting in the role of teacher or learner at the moment."
>
> By way of an easy concrete example I experienced yesterday:
>
>    1. A teacher I help with using social media, is using a web cam and
>    youtube to put out "mini lectures" to his students.
>    2. Before we started using Youtube, we attempted to load the same sorts
>    of videos to Moodle with various problems that lead us to Youtube instead.
>    3. Each week we load a new video to Youtube as unlisted - meaning only
>    people with the URL can see the video.
>    4. Youtube gives a variety of data on each video, both to the teacher and
>    the people studying his course. Most notably in this example is the Number
>    of Views.
>    5. I showed this teacher the various points of data that Youtube gives,
>    and he noticed that the Number of Views were considerably less than the
>    number of students he has, leading us into a very productive discussion on
>    various ways to improve engagement, one idea involving a negotiation with
>    the students based on their ability to see the number of views also. (A bit
>    like early comments in this group about only 10% of the people here being
>    active.)
>    6. The point here is, accessing that data is VERY simple. Especially
> > On Thu, Sep 9, 2010 at 2:11 PM, George Siemens <gsiem...@gmail.com> wrote:
> > > Hi,
> > > An overview article on using web analytics in
> > > DE:
> >http://www.aupress.ca/books/120177/ebook/12_Veletsianos_2010-Emerging...
> leighblack...@gmail.com
> Website <https://sites.google.com/site/leighblackall/>

Lynne Quartermaine

unread,
Sep 12, 2010, 2:27:12 AM9/12/10
to learning...@googlegroups.com
Gents???? There are some women contributing....

Lynne

________________________________

winmail.dat

Prashant Kaw

unread,
Sep 13, 2010, 9:47:37 AM9/13/10
to Learning Analytics
Lynne -
My sincere apologies. I am aware there are 100+ members in the group
including women. My reply was directed to the 3 gents (George, David
and Leigh) in that particular conversation! So I would be grateful if
you don't read more into my response that just that. I will be
mindful that a group discussion is broadcast to all members.
Thanks for understanding.
Prashant

On Sep 12, 2:27 am, "Lynne Quartermaine"
<L.Quarterma...@curtin.edu.au> wrote:
> Gents???? There are some women contributing....
>
> Lynne
>
> ________________________________
>
> From: learning...@googlegroups.com on behalf of Prashant Kaw
> Sent: Sat 9/11/2010 5:30 AM
> To: Learning Analytics
> Subject: [learninganalytics] Re: Use of web analytics in design/evaluation of distance education
>
> Gents -
>
> This is an extremely stimulating discussion and group!  @George, I
> really like how you have broken down Learning Analytics into micro
> segments.  I agree there is no sequential order, in fact there cannot
> be as created an integrated solution is a mammoth task.  In fact each
> stage is potentially an industry in itself.
>
> @David - While we are not building a system to automate instruction
> (even though the late Arthur C. Clarke would like us to) we are
> building tools that will easily synthesize data and create actionable
> information so instructors can make data driven decisions. My company
> Socrato.com is focused on Stages 6 (Knowledge Analytics) and 7
> (Intervention/Personalization/Adaptation). FYI - I come from a Web/
> Marketing Analytics background.
>
> Check outwww.websitegrader.comor yotta.com or compete.com.  These
> ...
>
> read more »
>
>  winmail.dat
> 19KViewDownload

Lynne Quartermaine

unread,
Sep 22, 2010, 9:52:17 AM9/22/10
to learning...@googlegroups.com
Thank you, Prashant, for your dignified response to my not very dignified blurt. I did not disappear, witch-like, in a cloud of huffy smoke - I have not been on-line for the past ten days as I have had the worst gastric 'flu in the world. Serves me right for getting in a snit - we ancient feminists do have our touchy sensitivities.
I do enjoy reading what everyone has to say- all new territory to me- thank you all for your generosity and sharing.
winmail.dat
Reply all
Reply to author
Forward
0 new messages