ANALYSIS OF DISTANCE ED COURSE WITH WEB ANALYTICS

One of the things I found most difficult in analyzing raw web statistics with this course (and it seems that I would find this frustrating with any similar project) was that I felt like I was doing some guesswork without having spoken to the client.

There are no goals set (perhaps that's part of the end assignment), and there is really no way for me personally to know what the client would like to see happening. Do they want more course evaluations? Would they like to know how many people finished the course? Do they care that people don't spend much time on any given page? These are all questions that could easily be answered and filtered through a couple of sit-downs.

My assumptions, along with the data I looked at, led me to these observations and recommendations:

• Of a total of about 4,300 page views of the first page of the first lesson (I have no access to actual enrollment data), only 15 or 16 visits were made to the course evaluation page. Again, not knowing how many actual course evaluations were submitted, this would reflect at most 15 or 16 course evaluations, or an evaluation rate of 0.37%. I don't know what Independent Study's intentions are for course evaluation, but this seems to be to be a very, very low number. My recommendation would be to integrate the course evaluation better into the course. It could be placed more strategically, perhaps, to give more attention or incentive to the student to complete the evaluation.
• Looking only at the past month, even the few evaluations that were visited (not necessarily completed) have quite a bit of variance. One student spent almost four minutes on the evaluation page, while two others spent 26 and 42 seconds--hardly enough time to fill out an honest or thought-out evaluation. It might be interesting to see if there was more than one evaluation submitted, and to be able to tie that evaluation to the time spent. My guess would be that if a student completed the evaluation in around 30 seconds, it's probably safe to throw that evaluation out. A recommendation could be to modify or take a new approach to the course evaluation so as to be both more effective and more inviting to complete. Maybe you could even scatter evaluation questions throughout the course?
  
• As an overall recommendation, the analytics for Math 110 needs filters and especially limited access. I can access any page on the course, which doesn't seem like it would help accuracy in determining which pageviews are from students/nonstudents. And any visits from students (such as those from our class) would cloud the undermine the integrity of the data.
  

Overall, I think I have more questions than I have recommendations or answers. I imagine we'll revisit this course.

EDUCATING ANALYSTS

After an exhaustive 1/2 hour search (hey, everything happens at greater speeds nowadays, right?) for the uses of web analytics in education, I came across a gem of a starting point. The title: Using Log File Analysis to Evaluate Instructional Design — A PowerPoint presentation used in Ken Fanser and Rod Riegle's presentation at the 2003 Sloan-C International Conference on Online Learning.


This resource hits a lot of really great ideas about using Web Analytics as a valuable tool for evaluation. It's dated but at the same time still very applicable. Some points I found interesting to consider after reviewing this presentation:

• One of the biggest barriers to entry for someone wanting to use a "log file" to evaluate their online instruction was getting the data in a readable format. No more. Tools like Google Analytics completely eliminate a lot of these painful and complicated steps of getting set up to see who's looking at your content.
  
  
• Some of the proposed limits to web analytics for use in education are still valid, while others I don't believe to be such a problem. Por exemplo, it still can be easy to misinterpret results from analytics tools, most of which are geared towards business goals rather than educational goals. Yet, the amount of data you can get from one user is not quite as limited as the presentation suggests, as well as some of the issues raised with rich media (Flash) aren't quite as valid. There are plenty of ways to work around/with rich media to cater to your analytics.
  
  
• The presentation raises some very important questions as to the effectiveness of analytics in online education.
  
  
• A fundamental question, quite possibly one of the only questions that needs to be asked in evaluation, is "Did learning occur?" How can that be determined from page views? Maybe with deeper analysis, we can approach a way to guess. Yet, isn't that a struggle of any educational medium, to assert whether or not learning did, in fact, occur?
  
  
• Another question that may be difficult to answer with web analytics: Was the content engaging?
  
  
• A more elementary question: How do I link student X to user X? Is it possible to evaluate effectiveness of instruction with web analytics at the single user level?

All in all, a great find for the subject. I wish I could get access to the recorded presentation.

METRICS FOR ANALYTICS

Some quick thoughts on useful things to measure for web analytics:

• Number of unique visitors: I know most analytics programs or server-side analytics engines can track this, but one of the most important lessons I've learned in analytics is that hits != visitors! With education, just as important.
  
  
• Time spent on a page. Difficult to measure (how to determine whether someone is reading or taking a break), but very useful. A lot of time spent on a page can be a very good thing.
  
  
• Conversion/goal tracking. Google analytics has a slick way of dealing with this. Basically, you want your users/learners to follow some sort of prescribed path--this can be tracked. If a student went from point A to point E, then back to B, and never ended up on the critical F, then something ain't right.

Also, some wish list items:

• More data per page. Imagine analytic technology that showed data as to what areas of the page were most focused on, not necessarily clicked on. In usability testing, this happens with a camera mounted on the user's computer that tracks eye movements. Not feasible from a broad analytical perspective, but I suspect it may not be as impossible as it sounds.
  
  
• More visualization. I'd love to see more dynamic, engaging, and at the same time useful visualizations of things like traffic demographics mashed with other tracked areas.