Learning Analytics for Educational Design and Student Predictions: Beyond the Hype with Real-Life Examples

Title:  Learning Analytics for Educational Design and Student Predictions:  Beyond the Hype with Real-Life Examples


  • Nynke Kruiderink, Teamleader, Educational Technology Social Sciences, University of Amsterdam
  • Perry J. Samson, Professor, Department of Atmospheric, Oceanic & Space Sciences, University of Michigan-Ann Arbor
  • Nynke Bos, Program Manager, Educational Technology, University of Amsterdam


SLIDE:  Lessons Learned:  February 2012 – Present

Our Proof of Concept is Two-Tiered

  1. Interviews with lecturers, professors, managers
  2. Gather and store data in central place for easy access

General things we learned

  • Emotional response to “Big Brother” aspect of accessing data
  • Data from LMS not detailed enough (folder based, not file based)
  • 50% of learning data available
  • Piwki, not secure enough


Next Steps

  • Focus group:  learning analytics
  • Professor Erik Duval – KU Leuven  (his advice: undertake one project involving all that proves learning analytics is useful)

What is the Problem?

Recorded lectures

Recording of F2F lectures

No policy at the University of Amsterdam

Different deployment throughout the curriculum

Not at all (fears/emotional)

Week after the lecture

Week before the assessment


Student vs. Policy

Students demanded policy

QA department wanted insight into academic achievement before doing so

Development of didactic framework

Research:  learning analytics



  • Two courses on psychology
  • Courses run simultaneously
  • Intervention in one condition, but not the other


Data Collection

  • Viewing of recorded lecture
  • Lecture attendance per lecture
  • Final grade on the course (with more segmented view)
  • Some other data (grades on previous courses, distance to the lecture hall, gender, age, etc.)


Lessons Learned

  • Let people know what you are doing
  • Data preparation:  fuzzy, messy
  • Choose the data
  • Simplify the data
  • Keep an eye on the prize


LectureTools:  Analytics

Females in class were much more likely to ask questions using a clicker (lovingly referred to as the “WTF” button).

90% of students at University of Michigan in the meteorology class cited felt they would have gotten the same grade if they had never opened the textbook








Ethics and Analytics: Limits of Knowledge and a Horizon of Opportunity

Title:  Ethics and Analytics –  Limits of Knowledge and a Horizon of Opportunity


  • James E. Willis III, Ph.D., Purdue University
  • Matthew D. Pistilli, Ph.D., Purdue University


(See my related post, “The Edward Snowden Affair: A Teachable Moment for Student Affairs and Higher Ed“)


Highly interactive session, sat in groups of five and discussed among ourselves.

SLIDE:  Lack of Discussion of Ethics in Data and Technological Innovation


  • Cutting-edge technologies are being developed every dy
  • Cost/profit margins are determined with speed of development
  • Typeicall education lines are split between science adn humanities/technolgoy and ethics
  • Difference between what can be done and what should be done


SLIDE:  Where to Go from Here?

  • Begin the discussion now
  • Have the difficult converations
  • Bring together the stakeholders:  data scientists, engineers, managers, ethicists
  • Establish a framework to adapt quickly to questions of whether or not an action or research agenda should occur


SLIDE:  Ethical Discussions = Innovation

Logic of negation

  • Why shouldn’t we do this?
  • What should we do instead?

Different paths emerge from divergent conversations

  • Stakeholders have different voices and understandings of potential paths of development


SLIDE:  Potter’s Box


  • ID values
  • Appealing to ethical principles
  • Choosing Loyalties
  • Responsibilities and recommendations

How Potter’s Box is useful for ongoing discussions

If you know a student is at risk, what do you do?  What is your responsibility?


SLIDE:  Research Questions

  • What do the ethical models look like?
  • How are these models deployed rapidly – at the speed of technology?
  • How are these models refined with time?


SLIDE:  Group Discussion 1

What are the current projects going on in learning analytics today?  What are the potential ethical pitfalls that surround these developments?  Why are they potentially harmful?  Are these things always wrong or are they contextually wrong?

Some of the ideas generated by the room:

  • Type 1 / Type 2 error:  will the student pass this class?  What’s my prediction – will the student pass or fail the class?  How accurate is your prediction – did your model work?
  • Is it right to get information from the students?  Where does the locus of control lie?  Does the institution need to take more responsibility for this?
  • University One (FYE equivalent in Canada) – informal collection and sharing with the next year.  Are we branding / labeling students appropriately?  Are we sharing this information appropriately?  Who should know what and at what point?  Will that affect their future studies?
  • Top-down approach using a product with funding from the Gates Foundation (similar to the InBloom project).  Goal is to make a product that analyzes what students should be taking.  Pitfalls:  don’t know model, don’t know the raw data.


SLIDE:  Group Discussion 2

What is the role of “knowing” a predictive analytics – once something is known, what are the ethical ramifications of action or inaction?  What are the roles of student autonomy, information confidentiality, and predictive modeling in terms of ethical development of new systems / software / analytics?

  • How much do we really know?  If we have a high level of confidence, what is our responsibility?
  • Could it be solved by giving students opt-in versus opt-out?
  • Discovering things, going down certain paths that may raise ethical questions about students who may not succeed…i.e. financial burdens that may be incurred due to failure.


SLIDE:  Group Discussion 3

How might we affect the development of future analytics systems by having ethical discussions?  What are possible inventions that could come from these discussions?

  • See more active participation from the students, with a better understanding of how their data is going to be used.
  • Obligation to allow students to fail; channeling people into things they can be successful with.
  • EU regs:  disclosure, portability, forget
  • Portability:  health care analog; your info can go from one hospital to another, and that data has real-world benefits to patients.
  • Information can also say a lot about faculty too.


SLIDE:  Group Discussion 4

What are the frontiers of ethical discussions and technology?  Are there new frameworks?  How can we affect the future by today’s discussions?

  • Maybe you can redefine what student success is?  Maybe it isn’t graduation rates…
  • How do we affect the future?  It’s all of our roles to be a part of this conversation!  You don’t really have the option to say “I don’t have the time for this”