Categories
Technology

Learning Analytics for Educational Design and Student Predictions: Beyond the Hype with Real-Life Examples

Title:  Learning Analytics for Educational Design and Student Predictions:  Beyond the Hype with Real-Life Examples

Presenters:

  • Nynke Kruiderink, Teamleader, Educational Technology Social Sciences, University of Amsterdam
  • Perry J. Samson, Professor, Department of Atmospheric, Oceanic & Space Sciences, University of Michigan-Ann Arbor
  • Nynke Bos, Program Manager, Educational Technology, University of Amsterdam

 

SLIDE:  Lessons Learned:  February 2012 – Present

Our Proof of Concept is Two-Tiered

  1. Interviews with lecturers, professors, managers
  2. Gather and store data in central place for easy access

General things we learned

  • Emotional response to “Big Brother” aspect of accessing data
  • Data from LMS not detailed enough (folder based, not file based)
  • 50% of learning data available
  • Piwki, not secure enough

 

Next Steps

  • Focus group:  learning analytics
  • Professor Erik Duval – KU Leuven  (his advice: undertake one project involving all that proves learning analytics is useful)

What is the Problem?

Recorded lectures

Recording of F2F lectures

No policy at the University of Amsterdam

Different deployment throughout the curriculum

Not at all (fears/emotional)

Week after the lecture

Week before the assessment

 

Student vs. Policy

Students demanded policy

QA department wanted insight into academic achievement before doing so

Development of didactic framework

Research:  learning analytics

 

 Design

  • Two courses on psychology
  • Courses run simultaneously
  • Intervention in one condition, but not the other

 

Data Collection

  • Viewing of recorded lecture
  • Lecture attendance per lecture
  • Final grade on the course (with more segmented view)
  • Some other data (grades on previous courses, distance to the lecture hall, gender, age, etc.)

 

Lessons Learned

  • Let people know what you are doing
  • Data preparation:  fuzzy, messy
  • Choose the data
  • Simplify the data
  • Keep an eye on the prize

 

LectureTools:  Analytics

Females in class were much more likely to ask questions using a clicker (lovingly referred to as the “WTF” button).

90% of students at University of Michigan in the meteorology class cited felt they would have gotten the same grade if they had never opened the textbook

 

 

 

 

 

 

Categories
Technology

Creating a Data Governance Program

Title:  Creating a Data Governance Program

Presenter:  Mike Chapple, University of Notre Dame

 

This presentation was one of those EDUCAUSE decided to webcast.  Primarily focused on events of last year, but will cover some things done over the last 5 – 10 years.

It All Starts with a Story…

One day, the President was wondering…how many students do we have?

Naturally, a lot of potential answers depending on who you ask.

SLIDE:  how Notre Dame views data governance, using a building to illustrate

Access to Data (Roof)

  • Quality & Consistency (current focus)
  • Policies & Standards (current focus)
  • Security & Privacy
  • Compliance
  • Retention & Archiving

Technology (Foundation

 

Data Driven Decision Making (D3M) = Business Intelligence (as it’s known everywhere else)

  • Definitions need to be agreed upon (i.e. – what is a student)

 

SLIDE:  Governance Model

  • Executive Sponsors (EVP, CIO)
  • Campus Data Steward
  • Unit Data Stewards
  • Coordinating Committees (Info Governance Committee, D3M Steering Committee)

 

SLIDE:  Domain Objectives

  • Data Steward(s) appointment
  • Data definitions and business rules
  • Data quality guidelines and monitoring process
  • Regulatory compliance plan (if applicable)

 

SLIDE:  Building Data Dictionary

  • Term, i.e. “Active Student”
  • Definition:  PLAIN ENGLISH DEFINITION
  • Source System, i.e. Banner
  • Source Detail, i.e. SQL query which explains gory details of how you get the data

 

SLIDE:  Data Definition Components

  • Definition
  • Source System / Detail
  • Possible Values
  • Data Steward
  • Data Availability
  • Classification

 

SLIDE:  Start with Executive Support

This is pretty much an admonition; it really helps.  At Notre Dame, responsibility for this campus function landed with IT.

 

SLIDE:  Identify and Involve Stakeholders

Each item to be defined takes a meeting…it’s very time-consuming because you need to have representation from each area.  Data is owned by the university, not specific departments!

Notre Dame uses a “RACI” matrix for each defined term

R – responsible (office)

A – accountable (who keeps the group on-track)

C – consult (you have a seat at the table)

I – inform (people who need to know)

The matrix is distributed to all stakeholders so they can fill it in with their preferences.

 

SLIDE:  Reconcile Differences Visually

ND had two competing student numbers:  “Registrar Count” and “IR Count”

IR count = Externally reportable enrolled student

“Registrar Students” includes some folks like students on leave, zero credit students, etc.

Use a stacked bar, starting with externally reportable enrolled students, adding registrar student populations on top of that.

 

SLIDE:  Give the group a Starting Point

  • Start with a draft
  • Counting matters!  Definitions help address this possible problem.
  • Don’t use Jargon!

 

Security and Privacy

Risk-based security program

  1. Highly Sensitive (SSNs, CCs, Driver’s Licenses, Bank Accounts, ePHI)
  2. Sensitive (Everything else)
  3. Internal Information (info that would cause minimal damage is disclosed)

 

Compliance

We have to be responsive to new legal realities, since our campuses are like small cities and any law passed probably affects some area on our campus.

All data must be auditable.

  • 75% of orgs have at least one person dedicated to IT compliance
  • 76% of orgs have a corporate executive-level compliance office or council

Build compliance plans

  • Document everything with respect to regulations, i.e. HIPAA
  • Everything should be substantiated

 

Questions

With so many stakeholders, how did you address and resolve differences in data definitions?  We didn’t really have very many of those disagreements, because each area was represented in each set of meetings, and there was a solid bond among the reps from each area.

What do you do with data NOT in the data warehouse?  You just have to find some way to “chunk the work out.”  The output of the program must be pristine, so naturally priorities must be set.

Did ND work with IU, since most of this is the same?  No.

What tools are you using to manage metadata?  Google Docs for now, great for getting started, but it’s not conducive to long-term maintenance.  We’re actually building our own graph database.  This tool will ultimately expose this data for other tools.

Any principle for prioritization?   Steering committees prioritize based on BI needs of the institution.

Is there an ongoing need for a campus data steward versus a department data steward?  In some areas, the data is general or applies to many different populations.  Campus steward plays an important coordination role.

Do you consider your work the beginning of a master data management program?  Yes!

Do you see shadow systems as being a problem?  We’re not really far enough along to have experienced this problem yet.  Data is not widely available yet.  We refer to this phase “taking it from the team to the enterprise.”

This is for administrative data, yes?  Yes, it does NOT include research data.

Categories
Uncategorized

Big Data Enables Student Retention: Student Success & SAP HANA

Title:  Big Data Enables Student Retention:  Student Success & SAP HANA

Presenters

  • James David Hardison, DMD, MBA, Industry Principal, Higher Education
  • Vince Kellen, Ph.D., Senior Vice Provost Academic Plan Analytics and Technologies

 

SLIDE:  Screen shots of the “Powers of Ten” film

What if you could…(I smell a sales pitch coming)

  • Manage data, have the current data, have the right answers

 

SLIDE:  30 year old data modeling is slow and inefficient, in-memory is the new hotness!  (My words, not theirs)

HANA:  High Performance ANalytic Appliance

 

SLIDE:  HANA for Higher Education through University Alliances

  • SAP donates licenses to 1,300+ univesities
  • 1.2 million students educated on HANA
  • Certifications, etc.

 

SLIDE:  Newton’s Second Law of Motion

Students at risk are not likely to change their behavior without intervention.  Used snowball rolling downhill as an example…easy to stop at the top of the hill, but irresistible at the bottom.

 

SLIDE:  Using Fast Analytics to Help Improve Student Retention

High response rates to mobile micro surveys, about 40,000 responses in 4 weeks.  Questions include things like “how much are you working this term, how stressed are you (1-5 Likert scale), do you think you’ll be successful this term, etc.

In-memory analytics requires different architecting of the data modeling; We don’t do traditional ETL (Extract Transform Load techniques).

Took an approach of making data available, rather than keeping it “close to the vest.”

We can answer the question:  do we know how many left-handed Hungarian ping-pong players we retained?  How do we break our analysis into little pieces to answer practical problems and questions.

Also collecting “technographic” information on student device types and OS, frequency of use, number of devices, etc.

Been prototyping this with lecture capture and full-text search (what he referred to as “wayfinding”), i.e. ability to find specific terms, apply metadata, ability to link out to other materials, and so on.

Graph databases are generally not used by university degree audit reports, which precludes the use of “what if” scenarios.  This is useful for day-to-day business needs.  Incidentally, graph database technology is widely used in the Social Media space, but for some reason it seems to be all new and exciting to many of the wide-eyed higher ed attendees.  Hmm…

They’re using HANA data with Tableau to make pretty graphs.  Also, showed an Enrollment Plot of Chemistry 105

 

SLIDE:  Academic Health Notifications: View in Student Mobile App

 

SLIDE:  Using this data in a personalized student profile

 

 

SLIDE:  Taxonomy?  Automatic metadata?  Automatic atomic metadata?

  • Let learners navigate an a/v stream
  • Let the system learn what the top terms are.  Let the system map terms to concepts.  Let instructional designers lightly bump the taxonomy.

Thinking of Google AdWords-style presentation of information that’s relevant to the student’s click-stream and status.

 

SLIDE:  Organizational Considerations

  • Hired Ph.D. level data scientists (there was some turnover)
  • Translated old architectures to HANA and retired old IR data warehouse
  • Opened data, many have access, personal data is protected
  • Raised skill sets in colleges and units and provide support.

 

SLIDE:  Some best practices

  • Be safe and secure
  • Be collegial
  • Help improve data quality
  • Be open-minded and inquisitive
  • Share (don’t be a taker)

 

QUESTIONS

Do you plan to include co-curricular data into your system?  Obvious items are participation in clubs and organizations, and surfacing internships and job opportunities.  Yes, it’s on the roadmap.

Are you collecting data on effectiveness of tutoring?  Yes.

 

Categories
Technology

Ethics and Analytics: Limits of Knowledge and a Horizon of Opportunity

Title:  Ethics and Analytics –  Limits of Knowledge and a Horizon of Opportunity

Presenters:

  • James E. Willis III, Ph.D., Purdue University
  • Matthew D. Pistilli, Ph.D., Purdue University

 

(See my related post, “The Edward Snowden Affair: A Teachable Moment for Student Affairs and Higher Ed“)

 

Highly interactive session, sat in groups of five and discussed among ourselves.

SLIDE:  Lack of Discussion of Ethics in Data and Technological Innovation

Why?

  • Cutting-edge technologies are being developed every dy
  • Cost/profit margins are determined with speed of development
  • Typeicall education lines are split between science adn humanities/technolgoy and ethics
  • Difference between what can be done and what should be done

 

SLIDE:  Where to Go from Here?

  • Begin the discussion now
  • Have the difficult converations
  • Bring together the stakeholders:  data scientists, engineers, managers, ethicists
  • Establish a framework to adapt quickly to questions of whether or not an action or research agenda should occur

 

SLIDE:  Ethical Discussions = Innovation

Logic of negation

  • Why shouldn’t we do this?
  • What should we do instead?

Different paths emerge from divergent conversations

  • Stakeholders have different voices and understandings of potential paths of development

 

SLIDE:  Potter’s Box

Definition

  • ID values
  • Appealing to ethical principles
  • Choosing Loyalties
  • Responsibilities and recommendations

How Potter’s Box is useful for ongoing discussions

If you know a student is at risk, what do you do?  What is your responsibility?

 

SLIDE:  Research Questions

  • What do the ethical models look like?
  • How are these models deployed rapidly – at the speed of technology?
  • How are these models refined with time?

 

SLIDE:  Group Discussion 1

What are the current projects going on in learning analytics today?  What are the potential ethical pitfalls that surround these developments?  Why are they potentially harmful?  Are these things always wrong or are they contextually wrong?

Some of the ideas generated by the room:

  • Type 1 / Type 2 error:  will the student pass this class?  What’s my prediction – will the student pass or fail the class?  How accurate is your prediction – did your model work?
  • Is it right to get information from the students?  Where does the locus of control lie?  Does the institution need to take more responsibility for this?
  • University One (FYE equivalent in Canada) – informal collection and sharing with the next year.  Are we branding / labeling students appropriately?  Are we sharing this information appropriately?  Who should know what and at what point?  Will that affect their future studies?
  • Top-down approach using a product with funding from the Gates Foundation (similar to the InBloom project).  Goal is to make a product that analyzes what students should be taking.  Pitfalls:  don’t know model, don’t know the raw data.

 

SLIDE:  Group Discussion 2

What is the role of “knowing” a predictive analytics – once something is known, what are the ethical ramifications of action or inaction?  What are the roles of student autonomy, information confidentiality, and predictive modeling in terms of ethical development of new systems / software / analytics?

  • How much do we really know?  If we have a high level of confidence, what is our responsibility?
  • Could it be solved by giving students opt-in versus opt-out?
  • Discovering things, going down certain paths that may raise ethical questions about students who may not succeed…i.e. financial burdens that may be incurred due to failure.

 

SLIDE:  Group Discussion 3

How might we affect the development of future analytics systems by having ethical discussions?  What are possible inventions that could come from these discussions?

  • See more active participation from the students, with a better understanding of how their data is going to be used.
  • Obligation to allow students to fail; channeling people into things they can be successful with.
  • EU regs:  disclosure, portability, forget
  • Portability:  health care analog; your info can go from one hospital to another, and that data has real-world benefits to patients.
  • Information can also say a lot about faculty too.

 

SLIDE:  Group Discussion 4

What are the frontiers of ethical discussions and technology?  Are there new frameworks?  How can we affect the future by today’s discussions?

  • Maybe you can redefine what student success is?  Maybe it isn’t graduation rates…
  • How do we affect the future?  It’s all of our roles to be a part of this conversation!  You don’t really have the option to say “I don’t have the time for this”

 

 

Categories
Student Affairs Technology

EDUCAUSE Student Affairs Constituent Group Meeting

Title:  Student Affairs Constituent Group

Facilitator:  David S. Sweeney, Director for Information Technology in Student Affairs at Texas A&M

 

David got us started by talking about some of the tasks this group wants to take on in 2013 – 2014

First up:  compiling a list of Student Affairs-related commercial software tools.  David was going to send out a survey asking for this, but hasn’t gotten around to doing this just yet.  He found out that EDUCAUSE has a database tool that does this.  Paul said he would assist David with this.  Perhaps this list / data repository can become a good resource for all of us?

Next:  what kind of IT-related things have surfaced this year within SAIT departments and/or Student Affairs?  David created a list of these things, and talked briefly about the EDUCAUSE “Horizon Report” – two of the items on David’s list are on the EDUCAUSE list.  Readers:  go look at the EDUCAUSE Horizon Report.  Don’t worry, I’ll wait for you to do that 🙂

…and the list, please…

  1. SoMe, a.k.a Social Media
  2. Explosion of tablets / BYOD (i.e. how do we support them?  How do we manage secure information on those devices?)  Do we go with permissive or restrictive models?
  3. Emergency reporting and the “dear colleague” letter (Title 9 sexual assault and reporting).  What mechanisms do we have for reporting these and OCR reports?
  4. Student Analytics, i.e Purdue’s “Signals” product.  How do we take that data and incorporate it into other datasets.  How do we count participation in particular events?  Texas A&M is using swipe cards for events to build up a co-curricular portfolio.
  5. Student Learning Outcomes and Co-Curricular Portfolio

 

In no particular order, we talked about the following stuff…my apologies to attendees for not capturing everyone’s thoughts!

 

There was some discussion about who IT reports to within Student Affairs.  Oddly, I’m the only one who reports directly to the VP of Student Affairs.  We also talked about centralized versus decentralized IT services.

 

It’s official:  student analytics is the hot topic of the year!

How many of us actively collect assessment data?  About five out of the 18 attendees are.  Assessment data collection by Student Affairs on many of our campuses is quite rigorous.  On some campuses, this effort was initially looked upon as a burden.  Many use the CampusLabs product (StudentVoice, Baseline, etc.).  Student Affairs clearly is clearly a leader in this space, but our ability to demonstrate the value of our data is spotty, and integration of this data appears to be a long way off in most cases.  I recently blogged about exactly this topic in this post:  “Should Co-Curricular Activities Contribute to Academic Early Warning Systems?”

University of Montana talked about being at “level two” of this process.  David asked a question:  are you running demographics analysis against this information?  Answer:  recently, yes.

University of Toronto pulls their assessment data into Oracle and use it to populate a co-curricular transcript (this sounds very cool).

There appears to be a general problem most of us face:  getting people to create / manage a data warehouse and do the data extraction!  Some campuses have invested in Crystal Reports and have built student data warehouses.  Where does the responsibility lie for doing this?  Should we educate others in our division so they can do it themselves?  One answer provided (sorry, can’t remember your name!) said that creating a department that professionally studies and manages assessment data has worked for them.  This is not a department of one, either, which was very encouraging to hear.

Texas A&M has a new (about 12 months old) student success department that analyzes data.

FERPA was mentioned by some in attendance as a perceived barrier for use of student data for analytics and reporting purposes.  This perception seems to vary from campus to campus.  FERPA is in fact NOT a barrier when it’s used by people who have a day-to-day business need for it.  Some campuses have data ownership “issues” that need to be overcome.

Paul talked briefly about how he met with LinkedIn CEO at Web2.0 conference about four years ago to talk about developing a service to generate a standardized format for co-curricular transcripts that could be easily imported into a student’s professional profile.  There was interest in this, but there was agreement that there isn’t one well-defined standard.  Maybe our group can think about what kinds of information should be in that kind of profile?  A number of great suggestions were made about taking advantage of Career Services’ expertise with what employers are currently looking for.

 

This group could undoubtedly have gone for a couple hours, but it was lunchtime so we reluctantly had to break.  There was consensus among the group to contribute to our constituent list serve, which can be found here:  http://www.educause.edu/discuss/information-systems-and-services/student-affairs-it-constituent-grou  Let’s get cracking, SAPros!

%d bloggers like this: