Tag: EDUCAUSE

Categories
Technology Uncategorized

How to Use the EDUCAUSE CDS to Support Student Success

Presenters

  • Susan Grajek, Vice President, Data, Research, and Analytics, EDUCAUSE
  • Laurie Heacock, National Director of Data, Technology and Analytics, Achieving the Dream, Inc.
  • Louis Kompare, Director, Information Systems and Services, Lorain County Community College
  • Celeste M. Schwartz, VP for IT & IR, Montgomery County Community College

Susan kicked off this session by describing what the CDS is.  It’s been around for over 10 years, includes data from over 800 institutions and allows members to use it to:

  • Study their IT org
  • To benchmark against past performance
  • To look at trends over time
  • To start gathering and using metrics
  • To have data available “just in case”

TOP IT ISSUE #4

Improve Student Outcomes Through an Institutional Approach that Strategically Leverages Technology. Data shared today come from module 3 of the CDS

Student Success Technologies Maturity Index

These 6 measurements are set by subject matter experts, and are measured against a 5 point radar scale

  1. Leadership and governance
  2. Collaboration and involvement
  3. Advising and student support
  4. Process and policy
  5. Information systems
  6. Student Success analytics

Maturity Index

  1. Weak
  2. Emerging
  3. Developing
  4. Strong
  5. Excellent

Deployment Index

  1. No deployment
  2. Expected deployment
  3. Initial deployment
  4. targeted deployment
  5. institution-wide deployment

Goal

Provide higher ed institutions with a reliable, affordable, and useful set of tools to benchmark and improve the cost and quality of IT services, improving the value and efficiency of IT’s contribution to higher education.

Process

Complete Core Data > order and configure reports > receive and use reports.  It takes between 40 and 70 hours to complete, but data is saved for auto-filling the following year.  This speeds the re-entry process considerably.

You can also use the reports for benchmarking against other institutions.  You can create your own, and some peer groups are pre-provided for you.

Achieving the Dream’s Institutional Capacity Framework

Montgomery County Community College (near Philadelphia), about 13,000 students, participating in CDS for about 13 years.  Celeste then went on…In the past, we used CDS more on the justification of new staff.  We used to look at numbers of computers for students, but we tend to look at those numbers less today.  What’s really helped us recently are in how we ask questions about technology.  While you only HAVE to complete module 1, I recommend you dip your toes in some of the other modules.  I’ve used SurveyMonkey to extend my reach and gather additional information from other folks, and then moved it into CDS.  The CDS is really helping to drive our own IT strategic plan.

Lorain Community College (near Cleveland), about 12,000 students, participating in CDS for 2 years.  Our enrollment is highly tied to local industry; local business cycles make make our completion rates look terrible!  CDS is the most valuable way I have to find out the various elements of IT in the higher ed world.  It really helps to discover the things that change from year-to-year.

Top 10 IT Issues Sneak Peek

Coming out in January in EDUCAUSE Review.  IT security is the #1 issue.  Three dimensions that will be discussed in the upcoming report:

  • Divest:  change the way you design, deliver and manage IT services.  Eliminate old processes and silos!
  • Reinvest:  to run state-of-the-art technology services, you need to double down on some things, like information security.  Hiring and retaining good talent, along with restructuring that talent to meet the changing needs of delivering IT services.  The ability to change funding models to meet those needs is also important.
  • Differentiate: institutions are now able to apply technology to strategically meet their goals and differentiate themselves from other institutions.  Ability to apply analytics against strategic objectives is hugely valuable to help provide feedback on where we are and what we need to do to improve.
Categories
Technology

Unifying Data Systems to Turn Insights into Student Success Interventions

Presenters:

  • Angela Baldasare, Asst.Provost, Institutional Research, The University of Arizona
  • Phil Ice, Vice President, Research & Development, American Public University System
  • Matthew Milliron, Senior Director, Solutions Engineering, Civitas Learning Inc.

Hypothesis

Unifying the data; connect disparate data systems, data and initiatives to gain insight into what’s working & what’s not & for whom

The area that’s of most interest to Phil is the LMS, because that’s where we have the most interaction.  Unfortunately, it’s mostly log file information.  Scroll and click information is not captured.  LTI integration does not help much because it’s based on an iFrame and we lose context.  Instead, they’re using Adobe Analytics (formerly Omniture).  We’re also using social sharing.

Institution-Specific Platform for Innovation

Unified Data Layer (Student Data Footprint – historic and incoming disparate systems) is connected to:

  • Institution-Specific Deep Predictive Flow Models
  • Frontline Apps & Initiatives
  • Robust Testing and Measurement

Matthew then talked about the Civitas Learning Platform components (not exactly a sales pitch, but not too far off).

Prediction

  • UA Historical Overall Fall to Fall Retention Rate = 87%
  • FTIC FT Freshman Historical Average First Year REtention Rate = 80%
  • Prediction for Fall 2015 Cohort = 80%
  • n=6,970 students

Data set used for modeling:

  • Train:  Fall 2012 to Fall 2013
  • Test:  Fall 2013 to Fall 2014

Model accuracy

  • AUC .844
  • 90% accuracy

Discoveries

For FTFT Freshman in their First Term, students with SAT Math >550 persist at a rate +1- percentage points higher than SAT Math <550

For FTFT Freshman in their First Term, an LMS Course Grade on day 14 that is lower than 75% is associated with lower persistence than students with grades over 75%.

For transfer students overall, following course pathways traveled by students who graduated is beneficial for persistence.

For FTIC students who deviate significantly from the course pathway, the effect can be very bad.

Angela mentioned how useful the toolset is for the ability to see a list of students that make up any active filter segment within the too, and dig deeper on the their activity to extract additional actionable insights.

Categories
Technology

Predictive Learning Analytics: Fueling Actionable Intelligence

Presenters:

  • Josh Baron, Assistant Vice President, Information Technology for Digital Education, Marist College
  • Shady Shehata, Principal Data Scientist, D2L
  • John Whitmer, Director for Analytics and Research, Blackboard Inc.

This is all about an ECAR report that was released a few weeks ago.  Report was compiled over about a year and had a large number of contributors.  It’s a great example of collaboration across organizations for profit and non-profit, scientific, etc.

39% is the Key Number

  • In the US, only this percentage completes a 4-year program.
  • This is a national challenge, because the US has slipped from #1 or #2 completion in the world to #12
  • How Can Predictive Learning Analytics Help?  It provides the ability to predict the future with a reasonable level of accuracy give you the ability to intervene on behalf of the student

What is Predictive Learning Analytics?

  • The statistical analysis of historical and current data derived from the learning process to create models that allow for predictions that improve learning outcomes
  • Subset of larger learning analytics field
  • Uses sophisticated mathematical models rather than user-defined rules.  Example:  Academic Early Alert Systems
  • OAAI:  the Open Academic Analytics Initiative
  • Apereo Learning Analytics Initiative

Data Sources, Relevance & Diversity

  • LMS:  academic technology’s first killer app
  • What’s been successful is the penetration and usage of LMS
  • What data from conventional data sources are systematic, significant predictors of course success?  High school GPA?  Race/Ethnicity?  First in Family to Attend College?  NONE OF THEM!
  • Academic Technology data is a systematic predictor of course success – caveat is that the academic material is connected in a deep and meaningful way.  Having a number of triggers help to track actionable details.
  • Embedding Predictive Analytics
  • Strategic Importance other Data Points:  underrepresented student groups.
  • Conclusion:  Learning Data Comes in Many Flavors and Relevance, i.e. Activity (behavioral) data and Static Data (survey data, student aptitude, extra-curricular activities, demographics and prior educational experience).
  • There are very few institutions that employe full-time Predictive Learning Analytics professionals.

How it Works, What is the Data Impact?

  • Historical data and predictive analytics are used to generate a predictive model
  • We want to tease out and surface those patterns that result in successful outcomes
  • After first month, predictive model can provide predictions of the final grades based on the # of content views of the students in the current course offering
  • Examples of what can go wrong:  what if students are viewing the content from mobile application (data incomplete); what if one of the historical course has hundreds of course topics, where other courses have tens of course topics?  (data is inaccurate)
  • Garbage In, Garbage Out
  • Data quality:  accurate, complete, unique, timely, consistent, valid, reliable, integrity

Brightspace Student Success System

  • Created a predictive model for a course
  • A number of screen shots showed how it was implemented with a group of students, with drilldowns on where students were having difficulty.
  • Good visualizations are critical to easily decoding information and making it useful

Data Strategy

  • ETL/Data Integration > Data Warehouse & MDM > BI & Data Visualization > Predictive Analytics

Strategic Implementation Considerations

  • Institutional Stages of analytics usage
  • Organizational leadership, culture & skills
  • gaining access to learning data
  • Ethics & privacy

Institutional Stages of Analytics Usage

  • Basic:  past trends & data observations
  • Automated:  automatically perform analytics & provide results directly to end-users
  • Predictive:  large amounts of data is crunched

Silos are antithetical to successful implementation; investing in new skill sets is imperative.

Gaining Access to Learning Data

  • Activity, clickstream data
  • It’s the fuel on which LA runs
  • Extracting sample data sets is often a good start

Ethics & Privacy

  • Ethics:  using LA for good and not evil
  • Privacy:  balance the need to protect confidentail records while maximizing the benefits of LA
  • Often requires new policies and procedures
  • LA “task force” to address ethics and privacy issues
  • JISC code of practice
  • SURF Learning Analytics SIG

Q&A

  • How do you best approach the introduction of PA to a group of people who don’t even know what it does?  EDUCAUSE has some great white papers on this (“Penetrating the Fog”).  Give examples of products, strategies and solutions.
  • Have you done anything to look at the performance of blended/flipped classrooms?  We get asked this a lot!  We’ve looked at some of the open course offerings of MiT, Blackboard usage patterns.  Many folks are interested in using it for academic course design.
Categories
Technology

Optimizing Business Intelligence at Lehman College/CUNY: A Road to Change

Presenters:

  • Ronald Bergmann, VP-CIO, Lehman College/CUNY
  • Richard Finger, Director, Graduate Studies, Lehman College/CUNY
  • Lei Millman, Oracle DBA, OBIEE Admin, Lehman College/CUNY

Ron Bergmann introduced himself and touted the Frye leadership program, encouraging involvement in the program.

Lehman College

  • A CUNY school, located n the Bronx
  • About 12,000 students, with 90 graduate and undergraduate programs

BI Solutions in Higher Education

  • Big data has changed the way higher education makes decisions  takes action
  • Dynamic, easy-to-use tools have given higher education leaders more power for decision making than ever.
  • What is your road map

Questions

  • What are your key BI needs and goal?
  • Where does data reside?   How is data shared, aggregated and analyzed?
  • What tools are you using/what’s the best fit?
  • What data is important to a key customer?
  • How would you describer your data “culture?”
  • What factors support/resist changes in the use of data?

Lehman College Dashboard (LCD) – BEFORE

  • Data unconnected
  • Users did not have a reporting tool
  • Devs had limited options
  • Report development took too much time

Lehman College Dashboard (LCD) – NOW

  • BI solutions using OBIEE
  • A common ecosystem for producing/delivering enterprise reports:  enrollment, graduation, faculty workload, etc.
  • Easy access to LCD in a comprehensive format
  • Actionable data drives more informed decisions

How Did We Get Here?

  • Culture Change / Buy-in
  • Data Governance
  • Stakeholder expertise and input
  • Access to multiple existing data sources and reports (enrollments, budget/expenditures, admissions, analysis, dashboards, data sharing with other CUNY schools, etc.)

User Experience at Lehman: Preparing end-users for the BI Tool

Adoption of BI has led to big changes in culture and expectations.  It has also led to significant changes in campus processes.  This section of the presentation included many screen shots of the reports that they run.  While the reports look pretty simple, they’re really helping us with enrollment management and understanding the effectiveness of our interventions.

  • Understand the difference between official and unofficial data
  • Develop a “data dictionary” to avoid confusion
  • Articulate report parameters
  • Understand that report writing is a collaborative effort
  • Be prepared to test and modify reports in draft form

LCD Example:  One Dashboard, Multiple Data Integrations

LCD provides actionable data.  The information LCD provided allowed us to hit our enrollment targets this past fall semester.

  • Current Semester Enrollment
  • Student Retention
  • Cohort Overall Analysis
  • Student Financial Aid Information
  • Student List by Advisors, Individual Student Detailed Academic Information Dashboard

Questions / Comments

  • Can you tell us about the data governance group?  We have it, but not every office is represented.  If I had a choice, I’d get everyone on board earlier into a formal governance group.
  • Can you talk about your data dictionary?  How big is it, how do you share it?  We created 2 document, one for users and the other for IT staff.  We share the users document with the campus.
  • Do you track interventions in the system?  Not yet, but we do use it to manage our communications outreach and advisor appointments.
Categories
Technology

The Second Machine Age: Work, Progress, and Prosperity in a Time of Brilliant Technologies

Presenter:

Andrew McAfee, Author, MIT, @amcafee

Erik Brynjolfsson and I wrote our book because we were confused about technology.  It’s doing things now that it’s not supposed to do…it’s affecting the real world in ways we don’t really understand.

State of Understanding a Decade Ago

  • Book:  The New Division of Labor.
  • Dealt with the question: “what are humans good at, and what are computers good at?”
  • Give all the rote work to computers, and leave the pattern-matching and complex communication to humans.  Example:  driving a car in traffic.
  • Andrew then related his experience of riding in a Google self-driving car through 3 phases of personal experience:  raw abject terror (first 10 minutes);  passionate interest in what was going on (next 20 minutes); mild boredom (rest of the ride).  My own thought at this point:  “I’ve seen the future, and it’s really boring”
  • Andrew then went through the example of IBM’s Watson computer participating in the game show Jeopardy!  Watson versus people in 2006 was terrible.  Watson today is now as good as – or better than – the best human champions.  Andrew included a photo of Ken Jenning’s funny parenthetical comment on his last question against Watson:  “I for one welcome our computer overlords.”  Indeed!

Minds and Machines

  • We need to rethink this combination…machine abilities are growing to match those of humans.
  • How did this happen?  Andrew alluded to Hemingway’s quote (regarding going broke):  first it happened gradually, then it happened suddenly.
  • A rough calculation of the “tipping point,” using Kurzweil’s first half/second half of the chessboard square-doubling analogy; 1958 was the first year the BEA measured computing power doubling every 1.5 years.  Thus, 1958 + 32*1.5 = 2006

 A Change in Approach

  • Rules-based approach is inferior and doesn’t work very well (i.e. learning a language as an adult using verb conjugation books).  There are too many rules to learn!
  • Kids learn language through listening and absorbing inductively what’s going on.  Humans are pre-wired for language.
  • “We know more than we can tell” – Michael Polanyi
  • The game of Go is way more complex than chess, and to date computers have not been able to beat the best human players.  However, this will likely change before the end of this year.  How?  Because we’re going to give computers a goal of maximizing the score in a game, via trial and error.  An example of this was shown with the game Breakout

Self Assessment

  • Let’s do a self assessment.  Compared to the people around me, I’m “” (score yourself on a scale of 1-100)
  • I have good intuition; I make good predictions; I’m a good judge of character
  • Now average the 3 values
  • We’re bad at self-judgement and are predictably irrational

Geeks Versus HIPPOS

  • Geeks (who are evidence and data-driven) versus HIPPO (HIghest Paid Person’s Opinion)
  • Robert Parker is the HIPPO of the wine world
  • Orley Ashenfelter (a wine geek) came up with a remarkably accurate algorithm that made wine HIPPOS largely irrelevant

What Do Humans Still Bring to the Table?

  • We have advanced social skills
  • We have good intuition
  • We have creativity
%d bloggers like this: