Tag: data

Categories
Technology Uncategorized

How to Use the EDUCAUSE CDS to Support Student Success

Presenters

  • Susan Grajek, Vice President, Data, Research, and Analytics, EDUCAUSE
  • Laurie Heacock, National Director of Data, Technology and Analytics, Achieving the Dream, Inc.
  • Louis Kompare, Director, Information Systems and Services, Lorain County Community College
  • Celeste M. Schwartz, VP for IT & IR, Montgomery County Community College

Susan kicked off this session by describing what the CDS is.  It’s been around for over 10 years, includes data from over 800 institutions and allows members to use it to:

  • Study their IT org
  • To benchmark against past performance
  • To look at trends over time
  • To start gathering and using metrics
  • To have data available “just in case”

TOP IT ISSUE #4

Improve Student Outcomes Through an Institutional Approach that Strategically Leverages Technology. Data shared today come from module 3 of the CDS

Student Success Technologies Maturity Index

These 6 measurements are set by subject matter experts, and are measured against a 5 point radar scale

  1. Leadership and governance
  2. Collaboration and involvement
  3. Advising and student support
  4. Process and policy
  5. Information systems
  6. Student Success analytics

Maturity Index

  1. Weak
  2. Emerging
  3. Developing
  4. Strong
  5. Excellent

Deployment Index

  1. No deployment
  2. Expected deployment
  3. Initial deployment
  4. targeted deployment
  5. institution-wide deployment

Goal

Provide higher ed institutions with a reliable, affordable, and useful set of tools to benchmark and improve the cost and quality of IT services, improving the value and efficiency of IT’s contribution to higher education.

Process

Complete Core Data > order and configure reports > receive and use reports.  It takes between 40 and 70 hours to complete, but data is saved for auto-filling the following year.  This speeds the re-entry process considerably.

You can also use the reports for benchmarking against other institutions.  You can create your own, and some peer groups are pre-provided for you.

Achieving the Dream’s Institutional Capacity Framework

Montgomery County Community College (near Philadelphia), about 13,000 students, participating in CDS for about 13 years.  Celeste then went on…In the past, we used CDS more on the justification of new staff.  We used to look at numbers of computers for students, but we tend to look at those numbers less today.  What’s really helped us recently are in how we ask questions about technology.  While you only HAVE to complete module 1, I recommend you dip your toes in some of the other modules.  I’ve used SurveyMonkey to extend my reach and gather additional information from other folks, and then moved it into CDS.  The CDS is really helping to drive our own IT strategic plan.

Lorain Community College (near Cleveland), about 12,000 students, participating in CDS for 2 years.  Our enrollment is highly tied to local industry; local business cycles make make our completion rates look terrible!  CDS is the most valuable way I have to find out the various elements of IT in the higher ed world.  It really helps to discover the things that change from year-to-year.

Top 10 IT Issues Sneak Peek

Coming out in January in EDUCAUSE Review.  IT security is the #1 issue.  Three dimensions that will be discussed in the upcoming report:

  • Divest:  change the way you design, deliver and manage IT services.  Eliminate old processes and silos!
  • Reinvest:  to run state-of-the-art technology services, you need to double down on some things, like information security.  Hiring and retaining good talent, along with restructuring that talent to meet the changing needs of delivering IT services.  The ability to change funding models to meet those needs is also important.
  • Differentiate: institutions are now able to apply technology to strategically meet their goals and differentiate themselves from other institutions.  Ability to apply analytics against strategic objectives is hugely valuable to help provide feedback on where we are and what we need to do to improve.
Categories
Technology

Unifying Data Systems to Turn Insights into Student Success Interventions

Presenters:

  • Angela Baldasare, Asst.Provost, Institutional Research, The University of Arizona
  • Phil Ice, Vice President, Research & Development, American Public University System
  • Matthew Milliron, Senior Director, Solutions Engineering, Civitas Learning Inc.

Hypothesis

Unifying the data; connect disparate data systems, data and initiatives to gain insight into what’s working & what’s not & for whom

The area that’s of most interest to Phil is the LMS, because that’s where we have the most interaction.  Unfortunately, it’s mostly log file information.  Scroll and click information is not captured.  LTI integration does not help much because it’s based on an iFrame and we lose context.  Instead, they’re using Adobe Analytics (formerly Omniture).  We’re also using social sharing.

Institution-Specific Platform for Innovation

Unified Data Layer (Student Data Footprint – historic and incoming disparate systems) is connected to:

  • Institution-Specific Deep Predictive Flow Models
  • Frontline Apps & Initiatives
  • Robust Testing and Measurement

Matthew then talked about the Civitas Learning Platform components (not exactly a sales pitch, but not too far off).

Prediction

  • UA Historical Overall Fall to Fall Retention Rate = 87%
  • FTIC FT Freshman Historical Average First Year REtention Rate = 80%
  • Prediction for Fall 2015 Cohort = 80%
  • n=6,970 students

Data set used for modeling:

  • Train:  Fall 2012 to Fall 2013
  • Test:  Fall 2013 to Fall 2014

Model accuracy

  • AUC .844
  • 90% accuracy

Discoveries

For FTFT Freshman in their First Term, students with SAT Math >550 persist at a rate +1- percentage points higher than SAT Math <550

For FTFT Freshman in their First Term, an LMS Course Grade on day 14 that is lower than 75% is associated with lower persistence than students with grades over 75%.

For transfer students overall, following course pathways traveled by students who graduated is beneficial for persistence.

For FTIC students who deviate significantly from the course pathway, the effect can be very bad.

Angela mentioned how useful the toolset is for the ability to see a list of students that make up any active filter segment within the too, and dig deeper on the their activity to extract additional actionable insights.

Categories
Technology

The 2015 Campus Computing Survey

Presenters:

Resources:

This is the 26th year of the National Survey of Computing, eLearning, and Information Technology in US Higher Education.  It’s the largest survey of it’s kind in the US.  This is a survey that I’m aware of, but I don’t think I’ve ever actually read (although I might have attended this session last year, I can’t remember).  I figure this is the year that I change that.

Intent of the project has been to provide insight for IT planning and policy.  There are 35 corporate sponsors of this project – none of which have ever seen individual campus stats.  Here are some top-level details about the survey’s data collection:

  • 417 campuses
  • Web-based data collection
  • Survey period:  9/17 – 10/21
  • 75% of participants also participated last year

Highlights

  • Priorities of focus on instruction, staffing, user support, advancing campus completion agenda, IT security
  • Big diff in CIO assessments of the things we do/provide vs. the things we buy
  • Great faith in adaptive learning & digital curricular resources
  • Transition to cloud

Challenges

  • Talent retention
  • Digital curricular resources make learning more efficient & effective for students
  • 3rd party cloud services are an important part of campus plan to offer high performance computing services

Top Priorities

  1. Assist faculty integrate tech into instruction
  2. Hiring / retaining qualified IT staff
  3. User support
  4. Upgrading / enhancing network security
  5. Leveraging IT resources for student success

Some High-Level Details

  • Among the range of priorities that we all have, there are lots of service items, and not nearly as many related to the things we buy.
  • CIOs Have Great Faith in the Benefits of Digital Tech for Instruction.
  • Rating the IT Infrastructure:  lowest rankings are services, highest are hardware.
  • CIO Assessments of Digital Resources and Services for Disabled Users:  only 50% have a strategy for ADA/Section 503 compliance.  This is litigation waiting to happen.
  • Mobile technologies over laptops!
  • CIOs rate the effectiveness of campus investments in IT.  Most scores are rather low.
  • Challenge of Effective IT User Support:  we think we’re doing better than our users think we are.
  • Budget cuts are still pervasive and affect us deeply.  Cuts versus gains across investments are interesting (refer to the report).
  • Disaster Plans:  most campuses have plans and even update them regularly.  22% DO NOT have a strategic plan for network and data security (this is an amazing stat to me).
  • Declining Confidence in MOOCs.  Completion rates are atrocious (although enrollment is voluntary).  Infrastructure could be a problem here.
  • We’re experiencing major cost over-runs / unexpected costs in our ERP deployment activities.
  • Two Views of the Cloud:  things may happen faster than we expect, but less than 25% think we’ll have mission critical systems in 5 years.  IT pros affirm the strategic importance of cloud computing.  There’s still significant concern over the security of the cloud.  Migration to the cloud is slow due to perceived risk, trust, control, limited options.  Interestingly, LMS has largely moved to the cloud. No mass movement to the cloud in 5 years.
  • Growing use of video lecture
  • Encouraging Faculty to Use Open Source / OER Content for Courses
  • Institutional demography of LMS providers:  decline in Blackboard, Canvas growing fast.  Market is volatile.  The LMS largely does not affect learning outcomes, but is used as a material delivery service.
  • Mobile apps are huge and an expected service.

Wonderful quote by Casey on his experience:  “In my 25 years of doing this survey, IT appears to be driven by epiphany and opinion, not evidence.”

Vendors: What You Need to Know

  • Partner is Not a Verb
  • Trust is the coin of the realm
  • No “logo buddies”
  • You are not your client
  • Your price is not your client’s cost
  • It’s a neural network
Categories
Technology Uncategorized

Opening Up Learning Analytics: Addressing a Strategic Imperative

Presenters:

  • Josh Baron, Assistant Vice President, Information Technology for Digital Education, Marist College
  • Lou Harrison, Director of Educational Technology Services, NC State University
  • Donna Petherbridge, Associate Vice Provost, DELTA, NC State University
  • Kenny Wilson, Division Chair-Health Occupation Programs, Jefferson College

This is actually a follow-up to one of my recent posts about a webinar I attended by Unicon on learning analytics.  We have representatives from three different LMSes:  Moodle, Sakai, and Blackboard.  Looks like Lou and Josh from that webinar are here…I’m looking forward to learning more about this effort!  Word of warning:  they moved fast, so I missed some detail, particularly around the workflow and data-heavy slides.  My Student Affairs colleagues will want to tune into the question I asked at the end…

Open Learning Analytics:  Context & Background

OAI, or the Open Academic Analytics Initiative:  EDUCAUSE Next Generation learning Challenges (NGLC).  Funded by Bill & Melinda Gates foundations, $250,000 over a 15 month period.  Goal:  leverage big data concepts to create an open-source academic early alert system and research “scaling factors”

LMS & SIS data is fed into a predictive scoring model, which is then fed into an academic alert report.  From there, an intervention is deployed (“awareness” or Online Academic Support Environment – OASE)

Research design:  rolled out to 2,200 students in 4 institutions:  2 community colleges, and 2 historically black colleges and universities.  More detail on the approach and results here.

Strategic Lessons Learned

Openness will play a critical role in the future of learning analytics.

  • Used all open source tools:  Weka, Kettle, Pentaho, R, Python, etc.
  • Open standards and APIs:  Experience API (xAPI), IMS Caliper/Sensor API
  • Open Models:  predictive models, knowledge maps, PMML, etc.
  • Open Content/Access:  journals, whitepapers, policy documents
  • Openness or Transparency with regard to ethics/privacy
  • NOT anti-commercial, commercial ecosystems help sustain OSS

Software silos limit usefulness

  • Platform approach makes everything more useful

NC State Project

  • Getting everyone moving in the same direction is a challenge.
  • The number one priority we have at NC is student success, and we know that data is going to help us get there.  However, we have different vendors approaching us independently, each with their own selling points on what they could do to help us.
  • Lunch and learn sessions, bring people up to speed on what questions to ask, and start thinking about who can generate answers.  It took us 10 months to get everyone together
  • Division of Academic & Student Affairs has purchased EAB; concurrently, we’re working on LAP.  Continued conversations with campus partners will have to happen.

From Proof to Production:  Toward Learning Analytics for the Enterprise

  • Initial steps:  small sample sizes, predictions at 1/4, 1/2, 3/4 points in course, multi-step manual process
  • Goal 1: make it more enterprise-y.  Use large sample sizes (all student enrollments), frequent early runs (maybe daily), automatic no more than 1 click
  • Currently in progress:  rebuild infrastructure for scale; daily snapshots of fall semester data; after fall semester ends look for the sweet spot.
  • Future goals:  refine model even more; segment model by population; balance between models and accuracy; refine and improve models over time; explore ways to track efficacy over time; once we intervene we can never go back to virgin state

Jefferson Project

  • Why is JC seeking LAP implementation?  First time pass rate of Anatomy and Physiology is 54%.  Only 27% re-take.  37% non-persistence rate (DFW).  Need to find ways to help students succeed.
  • How is it going?  We have a 4 year grant.  Compliance letter came in May of 2015.  Implement PREP program in October 2015, LAP roll-out in 10/1/2016, with one year to test.  We use Student Participation System data and feed it into the system.
  • Why use SPS data?  It’s readily available; part of HLC Quality Initiative; less politically charged; shown to correlate with student success; clear map of data schema; data is very robust, more data there than we are presently using; data is “complete” (better than Bb data; less complete than original LAP design).
  • Each instructor will receive an Academic Alert Report.

My question:  have you considered integration of co-curricular data into your models?  YES!  We’re very interested in integration of co-curricular data, because it’s often a better indicator for student success than LMS data.  Vincent Tinto’s research clearly indicates this, but our implementation of this is probably a phase 3 or phase 4 thing.

Categories
Technology Uncategorized

The Science of Predictive Analytics in Education

Presenters:

  • Patrick J. Bauer, Chief Information Officer, Harper College
  • Scott Feeny, Director of Policy and Research, Independent Colleges of Indiana
  • Vince Kellen, Senior Vice Provost for Analytics & Technologies, University of Kentucky
  • Jon Phillips, Managing Director – Worldwide Education Strategy, Dell Inc.
  • John K. Thompson, GM, Advanced Analytics, Dell Inc.

This session will focus on innovations in using data insights in decision-making.  What are the dos and don’ts that we’ve learned thus far.  We’ll start with stories from each panelist, then go into Q&A.  All material will be made available later (more to come on that).

Background

Patrick

  • William Rainey Harper College:  NW suburb of Chicage, a 2-year institution. 40,000 full time equivalent students
  • “Project Discover” leader Matt McLaughlin.  We got a title 3 grant to help do this project.  Includes Inclusion, Engagement, Achievement, Onboarding, Intervening, etc.
  • Data has been collected over 6 years.
  • We originally used a proprietary data warehouse
  • Grad rate increase in 10% in 5 years
  • New reactive programs:  early alert, supplemental instruction, completion concierge, summer bridge.
  • These were REACTIVE programs, we wanted PROACTIVE solutions.

Vince

  • University of KY
  • What have we learned?  We’ve integrated virtually everything we can, and are now moving into personalized learning and messaging.
  • Respect complexity in learning analytics!  I recommend reading “Arrival of the Fittest,” a book by Andreas Wagner.  Their research on genomics highlights and models that can help our process.  Instructional complexity is at least as complex as that of genomics.  We don’t have just one paradigm of instructional theory, but dozens.
  • Structure is important:  get the right people on the bus, remove rivalries within your organization, give groups distinct and clear missions, align with organizational strategy.
  • Engage the community:  transparency makes a big difference; democratize analysis; enforce community etiquette, bring in students & faculty researchers; engage the broader higher education community.
  • Use the right tools and techniques:  speed enables fast thinking, fast group decision-making, fast everything; maximum semantic expressiveness and rich detail improves data quality, analytic flexibility; visualization is important.
  • Conclusion:  respect complexity, attend diligently to the very human aspects of this puzzle, ignite the passion of the community, choose and use your tools wisely

Scott

  • I represent the Independent Colleges of IN
  • A statute required student record information needed to be shared back with the state
  • I needed to know how our institutions compared to others
  • We worked with vendor partners (Dell & Statistica) to run descriptive and predictive analytics against the data we had
  • We wanted to do card swipes, meal plans, and more for sub-group comparisons.

John

  • The Statistica product has been made free for higher ed faculty and students
  • I run the Statistica group at Dell
  • We’ve done a lot of work in universities and hospitals
  • We’re moving toward using data for real-time decision-making.  A specific example was given about reduction in surgical infections…pretty powerful stuff.

“Maslow’s Hierarchy of Data Management”

  • The spectrum:  Data Management > Business Intelligence > Analytics
  • The specific levels:  Data Foundation > Basic Reporting > Performance Mgmt > Predictive > Prescriptive

Challenges and Observations

  • Master organizational and technical planning, orchestrating organizational adoption.
  • Bringing in the “executive management hammer” can be useful
  • IR, advisor and counselor pushback, i.e. “you’re coming to take our jobs!”  Dashboards and forms are actually a value-add for these folks that let them do their jobs more effectively.
  • Usability testing and adoption feedback from students were interesting:  “Why do you give us a number?  Why don’t you just give us feedback and actions we can take?”
  • ROL (“Return On Learning”), how can we quantify what you’re seeing?  There is no control group!  Profound payoff is that you’re able to make informed changes to policies that have real impact.
  • Student subgroups with a GPA lower than X (not specified) were much more likely to stop out.  This challenged many people’s beliefs, i.e. “how is this even possible?”
  • University of Iowa cited an avoided cost of $31 million

Next Steps

  • Data sharing with school districts for a full life-cycle on our students as they go through our system
  • Classroom on realtime analytics, such as triggers set by faculty
  • Get a handle on what our students do when they leave, i.e. wage data
  • Improving the advising process
  • Sharing findings with our institutions
%d bloggers like this: