Using Splunk Data Analytics to Protect Students, Faculty, and the University

Presenters

  • Chris Kurtz, System Architect, Arizona State University

About Us

  • First Google Apps for Education customer
  • Multiple campuses with a diverse IT infrastructure
  • Large # of governing reqs: FERPA, HIPAA, DARPA, DoJ, NASA, JPL
  • Splunk is an Enterprise-level product, with easy access to all departments inside the University Technology Office (ISO/InfoSec, Ops, Dev, BA/BI, Accounting, Netcom, etc.). We wanted everyone to have equal access

The Power of Splunk

  • Is ASU’s universal aggregator of all machine generated logs
  • Typical response time to incident without Splunk: multiple days.
  • With Splunk, we have direct, immediate access…minutes!

Splunk and ASU

  • Had it for 4 years now.
  • It needs a lot of power to run properly
  • Use enterprise search head clustering and security
  • Licensing 1TB/day
  • Growth slowing down because we’re learning to better filter data
  • Admissions and payroll are beginning to use it

We Didn’t Know!

“It was like the invention of the microscope: we didn’t know what we couldn’t see” – Martin Idaszak, Security Architect, ASU

Use Case: Protecting Direct Deposit

  • Changing EE info online is great, but a target for hackers
  • ASU has international students, faculty and staff, just blocking other countries isn’ accessible
  • Before Splunk: whenever an EE was missing a direct deposit check, the investigation would take days, during which time it would sit between HR and Payroll systems. We were hand-protecting only a handful of people’s paychecks.
  • With Splunk, we check geo tag info, do an affiliate lookup, and put it into an unusual changes report which payroll checks.
  • Payroll will not run the payroll job WITHOUT this report now.
  • This is the most valuable data I have in Splunk, by far.
  • Where do you change your direct deposit from? Home and work. We take advantage of the “user’s center of gravity” to make a determination if the request is unusual.
  • False positives? YES. False responses? NO.

Use Case: Phishing as a Teaching Tool

  • We have 100K users. In 2015, we received 1 billion email messages, more than 750 million were spam and phishing.
  • We have students from all across the world, transient by nature, can’t assume traffic from Nigeria, China or Malaysia are hacking attempts. In fact, it’s probably legitimate!
  • Some Indian students were forced by their parents to give them their login credentials, which resulted in some interesting traffic and double-logins from completely different areas! We ended up setting up special limited accounts for these parents.
  • Do NOT store user emails in Splunk, only the headers that transit our system.
  • “This is the best tool we’ve seen in 10 years” – Jay Steed, AVP for UTO Operations, ASU

Leveraging Your Custom Data

  • It’s limited if you’re only reading logs.
  • If you don’t understand context of your data sources, you won’t get as much as you can get out of the product.
  • No schemas! No types! Eval is your friend.
  • Combine all data types in any way you want, on the fly.
  • “Think of it like a database where time is the primary key”
  • Don’t limit the power of Splunk!
  • Start using the Common Information Model now!
  • Not formatting data limits its value. Pull in secondary/ancillary data that makes sense of data in your logs. Makes the field extractions more valuable.
  • For ASU, the master datasource is the Data Warehouse. Affiliate ID is the unique ID.
  • Isolated Splunk server running Splunk DB Connect (DBXv2) runs SQL queries on several databases, and writes a series of lookup tables (with the Affiliate ID) every 4 hours. Linux ionotify monitors the lookup tables, and on write-close copies data to production systems (sanity checking applies).

Conclusion

  • Heavily invested in Splunk because it solves many of our outstanding problems.
  • 1st round of data onboarding concentrated on needs of ISO office
  • 2nd round focused on operations needs, with some interesting use cases thrown in as they appear
  • 3rd round is expanding Splunk usage and bringing it to the enterprise
  • Splunk’s savings in man hours, extreme flexibility, use to validate other systems, and goals to replace antiquated systems has very much paid off
  • Get your data into Splunk!
  • Modify it later.
  • Use the people who “get it” as evangelists
  • Don’t get caught up on “use cases.” Once you have the data in Splunk, use cases present themselves repeatedly. Think of it as use case on demand.