Presenters
- Chris Kurtz, System Architect, Arizona State University
About Us
- First Google Apps for Education customer
- Multiple campuses with a diverse IT infrastructure
- Large # of governing reqs: FERPA, HIPAA, DARPA, DoJ, NASA, JPL
- Splunk is an Enterprise-level product, with easy access to all departments inside the University Technology Office (ISO/InfoSec, Ops, Dev, BA/BI, Accounting, Netcom, etc.). We wanted everyone to have equal access
The Power of Splunk
- Is ASU’s universal aggregator of all machine generated logs
- Typical response time to incident without Splunk: multiple days.
- With Splunk, we have direct, immediate access…minutes!
Splunk and ASU
- Had it for 4 years now.
- It needs a lot of power to run properly
- Use enterprise search head clustering and security
- Licensing 1TB/day
- Growth slowing down because we’re learning to better filter data
- Admissions and payroll are beginning to use it
We Didn’t Know!
“It was like the invention of the microscope: we didn’t know what we couldn’t see” – Martin Idaszak, Security Architect, ASU
Use Case: Protecting Direct Deposit
- Changing EE info online is great, but a target for hackers
- ASU has international students, faculty and staff, just blocking other countries isn’ accessible
- Before Splunk: whenever an EE was missing a direct deposit check, the investigation would take days, during which time it would sit between HR and Payroll systems. We were hand-protecting only a handful of people’s paychecks.
- With Splunk, we check geo tag info, do an affiliate lookup, and put it into an unusual changes report which payroll checks.
- Payroll will not run the payroll job WITHOUT this report now.
- This is the most valuable data I have in Splunk, by far.
- Where do you change your direct deposit from? Home and work. We take advantage of the “user’s center of gravity” to make a determination if the request is unusual.
- False positives? YES. False responses? NO.
Use Case: Phishing as a Teaching Tool
- We have 100K users. In 2015, we received 1 billion email messages, more than 750 million were spam and phishing.
- We have students from all across the world, transient by nature, can’t assume traffic from Nigeria, China or Malaysia are hacking attempts. In fact, it’s probably legitimate!
- Some Indian students were forced by their parents to give them their login credentials, which resulted in some interesting traffic and double-logins from completely different areas! We ended up setting up special limited accounts for these parents.
- Do NOT store user emails in Splunk, only the headers that transit our system.
- “This is the best tool we’ve seen in 10 years” – Jay Steed, AVP for UTO Operations, ASU
Leveraging Your Custom Data
- It’s limited if you’re only reading logs.
- If you don’t understand context of your data sources, you won’t get as much as you can get out of the product.
- No schemas! No types! Eval is your friend.
- Combine all data types in any way you want, on the fly.
- “Think of it like a database where time is the primary key”
- Don’t limit the power of Splunk!
- Start using the Common Information Model now!
- Not formatting data limits its value. Pull in secondary/ancillary data that makes sense of data in your logs. Makes the field extractions more valuable.
- For ASU, the master datasource is the Data Warehouse. Affiliate ID is the unique ID.
- Isolated Splunk server running Splunk DB Connect (DBXv2) runs SQL queries on several databases, and writes a series of lookup tables (with the Affiliate ID) every 4 hours. Linux ionotify monitors the lookup tables, and on write-close copies data to production systems (sanity checking applies).
Conclusion
- Heavily invested in Splunk because it solves many of our outstanding problems.
- 1st round of data onboarding concentrated on needs of ISO office
- 2nd round focused on operations needs, with some interesting use cases thrown in as they appear
- 3rd round is expanding Splunk usage and bringing it to the enterprise
- Splunk’s savings in man hours, extreme flexibility, use to validate other systems, and goals to replace antiquated systems has very much paid off
- Get your data into Splunk!
- Modify it later.
- Use the people who “get it” as evangelists
- Don’t get caught up on “use cases.” Once you have the data in Splunk, use cases present themselves repeatedly. Think of it as use case on demand.